Striped Hyena - LLM's Beyond Transformers

 Striped Hyena: A Novel Language Model Architecture Beyond Transformers

In the rapidly evolving landscape of language models, a new architecture has emerged that challenges the dominance of Transformers. Together Computer has introduced Striped Hyena, a innovative approach to language model design that promises improved efficiency and capabilities for handling long sequences.

Introduction to Striped Hyena

Striped Hyena represents a departure from traditional Transformer-based architectures. Like its namesake - a carnivorous mammal from the Hyaenidae family - this model joins the trend of naming language models after animals, following in the footsteps of models like LLaMA and Alpaca.

 Key Advantages

Enhanced Context Length Handling

One of Striped Hyena's most significant advantages is its ability to handle extended context lengths efficiently. The model can be fine-tuned for tasks requiring context lengths of up to 128K tokens - surpassing even Claude 2's 100K context window. Most impressively, it can be fine-tuned with more than twice as many tokens as Transformers for the same computational budget.

 Technical Architecture

The model incorporates several innovative components:

- Fast kernels for GED convolution (Flash FFT Conv)

- Memory-efficient autoregressive generation

- State Space Model (S4 SSM) layers

- A hybrid architecture combining elements of Transformers and convolutional neural networks

Model Variants

Together Computer has released two variants on Hugging Face:

1. Striped Hyena Base - The standard language model

2. Striped Hyena Chat (NOS) 7B - Optimized for chat-based applications

 Performance and Capabilities

During testing, the model demonstrated strong capabilities across various tasks:

- Basic arithmetic operations

- Code generation (including AWS Lambda functions)

- Multi-language translation

- Factual knowledge (with training data up to 2021)

- Appropriate content filtering and ethical responses

The model has shown competitive performance with other open-source Transformer models, particularly excelling in tasks involving longer contexts.

Technical Implementation

The architecture employs several advanced techniques:

- Model grafting techniques that allow architectural changes during training

- Training on a mix of Red Pajama dataset, augmented with longer context

- Compute-optimized scaling protocols

- Reduced memory footprint (50% reduction compared to standard Transformers)

- Integration of signal processing principles into language modeling

Future Developments

Together Computer has announced upcoming features including:

- Multimodal support

- Integration into retrieval pipelines

- Additional fine-tuning options

Trying Striped Hyena

The model is available for testing on Together AI's platform. Users can experiment with various inference parameters including:

- Temperature

- Top P

- Top K

- Repetition penalty

Conclusion

Striped Hyena represents an important step beyond traditional Transformer architectures, offering improved efficiency and capability for handling long sequences. Its hybrid architecture and innovative technical approaches suggest a promising direction for the future development of language models.

As the field continues to evolve beyond Transformer-based architectures, models like Striped Hyena demonstrate that there's room for novel approaches that can achieve competitive performance while bringing new advantages to the table.