Understanding Mixture of Experts

Understanding Mixture of Experts: Beyond the Hype

The world of large language models (LLMs) is constantly evolving, and one of the most intriguing architectural innovations making waves is the concept of Mixture of Experts (MoE). While rumors have long swirled around whether models like GPT-4 use this approach, the reality of how MoE works is far more nuanced than many realize.

What is Mixture of Experts?

At its core, Mixture of Experts is a neural network architecture designed to solve a fundamental problem in large language models: computational efficiency. Traditional "dense" models like LLaMA 2 and LLaMA 3 activate every single parameter when processing a query. If you're working with a 70 billion parameter model, all 70 billion parameters need to process information for each token generated, requiring enormous computational resources.

MoE models take a different approach. Instead of activating all parameters, they use a "sparse" activation pattern where only a subset of the model's parameters are used for each query. This is where the "experts" come in – specialized sub-networks within the larger model that handle different aspects of processing.

The Reality vs. The Misconception

There's a common misconception about how MoE models work. Many people imagine eight distinct experts, each specializing in specific domains like mathematics, coding, or language translation. The reality is far more complex and interesting.

Take Mixtral, the well-known MoE model from French startup Mistral. While it's labeled as "8x7B" (suggesting eight 7-billion parameter experts), the architecture doesn't actually contain eight separate domain experts. Instead, each layer of the model has its MLP (Multi-Layer Perceptron) section divided into eight experts. This division is more akin to matrix factorization than explicit knowledge separation.

In Mixtral's case, the model uses a "top-2" selection mechanism, meaning that for each token, only two out of eight experts are activated. This results in roughly 25-30% of the model's parameters being active for any given query, providing significant computational savings while maintaining performance quality.

 The Connection to Model Merging

The relationship between MoE and model merging techniques becomes particularly interesting when we look at "sparse upcycling." This technique, implemented in tools like MergeKit, allows researchers to create MoE models by starting with multiple copies of a base model.

The process works like this: take eight identical copies of a model (say, a 7-billion parameter model), combine them into an MoE architecture, then perform extensive pre-training on trillions of tokens. During this training phase, each expert naturally develops different specializations, not through explicit programming but through the patterns in the training data.

However, there's an important caveat here: true sparse upcycling is reserved for organizations with enormous computational resources. We're talking about trillion-token training runs that are simply out of reach for most researchers and companies.

A Creative Merging Hack

One fascinating development in the MoE space is what experts call "the trench coat hack." When Mixtral was first released, some researchers realized they could take eight different fine-tuned Mistral models and combine them into a single MoE-style architecture. Instead of using the trained routing mechanism that determines which experts to activate, they approximated the routing based on the latent states of the model when processing specific types of prompts.

While this approach is "incredibly inefficient and quite hacky," as one expert puts it, it's remarkable that it works at all. Math questions can be routed to a math-specialized model, literature questions to a literature-focused model, and so on. However, this technique is more of an interesting proof-of-concept than a production-ready solution.

Looking Forward: Mixture of Agents

While the pure MoE approach has its limitations for smaller-scale implementations, there's growing excitement around "Mixture of Agents" approaches. Unlike MoE models where experts operate at the parameter level within a single model, Mixture of Agents systems actually use separate, complete models – each generating their own responses to queries.

This approach is more aligned with the intuitive understanding many people have of expert systems: discrete models that excel at specific tasks like coding, mathematics, or natural language processing. The outputs from these separate models are then combined to produce a final, hopefully superior answer.

The Economics of Efficiency

The appeal of both MoE and Mixture of Agents approaches ultimately comes down to economics. By activating only a fraction of available parameters or models, these systems can deliver high-quality results while reducing inference time by roughly 70% and cutting computational costs proportionally.

For organizations serving millions of queries daily, these efficiency gains translate to substantial cost savings and faster response times – critical factors in the competitive landscape of AI applications.

Conclusion

While the Mixture of Experts concept might not work exactly as many imagine – with neat, domain-specific expert divisions – the reality is arguably more sophisticated. These systems represent a crucial step toward making large language models more efficient and accessible.

As the field continues to evolve, we can expect to see more innovations in both MoE architectures and Mixture of Agents approaches. The key will be finding the right balance between computational efficiency, model performance, and practical implementation constraints.

For now, while the "GPU rich" organizations continue pushing the boundaries of what's possible with sparse upcycling and trillion-parameter training runs, the rest of the AI community can experiment with creative merging techniques and look forward to the promising developments in agent-based approaches that lie ahead.