← Back

MoM: Mixture of MoEs

Mom (Carol Miller) from Futurama, owner of MomCorp.

A mixture of experts gives a model several networks to choose from inside each expert layer. Each token uses only a few. In Mixtral, the router picks two of eight. That saves calculation, but all those experts still belong to one model.

I want to take that idea one level up. MoM, a Mixture of MoEs, would ask models from different families to work on the same request. They could challenge each other's answers, and the system would keep the reasoning and checks behind its choice.

The obvious problem is that several models can agree and still be wrong. Adding more votes does not help when everybody makes the same mistake. Here is a small example you can break yourself.

The choice of checks matters as much as the choice of models. A test that catches nothing gives you a more elaborate way to accept the wrong answer. MoM would need to preserve the failed proposals too, so we could see why one answer survived and use that comparison to train a model later.

A MoM could also ask another MoM for help. Each request would cap the number of recorded debate steps, including steps inside nested groups. It would need to know which models are already loaded. Waiting for another model can cost more than its answer is worth. The design still has to settle when another debate helps and how much of an answer to compare at once.

This is a proposal, not a working result. The Doppler runtime and its recording contract are related work; they do not establish that the full MoM system works. Happy Mother's Day.

Where the proposal comes from

RouteLLM studies choosing models; multi-agent debate studies comparing their reasoning. The proposed training step draws on knowledge distillation. Mixture of Agents combines outputs across layers, while Tree of Thoughts explores intermediate reasoning steps.

Mamba and Jamba change how a model processes sequences; choosing between model families is a different question. Geoffrey Hinton's AI-as-mother proposal is another use of the maternal analogy, about care rather than routing.

Sources

  1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
  3. Mixtral of Experts
  4. Mixture-of-Agents Enhances Large Language Model Capabilities
  5. Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  6. Jamba: A Hybrid Transformer-Mamba Language Model
  7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  8. RouteLLM: Learning to Route LLMs with Preference Data
  9. Improving Factuality and Reasoning in Language Models through Multiagent Debate
  10. Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks
  11. CNN on Geoffrey Hinton's AI mothers proposal