7 MoE Expert Routing Strategies
1 · Token-Choice (Top-k) · 2 · Expert-Choice · 3 · Dynamic Routing (Top-p) · Each token scores every expert and picks its top-k · Each expert picks its top-c tokens (capacity c = 2) · Number of exper…
On this board
- 7 MoE Expert Routing Strategies
- 1 · Token-Choice (Top-k)
- 2 · Expert-Choice
- 3 · Dynamic Routing (Top-p)
- Each token scores every expert and picks its top-k
- Each expert picks its top-c tokens (capacity c = 2)
- Number of experts varies with router confidence
- Confident (p_max = 0.9) → 1 expert
- E1
- T1
- Router
- E1 (cap 2)
- top-k = 2 selected
- p = 0.52
- Router (gate)
- E2
- T2
- Σp = 0.9 ≥ 0.8
- picks T1, T2
- Uncertain (p ≈ 0.4 / 0.3 / 0.2) → 3 experts
- Perfectly balanced load, but tokens can be dropped or get several experts
- token
- softmax scores
- p = 0.31
- picked by 2 experts
- E3
- T3
- 0.4
- E2 (cap 2)
- p = 0.17 · skipped
- T4
- picks T2, T3
- needs 3 experts
- 0.3
- y = Σ pᵢ·Eᵢ(x) over the k selected experts ⚠ Prone to load imbalance → needs aux load-balancing loss Examples: Switch (k=1), Mixtral (k=2)
- dropped (not picked)
- 0.2
- Add experts until cumulative Σp ≥ threshold (e.g. 0.8)
- 4 · Balanced Assignment
- 5 · Hash-Based Routing
- 6 · Soft MoE
- expert = hash(token_id) mod N — no learned gate
- Global matching: each token → exactly 1 expert, each expert gets exactly N/E tokens
- Each slot = weighted average of ALL tokens (no discrete routing, no dropped tokens)
- Slot 1
- idx 0
- hash(token)
- Dispatch
- Pro: no router training, no routing collapse Con: ignores context / semantics
- load 2 / 2
- Σ wᵢ·Tᵢ
- Slot 2
- Solved with linear assignment (BASE layers) or Sinkhorn. Unlike Expert-Choice: no token is dropped or gets 2 experts
- token_id = 4822
- 4822 mod 3 = 1
- idx 1 · selected
- softmax over tokens
- Y1, Y2, Y3
- Combine
- idx 2
- Deterministic: same token → always same expert
- per-token output
- softmax over slots
- 7 · Group-Limited Routing
- Step 1: pick top-m groups → Step 2: pick top-k experts within those groups
- Only experts in the selected group (G2) are considered
- G1
- not selected (masked)
- Group Selector
- G2
- Top-k within G2
- E4
- Why limit groups? • Each group ≈ experts on one device / node • A token is sent to at most m devices → lower all-to-all communication cost • Group score = sum of its top expert scores; groups outside top-m are masked before top-k Examples: DeepSeek-V2 / V3 (device / node-limited routing)
- top-m groups (m = 1)
- selected
- routes over E3–E5 only
- E5
- skipped