mixture of experts

An expert is just a feed-forward network and come in two forms: shared and routed.

Shared experts get every token.

Routed experts get only a fraction of the tokens.

Routed experts receive tokens based on a gating/routing mechanism.