284B model on one graphics card, because most of it is asleep

A mixture-of-experts model only wakes a fraction of itself per token. Six experts out of 256, in each of 43 layers. So the question stops being how to fit 96 GiB into 32 GiB of VRAM and becomes how fast you can read the missing pieces off a drive. Crow keeps the always-active part on the card, holds the 64 most useful experts per layer beside it, and streams whatever is missing while the GPU works. Peak host memory is 1.28 GiB. Decode is around 12 tok/s at a 200k context. The interesting part…