Engine Room: Inside an LLM Inference Server
shippedAn interactive sandbox that animates the three tricks behind DeepSeek's open-sourced inference engine — KV-cache growth, speculative decoding's draft/verify dance, and MoE expert routing — so you can watch why serving a big model is fast instead of just reading about it.
Approach
Everything is one index.html with inline CSS and JS — no backend, no API calls, no weights, so a bundler buys nothing and a toolchain would just be more that can break at 4am. The core is a deterministic pseudo-random generator (seeded PRNG, no Math.random) that produces four preset prompt-and-response token sequences of 22-28 tokens each at page-load time; every token carries a precomputed KV-cache size, a speculative-decoding verdict (drafted/accepted/rejected), and a routed expert id (0-7, top-2 of 8). On load, the first sequence is already fully rendered into a token table, a canvas line chart of KV cache size vs. token index, and an 8-cell expert-utilization heatmap, so the visitor sees a completed run immediately. A Replay button re-runs the same sequence as a token-by-token animation (~120ms/token) so the mechanics are visible in motion, not just as a static table; a New Prompt button cycles to the next of the four preset sequences, re-populating all three views. All data is generated client-side from the fixed seeds baked into the file — nothing is fetched, so there's nothing to 404.
The source post
https://x.com/TenzyXzy/status/2113022190090371218
Scoring
Pick
| surprise | 4 |
|---|---|
| demonstrability | 4 |
| self_containedness | 5 |
| honesty | 4 |
DeepInfer is the only candidate whose interesting mechanics (KV cache, speculative decoding, MoE routing) can be made tangible with code we write ourselves — a token-by-token simulation that runs entirely client-side, no weights or API calls needed, and is honestly labeled as a simulation of the technique rather than a wrapper around the real engine. The other three candidates are model/bot releases whose only honest demo would require fetching multi-gigabyte weights or a live API call at build time, which this sandbox's network-less build stage cannot do, or (for OpenBot/OpenDots/OpenMuse) aren't a single web artifact at all. Nothing in history.json is close to this slug or concept.
Review
| shipped | 5 |
|---|---|
| honest | 4 |
| worth_it | 4 |
| efficient | 5 |
No change: the gate passed on the first attempt, NOTES.md reports no omissions and nothing it could not do, and the only anomaly found during review (a transient 522 immediately post-deploy) was Cloudflare edge propagation that cleared within about a minute on its own recheck — not a pattern in scout/pick/build/routing, and the last 30 nights' failures in history.json are unrelated one-off causes (wrangler Pages API errors, a plan refusal, one build gate failure) already self-repaired or correctly handled, none recurring three nights running.
Cost
| total | $0.1398 |
|---|---|
| xai | $0.1398 |
What it looked at
Ornith-1.5-122B** — MIT-licensed 122B MoE model strong at coding/agent tasks (84.5 SWE-bench), runnable locally via Ollama/Unsloth GGUFs.
OpenBot / OpenDots / OpenMuse** — fully open-sourced self-hostable AI coworker bots (6.7k–5.3k stars) that run on any surface with any agent harness.
DeepInfer** — DeepSeek’s open-sourced production inference engine (KV cache, MoE, speculative decoding) with visual demo video.
Ming-3.0** — open-sourced family of expressive TTS, audio-understanding, and ASR models (0.5B–126B) for voice agents.
Gate
| pass | project directory exists — /Users/artax/code/builds/2026-10-22/project |
|---|---|
| pass | no build step needed — static project |
| pass | build output with index.html — /Users/artax/code/builds/2026-10-22/project |
| pass | index.html is a document — 15120 bytes |
| pass | local asset references resolve |
| pass | page loads without console errors |
Stages
| scout | grok · ok · 18.2s |
|---|---|
| pick | claude · ok · 67.8s |
| plan | claude · ok · 44.8s |
| build | codex · ok · 194.6s |
| gate | local · ok · 2.8s |
| publish | local · ok · 11.7s |
| review | claude · ok · 130.3s |
Notes
Omitted for lack of
None. This simulation requires no external service.