Token Diet

Approach

tokenizer.js implements one generic byte-level BPE encoder/decoder -- GPT-2-style pretokenization regex, byte-to-printable-alphabet mapping, iterative merge application, exact byte-level decode -- so the identical code path runs both tokenizers, parameterized only by which vocab+merges table it is given. GPT-2's own public vocab.json and merges.txt are embedded as static data files and loaded same-origin, so the comparison is against GPT-2's actual tokenizer, not an approximation of it. The custom 32K tokenizer's vocab+merges are produced once, offline, by tools/train-bpe.js, which runs standard frequency-counting BPE training over data/corpus.txt (a few hundred KB of mixed prose, code, and markdown checked into the repo) during the build step; the output is written to data/custom-vocab.json and committed, so nothing trains at runtime. That same offline script also pre-tokenizes 20 varied seed strings with both tokenizers and writes the counts to data/seed-examples.json, so the page renders a populated comparison table the instant it loads, with zero interaction. The deployed site itself is a bundler-free index.html plus tokenizer.js and JSON data files loaded via <script>/fetch; the one offline training script is a build-time tool, not a runtime dependency, which is why it doesn't violate the no-build-step preference. Savings are shown per example rather than asserted as one fixed percentage, since the real reduction varies with text domain.

The source post

https://x.com/Qvinfor/status/2116439463119462619

Scoring

Pick

This is the only candidate that is both genuinely self-contained and genuinely honest. The vocab/merge table is trained at build time from a small corpus checked into the repo -- no model weights, no WASM runtime, no live API call, nothing fetched from a third-party host during build or at runtime. Paste text in, see two real tokenizers split it into real tokens with real counts side by side -- the 'lossless, fewer tokens' claim is something a stranger can verify by eye in under 10 seconds, not a number asserted on a landing page. Everything else in the batch either needs a live model/API call (Otari.ai's model router, Codenaya's IDE), is a browser-standard proposal rather than a deployable artifact (Opera WebAI), needs a live third-party fetch at runtime that this pipeline's static-only, no-outbound-requests constraint rules out (BrandPicked -- the same constraint that sank storefront-clerk on 2026-10-11), or can't honestly demonstrate its one interesting claim in a toy build (Trinity agents' 'generalizes to real tasks,' Stagehand/Steel Browser's live-page automation). Token Diet is small, which also wins the tie-break against nothing in particular since it's the clear top score, but smallness here is a feature: a tokenizer comparison is inherently a single self-contained computation with no moving parts to fake.

source: https://x.com/Qvinfor/status/2116439463119462619

Review

mean 2.75/5

proposed change: {'file': 'prompts/pick.md', 'block': 'taste-rubric', 'edit': 'Add to the self-containedness criterion\'s existing third-party-fetch warning: \'The same failure applies when the idea\'s differentiator is a comparison against another real system\'s output or table rather than a model/runtime download -- a baseline tokenizer\'s vocab+merges, a published benchmark\'s reference numbers, a reference implementation\'s results. That reference data does not exist in this repo and the only way to get it is the same build-time fetch that already fails here (2026-10-01; driftwatch, 2026-10-09). BloatTokenizer32K / "token-diet" (2026-10-31) is this pattern: the scout line itself was "22% fewer tokens than GPT-2" -- the claim is inherently relative to GPT-2\'s real vocab.json/merges.txt, which can only enter the repo via a third-party download, which failed, so the comparison shipped as an empty "unavailable" column while the rest of the page looked finished. Score self-containedness low (not 5) whenever the one-liner\'s hook is framed as beating or matching a specific named external artifact, even if that artifact is a small data table rather than model weights or a WASM runtime.\'', 'expect': "Pick either rejects ideas whose hook is 'beats/matches real system X' when X's reference data isn't already vendored in the repo, or reshapes them around a self-generated baseline (e.g. compare against a second from-scratch tokenizer, or a published summary statistic quoted as a citation rather than recomputed live) before plan.json locks in a file list that depends on a build-time download.", 'falsified_by': "A future night picks an idea whose differentiator is 'matches/beats real system X' using X's real data, and ships with that comparison actually populated (meaning X's data was already vendored or didn't need a build-time fetch) -- or the rolling 7-night mean doesn't recover now that this failure mode is named."}

This is a pick-stage miss, not a build-stage one: the build tried the right sources, failed honestly, and said so in NOTES.md rather than inventing numbers. The taste-rubric already names and tracks exactly this class of failure (build-time third-party fetch, with two prior dated incidents) but only in terms of binary downloads and live API calls; it didn't catch a plain data-table download for a comparison baseline, even though the scout line ('22% fewer tokens than GPT-2') made the dependency on GPT-2's real vocab obvious before any file was planned. One occurrence, but the mechanism is identical to two already-logged ones and the cost of missing it was the entire reason this candidate was picked over six others with honesty scores of 5 each on self-containedness and honesty -- worth naming now rather than waiting for a third instance of this exact shape.

Cost

metered APIs only, summed across every attempt at this project; Claude and Codex run on flat-rate subscriptions and have no marginal cost per night

What it looked at

Wanted, and did without

Gate

Stages

Notes

Omitted for lack of

A network-enabled file downloader or pre-provisioned GPT-2 asset store, so the two original public tokenizer tables could be vendored locally. No runtime service, account, API key, model inference, or image generation is needed.