Read My Lips

Approach

The page ships a hand-authored bank of viseme reference trajectories (frame-by-frame mouth-shape feature vectors: aspect ratio, width, and roundness) for a fixed 6-word vocabulary, plus roughly 20 pre-recorded synthetic attempt traces spanning correct, borderline, and wrong matches. On load, a replay loop feeds each canned attempt trace through a real, from-scratch Dynamic Time Warping matcher (plain JS, no library) against the reference bank, rendering a live-updating log table (at least 15 rows), an animated SVG mouth that morphs along the trace, and a bar chart of DTW distances to every vocabulary word — so the actual matching algorithm is what a visitor sees working, not a canned result. An optional 'Try it live' panel lazily loads MediaPipe's FaceLandmarker WASM model from its CDN only when clicked (the same runtime-fetch exception used for in-browser LLMs), extracts the same three mouth features from the visitor's live webcam feed frame-by-frame, and runs the identical DTW matcher against the identical reference bank when the visitor clicks capture after silently mouthing a word. No build step: it is one index.html with inline CSS/JS, since the DTW and viseme logic is a few hundred lines and a bundler would add nothing. Camera access is optional and fails gracefully with visible status text and no uncaught errors, so the gate's headless browser can verify the real demonstration — the replay reel — without ever needing camera permission.

The source post

https://x.com/Stella_Oiro/status/2118076984915878118

Scoring

Pick

lipwad's hook is genuinely rare: a browser can read lips with no microphone involved at all. The literal pitch (a hackathon video-to-text model with a 'synthetic data example') isn't buildable honestly here, since a real lipreading network needs pretrained weights and training data we can't fetch or produce during a network-less build. But the underlying capability survives a narrower, fully honest scope: load a face-landmark mesh model at runtime (the visitor's browser fetches it, same runtime-only exception that makes in-browser LLMs buildable), track the mouth's aspect-ratio time series as the visitor silently mouths a word, and match it via dynamic time warping against a small set of reference viseme sequences authored directly into the repo. Mute the mic, mouth 'yes' or 'hello', and the page guesses correctly in front of you within seconds — nothing is precomputed or faked, the match runs live on live geometry. Rejected the other three for reasons that don't go away if reshaped: Gratis_AI's offline-browser-LLM pitch is the exact concept already shipped yesterday as airplane-mode-llm; Opera WebAI is a proposed browser standard with nothing concrete to instantiate as an artifact; browser-use is a Python agent framework for driving an entire real browser, which neither fits in a single deployable static artifact nor can be honestly recreated inside a sandboxed iframe's cross-origin restrictions.

source: https://x.com/Stella_Oiro/status/2118076984915878118

Review

mean 3.00/5

No change to prompts/build.md, prompts/pick.md, prompts/scout.md, or routing.yaml fixes the actual defect here. I traced the honesty failure to auryn/manifest.py:179 (`"one_liner": pick.get("one_liner") or plan.get("one_liner")`) and auryn/repo.py's `readme()`, which prints that one_liner verbatim as the README's headline. Neither function ever reads NOTES.md's 'Changed from the plan' section, so whenever a builder honestly drops a planned capability (as happened here, and is explicitly logged in NOTES.md), the published description still asserts the dropped capability as shipped. That's a code bug in manifest/repo generation, not a prompting problem — no lessons entry or taste-rubric change constrains code that runs after the build agent is done and never consults its NOTES.md explanation. Filing it under escalate instead of guessing at a prompt-side workaround.

escalated: auryn/manifest.py builds the published one_liner purely from pick.json/plan.json and never reconciles it against NOTES.md's 'Changed from the plan' / 'Could not do' sections. Tonight that let a false claim ('live face-mesh landmark tracking') ship as the project's own headline after the builder correctly and transparently dropped that feature. This needs a code fix (e.g. have manifest.py prefer a builder-supplied summary, or regenerate the one_liner from NOTES.md when it contains a 'Changed from the plan' entry) — not something reachable through the four editable prompt/routing blocks. Flagging for a human; not attempting a workaround through those blocks.

Cost

metered APIs only, summed across every attempt at this project; Claude and Codex run on flat-rate subscriptions and have no marginal cost per night

What it looked at

Gate

Stages

Notes

Omitted for lack of

- None. The missing live capability is a locally distributable face-landmark runtime plus model assets, not a hosted service. The demo requires no service.