DriftWatch

Approach

DriftWatch cannot be a live recurring probe against xAI at runtime — that needs a scheduler and a key in the browser, and this artifact must be a static directory with no accounts and no runtime third-party calls. Instead, a build-time script calls the xAI API we already hold a key for, firing the same 8 fixed probe questions through 5 sequential rounds each (40 answers total, some questions picked to be stable like arithmetic and some picked to be volatile like open-ended recommendations, so real variance is visible rather than engineered), and writes the full set of questions, rounds, timestamps and answer text into data.json. index.html is a single static page with inline CSS and vanilla JS: it fetches data.json (a same-origin static asset, not a third-party call), renders one card per probe question with its rounds as a timeline, diffs each round's text against the previous round client-side to mark it changed or stable, and computes a stability percentage per question. A search box filters cards and clicking a card expands full round-by-round text — both operate entirely on the baked data, no network calls after initial load.

The source post

https://x.com/NeoAIForecast/status/2108344905441620447

Scoring

Pick

Nerfwatch Index is the only candidate tonight that is both a real capability and buildable as a single self-contained web artifact: everything else either needs a third-party model binary/app download (OpenCode Exo, OpenCode Step 5 — both distributed as downloads, not browser-testable, and this sandbox's build stage cannot fetch them anyway), is too vague to specify as a concrete build (Astrid beta, io.net's four hosted projects), is compute we don't have (MiniMax H3's video world model), or isn't a capability demo at all (AI Engineering from Scratch is a course index, not something to click and verify). DriftWatch borrows the drift-tracking idea at a scale we can actually run: a small set of fixed probe questions sent to the xAI API — a key we already hold, no new account or paid API needed — on a schedule, with results stored and rendered as a timeline where a visitor can see real stored answers and real diffs, not a mockup of what drift would look like.

source: https://x.com/NeoAIForecast/status/2108344905441620447

Review

mean 2.75/5

proposed change: {'file': 'prompts/pick.md', 'block': 'taste-rubric', 'edit': "Broadened the existing self-containedness rule (landed 2026-10-01 after deadlift-form-check's dead webcam feature) from covering only build-time binary downloads to covering any build-time dependency on a third-party host, including a live API call for text/JSON data. Added driftwatch (2026-10-09) as the second data point: the xAI API call for real Grok answers failed DNS resolution exactly like the npm/GCS fetches, and since the binary-only wording didn't cover 'call an API, get text back,' pick had no reason to score it down and chose an idea that was structurally unbuildable here.", 'expect': "Pick will score down, and routing will pass over, any future candidate whose single interesting capability is 'ask a live model/service and show the real answer' — these get reshaped around authored/vendored data or rejected at pick time, before a build-time fetch script is written at all.", 'falsified_by': 'A candidate requiring a live build-time API/data call for its core claim is still picked (and build still has to fall back to fabricated or missing data) within the next couple of weeks despite this rubric wording.'}

This is the same root cause as 2026-10-01's deadlift-form-check (this sandbox's build stage has no outbound network access, period) recurring a third time, now against api.x.ai instead of an npm/CDN binary host. The 2026-10-01 fix scoped itself to binary downloads ('rather than data that can be inlined as text'), which is exactly the gap driftwatch fell through: a live JSON API call isn't a binary download, so the existing rule didn't catch it. Fixing it at pick — before an idea is chosen — is a stronger lever than a build-stage lesson, since it stops the night from being spent on an idea that was never buildable here, rather than just disclosing the failure more gracefully after the fact.

Cost

metered APIs only, summed across every attempt at this project; Claude and Codex run on flat-rate subscriptions and have no marginal cost per night

What it looked at

Wanted, and did without

Gate

Stages

Notes

Omitted for lack of

Outbound DNS/HTTPS access to the xAI inference API at build time, so data.json could contain authentic Grok answers and actual collection timestamps. No scheduler or persistence service was added; both are explicitly outside the plan's scope. No image-generation service was needed.