phronesis · laserbrain
Every number here was wrong first
Each threshold in laserbrain was set by a judgement and corrected by a measurement. The corrections were larger than the final values and more useful to know about: a gate that blocked 74.3% of correct answers tells you more than one that blocks 6.7%.
The way we found out was the same every time — point the thing at something real, then read what it said instead of counting it. Every run found a defect no test had caught, and the tests were green throughout, because a suite written by the person who wrote the code confirms the convention they had in mind rather than the one the world uses.
what is measured now
The corrections
Each row names the file that fails the build if the value drifts back, so this page can be checked rather than believed.
What blocks a generated answer
was quotations and figures both block — a number cannot be paraphrased
74.3% of correctly-grounded human summaries were blocked. 67 of 74 blocking findings were figures.
now quotations only. Paraphrase was never the question; whether a claim could be grounded in a source the checker was not handed is.
pinned · scripts/test-rag-gate.ts
What counts as a quotation
was any double-quoted span of three words or more
three of six sampled blocks were titles — "The Devolution Project", "Se Va el Caimán". Works are set in quotation marks by convention.
now Title Case excluded. Blocking fell 13.9% → 6.7%.
pinned · scripts/test-anchor.ts
How an endpoint is read from documentation
was a method followed by a path — GET /v1/widgets
zero endpoint claims from ten published READMEs. Every fixture used that form; almost no real documentation does.
now absolute URLs and curl invocations too, with backslash continuations joined first.
pinned · scripts/test-docs-drift.ts
When a documented status has drifted
was any status that is not the documented one
four separate false-positive classes — a 403 error envelope, an unauthenticated prober, a required query parameter, a 5xx read as a wrong document.
now one rule: a status is checkable only when the request this prober can make is the request the documentation describes.
pinned · scripts/test-docs-drift.ts
What a figure is
was any run of digits, with an optional leading minus
2026-06-21 split into three figures — 2026, -06, -21. Two invented from every date on the page.
now a date is one figure and stays a figure; a hyphen between digits is a range, not a minus.
pinned · scripts/test-anchor.ts
What a name is
was a capitalised word not at the start of a sentence
I've, I'd and I'll read as proper names; "Status: Ready" joined across a colon; UNDEFINED read as an organisation.
now contractions of pronouns dropped, runs broken at punctuation, all-caps over six letters treated as emphasis. Names still only flag — capitalisation is not an entity model.
pinned · scripts/test-anchor.ts
When two sentences say the same thing
was a 0.5 overlap threshold, with a test file that was never wired into the build
the threshold could be moved anywhere in (0, 1] without failing. Every fixture had overlap exactly 1.0.
now fixtures either side, at 0.429 and 0.714, and an assertion that they straddle it.
pinned · scripts/test-clarity.ts
One finding that was not ours
Pointed at published OpenAPI specs from the APIs.guru directory, laserbrain docs found that Vercel’s spec declares /edge-config under https://api.vercel.com. That path answers 404. The route is live one segment up, at /v1/edge-config, which answers 403 — so the spec’s server and path do not compose to the real endpoint, and anyone following it literally gets a 404.
Two other findings from the same run were triaged and reported differently: a double slash in clever-cloud’s own path, which is real but cosmetic, and a 404 at redirection.io that is either a moved route or a private one hiding behind 404 — nothing observable from outside distinguishes them, so it is reported as something to read rather than a defect.
What it still cannot do
A faithful paraphrase scores badly. A fabrication assembled from the source’s own words scores perfectly. A real name attached to the wrong claim passes, and so does a verbatim quotation attributed to the wrong speaker — it asks whether the words are in the document, not whether that person said them.
Each of those is pinned as a passing test. None can be quietly fixed without the build failing until this page is rewritten, which is the only mechanism we have found that keeps a limitation stated after the person who found it moves on.
run it yourself
npx laserbrain-check try
No files, no key, no model. The evidence harnesses are in the repository as scripts/evidence-*.ts and are repeatable — every number on this page can be re-measured rather than taken.