Engineering · 2026-07-15
The third verdict
Most checkers answer pass or fail. A QC tool that only knows two answers has to guess whenever it can’t actually tell — and a guess wearing a green check is the most dangerous output a validator can produce.
Here is the failure mode that shaped the whole engine. You upload a scanned PDF with no text layer to an accessibility checker. It looks for tagged headings, finds none it can read, and prints a clean pass — not because the document is accessible, but because there was nothing the checker could parse, and its vocabulary only contained “pass” and “fail.” Absence of a detected problem got reported as absence of a problem. Those are not the same statement, and the gap between them is where compliance tools quietly lie.
Three states, not two
Every check in Preflight can return one of three things: pass, fail, or not evaluable. The third is a first-class citizen, not an error path. It means: the tool looked, and it cannot honestly determine the answer from what it was given. No text layer to assess. No keyline template declared, so page-sequence integrity can’t be judged. A Data Matrix that would need image decoding we haven’t shipped yet. An AFP resource the parser reached but the module doesn’t cover.
The rule is absolute: a check that cannot run is never allowed to read as a pass. If we can’t evaluate it, we say so, in those words, and we say why.
Why it’s harder than it sounds
Adding a third verdict is easy in one function. Keeping it honest across an entire pipeline is not, because “not evaluable” is the state most likely to get flattened at a boundary. A module crashes on a malformed stream; if the surfacing layer coerces that into an empty findings array, the public API cheerfully reports complete, zero findings — a false green built from a swallowed exception. We have found and fixed that exact class of bug more than once: a not-evaluable or errored signal dropped at the CLI, the webhook, the scorecard, or the JSON boundary, each one turning “we don’t know” back into “looks fine.”
So the verdict has to be carried, not recomputed: coverage travels alongside findings all the way out. The CLI renders it and exits non-zero. The webhook marks the result partial. The in-app index labels how many checks actually ran versus were not evaluable versus errored. The counts have to add up, or a test fails.
The cost, and why it’s worth paying
A tool that says “not evaluable” a lot looks less impressive than one that prints green everywhere. That’s the cost, and it’s real: “not evaluable” is a worse demo than a wall of check marks. But the buyer for document QC is someone whose statements, checks, or explanation-of-benefits mail is already going out the door. For them, a false pass isn’t a missed feature — it’s the incident. An honest unknown lets them go look; a dishonest pass tells them not to.
A validator that never fails anything is just a mirror. A validator that never admits it doesn’t know is worse: it’s a mirror that lies about what it reflects.