Trust & evals

How every number stays honest, and what has been measured.

Every number is checked at three points before it is shown: the facts, the findings built from them, and the sentences written from those. Anything that fails is left out, never published. Below: the checks, then the scorecard, then what isn't measured yet.

1 · Facts

Code computes every number

The run computes facts from the dataset's rows. Each fact is a number with what it measures, who it covers, its period and how it was worked out. The Story and the answers never compute a number of their own.

2 · Findings

The AI chooses, code checks

For each question, the AI picks the findings a careful analyst would tell a business reader, by pointing at facts; it writes no numbers. Code then checks every finding: one measure, one population and one subject; a change cites a change, a share cites a share; a top list includes every member tied with the last one named; a trend needs six or more periods and two full years; a lift's figures are cited only by a finding about the lift. A finding that fails is dropped; one that links others is dropped with them.

3 · Writing

The AI writes, code checks every sentence

Every sentence names the findings it rests on, and must pass: every number equals a value of its findings, at the precision written; each direction word ("rose", "fell") agrees with the sign of its number; every condition on its findings is said in that sentence or the next; it uses no word the evidence can't support, from a published list; it names at most four periods' values. A draft with problems gets one repair attempt. A sentence that still fails is left out. Where the evidence doesn't settle something, the answer says "Not settled by the evidence", and the Story lists it under What is open.

The Checks tab of the Details panel for one chart: passed 9 checks when this page loaded, 01:44, with each check listed — **When results are read.** The Story's stored results are verified against their fingerprints on every load, and every number shown on a results page is matched to its fact when that page's read is built. The wording checks ran when the text was written and are not re-run. The Details panel says "Passed N checks when this page loaded" for each item. Synthetic sample data.

The Details panel for the whole Story page: 105 findings shown on this page (81 in the text, 24 in charts and figures), all matched their facts when it loaded; 14,672 facts → 156 findings → 137 passed every check → 81 cited here — **For the whole page.** Every finding shown on the page matched its facts when it loaded. Underneath, the run's own count: facts, findings, those that passed, those cited. Synthetic sample data.

The scorecard

What has been measured on the October 2026 release, and the result. All of it was run on synthetic data: planted datasets generated with known answers, and two sample files. Nothing here was measured on a client's real data; that is part of what a pilot is for.

Are the numbers right? (no model involved)

Computed cases that match an independent reference61 of 6169 cases in all: 61 computed and matched exactly, at 1 and 4 threads; 6 are requests the engine must refuse, and it does; 2 aren't run (one unsupported, one skipped).

Lift figures against a reference in exact fractionsAll matchEvery lift figure, band and check status, including 200 random cases.

Planted facts found in 11 test datasets120 of 120Datasets built with known answers, two of them tests against a control.

Do the checks catch the AI?

Findings the AI proposed that the checks stopped97 of 1,5516.3%, across 22 analyses on a development machine, 21 to 29 Oct. None reached a page.

Forbidden findings that got through1 of 29Claims or wording the analysis must not make, on 7 planted datasets, with recorded AI answers replayed through the release.

Planted findings the analysis stated67 of 74The same replay.

Does it read the request right?

The Brief's own checks, on 24 test requests100%With the default model, about 33 seconds a Brief (9 Oct).

How long does it take?

Time per analysisMedian 5.8 min90% within 10.5 minutes, across the same 22 runs, on a development machine.

All measured on the October 2026 release or the runs leading to it; rows measured on the release candidate were run again after the cut with identical results. The full scorecard, with each row's date, code and how to rerun it, is available on request.

Not measured yet. Every result above is on synthetic data; a pilot is the first test on a client's real data. Human judgment of the writing rests on a small hand-graded set, and how often every number matched its fact when read isn't recorded as a statistic.

Measure it on your data.A pilot is where Evident meets real data for the first time. Talk to Connor, who built it.

Book a demo Under the hood