Agent harness
How. A fixed pipeline of stages. Each model call is boxed between code that prepares its input and code that checks its output: Definitions, Brief, the Plan's Questions, Run, Claim ledger, Writing, Results, Editions, Ask. Durable command tasks in PostgreSQL are run by a worker; a task id is the run's effect-once key, so a retry resumes. No model memory; state is stored data. The model calls no tools; code calls the model.
Why. The output is an analysis someone acts on, so every number must be reproducible and checkable. Splitting the harness into stages gives each model call one narrow job with a contract, and puts a code check after each one.
Limits. The pipeline is fixed, so a request outside the catalog is refused rather than improvised. Stage outcomes are stored but not traced end to end in one view.
Workflow, not agent
How. Evident is a workflow on purpose. The order of stages, what each model call may do, and when the run ends are fixed in code. No model decides what runs next. The agent-like parts are small: the Brief's reply turns (a person drives each), one repair call in writing and editions, an edition re-plan (one call, one repair), and Ask (one retrieval, one call, code checks).
Why. An analysis must be reproducible, auditable and bounded in cost and time. An agent loop chooses different steps on different runs, can call a tool the designer didn't foresee, and is hard to test with fixed answer sets. A workflow can be replayed with recorded model answers and scored the same way every time.
Limits. A question the catalog doesn't cover is refused, where an agent might find a way. A bounded agent could fit later in two places, inside guards: Ask calling a read-only query tool, and a "dig deeper" step that proposes extra calculations for a person to approve.
Structured outputs everywhere, tool calling nowhere
How. Every model reply is a typed object; code runs every query and computes every number. One provider for every call; a reply that doesn't parse is an error. Claim selection cites cells by address and is told to write no numbers, dates or ids. Writing cites claim ids. Ask cites source ids.
Why. A model that can call a query tool can also compute, filter or round a number its own way. Pointing at precomputed cells means every figure can be traced to one certified fact, and the compiler can reject a claim on a rule without reading prose. Schema proves shape; code then checks meaning. One repair call; a reply cut off by its token limit is retried once with twice the room (up to 32,000 tokens).
Limits. The model can't ask for a calculation the plan didn't make; a missing number becomes "Not settled by the evidence". A read-only query tool for Ask is a candidate.
Retrieval
How. Two kinds. For Ask: BM25 over the 77 Help pages, with a stemmer, stop-words and a synonym list, the current page's entry weighted up, at most six pages given to the model. One structured call; each sentence cites source ids; code removes failing sentences. For the analysis: exact retrieval of certified facts into an evidence card (at most 40 members and 400 rows) per Question.
Why. The Help corpus is small, controlled and written in the product's words, so lexical search with a synonym list finds the right page and is deterministic and debuggable. For the analysis, similarity search would be wrong: the right facts are known from the plan, so retrieval is a lookup, not a guess.
Limits. Measured on 29 Oct, no model: every required page in the top six for 39 of 39 questions, all in the top three for 36 of 39. The synonyms were tuned on these questions, so this is closer to a training score. Ask repeats a wrong Help page faithfully; its checks test shape, not truth. The live answer pass is not yet measured.
Embeddings and hybrid search: not used, by decision
How. BM25 with a synonym list instead. The corpus is 77 controlled pages in the product's own words; there is no embedding call, vector store, migration or second provider; results are deterministic.
If that stops holding, the plan is hybrid search in PostgreSQL (vectors beside BM25), reciprocal rank fusion, an optional cross-encoder over the top 20, and the same 40-question harness plus a held-out set first. The comparison is scripted and not yet run.
Guardrails on the way in
How. Every dataset or user text sits inside data markers that the system prompts say can never give instructions. Values are cleaned (control characters, forged markers, instruction-like text replaced with "[text removed]") and capped: a request at 4,000 characters, a Question at 600, a label at 120. Personal columns send no values; personal member names become stand-ins; breakdowns over 50 members go as counts; a setup sample of up to 10 rows is a workspace choice; the model never sees rows otherwise.
Why. Data and user text are untrusted inputs to a model just as they are to SQL. Marking, cleaning and capping at the prompt boundary only means stored data and pages keep their real words while the model can't be steered by a cell value. The user's own words are capped but not stripped, because "only report Q4" is an instruction to the analysis.
Limits. Instruction stripping is pattern matching and can be evaded by phrasing it doesn't know; structured output and the output checks are the backstop. Non-personal member names of small breakdowns still reach the Brief, by decision.
Guardrails on the way out
How. Six code check points; anything failing is repaired once or dropped, never shown. Claims are compiled against the evidence card and kind rules. Every sentence is checked: citation, number, direction, condition, forbidden terms, list and span rules. Deck slots are checked. Ask's answer is checked. Stored outputs are re-hashed and every value matched on read. Check recomputes totals from rows, with integrity checks on labels, agreement, sums, shares and lift sums.
Why. A second model judging the first can be wrong in the same way. Rules written in code are testable, give the same answer every time, and say exactly why something was stopped. Code writes the sentences the model must not: "Not settled by the evidence", "Not answered", why a comparison was cut, a lift's checks and limits.
Limits. Measured: across 22 analyses, 97 of 1,551 proposed findings dropped (6.3%); on replayed planted sets, 8 of 214 claims rejected and 5 sentences dropped. Checks match words and values, not meaning; the number check tolerates rounding at written precision. A removed sentence can leave a thin chapter. Read-time outcomes aren't recorded.
Grounding: AI never writes a figure
How. Code computes facts from rows through the query layer (DuckDB over Parquet), each with measure, subject, period, method and lineage. The model points at cell addresses; the compiler builds claim sheets; the writer cites claim ids; the number check requires every number to equal a cited claim's value. On read, every chart, tile and claim value is matched to its fact and outputs are re-hashed. Check recomputes totals from rows.
Why. Asking a model to be accurate isn't a control. Making it structurally unable to introduce a number, and checking each hand-off in code, is. Measured: all 61 computed known-answer cases match an independent reference (of the other 8, 6 are correct refusals and 2 aren't run); the planted engine tier recovers 120 of 120 facts; on replayed planted analyses, 1 of 29 forbidden findings appeared.
Limits. The checks prove prose matches claims and claims match facts, not that facts match the data; that rests on the known-answer kit, planted datasets and Check. A correct number can sit in a misleading sentence. Cached reads (up to 8 entries) aren't re-verified. Ask can repeat a figure a Help page states.
Context engineering
How. Code assembles each call's context with fixed budgets and stated cuts. The Brief gets rules, the request, metric and breakdown names, categories of breakdowns of 50 or fewer, and dates; no rows or figures. Claim selection, one call per Question, gets a card of about 400 rows, 40 members per table, whole series up to 36 months, 16 quarters or 60 weeks. Ask gets at most six pages.
Why. The 30 September baseline found 5 answers using another subject's numbers. Citing cells by address removes that by construction, and the card's budget bounds what each call carries. Trimmed tables say what was cut; reading lines are written by code; one Question per call keeps failures local.
Limits. Members past 40 in a table can't be cited even when relevant; a daily series is never shown whole; limitations are judged per Question, so the Story can say something is "not supplied" that another Question has.
Evals, offline
How. A layered suite graded by machine against answers known by construction, in three tiers so most runs need no model. Query-layer known answers: 61 of 61 computed cases match (69 in all; 6 correct refusals, 2 not run). Lift reference: all pass, plus 200 random cases. Planted datasets: 11 sets; the engine recovers 120 of 120 facts; a replay of recorded answers scores 67 of 74 truths and 1 of 29 anti-facts broken; the live run of 22 October scored 62 of 73 and 6 of 29. Brief set: 24 requests. Lift and per-thing set-ups: 22 of 22, 40 of 40. Ask retrieval: 39 of 39.
Why. Planted data makes every expected answer right by construction, and the tiers make most measurement free, so it runs on every package. Live runs record every model answer; replay feeds them back in minutes at no cost. Evals drive the work: the 22 October run found an unanswered price-volume-mix question; the fix shipped the same day.
Limits. All data is synthetic; four sets lack live recordings; the live lanes for the lift and per-thing sets, and Ask's live pass, aren't measured; the machine grading was corrected after the 22 October live run, so scores before and after aren't comparable. The 30 September baseline was hand-graded (7 of 15 stated, 6 of 12 broken); hand and machine grading were never compared on the same run.
Evals, online and at runtime
How. Every run measures itself, and every read checks again, but only the first is recorded. Each run stores and shows its funnel: facts computed, claims proposed, checked, cited, shown. Across 22 analyses: 1,551 proposed, 1,454 passed, 97 dropped (6.3%), 955 cited. The sample run on the home page: 14,672 facts → 156 findings → 137 passed → 19 dropped → 81 cited. Reads re-hash outputs and match every value; failures are withheld with a reason code.
Why. The run-level counts were built for the reader ("19 findings failed a check and were dropped"), not for monitoring; the read-time checks were built to protect the page.
Limits. Read-time and integrity check outcomes aren't recorded, so "the share of page loads where every number matched" can't be reported yet. Drops by rule exist in the ledger but no screen serves them. Two small additions would make it a production monitor.
Observability
How. The pieces of a trace are recorded in Evident's own stores but not joined into one view: usage rows per model call (model, stage, tokens, repair flag) for claim selection, writing, editions and Ask; stored prompts and outputs for the Brief, Plan, ledger, writing and editions (Ask keeps no question text); run steps and timings (median 5.8 minutes, 90th percentile 10.5, across 22 runs); lineage per run table and fact; run manifests.
Why. Each record was added where a feature needed it. No external tracing service was added, which keeps prompts and data inside the platform.
Limits. Usage rows don't yet cover the Brief, the Plan's Questions or the definitions draft. The release that produced a run isn't recorded. Next: usage for those three calls, a release id per run, and a run-level trace view.
Determinism and reproducibility
How. Numbers are deterministic by construction; the model's words are reproducible by replay. Content-addressed storage re-hashed on read; query and snapshot fingerprints; each run table stored as Parquet with its query, SQL, fingerprint, engine version and dataset snapshot. Exact decimals and fractions; the known-answer kit runs at 1 and 4 threads and requires identical rows. A locked in-memory DuckDB, UTC. 51 versioned policies. Every model answer stored for replay.
Why. An analysis someone acts on must be re-creatable months later, and a reviewer must be able to see that the same data gives the same numbers. The release build runs the known-answer tests and the release's test list in a throwaway PostgreSQL before it can ship.
Limits. Runs don't yet record which release produced them. Model answers aren't deterministic across live calls; recording is the lever, not temperature.
A person in the loop
How. A person approves wherever the model's reading changes what runs or is published: Definitions (confirm and publish; analyses read only published datasets), the Brief (check "How we read it", answer readings, edit, approve), Readings (ambiguous choices become explicit questions, never silent defaults), the Plan (nothing runs before), a lift's members (leave out, rerun), the deck (pick slides and theme; a built version never changes).
Why. The model is good at reading intent and bad at knowing when it guessed. Putting the guess in front of a person before any compute, and making ambiguous choices explicit questions, catches misreadings where they are cheapest to fix. A person can't override the checks; they can only change the request and run again.
Limits. Approval is per analysis, not per sentence; a person can't mark a written sentence as wrong and have that recorded. A review-and-flag step on the Story would add that feedback loop.
Privacy and security
How. Rows stay on Evident's servers. Setup sends column names and counts, plus optionally up to 10 rows and 10 values per column with personal-looking columns hidden. The Brief gets the request, names, categories of breakdowns of 50 or fewer, and the date range. The run gets computed results and chosen findings, with personal names as stand-ins. The provider is OpenAI's business API with storing switched off. Sign-in through Cloudflare Access; PostgreSQL row-level security plus a code filter; a "Can see rows" right; content-addressed storage per workspace.
Why. Clients' data is the asset; an analysis needs the model's reading and wording, not the rows. Two independent locks on workspaces mean one bug doesn't leak another client's data. Proof tests try to read across two organizations on every route.
Limits. A client's own Azure OpenAI, or Evident in the client's cloud, are projects, not switches. Zero retention with OpenAI is arranged per client, on request. Data & security →
Model choice, cost and latency
How. Per-stage defaults, each changeable by setting. The Brief's "Thorough" choice scored 100% of its checks on 24 requests in about 33 seconds; "Fast" scored 93% in about 11 seconds and can miss parts of a complex request. Claim selection runs six Questions at a time with an 8,000-token output budget. The writer's model is a workspace setting.
Why. The Brief's quality is visible to the person immediately, so it was measured model by model; the writer is the most expensive step, so a workspace can lower it. Time: median 5.8 minutes, 90th percentile 10.5, on a development machine; writing is the longest step. Tokens: 74,441 to 143,867 per planted analysis, in 9 to 18 calls.
Limits. Dollars are not measured: usage rows don't cover the three planning calls and no price table is applied. Only the Brief has a recorded model comparison.
Prompt injection
How. Four threats considered: a cell value or column name written as an instruction; a label forging the data markers; a request or reply overriding system rules; echoed model output carrying instructions. Seven defences: nothing to hijack (no tools, typed output, checks); data marked as data; cleaning and caps; the user's request marked but not stripped; the number check; Ask bounded to its pages; a locked query engine.
Why. Pattern filters alone always miss something. Removing capability means a successful injection still can't do much: at worst it changes which claims are chosen or how a sentence is worded, and both pass through the compiler and the writing checks.
Limits. The instruction patterns are English phrases and can be evaded; an injected value could still nudge which true claims are chosen or how they're worded within the checks. Red-team cases in the planted harness would measure this; none are recorded yet.
MCP and agent loops: not in Evident today
How. No model in Evident calls a tool, and Evident exposes no MCP server, because a workflow is reproducible and testable against known answers while an agent picks different steps on different runs.
Later, maybe. Read-only tools over the query layer and Check (describe a dataset, query a metric, check a figure, draft a request) with guards; Ask answering "what does my data show" through such a tool, with the same number checks, is the likely first agent use. A sketch, not a build.
Fine-tuning: not used
How. Hosted base models through structured outputs instead, because correctness comes from code (facts computed, claims compiled, sentences checked), the rules change package by package, model choice stays open by setting, tuning on clients' analyses would put their data into a model, and there isn't enough failure data to show a gap only tuning closes.
When it would be worth it. A narrow, stable, high-volume step (claim selection is the candidate), trained on synthetic planted runs, never client data.
How Evident was built: AI-assisted delivery
How. Connor designed Evident and built it by directing several AI agent workstreams with written ownership (planning and data, analysis and evidence, deployment, design), through a harness of specs, self-validating packages and records. No agent changes the repository: each change is a package with files, starting hashes, tests and a script that checks, applies, runs the checks, commits, and restores on failure. Connor applies and accepts every change. Each package has an as-built record, a dated design-log row, and a weak-points register.
Why. Agents write fast and confidently, including when wrong. Hash-checked packages, tests in the apply step, and written records turn that speed into changes the owner can trust, undo and audit. The checks: the Python suite (343 test files) on PostgreSQL as an ordinary role with row-level security on; frontend validators and type check; the known-answer kit and planted datasets; independent review by a separate agent. The last full suite: 3,635 passed, with the same 13 failures and 1 error as the baseline.
Limits. The owner is the integration point; several agents can't merge in parallel. Specs and records are long, and an agent that skips reading them repeats old mistakes; a primer and a "which document wins" rule exist for that. About Connor →