When research is buried in a deck, it dies.
When it's built into the systems people use, it scales.
Research teams are starting to deliver research through AI systems, with people reviewing along the way. Building those systems takes a different discipline than using AI day to day, which Part 1 covers. Small error rates compound at every step of an automated pipeline, so trust has to be designed in.
What we built
Dual-agent tree testing
A synthetic user agent works through a navigation tree by following semantic scent, and an expert agent evaluates its path against defined logic and heuristics. We ran the same tasks across several models, including Claude, Gemini and Copilot.
Cognitive load crawler
A crawler that fully renders JavaScript-heavy pages, extracts their text, and scores reading difficulty across a large website with the Automated Readability Index. Scores feed a dashboard so teams can track them over time.
What we learned: labels, calls to action, page sections and navigation all needed scoring, not just body copy, and sites that restrict crawlers limit what it can reach.
Research repository
A taxonomy and content model that turned more than 1,000 past studies into a searchable repository ready for AI retrieval.
01Start from vetted data
Judge every system against a held-out set of prior research: vetted "golden" data, not whatever happens to be sitting in a shared drive. Otherwise it's easy to justify a system after it's built.
A Data Operator owns that set. Before anything is ingested, they audit the baseline data for bias and subjective noise, and label sources for structural bias, such as research drawn mostly from one market.
In practice. We pressure-tested the repository's taxonomy against real survey, persona and concept-test data, and scored it for confidence, before building anything on top of it.
02Separate rules from guesses
Split every system into two strict parts. Blurring them lets a probabilistic model quietly override a rule.
Hard rules: deterministic
Code-enforced, structured constraints that follow the standard rules of the research craft. Easy for an expert to review and audit.
Treat as fact
AI guesswork: probabilistic
Model output based on likelihood. Useful, but subject to made-up answers and drift.
Treat as a hypothesis to verify
In practice. The crawler scores readability with a fixed formula rather than a model's judgment, so every score can be traced. In tree testing, the synthetic user's navigation is the guesswork, and the expert agent's evaluation runs on defined logic and heuristics.
03Match the model and set safeguards
| Task | Model |
|---|---|
| Extraction and recognition | Lightweight, fast models with fixed output formats, for pulling interface elements, dates or other single facts |
| Reasoning and clustering | High-capability models with large context windows, for qualitative synthesis and spotting contradictions |
Three safeguards are necessities, not preferences:
- Null-handling. When data is missing, the model returns nothing. It never fills the gap with something made up.
- Defined failure states. Edge cases, such as a tie in clustering, stop the task and flag it for a person instead of guessing through it.
- Quarantine. Answer keys and test material never enter the agent's session, so there's no path to contamination.
04Enforce constraints in code, not prompts
Models quietly ignore instructions written as prose. When a constraint matters, it goes in the code, never in the prompt alone.
- Output schemas. Fixed output structures the model can't drift out of.
- Scope tags. Findings are tagged by scope, such as a single market, so a local finding can't quietly become a global one.
- Clear hierarchy. Headings and metadata, so the model knows what outranks what in the source.
05Test every change, then test against people
Every prompt change is measured against a fixed evaluation set before it's trusted.
| Change | What the evaluation showed | Decision |
|---|---|---|
| Retrieval-grounded extraction | Made-up citations disappeared, and every claim traced back to a source. | "No citation, no insight" became the rule for automated reports. |
| A higher-reasoning model for edge cases | It was noticeably better at spotting contradictory user signals. | Adopted for high-risk strategic discovery only. |
| Scope tags moved into the output schema | Findings from one market stopped being attributed to another. | A hard gate for all global synthesis. |
An evaluation set tells you whether a change helped. Only people tell you whether the system is right.
In practice. We validated the tree-testing tool against human tree tests. Synthetic users kept going where real people would give up, results changed from run to run, and they rarely matched what people did. We now treat its results as early flags on structure, never as a substitute for testing with people.
Framework, revised November 2026.