UXR × AI: Distributing intelligenceUX research and AI: Distributing intelligence

The right job, not just the right model. Identify where AI closes a real knowledge gap without producing research slop.

Utility must come before implementation. Adding AI without a specific user problem produces "vending machine insights": plausible-sounding answers with no tactical substance. Automate where signal is unreliable, missing, or too slow to reach a person. The point is not to replace the researcher, but to free them for the edge: the unsaid nuance a model can't perceive.

High-value use cases

Prioritize sources where volume already prevents human synthesis, provided each source is labeled for structural bias first. If most of a team's research comes from one country, an unconstrained model will read that country's patterns as global truths.

Data source	Strategic use
Support logs and transcripts	Surfacing unsaid knowledge gaps and sizing service issues at scale
App store and third-party reviews	Clustering feedback by theme fast enough to catch a sentiment shift while it's forming
Internal research repositories	Synthesizing prior research by its metadata, so studies aren't run twice
Unmoderated video sessions	Pulling verbatim quotes at volume, limited to what people say, not how they say it

Ground truth and the Data Operator

To prevent justifying a system after it's built, every system is judged against a held-out set of prior research: vetted "golden" data, not raw, unchecked history.

The Data Operator owns the repository's trustworthiness. Before any AI ingestion, they audit the baseline data for bias and subjective noise. Their sign-off is what lets a system be evaluated against verified data, rather than whatever happened to be sitting in a shared drive.

Hard rules vs. AI guesswork

Every implementation splits into two strict buckets. Blurring them lets a probabilistic model quietly "correct" an unfamiliar term into a more common one it has seen before.

Treating a guess as a fact is a failure of leadership.

Match the tool to the shape of the task, and enforce the trust math instead of trusting the speed.

Speed creates a false sense of reliability: the convenience trap. Every output must be grounded in the organization's authoritative data through retrieval, and every model choice matched to what the task actually is.

Model matching by task complexity

Task	Model
Extraction and recognition	Lightweight, fast models with fixed output schemas, for pulling UI elements, dates or other single facts
Reasoning and clustering	High-capability models with large context windows, for qualitative synthesis and spotting contradictions

Trust parameters

Small error rates compound at every step of an automated pipeline. That's why these three are necessities, not preferences:

• Null-handling. When data is missing, the model returns nothing. It never fills the gap with something made up.
• Edge-case logic. Defined failure states, such as a tie in clustering, stop the task and flag it for a person instead of guessing through it.
• Quarantine standards. Answer keys and test material never enter the agent's session, so there is no path to contamination.

Risk-based human review

The researcher stays the final authority. Review effort follows strategic risk rather than being spread evenly, with the most scrutiny on the smallest, highest-stakes slice of output.

Risk	Review	Applies to
High	Full review	Strategic hypotheses, executive summaries and any ambiguous data. Every claim is checked before it ships.
Medium	Targeted sampling	Samples of source-traced quotes are checked against the original transcript.
Low	Spot check	Administrative triage, formatting and routine wording, checked lightly at volume.

"It works" is not a finding. Enforce structure over prose, and measure success in decisions, not tokens.

Enforce constraints in structure

Models quietly ignore instructions written as prose. When a constraint matters, it goes in the code or the harness, never in the prompt alone.

• Output schemas. Fixed output structures, such as JSON schemas, that the model cannot drift out of.
• Scope tagging. Reports are explicitly tagged by scope, such as a single market, so a local finding can't quietly become a global one.
• Hierarchical structure. Clear headings and metadata, so the model understands what outranks what in the source.

Versioning and evaluation sets

Every prompt change is measured against a fixed evaluation set before it's trusted. Examples of what that looks like:

Change	What the evaluation showed	Decision
Retrieval-grounded extraction	Fabricated citations disappeared, and every claim traced back to a source.	"No citation, no insight" became the rule for automated reports.
A higher-reasoning model for edge cases	It was noticeably better at spotting contradictory user signals.	Adopted for high-risk strategic discovery only.
Scope tags moved into the output schema	Findings from one market stopped being attributed to another.	A hard gate for all global synthesis.

Success, defined by decisions

Every project documents exactly one of each: the decision it enabled, and the limitation it didn't overcome.

• Decision. For example: "Thematic clustering of thousands of support logs got a specific bug fix prioritized right away."
• Limitation. For example: "This model can't read expression or gesture in video sessions. Findings are linguistic only, and need a researcher to judge emotional intent."

The feedback loop: no citation, no insight

Verify, don't assume. If the model can't point to a timestamp, a quote or a raw data point, the insight is discarded before it reaches a deck.

1. The system produces an insight.
2. A citation check asks whether it points to a timestamp, quote or raw data point.
3. If it does, it goes into the report. If it doesn't, it's discarded and flagged.
4. Each flag becomes a correction to the ground-truth set the Data Operator maintains, which shapes the next insight.