When to trust AI agent output and how to verify it

You can trust AI agent output when every fact in it carries a source you can open, when a random spot check of those sources matches, and when the task was the kind an agent does reliably: reading, extracting, sorting, drafting. You cannot trust it, without your own check, for anything that depends on the current state of the world (is this open, is this price right, is this person still at this company) or on judgment (is this a good customer, is this claim fair). The verification habit is cheap: a few minutes per deliverable, in a fixed order.

Here is how I decide what to check and how, for research, lists and drafts an agent produces.

The four classes of agent output

1. Extracted facts: numbers, names, dates, quotes pulled from pages. Reliable when the page is cited; the risk is a stale page or a misread table.
2. Collected lists: companies, postings, products gathered from many sources. Reliable in coverage, unreliable in freshness; the risk is closed, moved or duplicated rows.
3. Judgments: rankings, scores, "best" picks, recommendations. Only as good as the rule you gave; without a rule, they are the model's taste.
4. Generated text: drafts, summaries, descriptions. Reliable in form, unreliable in specifics; the risk is an invented detail that reads naturally.

Each class gets a different check.

Check 1: a source next to every fact

Before the task starts, require it: "Put the URL you read next to every fact, in the same row or sentence. A fact without a URL is not allowed." This single rule converts an opinion into something you can audit. If the agent cannot cite a source, it says [MISSING], and you learn what it guessed.

Read the output once with that lens. A fact with no URL beside it is the first thing to question.

Check 2: the random spot check

Pick rows or claims at random (not the first ones; the agent is usually most careful at the start). For a list of 50, open 10. For a report with 20 claims, open 5. For each, confirm three things: the page exists, the fact is on it, and the page is current (dated, or showing signs of being live).

Decision rule: one mismatch in ten, fix that row and continue. Two or more, stop, tighten the instructions, and re-run the extraction; the rest of the output is probably as bad.

Check 3: freshness for anything that changes

Prices, availability, job postings, people's roles, event dates, product specs. For these the agent must re-fetch the page at the end and record the check date. "Checked on" is a column, and a row without it is unverified.

If the task matters, do the final freshness pass yourself for the top items. Fifteen links is ten minutes.

Check 4: the rule behind every judgment

For rankings and scores, ask to see the rule and the inputs per row: "Show the score for each row as its components." If the components are visible, you can disagree with the weights instead of the result. If the agent cannot show them, the ranking is a feeling, and you treat it as a suggestion.

Then ask for the counter-case: "Argue that the top-ranked item is the wrong choice." A real argument reveals what the rule missed.

Check 5: specifics in generated text

Read any draft once for every specific: a number, a name, a date, a quote, a product feature. Each one is either in the fact list you supplied, in a cited source, or invented. Invented specifics read smoothly, which is exactly why they need the pass.

For summaries, open the source and read the paragraph the summary leans on hardest. A summary that reverses a qualifier ("most" to "all", "may" to "will") is the common failure.

Signals that something is wrong

• Round numbers everywhere
• Every row has a value in every column (real data has gaps)
• The same phrasing in every "why" or "reason" cell
• Sources that are all from one site, or all undated
• Confident statements about things that change daily, with no check date
• No [MISSING] markers in a task where some facts were clearly hard to find

Any two of these together mean a re-run with tighter rules.

A two-minute verification routine

1. Scan for facts without URLs
2. Open five random sources, confirm fact and date
3. For anything that changes, check the "checked on" column exists and is recent
4. For any ranking, ask for the score components
5. For any draft, circle every specific and trace each to a source

Do this before reading the output as a reader. It is faster to check structure first and content second.

When to skip the checks

Low-stakes, reversible, internal: a first draft you will rewrite, a brainstorm, a reading list you will skim yourself. Everything you will send to someone, pay for, publish or act on gets the routine.

Related reading: Prompting patterns that save hours: brief, facts, checklist for the brief and fact-list patterns that make output checkable in the first place, and How to build a lead list with an AI agent (and check it) for the routine applied to one deliverable.

FAQ

How do I know if AI output is accurate? Require a source URL beside every fact, open a random sample of them, and confirm the fact and the date. One mismatch in ten is a fix; two is a re-run.

What kinds of AI output are most reliable? Extracting facts from cited pages, and drafting in a given form. Least reliable: anything about the current state of the world and anything that is a judgment without a stated rule.

How many AI results should I spot-check? About one in five for lists up to 50 rows, never the first few only. Increase the sample when the stakes are higher or the first sample finds a problem.

Can I trust an AI ranking? Only when it shows the rule and the per-row components. A ranking without visible inputs is a suggestion, not a result.

What are signs an AI made something up? Round numbers, no gaps in the data, identical phrasing across rows, undated or single-site sources, and confident claims about changing facts with no check date.

Should I verify AI output every time? For anything you will send, pay for, publish or act on, yes; the routine takes a few minutes. Internal first drafts and brainstorms can skip it.

If you want research delivered with sources beside every fact and a verification pass already done, that is how the research gigs on my store are built: https://rmgo.sites.tab.bot/leads?s=blog