Wrong number
82 editscaught. Every number in a claim has to appear in the evidence.
BeforeThey run for approximately 670 km in a south-west direction
AfterThey run for approximately km in a south-west direction
Open source · Runs on your laptop
AI cites its sources. upheld reads them.
Point it at an answer from ChatGPT, Perplexity, Gemini or your own RAG pipeline. It splits the answer into claims, reads every page it cites, and tells you which claims the sources actually back. Runs locally: no API key, no account.
Fig. 01 · Portico, in glyphs
Eleven columns hold. One doesn't.
macro-F1 on AttributionBench, in-distribution
GPT-4 with chain of thought: 73.3
macro-F1 out-of-distribution
Best fine-tuned model in the paper: 84.2
of wrong numbers caught
82 true claims, one number edited in each
API keys needed
Runs on CPU, Apple silicon or NVIDIA
The problem
AI search now mostly cites real, working pages. The trouble is what the answer says those pages say, and a link checker can't see it.
of cited facts were actually in the source, across 14 LLMs writing deep-research reports. Their links worked over 94% of the time.
arXiv 2605.06635of answers from a deployed clinical RAG system cited real evidence about the wrong drug, and passed faithfulness checks.
arXiv 2607.09349Watch it read the sources
These are real upheld audit --json reports, not mockups. Drag the slider or use the arrow keys to step through each claim: its verdict, the passage it was checked against, and the source's status.
An answer written with planted errors: a wrong height, an architect the page never mentions, and a link that 404s and was never archived.
verifier ensemble:minicheck+lexical+lettuce@0.5
Question
Tell me about the Eiffel Tower.
Answer
The Eiffel Tower was designed by the engineering company of Gustave Eiffel and completed in 1889 1. It stands 384 metres tall 1. It was the tallest man-made structure in the world until the Chrysler Building in New York was finished in 1930 1. Its design was led by the architect Frank Lloyd Wright 1. The tower is repainted every three years using a special bronze paint 2. It is the most visited paid monument in the world.
1 uncited claim is shown under All only.
- [1]en.wikipedia.org/wiki/Eiffel_TowerLive
- [2]en.wikipedia.org/wiki/Eiffel_Tower_repainting_scheduleDead and never archived
Claim detailthreshold 0.50
Claim 01 / 06
Waiting
This output came from upheld audit --json. Run the same on your own answer →
Four passes over every answer
Each claim ends up with one verdict per citation, the passage it was checked against, and a score. This is the run behind the report above.
01 · Parse
Parse
Splits the answer into sentences and maps [1], [1, 2], [^1] and inline links to their sources. Questions and lead-ins aren't claims.
claim 2It stands 384 metres tall.[1]claim 5repainted every three years[2]claim 6most visited paid monumentnone02 · Fetch
Fetch
Downloads each cited page or PDF and pulls out its main text. A dead link goes to the Wayback Machine: archived means link rot, never archived means likely made up. A dead or blocked page is read from its archived copy.
GET [1]Eiffel_Tower200GET [2]..._repainting_schedule404Waybackno snapshot, everfabricated03 · Retrieve
Retrieve
Cuts each source into ~120-word windows and ranks them with BM25 and bge-small embeddings, merged by reciprocal rank fusion.
claim 2384 metres tallbestThe tower is 330 metres ...0.9904 · Verify
Verify
MiniCheck, word overlap with exact number matching and LettuceDetect, combined by a logistic regression with four fitted numbers.
score0.24 < 0.50verdictunsupportedflagged384 metres
Six verdicts, each one specific
Every citation gets one. A claim keeps the best verdict among its citations. Each mark reads without color too: fill, wave, strike or dashes.
supported
It was the tallest man-made structure in the world until 1930.
Wikipedia, Eiffel Tower: support 0.84.
unsupported
It stands 384 metres tall.
The page says 330 metres. The flagged words are marked.
wrong_entity
A tirzepatide trial cited for Ozempic.
Reads as supported, but the name never appears in the source.
source_missing
Repainted every three years [2]
404, and the Wayback Machine never saw the page.
unverifiable
Trial data from the paywalled appendix [3]
Blocked with no readable archived copy, or a scan with no text. It says so instead of guessing.
uncited
The most visited paid monument in the world.
A factual-looking sentence with no citation at all.
Quality
How good is it
On 564 real 2026 citations whose page it could read, the default verifier catches 91% of bad citations and wrongly flags 38% of good ones. With the opt-in LLM judge: 86% and 18%. Treat the default's flags as a queue to review, and the judge's as an opinion you can act on.
Verifier alone, on the labels in the 2026 section below; bad means partial or unsupported. The whole pipeline, on 554 of those citations, is a little different and is labeled as such wherever it appears.
AttributionBench
Graded against 3,296 human labels
Claims from seven datasets, each with the evidence it cited and a human verdict. Thresholds were picked on the dev split, never on test.
In-distribution
macro-F1. With the judge: 76.7. GPT-4 + CoT: 73.3
Out-of-distribution
Best fine-tuned: 84.2
Average of ExpertQA, Stanford-GenSearch, AttributedQA and LFQA. Macro-F1, test split.
Best fine-tuned model
Best per column in the paper, fully fine-tuned
78.5
upheld ensemble + judge
Adds the Granite Guardian 3.3 8B judge, opt-in
76.7
upheld ensemble
MiniCheck + word overlap + LettuceDetect, the default
74.4
upheld judge
Granite Guardian 3.3 8B alone, run with llama.cpp
74.2
upheld minicheck
MiniCheck-Flan-T5-Large alone, 770M
73.9
FLAN-UL2 (20B)
Zero-shot, from the paper
73.6
GPT-4 + chain of thought
Zero-shot, from the paper
73.3
upheld ensemble + entity check
As shipped: adds the wrong_entity verdict
72.2
T5-XXL-TRUE (11B NLI)
Zero-shot, from the paper
72.2
upheld lexical
Word overlap with exact number matching
67.6
upheld lettuce
LettuceDetect-base alone, 150M
67.3
- upheld default
- other upheld verifiers
- from the AttributionBench paper
Lightly supervised, not zero-shot
The ensemble's four numbers were fit on 2,000 train rows. The clean zero-shot comparison is MiniCheck alone at 0.5: 73.3 in-distribution, level with GPT-4.
An LLM judge adds 2.3 points
Opt-in: Granite Guardian 3.3 8B as a fourth signal lifts in-distribution to 76.7, 1.8 below the best fine-tuned model. Search-engine answers gain most, 71.8 to 78.6.
ExpertQA is still hard
Long, expert-written claims score 57.8, or 59.3 with the judge, level with GPT-4's 59.2. Every model in the paper stays under 62.
The entity check costs 2 points
It false-alarms on 6% of supported claims, mostly names written differently. It stays on because it catches 98% of names the source never mentions.
Then we broke true claims on purpose
Take a claim humans marked supported, edit it so the evidence no longer backs it, and see if upheld notices. A catch only counts if the unedited claim passed first.
A name the source never mentions
150 editscaught with the entity check. 87% from the ensemble alone.
BeforeDry ether is used in the Wurtz reaction to ensure that the reaction is free from moisture.
AfterDry ether is used in the reaction to ensure that the reaction is free from moisture.
The wrong party, also named in the source
150 editscaught. The weak spot. 64% with the opt-in judge; MiniCheck alone catches 57%.
BeforeEpicureanism has also managed to adapt and blend with modern secular life
After has also managed to adapt and blend with modern secular life
Two claims joined, cited [1][2]
100 editscorrectly supported by reading both sources. No single source ever passed one on its own.
BeforeClaim A [1]. Claim B [2].
AfterClaim A and claim B [1][2].
2026 answers
Then we checked 641 real citations
25 current questions, answered by Claude, OpenAI and Gemini agents with their own web search. upheld fetched every cited page, and labelers who didn't know which agent wrote what judged each claim against it.
Bad citations flagged, whole pipeline
Partial or unsupported, of the 554 upheld could read
Flags that were right, whole pipeline
75% agreement with the labels overall
Each citation, judged against the page it points to. Sorted by share supported.
- Supported
- Partial
- Unsupported
- Can't judge
OpenAI gpt-6-luna, Codex CLI
125 citations
51% supported · 28% partial · 14% unsupported · 6% can't judge
Claude Sonnet 5.5, Claude Code
255 citations
47% supported · 25% partial · 17% unsupported · 11% can't judge
Gemini 3.8 Flash, Antigravity CLI
257 citations
41% supported · 45% partial · 7% unsupported · 7% can't judge
About half hold up
41-51% of each agent's citations back everything their sentence says. The ones that back nothing at all are 7-17%.
Gemini says more than its sources
It is rarely flat wrong, but 45% of its citations back only part of the sentence: a figure or a second fact the page doesn't have.
A first filter, not the final word
Whole pipeline: upheld misses few bad citations, but about 3 in 10 flags are false alarms. Block pages are now reported unreadable. Verifier alone, the opt-in LLM judge, reading 10 passages per claim, halves false alarms on good citations: 38% to 18%.
Labeled by Claude, checked by GPT and Gemini
Claude Sonnet 5.5 subagents labeled 639 of 641 citations blind. GPT and Gemini relabeled 150 and reached the same verdict on 92-93%, as often as they agree with each other, with no sign Claude favors its own answers.
Where it still misses
A checker you can't calibrate is worse than none. These are the known failure modes, each with the case that shows it.
- 01
Contradictions in the source's own words
“The Great Wall of China is visible from the Moon with the naked eye” passes at 0.59. The page says it cannot be seen.
MiniCheck scores it 0.08, but word overlap and LettuceDetect both give 1.0 and outvote it. Found while building this site.
- 02
One-step inferences
Source: “mean change in body weight ... -14.9%”. Answer: “an average weight loss of 14.9%”.
MiniCheck scores that 0.08, so a true claim is flagged. Same for dates that need arithmetic.
- 03
The wrong party
A result credited to Stoicism when the page says Epicureanism, and mentions both.
Caught 35% of the time. Word overlap finds every word of the edited claim in the source.
- 04
Same number, different unit
“It stands 410 metres tall” passes, because the page gives the base as 410 ft.
The number check matches digits, not quantities. Also found while building this site.
Coverage gaps
- Expert content
- 57.8 macro-F1 on ExpertQA. Expect more mistakes on specialist claims.
- Speed
- 0.3-1 s per claim per source on an Apple M5 Pro, several times slower on CPU.
- No OCR
- PDFs are read, but a scan with no text layer comes back unverifiable rather than guessed at.
- Sentence-level claims
- A sentence with two facts is judged as one unit.
One engine, four ways in
A CLI for spot checks, a Python library for your eval suite, an HTTP API to gate a RAG pipeline, and an MCP server so your agent checks its own citations. The first run is the slow one.
- No API key and no account. Nothing leaves your machine except the requests for the cited pages.
- The first run downloads about 4 GB of models into ~/.cache/huggingface, which takes minutes. After that, loading them takes about half a minute, and each audit takes seconds.
- The default verifier takes 0.3–1 s per claim per source on an Apple M5 Pro, and several times longer on CPU.
- The judge is opt-in. Its install builds llama.cpp, downloads 8.7 GB, and needs about 9 GB of free memory.
- Every report names its verifier and threshold.
Check one answer
Save the answer as answer.md, with its reference list at the end, and run the first line. The first run downloads about 4 GB of models into ~/.cache/huggingface, which takes minutes. Later runs take seconds.
It prints the verifier and the retrieval it used, then how many cited claims are supported, then a table with one row per claim: its verdict, its score, and the words the source doesn't back. A second table lists the sources and their status.
1pip install "upheld[models]"2 3# an answer with [1]-style markers and a reference list4upheld audit answer.md5 6# or a provider's response, saved as JSON7upheld audit response.json --from openaiChoosing a verifier
Agreement and flags are from the 564 labeled 2026 citations, with the verifier alone. Bad means partial or unsupported. See Quality above.
| Verifier | Passages per claim | Agreement | Bad citations flagged | Good citations flagged | Time per claim | Memory |
|---|---|---|---|---|---|---|
| ensemble | 3 | 76% | 91% | 38% | 0.3–1 s per claim per source, on an M5 Pro | about 4 GB, loaded |
| ensemble+judge | 10 | 84% | 86% | 18% | about 2.5–3 s per claim per source, with Metal | about 9 GB free, plus an 8.7 GB model |
Pick ensemble for speed, ensemble+judge if you'll act on the flags.
When it looks broken
| What you see | What's going on |
|---|---|
| A page blocks bots, or answers 401, 403 or 429. | It's reported unverifiable. upheld reads the Wayback Machine's copy instead when there is one. |
| A page returns 404 to bots but works in a browser. | If it was never archived, upheld reports it as likely fabricated. That's wrong here: the page exists and the bot was refused. Open the link yourself to check. |
| A scanned PDF. | It's reported unverifiable, because it has no text layer. upheld doesn't OCR. |
| A Gemini source link. | Gemini's source links are redirects that expire. Once one has, its claims are reported unverifiable. Audit a Gemini answer soon after you get it. |
| The judge runs out of memory. | It needs about 9 GB of free memory. Use ensemble instead. |
| You used --verifier lexical --retrieval bm25. | It needs no models and is instant, but it's much weaker: it checks word overlap. Use it when you can't spare the memory or the download. |