Field note
We failed our own accuracy test
Fifty records enriched, ten checked by hand against their sources. Five carried a material error. Here is what they were.
Drafted from 50 sourced records · reviewed by Saad Allahwala · 2026-08-10 · every fact links to its source
Before this record was allowed to grow, it had to pass a test set for it in advance: enrich fifty subjects automatically, hand-check ten of them against the documents actually cited, and find fewer than two with a material error. A material error means a fact that is not in the cited source, a wrong number, an invented contact detail, or a source that does not say what the statement claims.
Ten were checked. Five carried a material error. The test was failed, and by a wide margin.
Four of the five were one fault. LinkedIn company pages state staff numbers as a band — "Company size 201-500 employees" — and the extractor was reading the lower end of the band as an exact headcount. One organisation was published as having 201 employees, another 11, another 10. A fourth was published with a figure taken from LinkedIn's count of member profiles while the same page's own company-size field said something else entirely.
The fifth was subtler and worth knowing about. The guard that stops the extractor writing facts about the wrong organisation compares the documents against the subject's name. For "University of Agriculture Faisalabad, Department of Forestry" every document mentioned the University, so the guard passed — and the University's founding year, its type and its campus area were written onto the Department.
Both faults are now refused by the extraction rules, and a re-run over the same sources produced no material errors in the same ten. The rules that changed are public, and the failed records are still in the history where they can be inspected.
We are publishing the failure because a provenance platform that only reports its successes has already given up the thing it sells. Two comparable platforms in this sector were taken apart by journalists checking exactly this. We would rather be the ones who checked first.
What we couldn’t find
Six records enriched before the wrong-organisation guard existed have still not been hand-checked. We know they exist and we have not done it yet.
The corrected headcounts are not corrected everywhere: the bad values are still live on some records, because replacing them is a decision about the history and not only about the number.
And the deeper gap is the sample. Ten records is what the test called for, and ten records is a small sample.
Know any of it? Tell us — that is how the gaps close.
Every factual sentence, and what backs it
The Ten Billion Tree Tsunami Programme is published here with two different tree counts, 3.5 billion and 1.424 billion, both of them faithfully taken from the same article.
Ten Billion Tree Tsunami Programme · trees_planted_claimed = 3500000000 · source
Sindh’s Forest & Wildlife Department reports 241,198 hectares of riverine forest under its management.
Forest & Wildlife Department, Government of Sindh · forest_area_managed = 241198 · source