Zombie Stats: Where the Numbers AI Quotes Come From

We asked each assistant 200 questions that call for a number, word for word, through its company’s API with web search on and a US location: 160 everyday questions across 16 topics, and 40 famous statistics we suspected, reported separately. We pulled every statistic out of every answer, with the page it cited.

For a random sample of each assistant’s cited statistics (every one of ChatGPT’s, up to 300 from each of the others), we opened the cited page, found the figure, and followed the page’s own source, up to five pages deep. Where a page named a source without a link, we searched for it. An AI model judged each trail, and a stronger model re-judged 300 of them, agreeing on the outcome 82% of the time.

Then we checked 150 trails by hand, picked at random, with a third check wherever the hand check and the model disagreed. A trail we checked by hand counts as we found it; every other trail counts as the mix of outcomes we found by hand behind the model’s verdict. The ranges carry the uncertainty from both samples.

Rates for each assistant cover its own sample. The five-assistant rate weights each assistant by how many cited statistics it quoted, so it describes the statistics people were shown. Every statistic shown on its own on this page was checked by hand.