MacroGrowth, inflation & jobs
AI-assisted audit finds serious errors in 10 of 100 recent top-five economics journal papers
An AI-assisted audit of 100 papers published in 2026 in economics' top five journals found serious errors affecting key results in ten, despite peer review. Coming months after an AI disproof of an Erdős conjecture, it suggests AI's clearest economics role is error-checking, not discovery.
According to arXiv 2608.08437 (Aug 2026) [5]: Refine's pipeline audited 100 randomly drawn 2026 papers from the top five economics journals, all of which had passed peer review.
SOURCES[5] Fourie, AI and the research assistant (arXiv 2608.08437) arxiv.org
AI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- Refine's pipeline audited 100 randomly drawn 2026 papers from the top five economics journals, all of which had passed peer review. [1] [2]
- Original source
- The Economist · Finance & economics article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2608.08437 (Aug 2026) [5] date from arXiv ID
- When
- November 19, 2026 article published
- Covered
- Nov 21 edition
- Evidence
- 10 sources across 8 sites
- 1AI is swallowing maths. Is economics next?economist.com
- 2Refine: AI-assisted audit of 100 top-five economics papersrefine.ink
- 3TechCrunch: OpenAI claims it solved an 80-year-old math…techcrunch.com
- 4Matthew Kahn: Why AI has disrupted math but will improve…matthewekahn.substack.com
- 5Fourie, AI and the research assistant (arXiv 2608.08437)arxiv.org
- 6Marginal Revolution: AIs vs. humans in economics papersmarginalrevolution.com
Reach: not measured yet
Refine's pipeline audited 100 randomly drawn 2026 papers from the top five economics journals, all of which had passed peer review. AI models flagged 5,421 issues, a second model re-checked the serious ones, and experts then reviewed them. Ten papers had a serious error affecting an important result: four misstated headline numbers, two had missing or inconsistent supporting results, and four had failed proofs or assumptions. [1] [2]
In May 2026 an unnamed OpenAI reasoning model disproved a 1946 Erdős conjecture about unit distances. Mathematicians, including one who had called an earlier OpenAI claim a misrepresentation, backed the result. Its constructions beat grids only at about 10^2,000,000 points, though, and many of the ideas it needed were already in the literature. [3] [9]
Original economics research looks harder to automate. One economist argues that economic behaviour has no fixed parameters, so models trained on past data run into the Lucas critique. Papers written fully by AI agents lean heavily on difference-in-differences, show signs of p-hacking, and give quite different results depending on each agent's analytical choices. They score well mainly in crowded research areas. [4] [6] [7] [8] [10]
AI is already changing research staffing. Working papers that disclose AI use credit more human research assistants than other papers by the same author, and the difference comes from coding and data work. [5]
AI plus expert review caught misstated headline numbers and failed proofs that peer review missed.
Read the full assessment
Implication: journals, central banks and consultancies could use AI pre-review, though about 54 flags per paper need triage, and fully AI-written research still has method and error problems.
Executive brief
An AI-assisted audit run for The Economist checked 100 randomly drawn 2026 papers from economics' top five journals. All of them had passed peer review, and 10 still had a serious error affecting an important result (Refine). The Economist ran it alongside its Free Exchange column asking whether AI will do to economics what it is doing to mathematics. In May 2026 an OpenAI model disproved an Erdős conjecture from 1946 (TechCrunch). So far AI's clearest role in economics is checking errors, not original discovery.
What changed and event timeline
OpenAI model disproves Erdős unit-distance conjecture
A general-purpose reasoning model found a new family of point constructions. Mathematicians including Noga Alon and Thomas Bloom published supporting remarks ().
Economist argues economics is harder to automate
Matthew Kahn says maths is a stationary search space. Economic behaviour has no fixed parameters, so the Lucas critique limits AI trained on past data ().
Study links AI use to research-assistant work
In Johan Fourie's revised paper, working papers that disclose AI use credit more human assistants than other papers by the same author. The difference comes from coding and data work ().
Economist publishes top-five journal audit
Refine's pipeline plus human review found serious errors in 10 of 100 peer-reviewed papers (;).
Tyler Cowen rounds up AI-versus-human studies
His summary includes AI-written papers with high severe-error rates and AI doing better in crowded research areas ().
Capabilities and access
- Maths: OpenAI has not named the model. It is described only as an internal general-purpose reasoning model (TechCrunch).
- Auditing: Refine says it used "frontier AI models", with a second, different model re-checking serious flags. It does not name the models (Refine).
- Paper generation: Project APE at the Social Catalyst Lab uses AI agents to write complete policy-evaluation papers (arXiv 2603.16744).
Read the full section
- Maths: OpenAI has not named the model. It is described only as an internal general-purpose reasoning model (TechCrunch).
- Auditing: Refine says it used "frontier AI models", with a second, different model re-checking serious flags. It does not name the models (Refine).
- Paper generation: Project APE at the Social Catalyst Lab uses AI agents to write complete policy-evaluation papers (arXiv 2603.16744).
Technical analysis for researchers and developers
- Audit pipeline: AI flagged 5,421 issues (about 54 per paper). A separate AI checked them, then experts reviewed. 77% of flags held up fully.
- Reproducibility: The audit did not use replication packages. Refine says the true error rate is therefore probably higher.
- AI-written papers: APE papers lean heavily on difference-in-differences. About 74% mention it repeatedly, against about 35% of NBER working papers.
Read the full section
- Audit pipeline: AI flagged 5,421 issues (about 54 per paper). A separate AI checked them, then experts reviewed. 77% of flags held up fully. Only 14% bore on major results. The serious errors fell into three groups: misstated headline numbers (4 papers), missing or inconsistent supporting results (2), and failed proofs or assumptions (4) (Refine).
- Reproducibility: The audit did not use replication packages. Refine says the true error rate is therefore probably higher.
- AI-written papers: APE papers lean heavily on difference-in-differences. About 74% mention it repeatedly, against about 35% of NBER working papers. Results also vary a lot between agents making different analytical choices (arXiv; Causal Inference substack).
Claims and evidence
- 10% of top-five papers have a serious error. Refine reports this from its own pipeline.
- Cowen puts the total at 4,158 issues. Refine reports 5,421 (Marginal Revolution).
- 842 of 1,000 APE papers had severe errors. This is Cowen's summary of Yanagizawa-Drott and Willner. The primary paper was not checked.
Read the full section
- 10% of top-five papers have a serious error. Refine reports this from its own pipeline. Refine sells review software, and no independent replication was found (Refine).
- Conflicting count. Cowen puts the total at 4,158 issues. Refine reports 5,421 (Marginal Revolution).
- 842 of 1,000 APE papers had severe errors. This is Cowen's summary of Yanagizawa-Drott and Willner. The primary paper was not checked.
- Erdős disproof. Credible: Bloom backed it after calling an earlier OpenAI claim a misrepresentation, and Gowers would recommend it for publication (TechCrunch; The Conversation).
Context and prior work
- In October 2025, OpenAI's then vice-president Kevin Weil claimed GPT-5 had solved 10 open Erdős problems.
- A Google DeepMind team reportedly resolved nine lesser Erdős problems. This comes from search-result summaries only and was not checked in a primary source.
- Alexis Akira Toda (Emory) finds AI does not yet reliably match the human judgment needed to locate flaws in theory papers (Marginal Revolution).
Read the full section
- In October 2025, OpenAI's then vice-president Kevin Weil claimed GPT-5 had solved 10 open Erdős problems. The solutions turned out to be already in the literature, and he deleted the post (TechCrunch).
- A Google DeepMind team reportedly resolved nine lesser Erdős problems. This comes from search-result summaries only and was not checked in a primary source.
- Alexis Akira Toda (Emory) finds AI does not yet reliably match the human judgment needed to locate flaws in theory papers (Marginal Revolution).
Limitations, safety and contested findings
- Scale and novelty of the maths result: The constructions beat grids only at around 10^2,000,000 points.
- AI-written economics: Agents p-hack and default to a narrow set of methods (Causal Inference substack).
- Novelty: Michele Zampa (Geneva Graduate Institute) finds AI papers score well in crowded fields, while human papers perform consistently regardless of field.
Read the full section
- Scale and novelty of the maths result: The constructions beat grids only at around 10^2,000,000 points. Gowers notes that many of the ideas needed were already in the literature (The Conversation).
- AI-written economics: Agents p-hack and default to a narrow set of methods (Causal Inference substack).
- Novelty: Michele Zampa (Geneva Graduate Institute) finds AI papers score well in crowded fields, while human papers perform consistently regardless of field. Cowen summarises the human edge as ideas, not execution (Marginal Revolution).
- Reproduction: Benjamin Kohler (ETH Zürich) had AI reproduce 48 social-science papers. Discrepancies were blamed about equally on AI and humans, and stronger models tilted the balance in AI's favour (same source).
Business and practitioner implications
- Journals, central banks and consultancies: AI pre-review can catch misstated headline numbers before publication, at roughly 54 flags per paper to triage (Refine).
- Research staffing: AI is taking over coding and data-cleaning tasks once credited to junior research assistants (arXiv).
- Builders of auto-research tools: Narrow method choices, p-hacking and agent-to-agent variance are the main credibility risks.
Read the full section
- Journals, central banks and consultancies: AI pre-review can catch misstated headline numbers before publication, at roughly 54 flags per paper to triage (Refine).
- Research staffing: AI is taking over coding and data-cleaning tasks once credited to junior research assistants (arXiv).
- Builders of auto-research tools: Narrow method choices, p-hacking and agent-to-agent variance are the main credibility risks.
- Market facts: The reviewed sources contain no market data tied to this story.
The source trail.
Sources (10)
AI is swallowing maths. Is economics next?
Feed excerpt only.
economist.com