Link: https://hechingerreport.org/proof-points-ai-education-meta-analysis/

Studies synthesizing the evidence for AI in education are themselves unreliable — and the peer-review process failed to catch the problems.

“This is not trustworthy evidence,” said Justin Reich, an associate professor at the Massachusetts Institute of Technology and the director of MIT’s Teaching Systems Lab, who hosted a webinar on Oct. 20 to discuss the problem of the growing number of positive AI meta-analyses. “All of these meta-analyses are flawed.”

A forensic audit of 14 of the 39 meta-analyses has found serious conceptual and statistical flaws. Appearing in journals by prominent publishers, including Springer Nature, Wiley, SAGE and Taylor & Francis, these 14 papers have accumulated more than 3,000 citations and continue to rack up 200 citations a month, according to Google Scholar — implying that AI’s effectiveness is settled science. One meta-analysis has been retracted, almost a year after its May 2025 publication in a Nature journal after two Norwegian scholars lodged a complaint.

The new audit was conducted by Patrick O’Neill, a community college professor at the Bloomington, Indiana, campus of Ivy Tech, who examined a sample of the underlying studies and the statistical methods used to combine their results in meta-analyses. O’Neill found recurring patterns of errors, some of which he said could easily have been detected just by reading the study abstracts.

From the abstract of the actual report:

"The audit found that none of the examined meta-analyses provided a valid basis for the claims they advanced: none had a coherent construct, none sufficiently assessed publication bias, and all had severe heterogeneity. Statistics were misapplied, miscalculated, and misinterpreted. A majority of the randomly vetted primary studies were problematic, most commonly because the outcome measured didn’t match the meta-analysis."

For example, it mentions further in that of the 49 studies examined, six did not have a control group.

lol, it would take decades to get any decent meta-analysis. Anyone promoting that shit now is some sort of scammer, including researchers trying to get in early on the citation train.