Skip to content
Interface language
The Hechinger Report·(Date only in original)Featured

Why Did Two AI Studies Find a Positive Association in One Case and a Negative Trial Effect in the Other?

Same researcher, opposite results

Source image: Why Did Two AI Studies Find a Positive Association in One Case and a Negative Trial Effect in the Other?
Source imageThe Hechinger Report ↗
Reading guide

English translation

A reporter juxtaposes two October working papers involving the same researcher: use of a middle-school math platform in Arizona was positively associated with MAP scores, while a university randomized students by instructor to receive a course AI assistant and found lower grades and engagement with the learning platform. The tools, educational levels, and comparison methods differ, so the findings cannot directly cancel each other out or be generalized as a single overall effect of AI.

Why read this

Use the contrasting results to practice reading research: first distinguish an association with use from randomized assignment, then examine implementation conditions, engagement, and outcome measures.

Sources and editorial checks

Editorial check:

Source type
Preprint · not peer reviewed
Scope
Both studies were implemented in the United States; the report is from an independent U.S. education news outlet.
Study design
The middle-school study used a weighted propensity-score observational design; the university study was a teacher-level cluster-randomized trial.
Population and context
Middle-school students in one Arizona, U.S., school district; undergraduates across disciplines at one public U.S. university.
Sample
6,065 students in the middle-school analysis; 2,379 students and 30 instructors in the university trial.
Intervention and comparison
Comparison of high-use Goblins users with low-use or non-users; university classes were randomized by instructor to receive or not receive access to the AI assistant.
Main findings
Using the platform at least two problems per week was associated with MAP math scores about 0.11 standard deviations higher in middle school; in the university study, the course AI-assistant access group had final grades 0.37 standard deviations lower and learning-platform engagement 0.90 standard deviations lower.
Outcomes and uncertainty
The middle-school estimate cannot rule out unmeasured confounding such as motivation; the university findings depend on course grading and the coverage of platform logs and cannot be generalized to all AI tools.
Funding and conflicts of interest
The Goblins paper discloses funding from Accelerate and collaboration with the platform team; funding and conflict-of-interest disclosures for the university paper need to be checked in a later version.

Limitations

  • Both are working papers and have not completed formal journal peer review.
  • In the middle-school paper, 13% of platform users could not be matched to administrative records, and the analysis specification differed from the preregistered plan.
  • The university trial's course grades were not a common external test, and AI use outside this tool could not be fully tracked.

Editorial analysis

Original analysis based on the sources listed. Research findings, policy goals, and editorial judgment are identified separately; original rights remain with their sources.

Same Researcher, Two AI Studies: What Explains the Difference

On November 2, 2026, The Hechinger Report covered two October working papers under a headline emphasizing “the same researcher, opposite results.” That headline can invite a simple verdict for or against AI. The more useful part of the coverage is its step-by-step separation of the two studies' populations, implementation, and identification strategies. The first examines the association between use of the Goblins math-practice platform and achievement in one Arizona school district; the second reports a randomized trial of making a generative AI assistant available within courses at the University of Maryland. Jing Liu is an author on both papers, but they are not repeat trials of the same tool, nor positive and negative replications in the same population.

What did the first study compare?

Goblins lets teachers create classes and assign math problems. Students can type, draw, or speak to get feedback. The study covered middle-school math instruction in one district during the 2025–26 school year and analyzed 6,065 students; about half met the threshold of solving an average of two problems per week. Using observable information such as prior achievement and background, the researchers applied a weighted propensity-score design to compare students above the usage threshold with those who used the platform less or not at all. The abstract reports that high use was associated with MAP math scores 0.11 standard deviations above the district average; the estimated association was larger in the highest-use group. The key word is “associated.” Teachers may assign practice because students are struggling, while student motivation, family support, and platform use may all affect achievement. Weighting can balance recorded differences but cannot eliminate unmeasured ones, so 0.11 should not be described as a gain caused by the platform. The paper also reports that about 13% of platform users could not be matched to district administrative records, further limiting the population that could be analyzed. The study was funded by Accelerate and discloses collaboration with the platform team; those relationships should be presented alongside the results, but do not by themselves invalidate the study.

Why can the second study speak more directly to causation?

The university study recruited 30 instructors and 2,379 undergraduates at a U.S. public university. Researchers grouped courses or similar courses, then randomly assigned whether instructors' classes could use an AI assistant integrated into the course platform. The tool could answer using course materials. The comparison was between classes offered the assistant and classes not offered it, rather than requiring every student to use it in the same way. The original abstract reports that courses offered the assistant had final grades 0.37 standard deviations lower and learning-management-system engagement 0.90 standard deviations lower; the estimated grade loss was greater for first-generation college students. Randomization makes a causal interpretation of the comparison under this implementation more plausible, but execution still matters. The paper says that three instructors who were not enabled to use the tool were excluded from the analytic sample; grading and opportunities for online activity may also have differed across classes. Course grades were not a common external test, platform logs covered only visible learning activity, and use of other AI tools outside the platform could not be fully tracked. The findings therefore support caution that “embedding an assistant in courses may displace existing learning engagement”; they do not show that every AI tutoring tool reduces learning ability.

Why might the directions differ?

The educational level, subject, tool task, and instructor role all changed between the studies. The middle-school platform tied practice problems and feedback to teacher-assigned math activities and primarily observed sustained users. The university assistant appeared as an entry point to course materials, and the trial identified the effect of offering access. The first study may have attracted students who were already willing to practice; the second may have changed how students engaged with a course. The report raises possibilities such as students shifting to external chat tools, but the available data did not test each mechanism. An association study and a randomized trial should not be treated as equally strong votes, nor averaged into “AI has no effect.”

How to use this in teaching decisions

If a school wants to try AI, it should first specify the task it hopes to improve: helping students complete more practice, explaining course material, or improving the quality of questions. For each use, define a comparison group, records of actual use, an independent learning outcome, and possible activities that might be displaced. In particular, distinguish “tool made available,” “someone used it,” and “learning occurred”: launch rates are not engagement, and engagement does not automatically mean higher achievement. Teachers should check whether feedback keeps students thinking or simply helps them get an answer faster. For now, the more defensible conclusion is that the implementation context and measurement choices can change the direction of the results we observe, so schools need small, testable judgments in their own classrooms. Both papers are working papers; their later versions and peer-review status still need monitoring.

Source: The Hechinger Report · hechingerreport.org