🔍 Read the full analysis: ChatGPT Boosted Consultants’ Data Science—Then The Gains Faded on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A randomized trial at Boston Consulting Group found consultants using GPT-4 performed better on three data science tasks than a control group using online resources. After ChatGPT was removed, researchers found no evidence that the treated group performed better on technical questions, suggesting the observed gains did not establish lasting unaided skills.
A randomized trial involving consultants at Boston Consulting Group found that those using ChatGPT scored higher than a control group on three data science tasks, but researchers found no evidence of an advantage when participants later answered technical questions without the chatbot. Published October 26 in Nature Human Behaviour, the study suggests that AI can help workers perform tasks beyond their usual expertise without showing that they learned to do those tasks independently.
In the trial, consultants received 15 to 20 minutes of ChatGPT training and used GPT-4. The control group received instruction on searching the web, including resources such as Google, Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: Python coding, statistical analysis or prediction. The study was preregistered before data collection.
The researchers reported that the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. Those figures describe differences between the study groups on the respective tasks. The comparison benchmark was the average performance of 44 BCG data scientists, who completed the tasks without AI; the consultants using ChatGPT remained below that benchmark on average across all three tasks.
The main analysis included 487 consultants who submitted work on both assigned tasks, out of 986 allocated to the study. Afterward, participants answered five technical questions about subjects including coding, probability and machine learning without ChatGPT. The authors report no evidence that the ChatGPT group outperformed the control group on this unaided test. They also found that ChatGPT users were more confident that the system could solve problems it could not, a result the paper describes as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Performance Is Not Independent Skill
For employers, the distinction is between AI-assisted output and a worker’s ability to reproduce that work without assistance. The trial found sizable performance differences while the tool was available, but did not find a corresponding advantage on its later unaided questions. That limits what the results can establish about lasting training or learning.
The finding matters for companies deciding how to train, assign and review work. A chatbot may allow employees to tackle tasks outside their existing expertise, while leaving managers with the need to check accuracy and judge what workers can do independently. The study does not show that AI never teaches skills; it reports that this particular intervention did not produce detectable gains on the later test used by the researchers.
Popular right now
KitchenAid 3-Piece Pasta Roller and Cutter Set for Stand Mixers
eufy E25 Robot Vacuum and Mop with HydroJet Roller Mop, 20,000 Pa Suction
Ninja Flip Toaster Oven and Air Fryer SP151
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment took place at BCG in late March and early April 2024, with data collection beginning after preregistration on March 13. Most participants, according to the authors, reported no coding experience. The main sample of 487 workers completed two of the coding, statistics and prediction tasks.
Researchers designed the assignments with BCG data scientists and OpenAI’s Economic Impacts Research team. The tasks were constructed so ChatGPT could offer useful guidance but could not independently provide correct answers to every question. The study also compared participants’ work with that of 44 BCG data scientists, providing an expert benchmark separate from the comparison between the ChatGPT and control groups.
The paper’s title, “Generative AI and the temporary upskilling of knowledge workers,” reflects the authors’ interpretation of the results: AI can extend what some workers do while it is available, but the experiment did not find evidence that the benefit carried over to the subsequent unaided test. The trial examined a specific tool, short training and set of tasks, rather than every kind of workplace learning.
““We find no evidence” that the ChatGPT group performed better on the technical questions completed without the tool.”
— The study authors
Limits of the Unaided Follow-Up
The later assessment consisted of five technical questions, not a repeat of the full 90-minute work tasks. The results therefore do not establish whether participants retained particular methods or knowledge that the short test did not measure. The study reports no evidence of an advantage on that assessment; that is not the same as proving that no participant learned anything.
The experiment also covered one consulting firm, a short training session, GPT-4 and three designed tasks. It does not by itself establish how results would differ with longer instruction, repeated workplace use, other AI systems, different occupations or a different way of measuring learning. The authors’ finding about confidence in ChatGPT’s abilities was exploratory, so it should not be treated as a definitive measure of overconfidence across workers.
Although 986 consultants were allocated to groups, 487 submitted both tasks and made up the main analysis sample. The supplied study summary does not explain all reasons participants did not complete both assignments. It also does not establish the longer-term effect of using AI at work or whether later training might turn tool-assisted performance into durable skills.
What Workplace AI Studies Must Test
The immediate next step for employers is to evaluate both the work produced with AI and what employees can do without it, rather than treating a stronger assisted result as proof of independent skill. The trial’s comparison suggests that those are separate outcomes: participants performed better on tasks with GPT-4, while researchers found no evidence of a group advantage on the later unaided questions.
Further research would need to test whether longer or repeated training produces lasting learning, how performance changes across jobs and tools, and whether workers can reliably identify errors in AI-supported work. The study does not report a follow-up milestone for these participants. Its findings are a measured result from one experiment, not a final assessment of AI training across workplaces.
Key Questions
What did the BCG study find?
Consultants using GPT-4 scored higher than a control group on coding, statistics and prediction tasks. The reported differences were 49, 20 and 17 percentage points, respectively. The ChatGPT group remained below the data scientist benchmark on average.
Did ChatGPT help the consultants learn data science?
The study found no evidence that the ChatGPT group performed better than the control group on five technical questions completed without the tool. That result does not prove participants learned nothing; it means the researchers did not detect an advantage on that follow-up assessment.
How many consultants were included in the main analysis?
487 consultants submitted work on both of their tasks and formed the main analysis sample. A total of 986 consultants were allocated to the study groups.
Did the ChatGPT group match the data scientists?
No. The ChatGPT group scored higher than the control group on all three tasks but remained below the average of 44 BCG data scientists across the tasks overall.
Does the study show that AI cannot teach skills?
No. It tested a particular tool, short training and set of tasks. The findings show that this trial did not detect better performance on its subsequent unaided questions; they do not settle what longer training or other forms of AI-supported learning could achieve.
Source: ThorstenMeyerAI.com
