AIは子どもの学びを妨げているのか？（#1036）

This is episode number 1036 on whether AI is holding kids back from learning. Welcome back to the Super Data Science Podcast. I'm your host, John Krohn. Today's episode tackles a question that parents, teachers, and anyone who cares about the future workforce are asking. That's, does AI stop children from learning? Let's start with how widespread student AI use has become. A survey last year by the ed tech firm Chegg found that 80% of undergraduates in wealthy countries were using AI in their studies, and more recent polls in Britain and Germany put that figure at 93 to 94% of students. Teachers, unsurprisingly, report grading stacks of suspiciously similar essays that look like they came out of Chad GPT. Until recently, though, rigorous evidence on what all this AI use does to learning has been thin. That's changing. And the data released over the past few months tell a story that's more nuanced and more useful for

even us as adults than either AI is rotting kids' brains or AI is the best thing to happen to education since the printing press. Those are two extremes. All right, let's tease them apart. The biggest new study comes from researchers at Stockholm University and the University of Hong Kong. Published in June, it uses two and a half years of data on nearly 30,000 Chinese students in grades 7 to 12, so roughly ages 12 to 18. China makes for a fascinating natural laboratory because AI adoption among students there has been fast. About 80% of students reported using chat bots like DeepSeek or ByteDance's DuBao, while the remaining 20%, just a fifth, did not. Because students started using AI at different times, the researchers could exploit that staggered adoption in what's called a difference in differences design, comparing how each student's outcomes changed after adopting AI against the trajectories of students who hadn't adopted

yet. That's a big step up from the typical one-off correlational snapshot that we get in big studies of human behavior. The data cover monthly closed book exams, high school and college entrance exams, and homework scores and completion times across nine subjects. Here's what they found. After adopting AI, students' homework scores rose by 18 percent, and the time they spent on each assignment fell by about 30 percent, from an average of 64 minutes down to 45 minutes. That sounds like a productivity miracle! But monthly exam scores, of course, fell by 20 percent within six months, and the damage compounds. On high-stakes entrance exams, scores fell by 18 percent and 24 percent, with the full penalty emerging only after about two years. So as you might expect, it sounds like AI is not so much a productivity miracle as being used to cut corners on homework. That leads me to quite an interesting finding. Homework scores used to be a solid predictor of exam performance. With AI in

the mix, however, that relationship has flipped so that the students posting the highest homework scores are now more likely to do worse on exams presumably because AI is on average being abused to race through homework as opposed to being used as a tool to learn If you a teacher using homework grades as a signal of who understands the material that signal is now broken The losses were largest in social sciences followed by STEM and then languages and the losses were especially large for junior students, for high achievers, and for boys. So should schools ban AI like they have recently done in New York City where I live? Well, not so fast, I'd say. The learning losses were concentrated among roughly 80% of AI users whose behavior looked like homework outsourcing. Exceptionally short completion times paired with high homework scores. Meanwhile, among AI users who spent between 50 and 65 minutes on homework about the same as non-AI users, exam scores were effectively identical to those of non-users, even though

their homework scores were significantly higher. In other words, those 20% of students were using AI as a learning tool, presumably to do things like explain concepts or get unstuck rather than to produce answers wholesale. This kind of pattern, it isn't unique to China. A field experiment with nearly a thousand high school math students in Turkey, led by a researcher at the Wharton School in Pennsylvania, gave some students a standard GPT-4 chat interface and others a version called GPT Tutor with safeguards in that GPT Tutor designed to protect learning. Each GPT Tutor prompt ran to more than 500 words compared with about 50 words for the standard version. During practice sessions, performance improved by almost 50% with the standard GPT-4 and by 127% with GPT Tutor. But on the exam that followed, without AI, students who had used the standard version scored 17% worse than students who had had no AI access at all, while the GPT tutor groups' results were statistically indistinguishable from the

control groups. The control group, of course, not having access to AI at all. So having the standard GPT-4 improved performance in practice sessions, but having this program GPT-Tutor designed for learning, students performed much better. And then on the exam that followed, people who had access to normal GPT-4, they underperformed relative to the control group, but the folks with the GPT tutor did not. So yeah, it seems like there's something to potentially work with here, or at least it's not all bad news with AI. Regular listeners will already be aware that I'm obsessed with Anthropik's Fable 5 model, and it has taken over in my working life. I'm writing a technical book that includes LaTeX files, mathematical notation, Python code examples, and Fable 5 and Cloud Code handles requests I make across whole chapters with accompanying Jupyter Notebooks end-to-end. Work that a few short months ago would have been dozens of separate requests with way more manual fiddling required.

With Fable 5, it just works, essentially like magic, first time. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude slash superdata That claude slash superdata And check out Claude Pro which includes access to all of the features mentioned in today episode Claude.ai slash superdata. One interesting side effect from this Turkish study is that students who use AI were overly optimistic about how much they'd learned. Even the high achievers were overly optimistic. So weaker skills paired with greater confidence is a dangerous combination, but it's unsurprising that the AI users would have inflated egos given how flattering LLMs tend to be to us. I'm sure you're used to that experience. So far, all I've

discussed is students with AI access at best performing comparably on exams to students with no AI access. But there are some new studies suggesting AI can be leveraged in promising ways. In a randomized experiment, researchers at Middlebury College in Vermont had undergraduates in proctored in-person sessions, learn about an unfamiliar topic and write an analytic essay with or without off-the-shelf generative AI, and then take unaided assessments immediately and one week after. AI access raised immediate test scores by 0.27 standard deviations, which is quite a bit, and those gains persisted a week later. Interestingly, essay quality barely changed while students had AI, but it improved a week later when they wrote unaided. And that delayed gain was larger among augmentation users who had AI explain concepts, whereas automation users who had to generate text saw their short-run gains vanish once the AI was removed. The mechanism for all this? Students with AI shifted time away from drafting text and toward reading and searching for information,

and they reported enjoying learning more. There's additional good news. At Harvard, a randomized study of nearly 200 introductory physics students published in June of last year found that students learned more than twice as much from a custom AI tutor as from an active learning classroom and in less time. That tutor was no generic chatbot. It was fed the lecturers' own solutions, and the researchers instructed it to keep responses to a few sentences to avoid cognitive overload. And in another separate study in Nigeria, a World Bank randomized trial of a six-week after-school program using Microsoft Copilot powered by GPT-4 improved scores by 0.31 standard deviations, again, quite a bit, equivalent to one and a half to two years of business-as-usual schooling. Wow. And this outperformed, this GPT-4 randomized trial solution outperformed 80% of educational interventions that were evaluated through randomized trials in the developing world. The World Bank team behind that study is careful to note that the results reflect the full package of structured prompts,

curriculum alignment, teacher support, and peer learning, plus the AI itself. So that intervention was way more than just plopping kids down in front of a chat GPT session. So back to my question at the top of this episode. Does AI stop children from learning? The data suggest it depends on how AI is used. Learning requires effortful thinking. Cognitive scientists call this desirable difficulty. When AI removes that effort by producing the answer before a student has wrestled with the problem the homework looks great and the learning doesn happen When AI is used to explain concepts offer hints and give feedback it can be an outstanding tutor Given all that I've got three practical takeaways from all this for you. First, if you're a parent or educator, time on task looks like a useful warning signal. A student whose homework quality jumps while homework time collapses is probably outsourcing to AI. Second, tool design matters.

The Turkish, Nigerian, and Harvard results that I talked about in this episode all point toward AI configured to tutor rather than to just answer. So it's worth turning on the study mode or the learning mode that many AI platforms now offer. Third, assessments completed without AI, such as in-class closed-book exams, become more important now, not less, because homework grades no longer tell you what a student knows. This also makes me think of new testing methods that might make sense. For example, Professor John Rennie, with whom I co-instruct an introductory course on AI at Tulane University in New Orleans. Professor John Rennie is the co-creator of an AI tool called OralExam.ai. You can check that out online. And it has students engage in a bespoke, automated conversation about their written work after they submit it to the platform. John will be my guest on this show in a few weeks. So look out for more on oral exam.ai and keep an eye out for more clever AI powered educational solutions to AI created problems in the near future.

Finally, as might have been obvious to you as you listen to this episode, these lessons from the studies I covered in this episode aren't only for kids. Every one of us using AI at work faces the same tradeoff. Finish the task faster today or build the skill that lets you tackle harder tasks tomorrow. My guidance would be to try the problem yourself first. Ask the model to explain its reasoning and use AI to augment your thinking rather than replace it. Use it that way. AI may be the most powerful learning tool humanity has ever had. Go put it to work. All right, and haven't done this in a while, but it's time to check on our Apple podcast reviews and see if anyone has made any new ones. And we do have some. So we have one here titled Great Podcast. From the very clever and mysterious user named User21309. And User21309 says, love the work you do on this podcast. Returning to this community after years and super happy to see how much it has grown.

We're super happy to have you super listening. Thanks for all the recent ratings and feedback on Apple Podcasts, on Spotify, and all the other podcasting platforms out there, as well as for likes and comments on our YouTube videos. bonus points if you leave written feedback, which you can do on Apple podcasts. If you do that, I'll be sure to read your feedback on air like I did today. At this time, I really only see when people do that in the US, I think. But someday I will look around the world and see where else I can find reviews. All right, that's the end of today's episode. Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon. you