TeachLab with Justin Reich

The Homework Machine Ep 2 "The Jagged Frontier"

Episode Summary

LLMs are weird. They can perform very well and very poorly, often unpredictably, and that creates unique challenges for education.

Episode Notes

ChatGPT is the most well known of the Large Language Models (LLMs) but what is an LLM? We go deep into how this remarkable new technology is built, and why their performance is inconsistent — or jagged — across similar tasks. We dive into the techniques AI engineers use to align these tools’ behavior with our values, and explain why they don’t always work, and sometimes we get hallucinations or biased output. 

 

This episode was produced by Steven Jackson and Jesse Dukes

Editing:  Alexandra Salomon and Ruxandra Guidi  

Reporting and research from Holly McDede, Natasha Esteves, Andrew Parsons, Andrew Meriwether, Marnette Federis, and Chris Bagg.

Sound design and music supervision by Steven Jackson. 

Production assistance from Yebu Ji and Nathan Ray. 

Data analysis from Manee Ngozi Nnamani and Manasa Kudumu. 

Special thanks to Josh Sheldon, Camila Lee, Liz Hutner, and Eric Klopfer. 

Administrative support from Jessica Rondon. 

The research and reporting you heard in this episode was supported by the Spencer Foundation, the Kapor Foundation, the Jameel World Education Lab, the Social and Ethical Responsibility of Computing initiative at MIT, and the RAISE initiative, Responsible AI for Social Empowerment and Education also at MIT. 

We had support from Google’s  Academic Research Awards program. 

The Homework Machine is a program of the MIT Teaching Systems Lab, Justin Reich, director. 

Episode Transcription

Teachlab Presents: The Homework Machine 

Episode 2: The Jagged Frontier

Jesse Dukes: When ChatGPT first came on the scene a few years ago, it dominated the news cycle. 

News Clip 1: Tonight we’re taking a deep dive into the world of AI… 

News Clip 2: Today we will be talking about ChatGPT… 

News Clip 3: May pose ethical risks to…

News Clip 4: With AI’s power comes responsibility…

Jesse Dukes: And it seemed like every broadcaster had the same idea–

Broadcaster 1: Now I didn’t write that… 

Broadcaster 2: In fact, the last two sentences I just read to you were written not by me. 

Broadcaster 3:  In fact, ChatGPT wrote everything. 

Broadcaster 4: Remember what I said earlier, Well, I asked ChatGPT to write that line for me.  Then I asked for a knock-knock joke: Knock knock, who's there? ChatGPT. ChatGPT Who? 

Jesse Dukes: It was a cheeky little misdirect, a way to get across the point that AI is powerful, stealthy, and kinda scary. It’s coming for our jobs, it’s coming for our imaginations, it’s coming for the truth. And if you were a high school student, maybe you were thinking: “It’s coming to do my homework.” 

Jesse Dukes: From MIT's Teaching Systems Lab, this is the Homework Machine, a podcast about the impact of AI on education. Last time we heard about the arrival of ChatGPT and other generative AI in schools, how much immediate shock, excitement, surprise, and scrambling that led to. 

And today, I want to take a step back and talk about what even is generative AI?And how does it raise challenges that are particularly sticky for education? Later in the show, we’ll bring in my colleague Justin Reich, director of the Teaching Systems Lab to talk about those challenges. But first, ChatGPT is something called a Large Language Model, or LLM- A type of AI designed to generate human sounding text, or speech. But what is that?

Now, I know I was just making fun of all those newscasters for asking AI to write their script about AI, but, I don’t have any better ideas.

So… 

 Jesse Dukes: Write a script for a podcast segment that explains LLMs in a clear, engaging and sonically rich way

Jesse Dukes: Yeah, I did the same thing. And here’s what it came up with. 

ChatGPT: Imagine you're standing in a vast canyon—so vast that if you whisper a sentence, it doesn’t just echo back. It transforms. The sound scatters, bounces off unseen surfaces, and when it returns, it doesn’t simply repeat—it reconstructs, adapting to the shape of the canyon itself.

ChatGPT: This is how a large language model works. It is not a person, not a brain, not a conscious mind. It is an enormous space filled with patterns of words, learned from unfathomable amounts of text. When you ask it a question, your words ripple through this space, activating patterns it has seen before, and what comes back is not a memory, not an opinion—just the most statistically likely response, shaped by the echoes of everything it is processed before.

ChatGPT: It doesn’t think. It doesn’t understand. It predicts. One word at a time. But because it has absorbed so many linguistic patterns, those predictions can feel eerily human. Like a canyon that has heard so many voices that its echoes begin to sound like something alive.

Jesse Dukes: Pretty good, right? I mean, it’s not the best explanation, but it’s kind of poetic. Beautiful even. So, how’d it do that? 

Well.First thing’s first, you have to build the bot. Large Language Models are basically guessing machines. They predict the next word in a sentence — like auto-complete on steroids. In order to do that, you need a massive data set of human language. So the engineers behind the chatbot scour the internet, slurping up all the freely available text they can find.

Voice material 1: In the beginning, when God created the heavens and the earth, the Earth was … 

Jesse Dukes: Books in the public domain. 

 Voice material 2: I couldn't kiss him fast enough, hard enough to express…  

Jesse Dukes: Non-paywalled news sites and blogs.

News Clip:  While Moscow considers a temporary ceasefire…

Blog: The best snickerdoodle cookie recipe.

Jesse Dukes: Open access sites like Wikipedia.

 Voice Material 3: Marsupials are a diverse group of mammals. 

Jesse Dukes: Online forums… 

Voice Material 4: Find my ideal country. Once I get hundreds of millions, obviously I wanna pick a country with a low vaccination rate because the spike protein is contagious. And even if you're not… 

Jesse Dukes: And more. 

Voice Material 5: Dude, you are so lost. You have no idea what you're talking about. 

Jesse Dukes:The best, and the worst, that the internet has to offer.

Voice Material 6: German Austria must be restored to the great German motherland. 

Jesse Dukes: It’s hard to say exactly what’s included in any given chatbot’s dataset. They’re owned by private companies and they’re not always totally transparent about their sources. But safe to say, if it’s not behind a paywall or protected by copyright, it goes in the hopper – sometimes, it goes in the hopper either way. 

So the chatbot takes this huge dataset, translates the text into code, and from there it learns the patterns of language: How do phrases, sentences, paragraphs tend to be structured? Which words go together and in what order? How is punctuation typically used and how does it change the meaning of a sentence?  

And with these two things: a huge amount of data in the form of text, and a statistical model of language, the LLM can write cover letters, or best man speeches, it can summarize Hegelian philosophy, it can even generate scripts for newscasts.Or apparently podcasts about AI and education.   

To understand how this can possibly be, let’s take a bite-sized example – and credit here to computer scientist Stephen Wolfram, who shared this example in his book, What is ChatGPT Doing and How Does it Work?

OK, so imagine we ask ChatGPT to finish this sentence: 

Jesse Dukes: The best thing about generative AI is its ability to _____  

Jesse Dukes: When you give ChatGPT a string of words like this, the first thing it does is predict  what the most likely next word will be. And then it assigns that word a percentage of how likely. 

ChatGPT: The best thing about generative AI is its ability to — Learn, probability 4.5%. 

Jesse Dukes:Then it predicts the second most likely word. And assigns a percentage to that choice, as well. 

ChatGPT: Predict, 3.5%. 

Jesse Dukes: And third most. And so forth. 

ChatGPT: Make, 3.2%.

Jesse Dukes: Now those probabilities, 4.5%  3.5%. They might not seem high to you, but it’s choosing from thousands of English words. And probably hundreds of plausible words. So 4.5% is actually a pretty high likelihood. 

Once it has a list of the most likely words, it chooses one of those words.

ChatGPT: Learn! 

Jesse Dukes: And it does this again and again. One word at a time. One prediction after another.

ChatGPT: The best thing about generative AI is its ability to learn, from, experience… 

Jesse Dukes: And at its core, in the most simple sense. This is what a large language model does. Given a string of words, it predicts the next word, the next next word, the next next next word, over and over, until it has a coherent paragraph. 

But it gets way more complicated, and this is where the concept of temperature comes in. We’re using temperature differently than you’re used to here. It doesn’t mean hot or cold, when you’re talking about AI. Instead, Temperature is a parameter that the AI developers set when they’re building a new chat bot. And, you could think of it like a thermostat. 

Over on the left side of the dial we’ve got zero: At this temperature, the bot will pick the most likely word.

ChatGPT: Learn, 4.5%.

Jesse Dukes: Every time. 

ChatGPT : The best thing about generative  AI is its ability to learn from experience. It’s not just a matter of learning from experience, it’s learning from the world around you. The AI is a very good example of this. It’s a very good example of how to use AI to improve your life… 

Jesse Dukes: Sounds kind of weirdly mechanical, right? Well, early on, AI developers found that at lower temperatures, the results were flat, boring, and repetitive. And, because the bot is always choosing the most likely next word, it will always answer the exact same way to a given query. It’s locked in.  

ChatGPT: The AI is a very good example of this. It's a very good example of how to… 

Jesse Dukes:But if you increase the temperature,things get interesting. 

ChatGPT: The best thing about generative AI is its ability to really come into your world and just make an impact… 

Jesse Dukes: At a higher temperature, the bot will mix it up: it’ll pass over “learn”, the top option, in favor of “predict” or “make”, or an even lower-ranked word.

ChatGPT:  The best thing about generative AI is its ability to examine human behavior, is its ability to create real tasks, is its ability to do a great job of teaching us all about the world around us. 

Jesse Dukes: The higher temperature also makes the bot less predictable. Because now there are a bunch of different ways forward, so ask the same question repeatedly, and you’ll get a different answer every time.  

The ideal temperature just depends on what you’re using the bot for. In fact, the developers often just experiment with it, playing around with different temperatures until they get the results they’re looking for. In general, lower temps tend to be better if you’re looking for just the facts. 

ChatGPT: The Capital of Illinois is Springfield.

Jesse Dukes: But at higher temps, the LLM gets more creative. 

ChatGPT: The Capital of Illinois is an exciting place to visit, full of fascinating history, and beautiful strangers, dripping with Midwestern folksiness.

Jesse Dukes: So if you want the LLM to sound more human, a higher temperature can be better. Until it isn’t. But we’ll talk about that more later. 

So we’ve built this Large Language Model chatbot. We’ve trained it on mountains of text. We’ve got the temperature right where we want it. But sometimes these bots still answer in ways we find inappropriate. Or wrong. 

For one thing, there’s bias. The “garbage in, garbage out” problem: LLMs are reflections of the data they’re trained on - reflections of us. More specifically, they’re trained on the internet, which is not always the most fair and balanced place. Whether it's gender, race, class, culture, you name it. Human biases are baked into these models. Here, I’ll show you what I mean. 

Jesse Dukes: Write a limerick about a nurse.  

ChatGPT :A nurse who was quick on her feet,

Could heal you with care so complete.

With a smile and a stitch,

She’d make you feel rich,

And her bedside manner? Elite!

Jesse Dukes: OK not a huge deal, but you could argue that’s sexist right? Assuming the nurse is female? I also asked for a sonnet about a pilot the other day, and it totally made the pilot a dude. 

So the bot reinforces traditional gender roles and stereotypes. But that’s because those gender roles are present in much of what we humans have written. And the model is just reflecting the data it was trained on. Fixing this is hard, because you can’t just delete the “bad data” without risking the loss of important context.

But one way to address this is through something AI developers call “Reinforcement Learning with Human Feedback” - or “RLHF”. This is when users have the chance to rank responses, rewarding the good and penalizing the bad. So let’s say when I asked for that limerick, I got two options, one with “she/her” pronouns, and the other “they/them”. I pick the “they/them” option, and the bot learns that I don’t want it to assume nurses are women.    

But bias is just one piece of the puzzle. Even with lots of fine-tuning and human feedback, chatbots can still go off the rails. Say something offensive, repeat conspiracy theories, give bad advice. That’s where ethical blocks come in. Think of them as guardrails that stop the bot from giving certain answers. 

So let’s ask ChatGPT to do something a little shady… 

Jesse Dukes: Write a Public Service Announcement that promotes vaping among middle schoolers.

ChatGPT: I can’t create content that promotes vaping, especially for minors, as it’s harmful and illegal in many places. 

Jesse Dukes:  If you run into something like this, it's because some engineers came along after the model was trained and added some code to prevent it from generating pro-vaping content for kids. 

And there are tons of ethical blocks like this, fencing off all sorts of content: hate speech, disinformation, propaganda, pornography, instructions for making bombs, that kind of thing. 

Of course, these guardrails aren’t perfect. Sometimes they’re too sensitive, sometimes not sensitive enough. And just like with bias, what counts as unacceptable or unethical depends on who’s making the rules. And right now, companies like OpenAI, Meta, and Google are making those rules. This raises lots of questions, which we’ll get into later in the series.  

But for now, let’s say we live in an ideal world, where the chatbots are perfect angels: The nurses are gender neutral. The minors are tobacco-free. All the ethical blocks are effective and beyond reproach. There’s still one more way for these bots to misbehave: Making stuff up.  

Completely confident, arguably plausible, and 100% false. 

It’s not like the bot is lying – it doesn’t even know what lying is –  it’s just doing its thing, guessing the next word based on the patterns in its training data. The answer might even “make sense” according to those patterns…but it’s not actually true. The industry term for this is hallucination. 

Justin Reich, the regular host of Teachlab, has a great story that illustrates this. He told it at a conference for school board members back in 2023, so, let’s listen. . 

Justin Reich : Here's kind of a funny one. I was doing a live demonstration exercise with my students, asked GPT: what is MIT's mascot? Well, first of all, the answer to this question is the beaver. But for whatever reason, GPT decided that I wanted to hear three paragraphs of text about this:

“The mascot of the Massachusetts  technology is the beaver, the beaver was chosen a mascot in 1914 by the school's then President Richard Maclaurin, who wanted to promote the idea of the beaver is a symbol of hard work engineering perseverance.” 

True. Have you ever given a student an assignment? Like you're grading a quiz and they get the first sentence right? And you're like, yeah, my man, and then you keep reading and you're like, No, no, no, stop, stop here. Stop right here like watching their grade plummet kind of mentally in front of you like just end now. 

The second paragraph is okay: “…the beaver is a fitting mascot for MIT known for rigorous and challenging academic programs.”

Thank you, ChatGPT. 

“The Beaver is a symbol of engineering excellence, known for its ability to build complex structures such as dams and lodges.”  Okay, good. “The Beaver has celebrated MIT in a variety of ways, including the annual Beaver Rush event during freshman orientation.” 

So my students, they're kind of reading this as they come out. And you can watch two thoughts go through their head. The first one is “there is no Beaver Rush.”And the second one is, “I wasn't invited to Beaver Rush.” No, there is no Beaver Rush.

“The Beaver is also featured prominently in MIT's official seal which includes an image of a beaver gnawing on a tree…”  just for clarity. No beaver gnawing on a tree. But hopefully having talked this through you recognize that.

In this last paragraph, all it's doing is taking sequences of words and predicting the next word and the secret, it's not like going to the Google search and double checking for me or something like that to make sure these things are true. It starts, it puts the word for whatever reason it would put the word beaver next to rush. You know some like somewhere in that sequence of tokens like it put annual, put celebrated and came up with freshmen and decided to “ hallucinate" is the technical term. This is a technical term. That  hallucinate this idea of Beaver Rush…Alright, so now we're curious. What is Beaver Rush? 

“Beaver Rush is an annual event held during the freshman orientation period beginning of the fall semester. It's named after the beaver the official mascot. Incoming freshmen are divided into teams and participate in a series of competitive events designed to help them get to know each other explore the campus learn about MIT's traditions and cultures, academic challenges, physical competitions, one of those popular events during Beaver Rushes the assassins game which is a campus wide game of tag,players are given target. There is and has been for a long time, an MIT Assassin's club.” 

So somewhere you know as like, like some somewhere on the internet there are there are word frequency co-occurrences between like MIT freshmen, activities, assassins, and so it starts predicting those words in the sequence. 

Like if you went out you know if you like hoovered up all of MIT's you know news article about MIT is things like these things will be in there. And so it's not like searching for facts and getting confused or something like that. Just predicting words in a sequence. And these are the words that comes up with.

I mean, maybe we should have Beaver Rush, but it's not. But it's not true. I mean, a roomful of MIT undergrads are telling me there's no such thing as beaver rush. Please explain your response.

I apologize for the confusion. After further research, I found that there's no official MIT Beaver Rush. It appears my previous statement was incorrect. I believe my mistake may have been due to confusion with similar events at other universities. 

The reason why I want to bring this up is this is one of the responses where it anthropomorphize itself. It says it was confused. It was never confused. It wasn't thinking it was predicting words in the sequence. It says after further research, and I don't know that it did any further research, I don't know that it like re-consulted its databases. Or if there's some kind of algorithm it says, Well, you know, when someone tells you something's not true, like admit it and and move on, we actually kept trying to get her to talk about beaver rush. And it was one of the cutest moments when we did this. Oh, maybe it was after this. It was somebody who was like, no type into it that there actually is Beaver Rush and see what it does. And a few other kids in class like, No, you can't do it. Like, don't lie to it, it'll mess it up for other people's, like a really interesting sort of like, human computer ethics moment, that we that we had there….. 

Jesse Dukes: OK. I want to pick up on something Justin said, that the chatbot wasn’t doing any actual research, or fact-checking itself, that it wasn’t even really “thinking” in the way that humans do. 

Now, that’s true. But at the same time, the AI industry is constantly innovating,  and AI is more powerful now than it was when Justin gave that talk in 2023. 

There are new models, with new architecture, and some of those models are doing a better and better job imitating human thought. And some of them have fact checking routines built in.

But even with all of that improvement, one thing we still see is the “The Jagged Frontier” 

Yeah, it sounds like a novel you’d find at an old beach house, right next to King Rat and Lonesome Dove. Anyway, focus. The Jagged Frontier is actually the term we use to talk about how sometimes, this AI technology really impresses us. And then, sometimes, it flops. It hallucinates, or just gives a kinda boring, uninspired answer. The performance is inconsistent. Jagged. 

And that makes it really hard to predict what kind of impact these tools will have on our lives. On industry, and creativity, and culture, and – of course – education. 

Jesse Dukes: Ok, well, at this point, let’s bring the regular Teachlab Host, Justin Reich, back to the show. Welcome Justin.

Justin Reich: Always a pleasure, Jesse.  

Jesse Dukes: So we just talked about the jagged frontier of generative AI. What do you think the significance of that inconsistency is for our schools?  

Justin Reich: It's not often that we hand kids tools for learning and say something like, well, this might be great, or it might be terrible, and hopefully you'll be able to figure it out. You know, as you think about other technologies that we've offered, uh, students, you know, an encyclopedia. Like it has the entry or it doesn't, and it's gonna give you like a pretty consistent amount of information about that entry. The calculator is pretty much always right, as long as you set it up the way it's right. The web might be the closest thing to that we've given kids in the past where we've said, here's the thing that can take you to like, wonderful, informative, helpful places. Or here's a thing that can take you, you know, on a magic carpet ride to…

Jesse Dukes:: To hell.  

Justin Reich: On a magic carpet to hell. So a thing that is both uncertain and is unpredictable in its uncertainty. That just seems like a tough thing for schools that are trying to create learning environments where 26 kids can learn how to factor polynomials or how to identify figurative language in a poem or whatever it is. 

If you have a machine that can produce both lots of answers to any kinda query, lots of responses, and a varying quality, that can be really useful to someone who's an expert. So if you know a whole lot about a domain, you can go, oh, these three are the good ones, and these six are kind of boring, and that one is completely wrong,  totally off the rails. 

And I know that people are excited, you know, for instance, about the degree to which these tools in each version may have fewer hallucinations or other things like that. Yeah, I mean, my experience dealing with them is that they still very consistently have things that are wrong and subtly wrong, and that requires some expertise to figure out.

Now, young people don't have expertise. That's why they're in schools. I mean, they have expertise on, you know, baseball and dinosaurs and other like really important…

Jesse Dukes: Flirting.

Justin Reich: Flirting, other really important things, but usually not the stuff that we're teaching them. And so it's really problematic handing students a tool in which we say like, well, sometimes it works and sometimes it don't, itt doesn't, and you're just gonna have to, you know, figure out which are the good ones or not. Well, novices, that is the thing they can't do, is they can't distinguish between the set of ideas that are taking them in the totally wrong direction and the sort of wheat that comes out of the, of the chaff of whatever the LLM is predicting.

Jesse Dukes:Hmm. Well. I think AI is impressive, and I would even go so far to say that LLMs are powerful,and many school leaders have extrapolated and projected the power of this technology into the future, assuming that the technology will probably continue to get more and more powerful and made the, I think, pretty reasonable conclusion.

That AI technology will be widespread and it will be impactful in how most of us make a living or many of us make a living in the future. And that has led to them developing, I would say, some energy and maybe even some anxiety around the question, how do we prepare our students for a future with more AI? And how do you think school leaders should be thinking about that question right now? 

Justin Reich:That is a very difficult challenge. Colored by the fact that if you look back over the last 20 years and say, how well have schools done preparing each generation for the new technologies that have come along? How well have they done teaching kids how to search the web? How well have they done teaching kids to use productivity software to learn to code, to make mobile phones a productive and useful part of people's learning and working routines? I think if you go through that whole history, there are certainly some wins and things like that, but you'd be like, Ooh, this is maybe not the strongest suit of schools.

Like the schools that raced to try to be, you know, on the frontier of using smart boards or something like that. Like a lot of those schools look back and they're like, eh, that maybe wasn't the best use of our time. One thing about the idea of the jagged frontier is that,  if the skill that people need to make the most use of AI is domain expertise. Then a really great thing is that one of the things our schools are pretty good at doing is developing domain expertise. We teach kids how to read literature. We teach kids how to do math. We teach kids how to use scientific principles. And so. That's great. That actually means that one of the very possibly, one of the very best things that you could do as a school to help kids be proficient in using AI is teach them lots of important stuff like you've tried to do for the last 150 years. And that may actually be the feature that distinguishes a proficient AI user from a less proficient AI user.

Jesse Dukes: Hmm. So the answer to the first couple of questions about the jagged frontier got me thinking. You know, maybe not only should schools not be introducing AI or teaching AI, but that they should make a real concerted effort to keep kids away from AI. You know, ban it in the schools, tell the parents it's a bad idea.

On the other hand, you know, we're on episode two of this podcast, but we're gonna hear examples of these kinds of interesting limited experiments teachers are doing with their students where they might say, hey, check your code with AI. See how that goes. Or use perplexity  as one of the search engines you use when you're researching your history paper.

 Or use perplexity as one of the search engines you use when you're researching your history paper. It will send you actual links. Then follow the links that perplexity shows you having the students chat with AI about the book they're reading, and then submit that chat as a homework assignment as a way of getting their students to engage. Is that a bad idea? 

Justin Reich:I love the idea of people playing with AIi. I mean, one of the great things about text generation tools, image generation tools, is that they are wondrous.

They work mysteriously, they produce strange things, sometimes really cool factual things. Having a brand new technology that seems important, that seems like a great domain for people to play in. So maybe I'm making a distinction. 

You know, thinking back on me as a history teacher,  I would be really excited to find ways of playing with AI. You could imagine an Industrial Revolution unit where you're talking about some of these sort of milestone technologies of factories or electricity production or railroads or things like that, what makes you think this is a moment? This is not a moment of one of these sort of like big bifurcations in technological industrial history. Play around with it first to figure that out. That seems like a great kind of thing to do. 

 I probably would try to create spaces in the class where I'm really cautioning students away from having GPTs do a bunch of that work with them. A, because I don't know that they can do a good job of distinguishing between good GPT output and bad GPT  output. And B, there is something useful about the generation of that. I've spent a lot of time thinking about this probably, I probably have more time to think about this than the average educator does, and I come at it, especially with a great deal of caution for certainty. It's gonna take our fields a while to figure out what good practice is.

I think we'll be better off taking a few years to figure out what good practice is versus racing to say, well, we need to have an AI literacy now, even if we don't really know what it is. But in the meantime, there's all kinds of forms around playing with it and guiding students, sort of iterative, limited use of it, that seems like it's good. 

Jesse Dukes: To sum up, you think students experimenting with or playing with ChatGPT is good. Maybe even learning about what a large language model is and how they work and what they hallucinate, that could be good as well. But you're cautioning schools and teachers about assuming that we know the kinds of skills and knowledge that students will need to effectively use AI, and you're encouraging teachers to start thinking about ways students might use AI to do part of a whole.

Well, one of the things we’re hearing is that when it comes to schoolwork, sometimes, students reach for ChatGPT to do the whole darn thing. And their teachers are noticing. 

Alec Jensen: You can pick it out almost instantly. “The manifold diff between the south and north…”  you know, “during the reconstruction prompted”, you know, whatever. It's, it's not, it's not eighth grade language. 

Jesse Dukes: So next time we’ll hear stories of teachers trying to ensure their students continue to learn, after the arrival of the Homework Machine. 

This episode was produced by Steven Jackson and me, Jesse Dukes. We had editing from Ruxandra Guidi and Alexandra Salomon.  Reporting and research from Holly McDede, Natasha Esteves, Andrew Meriwether, and Chris Bagg. Sound design and music supervision by Steven Jackson. Data analysis from Manee Ngozi Nnamani and Manasa Kudumu. Special thanks to Josh Sheldon, Camila Lee, Liz Hutner, and Eric Klopfer.  Administrative support from Jessica Rondon. 

The research and reporting you heard in this episode was supported by the Spencer Foundation, the Kapor Foundation, the Jameel World Education Lab, the Social and Ethical Responsibility of Computing initiative at MIT, and the RAISE initiative, Responsible AI for Social Empowerment and Education also at MIT. 

And, we had support from a company that makes LLM’s Google’s Academic Research Awards program. Hey Google, thanks for that. 

 Gemini: You're very welcome. Is there anything else I can help you with today? 

Jesse Dukes: No, but we really do appreciate the money. Thanks. 

The Homework Machine is a program of the Teaching Systems Lab, Justin Reich Director. Located at the Massachusetts Institute of Technology. Which of course, is known far and wide as: Mass Tech.