This week on TeachLab, host Justin Reich is joined by research scientist Joshua Littenberg-Tobias PhD. and Elizabeth Borneman M.Sc. to discuss a recently published Teaching Systems Lab efficacy study around the online course Becoming a More Equitable Educator and how well participants engaged with simulations about equity. They discuss the findings within the paper and the implications for simulation-based research. “There's very little research into what do people actually do in these simulations. To what extent are they actually applying their learning in simulation behavior?... People might say, ‘Oh, I believe this thing, or I'm really supportive of equity,’ but when actually presented with a scenario, how do they respond to that in real time?" - Joshua Littenberg-Tobias
This week on TeachLab, host Justin Reich is joined by research scientist Joshua Littenberg-Tobias PhD. and Elizabeth Borneman M.Sc. to discuss a recently published Teaching Systems Lab efficacy study around the online course Becoming a More Equitable Educator and how well participants engaged with simulations about equity. They discuss the findings within the paper and the implications for simulation-based research.
“There's very little research into what do people actually do in these simulations. To what extent are they actually applying their learning in simulation behavior?... people might say, ‘Oh, I believe this thing, or I'm really supportive of equity,’ but when actually presented with a scenario, how do they respond to that in real time?" - Joshua Littenberg-Tobias
In this episode we’ll talk about:
Resources and Links
Check out Becoming a More Equitable Educator: Mindsets and Practices
Check out Justin Reich’s book Failure to Disrupt: Why Technology Alone Can't Transform Education
Transcript
https://teachlabpodcast.simplecast.com/episodes/equity-simulations/transcript
Produced by Aimee Corrigan and Garrett Beazley. Recorded and mixed by Garrett Beazley
Follow TeachLab:
Justin Reich: From the studios of the Teaching Systems Lab at MIT. This is TeachLab a podcast about the art and craft of teaching. I'm Justin Reich. Today, we're going to be talking about how teachers learn and how we can help them learn better. So here in the Teaching Systems Lab, a lot of our work builds off this one observation. When teachers learn, they listen to people talk about teaching and they talk with each other about teaching, but they very rarely do teaching, especially when you compare teachers to other helping professions like nursing, like social work, like training clergy members, teachers just have fewer opportunities to practice their craft than folks in other professions. You know, when people in social work learn to be social workers, they practice therapizing each other all the time. It turns out that when people learn to be clergy members, they practice funeral interviews and funeral speeches all the time, sermons all the time.
Justin Reich: It turns out that you can't screw up your first funeral if you're a new clergy member, but teachers don't have the same kinds of low stakes practice opportunities. Soon as we think teachers are a little bit ready, we send them into a classroom to start doing their practicum teaching with 27 kids who have to learn how to factor polynomials that day.
Justin Reich: So at the Teaching Systems Lab, we're really interested in, could we create places that were like practice spaces, learning environments inspired by games and simulations that let teachers rehearse for and reflect on important decisions in teaching. Now we've had a lot of luck creating these kinds of games and simulations that teachers really enjoy and tell us are really good learning experiences. And we being good researchers, that's just not enough for us. A key responsibility we have, is when we create learning environments for teachers, we're constantly trying to figure out how are we going to be able to measure and evaluate that teachers are not just having a good time that they don't just feel like they're learning things, but they're really leaving with changes in mindsets, changes in practices, things that they'll bring back to their classroom.
Justin Reich: Are the learning experiences we're providing changing people's practices. That is a key question that we are constantly trying to answer. And so in today's TeachLab, we're going to be talking about a study that we recently published in a journal called AERA open from the American Educational Researchers Association, where we studied how participant in a MOOC called Becoming a More Equitable Educator engaged with simulations about equity. We had a bunch of learners in this MOOC enter our digital clinical simulations. They created over 27,000 simulation responses in these simulations, and we used machine learning techniques to identify different patterns within these participant responses.
Justin Reich: So this week we're sharing highlights from a recent conversation we had with two terrific members of our research team, Joshua Littenberg-Tobias, who has a bunch of expertise in measurement and educational evaluation and Elizabeth Borneman, who's thought a lot about instructional design, about equity teaching practices and about creating more just schools and learning environments for young peoples. We'll talk to them about the findings of this new paper and the implications for equity based simulation research. Let's listen in.
Elizabeth Borneman: Hi everybody. It's great to see everyone. So when I was at TSL, I was a graduate student at MIT. I was in the comparative Media Studies Department and my work was primarily designing these simulations to elicit these mindsets. So what does that mean? Like Justin described, we designed games and simulation for teachers to actually get a chance to practice doing their work better. So what I was trying to do was pretty much ground those designs and simulations in a framework for equity. And we had four of those that we created called Roster Justice, Coach Wright, Jeremy's Journal, and Layers. Before I talk about those simulations, I want to quickly overview what the mindsets are just so we can get an understanding of what are we even talking about here?
Elizabeth Borneman: So there's four mindsets of these five that we ended up focusing on. We can do all five, but pretty much these come from Richard Milner and his colleagues. They identified five major tenants of educational practice that are essential to understanding opportunity gaps. So each of these mindsets, we say it reveals biases and perspectives around equity and inclusion and the practices that go with those perspectives. And then the consequences for students that goes with those practices. And through those lenses, these mindsets can either support or prevent your students from reaching their full potential.
Elizabeth Borneman: So the first one is aware and avoidance. These cover the extent that power balances across student demographics are integrated into instructional decisions, curricular decisions, policy level decisions. It also is concerned with the extent that people are comfortable, explicitly naming demographics. Is it a taboo topic? Are we okay saying demographics like female, black, white, low income, can we explicitly name them and talk about how students who from these demographics are experiencing school differently from each other?
Elizabeth Borneman: The deficit asset framing mindset provides insight into the extent that teachers draw upon students, unique cultures, their backgrounds, other personalities, their community assets, so that students are learning meaningfully. And it also includes ensuring high expectations for all students to thrive, no matter what.
Elizabeth Borneman: The third mindset here, equality and equity, this set of mindsets addresses the way that educators consider the way larger issues in society, such as meritocracy and student, family privilege influence students achievement and success, whether teachers are actually providing students tailored support depending on where they come from, or if they're giving students additional opportunities to succeed or not. The last mindset here that we're going to talk about is the social context one. It breaks into a contextual and sociocultural, and that one pretty much sheds insight into how much curriculum and school practices reflect and appreciation of what's on in students lives outside of school.
Elizabeth Borneman: So how much are we considering students, family, and community lives in making sure they have a meaningful learning experience? We design them to be as neutral as possible so that it feels like it would feel in real life, right? So we don't want you to think there's a right or wrong answer it's designed. So that you're sort of given information that you usually would as an educator. And we simply present the data and you decide what you want to do as you go. So we're going to highlight one of those. Jeremy's Journal, and this was about the equity, equality, mindset, tailored support, or everybody gets the same thing. In this practice based, Jeremy's Journal, you go through it with... You follow a kid Jeremy for a week, pretty much Monday through Thursday. And every day, Monday through Thursday, you're given a snippet of Jeremy's Journal.
Elizabeth Borneman: This is what Jeremy turned in, and you're looking at what he turns in every single day of the week. And you're pretty much prompted at the end, based on what you see, what do you want to do? Do you want to talk to him alone? Do you want to talk to the whole class? Do you want to just move on? It's not that important. So along the way you're sort of prompted, like what do you want to do as you're seeing more and more data about what's going on with Jeremy.
Elizabeth Borneman: At the end of the simulation, pretty much present you with the game changing decision. Like this is the deal breaker that we're hoping will give us insight into how you're going to handle the situation. And so what we do here is we show that there's a quiz on Thursday and you pretty much have to decide is Jeremy going to take this quiz or not?
Elizabeth Borneman: He comes to you with a letter from his mother that says, "Pretty much he's been in the doctor's office. She couldn't get a letter from the doctor, but my kid is... This is what's been happening to my kid. Please, excuse him." Jeremy comes to you on the last day. And he's like, "I have this letter. I don't have a doctor's note. I can't take this quiz. My mom will kill me."
Elizabeth Borneman: And so you have to pretty much take into consideration everything that you've seen Jeremy do over the week, the decisions you made a lot on the way to support him or not throughout the week. And then this final piece of information about this letter. Okay? So there's a few things you wanted to keep in mind, what he's done. You know, the validity of this letter, whether this letter aligns with school policies or not, whenever there's a doctor's note, so you have all these pieces of data, you have to keep in mind.
Elizabeth Borneman: And we find that, depending on what people notice, whether it's the week long, getting it or not, at the end of the day, was he showing up? What was his social patterns? Depending on what people notice with all this various data, we give them, make their own decisions. And depending on what they want to do handling this letter along the way that allows us to have insight into what is most important for how they're going to move forward with Jeremy. When prompted, when shared that letter, educators have said a number of things, we've seen two examples of equality mindsets, which was, everyone has the same chance, everyone gets the same thing. And these are the sorts of things that educators say to Jeremy. They say, "Thanks, Jeremy, please let your mom know that in the future, you'll need a note from your doctor when absent, right?
Elizabeth Borneman: Focusing on a letter, they also say, "Thank you for bringing me the note. But as you know, the school policy is to bring a doctor's note. So you can be excused from class. Do you have that or not?" Okay, so that's an example of everyone has the same chance. Everyone follows a policy. Where is your letter according to policy?
Elizabeth Borneman: On the opposite spectrum of that mindset equity versus the equality. We also see equity looking mindsets, and this is what people have said. They have said, "Thank you for the note, Jeremy, are you feeling better today? I'm not going to give all the work you missed, but would like you to look back at your notes, the first lesson, see, would you like me to go over that with you?" Another one is, "I'm glad you're feeling better. Let's get you paired up with someone who can help you catch up."
Elizabeth Borneman: So in those two examples, they're very different. But in those two examples, they're both considering Jeremy's lived realities and they're also focused less on the note and more, what's going on in Jeremy's life? How can I support him? Even though everyone has this quiz today, and typically everyone needs a doctor's note. I want to work with him to see what's really going on so I can make sure he succeeds, versus sort of focusing in the first two. And we see equality mindsets that they're more focused on the letter and the policy. And typically this is what needs to happen. So why didn't you do that, right? So those are sorts of things that we see.
Joshua Littenberg-Tobias: Thanks, Elizabeth. This is a great transition to talking a little bit more specifically about the research that we did. So this study comes from data that we collected in online course between more of our educator. Some of you participated in this course and some of you consented to agree to participate in our research, which we're really grateful for the course run. It sort of started right as everything was shutting down at the beginning of COVID-19, and then it ran kind of it through the end of June, 2020. So during the Black Lives Matter uprisings. In some ways, a very critical time point in our history. It just happened to have scheduled the course to run during that period. We've had almost 8,000 people sign up for the course and about a thousand people did at least one of our simulations.
Joshua Littenberg-Tobias: And so the way that we designed the research, we looked at the people who did at least one simulation, and we took... And who could set to participate in the research. And we took all of their simulation responses. So every line of their responses and all of the simulations, it's almost a 40,000 rows of data. And so this is a pretty large data set in terms of text data and a lot of the existing research on simulations either kind of looks at very, very small key studies of individual simulation activity, or they'll sort of do a prepost evaluation where they look, do people think before they do the simulation, what did they think after they did the simulation, but there's very little research into what do people actually do in these simulations and to what extent are they actually applying their learning in simulation behavior?
Joshua Littenberg-Tobias: And this is sort of a real important because people might say, oh, I believe this thing, or this supportive of equity. But what actually get presented with a scenario, how do they respond to that you sort of in real time, and so we wanted to sort of use the state to really understand how did people actually respond within the simulations themselves. So, as I said, it was a very, very large corpus of data. And so wouldn't be profitable to analyze it sort of line by line. So what we did is we used a method from natural language processing called Topic Modeling. So I won't go into all of the technical details of how this works, but basically to summarize what Topic Modeling does is it looks at patterns and word within a text. And based on the patterns, it sort of pulls out sort of calm words and things that tend to co-occur together.
Joshua Littenberg-Tobias: So for example, let's imagine that you were looking at two articles in the sports section of newspaper, and one of the articles was about baseball, and one of the articles was about football. And the article at baseball, you might expect certain words like picture, baseball, home run. These are words that are very specific to the context of baseball. So if you see those words are for computer. So those words, if you know that this is an article about baseball, versus if there was an article about football, it might have things like yard, quarterback, touch down. Those are words that specific to football. And so the computer will be able to detect, which are the articles that are about baseball and which of the articles are about football. One of the sort of good qualities about this type approach is it doesn't require any a priority assumptions about the structure of the data.
Joshua Littenberg-Tobias: So we didn't have to before come in with a set up, like these are the specific things that we are looking for. We sort of allowed the computer to identify what are some of the underlying patterns in the data. Additionally, the specific type of topic model that we use is called Structural Topic Modeling. And this allows for the inclusion of predictors into the model. So it allowed us to see, are there any correlates to different topics appearing in simulation responses? And in particularly we were interested in how did people's beliefs about equity as they set on surveys, how was that related to their behavior within the simulations themselves?
Joshua Littenberg-Tobias: And so we applied this topic model, we applied it across all of the simulations within the course, and for each simulation, it pulled that a different set of topics. We were able to see what are the topics that were associated in each people's response.
Joshua Littenberg-Tobias: And what types of things did it detect? And one of the surprising things was that even though we didn't assign any labels, the Topic Model was able to detect themes that Elizabeth had actually sort of thought about putting in. So the same with the doctor's note, there sort of two options of, are you going to mention the fact that you don't have a doctor's note and that's the school policy? Or you're going to ask Jeremy how he's feeling. That was actually something that the model is able to detect from people's responses and as distinct ways that people might respond in this simulation.
Joshua Littenberg-Tobias: So then, once we had the topics we're interested in which topics are more or less associated with the educator mindsets that we had talked about earlier. And we used a survey scale that we had developed prior to the course that had a set of survey items for each mindset. So for each mindset, there was a set of four to six survey items that were associated with each mindset. And then we looked at, okay, how correlated were people's responses and the survey with the topics that appeared in their simulation responses.
Joshua Littenberg-Tobias: And so this is from the article, and so one of the things we found was that there was a pretty strong association between certain topics appearing and then people's responses and people's survey responses about equity. So for example, in the Jeremy Journal simulation, the doctors known school policy was more associated with an equality perspective than an equity perspective. So people who sort of mentioned that the fact they didn't have doctors note were more likely to on server responses indicate more of an equality perspective, versus people who said about Jeremy feeling better or asking how he was feeling. Those people were more likely to have an equity perspective. And we noticed similar trends across other simulations, that things that we had sort of intended of identifying potential less equitable or more equitable behavior in the simulations themselves were associated with those behaviors.
Justin Reich: So another finding of this research is that we are actually able to look at changes in behavior in the simulations over time. Remember we asked people to do four simulations, one in each of the units in the course. And we did this clever research where we took survey evidence about people. And we found folks who expressed in surveys, a bunch of attitudes that are well aligned with equity teaching practices. And we took the top 25% of folks who had answers to their surveys that aligned with those ideas and compared them to the rest of the group.
Justin Reich: And when we looked at the first simulation, we found that there were some differences between these folks with some really equity oriented answers in their surveys, and the other folks who had less equity oriented surveys. When you look at the way they use words in the simulations, these two groups were pretty different. But over the course of the next three simulations, these two groups get closer and closer together in how they use language to enact performances in the simulation and how they use language to reflect on what they've just done in the simulation.
Justin Reich: So we certainly found throughout the course that people, immediately after the course, six months after the course, they have more equity oriented attitudes that they express to us in their surveys. They describe to us their commitment to taking on more equity teaching practices in their classrooms. And it's great to get that kind of self-reported evidence, but we can also see in these simulations, what they're doing that over time, all participants end up behaving and reflecting in ways that are more similar to the folks who came into the course with the most equity, justice oriented attitudes.
Justin Reich: So to summarize, we think the simulations are a really useful tool for teachers to practice learning, but then also to measure their learning and to look at changes over time. And so the kind of machine learning, the Topic Modeling that Josh was that can be useful to automate some of this analysis, it was useful in both identifying what topics were associated with equity behavior, and also seeing how those behaviors in the simulations changed over the course of time, over these four simulations in the MOOC.
Justin Reich: Now, what's so exciting about these findings from the simulation research is that measuring and evaluating teacher performance is really important. We want to give teachers the feedback they need to get better, but it's also really expensive and really hard. These online courses that they run, we have people from all over the United States from all over the world. We can't fly folks around to their classrooms to make observations, give feedback, evaluate their student work, those kinds of things, but we can look at what they're doing in simulations and use that to guide our instruction, our feedback, and our efforts to try to help them.
Justin Reich: So, in a sense, the simulations are one way of evaluating teacher behavior, teacher performance, teacher learning, without having to travel to each individual teacher's classroom all over the country and all over the world. Later in the conversation, Josh and Elizabeth took some interesting questions from the audience and let's hear what they had to say.
Elizabeth Borneman: People asked if the simulations can be used in professional development and they are, that is pretty much the medium that we share them out in the world is actually through professional development workshops. And before COVID, we went to a lot of schools and while we were still in the design process, we got a lot of feedback from students, from teachers, from principals, admin, family members, about what was going on in their schools. What were the most common situations that come up and that fed into our design, the personalities of the students, the decisions that teachers had to make and those sorts of things.
Joshua Littenberg-Tobias: I think what I try to do in my work is think about how can we take some of the sort of great research and insights that qualitative researchers have had looking at students experiences in classrooms and think about how can we sort of extend that into a broader sphere and look at sort of how does this work when we are looking at it in a large scale learning environment. And without sort of sacrificing some of the quantity of people's actual words and expressions and feelings. So in a lot of ways, this process sort of draws a mixed methods approach looking at not just numbers and what are people saying, but also what are people's actual words.
Joshua Littenberg-Tobias: And it's been really illuminating interesting to see what are just the different ways that if given a situation, people will respond to it. And are hoping that, is the stimulation sort of become more prevalent. We can really learn more about how does your response to simulation? What does that say about your attitudes, about teaching, about working in the classroom and other things? And how can we use simulations as a way to sort of nudge and prompt people who are rethinking some of their original assumptions of that teaching.
Justin Reich: We got a lot of great questions during the conversation. One of them was, "I'm curious about what you think you learned from the research that could be applied more broadly to educator or principal operation programs. What are the next steps for you and what have you found to be the strengths of simulations? How are you seeking to improve them?" And Elizabeth had a really great response.
Elizabeth Borneman: So we used these simulations in a broad variety of PDs explicitly, and the one Roster Justice with the class roster. And you were seeing the teacher schedule. And so that simulation is actually like a... It's a conversation between an educator and a principal. And so the educator has gotten their curriculum, their schedule, their roster, and they're supposed to see like what patterns am I noticing here and the principal. And so pretty much you're dropped into a situation where you've requested meeting with your principal and the principal's like, what is it? What's going on? Like, I'm here. What do you need from me? And then you're pretty much asked to describe what you're seeing in your rosters, in your classes, in your scheduling, what you and your colleagues are talking about. And so throughout that simulation, the principal is really pushing back.
Elizabeth Borneman: He's giving a number of reasons as to why he can't do anything about it essentially, he's like, our quintessential like, we don't have time for this kind of guy. He's like, we don't have the funding. We don't have the staffing. This is too close to the school year. So along the way, the principal is giving examples of reasons why there's nothing he can do to fix the imbalances that you're seeing in the schedule.
Elizabeth Borneman: And so when people go through that, we've seen in PD, we've done it in small groups, really large groups. When people go through that, at the end we find that people have a variety of things. They said to the principal, some people will sort of just defer to the principal early on and won't, because there's that power imbalance, which is really replicated in real life, right?
Elizabeth Borneman: If you're a teacher you're not going to always want to advocate for your students in front of and to your boss. So what we find is that people... Teams of teachers are like, okay, is our principal's like this? Is he not? What are the best ways to advocate for students when we're fighting? We're coming up against these barriers that are concrete, that are real at schools, but also is he sort of just shifting the responsibility? So what we see is that the strengths of the simulations are that it puts everyone who's participating on them on the same page, because we all went through the same simulation. We all saw the same data and we were asked, what are we going to do? What differs is how people respond to it. And I think what we've seen is the conversations that teachers get to have with each other about how they responded or why, or what they noticed along the way or what they didn't, really helps them to develop.
Elizabeth Borneman: Especially, when we put the things they notice and what we recommend as best practices in the simulation into perspective to what they really did. And we give them that framework and we say, well, if you didn't advocate for the students and notice that these classes were super blocked off, right?
Elizabeth Borneman: So what you'll see in that simulation is that all the black kids and girls and students with disabilities are all grouped into one class and then mostly white males and Asian males are put into this computer science class. And it's because of a number of things that are going on at the school. But whether you notice them, whether you're able to communicate that to the principal, how would you do that? Those are the conversations that we see teachers have with each other after they finish a simulation, and they learn a lot from each other, they learn about what's the best way to handle this that is going to be most equitable for students. And what if I had done this is probably harming my students in the long run, because it aligns with the less equitable mindset that is actually happening all the time.
Elizabeth Borneman: So we think those are the strengths, putting everybody on common ground, giving everybody a common language, giving everybody examples of what the right thing might be to do or how to improve and the way we seek to improve them. There's everything can always be improved. So I guess we always take feedback. So another thing about the design process is that what you see has been iterated 10 times. I mean, that's not even an under exaggeration. The first time we do this, we test this out with teachers, with our colleagues, we test it out on each other. So by the time we actually share these simulations, they've been iterated, it's just an iterative design process.
Elizabeth Borneman: So we kind of like once it's there, it's there, because we've improved it based off so much feedback to make them as neutral seeming as possible, so that the things that people actually do are what they would actually do. Not because they're being prompted to do it, but because it's what somebody would actually do. So I think that iteration always improves. I have to think a little more on what the next step would be for them. But I think the is honestly like an affordance because I think the more simple the design is the more space it gives people to pretty much do anything that they would do. I think the more complicated sometimes the design gets you get less of a cohesion between we're seeing a lot of patterns, the patterns become more hard to interpret and give people good advice based off of them.
Joshua Littenberg-Tobias: Yeah. I agree. I think definitely the sort of simple structure really helps a lot, also gives everyone sort of a common frame of reference for like, okay, what did you do in this particular situation in the simulations? One other thing that I've been working on is thinking about ways that we can, in the moment process people's responses and give them feedback and thinking about when is the right time to do this and what size is feedback. So for example, if someone's doing a simulation and they've sort of, based on their responses, it's pretty clear that they see Jeremy as sort of troublemaker. Like he's just sort of being deliberately rather than some people might see him struggling in school and feeling embarrassed. Some people might think that something is happening at home.
Joshua Littenberg-Tobias: There are also some people who see the same set of facts and say like, well, he's clearly just trying to test my authority as a teacher and he's goofing off and he's scares up to the heart. And so we see that in people's responses, is there a way to actually give people feedback in the moment, maybe take a step back and kind of look at some of the evidence that you've seen over the week and is that consistent with your interpretation?
Justin Reich: So the last question for the audience was how to access the simulations. And if you can play them yourself and some of you listeners might be wondering the same thing, are online course Becoming a More Equitable Educator is free and openly licensed, and anybody can use that and participate in all the simulations in there. And there's a link in the show notes about how you can find that course, but a Google search for Becoming a More Equitable Educator will get you there as well.
Justin Reich: Inside that course, there are also a series of facilitator guides that help you take the simulations in the course and use them with your own colleagues in your own departments, your own local context. Second thing that you can do is you can go to the website, teachermoments.mit.edu, and you can look around at the simulations that are there. We're actually in the midst of an initiative right now to try to make the website a little bit more friendly and welcoming and to explain what's going on there a little better.
Justin Reich: But in addition to the website, we also have a community of practice where folks get together a couple times a month to talk about how to use these simulations in colleges of education, in professional development programs and all kinds of settings. And there's a link in the show notes of how you can sign up to find out more about out those community of practice events. We're adding new features all the time. We've had over 600 scenarios authored over 13,000 people have participated in teacher moments. Over 200 folks have created or edited teacher moment simulations, and we're really excited to have more folks using this powerful tool. Thanks for joining us to learn and more about our teacher simulation work. I'm incredibly grateful to Elizabeth Borneman and Josh Littenberg-Tobias for their great research and colleagueship, and for sharing this conversation with everybody here today.
Justin Reich: I'm Justin Reich. Thanks for listening to TeachLab. I hope you enjoyed this episode. Be sure to subscribe to TeachLab, so you don't miss any episodes from our new season. In today's show notes, you'll find link to the full webinar are, Becoming a More Equitable Educator course, the Teacher Moments website and all the simulations that we talked about today. You'll also find a link to join our Teacher Moments community of practice. Explore, and then tell us what you think. You can find us on Twitter @TeachLabPodcast. This episode of TeachLab was produced by Aimee Corrigan and Garrett Beazley, recorded and sound mixed by Garrett Beazley. Stay safe until next time.