Justin Reich is joined by Barbara Means, author and executive director of learning science research at Digital Promise to discuss her research with digital learning before and during COVID. “There were quite a few universities that for equity reasons told their instructors, ‘Don't do any synchronous instruction in the spring, just put everything online and let students do it asynchronously, that is on their own time whenever they wanted.’ We found that when there were no synchronous sessions, which could have been either with the professor or it could have been online office hours, or it could have been working with a teaching assistant in a section. But if there were no synchronous sessions, the students were less happy with their course and their learning. So they really wanted that connection with a real person just like many of us do. You call up customer service. And it's so frustrating even if after the fifth click you can get to what you want. You just want to say, ‘I want a real person to talk to me and tell me they're sorry.’” - Barbara Means
Justin Reich is joined by Barbara Means, author and executive director of learning science research at Digital Promise to discuss her research with digital learning before and during COVID.
“There were quite a few universities that for equity reasons told their instructors, ‘Don't do any synchronous instruction in the spring, just put everything online and let students do it asynchronously, that is on their own time whenever they wanted.’ We found that when there were no synchronous sessions, which could have been either with the professor or it could have been online office hours, or it could have been working with a teaching assistant in a section. But if there were no synchronous sessions, the students were less happy with their course and their learning. So they really wanted that connection with a real person just like many of us do. You call up customer service. And it's so frustrating even if after the fifth click you can get to what you want. You just want to say, ‘I want a real person to talk to me and tell me they're sorry.’” - Barbara Means
In this episode we’ll talk about:
Resources and Links
Check out Barbara Means’ book, Learning Online: What Research Tells Us About Whether, When and How
Learn more about Digital Promise
Check out Justin Reich’s new book, Failure To Disrupt!
Transcript
https://teachlabpodcast.simplecast.com/episodes/barbara-means/transcript
Produced by Aimee Corrigan and Garrett Beazley
Recorded and mixed by Garrett Beazley
Follow TeachLab:
Justin Reich: From the home studios of the Teaching Systems Lab at MIT, this is TeachLab, a podcast about the art and craft of teaching. I'm Justin Reich. Today, we have Barbara Means. Barbara Means is the executive director of learning science research at Digital Promise. And she's the author of Learning Online, one of the great summaries of online and blended learning research from the past decade. Barbara, thanks for joining us on TeachLab.
Barbara Means: Oh, well, thank you so much, Justin. It's really a pleasure.
Justin Reich: For us to be able to get to know you, can you give us your ed tech story? What was the earliest, most significant encounter that you had with learning technologies that maybe planted the seed of interest that led you to where you are now?
Barbara Means: Boy, you're asking me to go back to the dark ages now.
Justin Reich: The dark ages, that's where we want to be.
Barbara Means: Okay, pre-internet. I would say the thing that got me interested in educational technology, probably a couple of events, and I think they're both still relevant today. One was I happened to be working for a nonprofit research organization that did work with the armed services, this is way back in the 80s. And we had a project with the Air Force human resources lab to study the parts of jobs for enlisted people that were really essentially problem-solving, the things you couldn't automate. So their vision was we're automating the things you can automate, but these things that are really much more conceptual, we can't automate, and we need to be able to train people to do those things better.
Barbara Means: So we worked at that time, I had a contract, and worked with Learning Research & Development Center at Pitt, and with Bolt, Beranek, and Newman, a nonprofit working in this space at the time. And we were working on intelligent tutoring systems, so my collaborators brought those in. And when you saw what people could do in terms of learning how to troubleshoot, they have 16 jet engine, and that they could learn as much in six months using the simulation and this intelligent tutoring system. These were Lisp tutors way back when for any old timers.
Justin Reich: Lisp was a programming language that would be a little bit like Fortran or other, that vintage anyway.
Barbara Means: That vintage. And at the time, we thought it couldn't be intelligent unless it was written in Lisp. But seeing how powerful that was, that really impressed me. And then I guess you flash forward some years later, I really became interested in how technology, not really for its own sake, but it actually makes people think about, what am I teaching? What is it I hope people learn and can do, and how do I structure a learning experience that works for them? And so in a way, technology is an invitation, whether it's for a curriculum designer, a teacher, a school leader to really rethink what they're doing, how they organize it, how they measure it, and what the experience is like for students.
Barbara Means: And seeing that happen in schools as I was studying early introductions of technology into schools for the Department of Education really got me interested in the potential learning impacts for people who otherwise wouldn't have high-quality learning experiences. And also I think just learning technology adoption as, if you will, kind of the-
Justin Reich: Like a catalyzing event.
Barbara Means: That's right. Yeah. I like that better than the Trojan horse, which actually is bringing bad things. The catalyzing event, exactly.
Justin Reich: But the Trojan horse metaphor is something like, it looks like you're putting the technology in, but you're actually trying to sneak in pedagogy, you're trying to sneak in organizational ideas. You're trying to sneak in all other kinds of ideas and change that people let in through the walls because they go, "Oh, look at this gift that we've been given in the device." I think there's something to the Trojan horse metaphor. I mean, that certainly resonates quite a bit with me that the learning technologies themselves can be interesting, but what's really interesting is the ... Maybe it's not completely unique, but a striking way in which communities of faculty and educators are willing to, at least for a period, rethink their practices in the face of this symbol of the future. Here's this shiny new laptop, here's this shiny new device. We don't know exactly what it can do yet, but we're willing to relax some of our assumptions about what the future should look like in order to spread our imagination here.
Barbara Means: That's right. One thing, there's no shame in saying, "I don't know how to use this new technology." And the other thing that often brings educators together who don't usually work together, that's one of the things we see in a lot of our work with various institutions. So suddenly you're team teaching or you're working with somebody who's an instructional designer that you normally wouldn't work with, but that instructional designer knows how to use the technology.
Justin Reich: Yep. Bringing different kinds of people with different kinds of expertise together. So tell us a little bit about what you're working on most recently because I know some of your recent research is focused on responses to COVID? What are you studying, and what are you finding that educators might find helpful right now?
Barbara Means: Yes, we have looking at COVID, and much of this has been at the undergraduate level. We did a survey in May of a nationally representative sample of over 1,000 students who had been taking a college course for credit that had in-person class meetings before COVID and then had to go entirely remote. And so we wanted to find out about the challenges they were facing, how common those challenges were, and how widespread they were for different kinds of students. We also wanted to find out what experiences they were having in their courses, that is what their instructors were doing and what their reaction was to those things.
Justin Reich: There's so little that we know about that even in normal times. One of the things that struck me in my earliest days as an education researcher is you'd come up with a question like, well, how many times do students talk during a typical class period in a social studies class? And we just have no idea. We had so little understanding of what typical classroom practice looks like in any granular way because it's so diffuse. It sounds like you really found a way of targeting the students who are stuck in this emergency pivot who happened to be there for this moment. What did you find out about their experiences and their responses that seems most important to you now?
Barbara Means: Well, one of the things that really hit me was when we asked them about what challenges they experienced. By far and away, the biggest one, the most commonly reported as a major problem for them was maintaining motivation for the course. And we asked them to describe the challenge first and then categorize it. And as we read the descriptions, a couple of things really hit home. One was that the loss of a routine was really hard for people. And this is apart from some other real challenges like having conflicts with a job you had to get to put food on the table or conflicts with family or childcare responsibilities. But just simply loss of that routine, students described it being hard to get out of bed or hard to make yourself go and get online and do work on your course.
Barbara Means: So that was one thing. The other thing that really came through was there was definitely a social aspect to this. I was somewhat surprised how much undergraduate students reported missing contact with their instructor, not being able to talk to my instructor, not being able to get an answer from my instructor right away. Feeling at a distance from their instructor and their peers. They also described missing their peers, which was less of a surprise. So that was on the challenge side what I was surprised by.
Barbara Means: On the side of in terms of practices and what they experienced, there of course was a great variation in what instruction they experienced and in whether it was just a matter of posting materials online for students to complete and then taking some online exams, or whether there actually were Zoom sessions for smaller courses or other kinds of synchronous instruction. But the interesting thing was when we asked them about a set of practices that we took because they're generally known to be correlated with student satisfaction and/or learning in courses, those courses for which the students said there was a higher number of these practices used, the students were also much more satisfied with their learning in those-
Justin Reich: What are some examples of those well-known practices?
Barbara Means: Well, the three that turned out to be the most influential, the first one was using real-life examples to illustrate the course content. So having the instructor actually bring it to something that comes from the real world. And of course, you do that offline not just online, but just-
Justin Reich: You can do it before a pandemic, you can do it during a pandemic, just good teaching.
Barbara Means: It's just good teaching. But some students did complain or did observe that there was less use of real-life examples during the pandemic than before. And in part, that was probably because instructional time was truncated and the instructor felt, "Okay, I've got to keep moving through the curriculum."
Justin Reich: Yeah. I gotta hit the core pieces, do the formulas, skip the application.
Barbara Means: Yeah, exactly. Anyway, so use those real-world applications was really important. The second one was getting personal messages from the instructor, so speaking to that interpersonal piece. Getting those messages, they could have been, are you okay? Are you in good health? Also about, do you have access to what you need in order to get online? And just, I noticed you're falling behind, is there anything I can do to help you?
Justin Reich: And instructors so often used those strategies in an in-person setting, you can use those things informally. You can be like, "I know I got to catch this student. I'm going to come to class 10 minutes early or I'll stay a little bit late and I'll just grab this person as part of my normal teaching routine." And you can do it in class, people are working on something and you can wander the aisles and touch base with them and things like that. To some extent, if you want to do that work in a fully online setting, it's gotta be a little bit more deliberate, I've gotta send that email or send that message to Barbara and say, "Hey, can we take five minutes to talk? I just want to check in," those kinds of things. It can happen in both places, but my sense is it's much easier to do informally in-person, and it really has to be part of a structured routine online.
Barbara Means: No, that's right. And I think normally it just is emergent in the in-person setting. Now, there are technology tools that can help you do this and systems that can identify students that are not participating and that can even set up the emails for you that they may look personal to the student, but in fact via emails that have... you've been given a shell for the email that you can customize as you like. And we know some faculty members use those. And then the third thing, which I thought was interesting was courses where there were assignments or activities that call for the student to reflect on her own learning. What is it you understand? What don't you understand? So those were the three practices that had the strongest relationship with satisfaction, with students' learning, and with the course in general.
Justin Reich: Good. So good teaching practices to be done at any time, but particularly important now. All the things you're saying resonate. We did a study in our lab where we interviewed 40 K-12 teachers across the country, public, private, independent, math, science, social studies, PE, music. It wasn't nationally representative, but we got a group that was pretty wide ranging. And certainly, student motivation was the top of their concerns along with connectivity in K-12. But that sense of, how do we keep folks connected? And that report was called What's Lost, What's Left, What's Next. And we can put both of these things in the show notes.
Justin Reich: But another thing that your comments, especially about the importance of connecting with instructors make me think of is in March, if you were to ask me what colleges should do to make it through the rest of the semester, I would have said something like, "In a lot of cases, people have spent a lot of money building online courses in the most common topics that are taught in higher education And instead of trying to, in the midst of a pandemic with two kids running around your feet, trying to whip together some Zoom class, you should just probably point people to what's ever on OpenStax or Coursera or edX and say, "Go have at it. And then come back and meet me once a week and we'll talk about what you're learning.""
Justin Reich: I was surprised at how little my perception is that happened. And I have the sense that there was no widespread student demand. I've heard nowhere where there are groups of students saying, "Why do we have to take these lousy courses that our instructors are whipping up out of nowhere when there's better materials available elsewhere?" And instead students saying, "No, I still want my version of introduction to microeconomics taught to me by my professor in my little university," whatever that happens to be. Does that resonate for you?
Barbara Means: That really resonates with me. I remember actually one student's comment was, and they were lauding their professor because they said instead of showing us videos of other people teaching the topic, she recorded herself doing it.
Justin Reich: Right, right. Other videos might've cost thousands of dollars for someone to put together, they might've been much better, but that student wants her professor.
Barbara Means: I think that's exactly right. And it really again points to this personal relationship aspect of teaching and learning that seems to be so important for maintaining motivation. There were quite a few universities that for equity reasons told their instructors, "Don't do any synchronous instruction in the spring, just put everything online and let students do it asynchronously, that is on their own time whenever they wanted." We found that when there were no synchronous sessions, which could have been either with the professor or it could have been online office hours, or it could have been working with a teaching assistant in a section. But if there were no synchronous sessions, the students were less happy with their course and their learning. So they really wanted that connection with a real person just like many of us do. You call up customer service. And it's so frustrating even if after the fifth click you can get to what you want. You just want to say, "I want a real person to talk to me and tell me they're sorry."
Justin Reich: For people who are immersed in the research on instructional design, this emphasis on the synchronous during the pandemic is a little bit odd because in fact I think many instructional design frameworks for people who are opting in to online learning, online college courses would say, "Yeah, there are a lot of good reasons to use things that aren't anchored to time. People work at different times, they live in different parts of the world." That we should, generally speaking, be creating mostly asynchronous curriculum, certainly with synchronous meetings kind of woven in, but kind of the heart of the experience should be ... I feel like when I was following instructional design Twitter in March and April, folks were saying, "Why are we doing all this synchronous stuff? We know that asynchronous can work really well."
Justin Reich: But to me, there's a signal there that asynchronous can work really well when you've opted into it, when you've chosen it, when you knew what you were up against and could prepare yourself to thread that approach to learning in your life. By contrast if the rug gets pulled out from you and your in-person school just stops, then you want something that's close to that, which is crummy Zoom school ends up being more desirable in some ways than even the best, most well-prepared asynchronous materials.
Barbara Means: Now, one of the things we did find, we also did case studies. We've been working with colleges and universities that serve large numbers of low-income students and students of color. We've been working with them on attempts to redesign and improve their introductory gateway courses, incorporating digital learning, typically adaptive courseware. So that's of course where that professional instructional designers developed and has some of the features you're talking about. But typically that's just a part of the course. Before COVID, there would be face-to-face classes plus work that's done on the courseware.
Justin Reich: And this is a research project that you had started pre COVID, but you just happened to be running it and the pandemic hits and so you're thinking about how to adapt?
Barbara Means: Yeah, yeah. And we're continuing now. So we're working with some of the same institutions as they go into this fall strange semester that we're in now. But one of the things we did find is that those instructors that had already done this and had some asynchronous use of professionally designed instructional materials that was part of their course and have built up routines and patterns before COVID. The transition was much easier for them and their students because there were parts of the course that could carry over that were resilient to this disruption. And there were many fewer things they had to figure out how to do in this new medium.
Justin Reich: If you already have a blended routine going that when you lose the in-person, you've got more of that online stuff to rest on, to go back to. Let me know if you saw this that some of those instructors might've said, like during those terrible weeks in March when everyone is scrambling, I could imagine saying to my students, "Just keep doing this online part for a week or 10 days or something like that, and I'm going to put together something that works for the synchronous piece." But you could almost really lean in to the asynchronous online routines just for a week and a half so that I can get my act together. Did you see patterns like that emerge or?
Barbara Means: Yeah. In a sense, we did. I didn't hear anyone say it as explicitly as you did, but we did see that they kept those routines up. There were many cases where instructors were a little more flexible on deadlines for completion than they had been before, but they tried to keep the same schedule. It was typically maybe a flipped classroom situation where students reviewed materials before coming to a synchronous session. It used to be in class, now the synchronous session is online.
Justin Reich: Well, pre COVID, if you had said, what are the best ways to improve gateway classes for students in community colleges or state colleges to help improve their persistence? Online learning, blended learning is one reasonable option. But in my view, it's not the dispositively obvious option. There are other things that you could choose about better advising, better coaching, more active learning in classrooms and things like that. Online blended learning, it's like a promising thing to try, but not the one obvious thing to do. But it does strike me that in a world of growing climate change, this is not going to be the only pandemic that we face in our lifetimes.
Justin Reich: Very likely we're going to continue to see more interrupted schooling from fires, from extreme weather events, for other kinds of things that schools just are going to find that they have to become more robust to more frequent interruptions. And that may require a school ... For the reasons that you just said. It's not that blended learning is the number one best way to make schooling better. But if you know that school is going to be interrupted a lot, having everyone have some experience and some practice with those routines is awfully helpful [crosstalk] for you.
Barbara Means: Yeah, no, I agree with that entirely. I do think that the, if you will, the secret ingredient, the biggest secret ingredient I think is around the active learning and trying to have more active learning in synchronous time as well as hopefully as courseware improves, there'll be really more active learning within that as well.
Justin Reich: And active learning here defined as not just lecturing to students, but interspersing lectures with turn and talk activities, with reflection activities, with peer teaching activities. Is that how you define it, a whole range of things that break up teacher talk and force students to articulate their thinking to themselves or to each other?
Barbara Means: That's right. Exactly. And to attempt to apply ideas in applications and in different situations. And I think those techniques are really important. And as we can perhaps offload some of the just transmitting information to some of the digital learning systems and to time outside of synchronous time. I think we can have a more constructive use of those precious minutes when students and instructors are actually together.
Justin Reich: So in Failure to Disrupt, one of the things that I try to teach readers a bit about is this idea of meta analysis. The idea that it's really hard from any individual study to make educational claims. And the way that knowledge moves forward in sciences and the way that we should think about applying science about learning and education to our work in schools is ideally by waiting until we have lots of studies done and can sum up what we learned from all of them. And certainly, one of the most important meta analyses about online learning in the last 10 or 20 years comes from you. I think originally published for the Institute of Education Sciences in 2010, correct me if I'm wrong there and then republished and expanded as a book called Learning Online in 2014. You've had lots of contributions to the field of learning science. But this is really a signature one that many people have come back to, where you took all of the really rigorous studies that you could find, compared online and blended learning or online and face-to-face learning and blended and face-to-face learning.
Justin Reich: And the summary that you point towards there is that in a lot of these research studies that were done, we find little differences in outcomes between online learning and face-to-face learning. And sometimes that we find that blended learning has outcomes, it's a little bit better than just online learning, maybe a little bit better than just face-to-face learning. Is that a fair summary of the argument that you made in 2010 and 2014?
Barbara Means: I think that's a fair summary. The only thing I would is we did find that blended learning on average was better than face-to-face, but we also found that in most of the studies going into the meta analysis, there really was a difference other than the modality. That is if you really had what an economist or some other methodologists might consider the PURIST Study, the Richard Clark study where everything is exactly the same only one is entirely face-to-face and the other is blended. I would argue it's probably an impossible dream. But if you really could do that, you probably wouldn't find a difference.
Justin Reich: Any difference at all. The part of what's happening in the blended learning is that you're having some other instructional improvement, you're having more time, more active learning, something else better that's happening.
Barbara Means: Right, right. That's likely where the improvement comes from would be my bet. Nevertheless, I remain ... And I think the book is pretty positive about blended learning as an approach, and I remain positive about it. Certainly, the COVID experience has added, I guess, another reason to be positive about blended learning as an approach. I do think it gives you more resilience, it can give you-
Justin Reich: Just [crosstalk] talking about.
Barbara Means: Just finished talking about. It can give the student multiple ways to learn the same objective because it's explained in different ways. You can keep that personal connection with an instructor when you've got blend rather than fully online, it's easier to have that personal touch. But you can also get some of the benefits of more sophisticated renderings of things like molecular structures or complex systems or going back and forth between equations and videotaped movement. I see blended learning as actually something I think is going to continue to grow, and I think has promise. So I'm a little more upbeat about that than perhaps your summary was.
Justin Reich: Yeah, that's great. That's a great clarification that the book is learning online, but certainly a theme in the book is let's really look at this blended stuff, and maybe some of that has grown in your thinking. This work as I was coming up as someone studying education technology was very influential to me. One thing that I've perceived, and I think you even see this come out during COVID times is one interpretation of the work is folks saying, "Well, we know that online learning can be just as good as in-campus learning. We may have to worry about connectivity issues, but we don't have to worry about kids going home from school because we know online learning can be just as good." To me, I feel like there's been a second stream of research that came after 2010.
Justin Reich: So you do this meta analysis in 2010, there's lots of randomized controlled trials. As you pointed out in the book, many of them happen in medicine, few of them happen in K-6, a number of them are scattered 6 through 12 and in different parts of higher education. But then since 2010, there's just this surge of online learning. There are many, many more people. One source of that surge is massive open online courses. But community colleges are offering way more of these courses, virtual schools are growing across the country. And so I feel like between 2010 and 2020, there's another wave of studies, which are typically not experimental, they're typically observational. So learning online mostly focuses on randomized controlled trials where there's some effort to control the difference between the people in the face-to-face condition and the people in the online or blended condition.
Justin Reich: There's a guy who was a community college researcher in California who in 2011 published an article in which he described the online penalty, which when he looked at this observational data from the California community college system, he said, "Actually, we're seeing that when people take online courses, their outcomes are a little bit worse than face-to-face, and that's a bit of a problem. But the real problem is that for our Latin American students, our Latino, Latina students, the outcomes are substantially worse online than face-to-face." We found similar kinds of things when we looked at massive open online courses that people, we found that globally, we found that people who came from countries with low levels of human development index, less education, less healthcare, less GDP, things like that passed courses at lower rates than folks from more affluent countries, even controlling for level of education and things like that.
Justin Reich: Susan Dynarski looked at a bunch of these kinds of studies and had an article in the New York Times, which says online learning is hurting the students it's most supposed to help. Ray Kaupp was onto something with this idea of an online penalty that for a lot of people when you switch from ... It seems like when you look at these things not experimentally, but observationally, when you send tens of thousands of people into a community college system and watch what happens to them that they do a little bit worse in online learning in general. But particularly people with low prior achievement, people who weren't well-served by the education system, people from racial minorities, people from lower class backgrounds, sometimes younger learners versus older learners, they do worse in online learning.
Justin Reich: And to me, the summary from that is a little bit different from learning online. The summary that I took away from the meta analysis and learning online is we can make online learning that's just as good as face-to-face learning. We can make blended learning that's maybe a little bit better. The second body of research points more in the direction of, boy, we ought to be pretty cautious when we do this online and blended learning stuff because there seems to be a population that when we do a lot of it is not well-served. And I'm wondering how you-
Barbara Means: Well, actually there is a chapter in Learning Online, which is learning for less prepared students, which actually takes up exactly this issue. And even at that time, by 2014, there were a number of studies coming out of the Community College Research Center analyzing data from large systems like Virginia, Colorado, I'm sorry-
Justin Reich: Washington State.
Barbara Means: Washington State.
Justin Reich: Teachers colleges, Community College Research Center, which has done some really terrific work over the years.
Barbara Means: Yeah. And showing that there were lower course pass rates and lower grades in online versions of courses for low-income students and students of color. So the book does take that up. My argument is that was a critique of the online learning that was offered, and it was a critique of online as opposed to blended learning. That's partly why the book comes out much more strongly in favor of blended learning. This does point up that as researchers we really need to desegregate our data and look for important student groups not just look at the average overall. I see it as kind of similar to the MOOC phenomenon you describe in your book where we know that for some people MOOCs are fabulous. They learn a lot of things, they do really well. But we also know most of the people that are participating in them in fact already have a bachelor's degree. So MOOCs are not going to be the answer to bringing undergraduate education to everybody. It helps, as you say, it helps the rich get richer.
Justin Reich: Is it fair to say that a bunch of the or most of the studies that we have, these randomized controlled trials pre 2010 just weren't big enough to do a lot of that disaggregation, that most of them don't attend to issues of race either because they were too small or even some of the ones that were bigger? I haven't gone through them all systematically, but it just seemed like that question of how do people from different backgrounds and different life circumstances use technology differently is one that has been taken up much more robustly in the last 10 years looking at online learning than in the years before that. How fair of a characterization do you think that is?
Barbara Means: I think that's very fair, I would agree with that. And you do need the really large numbers in order to look at some of these issues and have appropriate controls for other variables.
Justin Reich: And of course, the mess of the really large numbers is that a lot of these things are not experimental. So a lot of the research that the Community College Research Center does, it's excellent in the sense, it's like, here's the entire state of Washington for two or three years, but those students were not randomly assigned to be in the online condition or in the face-to-face learning condition or any other condition, they're just people who signed up for their courses. Whereas a lot of the research that you focus on in your meta analysis is experimental. It's like there's a strength of each of these bodies of research that they point in somewhat different directions. And part of that might be the nature of their design.
Barbara Means: Right, right. I think part of that is the nature of their designs. And there's also the issue that many of the things that can affect your learning online correlate with each other. In our survey this spring, one of the things we found when we looked at the different challenges students are reporting and encountering, the number of those challenges and the severity of them was greater for some kinds of students than for others. In particularly students that were from lower income households and students who were Latinx in particular had more challenges. They were more likely, for example, to be responsible for taking care of children during this period.
Justin Reich: Childcare is something that I've been thinking a lot about in online learning because one of the things that online learning researchers have become very interested is this idea of self-regulated learning. This idea that there are a set of practices that are good practices to have you be an effective, independent online learner, that you can set schedules, that you can hold yourself to deadlines, that you can have goals, that you can go back and review materials when you need to. There's a series of things, some of which we can abstract out from the learning logs that people leave in online systems like, "Look, this person always shows up at the same time week after week," or, "look, this person when they get things wrong, they go back earlier in the course." And I think a lot of learning scientists, particularly those with a psychologist bent have thought of those things as traits of individuals.
Justin Reich: We did this study of people who were taking a massive open online course based blended supply chain management degree at MIT. And one of the things that struck us was that the men persisted a little bit more than the women. And when we were talking to the men about how they got through this class, they said things like, "Well, I stay late at work, I find some time on the weekends, I take my lunch breaks off." And some of that is like, oh, that's really good self-regulated learning, that's really good setting these strategies. It is also shirking family responsibilities [crosstalk] describing those same characteristics is that I'm not meeting the responsibilities that I have to my family and my community. And in most cultures, men can do that more easily than women.
Justin Reich: Self-regulated learning, not so much a trait, but a condition that if you ... To bring it back to this group of folks that you were studying, there's maybe a little bit of instruction, of guidance that we can give to low-income Latinx students about how to balance childcare responsibilities with learning. But it's not a thing that you can fix someone's self-regulated learning traits and have be better. You have to fix the conditions under which they're learning, like society needs better childcare to be able to make these things more equitable.
Barbara Means: No, I totally agree. I think one of the things we need to be more aware of is what conditions people are learning under. And some of the things we heard about from students during the COVID this spring really pointed to things that were actually inequities that existed before.
Justin Reich: Long before, yep.
Barbara Means: So it may be that there was a computer lab at school. But if you have to arrange to be at school extra hours to use the computer and the internet there, you are at a disadvantage to the student who has a great connection at home and their own laptop. And those things were happening before, we just happened to become much more aware of them this spring.
Justin Reich: We became much more aware of them even as they were getting worse because those exact same populations that you're talking about, low-income, Latino, black students, those were also the same students who are living in communities more likely to be disproportionately affected by COVID, less healthcare, more likely to be affected by the recession. They're within educational system factors and beyond educational system factors. My colleague Tressie McMillan Cottom who's a great sociologist of higher education. She was talking about K-12 schools when she talked about this online, but she said, "Look, we have just dramatically underestimated the power of the building, the building that brings people into the same space to be in the same conditions with one another." There are a lot of really affluent people in college, there's a lot of much less affluent people in college. And they all end up sitting in the same crappy desks that are bolted to the floor if you've got [inaudible 00:41:23]. And there is a meaningful equalizing function of that building that we miss.
Barbara Means: That's right.
Justin Reich: One thing I took away from the beginning, the introduction to Learning Online, there's an argument that to analyze technologies, there's a lot of particularities to it. It's a particular technology in a particular subject area in a particular domain. And I thought hard about that as I was writing Failure to Disrupt because I think very highly of your work. And I decided to do something different, which is to say, for a lot of technologies, there's basically three types of them. There's these instructor-guided courses like massive open online courses where an instructor sort of sets the pace of things. There are algorithm-guided tools like adaptive tutors, and there are these peer-guided networks. And so I was curious to get your ...
Justin Reich: And the reason why those categories are useful is that new technologies are not new. Education technology, venture capitalists will come and tell you that they've built the brandiest new thing ever. And if you ask a couple of questions and if you know something about this ed tech history, you can go, "Oh, no, no, no, that's just an adaptive tutor. We know those things work in math, and they don't really work that well in reading. There's certain things they do in early language acquisition but not in later language acquisition." So I can make a good guess about how your brandy new thing works because I know a little bit about the past and can put it in this classification. And I was wondering if when you read Failure to Disrupt you thought to yourself, "Oh, these seem like three reasonable classifications, and this is working," or if you thought to yourself, "no, no, no, no, no, things are much more granular and much more complicated than this?" That there's some things that readers are going to miss if they only look at large scale learning technologies through these three categories.
Barbara Means: Well, I actually liked your three categories. I think the idea of who is controlling the content seems like a really important one to me. But I still would also go back to the complexity of this because I do think the nature of what you're trying to learn is also important. So whether it is a skill where a lot of practice is necessary just to fine tune that skill, even though you may know all the definitions, you may know all the concepts, but that doesn't mean you can execute it correctly. If you've ever had a spouse try to coach you on how to ski, you know what I'm talking about.
Justin Reich: Teaching my own kids is where I get ... I've pretty much retired now from teaching children to ski, and I've just told them they need to take lessons from somebody who's not me.
Barbara Means: That's one thing. On the other hand if you're trying to hone your understanding of a historical period or a historical phenomenon, I think when you talk about having the more collaborative peer discussion-based dialogue back and forth becomes more pertinent as opposed to building a model-based learning where you're trying to really understand a model or a system in which case-
Justin Reich: Like people often do in physics or chemistry.
Barbara Means: In physics or chemistry, in which case something else starts to make sense. So I do think it's important what the subject matter is and also the nature of the learners. We've talked a little bit about less prepared learners coming in and less prepared learners who may be less confident. I think that's part of the reason that having guidance from a person, an instructor whom I know as a person who is sending the message, "I believe you can learn this, and I'm going to help you do it. And we're going to do it together till you get there," is really, really important for some students. Whereas other students are just fine, "Get out of my way, let me look at the material. I'll get on Coursera, I'll be fine. Goodbye."
Justin Reich: Yeah. That's absolutely right. I think a real challenge of people working in education technology who are both trying to do really rigorous research but also trying to give advice that's useful to the public is you have to go between these levels of what's a pretty good summary, a pretty good heuristic, a pretty good thinking guideline. And then what's all the complexity of a particular circumstance because if you just give people all of the complexity, it's too much, it's too much to make all the many decisions that people need to make. Or you want to take the first pass screening like, is this technology someone's introducing me to reasonable to do the extra work to thinking about under all that complexity?
Justin Reich: And I think some of the things that your work, Learning Online have helped me really think through is what are the ways that we can respect all that complexity but also come up with some general guidelines for people that are pretty good first approximations of all that complexity and can help people think through that? And I think that's something that people have appreciated about your work is being able to try to balance both of those things.
Barbara Means: Well, thank you. I appreciate that. I want to return the compliment, I actually enjoyed Failure to Disrupt quite a bit. Your voice came through, what I think is a justified skepticism about over promising with technology came through. And a sense of the importance of not just thinking of technology as one unitary thing or the only thing that's happening. In our work recently, we've really emphasized what we call the implementation model, what actually gets implemented. And the technology is only one piece of that. The very same piece of software can be used in very different ways with very different effects. And-
Justin Reich: What are other common pieces of the implementation model? When people have implementation model in their head, what other kinds of things should they start thinking about?
Barbara Means: One of the things is just simply deciding how much of what parts of the course are students learning with digital tools, and what part are with the instructor? How much of it is collaborative? How much of it is interactive? Are there group projects? How often are you doing assessment? And are there assessments that are just for learning rather than counting towards your grade? So some of those things are things that are built into the way the course gets implemented. Hopefully by design in a very thoughtful way, sometimes it's more ad hoc. And those are things we see as having a big effect in concert with the use of a particular technology tool.
Barbara Means: One of the things I really liked about your book was the concept of getting better through tinkering. I've actually used that metaphor before myself, so maybe that's part of why I like it. We see with what we're doing now with instructors is these are instructors that really want to improve their teaching and learning. And they know it's going to take multiple changes, and it's not going to be perfect the first time. And what we try to help them do is collect kinds of data and analyze the data in a way that helps them see, okay, these things seem to be a step in the right direction. These others, it seemed like a good idea, but, oh my gosh, the students hated it or they didn't use it, or they were confused by it.
Justin Reich: Well, Barbara, I think that's a great place to end our conversation asking people just to continue to think about where does the technology fit in what you described as the implementation model in the broader context. How does it work not to sweep away the past but to iteratively step-by-step, tinker-by-tinker make our way to something better and something more robust for the future? So, Barbara, thanks so much for joining us on TeachLab.
Barbara Means: Oh, well, thank you, Justin, it was really a pleasure.
Justin Reich: That was Barbara Means, she's a learning scientist and education psychologist, and the executive director of learning science research at Digital Promise Global. I hope you enjoyed our conversation where we talked about COVID-19, its effect on students, how it differentially affects different students. And then being able to step back to one of her real signature works Learning Online, which we'll link to in the show notes and make some comparison to Failure to Disrupt, which was published by me just more recently. I'm Justin Reich. Thanks for listening to TeachLab. Please subscribe to TeachLab to get future episodes on how educators from all walks of life are tackling distance learning during COVID-19. If you're enjoying the show, go ahead and leave us a review or give us a five-star rating on your favorite podcast app. I've also just released a new book Failure to Disrupt: Why Technology Alone Can't Transform Education available from booksellers everywhere. You can read reviews, relate to media, sign up for online events at failuretodisrupt.com, that's failuretodisrupt.com. This episode of TeachLab was produced by Aimee Corrigan and Garrett Beazley recorded and sound mixed by Garrett Beazley. Stay safe, until next time.