Measuring Students' Self-Control: A 'Marshmallow Test' for the Digital Age

The "marshmallow test" invented by Stanford psychologist Walter Mischel and colleagues in the 1960s is famously known as a measure of willpower. The experiment gave preschoolers the option of either eating one mini-marshmallow right away or waiting 15 minutes to get two mini-marshmallows. Decades later, those who were better at delaying gratification, and resisted immediately snarfing the treat, ended up with stronger SAT scores, higher educational achievement and greater self-esteem and capacity to cope with stress in adulthood.

Now other psychology researchers have come up with a test that challenges the willpower of schoolkids to resist the brain-candy of today’s digital distractions -- the YouTube videos, Instagram and mobile gaming apps like Angry Birds. Some people are calling it a "digital marshmallow test," although it's tailored for an educational context and doesn't involve any sweets or near-term rewards.

Officially known as the "academic diligence task," the new computer-based test offers students a choice between doing math or watching videos or playing a video game. The test was created by postdoctoral research fellow Brian Galla and associate psychology professor Angela Duckworth of the University of Pennsylvania, with Sidney D’Mello of the University of Notre Dame, as a better (and free) research tool for measuring self-control. The researchers hope this new tool will advance their studies of ways to improve academic perseverance in students.

A report recently published online by the team documents the test's reliability and validity and shows that performance on the task predicts academic achievement -- including whether high school seniors graduate on time and enroll in college.

"It's a really creative and interesting approach to measuring an aspect of self-control," said Smith College psychologist Philip Peake, who has worked with Mischel on the longitudinal follow-up of participants in the Stanford "delay of gratification" studies. The new diligence task is quite different from the marshmallow experiment, so they can't be equated, he said, but both are research tools that can contribute to our understanding of the processes that underlie self-control. And both have the advantage of measuring how people actually behave, not just what they say or think they do.

"It's not just people filling out a questionnaire that says, 'Oh, I tend to persist in things' or 'I work hard at things,' " Peake said. It's assessing real behavior. "And you can see that that behavior has some consequential relations to real-life outcomes."

Studying Self-Control in the Face of Digital Distractions

The recent work by the Penn and Notre Dame psychologists is part of their ongoing national study of the role that non-cognitive factors such as "grit" and self-control play in students' persistence in school. That endeavor, which is funded by the Gates Foundation, is following about 1,800 high school seniors over six years to track who enrolls in and finishes college.

Duckworth is known for her work on grit, which she defines as a tendency to pursue long-term challenging goals with passion and perseverance. Self-control or "self-regulation," on the other hand, is more about the short-term exercising of self-discipline in the face of momentary diversions, an ability that also feeds into perseverance. The research team needed a standardized way to assess self-control, but most of the existing measurement tools were self-report questionnaires, which can give biased results.

So they devised a task that uses behavioral responses to measure academic diligence, which they define as "working assiduously on academic tasks which are beneficial in the long run but tedious in the moment, especially in comparison to more enjoyable, less effortful diversions."

The rationale behind the test is that with many subject areas or skills, such as mathematics, the basic process of building fluency and mastery involves a lot of practice. It requires "hard work that is perceived as tedious, even though people know it's immensely important," D’Mello said. "But that's just the reality."

With math, for instance, that means "studying your multiplication tables, solving equations, again and again and again," which is essential for building more complex numerical knowledge. However, "in the digital age, it's so hard to focus," D'Mello said. In psychology-speak, students are faced with having to "regulate" their emotions and impulses to overcome boredom and concentrate on homework instead of something more fun.

To measure this skill in a scenario simulating real life, D'Mello, who is an assistant professor of computer science and psychology, designed the diligence task with a split computer-screen interface (click here for a demo). On the left side, students can choose to do a series of boring skill-building math problems -- simple, single-digit subtraction. On the right side, they can play Tetris or watch short, entertaining YouTube video clips of movie trailers or sports highlights. The test is delivered online.

Road-Testing the Test

Galla, Duckworth and their colleagues took the diligence task into two large, ethnically diverse public high schools in Philadelphia, where they enlisted 921 seniors in early 2013. The students were instructed to answer as many math problems as they wanted, as fast as they could, in five consecutive four-minute sessions. They could take a break at any time to watch videos or play the game. The instructions informed them that practicing basic math skills could improve their problem-solving abilities for the future.

In total, the teens spent about half the time on math skill-building, answering an average of 244 problems, D'Mello said. Overall performance on the task consistently correlated with individual differences in conscientiousness, self-control and grit (which were also assessed in the students through questionnaires), just as the psychologists had theorized.

The kids who solved more math problems tended to have higher senior-year GPAs, better scores on standardized math and reading tests, and were more likely to graduate on time; they were also more likely to be enrolled in college at the end of the following fall semester, almost a year later. About 98 percent of the pupils who spent more than 17 minutes doing the math problems successfully graduated, compared with 95 percent of students who spent four minutes or less on math. While that's a small difference, it was interesting to see it even after the researchers adjusted for other factors, including intelligence, gender, ethnicity, interest in math and whether the kids were in the free lunch program, Galla said.

The diligence test "was able to pick up a signal in college enrollment, and this was above and beyond things like cognitive ability, socioeconomic status -- things that we know tend to correlate with or predict later college success," he said. So it isn't just IQ or braininess that matters for academic achievement, but self-control as well. Yet, unlike IQ, the researchers believe self-control in schoolwork is a skill that can be taught and developed.

The correlation between performance on the diligence task and academic achievement is modest, but it is significant and important, commented Peake of Smith College. Whether the task predicts long-term consequences, and whether those are limited just to academics or apply more broadly to other aspects of self-control, are interesting questions to further explore, he said.

Staying on Task

The results held some surprises. "I was really shocked that some people actually stayed entirely on the math problems the whole time," D'Mello said. "It's a really difficult task. I can't do it myself, frankly." One super-diligent student did 966 math problems; a few kids did none at all.

The teens used a range of tactics to resist the distractions. "Some students would turn it into a game for themselves," Galla said. “So they want to just see how many problems that they can solve in the four-minute task blocks." Others did math until they needed a break, then switched to the fun stuff, which is "very healthy behavior," D'Mello said. "We all know there's advantages to taking a break. ... It's just that if you do that too much, then you get into trouble."

The research team will track the students through the six-year national college persistence study, and is updating their diligence task to include verbal and spatial reasoning problems. "The hope is that by giving a good measure, you could really inspire a lot of science," said D'Mello. At this point, the researchers say they don't envision the test as something teachers would routinely use to assess students in the classroom; it isn't designed or validated for that purpose.

Different Takes on Willpower and Grit

Not everyone believes the growing focus on individual students' responsibility to demonstrate self-control or grit is the best way to support academic achievement. Some progressive education experts worry low-income or minority kids who are struggling in school might be blamed for lacking grit as the primary reason for underperformance. Those critics point out that in many schools, poverty and an inequitable lack of resources are much bigger roadblocks to teaching and learning that need to be addressed.

This viewpoint identifies an economic, social and racial overtone to the notion that if students "would just put their nose to the grindstone harder and work harder, and be more diligent and more resilient, that they will do better," said Grant Lichtman, education consultant and author of #EdJourney: A Roadmap to the Future of Education, who moderated an impromptu heated discussion on this issue on his blog. "It sort of plays into the mythology of the American dream. It sounds good, but it may be more relevant to some folks than to others," Lichtman said.

It's absolutely necessary to structurally address the income inequalities in society, said psychologist David Yeager of the University of Texas in Austin. "Increasing diligence is no replacement for that." But he believes cultivating skills like diligence and grit in students can still be valuable. Yeager points to high-performing urban charter high schools in the poorest areas that boosted their graduation and college enrollment rates with substantial financial investment, yet still find that 60 to 70 percent of their graduates drop out of college. At several such schools where Yeager worked with Duckworth in studying low-income students in their senior year, kids’ performance on the academic diligence task predicted their likelihood of dropping out of college a year later.

"Diligence still matters when [students] make transitions to the next setting," Yeager said. And the unfortunate and unfair reality is that it matters more for pupils who are disadvantaged than for those with ample resources, he added. When rich students fail at self-control and make poor choices, they can fall back on family support or finances to keep them pushing through school. In contrast, "disadvantaged kids simply have fewer opportunities to make up for poor decisions," Yeager said. But a "sense-of-purpose" mindset intervention and other strategies that boost self-control and academic perseverance might help to narrow the inequality gap in education, he said. And, he points out, teachers and schools could start these interventions tomorrow.

A Debate Over Drudgery

Meanwhile, the research on self-control and the diligence task also raise broader questions about drudgery and the definition of success in education. "If you had done this study with the metaphorical Bill Gates in his senior year in high school," Lichtman said of the diligence task, "he would have gone to the other side of the screen" -- skipping the math problems -- "or he would have tried to start hacking into the computer."

Lichtman is one of many progressive educators who think schools need to teach content in ways that are more engaging and relevant to students' lives, rather than just drilling them with monotonous math practice sets. More and more teachers, parents and students believe that academic success should be measured not by repetitive regurgitation of facts, high test scores or even a college degree, he said, but rather by whether kids learn skills like collaboration, creativity, communication, empathy, and, yes, persistence and resilience, too. Instead of trying to make the assembly-line education system work better by turning up kids' self-control, Lichtman says it's time to focus on alternatives such as the deep, project-based learning experiences offered at Expeditionary Learning schools, among others.

"We all support a forward-looking view of education and are excited to see the future of learning," said D’Mello in response. The research on the diligence task is an attempt "to study how learning occurs today for better or for worse," he said. Not everything taught in schools can be fun and easy, he said -- that's why he and his colleagues are focusing on "boring but important" skill-building tasks -- and a lot of research has demonstrated the merits of impasse-driven learning, desirable difficulties and productive failure in promoting deep learning. "I'm in favor of doing what it takes to make learning more engaging and intrinsically motivating when appropriate, and fortifying kids with the appropriate mindsets, emotion regulation strategies and cognitive strategies when things get difficult and tedious," D'Mello said.

Regardless of one’s educational philosophy, there’s no question that diligence is universally necessary for anyone to accomplish something important in life that they really want to do. Any job, career or project, no matter how inspiring, will at times require the self-discipline to resist distractions and plod through some drudgery -- whether it's proofreading a book chapter for the umpteenth time or hand-pipetting hundreds of samples of reagents for a molecular biology experiment.

While diligence may be an independent factor that contributes to academic success, "it's really important to know it's just one contributor," Peake said. "And it's not going to determine by itself whether or not kids do well in college -- there are so many other factors that are playing into this." The relationships between self-control and positive outcomes are correlations, "not determinative kinds of things."

That was certainly true for the pre-schoolers who couldn't wait to gobble the marshmallow in the Stanford experiments, Peake said: "There are many, many kids who didn't wait, who by all the standards that you put out there do perfectly well in life."