The cold shower

Common Core-aligned tests, career and technical education, liberal arts in elementary schools, and non-cognitive measures. 

Amber's Research Minute

SOURCE: Angela L. Duckworth and David Scott Yeager, "Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes," American Educational Research Association (May 2015).

Speaker 1:     This is the Education Gadfly Show!

This is Michelle Lerner of the Thomas B. Fordham Institute here at the Education Gadfly show and online at and now please join me in welcoming my co-host, the rock of ed reform Robert Pondiscio.

Robert:           As in Dwayne "The Rock" Johnson?

Michelle:       I guess. Is that who you want to be?

Robert:           Who wouldn't want to be Dwayne "The Rock"? The money? The charisma? The muscles? The box office!

Michelle:       The amount of food that man must eat.

Robert:           And can get away with it. Boy, I couldn't eat like that or I didn't eat like that since I was 12, probably.

Michelle:       I think I read that he eats like 40 pieces of cod a day.

Robert:           Ew.

Michelle:       Yeah. You want to rethink this one?

Robert:           No, I think I need to rethink this one. Although, you know what maybe even then. If that's what it takes to be Dwayne "The Rock" Johnson.

Michelle:       Just don't put the cod in the office microwave. That is bad co-worker attitude.

Robert:           That is bad behavior.

Michelle:       Yes. On that note, let's play "Pardon the Gadfly." Ellen, take it away.

Ellen:              A new article by Chester Finn argues that common coraline tests ought to be accompanied by reports that are blunt about whether a student is on track to be college and career ready, regardless of how parents might react to bad news. Thoughts?

Robert:           I was thinking, when was the last time you heard the words "blunts" and "Chester Finn" in the same sentence?

Michelle:       I was about to say, "Chester Finn being blunt?"

Robert:           Probably the last time you heard a sentence with either of those two words, right?

Michelle:       Yeah, but I think it was the last time he said something to me.

Robert:           I think that's probably right. It's a terrific piece and it's kind of an obvious point. I shouldn't say obvious, but it's like about time. Yeah, I mean that was the point of common core and the associated testing. Our standards have declined, the test scores are inflated, we're lying to ourselves about our kids' proficiency. This is supposed to be the moment where we tell the truth and because of all the stir and drum and controversy over common core, I think Chester Finn is right about this that we've been trying to lull ourself into a little bit of some nambulance if you like.

                        "It's going to be okay. Don't worry about it. The test will be ... It won't be that bad." It's supposed to be bad. It's supposed to be the cold wake-up call.

Michelle:       Well, I think the thing is here is states had the political will to adopt new standards, higher their standards.

Robert:           Right.

Michelle:       Then they apparently had the political will to survive a political fight.

Robert:           To withstand it, absolutely.

Michelle:       On the common core, really, only one state pulling out and now at this point we're just going to suddenly lack the political will to do the whole point of this thing, which is to be honest about where our kids are and performing.

Robert:           Well, to me I think, and to me Checker's piece is terrific but I think it's more about the messaging than anything else.

Michelle:       Oh, those communicators.

Robert:           There you go.

Michelle:       Always getting in the way of things.

Robert:           Evil, evil spin meisters trying to basically say, "No, it's not going to be so bad." Maybe it is going to be so bad and maybe that's the point.

Michelle:       I agree. I think this piece will be really well received by folks and the policy wonk world because this is exactly what it is that wonks want. They want the truth, they want honest feedback on where we're doing. I think the most important thing about Checker's piece is we can't wait until high school.

Robert:           Sure.

Michelle:       We have to - We know that kids fall behind really, really early and that is really hard to recover from that. Let's be honest early. Right, we don't want to say, "Your second grader will never go to college now because they're behind." No, that's not what we're - where we're going here.

Robert:           The implied point of common core standards and one assumed of the testing was to basically get your kids report card, as it were, test results and say, "Hey, is my kid on track for being college, career ready? Yes or no?" It looks like the reports that we're going to get from smarter balance and park are not going to say that at least according to Checker not until high school and by then the dye is cast.

Michelle:       I agree. All right on that dour note, Ellen.

Robert:           It's not dour. It is necessary. We have to be clear eyed. We have to tell the truth.

Michelle:       Amen. On that note -

Michelle:       Question number two.

Robert:           I'm practicing to be Checker. I'm being blunt.

Ellen:              A new MDRC report attempts to identify what makes certain career and technical education programs effective and how we can ensure that students are benefiting. Do you agree with the author's conclusions?

Robert:           Michelle, you read this report. I did not. I read your review of it but why don't you tell us what you found out.

Michelle:       This is a really great policy brief and CTE seems to be the new buzz for the Ed Reform community and everyone's talking about it. This is a great policy brief to read if you want to get the lay of the land. There's not a lot of research on CTE but they lay out what there is in research and the findings are slightly positive and stay tuned for future forum CTE report, I have to be the PR person and tout our own upcoming study.

                        They also lay out what CTE is, how it's changed from the voc ed of the past and there's a great table in the appendix that every CTE program in the country, and that's really fascinating because a lot of these are state-based reforms but they're also within schools so it's a lot of stuff out there. What's concerning is there are so many CTE programs and yet so few students in these programs.

                        I think what excites me most about CTE is it can be the both end of Ed Reform. It's for college and career and it's for everyone. We need to just forget about the voc ed of the past. This is not about pushing minority and/or low-income children into these programs. If anything, there is really great movements happening, especially in New York, where there's CTE's on engineering and computer science and all these things that I don't know anything about but I'd want my kid to do that even as a non-low-income person, a non-minority. These are really exciting programs that everyone should participate in.

                        The authors bring up two really interesting points that need to be considered. How we ensure that a cross section of all students are involved and, Robert, this second one will excite you; how to have an integrated curriculum. How to ensure that the technical part of this and the college path are integrated so that kids are learning algebra 2 and calculus or what have you, and learning coding or whatever the thing is. They interview a lot of teachers for this study or they code a lot of teachers and this seems to be the weak point. You've been on the curriculum bandwagon since, you want to give the year you started?

Robert:           I don't know. Since the earth cooled.

Michelle:       Exactly. I was excited to be reading a policy brief and even see the word "curriculum."

Robert:           Right.

Michelle:       That in itself is rare. It's a really exciting brief for folks to read and I think the key is everyone's talking about this, let's actually make it happen. Let's make sure it's high quality and how do we get there is still the big question.

Robert:           What I love about your review right now is where else but the Fordham Gadfly Podcast can you hear somebody say earnestly, "There's a really exciting table in the appendix."

Michelle:       That's true. I think I just signed my death sentence to always work at a think tank. There it is, it's done.

Robert:           Run, don't walk.

Michelle:       Oh, yeah. That's painful. I'm kind of regretting that. Alas, I will outlive that. I'm sure there are education researchers out there listening who are like, "Yes. I love the appendix. Someone reads it." There I am. Thanks, Robert.

Robert:           Busted. Sorry, Michelle.

Michelle:       All right. On that low point, Ellen, question number three.

Ellen:              Robert, you just published a piece in US News that made the case for restoring liberal arts in elementary schools. When did they go away and why should we bring them back?

Robert:           We all have our wonky moments. You have your appendix, I have my elementary school curriculas and now you can make fun of me. This is the thing that I've been concerned about since the earth cooled. As a former elementary school teacher, one of the things that I became militant about is how little content knowledge kids have, especially low-income kids like those that I taught in the South Bronx and is counter-intuitive to talents that has everything to do with reading comprehension.

                        The argument that I made in the US News piece, you hear these arguments from time to time, "Oh, we need to bring back the liberal arts in college. College isn't supposed to be about vocational training, it's supposed to be a broad, rich education." Well, so should elementary school because that's where it would really pay dividends in terms of building the stuff that we in reform care so much about, literacy, reading comprehension, the ability to write, speak, listen, etc. As counter-intuitive as it may sound and if you're familiar with the work of guys like E.D. Hirsch Jr and Dan Willingham then you know this to be true, kids really need a content-rich elementary school experience in order to communicate, to read, write, et cetera, speak and listen with comprehension.

                        Yeah, let's bring back the liberal arts. Let's start in kindergarten.

Michelle:       Who's against this? I don't understand. I just don't understand.

Robert:           Good question. Well, I mean how much time do we have. There are, I wouldn't say, valid reasons but understandable reasons. I mean we worship at the altar of student engagement in elementary school. We want school and we want curriculum to be about the student, to be student-driven, student-centered, these are these bromides that you hear all the time, but especially for low-income kids. I mean I've spoken and written about this at nauseum over the years that come from language poor homes, that don't have the enrichment that more affluent kids have. Not having a rich background or a rich, basic background knowledge can be just strictly determinative in terms of their ability to read and write clearly.

                        It's well-intentioned nonsense to say, no, well it should all be about student engagement and let the student follow their interests. There should always be room for that in any good education but it can't be dominated by that. We have to make some decisions as teachers.

Michelle:       Don't a lot of teachers bemoan the fact that we've taken away so many of these subjects in schools?

Robert:           Yeah. That's the great irony is and frankly I think -

Michelle:       I should say rightfully bemoan.

Robert:           Yeah, I think that's exactly right. We can't paint with too broad of a brush here but look some of the blame here frankly lies on us in the reform community who have pushed testing and I always say this, I've got a very complicated relationship with testing. I want to make sure that we have the data, I want to make sure that we use it to evaluate how schools are doing, but when you reduce reading to a series of strategies, tips, and tricks and that's the feedback that you get from standardized reading tests well, it should not surprise us that that's how it gets taught. We teach reading as a skill, I like to say reading's not a subject, it's a verb but we treat it like a subject a school subject.

Michelle:       All right. On that note, that's all the time we have for Pardon the Gadfly. Up next is everyone's favorite, Amber's Research Minute. David, notice I did not say Amber. She's on vacation, probably a well - I know a well-deserved one.

Robert:           Really? I didn't even notice she was away. Where did she go?

Michelle:       She's moving again.

Robert:           Again?

Michelle:       For the tenth time.

Robert:           Back into the witness protection program? That's what happens here at Fordham.

Michelle:       David, how do you even respond?

David:             I don't know how to respond to that, Michelle.

Michelle:       Would you just like to go to the research you have for us today?

Robert:           And then get on with his life.

David:             Sure thing. Today we have a new research brief from the American Educational Research Association entitled "Measurement Matters" assessing personal qualities other than cognitive ability for educational purposes, which was written by Angela Duckworth of the University of Pennsylvania and David Scott Yeager of the University of Texas.

                        In the brief, the authors compare three measures of students' non-cognitive skills. Student surveys, in which students self-report on their own non-cognitive skills. Teacher surveys, in which the teacher gives his or her assessment of a students' skills and so-called performance tasks such as the famous Marshmallow Test. After comparing these measures, the authors considered their suitability for various purposes, including individual diagnosis, improved practice, program evaluation and accountability.

                        According to the authors, every measure of non-cognitive skills has advantages and disadvantages. For example, student and teacher surveys are cheap, quick and generally reliable. However, they also suffer from various forms of bias such as reference bias, which occurs when individuals or groups rely on different frames of reference. For example, even though Kip Charter School students report spending more time on homework each night than students at control schools, they are no more likely to report that they come to class prepared. Presumably, because they are holding themselves to a higher standard. Unfortunately, the prevalence of reference bias means the schools that are best at promoting non-cognitive skills may actually score lowest on a survey measuring those skills.

                        Unlike surveys, performance tasks don't generally rely on the subjective judgments of students or teachers, making them less subject to many forms of bias. However, performance tasks have their own drawbacks. For example, to be a valid measure of a non-cognitive skill a performance task must be administered under carefully controlled conditions. More or less the opposite of the typical school environment. Furthermore, performance tasks are expensive and time consuming. A single performance task, such as the Marshmallow Test, can take as many as 20 minutes to administer by a trained experimenter. Administering several different tasks could take hours.

                        Obviously, using any of these measures for accountability purposes introduces additional dangers. For example, teachers could rate their students more favorably in surveys than they really perceive them to be or students could be coached to give the correct answers. In light of these dangers, the authors argue against incorporating surveys of non-cognitive skills into accountability systems or program evaluation. Although they have some hope that these surveys can improve practice.

                        As for performance tasks, the authors argue that it carefully designs sweep of performance tasks could be used in program evaluation if the tasks were brief and could be administered by computer. However, for the time being even this possibility is largely theoretical.

                        What should we make of all this? Well for starters, the authors are surely correct that existing measures of non-cognitive skills which were designed by researchers for use in their researchers are generally ill-suited for other purposes. As practical tools, these measures simply aren't ready for prime time.

                        Additionally, as I was reading the brief, I was struck by the inherent difficulty of the undertaking. Accurately assessing the character of America's youth on a massive scale. As well as the nearly incalculable risks associated with attaching consequences to session assessment. We already worry about teaching to the test, do we really want to teach to the performance task?

Robert:           To the Marshmallow Test. Well that's -

Michelle:       If it involved eating marshmallows every day I would be all for this, just keep practicing.

Robert:           There you go. Now wait a minute, David, maybe I missed this but is the upshot here that we should be very wary about expecting teachers to have an effect on things like grit and perseverance, isn't that the subtext of this?

David:             I'm not sure that that's necessarily the subtext.

Robert:           I may be remembering this completely incorrectly but Paul Tuft's last book I seem to recall that Kip, who I associate with, really valorizing grit and perseverance that they were ... Were they not issuing a report card about students' grit and perseverance? Does that not imply that the teachers should be in a position to improve it?

David:             Yeah, I don't think that the argument is that the teacher cannot improve it. I think the argument is that for each of the different measures there's just challenges with implementing it for purposes other than simply measuring it.

Robert:           Wait, is the measurement what's unreliable or the ability to effect change? Frankly, my impression was that we know that grit is helpful but we just, we're not sure how to improve it as an instructional matter.

David:             You know I think that's probably an additional matter. That's not really the ... That's not really where the authors go but at the end they sort of mention that well, even if we could measure this accurately we wouldn't necessarily know what to do with the information. I think, I mean we haven't even gotten to that problem yet, probably.

Robert:           There you go.

Michelle:       Fascinating. Thank you so much, David. You did an excellent job. This is your second time on the podcast?

Michelle:       I think you're a welcomed voice.

Robert:           A star is born.

Michelle:       All right. Thanks so much. That's all the time we have for this week's Gadfly show. Til next week ...

Robert:           I'm Robert Pondiscio.

Michelle:       And I'm Michelle Lerner for the Thomas B. Fordham Institute signing off.

Speaker 1:     The Education Gadfly Show is a production of the Thomas B. Fordham Institute located in Washington D.C. For more information visit us online at