Standards, Testing & Accountability

A new study by the NAEP Validity Studies Panel analyzes the alignment of the assessment’s 2015 Math Items (the actual test questions) for grades four and eight to the math Common Core State Standards (CCSS).

To do so, the panel enlisted as reviewers eighteen mathematicians, teachers, math educators, and supervisors who have familiarity with Common Core. This group classified all 150 items in the 2015 NAEP math pool for each grade as either matching a CCSS standard or not.

The reviewers determined that the Common Core and NAEP were reasonably aligned at both grade levels— not surprising, since CCSS writers had the NAEP frameworks at their disposal. Further, NAEP is by design broader than the CCSS and is supposed to maintain a degree of independence relative to the “current fashions in instruction and curriculum.”

Panelists found that 79 percent of NAEP items were matched to the content that appears in the CCSS at or below grade 4. The overall alignment of NAEP to CCSS standards at or below grade eight is even closer, 87 percent.

There is, however, variation in matches across content areas. In fourth grade, the least aligned content area was data analysis, statistics,...

Late last night, results were released from the National Assessment of Educational Progress (NAEP)—an exam that is widely considered the best domestic gauge of student achievement. NAEP is administered in each state, every two years, to a representative sample of fourth and eighth grade students in reading and math. With its rigorous content and stringent standards for meeting proficiency, NAEP provides a clear and honest view of student achievement in Ohio and across the nation.

The bottom line from these test results is that too many Buckeye children are struggling to meet rigorous academic goals. The NAEP results for 2015 show just 45 and 37 percent of fourth graders are proficient in math and reading, respectively. In eighth grade, only 36 percent of youngsters are proficient on each of the assessments. Relative to national averages, Ohio students achieve at somewhat higher levels—though some of that is due to its favorable demographics vis-à-vis poorer states. Yet their performance still trails well behind the top-performing states in the nation, such as Massachusetts and New Jersey. Compared to 2013—the last round of NAEP testing in these grades and subjects—student proficiency in Ohio was slightly lower (as were the national averages).

The following...

OK, everyone, back away from the ledge. With the release of NAEP data this week, the predictable deluge of commentary is well underway—mainly of the gnashing-of-teeth, rending-of-garments variety. NAEP may be the nation’s report card, but it is also the nation’s Rorschach test. Perception is in the eye of the beholder, and many see darkness and misery: “A Decade of Academic Progress Halts,” says the Los Angeles Times. “Student Score in Reading and Math Drop,” says U.S. News & World Report.

One of the frequent criticisms of NAEP punditry is “misNAEPery”—the sin of attributing fluctuations to particular policies, for example. One particularly virulent form of this fallacy—failure to account demographic changes in states over time—has become slightly less tenable this week, courtesy of this illuminating analysis by Matthew Chingos of the Urban Institute.

Not every state is the same. States with higher concentrations of black and Hispanic children, low-income families, and English language learners (ELLs) have a harder time rising to the top because they have more students mired at the bottom. But when you adjust for these demographic realities, a different NAEP emerges. There’s Massachusetts, still sitting pretty atop the tables. But Texas and...

Over the weekend, President Barack Obama received high praise from parents and teachers for acknowledging that testing is taking too much time away from teaching, learning and fostering creativity in schools, and recommending that standardized tests take up no more than 2 percent of total school instructional time. Frankly, this is arrant nonsense.

From time to time, I'm asked to give a talk about education. If I look at how I spend my time over the course of a year, giving presentations and speeches is a very small part of my job—less than 2 percent. However, if my effectiveness were to be judged on the audience response to the handful of talks I give each year, I'd spend a lot more time writing and practicing speeches. I'd fret endlessly over my PowerPoint slides and leave-behinds. I'd sprinkle in more jokes to be entertaining; I'd probably say whatever I thought would get audiences to like me more, rather than challenging my listeners. I'd definitely spend a lot more on suits and dry cleaning than I do now.

But most critically, I'd spend far less time on all the other things I do—writing, reading,...

Unfortunately, the rumors, predictions, and surmises were correct: Scores on the National Assessment of Educational Progress (NAEP) are mostly down or flat. The worst news came in eighth-grade math, where twenty-two states saw declines. One of the only bright spots is fourth-grade reading, where ten states (as well as Washington, D.C., Boston, Chicago, and Cleveland) posted gains.

Why this happened will be combed over and argued. So far, it feels like anyone’s guess (more on that below). But there’s no denying that it’s bad news. It had come to seem like NAEP scores would always go up, at least over the long term, just like it had come to seem like murder rates would always go down. Now the real world has intervened to remind us that social progress is not inevitable. Let’s not sugarcoat it: This is deeply disheartening for our country, our K–12 system, and especially our kids.

As our friends in the research community like to remind us, it’s impossible to draw causal connections from changes in NAEP data; doing so is “misNAEPery.” Yet we can’t help but search for explanations. And we can certainly float hypotheses about the trends—educated guesses that can then be tested using...

President Obama and Secretary of Education Arne Duncan deserve credit for acknowledging this weekend that there’s too much testing in our schools today and that “the administration bears some of the responsibility.”

Indeed it does. That’s because its decision to condition ESEA flexibility on state adoption of teacher evaluation systems has not only raised the stakes of reading and math tests (making them less popular and potentially more damaging to the educational enterprise). It’s also led to a proliferation of tests in “non-tested subjects”—everything from P.E. to social studies and beyond—for the sole purpose of collecting data to judge teachers’ effectiveness.

Yet, as Matt Barnum argues persuasively at the Seventy Four, the feds aren't willing to actually fix this problem:

The new report did not capture a precise measure on what proportion of tests were required by teacher evaluation, but it does point out that many states have put in place new assessments “to satisfy state regulations and laws for teacher and principal evaluation driven by and approved by U.S. Department of Education policies.”

But an initial reading of the department’s guidance suggests it is sticking to these policies: “The Department will work with states...

The French mathematician and philosopher Blaise Pascal famously posited that whether or not one believes in God, it behooves us to behave as if he exists. What have you got to lose? If you’re right, you wind up heaven and spare yourself eternal punishment in hell. And if not, well, what did it cost you apart from a few earthly pleasures here and there? Pascal’s Wager basically suggests that your upside is infinite, while your downside is relatively small. So do the right thing.

We need a Pascal’s Wager of curriculum. Schools are going to teach something, so it behooves us to ensure that the textbooks, workbooks, and software we put in front of students are coherent and of high quality. As this report from the Center for American Progress shows, crappy curriculum costs every bit as much as the good stuff. The authors found “little relationship” between the cost and quality of instructional products. And switching to a more rigorous math curriculum, for example, can deliver far greater returns on investment than other reforms. “The average cost-effectiveness ratio of switching curriculum was almost forty times that of class-size reduction in a well known randomized experiment,” the report notes.


  • If there’s one thing we know about standardized testing, it’s that parents absolutely loathe it. With outrage building across the country over Common Core and its affiliated assessments, it’s no surprise that scads of irate parents have been pulling their kids out of tests. Why, just look at the public opinion polli—oh, that’s weird. According to a new survey conducted by the Education Post, parents aren’t actually incandescent with anti-assessment fervor. Forty-four percent of polled parents say that the tests are fair, versus 38 percent who claim that they’re unfair (18 percent say that they’re unsure). The results pretty closely track those of the 2015 Education Next poll, which found that two-thirds of both parents and the public at large support federally mandated testing. All polls come with caveats (a slight manipulation of wording can skew results dramatically), but reformers should greet these results as welcome evidence of parental patience and wisdom.
  • Chicago was probably a lot more fun in the 1920s, when bootleg liquor flowed freely, gangsters and molls packed the speakeasies, and tough guys spontaneously broke into Bugsy Malone-style song. The good news is that the outlaw tradition carries on in the school district:
  • ...

Editor’s note: This post originally appeared in a slightly different form on the Seventy Four; that one also lambasted Arkansas for backpedaling on its cut scores. Since then, Arkansas acknowledged that it had erred in how it described the state’s performance levels and clarified that it would use the rigorous standards suggested by PARCC.

Way back in 2007, we at the Thomas B. Fordham Institute published a landmark study with experts from the Northwest Evaluation Association: The Proficiency Illusion. It found that state definitions for reading and math “proficiency” were all over the map—and shockingly subpar almost everywhere. In Wisconsin, for instance, eighth graders could be reading at the fourteenth percentile nationally and still be considered proficient.

This was a big problem—not just the inconsistency, though that surely made it harder to compare schools across state lines. Mostly, we worried about the signals that low proficiency standards sent to parents: the false positives indicating that their kids were on track for success when they actually weren’t. How were parents in Madison or Duluth supposed to know that their “proficient” son was really far below grade level, not to mention way off track for success in...

The donkey debate edition

Education in the first Democratic debate, whether Common Core is winning the war, Arne Duncan’s influence on President Obama’s education policy, and the effects of scaling-up pre-K. 

Amber's Research Minute

SOURCE: Dale C. Farran and Mark W. Lipsey, "Expectations of sustained effects from scaled up pre-K: Challenges from the Tennessee study," Brookings Institution (October 2015).


Kevin:                   Good morning Mr. and Mrs. America. From border to border and coast to coast, and all the ships at sea, this is your host, Kevin Mahnken at the Thomas B Fordham of Institute. Here at the Education Gadfly Show and online at Now, please join me in welcoming my co-host, a first time guest of the Gadfly show so take it easy on her, Ellen Alpaugh. The Lincoln Chafee of Education reform. I intentionally picked Lincoln Chafee because I figured there was a possibility that maybe you haven't been keeping up with the news or-

Ellen:                     Oh I know, he's running for President.

Kevin:                   No, no, I'm going to put this to the test. I'm going to see. I've written up a multiple choice question about who Lincoln Chaffee might be. Of these four, Ellen, try and see who or what Lincoln Chaffee is. I can give you a hint. It's concerned with Rode Island, okay? So Lincoln Chafee is either A-

Ellen:                     I knew that.

Kevin:                   A small Woonsocket law firm that handles trusts and estates. B, a soft rock duo famous for their 1978 hit Kiss on my Lips.

Ellen:                     Ohh, that's tempting.

Kevin:                   Or C, the parent company of Lincoln Logs. Which of those 3?

Ellen:                     Is it C?

Kevin:                   No, no, it's not.

Ellen:                     It's none, there's none of the above.

Kevin:                   It's D, Lincoln Chafee is the former Senator, current-

Ellen:                     Of Rhode Island.

Kevin:                   Recently former governor of Rhode Island.

Ellen:                     Yeah.

Kevin:                   As you may have seen, he was present at the Democratic presidential primary debate, first one of the season. Now, we were kind of looking at that. What did you think of sort of some of the performances? Were you like thumbs up, thumbs down on any of the candidates?

Ellen:                     Yeah. I think Hillary did pretty well and yeah, I don't know.

Kevin:                   Yeah, she looks like she is probably the front runner. I mean that's sort of what everyone assumed at the out said. However after like 6 months of relentless criticism, I think maybe people were starting to get doubts. I think she really reasserted herself last night as the candidate to be. I was not crazy about the performances of everyone else. Martin O'Malley looked like he's basically running for vice president, trying to flag the Glass Stegall issue and see if he'll get a vice presidential nod. Bernie Sanders looks like he's running for president of the student government at Sara Lawrence college. It's just not like the line up of Titans that you see for the Republican. Now, actually our first question has to do with the debate last night. So I'm going to throw it over to Clara to ask our first question.

Clara:                     The first democratic presidential debate was last night. Was the lack of discussion around education K-12 concerning?

Kevin:                   All right, that's an interesting question. As you mention, there was really very little discussion of education, I mean none of K through 12 education.

Ellen:                     Yeah, nothing on K through 12 but there was some mention of college.

Kevin:                   Yeah, higher ed got some discussion. Sort of all the candidates had college affordability plans, how to reduce tuition costs and so forth at state universities. But what about, is this basically just a case of it being a pre-non sexy issue? They're not going to mention it and no harm, no foul, what do you think?

Ellen:                     Well I think it goes beyond it not being a non sexy issue. I think the candidates are really scared to make a stance on this. It's super divisive. Anything they're going to make a stance on whether it be standards, or school choice or anything, they're going to lose as many voters as they may gain by doing so. That's my thought and it also speaks to, it's also supported by the fact that Democratic candidates just recently declined an invitation from Campbell Brown to speak about this very topic at the education summit later this month.

Kevin:                   Yes. It sounds like, yeah you're quoting from our brilliant college Kate Stringers. He's castigating the Democratic candidates for not going to this 74 candidate form.

Ellen:                     That's correct, Kevin.

Kevin:                   And that does strike me as sort of a case of political cowardice on their part. Do you, I mean, my impression because I also happen to write a little piece about this, still though you weren't good enough to bring that up, it was that basically these issues that don't get brought up in political campaigns, they're ... Issues that don't end up being talked about, they don't end up ever being addressed either when a candidate comes into office. That is, political scientist have sort of looked at this and what they found is that candidates end up making promises and presidents end up keeping promises. Presidents prioritize delivering on things they have promised to their base, even if that's just in the primary. You know, and then they get through the primary and they're running in the general election. They're going to do fine but they always end up going back.

                                I mean President Obama campaigned on health care reform. He campaigned a little bit on climate change. I've seen action on both of those issues. Now something like Guantanamo Bay was like a want to have but it’s not like he really ran on it in 2008. So you see a lot less activity on that issue. Do you suppose that if we heard anything on K through 12 from these guys it would be anything ground breaking or ...

Ellen:                     Well, not so much. I think that everyone's track records on the democratic side show that they're pretty much in support of most of the reforms, the standards at least. And whatever name they are.

Kevin:                   Yeah, yeah, yeah I think you're right. Like Hillary last night had to dodge a lot of accusations that maybe she's running for Obama's third term. And it sounds to me like probably that's ... They're going to end up lining up behind high standards, probably going to have to square the circle with charter schools which democrats have had a hard time with for a long time. Maybe in fact it's kind of the dog that didn't bark. We'll see. I hope we'll get more talk from both Democrats and Republicans about this. Because it’s not as if the Republicans have been chatter boxes on education either. Clara, how about number 2?

Clara:                     Secretary of Education, Arne Duncan's tremulous education reform policies prove divisive amongst liberals. Did he manipulate the Obama administrations vision for eta form?

Kevin:                   Excellent question. Clara is getting down to the bottom of the mystery of just who has been pulling president Obama's puppet strings all these years.

Ellen:                     It's been me.

Kevin:                   It's been, Ellen Alpaugh from her perch in the corner at, pushing a mop and a broom around, in the office at the Thomas B Fordham of Institute is actually been dictating education policy. This is just interesting. This question comes from an article that was written by Jonathan Chat from New York magazine. I highly recommended it. It’s called Was Arne Duncan Secretly Obama's Boss All Along? And what he effectively says is that Arne Duncan resigned a few weeks ago. That's sort of old news but what you're starting to see now is the retrospectives from different political commentators, liberals and he's quoting a piece specifically from Charles Pierce of formally the Boston Phoenix, now of Grantland and an idol of mine journalistically. Pierce says that, I can't quote it exactly but basically the reform type policies, things that race to the top, prioritizing high standards, these were policies that President Obama basically implemented reluctantly. He uses the word at the behest of Arne Duncan. That he went along, that he went along to go along. Ellen, is that your impression of how the White House works, that the Secretary of Education who calls the shots?

Ellen:                     I wouldn't say so, no.

Kevin:                   I shouldn't think so.

Ellen:                     I hope not.

Kevin:                   Yeah, I mean, it seems to me that ... You're work on most in the comms side I mean, what has your impression been on people's views of Arne Duncan? I mean, I think he's sort of been made to take the fall here.

Ellen:                     Yeah, and it’s really easy to beat up on somebody who is the face of education reform for the United States, especially when a lot of people don't want anyone in the federal government to be talking about it at all.

Kevin:                   Yeah, quite so, quite so. Duncan in a lot of these cases, the fall guy like you say is an un-elected person. It's somebody that was sort of thrusts in a roll and made the face of a movement of a set of policies. Arne Duncan is convenient for that in that way he sort of reminds me of Eric Holder. Former attorney general who people speculated was like Obama's lightning rod. That these figures were being kept in place because they sort of allowed liberals and conservatives to both beat up on the straw man and Obama would escape unscathed.

                                I have to say that the argument to be seems to be a fallacious one. These liberals are saying, you know it’s too bad that Arne Duncan got to run education for the last seven years if only the president had known, had he ever looked at what was coming out of his department of education. Which is, you know, obviously absurd. It reminds me of like a Gallo prisoner in Russia in the 1950's saying like if only Comrade Stallings knew how bad things were here, he wouldn't stand for something like this. I mean, this is the president. The president makes the pick, he makes the selection for personnel. And there after, his policies are implemented. So I guess I'd like to see from now on for a little more accountability. If you have something against the president, well then NEA, whoever the critic from the left may be, direct it towards the guy who is probably making the call, make sense?

Ellen:                     Well said, Kevin.

Kevin:                   Aw, thank you. I'm going to throw it over to Clara for number 3.

Clara:                     Politico recently wrote that Common Core has recently won the war. Is it true? Is the great battle of education reform over?

Ellen:                     Well, I guess we can all go home now.

Kevin:                   Yeah. I was hoping so you know. Our human resources guy will be happy we don't have to rent out the office space anymore.

Ellen:                     Yeah, job well done, job well done.

Kevin:                   We can just, I can go back to hand gliding lessons. This is a claim that was made in Politico the other day. It was two days ago, basically speculating that after years now of work setting up the conservator, writing the standards, getting states to buy in from the National Governor’s Association, that effectively Common Core is now just a fact of life. It's in place now I believe in 42 states, the District of Columbia also has it. You've had 2 or 3 states like South Carolina and Oklahoma who either initially adopted it, then pulled out or who simply never adopted it in the first place. It's, but for the mast majority, what was the figure for the kids who are in class rooms with Common Core?

Ellen:                     I think it's about 40 million right now, that's 4 in every 5 kids. Is it 40?

Kevin:                   Yeah, 40 million. Okay, yeah, yeah, yeah, it's 80 percent so yeah that'd be 4 out of 5, so 40 million. The vast majority therefore are in states with Common Core. Does this strike you, Ellen, as kind of a bit of a preemptive declaration of victory? I mean are we actually ... What's your stance, I mean are we actually there yet?

Ellen:                     The battle is only half won I'd say.

Kevin:                   Okay General Eisenhower. All right.

Ellen:                     Yeah, so Common Core has pretty successfully been adopted by most states. I think that's hopefully the way it's going to be. The next challenge is definitely making sure that the tests that are supposed to measure how kids are doing with these standards are actually measuring what they say they're going to do. They are actually aligned to Common Core. And beyond that, there's communicating those results to parents and students and finding a way to deal with the fact that a lot of these kids who have previously been told they are doing okay, are actually really not.

Kevin:                   Yeah, yeah. That's what we've addressed on the Fly paper blog a lot of it, you know, fits the proficiently illusion. The idea that we've been, we've now been telling generations of kids, generations of parents, your kids doing fine, don't worry about it, yes of course there's an education crisis in this country but it has nothing to do with darling little Aiden. Unfor- I hate that name- unfortunately, unfortunate now eventually there's going to come a time where you set your sights a little higher and unfortunately you're not going to be able to deliver that comforting message anymore.

                                I think the other point, the one last thing we have to settle here is the political question. You're very right Ellen that implementation is important and that's on the job of ed reformers to hold people's feet to the fire. But politicians have a roll here too. Once a policy, especially one that's not produced by federal statue, like a lot of Common Core critics claim that it was, one that was voluntarily adopted by the States. Once it's put into place, that doesn't mean it's set in stone, right? Then President Bush, again through statute. Then these things went to congress, initially they could have a little more staying power but President Bush put in place tax cuts in 2001 and 2003. Those tax cuts have changed because government changed. Just off the top of my head, President Obama made Obamacare a thing that the future of Obamacare is not set in stone either. So if we think we have a good thing here, my understanding is we need to make sure we preserve it. Especially where now in 2016, we've got politicians running for president who are making it basically the point of their campaign, some of them, repeal Common Core. It seems to me there are, who was it, Bobby Jindal, Chris Christie, I mean who's running against it this? I mean there are figures who actually hate Common Core.

Ellen:                     Oh yeah. I'd say that most of the Republican cohort, right?

Kevin:                   Well it's not all, prominently Jeb Bush but if you care to speak about Common Core, it seems as though, and not a few liberals as well, if you speak about Common Core, generally speaking, you're bringing it up to condemn it. So I don't think we've turned the page yet. But hopefully we've reached if not the beginning of the end, the end of the beginning. I believe now we are going to transition to my main man David Griffith for our research minute after a quick word from our sponsor Lincoln Chafee, state trusts serving Woonsocket, Rhode Island since 1931 ... And now welcome our guest today, the Jim Webb of education reform, could we call him that ?

David:                   You could not.

Kevin:                   I could not call you the Jim Webb, how about he Jeff Merkel? The Oregon senator-

David:                   Merkley, Merkley.

Kevin:                   Merkely?

David:                   Yeah.

Kevin:                   I'm getting confused with the German Chancellor. I think they share one another's charms. It’s that native charisma.

David:                   Are you going to introduce me?

Kevin:                   Yeah, yeah, David Griffin, everyone knows who the Jeff Merkel of education reform is.

David:                   Okay, glad to be here Kevin.

Kevin:                   And he's going to share his homework, which was a study of the preliminary results of the Tennessee volunteer pre-k effectiveness study.

David:                   All right, thank you Kevin. That is the study we will be talking about today and it’s an important study. And what makes it important is its design. It’s based on a randomized control trial which is though sort of the widely known gold standard for rigorous research in education and elsewhere. So this isn't the first time such a design has been used to evaluate a pre k program. But it is the first time this method has been used to evaluate a scaled up state funded pre k program and we'll talk about the difference between those. For this study, the researchers have, basically they've been tracking the progress of about 3,000 students in Tennessee as they enter elementary school. About two thirds of these students participated in Tennessee’s pre K program. Of those 3,000, about a thousand were evaluated more intensively so for this group the researchers sort of directly administered a number of scales test that are gauged at the pre k level obviously. And also their teachers provided sort of annual ratings of their non-cognitive skills. What did they find? Well unfortunately this is yet another disappointing pre-K study.

                                The results are extremely discouraging. Like a lot of other studies, this one finds that participating in Tennessee program does give kids a head start when they enter kindergarten in a lot of measures, but by the end of kindergarten, this advantage had basically disappeared a lot like it did in the famous head start impact study. And worse, by second grade, the kids that participated in the Tennessee pre k study actually scored lower than the kids in the control group on most of the measures. So since the studies come out, lots of some pre k advocates have sought to down play it and arguing that Tennessee offered lower, has a lower quality pre k than other states. It's not clear that there's much evidence that that's actually true, and more over these results obviously fit into a pattern that we've seen elsewhere which is the clear and initial benefits followed by this really rapid fade out as the kid, the kids enter the K-12 system. So as authors Dale Farren and Mark Wipsy note, there is some as yet poorly interaction between the pre-K experience and the experience children have in subsequent grades that fails to carry forth the momentum they gain in pre k. So at least for me all of this is pretty depressing.

                                If you believe as I do that high quality pre-K does have the potential to change the lives of many underprivileged kids as it seems to have done in a lot of the early studies like the Perry Preschool study and Advocacy Darwin study. So unfortunately were operating with fuzzy definitions of pre k and high quality and it's not clear if we can scale that sort of success.

Kevin:                   Now, commentators on this study, it sounds as though you may share this, there's a certain amount of fatalism. You mention the head start study as well. Are there and I can't blame you for being fatalistic, you're a trail blazers fan, is there reasons for hope on the horizon? I mean it does seem dispiriting, I agree.

David:                   Well, so as Audrey was mentioning before we recorded this, there is this evidence from these long term studies that actually the benefits of pre k can actually disappear or become dormant for a number of years and then sort of reemerge from the data later in life. So even though they don't show up in test scores in the K-12 years, it sometimes does seem like kids do better once they are adults. The problem is, there’s a 30 year time lag between the point of which we offer pre-K and the point of which we know if those results actually occur. Right, so it's pretty difficult to make a policy I think based on benefits that won't show up for you know till 2030. I buy those studies but it also it still leaves us in a pretty tough place I think.

Kevin:                   Yeah it certainly does. A tough and lugubrious place. Thank you very much David Griffith for stopping by to bring down the Educational Gadfly show. And on that sour note, that's all the time we have for this week’s Gadfly show. Until next week-

Ellen:                     I'm Ellen Alpaugh.

Kevin:                   And I am Kevin Mahnken for the Thomas B Fordham Institute signing off.

Radio:                   The Education Gadfly show is a production of the Thomas B Fordham Institute located in Washington DC. For more information visit us online at