While public discussion of the education bill has focused on such hot-button issues as vouchers, much of the real drama in Washington-"what everybody was E-mailing and voice-mailing everybody else about"-is the "adequate yearly progress" or A.Y.P. formula, writes Nicholas Lemann in a narrative account of the progress of Bush's ambitious education plan through the Congressional gantlet. (Lemann's piece appears in this week's New Yorker.)
As if Lemann were a fly on our wall, a group of us inside-the-beltway types got together the other day to parse some of the perplexing dilemmas that will face Senate and House conferees when they turn to the A.Y.P., testing, and accountability sections of the pending E.S.E.A. bills. The deeper we went, the more alarmed we became at just how knotty some of these issues are. This section of the legislation simply doesn't lend itself to "splitting the difference" or melding rival versions. To have any realistic hope of ending up with something in this area that can actually be implemented without causing untold mischief, the conference committee must, in essence, start afresh. Here are some unsolicited precepts to guide that difficult process.
First, the gold standard for analyzing student achievement is value-added analysis that employs annual test scores for individual students, with results aggregated for schools, districts, states and other relevant institutional units-and whatever demographic groupings need their academic progress tracked.
Second, we must understand the key distinction between value-added and achievement-level analysis, also sometimes known as "standards-based" analysis. In the latter case, states and schools are judged based on the percentages of their students who achieve proficiency-or make some prescribed level of progress over time toward proficiency. The E.S.E.A. bills assume achievement-level analysis, which is certainly important, but manifestly incomplete. Far more useful is value-added analysis that concentrates on gains made by students. For such analysis to be practical, however, (at least) annual data are needed. This poses a problem, for most states lack such data now and the pending legislation doesn't mandate annual testing until four or five years after it demands the onset of "adequate yearly progress."
Third, whether value-added or achievement-level assessment is used, high-stakes rewards and sanctions for states, districts and schools oughtn't be based on a single year of data. They should be based on three-year (or longer) rolling averages. If states identify "failing" schools based on just one or two years of data, they may well "punish" the wrong schools. Hence until several years of data are available, Washington shouldn't prescribe the method by which a state or district identifies a failing school. People on the ground are likely to possess far more complete information about schools that allows them to identify those that are truly failing and in need of corrective action.
Fourth, it may be wise to keep the current accountability system (from E.S.E.A.'s 1994 amendments, though these have not yet been fully implemented) in place until there are enough data to install a new system using annual tests that have yielded several years of data. (The current system could be supplemented with a data-disaggregation requirement that isn't there now.)
Fifth, we need to recognize that annual administration of the National Assessment of Educational Progress (NAEP), as it currently operates, is extremely unlikely to yield reliable external audit data that can "confirm" state-level academic gains and losses. The most noteworthy characteristic of NAEP trend lines is their flatness. Odds are almost nil that state-level data disaggregated by race will show significant yearly change. Indeed, state-level data of every sort probably won't show significant changes from one year to the next. Unless one relishes the prospect of an interminable national debate about statistical significance, Congress needs to weigh new options here. A comprehensive overhaul (and expansion) of NAEP is one possibility. Better, I think, is to decouple NAEP results from high-stakes decisions about state-level academic progress. Require NAEP participation, by all states, yes, but test every two years, not annually, and use the resulting data only to inform the public about how their states are doing. NAEP data are fine demonstrating whether a state's standards are high or low, whether its students are doing better or worse over time, and how it is faring in relation to other states and the nation as a whole. But don't try to base federal rewards and sanctions on NAEP results.
Sixth, while we're at it, let's think again whether it makes sense for Washington to bestow any rewards or sanctions on states based on whether all children reach proficiency as defined by the state's academic standards. Our group was struck by the fact that this could easily penalize states that set high standards and thus encourage them to set low standards. Under American custom-as well as the pending E.S.E.A. bills-each state sets its own standards, then tracks its students' progress toward them. This means that a state that wants all its children to become academic lions will inevitably have greater difficulty attaining that standard than a state that sets about to create academic hamsters. Do we really want Washington encouraging hamsters by penalizing the failure to produce lions while rewarding states that do a good job of hamster creation?
Worth reading: "Testing Limits: can the President's education crusade survive Beltway politics?" by Nicholas Lemann, The New Yorker, July 2, 2001. Available for $3.50 at newsstands; not available at www.newyorker.com)
One highlight: (Lemann on testing, from "Testing Limits") "Bush is right that you can't figure out whether schools are doing a good job unless you have some way of measuring how much their students are learning. Go on the Web and read a Texas Assessment of Academic Skills test-are you really comfortable with kids' not knowing that material?...A guarantee by the national government of a decent education for every child is a noble cause, and so is the idea that all Americans will acquire a common body of skills and knowledge as they come of age. (Memo to Scarsdale, New York, where parents have been boycotting state tests: Your kids should learn that stuff, too, after which they can go back to being little geniuses.)"