Critics of Fordham's superintendents survey missed their mark

Steve Farkas is
veteran public opinion researcher, co-founder of
FDR Group
, and author of the
Fordham Institute’s recent report,
Yearning to Break Free: Ohio
Superintendents Speak.  The following was written in response to a review of the report by the Think Tank Review Project at the
National Education Policy Center.

The FDR Group’s recent survey of Ohio’s school district
superintendents, Yearning to Break Free (online
conducted on behalf of the Thomas B. Fordham Institute, found these local
education leaders eager to overhaul the collective bargaining process and to
increase their authority over staff and money. “Give us autonomy,” they said,
“and hold us accountable for getting results.” Easy to understand, right? Well,
two unhappy University of Houston professors reviewing the study ­for the
National Education Policy Center’s “Think Tank Review Project” had a lot of
trouble comprehending it.

From the get-go in their review, the professors failed to
realize that the study was giving voice to the opinions of these school leaders
– not our opinions as researchers or
even Fordham’s but those of Ohio superintendents. The professors say, “the authors of the study recommend
[emphasis added] that ‘two promising ways to save districts money are to give
superintendents greater control over combined state revenue streams and to
mandate a statewide health insurance plan…’” But the report itself stated “Ohio’s superintendents think [emphasis
added] two promising ways…” We (neither the FDR Group nor Fordham) didn’t
recommend anything. Yearning to Break
is a study of perceptions so the whole report is filled with such
modifiers as “superintendents believe, say, think.” It’s hard to see how the
professors missed that.

The professors had other troubles. They didn’t understand
why we would force superintendents to choose between what they called
“inappropriate dichotomies.” They balked when we asked superintendents what
would be more likely to lead to improved student achievement: “significant
expansion of management authority over staff” or “significant increases in
school funding.” But the dichotomy is valid and interesting – and it pushes
respondents to think harder and prioritize. (By the way, the result was 50
percent to 44 percent.) I’ve got a really unrealistic question for the
professors: If you had to choose, would you rather have lunch with President
Obama or Lady Gaga? See, even a far-fetched dichotomous question can tell you
something interesting about your preferences.

I have my own complaints about what the professors say. The
professors said the study aggregates responses for superintendents and charter
leaders. That’s just plain wrong – it doesn’t. They talk about a poor response
rate when the response rate was very high for a study of this type – 40 percent.
Real-world survey researchers – myself included – dream of getting such high
response rates on a regular basis. The professors say we don’t know how the
superintendents who didn’t answer the survey differ from the people who did.
Well, by definition that’s almost always the case. We made extraordinary efforts
to get non-respondents to take the survey (see the appendix of the report for
more on those efforts). That’s how we got such a high response rate.

Perhaps most amazing to me: in the end, after all the
mistruths and mistaken comprehensions, the professors agree that the survey correctly captures the attitudes of
superintendents! They say: “some aspects of the report are not surprising…it is
entirely predictable that superintendents would like to have greater control
over both teachers’ salaries and state regulations.” Oh. So this survey that
they claim is badly designed, badly worded, not representative, biased, and
narrow somehow got the right answer? Now that is interesting.