Professor develops ‘bullshit index’ for policy research

Academics should harness a “bullshit index” to call out questionable research that underpins policy interventions, an Australian educational scholar has suggested.

Adelaide University professor Pat Thomson has proposed a five-tier “policy bullshit index” – from level zero (no bullshit) to level four (extreme bullshit) – for appraising documents that mimic the “conventions” of evidence-based research “while failing to meet evidence standards”.

Writing in the Journal of Education Policy, Thomson said thinktank reports, advocacy briefs and government proposals routinely included claims that their evidence did not support. But rebuttal mechanisms were too technical, nuanced or drawn-out to garner attention in an age of soundbite journalism and populist politics.

“The incentive structure of the policy ecosystem rewards bullshit,” Thomson writes. “If we critical policy scholars are to…speak up for the public good, then we may need to become somewhat less polite.”

The idea evolved from Thomson’s “ongoing frustration” at how governments and lobbyists used “not very well-conducted research” to their advantage. “It’s almost impossible to do a piece of research that doesn’t have…some kind of limitation,” she told Times Higher Education.

“We teach in universities that you have to be transparent about those; you have to say what you can and can’t claim on the back of the research that you’ve done. That kind of hedging your bets…doesn’t necessarily produce the kind of recommendations that might be attractive to policymakers.”

Thomson’s index has characteristics in common with fact-checking and peer review, requiring knowledge of “basic principles of research” and ideally the subject matter under discussion. “But it is [a] set of criteria that you could use probably in…24 hours rather than three weeks.”

While there were now “very good” fact-checking sources, they could take weeks to work through the references in any policy document, she said. “By then it’s an old story. I’m really just suggesting, in a fairly…light-hearted but pointed way, that we’ve got to be able to do a bit better than that.”

The index methodology poses 19 questions to gauge whether documents meet “basic research standards” such as a thorough literature review and a study design that is fit for purpose and acknowledges its own limitations. “It’s likely to be more useful in teaching than…for any other purpose,” Thomson said. “It’s a way to help people critically read a text [with] a set of questions to see whether it stands up to scrutiny.

“I don’t really imagine people are going to be using it in public discourse, although it’d be interesting if they did. Somebody could use it for an annual set of awards.”

Thomson trialled the methodology to appraise a thinktank’s recommendations that public schools share resources and executive leadership teams. The report earned an overall assessment of level three (high bullshit).

She said the technique could be deployed in any area of public policy. In higher education, “lazy scholarship” justifying cuts to humanities programmes or new constraints in student visas, for example, might be an “eligible candidate for analysis”.

Thomson joked that “bullshit studies” was an emerging research discipline in fields from anthropology to education to philosophy. “It’s very big in business, for example, where people talk about ‘organisational bullshit’.

“It’s got…rhetorical appeal. It’s the kind of expletive you can just about get away with.”

john.ross@timeshighereducation.com