The dangerous Dr Goldacre: Cochranian hero or garbage peddler?
A few months ago I launched a pre-emptive strike on Dr Ben Goldacre:
It strikes me that the impulse to control a problem rather than ‘uncontrol’ it away finds its most dangerous expression in the growing ‘evidenced-based policy’ campaign being headed up by all-round-good-guy-civil-liberties-defender Ben Goldacre, in association with the Cabinet Office’s Behavioural Insights team. Between them, they have produced a convincing story about how society would be much better off if all social problem interventions were assessed through Randomised Control Trial methodology developed by the biomedical sciences. This is all very well, but the promotion of such scientific rigour overlooks the need, in order to fit the method, to atomise problems and interventions in a way which embeds garbage can model social policy practice. Ultimately, a problem and intervention focus diminishes the power of people to make their problems go away, and enhances the power of the state to make interchangeable those problems and the people who have them. But that’s a longer blog.
I never did get round to that longer blog. Fortunately, though, someone much cleverer than me has. Here’s Will Davies, Assistant Professor at the Centre for Interdisciplinary Methodologies (and the brilliant Potlatch blog):
[T]he spread of medical epistemology into public policy is strangely anti-theoretical, thanks to a somewhat naively optimistic view of a single technique: the randomised controlled trial (RCT). RCTs operate according to induction. The facts are meant to speak for themselves; the data and the theory are kept neatly and self-consciously separate from each other. A medic, Ben Goldacre, has co-authored a paper on the policy applications of RCTs for the British government, which opens with the line ‘RCTs are the best way of determining whether a policy is working’…….
By adopting the inductivist epistemology associated with RCTs and Big Data, social policy-makers may learn a great deal more about the world, but may also become commensurately less sure of what it even means for a policy to work in the first place.
There is a risk that, as with RCTs in psycho-pharmaceuticals, diagnoses of social pathologies might start to spiral. Whole new problematic demographic sub-groups will start to appear to the gaze of the data analyst; new correlations of behavioural problems will be spotted; the perceived sources of our social, psychological and neurological malaises will simply multiply, and we’ll long for an age when it was all just a problem of the wrong ‘incentives’. Tesco’s Club Card is rumoured to produce 18,000 sub-groups of customer; the equivalent for the state would be 18,000 sub-groups of pathological behaviour to be nudged back into line. Without the extreme simplifications of rationalist theories, society would appear too complex to be governed at all. The empiricist response to the government’s paper title, ‘What Works’, might end up being ‘very little’, unless government becomes frighteningly ‘smart’. Alternatively, if theory no longer provides the procedures of evaluation, there is a risk that private backroom politics will do so instead…..
Nothing simply works unambiguously in social policy, gold standard or no gold standard. No policy delivers benefits without any ‘side-effects…….A policy might ‘work’ in terms of reducing unemployment*, but lead to an increase in family break-down. The inductivists response would be – yes, and that’s precisely the type of pattern that our new evidence centres will detect! So why use the rhetoric of ‘what works’, when it is plain that nothing unambiguously works, at least without also offering the standard (the QALY for social policy, if you like) through which ethical dilemmas and trade-offs will be addressed?
Will’s article is – like my earlier, less well formulated one - a call for the retention of political judgment at the heart of public policy, and a warning of what we risk by simply handing over control to the positivists, as the government now seems intent on doing. This is not to say that RCTs have no value, of course. But their use needs to be, of itself, a political matter, in which the area of research and the outcomes being tested are a matter for public and political argument.
This piece, therefore, isn’t a personal attack on Ben Goldacre (the title is just a brazen attempt at linkbait readership). It’s not an attack precisely because Ben recognises appears to recognise the issues raised above, in a way which Will doesn’t acknowledge.
In his most recent paper, on how RCTs might be become the norm in education policy-making, he stresses the need for professional involvement not just in the research, but in decision-making over exactly what research happens and why. The problem is that such a worthy aspiration can all too easily be hijacked by a governments (not just the current one) very keen on the idea of using research evidence to impose their own views of what success in education looks like, and less, but much less keen on the development of the kind of ambitious, democratically oriented research governance infrastructure that he advocates, and which he suggests will provide the ”opportunity…..to become and evidence-based profession, in just one generation”.**
So, for example, where Ben may take reassurance that “performance on specific academic or performance tests” have quite measurable outcomes, and means to improve such performance are therefore open to RCT methods, I see risks that RCTs might be used to embed such tests (and their associated curricula) at the cost of the wider educational enrichment of children.
Overall, I sincerely hope that Ben does turn out to be the leader, or a key opinion former in what would be a genuine Kuhnnian paradigm shift in governmental policymaking (or, as in the terms he uses in his new paper, that he becomes one of a new breed of Cochrane-style ”mischievous leaders, unafraid to question orthodoxies by producing good quality evidence”. Nevertheless, I can’t help worrying that Ben’s undoubted talents both as scientist and salesman are being co-opted for a deeply political process of depoliticisation.
To allay my fears, and get me (for what I’m worth) on side with his campaign for research both well-done and well-chosen, I’d like to set Ben a challenge – a challenge emanating from this sentence in his new paper:
Nobody in government would tell a doctor about what to prescribe, but we expect all doctors to be able to make informed decisions about which treatment is best, using the nest currently available evidence.
Quite right. There’d be uproar if a government minister started telling the medical what drugs were best, and s/he just wouldn’t dare.
But, as I set out at some length here, that’s precisely what the Prime Minister did to the nursing profession last January, when he casually announced that nurses would now be required to reorganise the way they work and institute ‘hourly rounding’, because he had heard that this was a good thing.
As I set out back then – using the available evidence - this was not just an utter outrage and insult to nurses, it also creates huge long-term risks of a downward spiral in care standards (as I set out more fully later, there may well be very different reasons for the apparent decline in hospital care standards, none of which have been picked up as possible causes by Cameron’s Nursing and Care Quality Forum (who now simply ignore my letters and emails), which was set up after he had made his intentional rounding decision and which has duly complied with his wishes without undertaking any further research.
But no-one but me noticed what was going on. Not even Ben, who you’d have thought might be alive to something so close to the medical profession. (I did tweet the importance of what was going on at him, in the light of my article, but he didn’t respond.)
So here, better late than never, is the challenge. If Ben really is a Kuhnian/Cochranian hero for our times, he’ll join me in a concerted call for a Randomised Control Trial around the use of intentional rounding in British hospitals, to establish whether it does actually improve care standards. The methodology will be necessarily complex, as outcomes are not tha easy to measures, but there are probably still enough wards left who have not instituted the new process to make it feasible to undertake the trial on the basis of existing organisational patterns (and therefore cheaply).
If he doesn’t bother, then I’m afraid I’ll be a little more convinced that, far from being a new Cochrane, he is indeed just a good exemplar of an actor in Olsen’s Garbage Can Model (this takes us neatly back to where I started): “a decision-maker looking for work”, content to take commission from the land of Gove for what is, if he’s honest, a somewhat thin paper bashed out on a wet Tuesday afternoon, happy to persuade government that they should look in his garbage can for a solution he came up with earlier.
*Will Davies’ example is an apt one, since reducing unemployment through ‘nudge’ tactics, and an RCT to prove that such tactics do ‘work’, is the example the Behavioural Insights team in Cabinet Office are most keen to promote as evidence of their worth. However, I think there’s a somewhat wider ‘side-effect’ to be considered, which exemplifies more clearly than Will’s postulate the way in which the unchecked growth of RCTs (and other positivist methods) might act to depoliticise social policy. This side-effect is that reducing unemployment/getting people into work is embedded, via the research, as sine qua non of economic as well as social policy. While this may seem reasoable at the moment – after all, full employment remains a mainstream aspiration – it may well be that it is no longer an appropriate policy aim. As Chris has set out, it may be that, whatever macro-economuc policy decisions are taken, we are in for a long period of stagnation or low growth. Says Chris:
[I]n a stagnating economy, aspirations are dangerous. When there’s no aggregate growth, one person can “get on” only at the expense of another. Aspiration thus becomes a (near) zero-sum game, which is a recipe for conflict and social tension.
In such circumstances, it’s arguable that getting everyone into work becomes a very bad social p0licy (and more recently Chris sets out a ‘supply-side socialism’ alternative of a citizen’s income). But if, in the meantime, policy comes to be driven by research into the best way to attain a harmful and decisive outcome……..well, you know where I’m headed.
** It’s worth noting here that by far the biggest challenge such a step change is likely to face is the uncomfortable reality that success or failure in education is currently caused mostly by factors way beyond the scope of anyone involved in education, and not therefore amenable to teacher-based solutions.