Kirsty Newman was very nice to answer this post in which I bring my very own evidence-based policy flowchart®. She even said she liked it. However, there seems to be two objections. She says – correctly – that much of policy making involves personal views, politics etc, which are the factors that ultimately decide what policies should be implemented or not. But the flowchart talks about recommendations of policies, not decisions. The second objection is that it would not apply to many situations but only a very specific subset of policies.
There is an important difference regarding the first objection. It is true that democracy presupposes that experts won’t decide everything according to their own beliefs and priorities. The people have ultimate sovereignty over decisions. Politicians, being representatives of the people, have ultimate power over experts. That’s how things are and – in the humble opinion of this intermittent blogger – that’s how they should be.
But experts are called upon by democracies to give their, well, expert opinions. And that should not be confused with the opinion that the people or the politician already have and think the expert should share. This is why all of the paths of the flowchart end in a recommendation, not a decision. Too often, however, experts seem to take into account political factors to make a recommendation, and then politicians and the public use those recommendations to form their preferences about policies and a vicious circle is formed. So I still do not see anything wrong with the flowchart.
Maybe an analogy will make my point clearer. Suppose you are a patent lawyer, and an inventor comes to you and says he wants to apply for a patent on what effectively is a car, telling you how certain he is that the patent will be granted because it is such a novel and non-obvious idea. You have the duty of telling him that his chances of getting the patent are very slim. Suppose, on the other hand, that he says: “I don’t care what you think my chances of getting a patent on my invention, I’m asking for your expertise on how to produce patent applications and you let me worry about my odds”. I would be perfectly fine doing that job.
With regard to it being applicable to only a very specific subset of policies, I don’t see why would that be the case. Blame it on my limited imagination, but I can’t think of an example of a policy where that flowchart would not be applicable – with the important caveat that, as said above, it concerns only recommendations, not decisions.
Let’s use, as an example, GiveWell recent discussion of taxes on alcohol. This is the kind of policy issue that I can envision some people not wanting to use the flowchart. But let’s apply it, then. Is there plenty of evidence that the policy would be beneficial? I hardly think so. For one, there are a number of studies indicating benefits of alcohol consumption, which might even be large. Now these are observational studies, and they have the issues that observational studies often have. So while I would not count that there is plenty of evidence for it, I would not say there is plenty of evidence against it.
The next step would be harder. Can we get evidence with an RCT? It would take a long time. So both “yes” – in the future – and “no” – right now – are valid answers. So let us go with “no”, because if we answer yes everything would be simple.
We would then ask ourselves: can we get the evidence if we run the full project (in this case, establish the tax)? We sure can. We could, after establishing the tax, randomize a few not to be affected by it and even establish some kind of compensation for them to participate in the project. Considering the potential net benefits for the whole world of getting this policy right, this compensation could even be pretty large.
The reason I think this flowchart can be applied across the board is that it has a clear answer for when there are not clear answers: don’t do anything. Policies involve limiting people’s activities, so we should only implement them when we have very good reason to think they are actually going to work.
Can you think of a counterexample where the flowchart would not apply? I would like to hear about it.