What the moral case for evidence means for impact evaluation
Editor’s note: Ruth Levine, director of the Hewlett Foundation’s Global Development and Population Program, delivered these remarks on November 8, 2017, at the close of the International Initiative for Impact Evaluation’s (3ie) London Evidence Week, which included a series of public events focused on the importance of using evidence to inform international development research, policies and advocacy.
These ideas echo the moral case for evidence in policymaking remarks she delivered at the U.S. Agency for International Development’s Evidence Day earlier this year, and include a new call for evidence enthusiasts to ensure that the way they conduct and use impact evaluations reflects a larger moral aspiration toward truth, distributive justice, and human progress. Remarks below as prepared for delivery.
This afternoon I’ve set out for myself one easy job and one that’s a little harder. The easy job is to help you to see and articulate how your work is not only valuable as an instrument for technocrats, but is an expression itself of the timeless value of pursuing truth, justice, and human progress.
Building on some remarks I had the opportunity to make a few weeks ago at USAID in Washington, I’m going to try to make a moral case for the work behind evidence-informed policymaking, including but not limited to impact evaluation.
The second job is harder: to induce you to reflect in your daily work on what it means to engage with the morality of evidence-informed policymaking. Within the world of impact evaluation, how do we translate a moral idea into ethical practices? What are the methods behind the morals?
It’s an unusual topic for me to take up, since I’m neither a philosopher nor clergyperson; and it’s an unusual topic to hear in a conference of the International Initiative for Impact Evaluation, because the narrative around impact evaluation, and evidence-building more generally, invokes rationality more than morality. But no field can be sustained and advance unless it confronts and articulates its moral core. I want to think through with you how the conduct and use of impact evaluations answers a call to be our best selves, individually and as a society. Fittingly, we are having this conversation in the George Fox Room in the Friends House, and Quakers are known for their commitment to peace, religious freedom, participatory worship, integrity in personal and business affairs, and social justice.
If I were to ask each of you what draws you to work on ensuring that the most valid and reliable information – the most rigorous evidence – is available for public policymaking, and used by decision-makers, I’d hear a range of answers. Maybe: “I want people who design and implement programs and policies to know whether the stated goals are being met, so they can make adjustments if things aren’t going well.” Maybe: “I want to make sure that tax dollars aren’t wasted on ideas that don’t work.” Or “My graduate advisor was a ‘randomista’ so I am, too.”
I think if we talked a little longer, though, deeper meaning would emerge.
You might not think about it every day, while you’re writing papers or scanning the literature for information that can help you give the best advice to your minister, but impact evaluation calls us to something higher – to truth, justice, and the pursuit of – sometimes the struggle for – human progress.
These are deep topics and the hour is late, so I’m just going to touch on a few of the strong links between the work we’ve heard about today and this moral core. I’m pretty sure you will be able to improve on these ideas in the future.
First, the value of pursuing truth. Across all cultures, the value of telling the truth is upheld over the sin of lying. One of the main tasks that parents have is instilling in their children a feeling of self-love when they tell the truth, even when it is difficult, and the pain of a guilty conscience when they tell a falsehood, even when – or especially when – it is self-serving.
Well, the work of data scientists, evaluators, researchers – the people who engage in generating the information and evidence for program design and policymaking – is fundamentally about revealing truth. Often the truths are specific, not universal: This program was supposed to improve nutritional status a lot but it only improved it a little – or maybe not at all. Or, this change in policy will leave millions of people without basic health care.
Specific, yes, but intrinsically important, because they are part of an aspiration that we be truthful in public affairs – that true, verifiable, reproducible information is shining a light to frighten off the purveyors of falsehoods and ignorance.
Now, associating our work with the value of truth does not mean that we need to be reductionist or rigid about one unique truth or one way to understand the world. But it does mean that we are, each in our own way, trying to contribute to a full comprehension, and an appreciation for distinguishing facts from opinion, even when they are inconvenient.
But this work is not only about truth. It’s also about justice.
Analytic types have often left the fight for justice to others, to people who say they take a “rights-based approach” or who self-identify as human rights activists. But I believe that we need to state, affirmatively, clearly, and repeatedly, that what we’re doing is part and parcel of a millennia-long fight for justice – the struggle for just allocation of resources in a society, or distributive justice.
Let’s consider the elements of distributive justice. One important part is the question of what constitutes “just allocation”? As a society, decision-makers are routinely asked to make tradeoffs between conditions that are not intrinsically comparable: the wellbeing of a baby versus an elderly person, the welfare of those of us living today versus our descendants two generations hence. Societies have to make these choices around which there are no objective guides or guardrails, and individuals and groups draw on profound beliefs, whether from religion, historical precedent, philosophical tenets, or other sources. In the abstract, these are not issues on which the evidence community has any special claim, although we can and, in my view should, engage in our personal capacity.
What many in this room, and empirical analysts around the world, do contribute every day are the essential ingredients that make a quest for distributive justice real. They – we – offer the necessary information that permits those who are pursuing social choices to know what they are actually doing and how to do it better.
Values abstracted from implementation are insufficient to achieve distributive justice – and that’s where the evidence comes in. Fairness can be achieved only if full and unbiased information is available about current conditions, and about the costs and benefits of one way of acting – one policy option – versus another. Yes, we could guess or assume. But when measurement is possible, guesses and assumptions are lazy and irresponsible; not for the people most dedicated to a just outcome.
I am making a simple point: Empirical analysis is not a substitute for the value judgments that inform a theory of justice in any society. But empirical analysis is an essential complement to those value judgments, helping to turn the “what we believe” into the “what we do.”
As a final part of this big-picture discussion of the moral core of your work, I want to talk about the contributions that evidence-informed policymaking can make in service of the value of human progress. On any given day, human beings have choices to make about whether we maintain the status quo, doing things as we have always done them, or we work to advance, perfect, optimize, and seek a better form of ourselves as individuals and our societies as collectives.
Virtually all religions encourage or mandate ever-purer and more loving thoughts and behavior, whether for benefit in this life or the next one. The size of the self-help industry alone speaks to the impulse toward continuous improvement; businesses are judged on the basis of year-on-year growth; and nation-states are constantly trying to grow their economies, improve the health and wellbeing of their citizens, and sometimes expand their global power through military means. I think it’s safe to say that, in general, human beings seek to create better lives and better societies.
It is no stretch to say that this value – the value of human progress – is amplified and served by the work of people in the evidence community. We have to answer questions like, “As a society, where are we now?” That is a question that can be answered fully only by assembling facts in an organized way. Guessing will lead us astray. We have to answer questions like, “How effective are we now in solving specific problems?” Careful, thoughtful evaluation serves us, while wishful thinking does not. We have to answer questions like, “What are other approaches that might work better?” Innovation, based on reason but spiced with creativity, gives us the seed from which improvements grow. We’ve seen examples of all of this today, striving for more, better, stronger, and more lasting.
These are some elements of the moral core of evidence-informed policymaking, although they are not the only elements. If you are like the audience I spoke to about this in Washington a few weeks ago, this message may be warmly received – and perhaps I should stop here and let you go home feeling good about yourselves.
But I’m not going to do that. What I am going to do is ask that we all push ourselves to figure out whether our daily practices – the way we conduct and use impact evaluations – reflects a larger moral aspiration toward truth, distributive justice, and human progress. I do not have the answers, but I have personally found it helpful to ask challenging questions that connect imagination to implementation, aspiration to action. What do ethical practices in impact evaluation look like? The first response to that question is likely to be something about adhering to guidelines protecting human subjects – informed consent and ensuring confidentiality of sensitive information, for instance. This is so fundamental that research institutions and funders demand that investigators guarantee their protocols follow these standards. Good – but not good enough. There is so much more to do.
There are ethical considerations is selecting which programs to evaluate. Impact evaluations use scarce resources, so no impact evaluation should be proposed (or funded) unless it addresses a question that is truly important to the welfare of the people being studied (or people like them) and there is a reasonable chance that the findings from the impact evaluation will affect decisions, either directly or indirectly. We are not in a field where “academic interest” can be the sole or central driver of what work gets done.
Does this mean we should ensure that key stakeholders (or their proxies) are involved in setting research priorities? Yes, but we need to broaden our view of stakeholders to include both power-holders – those who make decisions about policies and programs – and those who are likely to be affected by the decisions. An ethical practitioner finds ways to include all stakeholders in setting out priorities.
A fundamental requirement for ethical impact evaluations is that we are seeking to answer genuinely new questions. We take the time to learn what’s already been done, and we build on that rather than recreating existing research. One tool toward that end is 3ie evidence gap maps, which clearly show where research effort has concentrated, and what the frontier questions truly are – whether they are about a new intervention, a new outcome of interest, or a context that is different in material ways from the places in which studies have already been done.
Ethical design and conduct of impact evaluations also demands that research isn’t extractive, but instead contributes to and builds on local capacity. This means meaningful involvement of researchers as close as possible to the communities. It means feeding back findings to those who have participated in studies.
The ethical dimension of impact evaluation goes deep – all the way to sample size. Thinking through the ethics of research challenges us to ask: when is it okay to estimate average effects, and when do we have to take the time and find the resources to look at the effects on important subgroups – poorer people, for instance, or women? This question, which can be inconvenient because it often leads us to recognize the need for larger and more complex study designs, is essential to ask when we have reason to believe that overall population averages silencing an important part of the story.
Ethical impact evaluation also depends on a very high standard of truthfulness and accountability. Researchers have a responsibility to do everything they can to support valid and reliable findings – from preregistering studies to prevent suppression of results, to being fully transparent about methods and data, to stating clearly the limitations of single studies.
And, on top of all this, the imperative of impact evaluations to be useful – to make the most of those precious resources – is fulfilled only when researchers make every effort to communicate their findings clearly, to the right people, at the right time. They may do this by themselves, or with support from others who are skilled at the fine art of translation from research to policy domains.
These are just a few of the ways in which how we do the work connects to the moral core of impact evaluation.
We have to stop the well-rehearsed and defensive arguments about the superiority of specific research methods, abstracted from an understanding of why we do this work, and what the why implies about how. As an evidence community, our shared moral values and commitment to ethical practices are far more important than our methodological differences. It’s time to lift up the conversation to a place that will be, in the end, far more important when the history is written about the early days of impact evaluation in international development.
This community has systematically built evidence about what works in social development, shown it’s possible to improve how donors and governments spend their money, and helped more people around the world improve their lives. But have we established the professional norms, standards, and skills that are aligned with fully ethical practices? I hope when you leave here today, you will do so with a renewed commitment to focus on work that informs real decisions, answers new questions, and does so in a way that is respectful and attentive to the interests of those whom we expect the evaluations to benefit. And, I’d ask 3ie, as a leader in the field, to find ways to deepen and formalize all of our thinking about how impact evaluation practices can serve what really matters: justice, truth, and a better future.
