Like a whole lot of other people, I’ve become engrossed by Serial. (If you’ve somehow managed to avoid its seeming cultural omnipresence: Serial is a weekly podcast in which a reporter sifts through clues and patches together fragments of people’s memories to try to solve a long-ago real-life murder mystery—and possibly exonerate the man convicted of the crime.) It’s a messy story, complete with ambiguous evidence, false leads, and contradictory points of view. And that’s what’s so captivating. Listeners must puzzle it out along with the reporter, applying powers of observation, inquiry, and judgment to understand both what happened and what it means. The untidiness of the reporter’s investigative process as we follow along with her creates a kind of audience engagement that no polished three-act structure, complete with a tidy ending, ever could. For a taste of it, just tune in to the “meta-conversation” on Reddit, in Slate, or around the water cooler at many offices (including our own).
So why am I writing about this when I’m supposed to be focusing on global development policy and programs? Because the phenomenon of Serial, and the popularity of detective stories in general, gives us a clue about one of the great unsolved mysteries in global development: Why don’t people who want to get the most good out of every development dollar spent pay enough attention to what works and why? Why don’t more people truly learn from program evaluations?
I think one of the reasons that the findings from program evaluations don’t gain traction—either in the minds of individual development practitioners or with the institutions whose programs might be improved—is because of the way they’re presented: dull, predictable, and unengaging, tied up in a too-neat bow. They are not only easy to ignore; they’re actually hard to pay attention to.
But what if program evaluations were constructed as detective stories? There are lots of similarities, in fact, between a good mystery and the assessment of a particular program’s effectiveness. Maybe there are no dead bodies or court records in program evaluations, but there are lots and lots of questions we really need the answer to—lots of questions we should be just plain curious about. What happened, and why? Does everyone agree or are there differences of opinion? What were the motivations for decisions, and the impact of them? Inquiring minds want to know!
And, like true crime dramas, evaluations are stories necessarily built on incomplete facts and conflicting opinions that have to be poked, prodded, and sorted. An open-minded, curious evaluator has to take it in and make enough sense of all the information to know what conclusions she is confident about, and what remains unknown. She not only has to sort fact from fiction, but also figure out what the findings mean: Yes, the implementation was slower than expected, but how does that (or does it even) affect what was actually accomplished? Was the loss of three key staff members during year two a result of disastrous management or just bad luck? And then there’s the big one: even if all the outcome indicators are trending in the right direction, can we say with confidence that it’s due to the program itself?
I know it’s a stretch to imagine that evaluation reports, which are notoriously banal, could be as compelling as Serial. But surely when we’re designing evaluations we could try to tap into our natural human curiosity. We could ask ourselves the question, “What are we genuinely interested in knowing?” This is a question that might lead us in a novel direction—not the well-trod paths of bureaucratic exercises to tick the “evaluated” box or the careful marshaling of evidence to confirm what we already believe. Then, in the course of the evaluation, the evaluator could present the information not as a neat set of conclusions backed up by selected and sanitized facts, but as a puzzle for those who care to join in the solving of it.
In the end, I’d bet that the findings from an evaluation that engaging would be remembered for a long time, and might even have a chance of influencing decisions far into the future.
