The summer season means a steady stream of Hollywood blockbuster premieres. Many of these will be in the form of sequels—there are over 20 planned for 2014, including Spiderman 2, Transformers 4, and 22 Jump Street. Evidently, past success at the box office is indicative of future returns; so why not bet on the sure thing instead of taking risks on new ventures?
With grant-making too, past success is a big factor in deciding whether to continue investing in an organization or a project. But the metrics we have to measure success often aren’t as clear as box office sales. Evaluations can be a useful tool to assess how well things are going (or not), and how future investments should be directed. In Hollywood speak: should we greenlight the sequel, and if yes, what plot line should it follow?
Since 2005, the Hewlett Foundation has invested more than $25 million in the Population and Poverty Research Initiative (PopPov), a body of research on the relationship between population dynamics and micro- and macroeconomic outcomes. As part of the initiative, researchers looked at the effects of obstetric complications on long term economic and social well-being of women and their families in Burkina Faso, and estimated the quantitative impact of reductions in fertility on a country’s long-term output per capita, to take just two examples. The funding provided by the Hewlett Foundation stimulated another $10 million of support for the Initiative from several European research councils and the World Bank, and the sheer amount of research undertaken to date is impressive: PopPov has supported 56 doctoral fellows and 61 research projects, which have resulted in 260 papers. About 70 of these have been published in peer-reviewed journals, and another 30 are in pre-publication review.
Numbers alone can’t truly measure the impact of the initiative, of course. The primary objective of PopPov was to advance and re-energize the field of economic demography, which had lost some of its earlier salience in social science research and in international development policy debates. The initiative was premised on the idea that attracting top talent and supporting cutting-edge research on population and economic development would yield compelling results that could help policymakers, particularly those in sub-Saharan Africa, to make informed policy decisions. In late 2012, we decided to evaluate PopPov in order to draw some summative lessons from our investment in the initiative, and selected the RAND Corporation to conduct the evaluation through an RFP process.
The evaluation was designed to help guide decisions about the PopPov network, as well as future investments in other policy-relevant research. The evaluation focused on four questions: 1) To what extent did PopPov strengthen the field of economic demography? 2) What contribution has PopPov made to the evidence base? 3) To what extent did PopPov yield policy-relevant research? And 4) How did the design and implementation of PopPov affect outcomes? The final evaluation report, completed in 2014, is based on document review, key informant interviews with nearly 100 individuals, and an online survey of research grant recipients.
The report provides a comprehensive summary of the PopPov network, insightful observations about the four questions above, and valuable recommendations about the various aspects of PopPov. The Hewlett Foundation will likely pursue a more focused research agenda in the future. Based on the strategic objectives of the Global Development and Population Program, we plan to prioritize recommendations in the three areas below. The Hewlett Foundation Board approved a renewal grant to the Population Reference Bureau, the Secretariat for PopPov, at its meeting last week to help carry out these activities.
Maintain the “network”—strengthening the field of economic demography was a key objective of PopPov. The initiative has helped to increase the number of economists working on population and development issues and enabled researchers already in the field to do more than they otherwise would have. In particular, through the annual research conference, PopPov has also created a community of researchers with shared interests who value the opportunity for meaningful and productive exchange. In order to help sustain the PopPov network and maintain the field, the Foundation plans to continue supporting the research conference through 2015. We will also explore ways to strategically support activities to engage and connect researchers in the field, for example by providing small grants to young scholars or creating networking opportunities at such venues as the annual Population Association of America (PAA) conference.
Strive for greater policy impact—PopPov research was funded through open calls for research, and as a result the body of work includes a wide range of topics. With some exceptions, producing high quality research was prioritized over translating research into policy messages or presenting findings to policy audiences. In order to maximize policy impact of the research funded to date, the Hewlett Foundation intends to support a number of policy communication activities, such as policy messaging and writing workshops for researchers, creation and dissemination of policy briefs, and presentations to advocacy groups and other policy audiences. A couple of examples include providing travel support to researchers to speak at the World Economic Forum or other events, and organizing a policy discussions with high level decision-makers around the African Union meetings.
Build research and policy engagement capacity in sub-Saharan Africa—approximately two-thirds of PopPov research projects focus on sub-Saharan Africa, and many of these include African collaborators. Although capacity building was not an explicit aim of PopPov, there is an opportunity to further increase research capacity in sub-Saharan Africa. The Foundation plans to improve coordination with its demographic training portfolio and other research funding efforts, for example by inviting staff and students from African training and research institutions to PopPov conferences and workshops.
So was PopPov a success? I invite you to read the report to get the nuanced answer. In brief, PopPov has enabled researchers to do more fieldwork, increased the number of European researchers working on population and development issues, and brought a new generation of scholars into the field. It has made important contributions to the knowledge base on the economic impacts of fertility and stimulated renewed interest in research on the demographic dividend—accelerated economic growth that may result from changes in a country’s population age structure. Studies using natural experiments and randomized control trials have helped to make methodological advances, too.
So, will we film the PopPov sequel? Perhaps it’s less of a sequel than a spin-off. We will not pool resources for a large, wide-ranging research program, but instead build on the pieces that get us all closer to connecting research to policy. Instead of a feature length film, we are more likely to support shorter pieces that can be in tune with what the audience demands. PopPov on YouTube? Why not?