Evaluating Community-Driven Reconstruction: Lessons from post-conflict Liberia

Reconstruction in northern Liberia presented daunting challenges when the International Rescue Committee launched a community-driven reconstruction (CDR) program there in 2006. The area had been racked by civil war since 1989, when Charles Taylor’s rebel group, the NPFL, launched its attack from Cote d’Ivoire. The conflict was devastating, with 85 percent of the local population displaced at some point during the fighting and over four percent killed directly from war-related violence. The conflict was finally brought to an end with Taylor’s exile, a transition government, a UN peacekeeping mission and elections in 2005.

The CDR project, financed by the British government’s Department for International Development (DFID), sought to improve material well-being, reinforce democratic political attitudes, and increase social cohesion. CDR projects like this are being employed in post-conflict situations around the globe. They are a mechanism to disburse development funds rapidly and support the war-to-peace transition by strengthening local communities and rendering decision-making more transparent and accountable.

That’s the theory. In fact, although billions of dollars are disbursed on community driven development programs, the evidence that these programs are effective is quite thin. Indeed, many social scientists might contend that the idea that a short-term injection of funds coupled with an imported model for collective decision making would alter local community dynamics—as many aid organizations hope—appears fanciful. Community cohesion and decision-making practices reflect long histories and real distributions of power and wealth; it is plausible that they are not sensitive to short-term external interventions. It is also possible that short-term cash injections in a post-conflict setting could even exacerbate existing social divisions.

So which account is correct? Do CDR approaches actually deliver the many benefits claimed for them? To find out, our group partnered with the International Rescue Committee (the IRC) to undertake a rigorous evaluation of post-conflict community-driven development in Liberia.

The evaluation challenge

Attempts to ascertain whether CDR programs actually improve community cohesion and reinforce democratic practices face two major challenges. The first concerns how communities are usually selected to participate in CDR programs. If pre-existing differences between communities are related both to outcomes and selection into the program, we face a problem of “confounds” which makes evaluation extremely difficult. If, for example, communities are selected because they are especially easy to work with, then a simple comparison of outcomes between project and comparison communities would lead us to overestimate program effects; if, however, the neediest areas are selected, then a simple comparison could lead us to underestimate the impact.

From an evaluation perspective, the best way to address these problems is to select the areas for the program so that there is a control group that is similar to the treatment group in all respects except for the presence of the treatment. This is best achieved through a “randomization” process in which eligible communities are divided into treatment and control communities by a lottery. Then we can know with confidence that programs are not in operation in particular sites because attributes of those sites and individuals were not exposed to the program. While it is possible (and to some extent inevitable) that there are differences between program and control areas, there is no reason to expect there to be any systematic bias one way or another. This is the solution we adopted in Liberia, where communities were selected into the program though public lotteries.

The second major challenge concerns measurement. Democratic political attitudes and patterns of social cooperation are difficult to measure, much more so than outcomes such as educational attainment and health status. Moreover, one might reasonably believe that communities exposed to CDR would be more likely to provide survey responses that indicate support for democracy, less community tension, and greater collective efficacy even if there has been no underlying change in attitudes and behavior. If we observe positive effects of a CDR program, how can we be sure that these reflect real changes in attitudes and behavior, and not simply the fact that CDR treatment communities learned what to say and how to act in order to please outsiders?

In response to this concern, we gathered behavioral measures of political practices and social cooperation (to complement our standard household surveys) through a set of experimental games. Six months after the CDR program was completed, all treatment and control communities were given the opportunity to raise funds from a Liberian NGO to implement a public goods project. The only requirements were that the communities choose a project in advance and identify three individuals to handle the money. Communities were also told that the amount of money they would receive (ranging from $0 to about $500) would depend on the results of a village-wide public goods game, in which 24 randomly selected individuals could choose privately to retain a sum of money for their own use or contribute it to a community fund (with an additional contribution supplied by us) to be used for the public good. We then gathered data on how communities selected projects and representatives and observed patterns of play in the public goods game.

This behavioral approach has an important advantage over traditional methods for measuring social cooperation. Unlike with survey responses, there is a real cost in the games to taking an action motivated purely by a desire to please outsiders; instead the focus is directly on the question of interest, the willingness to make a private sacrifice for public gains.

What we found

The data collected through our surveys and behavioral games suggest the following key conclusions:

• The CDR program had a measurable, positive impact on the level of community cohesion. Communities exposed to the CDR program appear better able to act collectively after the program’s completion to improve their own welfare. The evidence for this in the surveys is mixed, but our behavioral measures provide strong evidence that CDR communities can significantly outperform control communities in raising funds to implement a community project.
• Exposure to CDR also appears to have increased social inclusion in beneficiary communities, especially for marginalized groups. The behavioral evidence on this is powerful, with greater knowledge and awareness of the community project and faith in the representatives and projects chosen. Traditionally marginalized groups made significantly higher contributions to the community project in CDR communities. Survey evidence also supports this conclusion, with individuals in CDR communities reporting less social tension and exhibiting greater acceptance of traditionally marginalized groups.
• The CDR program reinforced democratic values and practices. The evidence in the surveys provides some evidence for greater support for elections and participatory processes in treatment communities. The behavioral games demonstrate these values in practice, with a higher likelihood of democratic selection processes for community representatives and projects in communities that experienced CDR. Baseline support for these values and practices was already high, however, and there is no evidence that the program altered these values among our sample of town chiefs.
• There is little evidence of positive improvements in material well-being related to the CDR program. While the data do suggest an improvement in local public goods—consistent with the focus of communities on the construction of community facilities—this does not appear to have translated into household-level welfare improvement. The main positive finding on welfare is that access to education improves significantly. Evidence for gains in livelihoods and asset holdings are, however, weak.

The findings were quite surprising to us. The program had a demonstrable effect on more abstract and difficult-to-measure outcomes, such as social cohesion. It had much less impact on material wellbeing. These results went against our expectations, but we were conscious of the weak empirical base that informed those expectations. In fact, to the best of our knowledge, this is the first time a CDR project of this type has been evaluated with a randomized control trial (though other trials are ongoing). The evidence of positive program impact, and absence of evidence of any negative effects, is thus considerably stronger than in most previous evaluations done on community-driven reconstruction.

What we learned about evaluation

In the course of this work, we found that we learned as much about the challenges of doing evaluation – and the linkages between program design and evaluation – as we learned about the program itself. Three issues merit particular emphasis:

Program assumptions. Evaluations often focus on differences between treatment and control groups and not on the levels of the outcomes of interest. Yet, we found in many cases that the levels of outcomes such as cohesion, democratic attitudes, or social acceptance, were high both in treatment and control communities. These findings are inconsistent with some of the assumptions underscoring the CDR approach. Plausibly the really important divisions exist between communities rather than within them, but this was not the focus of the program. Having this information at the outset might have shaped the design of the program, but in standard practice, baseline measures are taken only after the program design has been completed. Our results suggest that these data may be even more useful if they are gathered earlier in the process.

Ethics of randomization. From the beginning, we were conscious of the ethics of using randomization; in particular, we worried that this strategy could give rise to jealousies between communities that did and did not receive the program, thereby exacerbating conflict. At the same time, we were aware of ethical arguments suggesting the equitability and transparency advantages that derive from the use of lotteries when insufficient resources are available for all needy communities. Because of these concerns, we monitored the lottery process. Reports from the field were encouraging: not only were there no conflicts resulting from the randomization, but communities actually viewed the process favorably and appreciated the equity of the procedure. Though randomization processes should continue to be monitored closely, our experience suggests that the potential for using lotteries is greater than many believe.

Assessing the magnitude of program impact. The randomized design and our behavioral measures allow us to conclude with some confidence that the IRC CDR program had a positive impact, but how big are these improvements and were they worth the cost? When an aid program seeks to reduce infant mortality or to improve material welfare, the outcomes in question usually have natural metrics that allow for cost-benefit comparisons with other types of aid projects. This is much less the case for aid projects that seek to improve governance or to promote post-conflict reconciliation and we faced real difficulties in assessing the magnitudes of the effects we find. With time we hope that it will be possible to develop more standardized governance and cohesion measures that allow comparisons across programs.

Looking forward

Exercises in post-conflict reconstruction and local institution-building are increasingly common across the developing world. But despite major investments, few efforts have been mounted to assess the efficacy of external interventions designed to alter local dynamics and patterns of social cooperation. Our work with the IRC and DFID in Liberia shows that rigorous program evaluation is possible, even in post-conflict contexts. We also learned that distributing program benefits through a public lottery not only facilitates evaluation but may also increase the satisfaction of potential recipient communities with the equity and transparency of the aid distribution process. Finally, we found that innovations in measurement can be used to demonstrate believable program effects on difficult-to-measure outcomes such as social cohesion and a commitment to democratic practices.

Strategies for post-conflict peace-building should be held up to the same evidentiary standards of efficacy now being applied to development efforts in health, education, and agriculture. But this requires changes to current practice.

• First, we need to focus on substantive outcomes, not just process. Most evaluation still focuses on how well programs are implemented and not on whether they have an effect. This needs to change.
• Second, there needs to be early coordination between practitioners and researchers. It was essential for our work that the evaluation process began before the program started; but for many programs, evaluation is treated as an afterthought and implemented when it is often already too late.
• Third, researchers and practitioners will need to further innovate in the measurement of outcomes such as social conflict, social cohesion, and political values and beliefs. While attitudinal surveys remain standard, our results suggest that they may fail to pick up underlying behavioral change. Actual behavioral measures offer greater promise, but the twin issues of how to ensure comparability across contexts and how these behavioral outcomes can be assessed in a cost-benefit framework are the critical challenges that will shape the next generation of CDR evaluations.

James Fearon is Professor of Political Science, Stanford University.

Macartan Humphreys is Associate Professor of Political Science, Columbia University.

Jeremy Weinstein is Assistant Professor of Political Science, Stanford University.

Download the PDF to view all images, graphs and additional information.

Although billions of dollars are disbursed on community driven development programs, the evidence that these programs are effective is quite thin.