3 Emerging Good Practice
- Caroline Kende-Robb
- Published Date:
- January 2002
This chapter identifies good practices that should be considered when undertaking participatory policy research for policy change. Emerging good practice builds on the diverse impacts of key variables discussed in the previous chapter. It is divided into three main areas in which issues are similar and linked: first, issues to be considered from an institutional perspective within the World Bank;1 second, good practice when managing a PPA in country, at the national level, including how to open up the dialogue in participatory policymaking; and third, emerging good practice in conducting participatory research with the poor at the community level, and the principles behind this method of data collection. There is no unconditional good practice in this type of work because the best approach will be determined by the context. However, box 8 gives some suggestions for good practice and minimum standards that have emerged from experience with the Bank’s PPAs. These issues are then discussed in more detail throughout the chapter.
At the World Bank: Initial Steps and Follow-Up
This section is divided into five main parts: (a) professional input and commitment, (b) ownership of the PPA in the Bank, (c) management support and follow-up, (d) PPA design, and (e) linking to the Bank’s country assistance strategies.
Professional Input and Commitment
The first step in initiating a PPA and poverty assessment is to ask, Will the outcome drive policy reform within the country and in the work of the Bank? Whether PPA and poverty assessment will move from an academic exercise to influencing policy depends on the extent to which the Bank and, more specifically, the sponsoring Country Department is committed to poverty reduction. Although there is no one approach to poverty reduction, and the definition of poverty is broadening (see box 9), PPAs have yet to be as generally accepted as traditional household surveys. However, the Bank is now moving toward undertaking ongoing poverty analysis and monitoring as opposed to one-time poverty assessments, and is recognizing the importance of including the poor in this analysis.
Box 8.Twelve Hallmarks of Good PPAs
- 1. The Bank’s role is to provide support (technical or financial) to the government to enable the government, not the Bank, to lead the process.
- 2. The PPA research agenda is discussed with country team members, leading to broader ownership and understanding of results within the Bank.
- 3. Results of the PPA are combined with household survey data by the government, with support from donors if necessary. Such analysis should then influence World Bank poverty assessments and the country assistance strategies.
- 4. Bank managers and staff observe the PPA being conducted in the community so as to better understand the strengths and weaknesses of the data.
- 5. Government leads the process, and support should be secured from the beginning. Key policymakers and administrators are included in designing, planning, and implementing the PPA and analyzing the results.
- 6. Timing and extent of involvement of other stakeholders (NGOs, line ministries, unions, religious groups, local social science institutes) are attuned to the social and political environment.
- 7. In-country capacity to conduct ongoing PPAs is strengthened so PPAs can feed into the policy dialogue.
- 8. PPAs can contribute to the development of Poverty Reduction Strategy Papers by including the poor in the consultation process; providing data for poverty analysis; and where the PPA is an ongoing process, building country capacity to monitor poverty.
- 9. Local research teams are trained to conduct high-quality participatory research, with an understanding of both the principles and techniques.
- 10. Communities are involved in analysis of the data.
- 11. Results of the PPA are disseminated to the communities involved in the policy research and to agencies that can follow up at the community level with action and projects.
- 12. Results are presented in a clear and concise manner
Experience has shown that the PPA manager needs to have a diverse set of skills, including technical methodological skills and skills in managing a participatory policy dialogue. Operating at a policy level and opening up the policy dialogue in country often mean that conflict will occur. Conflict is not always negative—from it, greater understanding of the problems of the diverse groups involved can evolve. An understanding of people and their motivations, as well as sensitivity, tact and diplomacy, is required when opening up a policy dialogue. This is never a smooth process: it is unpredictable, and no matter how skillful the PPA manager, the process might not go according to plan.
Who Owns the PPA in the Bank?
Ownership within the World Bank, across departments, emerges as a key issue when considering the impact of PPA exercises on World Bank policy and projects. For the PPAs in Pakistan, for example, there was limited ownership and understanding of the process. Consequently, the results were not reflected to a significant extent in other World Bank documents. In Cameroon, changes in the team managing the country program occurred while the poverty assessment was being prepared. Within the country department, the PPA results had limited credibility with those who were not part of the process. Additionally, keeping poverty issues on the agenda proved difficult when the CFA (Communauté Financière Africaine) was devalued as the emphasis shifted to macroeconomic issues.
Box 9.What Is Poverty Reduction?
The Bank is broadening its view of poverty reduction, as reflected in the comments below by economists and sociologists affiliated with the Bank.
- “Commitment to poverty reduction is dependent upon the government’s public expenditure priorities. An example may be the targeting of clean water for the poor, which would consequently improve their health and thus increase their productivity.”
- “[Poverty reduction is] increasing income and general assets to a level where the poor are less vulnerable to risks and falling below a certain level.”
- “Poverty reduction is giving people greater control and the means to determine their lives.”
- “Poverty reduction starts with the poor’s perceptions of their own poverty in a process of sharing strategies, priorities, and solutions of various stakeholders.”
To achieve greater policy relevance and broader ownership, a broader team approach is important. In Armenia, for example, the manager of the poverty assessment had in-depth country knowledge, built up respect among key policymakers and within the country’s academic community, and encouraged a team approach within the Bank. In addition, the PPA manager worked closely with those managing the household surveys and the country department’s macroeconomist to establish a research agenda for the PPA. As a result, the process had the following outcomes: first, the results of the PPA were reflected in the poverty assessment; second, the country program and the CAS integrated the results of the poverty assessment; and third, the poverty assessment was well received and used by policymakers in Armenia.
Management Support and Follow-Up
Limited management support and follow-up within the Bank have sometimes led to lost opportunities. In Madagascar, for example, there was a high degree of in-country support because key policymakers were included from the beginning. With changes in the Bank management, however, there was a delay in follow-up of more than a year and a half, and the commitment and interest of the government consequently weakened. In Equatorial Guinea, the information was controversial and the Bank was reluctant to continue the process.
Appropriate follow-up measures are sometimes difficult to identify because the outcomes of the PPA and poverty assessment consultations are not always reported accurately. In one country, many NGOs and high-ranking government officials openly opposed the results of the poverty assessment. In addition, many felt unhappy that their views were sought during the consultation but then not included in the final poverty assessment. Yet within the Bank this poverty assessment was considered technically sound and successful. A recommendation for good practice is to monitor not just the outcome of the policy dialogue or the poverty assessment but also the process and outcomes of the participation and consultation. For example, CASs, poverty assessments, and PPAs could document who was consulted and how, and the major lessons learned from consulting each of the key stakeholders.
Decentralizing the management of the PPA to resident missions might be appropriate in some countries because it is difficult to coordinate the PPA process from Washington. The manager of the Tanzania PPA, for example, suggested the need to strengthen that resident mission to enable it to undertake frequent PPAs and contribute to a broader poverty assessment. Teams could be located in the field, and people skilled in the analysis of poverty could be located within the mission. To increase the capacity of the resident mission, training in participatory policy research could be conducted and tool kits provided. Where appropriate, the NGO officers and social scientists (recently recruited in many resident missions) could assist in such poverty-focused work.
Some PPAs can be strengthened by the use of sampling methods. For example, the selection of PPA sites could be informed by traditional household survey data. In Kenya, the Welfare Monitoring Survey (WMS) was used to identify the poorest districts in each of six provinces.
Trust and understanding should be developed among those who use different approaches to defining research agendas and collecting and analyzing data with the aim of influencing policy The integration of data sets will evolve through this trust building. Both survey and participatory assessment practitioners need to understand the limitations of various data sets, appreciate the biases in their own research methods, and know when alternative methods can compensate for some of these limitations.
In an attempt to better understand the various approaches to poverty analysis, the local NGO research team (Participatory Assessment Group) in Zambia is currently undertaking participatory poverty monitoring exercises and combining the results with those of the household survey carried out by the Central Statistics Office. In other countries, policymakers have visited research teams in the communities. In Costa Rica, for example, a senior official from the Ministry of Economic Planning was involved with the research and consequently was better able to appreciate the value and limitations of the PPA. And in Armenia, where the manager of the poverty assessment built trust among those managing the household survey, the Bank’s PPA team, and government policymakers, the resulting integrated analysis of poverty was widely used both by the Bank and by government policymakers.
Another PPA design issue is the need to ensure that the results are shared with various stakeholders. Dissemination of results should be part of the PPA planning and budget, but in most PPAs this has not been the case. If the information gathered is not fed back to the communities, the participatory nature of the work is incomplete. There are several important reasons to feed back information from the PPA: to validate the information; to continue the process of constructing a dialogue with communities so that semipermanent linkages are created; to show respect for the partnership with the community by sharing the information; to continuously reevaluate the relationship of the PPA facilitator (for example, the Bank) with the various stakeholders, especially the poor; to increase the credibility of the information and thereby enhance the potential of the PPAs to influence policy formulation and delivery; to facilitate a process whereby the poor monitor and evaluate the impact of the PPA; and to encourage action at the community level.
In designing a dissemination strategy, the demands of the various stakeholders should be considered and key stakeholders should be involved. Where governments have not been involved, results have been mixed. In Cameroon, for example, the results of the PPA were perceived as threatening to the government Thus, the impact of the PPA was greatly reduced. To build a political base for policy change, effective use of the media as part of a communications strategy can help to increase understanding between the government and the public.
Different documents might be needed to meet various stakeholder demands. For example, those at the community level might be more interested in detailed site reports of their community, whereas line ministries might want a country-level document. To increase awareness and to disseminate the broad results and policy recommendations of the PPA, countries such as Zambia, Guatemala, and Lesotho have produced clear, well-written, short summary documents that have contributed to widespread ownership and understanding. Another suggestion is for the Bank to produce a separate document on the PPA results in addition to integrating these results into the overall poverty assessment. This could give the PPA managers more autonomy and accountability.
The design stage should include consideration of how the data will be presented. In Ghana, for example, the information from the PPA was relatively complex and extensive, making incorporation into other World Bank reports time-consuming. In other PPAs, it might be appropriate to use the visual diagrams from the PRA exercises (see appendix D) in the final report as a means of conveying information.
One reason that PPAs have not resulted in more action has been the lack of specificity in the presentation of results. Wherever possible, therefore, proposals should be presented in matrix form, detailing the following:
- Actions that could be taken immediately
- Actions that require policy change
- Cost requirements
- Whether a short or long time is required for results
- Administrative order or legislation required
- Which ministry, donor, or NGO could take responsibility for carrying out the action.
To follow up on such proposals, the PPA should include a monitoring component.
Link to Country Strategies
To better reach the poor, the results of PPAs and poverty assessments should be closely linked to the World Bank’s CASs. Their impact on CASs has been weak because of broad constraints on adopting participatory approaches in both projects and policy work. These constraints include the following:
- Accountability. In some areas, it is not always possible to assess the quality and extent of participation. Stakeholder analysis and a plan for including stakeholders in the evolving dialogue are not always clearly presented. Thus, it is difficult to track the process and compare the level and quality of actual participation with the level and quality of planned participation,
- Support. Some participatory activities are limited because of the lack of time and funding. In both project and policy work, it is sometimes difficult and time-consuming to obtain funding to include a wider cross-section of stakeholders. Trust funds (grants given by bilateral aid agencies) are available but can be difficult to access for policy work. Core Bank funding is often not available, and many governments are still unwilling to borrow money for such activities. Pressure to conform to ever-tightening deadlines often undermines broader participation and consequent ownership and commitment.
- Evidence. A few people in the Bank and some government officials still question the cost benefit of participation.
In 1996, the Operations Evaluation Department surveyed the managers of completed and ongoing poverty assessments (see World Bank 1996j). Only 46 percent of those who answered the question, “What influences the impact of poverty assessments?” believed that the poor should participate in the design and preparation of such assessments. A draft OED report, “Participation Process Review” (World Bank forthcoming), took note of a World Bank task manager survey on participation, conducted in May 1999, which revealed that the most significant constraints to participation in Bank operations were a lack of time and money, rigid project cycles, and inadequate incentives and management support. In this survey, 81 to 88 percent of task managers agreed or strongly agreed that primary stakeholders should participate extensively in Bank-supported work. For structural adjustment loans and sector adjustment loans, 68 percent agreed.
There is now a move to increase the poverty focus of operations by overcoming barriers such as strategic issues—weak links between the PPA and the poverty assessment, between the poverty assessment and the CAS, and between the CAS and operations; lending—emphasis on loans approved rather than on poverty reduction goals; and impact—focus on input and disbursement indicators rather than on laying the foundation for assessing impacts on the poor.
Another Bank report, “Taking Action for Poverty Reduction in Sub-Saharan Africa” (World Bank 1996i), notes that “poverty reduction is rarely a central or motivating theme for the business plan or country assistance strategies, although responsiveness on this issue has recently improved” (p. 15). The report contends that CASs are too general to address poverty and that much of the poverty focus of projects is lost by the time the lending program is implemented. The report further states that CASs usually do not make poverty reduction a core objective of economic development programs, that poverty reduction is incidental to macroeconomic stability or lending, and that the link between the reform agenda and poverty reduction usually is not made. The report adds that past CASs have lacked a “strong strategic vision on poverty reduction and clear monitorable actions for reducing poverty” and argues that
this shortcoming at the operation level is often rooted in: (a) a lack of information on poverty, (b) inadequate analysis, (c) a disinterested attitude toward poverty reduction, and (d) Bank management’s willingness to compromise on poverty reduction to maintain good country relations and to be satisfied with lending operations that address aggregate growth with little attention to the distribution of growth …. Operational decisions, therefore, tend to be based more on sector interests than on poverty reduction [whereas poverty is] a multisector issue requiring an integrated strategy (p. 20).
The report calls for the Bank to revamp its strategy to include responsiveness to the needs of the poor, which in turn, requires a better understanding of poverty—precisely what the PPA can deliver, in conjunction with household surveys.
There is now a demand for better poverty analysis to help both the Bank and governments focus their projects and policies more effectively. To achieve this, PPAs should aim to become a building block and not just an adjunct to CASs and policy framework papers. Experience from past PPAs shows that this linkage is greater where the research agendas for the PPA and the poverty assessment have been developed with those working in country departments and on CASs. This cooperation can be time-consuming and requires more preparation, but the payoff is a greater impact on the CAS. Another report, “Poverty Reduction and the World Bank” (1997c), details how many of the CASs have become distinctly more poverty focused, particularly those for Sub-Saharan Africa, ever since the May 1996 directive from senior management to put poverty reduction at the center of the country assistance strategies. Other recent developments include more poverty-focused guidelines for CASs and the rewriting of the Operational Directive on poverty. The reports note that all CASs that are written a year or two after a poverty assessment incorporate the main findings of the assessment, although some do so more comprehensively than others.
Since the introduction of the PRSP in 1999, the nature of the CAS has been gradually changing. The CAS will become more like a “business plan” linked to the country’s poverty reduction strategy (see figure 8 in chapter 4).
A summary of this section is provided in box 10.
At the Country Level: Linking PPAs to the Process of Policymaking
Using the PPA examples, this section looks at the major issues to be considered when working with institutions in country. It is divided into the following parts: starting point—understanding the political environment, creating a conducive policy environment, who controls the research agenda and outcome, and strengthening the policy delivery framework.
Starting Point—Understanding the Political Environment
Participatory policymaking involves linking information from communities into a broader policy dialogue that includes a cross-section of stakeholders. In moving from community-level research results to policy analysis, issues surrounding policy change should be considered. For example, policy formulation is an inherently political process. Rules, legislation, traditions, networks, ethnic alliances, patronage, political allegiances, and bureaucratic structures all interact to form a complex and fluctuating policy environment. Key questions, therefore, include what factors affect policymakers’ decisions to create, sustain, alter, or reverse polices; what are the legal complexities of policy change; and what influence does individual survival in an institution, institutional survival in a government, and the maintenance of a regime within a country have on policy choice?
Box 10.Factors for the World Bank to Consider to Increase the Impact of PPAs
Professional input and commitment
- Promote poverty reduction as a clear commitment. The extent to which country departments and country directors at the Bank are committed to poverty reduction will affect the impact of the PPA and poverty assessment. Where this commitment is not clear, operations will tend to be biased toward sector interests rather than poverty reduction.
- Measure performance of country directors by the poverty focus of the country assistance strategy (CAS), pipeline projects, and adjustment policies.
- Develop skills to conduct poverty assessments and PPAs. Challenge individual behavior, approaches, and motivations.
- Observe participatory research in communities to understand the strengths and weaknesses of PPAs.
- Develop trust and understanding between those who managed data collection for the various approaches (surveys and participatory research) and those who are doing the poverty analysis.
- Promote a team approach within the Bank and include different disciplines to enhance the understanding of the various dimensions of poverty.
Ownership in the Bank
- Establish broad ownership within the Bank for greater policy relevance.
- Create the research agenda for both the poverty assessment and PPA with others working in country departments.
Management support and follow-up
- Increase the capacity of resident missions for poverty analysis.
- Support governments to undertake continuous participatory poverty monitoring, as in Zambia (see box 2), to build up time sequence data.
- Monitor not just the outcome of the policy dialogue (the PPA, poverty assessment, and CAS) but also the process of participation and consultation. Also monitor the follow-up of the PPA and poverty assessment recommendations.
- Build government capacity to link participatory research with household surveys (as in Vietnam). Support the building of an iterative process whereby traditional surveys and participatory research inform each other on an ongoing basis.
- Help design PPAs that include dissemination strategies.
- Build government capacity to produce clear, well-laid-out reports and different reports for different audiences. Detail the process of consultation in each report
- Assist the government in the use of the media to promote communication with the public and increase the political base for policy change.
Link the PPA and poverty assessment to the CAS
- Ensure that PPAs and poverty assessments are building blocks for the CAS.
- Work to ensure that the poverty assessment drives policy reform, both in country and in the work of the Bank.
- Identify, in the CAS, clear, monitorable actions for reducing poverty that link to the poverty reduction strategy papers.
- Build on existing social knowledge in the country.
A further complexity of the policymaking process is the relationship between policy formulation and implementation. Policymaking and implementation are not disconnected but are part of ongoing interrelated processes of change (Grindle and Thomas 1991). But while some policymakers might be willing to incorporate certain issues in the policy agenda as statements of intent, they might be less willing to implement the resulting policies because of the political dimensions of implementation (see Wildavsky 1979; Moser 1993; and Wuyts, Mackintosh, and Hewitt 1992).
It is within this dynamic that the World Bank is trying to influence policy and therefore needs to understand the often hidden influences on policy decisions, including the many institutional, formal, personal, and informal networks that can either help or hinder implementation.
For example, in some of the countries where PPAs have been undertaken, poverty has not been high on the political agenda. Limited political support, or a lack of trust between the government and the World Bank, has led to a lack of support in country for some PPAs. In Cameroon, there was a perceived lack of support from the central government, in part because some key policymakers felt excluded from the PPA dialogue. Although the fieldwork was considered to be good quality and the results relevant, the government was not willing to embrace the findings of the PPA or to initially include in the political agenda controversial issues emerging from the PPA.
In general, open political environments provide greater opportunities for building consensus in regard to poverty issues. For example, in Costa Rica, where there is a tradition of bringing marginal groups into the political sphere, the government was eager to better understand poverty from the perspective of the poor and welcomed the PPA. Similarly, in Argentina, the government requested assistance from the World Bank to undertake participatory research. As a result, a strong level of commitment and coordination existed between the Bank and the government in the preparation of the poverty assessment and the PPA. In contrast, in Mali, because of the sensitivity of the poverty issue, the PPA had to be renamed the Living Conditions Survey and open dialogue on poverty was constrained.
In countries where poverty is highly sensitive, however, not all policymakers will be opponents. Individuals respond to a great many factors, including bureaucratic structures, political stability and support, technical advice, and international actors (see Grindle and Thomas 1991). Some might support the PPA if they perceive it to be for the good of their society, since not all policymakers are just rent seekers. It is good practice to identify and include those who support the idea of the PPA at the beginning of the dialogue and gradually build up broad-based support. Such good practice requires that Bank teams have an in-depth country knowledge of policymakers and that they develop relationships with and understanding of the key players.
The experience with PPAs is showing us that merely presenting to policymakers the results of new information generated through PPAs does not guarantee policy change. As a result, more recent PPAs have also focused on the policymaking process and the political context of policy choice and policy change.
Creating an Environment Conducive to Poverty Dialogue
Without government support, or even with limited support, the impact of the PPA is lessened. Because the ultimate objective is to influence policy rather than just produce technically sound documents, the value of conducting a PPA with little government support should be questioned. With limited support, a key issue will be what happens when the research results run counter to the government’s interest. Thus, dialogue is needed to build trust and understanding between the Bank and the government before the PPA is undertaken. Generating a more open climate can help ensure that the government is less threatened by the PPA results and that the PPA thus will have greater impact.
The participatory process will vary greatly from country to country, and the inclusion of different stakeholders within the PPA and poverty assessment should be attuned to the country’s overall political, social, economic, and institutional environment. In this kind of highly context-specific work, it is not possible to provide a blueprint; personal judgment is required. In some countries it might be appropriate to include a cross-section of stakeholders rather than targeting only a few policymakers. In South Africa, for example, the unexpected closure of the South African Reconstruction and Development Office meant that the initial strategy of focusing on one particular department was rendered inadequate (see May and Attwood 1996).
Maintaining a receptive attitude is not easy in a dynamic environment, where unexpected conflict often occurs and agendas and people change. Continuous follow-up and dialogue with various stakeholders are therefore recommended. This approach requires a shift from top-down prescription to a more flexible process approach, with local dialogue being maintained in country. The challenge for many PPAs, and for the Bank’s wider country programs, is to maintain the new partnerships created through such dialogue.
Who Owns and Controls the Research Agenda and Outcome of the PPA?
At the national level, ownership and commitment of stakeholders have varied among the PPAs. The Bank’s experience has shown that the involvement of key policymakers from the beginning enhances ownership and commitment. Where appropriate, the following measures can help to increase policy impact:
- Involve policymakers in the early planning of the PPA
- Bring key policymakers into the field to participate in the PPA
- When sharing a report with government policymakers, include local communities who contributed their analysis
- After the results are presented, convene workshops with policymakers and local people
- Negotiate high-level commitment to follow up the PPA and monitor the implementation of key recommendations.
In Argentina and Zambia, key government officials were included from the beginning and often led the process. As NGOs and other stakeholders were gradually included, the room for dialogue on poverty increased. This approach led to greater understanding and trust between the government and the NGOs. In South Africa, stakeholder involvement from the beginning was a time-consuming but important step in a complex process of dialogue, with a high level of ownership and commitment evident. In contrast, in Togo and Cameroon, key policymakers were not included early in the process and, therefore, the PPA’s impact has been limited. Similarly, in Lesotho, the government was initially not included and there was limited ownership. Local ownership was created only when the action plan was formulated by the government with a cross-section of stakeholders.
In regard to control, Owen (1996), in his analysis of the PPA in Mozambique, discusses the difficulty of satisfying the demands of multiple stakeholders. He asks, “Whose PPA is this?” Diverse and sometimes conflicting demands have the potential to undermine the participatory nature of the PPA, with the institutions that control the process wanting to produce documents according to predetermined deadlines and documents that represent their point of view. Owen further points out that where control has been relinquished there may be a tradeoff between ownership and quality. Box 11 discusses the complexity of achieving ownership even where a participatory process has been adopted.
Control and ownership of the PPA are also linked with the government’s ability to negotiate with the World Bank. Generally, if donors adopt a top-down approach to assisting in policy formulation, there will be limited ownership and commitment on the part of the government. Several government officials in Guatemala felt excluded from the PPA process, and relations between the Bank and the university that undertook the PPA were weak and antagonistic. Ownership of and commitment to the PPA results were, therefore, limited until the university published an independent document on poverty in the country, without any World Bank input.
Although the information from PPAs might be relevant and result in changes to policy documents, without ownership there will be no long-term shifts in attitude. It is recommended that for greater ownership, the research agenda should not be determined solely in Washington. Those who influence policy in country should be part of the discussion. This process might take much longer than anticipated, so the PPA design should be flexible to accommodate unexpected delays. Delays become more likely as more stakeholders become involved, and it is not always possible to predict how or even if consensus will be achieved (see box 12).
Strengthening the Policy Delivery Framework
Policy change is not just about writing a new policy document—it is also about implementing that policy. To link policy formulation to implementation, good practice is to focus on the following:
- Increasing in-country capacity for ongoing research
- Creating channels for ongoing dialogue among a cross-section of stakeholders
- Opening up a process of continual negotiation on the political agenda, in which the views of the poor are taken into account
- Maintaining partnerships.
Box 11.Handing Over the Document Does Not Equal Ownership
Zambia: There was extensive dialogue with a cross-section of stakeholders in the Zambian poverty assessment and PPA and, as a consequence, there was a strong and widely shared feeling of ownership of the process and the action plan. The Zambians drafted the recommendations section of the poverty assessment. However, in discussions with the local research team in Zambia,a one government official asked about the PPA:
What is there on this document’s cover to show that it is owned by the government? There is no coat of arms or government logo, no preface by any government official.
An NGO representative added,
The World Bank calls a national workshop at Mulungushi International Conference Center, introduces the poverty assessment, and hands over the ownership of the poverty assessment to the Permanent Secretary chairing the workshop. Just like that and the Bank thinks it has resolved the ownership issue.
It had been clearly stated and widely understood from the beginning that the poverty assessment was a Bank document. Although one objective is government ownership, it might not be appropriate to expect some governments to feel ownership of documents that were initiated in Washington and carry the World Bank logo. Some governments might not even want ownership, but might want the document to remain identified as a Bank document in order to promote an independent assessment. However, in other cases it might be appropriate for the Bank and the government to publish a joint document.
South Africa: The PPA included key policymakers from the beginning and ownership gradually developed among high level stakeholders. For example, the cabinet met twice to discuss the PPA. The first meeting took two hours and was chaired by Thabo Mbeki, the Deputy President of South Africa.a For this study, a local research team was contracted to review the process and impact of the PPA, For a full report, see Mutesa and Muyakwa (1997).
Box 12.Participation Is More Than Holding Workshops
Pakistan: The poverty assessment was the first economic-sector work in Pakistan to be widely disseminated and discussed. The workshops were followed by many positive press reports and increased awareness of poverty issues. The process helped encourage the government to form a group specifically to look at poverty issues.
There was a general feeling that the poverty assessment was a good analysis but that it was too narrow because it used only the consumption measurement of poverty. How to measure poverty was the subject of extensive debate. Government officials and NGOs felt that the main message from the assessment was that poverty in Pakistan had declined. This was disputed by some Pakistani economists, who stated that different measurements would produce different results, and by NGOs that had extensive countrywide experience.
Stakeholder views had been expressed in various workshops for the poverty assessment but many felt these views had not been adequately reflected in the final document. As a consequence, some commented that the assessment was the Bank’s “justification for structural adjustment” and challenged its objectives. One senior government official had attended many workshops but felt that his extensive participation during the workshops and written comments had not been considered. The question was raised about which institution controlled the research agenda and outcome of the poverty assessment.
The main message from this experience is that participating in workshops is not the end of a process of participation. A final consensus might not be feasible, so differing views should be reflected in the final document. Furthermore, if people’s views are not included, that should be explained. A recommendation is that PPA and poverty assessment managers should know how to organize workshops and do appropriate follow-up, including incorporating the views of all participants in the research results where possible, or at least the main themes emerging from the research. The quality and follow-up of workshops will affect both the impact of the PPA and the relationship among participating stakeholders.
In most countries, it will be important to build a constituency for reform beyond the government because societies are becoming increasingly pluralistic and change often depends on a variety of partnerships. The role of other international donors, which have the power to influence national policy, should also be considered. The United Nations Development Programme (UNDP) is currently undertaking poverty analysis in some countries using participatory methods. In Togo, the UNDP was a partner in the PPA exercise, and its resident mission continues to promote participatory analysis. In Ecuador, UNICEF used PPA methodologies to evaluate the impact of its program.
Some PPAs have been carried out in partnership with institutions that specialize in social research (universities, networks of social scientists, etc.). Such partnerships help to increase the capacity of such institutions while avoiding the duplication of research and helping to ensure that PPAs become part of the body of social knowledge.
The process of policy implementation often alters intended policies. It is, therefore, important to understand the linkages between intention (policy) and outcome (implementation), and identify and include those who will implement policy in the policy dialogue. Administrators at the central and local levels must be included in the PPA. To increase understanding of the various research approaches, it is also crucial to include statisticians from line ministries. For example, in Kenya, the Central Bureau of Statistics assisted in coordinating the PPA.
Because governments and donors have traditionally focused on sectors as opposed to cross-cutting themes, it might be difficult to place participatory research results within one institution. A recommendation is to identify an institution in country where such data could be analyzed, coordinated, and disseminated. Many countries have collected great quantities of participatory data but lack follow-up and coordination. Finding an entry point for participatory research results might encourage more continuous research by a cross-section of institutions, thus contributing to broadening the policy dialogue and eventually to an increased government and Bank commitment to poverty alleviation.
See box 13 for a summary of this section.
At the Community Level: Including the Poor
This section analyzes how to undertake participatory research at the community level, focusing on good practice to achieve credibility and legitimacy of the PPA. The section is divided as follows: research teams, management of research teams, and research process.
The composition of the research team working at the community level is usually context specific. In general, men and women should be equally represented, and familiarity with local culture, especially a knowledge of local languages, is essential. In Zambia, for example, the research team comprised one manager (male), and five male and four female facilitators of mixed ages and ethnicity. This team then split into mixed gender groups of three to four researchers and spent two to three days in each community. In Tanzania, 35 researchers split into teams of five or six and worked in six different provinces.
Box 13.Factors to Consider at the National Level to Increase the Impact of PPAs
Understand the political environment
- Secure support from the beginning; government leads the process.
- Undertake the PPA only after potential political implications have been thought through.
- Use the institutional, formal, personal, and informal structures and networks, and understand the impact they have on policymakers. This requires Bank teams to have an in-depth knowledge of the country.
Create a conducive policy environment, if possible
- Question the value of conducting a PPA for which there is limited government support.
- Build dialogue to create a more open climate, so that governments feel less threatened by the resulting data.
- Maintain a policy dialogue through continuous follow-up with various stakeholders.
- Attune stakeholder involvement to the overall political, social, economic, and institutional environment in country. There is no blueprint approach to the timing of stakeholder inclusion in the policy dialogue.
- Include key policymakers from the beginning. Develop relationships with and understanding of the key players.
- Include key policymakers and administrators in designing, planning, and implementing the PPA and analyzing the results.
- Consider publishing PPA results as a government document where possible. Data should be government owned.
- Know how to organize workshops with appropriate follow-up. Workshops are not the end of a process of participation. Final consensus might not be achieved, so the documents should reflect the differing views. If people’s views are not included, that should be explained. The quality and follow-up of workshops will affect the impact of the PPA and the relationship among participating stakeholders.
- Use PPAs to contribute to the development of Poverty Reduction Strategy Papers by including the poor in the consultation process, providing data for poverty analysis, and where the PPA is an ongoing process, building country capacity to monitor poverty.
Strengthen the policy delivery framework
- Identify a credible institution in which participatory research could be analyzed, coordinated, and disseminated. Investigate provincial capacities.
- Work with institutions (universities, networks of social scientists, and the like) already undertaking social research to ensure that research is not duplicated and the PPA becomes part of the body of social knowledge.
Teams should be well prepared before going to research sites. PPA experience has shown that even where teams are experienced in participatory methods, at least two weeks of training are required to discuss the complexities of undertaking national-level policy analysis; match participatory tools with the research agenda; decide on methods of recording and reporting; create an initial framework for analysis of results; build up a team spirit; and discuss attitudes and behavior. Compromising on training time leads to poor-quality research. Teams should also be aware of major policies linked to the research agenda before going to communities.
The skill and role of facilitators become increasingly important to achieving credibility when participatory exercises are extended from the project level to the national level for large PPAs. The speed of scaling up, often to fit with donor agendas, has often led to compromises on the quality of research. If the facilitation of participatory methods is poor, data could be biased, vulnerable groups excluded, and outcomes inaccurately analyzed. This bad practice has hurt the credibility of participatory methods. Good-quality work requires a combination of factors, including a good attitude, technical skills, and experience on the part of the facilitator.
In Mexico, it was difficult to find a suitable national consultant who was not politically affiliated to coordinate the PPA. In addition, controlling the process of gathering information proved problematic because the teams attempted to follow their own agenda. In Togo, the teams in the field had limited skills to analyze the results. In Mozambique, in an internal evaluation of the preliminary research phase, it was concluded that teams were too unfamiliar with the communities to develop trust, and some were not able to apply the methods effectively.
The major question now emerging is how to integrate the diverse data sets into a comprehensive analysis of poverty. Some have also argued that integration could be relevant at the data collection stage (see Chung 2000; Ravallion 1996). Integrating quantitative and qualitative research using the same teams has implications for the types of skills required by research teams. Whereas questionnaire surveys require enumerators, participatory research requires facilitators who have a completely different set of skills, behaviors, and attitudes. Therefore, although it might not be feasible to expect a team of enumerators to conduct credible participatory research, different teams could be used for different research techniques (for example, PPAs in India [Uttar Pradesh and Bihar] and Kenya).
Management of Research Teams
A key issue for good-quality participatory research that is emerging from this study is how to effectively manage research teams. Two major concerns require further investigation:
Diverse team structure
Most PPA research teams have been selected to represent the major groups in society. In Tajikistan, where participatory research was undertaken to support a World Bank poverty alleviation program, team members were selected to reflect the composition of Tajik society. The team consisted of men and women of all ages (college students, middle-aged people, elders) and education levels (from village schoolteachers to doctors and academics), from rural and urban areas, and from all major ethnic groups. The objective was to design a team that was not biased toward any one subgroup, especially the more educated urban elite.
During debriefing sessions and informal discussions with fieldworkers, the research manager was able to gather a great deal of information as long as she did not show preferential treatment toward any group. This meant breaking some social rules in Tajikistan by making room for the less-educated rural woman to voice her opinion. However, it also meant creating opportunities for the elder male to represent the team in meetings with local officials. The manager stated, “While on the whole this choice had positive results for the team, participation practitioners need to be aware that this minisociety is not necessarily easy to manage.”2
Certain members of the team tried to control the discussions based on their societal role. In Tajikistan, social hierarchies are designed along education, age, and gender lines. There is also a hierarchy among regional ethnic groups and among castes within some groups. The manager noted that although she was able to supervise and effectively manage the debriefing sessions, the dominant people were able to take over the report writing, which was done in separate groups.
Psychological toll of poverty research
Another challenge is managing the psychological toll of poverty research. PPAs, which are based on the premise of seeing poverty from the point of view of the poor, might expose field workers to some degree of trauma for which they are not prepared. In Tajikistan, although the fieldworkers had been involved in surveys and other studies, most of them lived in the capital and had little information about the depth of poverty in the regions. The manager stated that as fieldwork progressed into its second week, some fieldworkers broke down as they described their day’s work. In Equatorial Guinea, as well, the poverty was more severe than expected and in this case also, fieldworkers broke down during debriefing sessions. The outcome is often that fieldworkers feel depleted emotionally and physically, which could affect the quality of their analysis.
Selection of an institution
Identifying an appropriate institution to undertake the research can be difficult. Local knowledge of credible, neutral institutions is required. In general, PPAs have been more successful when the selected institution has some existing capacity to undertake participatory research; for example, a research institute, NGO, or social science network. However, some organizations claim to have experience in participatory research but do not have the capacity to undertake good-quality research, thereby compromising the credibility of the PPA.
To increase credibility, it might be appropriate to use an existing NGO network, where there is often a wealth of knowledge and skills. The advantages of using these networks, as opposed to training new teams, are as follows:
- Many NGOs have already established trust with communities and undertaken participatory research.
- The results could be followed up by the NGOs working in the communities, thereby ensuring that the research is not purely extractive. The limitation here is that the research results would be biased toward communities where the NGO has already had some impact, and the poorest communities might not be included.
- The PPA research could help to strengthen the capacity of existing NGO networks.
- Information could be collected by NGOs over time, and links established between the NGOs, policymakers, and statistical departments.
It should be noted, however, that few NGOs have the skills and capacity to undertake good-quality research on a large scale and that some NGOs may have sector biases.
The research process in some PPAs has been viewed as exploitative because it takes the community’s time, raises expectations, and undermines self-reliance. Facilitators should, therefore, clearly state the objective of their visit. An example of bad practice is producing community wish lists instead of analyzing the community’s needs. Furthermore, if the agency then funds the priority identified on the wish list without community participation and capacity building, dependence on the outside organization increases, community self-reliance is undermined, and false expectations are raised.
PPA researchers in Pakistan, Mozambique, and Zambia reported that some communities expressed hostility toward the research teams, especially where there was extensive research with limited follow-up. In Armenia and Moldova, communities expressed frustration and anxiety over being involved with many research exercises with no improvement in their situation. In these countries, the fieldworkers also reacted with frustration and some accused the participants of complaining rather than doing and being stuck in old ways. The manager of these assessments suggested that the fieldworkers were reflecting the frustration of the participants.
Time spent in communities
In many communities, it is easier and quicker to interact with the local elite, thereby missing the poorest (who are often less articulate, overworked, and unable to attend meetings) and women (who do not often leave their homes and are used to being excluded). To overcome this limitation, facilitators need to be aware of the power relations in the community and the composition of the community as a whole. Some PPAs have rushed the research process in order to meet deadlines, often leaving out the poorest and those on the periphery.
The difficulty of undertaking participatory research in urban areas has been an issue in many PPAs (see Norton 1994) where more time and flexibility are required than in rural areas. For example, Moser and Holland (1996) highlight the issue in Jamaica of confidentiality in wealth ranking and fear of being identified as part of the research because of safety. In the urban areas in Zambia, it was difficult to identify social groups, and people were occupied and not willing to participate. In other urban areas there might be a question of safety for the research teams, especially for women researchers, as was the case in Costa Rica and Zambia.
There is a widely held belief that for participatory research to be more accurate, the tools and techniques should be standardized. However, flexibility can be strength, for the approach, tools, and techniques will vary depending on the community. But in some circumstances it is possible to use certain standardized participatory methods on a wide scale to generate numeric information. Beneficiary assessments have quantified results based on a sampling frame, as in Costa Rica and Madagascar. The UNDP’s PPA in Bangladesh used standardized methods for focus discussion groups and the identification of priorities (see UNDP 1996). The utilization survey conducted by Action Aid in Syndhupalchowk, Nepal, used participatory mapping in more than 130 villages to generate service utilization data.3
In some household questionnaire surveys, questions are preset by outsiders and the respondent is not likely to know the interviewer. PPAs that use the participatory rural appraisal tools (visuals and group analysis) typically elicit more accurate responses when
- Institutions conducting the research are known and trusted by the communities
- Group dialogue and analysis encourage people to challenge inaccurate responses
- Data are triangulated (checked with informants and data sources) to test for accuracy and to find areas that need probing
- Researchers and local people learn from the process
- Marginal groups are targeted
- Data are analyzed by the community.
Skilled facilitators are needed to conduct this type of participatory research. Where skills have been lacking, the accuracy of PPA data has suffered.
During the research process, teams can learn from each other in regular meetings where tools and approaches are reviewed and differences between various social groups discussed. Site reports could be compiled as a result of the meetings and later disseminated to communities. Local officials should be included where appropriate and results of the participatory research at the community level shared with them. See box 14 for a summary of this section.
Analysis and Synthesis: Combining PPAs with Household Survey Data
In the past, poverty analysis was dominated by quantitative data derived from nationally representative household surveys. Since the beginning of the 1990s, participatory research has been increasingly used to define poverty and influence policy. The need to combine participatory research with household survey data is now more accepted, as illustrated in recent literature. However, at the operational level, household survey data are used more extensively in poverty analysis and are still seen to be more credible than data from PPAs. This section discusses some of the strengths and weaknesses of the most important surveys now used in poverty analysis, focusing mainly on household surveys and PPAs; and highlights some of the tensions that have arisen from attempting to combine these data sets.
Box 14.Factors to Consider at the Community Level to Increase the Impact of PPAs
- Develop trust between the research teams and communities.
- Be aware of bad practice in participatory rural appraisals (PRAs). Facilitators need experience, skills in applying the tools, and the ability to hand over control.
- Training of teams takes at least two weeks to discuss the complexities of undertaking national-level policy analysis; match participatory tools with the research agenda; decide on methods of recording and reporting; create an initial framework for analysis of results; build up a team spirit; and discuss attitudes and behavior. Experience has shown that compromising on training time leads to poor-quality research.
- Be aware of major policies linked to the research agenda before going to communities.
Management of research teams
- Be aware of the difficulties in managing diverse research teams that often represent different ages, genders, and ethnic groups.
- Be aware that research teams working with poor communities may experience some degree of trauma for which they are not prepared. Managers should understand that this outcome is possible. This is an emerging issue, and more training is required for both field researchers and managers to find ways in which such outcomes can be better managed.
- Share information with communities on an ongoing basis.
- Do not undermine community self-reliance.
- Be aware of respondent fatigue and raising expectations. Many communities—especially those accessible from major cities—are the subject of excessive research.
- Review existing data and material pertaining to the area before initiating any study.
- Identify credible, not just experienced, institutions to undertake research. Use existing NGO networks where appropriate to promote follow-up.
- Allow for more flexibility in urban than in rural areas.
- Link results of PPA with other institutions for follow-up.
- Write clear site reports to disseminate to communities.
- Recognize the limitations of the PPA. Participatory poverty research is not a methodology for empowerment.
- Adapt the methodologies to the research agenda.
- Use PRA for greater community-level analysis and ownership. Be aware of the dangers of rapidly scaling up PRA methods, which can undermine the quality of the research.
- Avoid biases—triangulate data.
- Quantify and record the number of people involved in the participatory research.
Analysis and synthesis
- Understand the difficulties of drawing macro conclusions from micro analysis.
- Present clear policy messages—do not present everything
Background to Household Surveys
World Bank poverty assessments have used different types of household surveys since underpinning all poverty statistical analysis is a need for a wide variety of data. Table 8 provides a summary of the main household survey types.
|Multitopic surveys (for example, LSMS and Priority Survey)||Measurement and analysis of different poverty dimensions, their interrelationships and correlates||Measurement and analysis of different poverty dimensions, their interrelationships and correlates|
|Demographic and health surveys||Health-poverty measurement, health behavior analyses, basic poverty diagnostics||Measurement of other dimensions of poverty limited, diagnostics limited|
|Employment surveys||Analysis of employment patterns, wage income analysis (linked to education)||Limited use for poverty measurement and diagnostics|
|Single-topic surveys||Income-poverty measurement (or another single dimension)||Limited diagnostics possible|
|Rapid monitoring surveys and service satisfaction surveys (for example, Core Welfare Indicators Questionnaire)||Quick and cost-effective monitoring of key welfare indicators, often with a focus on measuring beneficiaries’ access to, use of, and satisfaction with services||Income-poverty measurement not possible, limited diagnostics|
Typically, a national household income or expenditure survey, or a multipurpose Living Standards Measurement Study (LSMS), is undertaken to provide basic information on the patterns of poverty. A number of countries have based their poverty analysis almost exclusively on national income and expenditure surveys. It is now widely recognized that this approach is one-dimensional, and where possible, these data are supplemented with other data from such sources as demographic and health surveys. In order to have a single survey cover a range of topics, multitopic surveys were developed. In Africa, a common form of multitopic survey has been the Priority Survey—a single-visit survey that includes household consumption estimates.
The most comprehensive and ambitious multitopic survey is the LSMS. Many World Bank poverty assessments use data collected through the LSMS. Data are collected by an enumerator who typically makes two household visits, each usually lasting three to four hours. The Priority Survey is similar to the LSMS but has a shorter questionnaire and usually covers a larger sample of households—8,000 as compared to the 2,000 to 5,000 covered by the LSMS (Carvalho and White 1997). The first LSMS in a country can take between 18 and 36 months and costs between US$500,000 and US$1 million.
More recently, the World Bank has been developing the Core Welfare Indicators Questionnaire (CWIQ).4 The CWIQ is a household survey that uses structured questionnaires and probability-based samples. It draws extensively from market research methodologies. It is used mainly to monitor development objectives through the use of leading indicators such as beneficiaries’ access to, use of, and satisfaction with services. The CWIQ is based on large samples (in Ghana, the sample was 15,000 households), short questionnaires, easy data collection, quick data entry and validation, simple reporting, and fixed core and flexible modules. The CWIQ, by virtue of its streamlined format, can yield results more quickly than other household surveys.
The World Bank is planning to pilot, in Tanzania, a Community Service Delivery Survey, which combines a simple household survey based on the CWIQ model with a participative needs assessment survey. The main objective will be to monitor the delivery of local government services. The survey will be facilitated by local government and filled in by communities. The idea will be to aggregate the results for each district and then present them to communities so that they are able to compare their village’s services with those of others in their district.
Key Differences and Similarities of Household Surveys and PPAs
Table 9 compares the kinds of data collected by household surveys and PPAs, as well as differences in collection, analysis, and synthesis of data.
|One reality||Multiple realities|
|Representative sampling||Purposive sampling|
|More breadth||More depth|
|Structured interviews are used to collect data||Semistructured interviews, focus group discussions, and participatory visual exercises are used to collect data|
|Noncontextual methods||Contextual methods|
|Collects both quantitative and qualitative data||Collects both quantitative and qualitative data|
|Seeks statistically significant relationships||Looks for meaningful patterns; identifies causality and explains statistical correlation|
|Less rapid: can take 18–36 months to complete||More rapid: can take 6–9 months to complete|
|Methods drive the questions||Questions drive the selection of participatory methods used|
|Outcome oriented||Process oriented|
|Household as primary unit of analysis||Intrahousehold relations, social groups, and community as the primary units of analysis|
The different survey approaches have evolved from different traditions. Whereas household surveys determine one reality and attempt to predict behaviors by testing hypotheses (positivist tradition), PPAs seek diversity and present what can sometimes be uncomfortable results reflecting many realities of a diverse and unpredictable environment (post-positivist or constructivist). Unlike household surveys, which collect statistical data on the extent of poverty through standardized methods and rules, PPAs focus on processes and explanations of poverty as defined by individuals and communities within an evolving, flexible, and open framework. Participatory research is more open-ended and interactive. Rather than looking for statistically significant relationships to explain behavior, it emphasizes multiple realities and divergence. Thus, it is important to be clear about which paradigm informs and guides the researcher’s approach (Guba and Lincoln 1996).
Traditional survey data can be used to count, compare, and predict. The strength of the PPA is not in counting but rather in understanding the hidden dimensions of poverty and analyzing processes by which people fall into and get out of poverty. PPAs also seek diversity and recognize that behavior is difficult to predict; moreover, comparisons are often not possible in a dynamic situation. Booth and others (1998) make a distinction between contextual methods (for example, PPAs) that aim to capture social phenomena within their social, cultural, economic, and political context, and noncontextual methods (for example, household surveys) that are designed to collect information that is untainted by the context.
Quantitative and qualitative
There has been a tendency to see a dichotomy between traditional household surveys, which are considered quantitative and objective, and PPAs, which are considered qualitative and subjective. In practice, however, these divisions are not as clear and are often misleading, since subjective questions are increasingly being used in traditional surveys and many PPAs contain quantified information and analysis. Further, there is a qualitative dimension to traditional survey work. In household surveys, for example, interviewers and analysts will interpret informants’ answers subjectively. In the best poverty analysis, the two merge into one integrated analysis (for example, the World Bank’s poverty assessments for Armenia and Zambia, and the Ugandan government’s 1999 Poverty Status Report). For clarification, methods and data shoul be clearly separated. For example, PPAs are a contextual method that collect quantitative and qualitative data, and household surveys are a noncontextual method that produce quantified data (see Hentschel 1999 and Booth and others 1998).
Types of data collected
Neither household surveys nor PPAs are more accurate than the other, since each produces different types of data, fulfilling different informational requirements and illuminating various dimensions of poverty and what it means. While household surveys can provide information on the extent of poverty, PPAs provide explanations, shed light on complexities, and identify the priorities of the poor, making possible new levels of analysis. A PPA alone will not give the whole picture; neither will a household survey. Household surveys often interview only the head of household (usually a man). PPAs typically gather information on intrahousehold issues from more than one perspective, and also explore interhousehold and community-level social issues in addition to gathering household data. The accuracy of a method should also be judged by the extent to which it yields fruitful answers to the questions being asked. Further, a technically accurate method can be inappropriate if it is not the best and/or most feasible way to answer a given question.
Quality of collection and analysis
It is now more widely recognized that scaling-up (carrying out largerscale research beyond individual communities) of participatory techniques has led to the quality of information being compromised. Likewise, household surveys have been criticized for manipulating data and producing misleading results—that is, that which is measured in household surveys is all that matters. What is measured is usually determined by outsiders who may have limited knowledge of local people’s realities. Chambers (forthcoming) states that there is a need for codes of behavior when analyzing participatory research. He adds that it is necessary to repeatedly examine how information and knowledge are generated. “This means critically straining for honest reflection on how one’s own ego, mindset, institutional context, and social and political interests combine to select and shape both personal knowledge and the form it is given when passed on to others.” A further limitation of both household surveys and PPAs is respondent fatigue—many respondents have complained about the demands on their time.
For PPAs, research teams should begin to analyze information during the research process. However, analysis and synthesis require highly trained teams to ensure that the results are valid. In South Africa, a two-day workshop on report writing was convened for the PPA researchers, and card sorting techniques were used with the communities to analyze the material and determine categories for the reports. Policymakers could be involved at this early stage of analysis to better understand the process. Quick and early feedback to key individuals could help policymakers understand the preliminary findings and feel some early ownership before the final report is issued. Some PPAs have collected valuable information, but not all of it has been useful to policymakers. PPAs should try to achieve “optimal ignorance”5 (Chambers 1993), so that information is collected only on issues relevant to policymaking. Careful selection of methods that link to the identified research issues is required. In Mozambique, the PPA presented too much information to policymakers. This was the result of a lack of coordination between the research agenda and methods applied in the field, as well as the reporting style of the coordinating institution. PPAs have achieved less credibility when the results have been too broad, too obvious, or too complex for policy use.
Influence of knowledge and power
In the process of data collection, analysis, and synthesis, the key question is, Who controls the selection of data used to influence policy? The handling of data is often determined by power relations, and power influences the construction and use of knowledge. As Chambers (forthcoming) states, “It is that power forms and frames knowledge and that interpersonal power distorts what is learnt and expressed.” In most traditional surveys, control remains in the hands of those outside the community, especially in
- Designing the questionnaire, which is inflexible and based on what policymakers want to know. Questions are generated in the capital city, reflecting the researchers’ bias.
- Asking the questions, with control remaining with the interviewer, and respondents often feeling inhibited by power differentials (especially between educated enumerator with paper and pen and illiterate respondent; male enumerator and female respondent; urban enumerator and rural respondent). Respondents frequently react to such power relations by telling the enumerators what they want to hear.
- Analyzing the data, which remains outside the community, as does control of its publication.
Once results are accepted, they are repeated and consequently widely believed. Chambers (1997) notes, “To this day, the extent to which survey results are socially and personally constructed remains underresearched, under-reported and under-recognized” (p. 95).
Participatory research is undertaken by facilitators using a diverse set of participatory tools determined by the research agenda and local context. Enabling the poor to participate leads to a reversal in the relationship between the community and the outsider that is implicit in traditional surveys. Facilitators of participatory research need different skills and behavior, including listening to and respecting the expertise of participants, building trust, handing over control, and allowing the community to define the poverty issues that matter. The poor are viewed as participants or partners in the research process, data are shared with them, and the analysis of research results takes place within the community. The poor thus have more control over the research process, and their capacity to appraise, analyze, plan, and act is recognized.
However, even though the analysis of PPAs is controlled by communities to some extent, when this information is translated into macro policy messages and results are aggregated, local people may lose the control and results may not be fed back to communities for comment and verification. Complex and detailed community-level information is valuable for both project design and policy formulation. However, inaccuracies sometimes arise from extrapolating to the national level for policymaking as information becomes too generalized and the context ignored (see Attwood, 1996 for a case study in South Africa of this issue).
Criteria for measuring robustness
It is recognized that there is a need to strengthen the rigor and thus the quality of participatory research. Sample surveys, including the LSMS, are based on the principle that behavior can be measured, aggregated, modeled, and predicted according to statistical measures of reliability or “robustness.” One way to test the robustness of PPA data is to add questions to household surveys based on the findings of the PPA. However, many key PPA conclusions may be context specific and it is not always possible for the results to be representative at a national level, standardized, or aggregated. Many PPA practitioners would argue that it is not always desirable to aggregate and generalize. The strength of some PPA data is their diversity and context-specific nature, as priorities vary across different communities/districts/regions.
Guba and Lincoln (1981) argue that traditional criteria for robustness are inappropriate for participatory research and should be defined differently. Moreover, many surveys are judged on sampling error, but Stone and Campbell (1984) contend that there is also a nonsampling error such as “contextual bias” (for example, cultural differences), which surveys do not take into consideration but which influence the robustness of the results.
There is a widely held belief that for participatory research to be more robust, the tools and techniques should be standardized. Flexibility, however, can be a strength, as the approach, tools, and techniques will vary depending on the community. But in some circumstances it is possible to use certain participatory methods on a wide scale to generate quantitative information. Beneficiary assessments have quantified results based on a sampling frame. The UNDP PPA in Bangladesh used standardized focus discussion groups and the identification of priorities. Also in 1991, the Utilization Survey conducted by Action Aid in Syndhupalchowk, Nepal, used participatory mapping in more than 130 villages to generate data about service utilization.6Chambers (1997) argues that relevance is also an important consideration—can the results be used for learning and action?
In both PPAs and traditional surveys, bias emerges through the interpretation of answers and, most critically, the analysis of results. In participatory research, changing the relationship between the outsiders undertaking the research and members of the community is not an easy process. In a few recent PPAs, the outside facilitator remained dominant and community members tended to say what they thought the facilitator wanted to hear. PRA visual exercises can help to reduce such distortions by opening up the discussion and analysis. But some distortions might still exist, because the process of compiling PPA results involves many stages of information filtering (see figure 7). Where PPAs are more closely linked to the policymaking process, it should be recognized that they may not be politically neutral. In household surveys, bias emerges through preset questionnaire designs, the enumerator’s interpretation of answers to the preset questions, and analysis of results.
Figure 7.Information Filters and Biases: Case Study of the PPA in Zambia
From micro (community level) to macro (policy level)
Although community-level PPA information is valuable for project design, inaccuracies sometimes arise in extrapolating from the community level to the national level for purposes of policymaking, as it is not easy to filter and translate complex messages from the local level. These inaccuracies, however, do not always occur. For example, the Zambian PPA used a small number of communities in different parts of the country, and they had certain characteristics that were administratively uniform and climatically similar. School fees had to be paid in December and January all over the country, and these months were stressful for all rural areas. Thus, a simple message was created. The lesson is that different types of conditions need to be better identified, as well as the degree to which they can be generalized for policy purposes.
Combining PPAs with Household Surveys
To ensure that PPA data do not remain an “add on” to the poverty analysis, the surveys should be combined throughout the research process into four stages: design, implementation, analysis and synthesis, and dissemination (see table 10).
|Data analysis and synthesis|
Trust and understanding should be developed among those who use different approaches to defining research agendas and collecting and analyzing data with the aim of influencing policy. Both survey and participatory assessment practitioners need to understand the limitations of various data sets, appreciate the biases in their own research methods, and know when alternative methods can compensate for some of these limitations. In Zambia, in an attempt to better understand the various approaches to poverty analysis, the government has located the local NGO research team (Participatory Assessment Group, PAG) at the Central Statistics Office (CSO). PAG is currently undertaking participatory poverty monitoring exercises and combining the results of these exercises with the household survey work of the CSO.
The selection of PPA sites can be informed by traditional surveys. For example, in Kenya, the Welfare Monitoring Survey (WMS), which was based on a nationally representative sample of some 12,000 households, was used to identify the poorest districts in each of the six provinces. These districts became the center of focus for the PPA. Within each of these districts, two WMS clusters (roughly equivalent to a village) were randomly selected for the PPA, and the WMS survey enumerators most familiar with the selected clusters were then attached to the PPA teams to serve as guides. Thus, the PPA was conducted in a subsample of clusters used for the WMS. In Guatemala, detailed participatory research was conducted in 10 villages that were later included in the LSMS.
Chambers (forthcoming) states that there are important tradeoffs when designing participatory research. These include the following:
scale and representativeness versus quality—the larger the scale the more representative, but the more difficult to assure quality; scale versus timeliness for training, fieldwork and analysis—the larger the scale, the more time and resources needed for training, fieldwork supervision, and analysis; scale versus resources for follow-up—for a given level of resources, larger scale diminishes scope for follow-up with communities, and for policy-related workshops; standardisation and analysability versus open-endedness and difficulty of analysis; care and comprehensiveness of analysis versus timeliness in influencing policy; the qualifications and nuances of academic standards versus simplified messages for policy influence.
Participatory research and household surveys may be conducted interactively, so that they enhance each other. If a PPA is conducted after the household survey, the results will explain, challenge, reinforce, or shed new light on household survey data. The results of the household survey can also, of course, explain, challenge, or reinforce the PPA (see Carvalho and White 1997; Chung 2000). If the PPA is conducted before the household survey, the PPA results could assist in generating hypotheses, shaping the design of the household survey, and developing survey questions appropriate for the respondents. Ideally, this should be an ongoing process whereby both PPAs and household surveys are conducted periodically and feed into each other (see figure 1). The results of past PPAs indicate that when they are used in conjunction with household surveys, the final assessment is a much fuller analysis of the varying dimensions of poverty, and the policy recommendations are more relevant and informed. The sequencing will be determined by the context in country. In Armenia, for example, the PPA was conducted after the survey work and was able to illuminate areas not covered in the survey, such as reciprocity and kinship networks and the impacts of crime. In Mongolia, the results of the PPA will be used to determine the research agenda of the next LSMS.
Increase awareness of methods by going to the communities: A key recommendation is for policymakers to go to the field and be involved in the research process, in order to understand the strengths and limitations of different approaches and to gain insight into the reality of poor communities. For example, in Costa Rica, a senior official from the Ministry of Economic Planning went with the research teams to the communities and consequently was better able to appreciate the value and limitations of the PPA.
Composition of PPA teams: In some countries, it may be appropriate to include statisticians from the central statistics office in the PPA field teams. This was done, for example, in Pakistan and Mongolia.
Gather perception variables in household surveys: Household surveys could be used to gather data on perceptions using different approaches, such as using more open-ended and semistructured questions with a random group of people, or by including subjective questions on welfare. In general, World Bank LSMSs have yet to incorporate such variables. However, the recent LSMS in Guatemala included questions on trust and social organizations.
Analysis and synthesis stage
Comparing survey results: As stated above, Carvalho and White (1997) argue that it is possible to examine, explain, confirm, and/or enrich information from household surveys to PPAs and vice versa. The results from household surveys and PPAs can be triangulated (validated through cross-checking). Apparent conflicts in data can be further researched. To test the robustness of PPAs, key results can be included in the more representative household surveys. PPAs can assess the validity and interpretation of household data at the local level.
However, when one is synthesizing data, it is not always possible to directly compare different data sources. For example, it can be misleading to use aggregated wealth-ranking PPA data from different communities since communities determine the ranking criteria, which will most likely vary from community to community. As a result, comparisons of such PPA data with household survey data are not very meaningful. Some PPAs have attempted to undertake such comparisons. For example, in the Kenya PPA (see Narayan and Nyamwaya 1996), the results of the Poverty Profiles from the Welfare Monitoring Survey (1992) were compared with the PPA. The report concluded that in three of the five districts, the results of the two approaches were almost identical, with a similar percentage of people falling below the conventional poverty line. And in Tanzania, the PPA report noted the similarity of results from two separate surveys: 50.3 percent were identified as poor and very poor by the PPA, while 49.7 percent fell below the poverty line in the Human Resources Development Survey (HRD) (Narayan 1997). It should be stressed that although such comparisons may have value as an indicator for further investigation, these data sets are not directly comparable.
Combining results for policy recommendations: Some countries are beginning to produce a clear set of policy recommendations based on the results of PPA and household survey data (for example, Armenia, Uganda, Zambia, Vietnam).
As stated above, one of the principles of participatory research is to feed back the findings to communities. This is rarely done in the case of household surveys. However, in some countries the PPA and household survey data have been integrated into one set of policy recommendations, which have then been discussed with communities for further policy feedback (for example, Uganda).
Mongolia Case Example
Mongolia is a good case example of how a PPA has been combined with the household survey.7 The PPA, called the Participatory Living Standards Assessment (PLSA), was the first exercise of its kind in Mongolia to use participatory learning and action methods to broaden and deepen understanding of poverty at the national level. It was conducted by the National Statistical Office (NSO) in 2000 with assistance from the World Bank and other international agencies. It was intended to inform national policy, in part as an essential building block for Mongolia’s Poverty Reduction Strategy Paper. The LSMSs were conducted in 1995 and 1998 and remain the most reliable sources of quantitative data on poverty in Mongolia.
The PLSA is linked to the LSMS in three main ways: (a) research sites for the PLSA were selected to correspond to the 1998 LSMS sites; (b) the results of the PLSA will be used to determine the research agenda for the next LSMS; and (c) capacity was built at the NSO to conduct similar participatory assessments in the future and to promote better integration of data derived from both household surveys and PPAs.
The PLSA aimed to complement and, to the extent possible, update and expand earlier poverty analyses carried out on the basis of the 1995 and 1998 LSMSs, as well as to broaden public discourse on poverty in Mongolia, which has turned largely on distinctions between deserving and undeserving poor. In the past, antipoverty strategies such as the National Poverty Alleviation Program (NPAP) have been conceived as social assistance and formal public safety nets, rather than as public action to enhance the capabilities of poor and vulnerable groups to sustain their own livelihoods. In addition, there has been little understanding of the multiple dimensions, causes, and consequences of impoverishment and vulnerability; of differentiation among the poor and the places they live (implying that very different forms of public action may be required to reach different groups of poor people); of poverty dynamics and distinctions between chronic and transitory poverty; and of how the poor themselves define ill-being and wellbeing. Although there was some participatory action-research in particular localities throughout Mongolia during the 1990s, and some strengthening of local capacity to carry out such analysis, the PLSA represents the first exercise of its kind to bring these skills to bear on national-level understanding of poverty and the formulation of future antipoverty strategies. It was also the first experience on the part of NSO in applying participatory methodologies in poverty analysis.
Using participatory research methodologies, the PLSA permitted a deeper analysis of certain issues that LSMS and other household survey methodologies are often not well equipped to address, such as poverty dynamics over time, spatial dynamics in livelihood strategies, and processes that affect individuals and communities as well as household units. Headcount data mask the fact that the location of poverty may shift over time through migration, for example; and they disguise complex rural-urban linkages which are themselves dynamic. In the future, a time series survey could also yield data on these issues. Such data could then be integrated with results of the PPA.
The PLSA was designed to ensure a high level of complementarity between existing quantitative data from the LSMS and other surveys, and the new qualitative (and, to a lesser extent, quantitative) data arising from the PLSA. This was achieved in the following ways:
- First, the field research was guided by hypotheses that emerged from an initial desk study of the 1995 and 1998 LSMSs and other surveys.
- Second, the sampling approach (see below) entailed revisiting many of the same clusters that were sampled under the 1998 LSMS.
- Third, an attempt was made to include in the analysis, where possible and relevant, newly analyzed and previously unavailable quantitative data from the 1998 LSMS.
- Fourth, an action plan was prepared to guide further analysis of the 1998 LSMS data by NSO staff, which would assist in deepening the poverty profile when used in conjunction with the PLSA findings.8 This action plan dealt with, among other things, the construction of simple household-level asset indexes to complement the analysis of vulnerability.
The selection of provinces (aimags), districts (sums), and communities that participated in the PLSA was guided by three principles: (a) the need to ensure complementarity and comparability with existing quantitative data; (b) the need to capture as much as possible the diversity in living conditions among rural and urban communities; and (c) the need to balance sample size (number of participating communities) with depth of analysis.
In accordance with these principles, the PLSA followed the broad-level sampling frame used for the 1998 LSMS. At each site, the research teams held focus group discussions with three men’s groups, three women’s groups, and one youth group, with 7 to 15 people in each group. A total of 220 focus group discussions and 269 individual household interviews were conducted, involving more than 2,000 participants. This sample is of the same order of magnitude as the 1998 LSMS.
Certain logistical factors impeded this ideal sample frame from being followed in all cases, however. In the rural field sites, the considerable distances that research teams had to travel, owing to extremely low population density, presented a significant challenge to gathering together sufficient people for focus group discussions. In Ulaanbaatar, some community members were reluctant to participate when they realized they would receive no payment or other material incentive for doing so. These problems probably led to some degree of sampling error or selfselection bias in some groups.
Within these practical constraints, the sampling of households and individual participants in communities was guided by participatory wealth ranking. Using this technique, focus groups stratified their communities according to locally relevant parameters of different levels of household well-being. The parameters were themselves elicited through the use of the wealth-ranking method. Using the resulting stratification as a sampling frame, individual households (and individuals within them) were then randomly selected within each stratum, to generate a purposive-random sample. This method combined the advantages of purposive sampling, to ensure that the full range of diversity in living standards was represented, with some measure of random sampling.
The PLSA did try to disseminate the results through the media. However, this proved to be difficult, since the media were at that time more occupied with the national elections. The government intends to disseminate the results of the PLSA by publishing a shortened version with the key policy messages.
Box 15 provides a summary of this section.
Box 15.Summary of Emerging Good Practice for Integrating Data Sets
Develop trust and understanding
The starting point is developing an understanding of the different approaches, and trust among those who control research agendas and those who select, collect, and analyze data used to influence policy.
Confront limitations of own data sets
Practitioners need to confront the limitations of various data sets and to appreciate the hidden biases, lack of objectivity, and the like, in their own research methods. It is important to see where alternative methods may address some of these limitations.
Ongoing process of household surveys and PPAs
If the PPA is conducted after the household survey, the results may explain, challenge, or reinforce household survey data. If the PPA is conducted before the household survey, the results of the PPA may assist in generating hypotheses, shape the household survey questionnaire, and design and develop appropriate survey questions that will be understood by the respondent. Household surveys can help define a research agenda for the PPAs. Ideally, this should be an ongoing process whereby both the PPA and household surveys are done intermittently and feed into each other.
Use survey data to select sites for the PPA
Traditional surveys can identify the poorer areas for PPA research. Increase awareness of methods by going to the communities A key recommendation is that policymakers go to the field and be involved in the research process, in order to understand the strengths and limitations of different approaches and gain insight into the reality of poor communities.
The existing institutional frameworks, both in country (central statistical offices) and within the World Bank, currently provide an entry point for the quantitative data sets and the link with policy analysis, but this is less so for participatory data. The challenge is how to move from research results to policy analysis by finding an appropriate institutional entry point for various data sets and for longer-term analysis.