Evolution of World Bank interest in Multi-Method Approaches
Defining Quantitative and Qualitative Methods
Table 1.1 Characteristics of Quantitative and Qualitative Approaches
Toward an Integrated Approach to Development Research
Table 1.2 Elements of an Integrated, Multi-Disciplinary Research Approach
However, despite significant progress in promoting integrated approaches, many researchers from both quantitative and qualitative traditions still often find it difficult to make full use of the data collection methods and analysis from the other tradition. Some quantitative survey researchers may find it difficult to make full use of the wealth of case studies, PRA maps, calendars, and key informant interviews they have commissioned. Quantitative researchers also complain that their important messages on the incidence and determinants of key development variables such as malnutrition, usage of health services and consumption-expenditure measures of poverty; and on the poverty consequences of economic variables such as price changes, agricultural marketing policies etc are dismissed as being "too macro" by many qualitative researchers. On the other hand, qualitative researchers often complain that their findings , may be dismissed by survey researchers as not being sufficiently representative or rigorous. Qualitative researchers also express the concern that even after collaborative research efforts, the survey researchers still do not understand the true nature of a complex phenomenon such as poverty.
The message is that there is a growing consensus on the value of integrated
approaches, but that further work needed to develop guidelines for the
effective use of these integrated approaches. Most of this publication
is devoted to presenting examples of promising approaches to integrated
quantitative and qualitative research which have been used in the World
Bank and other development institutions. The purpose of this chapter is
to provide a framework for assessing these experiences and for discussing
the lessons that can be learned. After tracing the increasing interest
of the World Bank and other development agencies in the use of integrated
research approaches, this chapter addresses the following four issues:
(a) the definition of qualitative and quantitative research methods; (b)
potential benefits from the use of integrated approaches; (c) issues and
challenges in the use of integrated approaches in development work; and,
finally, (d) whether there is an integrated research approach that consists
of more than simply using a wider range of data collection methods.
Return to top of page
Evolution of World Bank Interest in Multi-Method Approaches
A number of factors have contributed to the World Bank’s growing interest in the integration of quantitative and qualitative research methods. The first is the Bank’s focus on poverty. Much of the early work on poverty was highly quantitative: how many people fall below the poverty line, how this number varies during different kinds of economic change, and so on. It became increasingly clear, however, that while numbers are essential for policy and monitoring purposes, it is also important to understand people’s perception of poverty and their mechanisms for coping with poverty and other situations of extreme economic and social stress. Kozel and Parker’s study on poverty in Northern India (Chapter 4) illustrates the wide range of coping mechanisms and different attitudes to the possibility of escaping from poverty. Although many poverty alleviation programs are based on providing opportunities for economic improvement within the community where people live, their study showed that many families believe they have no way to escape from poverty, while others see migration as the only possible escape.
Poverty Reduction and the World Bank: Progress in Fiscal 1998 summarizes recent developments in the use of mixed-method approaches in the Bank’s work on poverty analysis. While almost all World Bank poverty assessments (PAs) rely on data collected through sample surveys, an increasing number combined this with data collected through participatory poverty assessment (PPA) techniques and analyses carried out with and by poor people in the field. The report states that while earlier PPAs contributed mainly to enriching the description of poverty and seldom followed an integrated approach, more recent work shows three important changes: greater integration of methods, more emphasis on understanding the causes of poverty, and greater participation.
The report distinguishes between data, which can be either quantitative or qualitative, and data collection and analysis methods which can be either contextual or non-contextual. "Contextual methods attempt to understand human behavior within the social, cultural, economic, and political environment of a locality, usually a village or neighborhood or social group." Non-contextual methods, on the other hand, "abstract from the particularities of a locality to gauge general trends." Recent poverty assessments seek to enrich the understanding of poverty through the integration of contextual and non-contextual methods. The report argues that it is helpful to think of two continua: one where methods used are more or less contextual, and the other where data gathered are more or less quantitative.
Hentschel (1999) illustrates the use of contextuality to characterise information needs for planning the utilization of public health services. He argues that in the public health sector, and in the social sciences more generally, rather than considering quantitative and qualitative methods as describing two different realities, they are both needed to describe and understand one reality. He also argues that "labeling both methods and data as quantitative or qualitative creates a problem with regard to analyzing what the comparative advantages of different methods and data types are to understand human behaviour like the ulitisation of health facilities." Hentschel’s paper illustrates three ways in which contextual and non-contextual methods can be combined:
Mangahas describes a different approach to poverty assessment adopted by the Social Weather Stations program in the Philippines, which has used a small set of qualitative measures to monitor changes in the level of poverty in the Philippines for over twenty years. These self-rating indicators consistently estimate a higher proportion of households experiencing poverty than do the official poverty indicators. In addition to the economy and speed with which these measures can be applied and analyzed, Mangahas argues that these indicators provide a better reflection of the impact of short-term economic and political changes on poor and vulnerable groups. Somewhat similar methodologies are used, although for different purposes, by the Eurobarometer’s "gainers/losers" and "optimists/pessimists" indicators, and the U.S. Conference Board’s consumer confidence index.
The Bank’s work on resettlement is another area where there has been a great deal of interest in integrated research methods. The Bank has had a strong focus on understanding the processes of resettlement, which traditionally has been an area for anthropological research. However, there is a need for quantitative data when it comes to evaluating the impact of a large-scale resettlement program that involves the relocation of tens of thousands of people.
As the Bank has become more involved in dialogue with civil society—including non-governmental organizations (NGOs), women’s organizations, and academic groups—it has become apparent that different groups employ different paradigms for addressing issues such as structural adjustment, gender, or household structures. As a result, there have been some major efforts to develop a common framework for assessing whether people are better or worse off as a result of these economic reforms. One example is the Structural Adjustment Participatory Research Initiative (SAPRI), in which NGOs and Bank researchers are trying to come up with a broader, more integrated approach for evaluating the impacts of structural adjustment. That is still very much an ongoing dialogue, but there has been some important progress.
As the Bank has moved into the field of social development, there has been a greater focus on participatory assessment methods and other qualitative approaches, such as rapid rural appraisals. This has given rise to a number of methodological questions regarding how to present qualitative research findings so as to increase their legitimacy in the eyes of quantitatively oriented policy makers and planners who wish to know whether the findings of specific cases can be generalized to wider populations. For example, how can you determine whether the findings of a focus group represent the views of the community, and not just the position of an influential individual or minority?
Another promising area for the use of a multi-method approach is the Bank’s social assessment work, in which researchers are using a variety of quantitative and qualitative tools to plan and monitor development interventions. The objective of these studies is to understand the processes of change, determine the number of people affected, and identify socially and politically viable options for interventions such as introducing new transport systems, privatizing industries, improving the economic viability of water supply systems, and so on. Chapter 9 illustrates how integrated approaches were applied in a recent social assessment study for a proposed water supply, sanitation, and health project in Uzbekistan.
Incorporating quantitative methods of data collection and analysis into previously qualitative research areas is becoming important, both in the Bank’s operational work and to increase the legitimacy of qualitative data. For example, even when they themselves agree with the findings of the qualitative studies, policy makers often want the qualitative findings to be presented using conventional statistical principles, in order to make these findings more acceptable to other agencies. The need to increase the reliability of qualitative findings through the use of appropriate statistical techniques is often important to ensure the acceptability and utilization of findings
There has been significant interest in developing rapid, cost-effective assessment methods to provide timely feedback on the likely social costs and outcomes of proposed projects and policies. There may only be a few weeks to conduct such a study, either for administrative reasons (for example because loan documents must be prepared by a certain date) or because decisions must be made in the midst of a crisis. Consequently, there is a strong desire to find a way to do things quickly, but with sufficient rigor that the findings can be defended and generalized. The recent financial crises in South East Asia stimulated interest in this application of rapid social assessments, because the crises, and the measures taken to address them, often had serious repercussions for poor or vulnerable groups.
The need for a different research paradigm has also been identified in the area of gender analysis. Conventional survey methods are often inadequate for capturing the views of women in male-dominated societies where only the men provide information to outsiders, or where the women feel intimidated about responding freely to research questions. The growing interest in participatory research has emphasized the fact that the views of women are frequently not captured unless special gender-sensitive methods are devised to give them voice (Bamberger, Blackden and Taddese 1994). There has been a great deal of discussion about ways to integrate economic analysis with gender analysis in order to adequately give voice to women and to understand how development affects women and men. An important area in which innovative research methods have been conducted concerns the assessment of women’s time poverty through the measurement of the hours per day which women spend on each of their multiple tasks (Morris-Hughes and Blackden 1993). Time poverty analysis and the development of methods to obtain more realistic estimates of the value of women’s time is proving to be a key issue in making transport projects more gender sensitive (Barwell 1996, Bryceson and Howe 1993, Bamberger and Lebo 1999). Some of these issues are discussed in Chapter 3.
The demand for integrated research approaches comes from researchers
with a quantitative background (for example, economists working on poverty
assessments) as well as from those who have traditionally used more qualitative
field studies (for example, work on resettlement and social assessment).
The case studies presented in Chapters 4-11 show that integrated approaches
appeal in different ways to researchers coming from different traditions.
Return to top of page
Defining Quantitative and Qualitative Methods
Most attempts to draw clear distinctions between quantitative and qualitative research have floundered in the face of the many counterexamples that can be found to challenge each categorization. Chung (Chapter 2) cites a number of caveats for every simple definition. For example, while textual reports on key informant interviews and group discussions will often be analyzed using broad qualitative interpretations, it is also possible to conduct the analysis using rigorous statistical content analysis. On the other hand, the design of the most highly structured questionnaires usually involves subjective, qualitative decisions on the selection of topics to be included and the way that questions are worded.
This section discusses the distinction between quantitative and qualitative methods in different stages of the research process. The next section addresses the question of whether it is possible to identify an integrated approach to, or a unique comparative method for, development research that consists of more than simply broadening the range of data collection and analysis methods.
When attempting to define the differences between quantitative and qualitative research methods, it is useful to think of methods of sample selection, design of the research protocol, data collection and recording, and data analysis as each being ranged along a quantitative/qualitative continuum. While some studies rely exclusively on quantitative methods for sampling, data collection, and data analysis, and others rely exclusively on qualitative methods, many studies mix and match statistical sampling techniques, qualitative data collection, and statistical analysis from the qualitative and quantitative traditions. It then becomes an empirical question as to whether a particular discipline (for example demography or economics) tends to rely more heavily on tools and methods from the quantitative end of the continuum than does another discipline or sub-discipline (such as empowerment evaluation or participatory research). Even when a particular discipline has a general penchant for either quantitative or qualitative methods, many exceptions will be found.
Hentschel’s (1999) distinction, discussed earlier in this chapter, between contextual and non-contextual methods of data collection and data analysis is also helpful in this context. Contextual methods attempt to understand human behavior within the social, cultural, economic, and political environment of a locality, usually a village or neighborhood or social group; while non-contextual methods abstract from the particularities of a locality to gauge general trends.
It is also helpful to distinguish between the two main purposes for
qualitative research. The first is exploratory research, in which
the objective is to understand the context within which behavior is determined
or processes take place in order to develop hypotheses, or to present case
studies of particular communities, groups, or individuals. In these studies
methods can be flexible and unstructured, with little or no concern for
comparative rigor. The second purpose is directed research, in which
hypotheses are tested, or rigorous comparative findings are required. For
these latter kinds of studies, both qualitative and quantitative researchers
use similar sampling methods, follow standard measurement and reporting
procedures, and observe careful documentation requirements.
Return to top of page
Table 1.1 describes typical quantitative and qualitative approaches to sample selection, design of the research protocol, data collection techniques, and data analysis. The primary characteristics of the two approaches in each of these areas are described in more detail below.
|Table 1.1 Characteristics of Quantitative and Qualitative Approaches to Sample Selection, the Research Protocol, Data Collection, and Data Analysis|
|Research Activity||Quantitative Approach||Qualitative Approach|
|Selection of subjects or units of analysis||
|Data collection and recording methods||
|Table 1.1 (continued)|
|Research Activity||Quantitative Approach||Qualitative Approach|
|Role of the conceptual framework||
The quantitative framework can often, but not always, be characteristized as:
Return to top of page
Procedures for Selection of Subjects
An essential characteristic of quantitative research is random selection of subjects so that each subject has an equal or known probability of selection. This makes it possible to generalize from the sample to the total population. Equally important, the selection procedures make it possible to determine the statistical significance of differences between subgroups.
Qualitative research, by comparison, has no single defining sampling procedure; rather, the choice of sampling method is determined by the purpose of the study. Miles and Huberman talk about the "bounding function" of the conceptual framework in defining sample selection procedures. In exploratory studies, or where the purpose of the study is to obtain a general understanding of the attitudes or priority concerns of a community, there may be no clearly defined selection procedure; any interested members of the community may be invited to participate in focus groups. One of the most common methods is purposive or theoretical sampling, in which interviews are conducted with representatives of each category, stakeholder, or socioeconomic group of interest to the objective of the study, but without random selection of the particular subjects who are studied in each group. Kozel and Parker use this approach in their study of poverty in India presented in Chapter 4.
There are, however, many examples in which random sampling procedures are used to select the subjects for qualitative interviews. A promising and practical approach to ensuring a certain degree of representativity is to draw the sample of communities or individuals for the qualitative research from the sampling frame that is used for the quantitative stage of the research (see Chapters 4, 6, and 8). Qualitative research may also be interested in the selection and analysis of outliers in order to explain the reasons for the deviation from the general pattern observed in the data. In the Indonesia water supply study (Chapter 8), this approach is used to follow up on the one community where women were not involved in water supply management.
Another difference between qualitative and quantitative work is found in the area of research protocol. Whereas in quantitative research it is extremely important to administer a survey according to a standard protocol, many qualitative research protocols are relatively unstructured and flexible. When the purpose of the study is to understand the unique characteristics of a particular community, organization, or group, qualitative researchers will often modify the format or content of a study while it is in progress in order to capitalize on what is learned in the field and to pursue certain preliminary findings. However, as indicated above, when the purpose is to ensure comparability, a precisely defined research protocol will also be used for qualitative research.
Data Collection and Recording Methods
In quantitative research, information is usually collected and recorded either numerically or in the form of pre-coded categories. The principal advantage of such surveys is that they can be administered to large numbers of individuals, organizations, or households using standardized methods.
In qualitative studies, information is most frequently recorded in the form of descriptive textual reports with little or no categorization. The documentation may consist of subjects’ responses to semi-structured interview questions, notes taken during focus groups or other kinds of group interaction, or the researcher’s observations of relevant aspects of a community or organization. In other cases the information may be recorded within pre-defined categories, but with reports presented in an unstructured or semi-structured form within each category.
It should be noted, however, that some qualitative studies do rely on pre-coded classification of data, such as the perceived level of understanding of campaign communications, level of formality of community groups, and so on. The Indonesia water supply study (Chapter 8) uses this approach to classify the effectiveness of community organization in project implementation, while the Nicaragua and Pakistan education studies (Chapters 6 and 7, respectively) use it to evaluate different aspects of the quality of education in those countries.
Triangulation is the principle of increasing the validity of the data by looking at different data sources or by going back to the same subjects at different periods of time and asking the same kinds of questions. The purpose of triangulation is to improve the validity of one’s findings.
Some researchers claim that triangulation is a characteristic of qualitative research. However, triangulation is also used in quantitative research, usually by comparing findings from different surveys, or by comparing survey findings with census data.
One of the most important ways in which triangulation can be used is to compare the findings of qualitative and quantitative studies. The Cartagena study (Chapter 4) uses participant observation and informal interviews to double check the sources and volume of transfers reported in the sample survey.
Quantitative methods for data analysis are most commonly analyzed using descriptive statistical methods such as measures of dispersion or central tendency, or multivariate analysis to examine the factors contributing to the direction and magnitude of change. In addition, statistical tests will often be used to test the significance of differences between groups or of changes over time. Standardization across observations makes it possible to aggregate measures and to make statistical comparisons among individuals, households, regions, and time periods.
Data collected using Qualitative methods can be either be analyzed and interpreted descriptively or numerically. The objective of descriptive analysis is to understand the unique characteristics of a particular context (i.e., community, organization, event), household, or individual. Numerical analysis permits systematic comparison of these entities. A wide range of analytical methods are available for converting textual and image data (photographs, advertisements, etc.) to a numerical format.
Qualitative research is frequently more interested in eliciting the stories behind particular individuals or groups. In some cases the emphasis will be on statistical outliers or unusual cases that are not behaving as expected, whereas in other cases the purpose is to put "flesh and bones" on the findings of the statistical analysis. For example, a researcher conducting a nutritional study might notice that in a high-income household, where one would expect the children to be well nourished, they are instead quite thin and stunted. This finding might lead the researcher to do a case study of that household to try to uncover the dynamics behind that unexpected outcome.
In other instances, case studies may be prepared on typical households or communities to help understand the meaning of the statistical findings. In an earlier study on inter-household transfers on which the research described in Chapter 4 was based, the statistical analysis identified a number of inter-household transfer strategies including transfer avoiders, transfer givers, transfer receivers, and households seeking to develop transfer networks. Case studies were then prepared to illustrate each of these strategies.
Qualitative and quantitative research frequently differ in terms of the role of the conceptual framework. In quantitative research, the researcher’s conceptual framework usually leads to the formulation of hypotheses, which are then tested. This can also be true for qualitative research, but when the purpose is to explore the uniqueness of each situation, the conceptual framework and research protocols may evolve in the field as data are obtained. This occurs through the process of iteration, the purpose of which is to clarify or follow up on information that was obtained in an earlier stage of the research.
While quantitative research ensures comparability over time by applying standard questions to the same or statistically comparable samples of households or individuals at different points in time, iteration in qualitative research does not always involve repeated contact with the same or comparable respondents or groups. In a qualitative study, the purpose may be to react to, and build upon, what was learned in the previous round. In some cases this may require that different kinds of respondents may be added as the study progresses.
While there are many exceptions, qualitative conceptual frameworks can
often be characterized as having a micro rather than a macro focus, seeking
to understand processes starting at the individual rather than the aggregate
level, and having a holistic focus. The framework and analysis also tend
to be interpretive, often relying on naturalistic observation to capture
the constraints of everyday life.
Return to top of page
Toward an Integrated Approach to Development Research
Although some writers are concerned that the over-enthusiastic adoption of qualitative methods may negatively affect the quality of quantitative research, it is now widely acknowledged that there are considerable benefits to be gained from combining quantitative and qualitative methods. Kertzer and Fricke (1997) state that there is a " . . . growing recognition of the limitations of the survey mode data collection for gathering accurate, fully textured, and nuanced data at multiple levels of social reality" and emphasize that surveys must be complemented by qualitative methods.
Despite increasing eclecticism in the combination of data collection methods, there is much less integration at the level of the conceptual framework and the overall research approach. A demographer or economist may use focus groups or observational techniques to enrich survey data, or a social anthropologist might include a rapid sample survey to compare the socioeconomic characteristics of the case study households with those of the community. In general, however, most researchers have found it difficult to break out of the conceptual framework of their own discipline.
Although quantitatively oriented researchers such as demographers and economists make increasing use of qualitative data collection methods, Obermeyer (1997) argues that it has proved much harder for them to accept the analytical frameworks used in the disciplines from which qualitative data collection methods are borrowed. Some have argued that the popularity of focus groups is due to the fact that they are much less tied to particular analytical frameworks than are some other qualitative methods. Similarly, some economists feel that many qualitatively oriented sociologists and anthropologists are unwilling to give up their skepticism about sample survey research findings on subjects such as poverty.
A number of writers have argued that the greatest potential benefit from cross-disciplinary research would come from developing new integrated analytical frameworks where two disciplines could each draw on the conceptual and analytical frameworks of the other. Rao, for example, would like to see ethnographic analysis being used to inform the development of rational choice models. While progress has been made, the potential for sharing disciplinary frameworks remains largely untapped.
Ragin argues in favor of a distinct comparative research approach that is more than the simple integration of quantitative and qualitative methods. While qualitative researchers tend to consider multiple cases as many instances of the same thing,
A fully integrated research approach would draw on the conceptual and analytical frameworks of at least two disciplines in the design, analysis, and interpretation of the research, while at the same time combining a broad range of data collection methods¾ in most cases including both quantitative and qualitative methods. Although the integrated approach must be adapted to the needs of each specific study, a fully integrated research approach will normally seek to ensure integration at the following stages of the research process:
An integrated approach can broaden the conceptual and analytical framework of a study. In many demographic or economic research projects, for example, the study is designed to test hypotheses concerning quantitative relationships among the variables in the model. Frequently non-contextual data collection and analysis methods are used, and no contextual variables are included to take into consideration the unique characteristics of the social, economic, political, and cultural context within which the study is conducted. In these cases an integrated approach could strengthen the analysis by taking into consideration the influence of contextual variables such as social organization, culture or the political context.
In demographic research in particular, there is increasing interest in the effect of culture on demographic outcomes, but conventional quantitative research approaches have not been able to capture the subtleties and complexities of culture. Culture is either ignored in the modeling, or it is reduced to one or more dummy variables in what Obermeyer (1997) describes as the "add fieldwork and stir" approach. Many demographers, and some economists, believe that one of the greatest potential contributions of anthropology to their discipline is to permit a fuller understanding of culture to be built into their models. The analysis of culture requires that the conventional demographic or economic analytical framework be broadened to incorporate cultural concepts and methods of data collection and interpretation into the research design¾ it is not just a question of introducing a new data collection instrument.
Kertzer and Fricke discuss several factors which may constrain the full adoption of the anthropological approach to culture by demographers; the second and third of these points may also be relevant to economics and other quantitatively oriented disciplines. First, most demographers are trying to adopt a structuralist-functionalist model of culture, which most anthropologists would consider to be at least thirty years out of date. Second, many demographers who are unfamiliar with anthropological theories and methods have tried to adopt certain concepts concerning culture without understanding the theories underlying the concept. Such borrowing out of context tends to greatly limit the utility of the ideas that are borrowed. A related point is that demographers have usually not wished to become involved with the ideological debates surrounding modern anthropology, some of which concern the interpretation of culture.
On the other hand, sociological, political science, and anthropological frameworks can also be incorporated into the design of economic research in areas such as poverty assessment (see the India poverty study described in Chapter 4), social capital, inter-household transfers (see the Cartagena income transfer study in Chapter 5), and the evaluation of the impacts of many kinds of development interventions (see the evaluation of education projects in Chapters 6 and 7, and the Indonesia water supply project in Chapter 8). In each case the analytical framework and the whole research approach would have to be broadened to incorporate these new concepts.
Similarly, many kinds of anthropological, sociological, and political research can benefit from the incorporation of economic approaches. Rao argues that economics can play an important role by offering a number of analytical approaches¾ such as rational choice models and game theory¾ which can be applied to the kinds of rich observation of human behavior provided by anthropology and sociology and political science. He states:
Lampietti (Chapter 10) illustrates how the economic concept of contingent valuation was combined with anthropological studies of household and community knowledge and attitudes about malaria to provide new insights into household decision making with respect to the purchase of a hypothetical malaria treatment in Northern Ethiopia.
Qualitative methods can be used to conduct exploratory analysis during the preliminary stages of a survey to understand the social, cultural, and political context affecting the communities to be covered by the study. These methods can also help with hypothesis testing, and the definition of key concepts such as "the household," "work," and "vulnerability." The full value of this exploratory analysis will only be obtained if it is conducted early in the research design, and if a sociological, anthropological, or political science framework has been built into the conceptual framework. In the India poverty study (Chapter 4), the exploratory analysis highlighted the importance of the caste system as a constraint on household perception of the possibility of escaping from poverty, and it showed that many families believed that leaving the village was the only possible way to escape.
The use of statistical sampling procedures can help to ensure that the results of case study research and other qualitative methods can be generalized, thereby increasing the likelihood that the findings will be accepted and used by quantitatively oriented policy makers and planners (see Chapters 4,5,6,7, and 8 for alternative ways in which researchers tried to ensure the generalizability and credibility of their findings). On the other hand, exploratory research methods can be used to determine the criteria for the formulation of cluster-sampling and stratified-sampling designs by helping to identify some of the important social and cultural characteristics of different groups which could not be obtained from the kinds of statistical sources normally used in sample selection. Exploratory research can also be useful in multi-stage sampling, where it is important to understand the composition of individual households within a multi-unit building or compound.
Data Collection Methods
This issue has been discussed earlier. Data collection is the area in which the procedures and benefits of integrated approaches are best understood.
Understanding Context: The Political and Economic Environment, Project Implementation Processes, the Structure and Operation of Organizations, and Culture
The traditional approach to project planning and evaluation looked at inputs, implementationprocesses (how the inputs are used), outputs, and impacts. However, researchers are increasingly aware that projects take place in a certain context, and they are recognizing that it is important to understand the household characteristics, the socioeconomic environment, and the political and institutional environment within the project is planned and implemented.
Although a quantitative survey is usually the best way to estimate the magnitude and distribution of poverty, the proportion of the population with access to different public services, or the quantitative impacts of projects;, it is usually not the best method for understanding the socioeconomic environment, institutional and political processes, or how different kinds of households within different cultural contexts are going to respond to a project. Some researchers would go even further and challenge the conventional wisdom that survey research is an objective process for collecting "facts" about a community or activity. Instead they would argue that any kind of interview must be regarded as a social process in which the outcome is dependent upon the characteristics and expectations of the interviewer and respondents, and of the context in which the interview takes place.
Qualitative methods, by comparison, are well suited for the analysis and interpretation of the context within which families live, or within which organizations or groups are operating and projects are implemented. Ethnography, sociology, and political science can evaluate contextual variables and explain how they affect the behavior and attitudes of the individuals or groups being studied. The analysis of these contextual and cultural factors should be an integral part of the research design. If this analysis is incorporated into an exploratory study during the research design phase, communities can be ranked on the contextual variables of interest to the study so that this information can be used in the design of stratified or cluster samples.
However, qualitative methods of contextual analysis are often in-depth studies of a single, or small number of communities or areas, and consequently it may be difficult to generalize from these studies to assess the overall impact of these contextual factors at the regional or national level. In cases where it is necessary to generalize to large populations, the findings of the qualitative studies can be used to identify a few numerical indicators which can then be incorporated into quantitative surveys as part of large-scale studies.
Qualitative methods are particularly useful for the description of the project implementation process and for assessing the quality of implementation. Differences in outcomes and impacts for projects with similar resource endowments can often be attributed to differences in how the projects were implemented. Effective participatory approaches, adapted to local conditions, may be used in one project, while another project might use more rigid procedures that are less responsive to local conditions. As Sedlacek and Hunte (Chapter 7) show, individual personalities (in this case the characteristics of the head teacher) can also have a major impact on project outcomes.
Qualitative methods can also be used to assess the quality of participatory planning and implementation methods. Every community and group has its own distribution of power, which is based partly on local traditions, partly on linkages to external political and economic systems, and partly on personalities. As a result, some participatory processes are dominated by a few traditional leaders or powerful people, while others are more open. It is frequently the case that certain groups, such as women, young people, economically weaker groups, or certain ethnic groups are at least partially excluded from effective participation in decision-making. Brown (Chapter 8) reports that women were largely excluded from the planning and management of local water supply projects in Indonesia, even though water management is traditionally the responsibility of women. Surveys and methods relying on administrative information or the reports of community leaders are usually inadequate for capturing these political and cultural nuances.
Some of the recent World Bank poverty assessments (World Bank 1999) show how contextual and non-contextual methods can be combined with quantitative and qualitative data to explain how contextual factors such as local culture interact with standard indicators of poverty and welfare to influence responses to poverty alleviation programs as well as feelings of family members about their present and future conditions.
The following chapters provide examples of how qualitative methods have been used by researchers to clarify various contextual issues:
Consistency checks and alternative measures of key variables should always be an integral part of integrated approaches. These alternative indicators should be used both as consistency checks and as a means of obtaining a deeper understanding of the variables being studied. Poverty research has shown, for example, that women and men may have a different understanding of the concepts of poverty. In some cases, Participatory Rural Appraisal (PRA) techniques such as wealth ranking produce rankings of the relative poverty of households or communities that are consistent with the survey estimates based on expenditure, consumption, or income, but in other cases the rankings may produce significant discrepancies. Women, for example, may place greater emphasis on the concept of vulnerability, which frequently concerns a lack of access to social support networks, than they do on current income or consumption (see the India poverty study in Chapter 4). A widow, for example, may be ranked as more vulnerable to the consequences of drought or family crises than a married woman, even though the former may have a higher current income or expenditure.
The Indonesia water supply study (Chapter 8) used triangulation to compare survey findings and observational estimates of the quality of community organization and the effectiveness of project implementation. Meetings were held to discuss and resolve any inconsistencies between the two estimates.
Data Analysis and Follow-Up Field Work
The data analysis must incorporate both quantitative and qualitative analysis into an integrated analytical framework. The integrated sampling procedures discussed earlier must be used to assess the representativity of the findings from the case studies and the extent to which they can be generalized. The qualitative analysis of institutions, cultural variables, and the effectiveness of project implementation processes may be used to create dichotomous (dummy), nominal, or ordinal variables which can then be incorporated into the multivariate analysis to help analyze and explain differences in outcome variables in different communities or projects.
In the majority of research studies, inconsistencies between triangulated estimates are either ignored (the most common approach) or explained away in an unconvincing manner; and it is unusual to find a systematic treatment of inconsistent findings. A truly integrated analytical approach should include systematic procedures for identification and analysis of triangulation discrepancies and for addressing and resolving inconsistencies. The researcher should anticipate that such discrepancies will be found and should use them to enrich the analysis and interpretation. For example, systematic differences between the household income reported by women and men might reflect a weakness in the survey design, or it might provide some important insights into gender differences in control over, and information about, household resources.
Wherever possible the research design should include time and resources to permit a return to the field to follow up on these inconsistencies and to provide further insights into unexpected or difficult-to-explain findings. Rapid qualitative studies can make a valuable contribution in this area, because these methodologies are designed to provide greater depth of understanding concerning differences between communities, organizations, or households and to explain subtle differences that were not captured by the survey instruments. The study on village water supply in Indonesia (Chapter 8) found that the water supply in one community was managed by men, whereas in all other cases this was the responsibility of women. The follow-up study revealed that women in this community were able to earn exceptionally high income from diary farming, thus the men were willing to take over responsibility for water management in order to share in the earnings of their wives. Without the follow-up study this unusual finding might have been ignored or even considered as a reporting error.
Whenever possible, follow-up fieldwork should be conducted to
It is also important to ensure the integration of quantitative and qualitative data in the presentation of findings. The data gathered through case studies, wealth rankings, focus group reports, and other qualitative tools must be linked to, and compared with, survey findings. Case studies should be used not only to illustrate and enrich statistical tables, but also to combine analysis on multiple levels. For example, how do cultural practices with respect to girl’s education, water management, sexual division of labor, marriage, and other issues affect the observed statistical regularities concerning labor force participation, willingness to pay for community projects, the sustainability of infrastructure investments, and so on? The challenge is to capture the complex interactions between these different levels.
Table 1.2 proposes a set of guidelines for the design and implementation
of a fully integrated, multi-disciplinary research approach. A further
discussion of how quantitative and qualitative methods can be used at each
stage of the research process is provided in Chapter 12.
Return to top of page
Table 1.2 Elements of an Integrated, Multi-Disciplinary Research Approach
Return to top of page
Chapter 2. Issues and Approaches in the Use of Integrated Methods (Kimberly Chung)
This chapter presents a short introduction to research with qualitative methods and identifies some of the distinguishing characteristics of quantitative and qualitative methods as they are typically used at different stages of the research process. Chung stresses that there are very few clear distinctions between quantitative and qualitative approaches that hold in all cases. Whole volumes are written on qualitative methods, hence it is impossible to give a comprehensive overview in this chapter. The author addresses objections that are commonly cited as obstacles to accepting qualitative research and suggests reasons for using an integrated approach. Options for structuring an integrated research study are presented, along with prerequisites for conducting such research. The author cautions that no single approach will be best for all possible situations. The choice of methods will depend on the goals and budget of the study as well as the time and personnel available to it.
Chapter 3: Gender Issues in the Use of Integrated Approaches. (Roberta Spalter-Roth)
This chapter presents a sociologist’s perspective on research methodologies, with an emphasis on approaches for studying gender issues. The author stresses the danger of creating a false divide between quantitative and qualitative methodologies and the need to avoid the assumption that only qualitative research can give women voice and provide insights on gender relations. She urges combining a variety of research strategies, including survey research done for other purposes, and provides examples of studies that use an integrated approach.
Chapter 4. Integrated Approaches to Poverty Assessment in India (Valerie Kozel and Barbara Parker)
This chapter summarizes an innovative study of poverty in rural India that combined quantitative and qualitative research approaches. The authors discuss the objectives, methodological approaches, and preliminary findings of the study, with particular attention to issues such as underlying assumptions, sampling techniques, and lessons learned in the process of implementing an integrated research design. Recommendations for future research are also presented.
The study shows that poverty is an extremely complex phenomenon, and that an interdisciplinary approach is required to understand the socio-cultural, political, economic, and institutional context within which people live their lives and which determines how poverty is experienced and perceived by different groups. Qualitative methods were essential for understanding the caste system and its role in determining attitudes to the possibility of escaping from poverty. The interdisciplinary approach also proved valuable in assessing the effectiveness of various government and donor-funded poverty alleviation programs, providing new insights into the strengths and weaknesses of some major initiatives.
Chapter 5: Studying Inter-Household Transfers and Survival Strategies of the Poor in Cartagena, Colombia. (Gwyn Wansbrough, Debra Jones, and Christina Kappaz)
This chapter presents the experience of using a combination of quantitative and qualitative research methods in a study of inter-household transfers in the Southeastern Zone (SEZ) of Cartagena, Colombia, which was conducted as a follow up to World Bank research on this topic conducted in the SEZ in 1982. The authors discuss the design and implementation of the study, which included both quantitative and qualitative approaches, and present their observations regarding integrated research. Issues addressed include survey design, sampling techniques, selection and training of interviewers, and timing and cost of research activities.
The researchers found that inter-household transfers provided a significant proportion of the total household income, particularly for the poorest households and for female headed households. The study demonstrated the value of an integrated approach in the analysis of inter-household transfers. Transfers are determined by a complex set of social rules that are difficult to identify through formal surveys, and the transfers often take place in ways that are difficult to capture through surveys. The researchers concluded that once an exploratory qualitative study has been conducted to understand the dynamics of transfers in a particular community, it is possible to obtain reasonable estimates of the magnitude, distribution, and use of transfers through quantitative surveys. However, surveys will frequently fail to capture some of the transfers, and higher and more reliable estimates of the total volume of transfers will be obtained if surveys can be complemented by extended, informal interviews and participant observation with a sub-sample of households.
Chapter 6. Evaluating Nicaragua’s School-Based Management Reform (Laura B. Rawlings)
This chapter discusses how a mixed-method approach was used to evaluate the impact of Nicaragua’s school decentralization reform. Following an overview of the reform program and the objectives of the evaluation, the author discusses the quantitative and qualitative techniques utilized in the study, sampling issues, and the sequence of activities. The chapter summarizes the findings of the first phase of the study regarding the role of the school in governance, the perceived level of influence of key stakeholders (directors, teachers, and school council members), and the impact of the reform on school performance. This chapter contributes to the evaluation of the Nicaraguan reform by clarifying the objectives, methods, and value-added of the mixed-method approach.
Chapter 7. Evaluating the Impacts of Decentralization and Community Participation on Educational Quality and the Participation of Girls in Pakistan (Guilherme Sedlacek and Pamela Hunte)
This chapter focuses on the qualitative component of mixed-method research conducted in Pakistan to evaluate the impacts of decentralization and community participation on the quality of schools and to identify factors that influence the educational participation of girls. The authors discuss issues such as research design and sampling methods, and present preliminary findings and their implications for the project. The authors also suggest a model for conducting follow-on studies in consultation with the community and offer recommendations for conducting integrated research.
The authors stress the importance of linking the selection of the areas for qualitative studies to the statistical sampling frame used for the quantitative studies in order to ensure the representativity and generalizability of the qualitative findings. The qualitative study was important in understanding what parents consider important in the assessment of a successful school. It suggested that the original project design might have placed too much emphasis on parental and community participation, and might have under-estimated the importance of school organization, discipline, the critical role of the head teacher, and the importance of alliances between the head teacher and the community. While the integrated approach significantly increased the operational utility of the study, the costs and complexity of conducting this kind of research in remote areas such as Northern Pakistan, may limit the opportunities for replicating these research approaches in similar projects.
Chapter 8. Evaluating the Impact of Water Supply Projects in Indonesia (Gillian Brown)
This chapter describes integrated research methods that were utilized in the evaluation of rural water supply projects implemented by the Government of Indonesia with support from various multilateral and bilateral agencies. The author describes the research methodologies and data analysis techniques used in these studies and presents preliminary findings of Phase 1 research, including analysis of gender issues. Advantages and disadvantages of using an integrated research approach are discussed, along with recommendations for future research.
The samples for the qualitative work were selected to ensure their statistical representativity and comparability with the survey research work. Procedures were used to ensure that the rankings of communities (in terms of the effectiveness of project implementation and other variables) on the basis of survey reports were consistent with the more detailed qualitative assessments of the same communities.
The integrated approach proved valuable in several ways. First, it demonstrated a statistical correlation between the level and quality of community participation and the effectiveness and sustainability of implementation, and it provided an assessment of the quality of project implementation. Second, it questioned the conventional assumption that community contributions to the cost of a project are a good indicator of willingness-to-pay. In fact, many families were pressured by community leaders to make contributions even if they did not wish to do so. Third, it showed the need for qualitative measures of the level of women’s participation, as very few women were actively involved in project management even though many indicated that they would have been interested in participating if they had been given the opportunity. Fourth, the qualitative methods provided an opportunity for rapid follow-up studies to assess unexpected outcomes, such as the fact that one of the best managed water projects happened to be the only one in which women were not involved in water management.
Chapter 9. Social Assessment of the Uzbekistan Water Supply, Sanitation, and Health Project (Ayse Kudat)
This chapter describes a social assessment process that was initiated in Uzbekistan as part of the preparation of a water supply, sanitation, and health project. The social assessments’ focus on social development, participation, and institutional issues required empirical research using a combination of qualitative and quantitative methodologies. The author presents the objectives of this research and the range of methodologies that were used, including the decision to establish a local social science network. The findings of the impact of the various studies are also discussed.
Social assessment had a number of benefits as part of the overall planning and implementation process. First, it helped focus on the complementarities between the water supply, sanitation, and health sectors and helped develop integrated approaches covering all three sectors. Second, the social assessments helped the World Bank and many national agencies to understand the socioeconomic structure and its effect on how projects should be selected, designed, and implemented. Third, social assessments helped understand the relationships between the Uzbeks and the Karakalpak regions and peoples, a major consideration in the design of projects that would affect both regions. Finally, the social assessments helped in conducting needs assessments for the more than one thousand villages in the region and ensured that the proposed programs would address the priority needs of each sub-region.
Chapter 10. Using Qualitative Methods to Strengthen Economic Analysis: Household Decision-Making on Malaria Prevention in Ethiopia (Julian Lampietti)
This study illustrates how qualitative methods can be used to enhance and complement the findings of quantitative survey methods. The chapter describes how qualitative methods are used at various stages of the research process to confirm the validity of quantitative results. The objective of the research was to measure the value people place on preventing malaria in themselves and members of their household in Tigray, a province in northern Ethiopia. While it is too early to discuss the actual valuation results, it is possible to highlight the interaction between qualitative and quantitative methods. An important methodological contribution was to show how qualitative methods can be used to provide the information and check the assumptions which will be built into the contingent valuation models. In particular, the qualitative studies were able to verify that the communities covered by the study fully understand the impacts of malaria on children and adults and were aware of the benefits and costs of currently available malaria prevention treatments.
Chapter 11 UNICEF’s Use of Multiple Methodologies: an Operational Context. (Mahesh Patel)
This chapter describes how UNICEF uses quantitative and qualitative methodologies to collect information for the purposes of monitoring, evaluating, and researching its programs and policies. It provides an overview of how UNICEF uses mixed-method research to evaluate and improve the effectiveness of its programs. The author concludes that while there are cases in which it may be sufficient to use a purely quantitative or purely qualitative approach, it is usually more cost effective to use different methods to study different aspects of the research question.
Chapter 12: Lessons Learned (Michael Bamberger)
This chapter brings together all of the lessons learned from the workshop
and the papers that were presented. It summarizes the benefits obtained
from integrated approaches and shows how these can be implemented at each
stage of the research process. The operational implications of integrated
approaches with respect to cost, timing, and coordination are discussed,
and some of the major challenges in using integrated approaches are identified.
*The author is Senior Sociologist in the Gender and Development Group
of the World Bank.
Return to top of page
Bamberger, Michael; Mark Blacken and Abeba Taddese. 1994. Gender Issues in Participation. Environment and Socially Sustainable Development Department. The World Bank.
Bamberger Michael and Jerry Lebo. 1999. Gender and Transport: A Rationale for Action. PREMNOTE No. 14 Poverty Reduction and Economic Management Network. The World Bank.
Barwell , Ian. Transport and the Village. 1996. World Bank Discussion Paper No. 344. The World Bank
Blackden, Mark and Elizabeth Morris-Hughes. 1993. Paradigm Postponed: Gender and Economic Adjustment in Sub-Saharan Africa. AFTHR Technical Note No. 13. Africa Human Resources Department.
Brewer, John and Albert Hunter. 1989. "Multimethod Research: A Synthesis of Styles." Sage Library of Social Research 175. Thousand Oaks. California: Sage Publications.
Bryceson, Deborah Fahy and John Howe. 1992. African Rural Households and Transport: Reducing the Burden on Women? IHE Working Paper IP-2. International Institute for Hydraulic and Environmental Engineering. Delft. The Netherlands.
Cernea, Michael and Ayse Kudat, eds. 1997. "Social Assessments for Better Development: Case Studies in Russia and Central Asia." Environmentally Sustainable Development Studies and Monographs Series 16. Washington, D.C.: The World Bank.
Datta, Lois-ellin. 1997. "Multimethod Evaluations: Using Case Studies Together with other Methods." In Eleanor Chelimsky and William Shadish, eds., Evaluation for the 21st Century. Thousand Oaks. California.: Sage Publications
Hentschel, Jesko. 1998. Distinguishing Between Types of Data and Methods of Collecting Them. Policy Research Working Paper 1914. Washington, D.C.: The World Bank.
Hentschel, Jesko. 1999. "Contextuality and Data Collection Methods: A Framework and Application to Health Service Utilization." The Journal of Development Studies Vol. 35, No. 4, April 1999, pp.64-94.
Kertzer, David and Tom Fricke. 1997. "Toward an Anthropological Demography." In Kertzer and Fricke, eds., Anthropological Demography: Towards a New Synthesis. University of Chicago Press.
Mangahas, Mahar. 1995. "Self-Rated Poverty in the Philippines 1981-1992." International Journal of Public Opinion Research 7:1.
Mangahas, Mahar. 1999. "Monitoring Philippine Poverty by Operational Social Indicators." Paper presented at the World Bank’s Poverty Reduction and Economic Management Week, July 14, 1999.
Miles, Matthew and Michael Huberman. 1984. Qualitative Data Analysis: A Sourcebook of New Methods. Thousand Oaks. California: Sage Publications.
Obermeyer, Carla Makhlouf. 1997 "Qualitative Methods: A Key to a Better Understanding of Demographic Behavior?" In Obermeyer, ed., Qualitative Methods in Population: A Symposium. Population and Development Review 23(4): 813-818.
Pradhan, Menno and Martin Ravallion. 1998. "Measuring Poverty Using Qualitative Perceptions of Welfare." Policy Research Working Paper No. 2011. Development Research Group, Poverty and Human Resources, The World Bank, Washington, D.C.
Ragin, C. 1994. Constructing Social Research: The Unity and Diversity of Method. Thousand Oaks, CA and London, England: Pine Forge Press.
Rao, Vijayendra. 1997. "Can Economics Mediate the Relationship Between Anthropology and Demography?" In Carla Makhlouf Obermeyer, ed., Qualitative Methods in Population: A Symposium. Population and Development Review 23(4): 813-818.
Ravallion, Martin and Michael Lokshin. 1999. "Subjective Economic Welfare." Policy Research Working Paper No. 2106. Development Research Group, Poverty and Human Resources, The World Bank, Washington, D.C.
Riffault, Helene. 1991. "How Poverty is Perceived." In Karlheinz Reif and Ronald Inglehart, eds., Eurobarometer: The Dynamics of Public Opinion. London: Macmillan Academic and Professional Limited.
Sanders, James. 1997. "Cluster Evaluation." In Eleanor Chelimsky and William Shadish, eds., Evaluation for the 21st Century. Thousand Oaks, California: Sage Publications.
World Bank. 1999. Poverty Reduction and the World Bank: Progress
in Fiscal 1998. Washington, D.C.: The World Bank.