Learn how to collect your data and analyze it, figuring out what it means, so that you can use it to draw some conclusions about your work. Show
In previous sections of this chapter, we’ve discussed studying the issue, deciding on a research design, and creating an observational system for gathering information for your evaluation. Now it’s time to collect your data and analyze it – figuring out what it means – so that you can use it to draw some conclusions about your work. In this section, we’ll examine how to do just that. What do we mean by collecting data?Essentially, collecting data means putting your design for collecting information into operation. You’ve decided how you’re going to get information – whether by direct observation, interviews, surveys, experiments and testing, or other methods – and now you and/or other observers have to implement your plan. There’s a bit more to collecting data, however. If you are conducting observations, for example, you’ll have to define what you’re observing and arrange to make observations at the right times, so you actually observe what you need to. You’ll have to record the observations in appropriate ways and organize them so they’re optimally useful. Recording and organizing data may take different forms, depending on the kind of information you’re collecting. The way you collect your data should relate to how you’re planning to analyze and use it. Regardless of what method you decide to use, recording should be done concurrent with data collection if possible, or soon afterwards, so that nothing gets lost and memory doesn’t fade. Some of the things you might do with the information you collect include:
What do we mean by analyzing data?Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc. that can be found within it. That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. The point, in terms of your evaluation, is to get an accurate assessment in order to better understand your work and its effects on those you’re concerned with, or in order to better understand the overall situation. There are two kinds of data you’re apt to be working with, although not all evaluations will necessarily include both. Quantitative data refer to the information that is collected as, or can be translated into, numbers, which can then be displayed and analyzed mathematically. Qualitative data are collected as descriptions, anecdotes, opinions, quotes, interpretations, etc., and are generally either not able to be reduced to numbers, or are considered more valuable or informative if left as narratives. As you might expect, quantitative and qualitative information needs to be analyzed differently. Quantitative dataQuantitative data are typically collected directly as numbers. Some examples include:
Data can also be collected in forms other than numbers, and turned into quantitative data for analysis. Researchers can count the number of times an event is documented in interviews or records, for instance, or assign numbers to the levels of intensity of an observed event or behavior. For instance, community initiatives often want to document the amount and intensity of environmental changes they bring about – the new programs and policies that result from their efforts. Whether or not this kind of translation is necessary or useful depends on the nature of what you’re observing and on the kinds of questions your evaluation is meant to answer. Quantitative data is usually subjected to statistical procedures such as calculating the mean or average number of times an event or behavior occurs (per day, month, year). These operations, because numbers are “hard” data and not interpretation, can give definitive, or nearly definitive, answers to different questions. Various kinds of quantitative analysis can indicate changes in a dependent variable related to – frequency, duration, timing (when particular things happen), intensity, level, etc. They can allow you to compare those changes to one another, to changes in another variable, or to changes in another population. They might be able to tell you, at a particular degree of reliability, whether those changes are likely to have been caused by your intervention or program, or by another factor, known or unknown. And they can identify relationships among different variables, which may or may not mean that one causes another. Qualitative dataUnlike numbers or “hard data,” qualitative information tends to be “soft,” meaning it can’t always be reduced to something definite. That is in some ways a weakness, but it’s also a strength. A number may tell you how well a student did on a test; the look on her face after seeing her grade, however, may tell you even more about the effect of that result on her. That look can’t be translated to a number, nor can a teacher’s knowledge of that student’s history, progress, and experience, all of which go into the teacher’s interpretation of that look. And that interpretation may be far more valuable in helping that student succeed than knowing her grade or numerical score on the test. Qualitative data can sometimes be changed into numbers, usually by counting the number of times specific things occur in the course of observations or interviews, or by assigning numbers or ratings to dimensions (e.g., importance, satisfaction, ease of use).
Qualitative data can sometimes tell you things that quantitative data can’t. It may reveal why certain methods are working or not working, whether part of what you’re doing conflicts with participants’ culture, what participants see as important, etc. It may also show you patterns – in behavior, physical or social environment, or other factors – that the numbers in your quantitative data don’t, and occasionally even identify variables that researchers weren’t aware of.
Quantitative analysis is considered to be objective – without any human bias attached to it – because it depends on the comparison of numbers according to mathematical computations. Analysis of qualitative data is generally accomplished by methods more subjective – dependent on people’s opinions, knowledge, assumptions, and inferences (and therefore biases) – than that of quantitative data. The identification of patterns, the interpretation of people’s statements or other communication, the spotting of trends – all of these can be influenced by the way the researcher sees the world. Be aware, however, that quantitative analysis is influenced by a number of subjective factors as well. What the researcher chooses to measure, the accuracy of the observations, and the way the research is structured to ask only particular questions can all influence the results, as can the researcher’s understanding and interpretation of the subsequent analyses. Why should you collect and analyze data for your evaluation?Part of the answer here is that not every organization – particularly small community-based or non-governmental ones – will necessarily have extensive resources to conduct a formal evaluation. They may have to be content with less formal evaluations, which can still be extremely helpful in providing direction for a program or intervention. An informal evaluation will involve some data gathering and analysis. This data collection and sensemaking is critical to an initiative and its future success, and has a number of advantages.
When and by whom should data be collected and analyzed?As far as data collection goes, the “when” part of this question is relatively simple: data collection should start no later than when you begin your work – or before you begin in order to establish a baseline or starting point – and continue throughout. Ideally, you should collect data for a period of time before you start your program or intervention in order to determine if there are any trends in the data before the onset of the intervention. Additionally, in order to gauge your program’s longer-term effects, you should collect follow-up data for a period of time following the conclusion of the program. The timing of analysis can be looked at in at least two ways: One is that it’s best to analyze your information when you’ve collected all of it, so you can look at it as a whole. The other is that if you analyze it as you go along, you’ll be able to adjust your thinking about what information you actually need, and to adjust your program to respond to the information you’re getting. Which of these approaches you take depends on your research purposes. If you’re more concerned with a summative evaluation – finding out whether your approach was effective, you might be more inclined toward the first. If you’re oriented toward improvement – a formative evaluation – we recommend gathering information along the way. Both approaches are legitimate, but ongoing data collection and review can particularly lead to improvements in your work. The “who” question can be more complex. If you’re reasonably familiar with statistics and statistical procedures, and you have the resources in time, money, and personnel, it’s likely that you’ll do a somewhat formal study, using standard statistical tests. (There’s a great deal of software – both for sale and free or open-source – available to help you.) If that’s not the case, you have some choices:
Who should actually collect and analyze data also depends on the form of your evaluation. If you’re doing a participatory evaluation, much of the data collection - and analyzing - will be done by community members or program participants themselves. If you’re conducting an evaluation in which the observation is specialized, the data collectors may be staff members, professionals, highly trained volunteers, or others with specific skills or training (graduate students, for example). Analysis also could be accomplished by a participatory process. Even where complicated statistical procedures are necessary, participants and/or community members might be involved in sorting out what those results actually mean once the math is done and the results are in. Another way analysis can be accomplished is by professionals or other trained individuals, depending upon the nature of the data to be analyzed, the methods of analysis, and the level of sophistication aimed at in the conclusions. How do you collect and analyze data?Whether your evaluation includes formal or informal research procedures, you’ll still have to collect and analyze data, and there are some basic steps you can take to do so. Implement your measurement systemWe've previously discussed designing an observational system to gather information. Now it’s time to put that system in place.
Organize the data you’ve collectedHow you do this depends on what you’re planning to do with it, and on what you’re interested in.
Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriateWe’ve referred several times to statistical procedures that you can apply to quantitative data. If you have the right numbers, you can find out a great deal about whether your program is causing or contributing to change and improvement, what that change is, whether there are any expected or unexpected connections among variables, how your group compares to another you’re measuring, etc. There are other excellent possibilities for analysis besides statistical procedures, however. A few include:
Take note of any significant or interesting resultsDepending on the nature of your research, results may be statistically significant (the 95% or better certainty that we discussed earlier), or simply important or unusual. They may or may not be socially significant (i.e., large enough to solve the problem). There are a number of different kinds of results you might be looking for.
Interpret the resultsOnce you’ve organized your results and run them through whatever statistical or other analysis you’ve planned for, it’s time to figure out what they mean for your evaluation. Probably the most common question that evaluation research is directed toward is whether the program being evaluated works or makes a difference. In research terms, that often translates to “What were the effects of the independent variable (the program, intervention, etc.) on the dependent variable(s) (the behavior, conditions, or other factors it was meant to change)?” There are a number of possible answers to this question:
If your analysis gives you a clear indication that what you’re doing is accomplishing your purposes, interpretation is relatively simple: You should keep doing it, while trying out ways to make it even more effective, or while aiming at other related issues as well.
If your analysis shows that your program is ineffective or negative, however – or, for that matter, if a positive analysis leaves you wondering how to make your successful efforts still more successful – interpretation becomes more complex. Are you using an absolutely wrong approach? Are you using an approach that could be effective, but is poorly implement? Is there a particular contributing factor you’re failing to take into account? Are there barriers to success – of culture, experience, personal characteristics, systematic discrimination – present in the population from which participants are drawn? Are there particular components or elements you can change to make your program more effective, or should you start again from scratch? What should you address to make a good program better? Careful and insightful interpretation of your data may allow you to answer questions like these. You may be able to use correlations, for instance, to generate hypotheses about your results. If positive or negative changes in particular variables are consistently associated with positive or negative changes in other variables, the two may be connected. (The word “may” is important here. The two may be connected, but they may not, or both may be related to a third variable that you’re not aware of or that you consider trivial.) Such a connection can point the way toward a factor (e.g., access to support) that is causing the changes in both variables, and that must be addressed to make your program successful. Correlations may also indicate patterns in your data, or may lead to an unexpected way of looking at the issue you’re addressing. You can often use qualitative data to understand the meaning of an intervention, and people’s reactions to the results.The observation that participants are continually suffering from a variety of health problems may be traced, through qualitative data, to nutrition problems (due either to poverty or ignorance) or to lack of access to health services, or to cultural restrictions (some Muslim women may be unwilling – or unable because of family prohibition – to accept care and treatment from male doctors, for example). Once you have organized your data, both statistical results and anything that can’t be analyzed statistically need to be analyzed logically. This may not give you convincing information but it will almost undoubtedly give you some ideas to follow up on, and some indications of connections and avenues you might not yet have considered. It will also show you some additional results – people reacting differently than before to the program, for example. The numbers can tell you whether there is change, but they can’t always tell you what causes it or why (although they sometimes can), or why some people benefit while others don’t. Those are often matters for logical analysis, or critical thinking. Analyzing and interpreting the data you’ve collected brings you, in a sense, back to the beginning. You can use the information you’ve gained to adjust and improve your program or intervention, evaluate it again, and use that information to adjust and improve it further, for as long as it runs. You have to keep up the process to ensure that you’re doing the best work you can and encouraging changes in individuals, systems, and policies that make for a better and healthier community. You have to become a cultural detective to understand your initiative, and, in some ways, every evaluation is an anthropological study. In SummaryThe heart of evaluation research is gathering information about the program or intervention you’re evaluating and analyzing it to determine what it tells you about the effectiveness of what you’re doing, as well as about how you can maintain and improve that effectiveness. Collecting quantitative data – information expressed in numbers – and subjecting it to a visual inspection or formal statistical analysis can tell you whether your work is having the desired effect, and may be able to tell you why or why not as well. It can also highlight connections (correlations) among variables, and call attention to factors you may not have considered. Collecting and analyzing qualitative data – interviews, descriptions of environmental factors, or events, and circumstances – can provide insight into how participants experience the issue you’re addressing, what barriers and advantages they experience, and what you might change or add to improve what you do. Once you’ve gained the knowledge that your information provides, it’s time to start the process again. Use what you’ve learned to continue to evaluate what you do by collecting and analyzing data, and continually improve your program. Online Resources My Environmental Education Evaluation Resource Assistant (MEERA) provides extensive information on how to Analyze Data. Within their guide, they answer various questions such as: What type of analysis do I need?, How do I analyze qualitative/quantitative data?, and What software can I use to analyze qualitative/quantitative data? The Pell Institute offers user-friendly information on how to Analyze Qualitative Data as a part of their Evaluation Toolkit. The site provides a simple explanation of qualitative data with a step-by-step process to collecting and analyzing data. Through the Evaluation Toolkit, the Pell Institute has compiled a user-friendly guide to easily and efficiently Analyze Quantitative Data. In addition to explaining the basis of quantitative analysis, the site also provides information on data tabulation, descriptives, disaggregating data, and moderate and advanced analytical methods. CDC’s Analyzing Qualitative Data for Evaluation provides how-to guidance for analyzing qualitative data. CDC’s Analyzing Quantitative Data for Evaluation provides steps to planning and conducting quantitative analysis, as well as the advantages and disadvantages of using quantitative methods. Charts and Graphs to Communicate Research Findings, from the Model Systems Knowledge Translation Center (MSKTC), will provide guidance on which chart types are best suited for which types of data and for which purposes, shows examples of preferred practices and practical tips for each chart type, and provides cautions and examples of misuse and poor use of each chart type and how to make corrections. Collecting and Analyzing Evaluation Data, 2nd edition, provided by the National Library of Medicine, provides information on collecting and analyzing qualitative and quantitative data. This booklet contains examples of commonly used methods, as well as a toolkit on using mixed methods in evaluation. Compiled for the Adolescent and School Health sector of the CDC, Data Collection and Analysis Methods is an extensive list of articles pertaining to the collection of various forms of data including questionnaires, focus groups, observation, document analysis, and interviews. Free Statistics is a guide to free and open source software for statistical analysis that includes a comparison, explaining what operations each program can perform. Provided by the U.S. Department of Health and Human Services, this HRSA Toolkit offers advice on successfully collecting and analyzing data. An extensive list of both for collecting and analyzing data and on computerized disease registries is available. This Human Development Index Map is a valuable tool from Measure of America: A Project of the Social Science Research Council. It combines indicators in three fundamental areas - health, knowledge, and standard of living - into a single number that falls on a scale from 0 to 10, and is presented on an easy-to-navigate interactive map of the United States. Open Directory Project links to statistical software. Research Methods Knowledge Base is a comprehensive web-based textbook that provides useful, comprehensive, relatively simple explanations of how statistics work and how and when specific statistical operations are used and help to interpret data. Print Resources Bazeley, P. (2013). Qualitative data analysis: Practical strategies. New York, NY: SAGE. Brown, M. & Hale, K. (2014). Applied research methods in public & nonprofit organizations. Hoboken, NJ: Wiley. Creswell, J.W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches, 4th edition. New York, NY: SAGE. Guest, G.S., Namey, E.E., & Mitchell, M.L. (2012). Collecting qualitative data: A field manual for applied research. New York, NY: SAGE. Longest, K.C. (2014). Using Stata for quantitative analysis. New York, NY: SAGE. Miles, M.B., Huberman, A.M., & Saldana, J. (2013). Qualitative data analysis: A methods sourcebook. New York, NY: SAGE. Vogt, W.P., Vogt, E.R., Gardner, D.C., & Haeffele, L.M. (2014). Selecting the right analyses for your data: Quantitative, qualitative, and mixed methods. New York, NY: Guilford Press. Is a change process based on systematic collection of data?A change process based on the systematic collection of data and then the selection of a change action based on what the analyzed data indicate. An approach to change that seeks to identify the unique qualities and special strengths of an organization, which can then be built on to improve performance.
What are the various forces that make it necessary for organizations to change quizlet?What are the various forces that make it necessary for organizations to change? The six specific forces for change are the nature of the workforce, technology, economic shocks, competition, social trends, and world politics.
Which of the following is the correct order for the three steps included in the process for organizational change?The three steps of Lewin's model of planned change are unfreezing, change, and refreezing. Unfreezing is the process by which people become aware of the need for change.
Which is an example of the changing nature of the workforce?1) An example of change in the nature of the workforce is an increase in ________. Explanation: E) Almost every organization must adjust to a multicultural environment, demographic changes, immigration, and outsourcing. This represents the changing nature of the workforce.
|