Summative or Impact Evaluation

This week are discussing summative or impact evaluation. Impact evaluation seeks to determine if a program has an impact based on the intended outcomes. To measure the intended outcomes, it is important that the evaluator understand how the program operates or how the program conceptualizes its framework. And this is called "Theories of Change" or "Program Theory." As Gertlet et al. noted, "A theory of change is a key underpinning of any impact evaluation, given the cause-and-effect focus of the research. As one of the first steps in the evaluation designs, a theory of change can help specify the research questions" (p.22). It is the evaluator's task to identify the program theory or the theory of change.

A well operating program may not have a specific, explicitly stated theory, although they know what they are trying to achieve. Helping program staff or primary stakeholders to understand a clearly stated program operation with explicit operating framework will be helpful for the organization itself in terms of having a valid documentation with rigorous program theory necessary for their fund raising purposes, and for the evaluator in having a clearly defined program goals/objectives that they can base on to help them develop their evaluation criteria to determine the outcomes of the program being evaluated. Program theory development/identification can be done with discussion with program staff and other primary stakeholders.

There is an excellent article on how to establish a program theory published in the American Journal of Evaluation by Frans L. Leeuw (2003) entitled, "Reconstructing program theories: Methods available and problems to be solved." Leeuw offered three approaches to help uncover the mystery of a program theory, making the theory more explicit.

First approach is using the "policy-scientific" method which involves reviews of empirical as well as program documentation, evidence (pay attention to 'our goal is to improve...' or 'we argue that...' and a propositional statement such as "if, ...., then, ...."  The second approach is a strategic assessment method which refers to the process of uncovering all possible assumptions about the program in which group dialog is central to this approach. The third approach is "elicitation" method that involves a mental mapping process (a method used in cognitive psychology) where participants (stakeholders) are asked to provide their thoughts on the program model (or logic of the program) and then compare it to evidence from scientific organization studies.

Leeuw mentioned that the first approach is "best suited for ex post evaluations (after the fact) of programs and policies back by documentary evidence (i.e., often public policies), while the strategic assessment and elicitation approach appear to be more relevant for ex ante evaluations under other conditions" (p. 16). Leeuw's article offers step by step methods to help you digest program theory efficiently. This is the citation to the article:

Leeuw, F. L. (2003). Reconstructing program theories: methods available and problems to be solved. American Journal of Evaluation, 24(1), 5-20.

And this is the link to the article (you may need to be connected to the campus network to access to this article). Worth mentioned is the Figure provided in the paper which illustrates how you can use policy-scientific method to establish a program theory:

When a program does not have any explicit theory stated, the use of needs assessment is helpful. This can be done in a backward process where the evaluator asks the program staff and primary stakeholders about why the program is needed--why this program exists in the first place and who needed it, for what purpose, and outcomes to be accomplished. Discussion with primary stakeholders to gain a better understanding of the needs and theory that it generates is necessary. This needs assessment can also be useful when you describe the program--in your program description where you will write the background, mission, and activities of the program.

In addition to using needs assessment as theory development and identification, needs assessment is also relevant for both process and outcome evaluation. For process evaluation, the evaluator is interested in knowing if what was needed got implemented, whether those who needed it received the service, and whether the program staff has the capacity to responsible for the need to be implemented. In addition, need assessment enables the evaluator to understand if people need the offering services or if they think the services are relevant to their life based on the context in which they are situated. The response allows the evaluator to make recommendations for program improvement. For example, Weaker Student Program implemented by CFC in 2013, received less attention and participation by students and parents even though the teachers reminded them--the parents-- possible improvement of their children's participation in the program. In this case, parents may not see the real need of this program for their children. No evidence was gathered from the parents at the time--only students' data were collected. Needs assessment from parents via face to face interviews should be helpful in understanding program theory and actual need perceived by the beneficiaries. For outcome evaluation, the evaluator is interested in knowing if the needs impact participants' economic, social, psychological, or academic functioning. In other words, is there a correlation between the stated needs and all of these functional outcomes.

Need assessment can be done using different strategies including:

Personal observations on the resources and the needs in the community
Social indicators of need (via national survey data)
Community surveys of need--including attitudinal surveys
Service availability in the areas (duplicity check)
Key informants interviews and information (usually village chiefs or school principals)
Focus groups
Community forums--via parent-school meeting day or occasional events happened on the campus

For your own program evaluation, I also recommend doing organizational assessment by discussing with program staff. Organizational assessment allows the evaluator to understand:

1. Location and facilities of the program: infrastructure in different locations and staff delivery of the services.
2. Program personnel structure: who is doing what, and who make certain decisions. Here a diagram of the organizational structure is useful to be reported in your evaluation.
3. Values and interaction quality within the organization: interactions between staff members and between their clients.
4. Qualifications of the personnel
5. Frequency of meeting and communication
6. Staff training opportunities
7. Ongoing program monitoring: how do they know the program is running smoothly?

Gertlet et al. also discuss "the results chain" as part of program theory or theory of change. The results chain outlines 5 elements that serve as a map of the program to help evaluator design their evaluation and to select measures (or performance indicators) corresponding to the program activities. Below is the results chain taken from the Gertlet et al.'s chapter 2.

I suggest that you model your program based on this results chain flow chart.

Questions for Discussion

As an evaluator, how do you decide if different stakeholders have different theories of a given program?

Tasks for this week

1. Begin drafting your IRB application that includes filling out  (1) HUMAN SUBJECTS APPLICATION (New Studies) MS Word03.doc and (2) Consent Form Template (11-21-14).docx.

2. Try to identify program theory or results chain of the program you are evaluating. Understanding a program theory or results chain helps you develop appropriate measures to evaluate the outcomes. Leeuw's article helps you develop a program theory. In addition, try to put the program theory or results chain visually by drawing a flow chart of it. That will be also used in your program review and description section of your evaluation report.

3. You have spoken with Natalie and Lydia about your project last week. With their input, please draft the summary of your project and questions for clarifications and send to Jamie, CFC founder, for her input. The purpose is to make sure that all the primary stakeholders have been consulted and to make sure that everyone agrees on your project before you proceed. This has to be done ASAP.  

4. By now, you should have a complete understanding of the program operations and your project design.

5. Make sure that everyone of your team members has a shared understanding of the project.