Formative Evaluation

This week we will go through formative evaluation. It can be used for a program that has already been running as well as a program that is being created. It's more often used to evaluate the latter. Beyer (1995) defines it as "evaluating or assessing a product while that product is in the process of being created and shaped." Sometime it is called "improvement-oriented evaluation."Rather than making a judgement or determining the impact of the program (summative), formative evaluation enables ways in which a program can be improved. There is a metaphor for formative and summative statement: "when the cooks taste the soup, that's formative; when the guests taste the soup, that's summative."

Beyer discusses primarily on using formative evaluation for a product that is being developed before it is put into a regular use. For example, take a look at Apple's iPhone product with an antenna issue. If the formative evaluation was done well, Apple would have found out and fixed it before it went on sale (or before a regular use). Beyer offers four stages of which formative evaluation of a product or a program needs to be occurred:

1. Design
2. Prototype 
3. Pilot
4. Field test

What happens when a program has already been running or a product has gone on sale? In what way that formative evaluation can be used? Is it too late to evaluate formatively? If a program has already been running and a product has already gone on sale, why bother with formative evaluation? Some funders would just want to see the final outcomes of the program; how would you convince them that formative evaluation is needed?

Formative evaluation can also be used to evaluate a program while it is already in operation. According to Michael Patton, formative evaluation asks the following questions:

1. What are the program's strengths and weaknesses? 
2. To what extent are participants progressing toward the desired outcomes?
3. Which types of participants make good progress and which types aren't doing so well?
4. What kind of implementation problems have emerged and how are they being addressed?
5. What's happening that wasn't expected?
6. How are staff and clients interacting? 
7. What are staff and participant perceptions of the program? 
8. What do they like? dislike? want to change? 
9. What are perceptions of the program's culture and climate?  
10. How are funds being used compared to initial expectations? 
11. How is the program's external environment affecting internal operations? 
12. Where can efficiencies be realized? 
13. What new ideas are emerging that can be tried out and tested?

What are your data sources? Who, what, and where are the best sources of this information? Beyer mentioned 2 things: people and well-established standards. Who are the people? 

1. Experts 
2. Users (e.g., intended beneficiaries) 
3. Stakeholders (e.g., providers of services, teachers, parents, community members etc.) 

Class activities

Below are tasks for you to do as a group: 

1. With your project team members, identify experts related to your project, locally or internationally. They can be anyone who can provide you feedback to the program activities and framework. How many of them and in what sub-areas? 
2. Once identified, think about what kind of things you would like the experts to help with? List  all the things you would like them to help with. 
3. Identify users of your project. Who are your primary and secondary users? 
4. Identify stakeholders of your project. Who would you like to include and why? 

Tasks for you to do this week

1. Schedule a Skype call with your primary stakeholders (Jamie and Natalie) 
2. Prepare for questions to ask both of them 
3. Start collecting all the information about the program you are evaluating--everything about the program because it allows you to have a comprehensive understanding of the whole program. Don't wait till you know all the things you are asked to evaluate. 
4. Create an online shareable folder that your team members can upload the documents 
5. Always look into the government policy documents related to your program as it gives you a big picture from the national level. It is required that you know what's going on with the national policy on your program. Here is the website to the Cambodian Ministry of Education: http://www.moeys.gov.kh/en/policies-and-strategies.html  
6. Start writing up program description. For example, if you are working on your preschool program, then write all about the program background and overall activities.