Aug 31st: Plan and Essential Information for your Evaluation

After you have selected your evaluation topic, here are steps that could help you plan for your evaluation, especially for you to complete your first part of the assignment:

1. Schedule a meeting with your team members to plan for your evaluation. Kelly and I can also join your meeting to help focus your topic. With your team members, start brainstorming all kinds of questions you want to ask the primary stakeholders (those who primarily manage the program on day to day basis), and write them down. Questions should be as detailed as possible, covering background and history of the program, program's mission or goals, day-to-day activities, budget, staff involved with program operation, number of beneficiaries it serves, the purpose of this evaluation, and the intended uses of the results from this evaluation etc.     This needs to be done this week by Friday 9/4th. 
2. Part of your initial contact, send a brief note (via email) of the program you are evaluating to the best of your understanding what is being evaluated, to your primary stakeholders (for CFC project send to: Jamie and Natalie) to obtain their feedback. This needs to be short and right to the point. It needs to be done by Friday 9/4th. 
3. Once you receive their feedback, the next step is to schedule a meeting (via Skype) with the primary stakeholders (for CFC project: Jamie and Natalie), again, to explore the program being evaluated more in details. Make sure that your questions allow you to understand their needs and their intended uses of the program results. It would also be important to ask them who should be included in your sample (data to be collected). Right now, their schedule is open on Tuesday September 8th at 9:30am till noon. 
4. Depending on who they refer you to talk to, please then schedule a Skype call with them. Those may be staff on the ground including Savy Ung, CFC's superintendent and Christin Spoolstra, CFC's deputy country director.  
5. Set up your evaluation timeline--when to finish what. 
6. Discuss individual team member's roles--who is doing what? It is also helpful to assign one team member as a contact person to primary stakeholders and Kelly and I. 
7. Conduct literature search related to your evaluation topic including relevant government policy, similar programs run by other NGOs or institutions, as well as empirical findings related to the program.  
8. Begin drafting your IRB Human Subject Application. You have almost 4 weeks to do this. Make sure it is ready to be submitted by September 28th. 
9. The next step would be to meet up with your team members to work on writing up the first part of your evaluation.   

Questions for discussion

1. Think of your project, in what way that it can be a needs assessment?
2. You are planning your talk with primary stakeholders (e.g., those who sponsor the needs assessment), think of some questions to ask that show your understanding of needs assessment project you are undertaking.
3. Think about who should be included as your stakeholders in your needs assessment project.

Due next week, September 7th 2015 

IRB package will be created on the irbnet.org website. This package is the registration of your project on the website that includes project information, IRB certificate upload, and names of all your team members added to the package.

Task division of your team members should have already be defined. Please be prepared to share everyone in class.

Your team will also be asked to briefly described the progress of the project. 

Welcome to CIE 402: Development and Evaluation of International Educational Projects

Welcome to CIE402!

There are a several things that we will go through in this first class:

1. Introduction
2. Syllabus
3. Introduction to Caring for Cambodia (CFC) project at Lehigh with a video 
4. Choosing an evaluation topic of your interest
5. Selecting a team members for your evaluation group 
6. Cambodia trip: November 4th-14th  
7. Expectation of the report, style and contents 
8. Example of evaluation reports from past semesters (available in coursesite) 
9. IRB certificate and submission of your project  
10. IRB proposal submission 
11. Resources for evaluation (conferences, journals, etc.) 
12. Evaluation topics 
13. Overview of program evaluation: definition, purposes, types, standards etc.  
14. Basic facts about CFC
15. Contact information of CFC primary stakeholders that you can reach out to negotiate your project 

7. Expectation of the report, style and contents

Let's talk about item #7: Expectation of the report, style and contents. I created this template for the consistency of use in this particular class. Please use this template file for your evaluation report. Word file created on your own will not be accepted.  

Click here for the link to the template

8. Example of evaluation reports from past semesters 

I would also like to share with you a sample evaluation paper that was written in a previous class on student council evaluation at the Caring for Cambodia schools. This is the link the paper (available in coursesite too).  

9. IRB certificate and submission of your project  

Since students are required to conduct data collection and data analysis as part of their evaluation project, obtaining an IRB certificate is required. This assignment accounts for 5 points of the total grade. For those of you who have already completed the training and received the certificate, the grade will be automatically applied to your overall grade. To take the training and certificate, please go to this website: http://phrp.nihtraining.com/ . Once you received the certificate, please save it in a PDF file, and email my TA, Kelly Grace (krg314@lehigh.edu).   

10. IRB proposal submission

Students are required to submit their evaluation proposal for IRB approval for their evaluation to take place. Please see the dateline of this proposal submission in your syllabus. Here is a sample of IRB proposal submitted by a group of students in previous class: link to the file. 

11. Resources for evaluation

American Evaluation Association is perhaps one of the largest, well recognized organizations in the United States and in the world with about 7700 members from within the US and over 60 other countries, as of 2014. It's mission is to "improve evaluation practices and methods, increase evaluation use, promote evaluation as a profession, and support the contribution of evaluation to the generation of theory and knowledge about effective human action." An article by Donaldson & Christie (2006) lists all of the major associations worldwide. It publishes a well-known journal in this field called, American Journal of Evaluation. Here is brief introduction to the journal: 

"The American Journal of Evaluation (AJE) explores decisions and challenges related to conceptualizing, designing and conducting evaluations. Four times/year it offers original, peer-reviewed, articles about the methods, theory, ethics, politics, and practice of evaluation. AJE features broad, multidisciplinary perspectives on issues in evaluation relevant to education, public administration, behavioral sciences, human services, health sciences, sociology, criminology and other disciplines and professional practice fields."

This association holds an annual conference where both academics and practitioners come together to share their results and lessons learned. This year's conference will be held on Nov 9-14 in Chicago. It's its 29th annual conference. This year's conference theme is: Exemplary Evaluations in a Multicultural World.  

What can you do with evaluation skills?   

Well you can be an evaluator or evaluation specialist, consultant, program analyst, research analyst, based on your area of specialty. You can find a whole job listing in the American Evaluation Association's website. Let's pick one and see what they are looking for. 

This is an excellent article that talks about "Emerging Career opportunities in the Transdiscipline of Evaluation Science" by Donaldson & Christie (2006). 

12. Evaluation Topics 

Students will work in a group of three to carry out an evaluation project below. 

CFC Project 1: Teachers’ perceptions toward database project: A needs assessment 

Lehigh’s CSB (Computer Science and Business) faculty and students have developed academic and health database system for CFC schools. The database system allows the teachers to enter students information that include their academic activities and performance as well as their health record. The system has yet to be implemented, but training in how to use the database has been provided to the teachers and CFC. Little is known how receptive the teachers are toward this database system participation and implementation. The purpose of this evaluation is to assess the needs of this program by interviewing teachers and CFC staff their perspectives in implementing the database program.

CFC Project 2: Preschool Program Evaluation: Formative and Summative Evaluation

CFC has been implementing preschool program since 2008. Two evaluation studies have been conducted previously, one in 2010 and the other one in 2012. For this evaluation, we aim at examining the progress of the program, strengths and challenges facing program operation (formative evaluation), as well as the impact of the program on students’ school attendance and performance (summative evaluation). In this evaluation, we plan to draw the data from school record of students who participated in the program and those who did not, in terms of their school attendance and performance as well as teacher evaluation on the two groups of students. So this evaluation utilizes both formative and summative methods.

CFC Project 3: Needs Assessment of Community Contributions to Education

This evaluation seeks to assess community’s willingness or interests to contribute to schools in terms of materials and finance in order to sustain the schools and all the existing programs created by CFC. The evaluation will seek to gain perspectives from relevant stakeholders within the school community, namely, students, parents, teachers, principals, village and commune chiefs, and relevant government officials. In addition, the review of existing programs related to community contributions in Cambodia or elsewhere will need to be included in order to understand best practices of these similar programs that could be adapted for the CFC schools. Previous research by UNESCO in five post-conflict societies found that “the most community involvement involved the provision of material and financial resources, primarily in the form of providing land for building school or classroom venues, contributing materials for school rehabilitation and maintenance. Some communities also contributed human resources through the selection of teachers or involvement in governance structures such as Parent Teacher Association (PTAs).” However, the research also showed that trust is important in determining whether the community will contribute. In other words, trust is a prerequisite to gaining community contributions. With this idea in mind, this evaluation will also seeks to understand:

1. how the community perceives the role of CFC in its educational provision to the children in their community. 
2. how the community perceives its role to be in its educational contributions. In other words, how do they think their roles in the education that their children receive from CFC schools.
3. what are the existing community resources, tangible and intangible, that readily available for the schools to use
4. finally if the community is willing or interested in contributing to education. 

CFC Project 4: English Language Program 

We have spent the last three years working on the English Language Program in CFC primary schools. Through research studies and evaluations we determined that the ELP needed to be changed, as students were struggling in all areas of language acquisition. We are currently in the second pilot stage of a newly written ELP that incorporates interdisciplinary content linked to traditional classroom learning objectives in order to promote transfer of learning, as well as provide students with supplementary exposure to traditional classroom content. The ELP employs a student-centered, project-based learning approach, and initial findings from a short pilot in the spring proved very promising.

The first full-year pilot of the new program will take place this fall, and we need to establish a clear plan for monitoring and evaluating the program throughout the year. The group working on this project will establish a baseline assessment of students' English Language proficiency, and work to frame both formative assessments that will monitor progress throughout the year, as well as and end-of-year summative assessment to help teachers and administrators understand the strengths and weaknesses of the program.
Other Project 5
Other Project 6 

13. Overview of program evaluation

There are a few things that you need to do to get you started with an evaluation project:

1. First of all know the purpose of your evaluation or the intended use of the evaluation: to improve or to determine the outcomes? To improve the program, we use "formative evaluation" as it helps an evaluator understand the process of the program and how it runs. In other words, it aims to find out what works and what does not work and what to do to improve the program. To determine the program's outcomes, we use "summative evaluation" as it helps an evaluator determine if the program impacts targeted stakeholders with the intended benefits. This is such as interesting analogy to better understand the two inquiries: "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative" (Professor Bob Stake, U. of Illinois, Urbana-Champaign."
2. Then, you need to identify relevant stakeholders (U1), starting from primary stakeholders, those who are closely managing the program (e.g., director, program manager, program officers etc.). Primary conversation will need to start taking place with these above mentioned people in order to understand their needs for program evaluation: do they want to improve the program or examine the impact of the program, or both? Different stakeholders may have different expectations and so yourself. What you plan based on program description you saw on the ad may be different once you start communicating with your stakeholders. This is an important step to be taken to ensure that the evaluation does what everyone or at least the majority of the stakeholders expected. 
3. This next step is very important: to obtain formal agreements with your stakeholders after your negotiations to undertake the evaluation project (P2: Formal Agreements). The goals have to be clear at this point--although things can change later as the evaluation takes place, the evaluator will need to keep communicating with the stakeholders to modify the agreement terms. This will help with any political conflict from various interest groups (F2: Political Viability or P7: Conflict of Interest). "Any possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted (The Joint Committee on Standards for Educational Evaluation, 1994, p. 71)." In addition, the agreements should also document if the procedures to be taken during the evaluation (ongoing communication, data collection, or information gathering) is practical and feasible (F1: Practical Procedures). For example, the evaluator needs to ensure that qualified personnel are available to help out during the whole evaluation process. 
4. Next step is to start gathering information and documents (A1: Program Documentation), examine the context of the program (A2: Context Analysis), reliability and validity of the program information (A5: Valid Information & A6: Reliable Information). At this step, the evaluator needs to be in constant contact with personnel assigned to assist with the program evaluation to ensure that all the information obtained is correctly reflected the program's theory or activities.   
5. Once information has been obtained, the evaluator is now ready to design their evaluation. He or she can start planning on sampling (participant selection), procedures of data collection, designing questionnaire or developing interview questions for corresponding stakeholders, and finally collecting the data. Here the evaluator needs to ensure that the procedures are practical as mentioned above in F1: Practical Procedures. The evaluator needs to discuss with the primary stakeholders in the process of identifying stakeholders that they deem important to include in the evaluation. This process is done to ensure that your data are completely collected from various stakeholders identified by you and by the primary stakeholders. During data collection, the evaluator needs to ensure that participants' rights are respected and protected (P3: Rights of Human Subjects." If participants do not want to answer any questions due to the information sensitivity, the evaluator should respect their wishes and move on the next questions. 
6. The next step is to analyze the data collected and put together a complete report including recommendations based on the findings. Results dissemination may be expected via printed materials or oral presentation by the evaluator. This is when the evaluator needs to prepare a racing car in case the sponsoring organization finds the results disturbing :-). Findings should be disclosed to those impacted by the evaluation (P6: Disclosure of Findings). The Joint Committee noted that "the formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation, and any others with expressed legal rights to receive the results (p. 109).  
7. At the same time, the evaluator needs to be knowledgeable in statistical analysis, at least at the basic levels. The evaluator needs to also be equipped with knowledge in interview response analysis (qualitative method). 
8. The next step, the evaluator should consider submitting their evaluation report for a national or international conference, pending on approval from the sponsoring organization.  

14. Basic Facts of CFC

Below are basic facts of CFC written by Alyssa Buccella, Busra Ozturk and Amanda Pritt of Comparative and International Education, College of Education, Lehigh University.

• Eleven years ago, Jamie Amelio founded Caring for Cambodia (CFC), a non-profit organization that provides quality education for 6,400 students across 21 schools in Siem Reap.
• Caring for Cambodia provides teacher training, hygienic classrooms, and essentials for students’ health and welfare such as meals, hygiene kits, uniforms, school supplies, etc. 
• According to national estimates, only 28% of students graduate from high school across Cambodia. CFC is dedicated determining how they could assist in making a difference in the lives of these children. 
• With an estimated 44% of children facing chronic malnutrition (CDHS, 2010), ensuring that students are fed twice a day serves as both an incentive to attend school and a means to focus in class with a full stomach. 
• Further, many students weren’t able to attend school due to lack of transportation. In 2012 alone an extra 197 bicycles were distributed to students, making their commute easier. 
• Financial restraints pose an immense barrier to education in Cambodia, but with a free education, meals, uniforms, and school supplies, CFC has greatly lightened the burden from families, encouraging them to send their children to CFC schools. 
• CFC has proven to have the lowest dropout rates in all of Cambodia (less than 2% for its primary level compared to the national average of more than 10%), consistently achieving successful results over the past decade. 
• Total revenue and support in 2012 amounted to $1.26 million, and expenses totaled $1.19 million. Teacher training alone costs about $56,000 in 2012, but well-trained and incentivized teachers are perhaps one of the most important factors in ensuring a high-quality education for the students in CFC schools. 
• When taking science and math curriculum into account, it is important to note that CFC operates within standards set by the government. Still, they have yearly goals and by working closely with child-friendly schools, they aim to supplement the national curriculum in order to provide students with a quality education. 

15. Contact information of CFC primary stakeholders that you can reach out to negotiate your project 

Here are some key contacts at the CFC schools for your evaluation:

Mr. Savy Ung, Superintendent. Savy has been with CFC since the start of the organization, for about 13 years. Savy is managing all schools and overseeing all the programs being run at all the CFC supported schools, on the ground in Siem Reap province. Savy could provide background information of most of the programs including day-to-day activities, the programs' mission, and his intention of using the results of all of the program evaluations.

Ms. Christin Spoolstra, our new Deputy Country Director. Originally from Indiana and a graduate of Albion College in Michigan, Christin has spent the last three years working for the Peace Corps in Cambodia. Initially, she worked in the English Teaching and Teacher Training Project in Svay Rieng Province, and then, most recently, she was the Volunteer Coordinator in Phnom Penh. Christin's knowledge of the Khmer language, her familiarity with the Cambodian culture, and her passion for education will serve her well in her new role with CFC. As Deputy Country Director, Christin will be Country Director Ung Savy's right-hand woman and the central point of communication. Her primary responsiblilities will include helping to establish strategic direction and execution of key CFC programs, working with the Program Managers to strengthen current programs and implement new community programs, development of Human Resource policies and procedures, and training of local staff with communication and PR skills.

Ms. Natalie Bastow, Chief Operating Officer. Natalie manages all aspects of CFC including program initiatives, international network connections, fund raising activities, and decision making. Natalie has been a great resource in linking Lehigh with CFC on the ground and worldwide. She could provide larger picture information related to CFC visions and how evaluations would benefit CFC in a long run, ensuring all aspects of sustainability of CFC.

Ms. Lydia Breckon, Development Director. Lydia manages fund raising activities corporate relationships / opportunities and grant applications. As one of the primary stakeholders, Lydia plays an essential role in evaluation topic selection as it provides research-based evidence that she could use to support her potential grant proposals.