Each year billions of federal, state, and foundation dollars fund local educational initiatives designed to make a difference in the lives of children, adults and families. Practitioners concerned with the delivery of large and small scale initiatives devote countless hours of their time to the nuts and bolts of program delivery.  They provide direct services to participants, facilitate discussions with program partners, and address unanticipated issues as they arise. Meanwhile, uneasy thoughts are often lurking somewhere in the back of their minds - we need to evaluate this program, but how the heck are we going to do that? Program evaluation can seem daunting (and maybe even a waste of precious time, energy, and other resources). And yet, inevitably, evaluation questions start surfacing: How are things going? Are we making a difference? Are the participants benefiting? How can we demonstrate that this program should be re-funded?

In this article we introduce and demystify common evaluation terminology and present a series of strategies to help those delivering direct services and programs to begin down the road of program evaluation. The approaches described in this article will help those with little or no experience in program evaluation to:

• identify and document the outcomes, activities, and indicators to be evaluated, and
• assess the quantity and quality of the program’s achievements.

By way of illustration, we showcase how these strategies were used by a school-based program committed to enhancing the lives of young children and supporting the professional development of their educational providers.  This article demonstrates how these approaches can be used to assess education, human service, and/or health care goals, and discusses how they can be adapted for use with various initiatives.

Determine and Document Desired Outcomes, Activities, and Indicators

The determination of desired outcomes, activities, and indicators should take place during the planning stages of project development, such as during the grant writing process.  However, it is never too late to get strategic about program evaluation.  Regardless of how far you have traveled down the road toward implementation, for evaluation purposes it is essential to identify and document the program outcomes, activities, and indicators that will be evaluated.

Think of the desired outcomes as what you ultimately want the program to accomplish, the activities as what you will do to get there, and the indicators as the gauge of whether, and to what degree, you are making progress.

Outcomes should be consistent with what could reasonably be accomplished and not overly idealistic. Reasonable and realistic doesn’t mean you won’t strive for more, but in terms of carrying out an evaluation the more clearly defined and measurable the outcome, the better.  (See Patton (1997) for a detailed discussion on the development of outcomes.)  Your outcomes provide a foundation for all subsequent program implementation and evaluation activities, and each of the outcomes will need to be evaluated. While you and your program partners will undoubtedly seek to obtain a vision that is much bigger and beyond the scope of the grant, at this point, focus your outcomes on what can realistically be accomplished within the period of program funding.

The activities are the interventions that your program will provide in order to bring about the intended outcomes.  Programs offer all sorts of different activities to address their desired outcomes.  For the most part, program activities can be classified as any type of direct service or information that is provided to participants.

Indicators act as the gauge of whether, and to what degree, your program is making progress. Your program’s progress needs to be examined in two distinct ways:

1. the quantity and quality of the program activities you are delivering, (commonly referred to as process indicators), and
2. the quantity and quality of the outcomes that your program is achieving. (commonly referred to as outcome indicators).

Therefore, indicators must be developed to measure both of these types of program progress. Process indicators help track the progress that your program is making as you work toward achieving the desired outcomes. Process indicators often provide important feedback to program providers long before you can expect to see evidence that outcomes are being achieved. Outcome indicators provide the most compelling evidence that the program is making a difference in the lives of program participants (and other beneficiaries, such as the young children who are intended to benefit from the ECE Cares initiative.)

Now that you have an understanding of what outcomes, activities and indicators entail, the question becomes – how best to develop them for your particular program?  To effectively determine program outcomes, activities, and indicators, it is essential to engage a broad range of stakeholders in a purposeful discussion. It is important to convene the appropriate group of stakeholders (such as organizational leaders, program staff, program partners) and ask yourselves the following questions. 

Suggested Discussion Questions:

1. What are the desired outcomes of this program? What are the goals? What are we trying to accomplish within the next month/quarter/year(s)?
2. How will we get there? What activities will enable us to reach our outcomes? 
3. What will indicate to us that we are making progress toward the desired outcomes?

Be sure to allocate enough time to thoughtfully and thoroughly engage in discussion.  The time required depends upon the size of the group, the skills of the facilitator, the scope of the initiative, and the degree of collaboration and consensus developed to date.  Most groups will require between 3 and 5 hours in order to achieve consensus and many groups will need more time for discussion allocated over several days. While this process is not quick or easy, it empowers program stakeholders and enables them to come to consensus on the most important program outcomes, key activities, and realistic indicators. When done with integrity the discussion will have a dual effect: it will clarify the central purpose of the initiative and its corresponding outcomes, activities and indicators, and it will foster collaboration among stakeholders. (See Gajda (2004) for information about the assessment of collaboration and strategic alliances). Once you are relatively certain that group consensus has been reached, you should record your program outcomes, activities, and indicators in the form of a spreadsheet.

The collaborative discussion process has been used successfully by many groups, including the stakeholders involved in the Early Childhood Education Cares (ECE Cares) initiative in Colorado (funded through a U.S. Department of Education’s Safe Schools/Healthy Students Demonstration Grant). The ECE Cares stakeholder discussion permitted them to share ideas about the ECE Cares program, which seeks to deliver training and professional development to early childhood education teachers.  Ultimately, the discussion afforded them the opportunity to generate and document several essential outcomes, supporting activities, and corresponding indicators.  By way of example, the documentation of ECE Cares' first intended outcome and corresponding activities and indicators is provided in the following chart.

Early Childhood Education Cares (ECE Cares) Initiative
Outcome 1, Activities and Indicators

Notice that the number “1” refers to the first program outcome and the number following the decimal point refers to the specific activity.  Subsequently, the third number refers to a corresponding indicator.  By labeling outcomes, activities, and indicators with numbers you will be able to keep track of and refer to them more quickly as you expand the chart to include additional outcomes, activities and indicators.

• During your discussion, keep in mind the outcomes, activities and indicators that may already be expected of or available to you.  Scan relevant documents from the funding agency (such as the request for proposals) and your larger organization (such a school district’s strategic plan) to determine what outcomes are of greatest priority.  Ascertain what indicators of effectiveness are used by other agencies (governmental, non-profit, etc.) and determine if it makes sense for your program to use the same or similar indicators.

• Be sure that your outcomes, activities, and indicators are representative of the spirit of the grant for which you received funding. Too often, project leaders focus on outcomes that are not in alignment with the “absolute priorities” of the funding source. A mismatch between desired outcomes and your funder’s intentions will inhibit the relationship with the program officer and could jeopardize continued funding.

• The outcomes articulated in your grant proposal represented your “best thinking” at the time. Once you receive your grant award – and then again on a regular basis (at least annually) – it is important to revisit your stated outcomes, activities, and indicators to ensure that they are focused, pragmatic, and provide a current roadmap for your program.

• The outcomes, activities, and indicators that are developed and documented must be disseminated to all appropriate program staff and practitioners as soon as possible. They need to know where the program is headed and how they will know if things are moving in the right direction. As Hogue et al, (1995) states, “it is critical to move from problem driven to vision driven, from muddled roles and responsibilities to defined relationships and from activity driven to outcome focused” (p. 4).

Develop the Strategies to Assess the Quantity and Quality of the Program’s Achievements

Once program outcomes, activities, and indicators have been drafted, you need to revisit the indicators and consider whether they capture the different types of information that will be necessary to evaluate the program. From the ECE Cares example, we can see that the data tend to fall into two categories of information: quantity and quality. Typically, quantity measures are numerical descriptions of program activities and achievements, while quality measures often portray program activities and achievements through narrative descriptions. ECE Cares program staff/leaders are concerned with determining how many providers they train (numerical) and documenting the personal perspectives of early childhood providers regarding enhancements in their teaching ability and improvements in classroom climate (narrative).

Quantity

Quantity measures are used to evaluate both process and outcome indicators. Very early on in the first stage of program implementation it will be important to establish and document what tool(s) and strategies will be used to collect data that corresponds with each process and outcome indicator.  Quantity measures may examine how many people are being served and how often (process) – they may also examine change in rates of achievement, such as academic test scores and high school completion (outcome). The identified data collection tools and the person(s) responsible for gathering the documentation can be inserted into the chart that was introduced earlier. For example, the ECE Cares initiative documented their quantity measures, corresponding data collection tools, and persons responsible in the format illustrated below. 

Early Childhood Education Cares (ECE Cares) Initiative
Outcome 1, Activities, Quantity Indicators, Data Collection Tools and Person Responsible

As indicated in the spreadsheet, to collect and record the quantity of ECE Cares curriculum training (1.1.1) they used a Monthly Training Update Form.  It was important for ECE Cares project personnel to document that training participants included both school district and community early childhood providers.  In addition, they wanted to record the number of children in classroom settings with people trained in the ECE Cares curriculum. The Monthly Training Update Form was used by the ECE Cares trainer to compile training log attendance information and to report program quantity information on a monthly basis.  To modify this form for use with your own program, simply re-label the column headings to reflect the quantity measures that you want to capture and report on a regular basis.

Quality

Quality measures determine and document the effectiveness of the program’s activities and services. Evidence of a program’s quality can be gathered and reported through the use of narrative and/or numerical approaches. Mechanisms for gathering narrative information include individual or focus group interviews, open-ended survey questions, and observations of the program in action. The perspectives of program participants, program staff, and other stakeholders are often captured through interviews, surveys, and/or observations conducted by a program evaluator (Patton, 2002). Mechanisms for gathering information about a program’s quality include close-ended survey questions, such as those that ask participants to rate their level of satisfaction with the services and information provided. More powerful evidence is often generated when survey participants are asked to rate the degree to which they have gained new skills or information, or changed their behavior as a result of their involvement in the program. Likert scales are commonly used to elicit numerical ratings from survey respondents about the quality of a program.

As is the case with most initiatives, quality measure are important to ECE Cares.  Specifically, project leaders anticipate that providers will report through written surveys and in interviews that the curriculum training was effective and enhanced their ability to improve classroom climate.

Early Childhood Education Cares (ECE Cares) Initiative
Outcome 1, Activities, Quality Indicators, Data Collection Tools and Person Responsible

In order to devise the data collection instruments, such as interview questions and written surveys, it is important to revisit your program’s outcomes and indicators to develop questions that specifically address what you want to know and, therefore, what information you need to capture. There are several questions that could be asked of program staff and participants involved in almost any program to capture their perspectives on the effectiveness of the program. The questions posed to ECE Cares staff and participants were similar to those provided here.

Suggested Interview Questions for use with Program Personnel

1. From your perspective, in what ways has the program been effective or successful? Please share specific examples.
2. In what ways has the program made progress toward the desired outcomes and indicators?
3. From your perspective, what challenges or concerns have you encountered with this program?  Please describe.
4. What could be done to improve or enhance the program in the future?
5. Is there anything else that you would like to add at this time?

 Suggested Interview Questions for use with Program Participants

1. What did you gain as a result of participation in this program/service/activity?
2. How do you anticipate using the knowledge/skills that you gained as a result of your participation in this program/service/activity in the future?  Please describe.
3. What aspect of this program/service/activity did you find to be most valuable? Least valuable?
4. What suggestions do you have for improving this program/service/activity in the future?
5. Would you recommend this program/service/activity to others?  Please explain your response.
6. Is there anything else that you would like to add at this time?

Your program’s particular outcomes, activities, and indicators will provide specific language that you can use to tailor these interview questions. Another possibility is to pose the above questions as is, and then follow-up with interview probes or prompts that elicit responses regarding the various components of the program. For example, when asking about suggestions for improving the program, it is helpful to jog the interviewee’s memory by listing the various program activities (such as curriculum training and professional mentoring in the case of the ECE Cares initiative.)

Along with interviews, written surveys are another powerful means of collecting data about program quality.  The ECE Cares initiative utilized a written survey of early childhood education providers to determine the extent to which the training helped practitioners improve their skills in relation to Outcome 1.  The following survey provides a simple example of how to construct a written survey to gather data about the degree to which your program participants have gained new skills or information, or changed their behavior as a result of their involvement in your program.

ECE Cares Initiative: Example Written Survey Format

The data that you gather about your program’s quantity and quality can be used to make decisions about program implementation and development. Quantity and quality measures enable you to make informed, mid-course adjustments in program implementation and help you to accurately showcase the effectiveness of your program’s services and activities.  It’s essential to utilize quantity and quality data collection tools from the outset of the project. If a data collection system is not developed during a grant application process, or prior to or immediately following project implementation, it will be nearly impossible to generate with accuracy all the data that you will need to determine whether you have made a difference and to satisfy reporting requirements.  

• The quantity and quality data that you collect will be invaluable in fulfilling multiple reporting requirements (eg. bi-annual or annual updates to the funder and/or program sponsor, summaries for other key stakeholders such as boards of directors). 

• To help keep the process of tracking the quantity and quality data manageable, be sure to request information from all project personnel in the same basic format, using the same fundamental process throughout the duration of the project.  It’s important to note that some project activities might require that additional data collection categories be added

• Collect data early and often.  If you articulate who will collect what types of evaluation data, the tools to use, and where it can be found, there won’t be a mad scramble to collect information and generate summaries when performance reports come due.

• Use technology to the greatest extent possible.   Using Microsoft Excel or other spreadsheet software to create documents will make it easier to modify the format as needed.  Consider sending written surveys via email attachment or put the survey right into the body of your message.  If you and your survey respondents have the capacity, design a website that allows people to take surveys on-line -- you’ll save paper and time.

Conclusion

As program providers, you devote an incredible amount of time to the implementation of services and activities.  You have a vested interest in how things are going, whether the program is making a difference, and if the participants have experienced success.  In addition, those concerned with program delivery are increasingly being expected to carry out  rigorous program evaluations, yet many project directors do not know where to start and don’t have access to the user-friendly strategies and tools that they need. The approaches presented in this paper are appropriate for those just beginning down the road of program evaluation and are not intended to be all-inclusive or an evaluation panacea – program evaluation is a highly complex and exceptionally challenging endeavor. One thing is clear - if you are able to articulate program outcomes, activities, and indicators and have the strategies in place to assess the quantity and quality of the program’s achievements, it is all the more likely  that you will make a positive difference in the lives of those you serve. If you are thinking about how to evaluation your program, the strategies and tools described in this article can get you started on the right path.

References

Bailey, D. & Koney, K. (2000). Strategic Alliances Among Health and Human Services Organizations: From Affiliations to Consolidations [ABRIDGED], Sage Publications: Thousand Oaks

Gajda, R., (2004, Spring). Utilizing Collaboration Theory to Evaluate Strategic Alliances. American Journal of Evaluation. 25, 65-77.

Hogue, T., Perkins, D., Clark, R., Bergstrum, A,. Slinkski, M., & Associates. (1995). Collaboration framework: Addressing community capacity. Columbus, OH: National Network for Collaboration.

Patton, M. (1997). Utilization-Focused Evaluation: The New Century Text. Thousand Oaks: Sage Publications.

Patton, M. (2002). Qualitative Research and Evaluation Methods. Thousand Oaks: Sage Publications.