9.2 Developing Outcomes and Theories of Change that Capture the Goals of the Program and Assumed Causal Pathways to those Outcomes are Critical to Building Evidence about a Program’s Impact

YEO practitioners and researchers need to build evidence about a program and its impact. That process needs to begin earlier than most assume. Effective impact evaluation begins during program design and continues throughout implementation. By building evidence as the program develops, practitioners can make course corrections, make effective decisions about program activities, and document impact. Evaluators should also acknowledge that youth have their own outcomes in mind when they participate in programs. Those should be surfaced in initial stages.

Outcomes reflect the new knowledge, skills, behavior, and/or circumstances that the program will help the youth to experience or acquire. Practitioners should consider if these outcomes represent meaningful quality of life improvements for youth and whether or not youth value these outcomes and want them to be true. These outcomes, like all indicators, should be SMART (see Box 9.2.1). Theories of change reflect the causal connection or pathway between activities and outcomes. They should reflect assumptions about how and why activities will project result in outcomes.

9.2.1 Checklist: UNC on Getting SMART about Theories of Change and Indicators

Mat Despard, Clinical Assistant Professor with the University of North Carolina, advocates for evaluation that starts with the end in mind–determining what the project (and youth served) hope to be different about youth after they participate. That might include new knowledge, new skills, and/or changed behaviors. The theory of change should reflect how the program will achieve the end. Despard discussed how to test theories of change related to financial capabilities.

When developing theories of change, have you considered the following:

✔Do activities represent necessary and sufficient conditions for change?

✔Are ideas reasonable? Do youth think they are reasonable?

✔Are ideas supported by and consistent with evidence from market research, community assessments, and/or studies with similar target groups?

Once theories of change are developed, choosing the right indicators will facilitate data collection.

The “SMART” convention is widely used in behavior change projects. To tell if your outcomes indicators measure up, ask yourself if they are:

✔Specific – clear and well defined, not vague

✔Measurable – we can find out if it happened

✔Achievable – it is something that is within our project’s power to influence

✔Relevant – it is something that is valued by and relates to the lives of youth

✔Timely – it is something that we can expect to happen within the project’s time frame