5.2.2 Locate a proposed evaluation within a “spectrum” of scientific rigor
Over the years, members of the YEO field have been urged by funders and policy-makers to provide clear evidence for program effectiveness. Decision-makers look to rigorous, scientific studies to provide that evidence. However, many practitioners still have questions about how to build that evidence base in a way that works for their organization. Many assume that randomized control trials (RCTs) are the only way to achieve scientific rigor. This is not the case. Impact evaluation includes methods that span a spectrum of scientific rigor. Programs must decide the level of rigor they need to make decisions, promote their program, convince stakeholders, or contribute to learning. Alyna Wyatt of Genesis Analytics provided the following pointers to keep in mind when deciding how rigorous your evaluation needs to be.
Different types of studies can be employed in impact evaluation. Organizations can decide on which studies meet their needs, circumstances, and capacity. In general, the following guidelines provide an overview of which types of studies will provide more scientific rigor.
- Randomized studies are stronger at proving causation than non-randomized studies.
- Prospective studies are stronger than retrospective studies (baseline and endline).
- Large studies are stronger than small studies (sample sizes).
- Contemporaneous controls are stronger than historical ones.
- External controls (multiple-group designs) are stronger than self-controls (one group designs).
The following graphic provides a unique visual to understand the valid options for impact evaluation along the spectrum of rigor.