5.3.4 Understand technical and ethical implications of effect size, sample, and selection prior to planning

Sample and effect size are key technical components of randomized studies. The effect size refers to the size or magnitude of the intervention’s effect. Calculating effect size should consider both statistical significance and policy significance. From a policy standpoint, programs would want to know the minimal effect size that would justify the adoption of the program. From a statistical standpoint, the sample size should be large enough to detect that minimal effect size. Miscalculations can lead to costly failures as the RCT may not be sensitive enough to answer research questions.

An effect size of… Is considered… …and it means that…
0.2 Modest The average member of the treatment group had a better outcome than the 58th percentile of the control group
0.5 Large The average member of the treatment group had a better outcome than the 69th percentile of the control group
0.8 Whoa…that’s a big effect size! The average member of the treatment group had a better outcome than the 79th percentile of the control group 1

Once the program determines the type of effect size it hopes to prove, researchers can determine a sample size. Sampling can happen at an individual or cluster level but sample sizes must be large enough to detect differences (assuming there are any) between treatment and control groups. In turn, sample size influences the total cost of the RCT—in general, a smaller sample size will lead to a less costly study.

RCTs are characterized by random assignment of control and treatment groups. This can create ethical dilemmas. Many organizations chose to waitlist or stagger services so that control groups receive interventions after the endline has been completed on the treatment group.

Practical Tips: IPA on Communicating with Young People, their Families, and Communities

Local stakeholders should understand why evaluators choose RCTs and how they work. It is imperative that young people, their families, and communities understand randomization or there is a risk of alienating potential beneficiaries. Below are steps you can take to enhance overall understanding of the process.

  • Explain the mechanics of randomization.
  • Transparency is key: public activities (such as pulling names out of a hat) can assist in transparency.
  • Randomizing at the school or group level may ease perceptions that particular individuals are denied services.
  • If randomized assignment is not possible, consider a comparison group.

 

  • 1. Excerpted from the presentation of Ms. Kelly Bidwell, Director of the Post-Conflict Recovery and Fragile States Initiative at Innovations for Poverty Action,” at the 2012 GYEOC.