5.4.1 Match the right question to the right method

Good evaluation starts by formulating the right questions. Different evaluators take different approaches to formulating questions. Some evaluators and program designers focus on the effectiveness of the program in achieving its objectives, as would be the case for an impact evaluation. Other evaluators and program designers have a broader learning agenda that seeks to gather information above and beyond, “does the program work?” Those questions may address nuanced aspects of learning, product use, program participation, or behavior. In general, questions should be narrow, relevant to youth and the program, and they should contribute to knowledge of the program or field in a key way.

Once questions are identified, they need to be paired with the appropriate method. The Youth Save program, funded by The MasterCard Foundation and implemented by a consortium that includes Save the Children, the Center for Social Development at Washington University in St. Louis, the New America Foundation, and CGAP (the Consultative Group to Assist the Poor), investigates savings accounts as a tool for youth development and financial inclusion in Ghana, Colombia, Nepal, and Kenya. Their learning agenda, shown below, is comprised of a series of questions supported by researchers at the University of North Carolina and the Center for Social Development at Washington University 1.

Learning Objective Method
Is there a business case for YSAs? Business Case Studies
Which youth, household, and YSA characteristics are associated with savings? Savings Demand Assessment
How do YSAs impact developmental outcomes for youth? Cluster randomized experiment in Ghana
How are youth experiences in saving informed by context? Integrative Case Studies

Not all programs have the luxury (of time, funding, and expertise) to support a broad learning agenda. Program effectiveness may be their principal concern. In that case, evaluators may devise questions directly from program objectives. RCTs would be one example of a method that requires a specific type of question.

Checklist: IPA on the types of questions that lend themselves to RCTs

Not all types of questions can be addressed via an RCT. In order to decide if your question might work with an RCT, ask yourself if it is:

  • Specific – Targeted, focused: test a certain hypothesis
  • Testable – Has outcomes that can be measured
  • Important – Will lead to lessons that will affect the way we plan or implement programs
  • Micro – Macro, expansive questions not appropriate

 

  • 1. Excerpted from the Youth Save Consortia presentation at the 2012 GYEOC