5.2.4 Multinational impact evaluations should allow flexibility within parameters
As the YEO field replicates program approaches in new contexts and countries, stakeholders have become increasingly interested in evaluating programs in various countries. Donors in particular need to know that their funding portfolio is advancing their goals across national contexts. The use of technology as a vehicle for program delivery across national boundaries has also increased the need for multi-country impact evaluations. Distinct challenges need to be considered when evaluating in different countries. Even within one country, differences at the sub-national level can be significant. Across countries, differences in educational systems, regulatory frameworks, capacity levels, and policy can be vast. Nevertheless, identifying core programming elements that work in multiple contexts can advance learning for the field significantly.
Donors of multinational initiatives frequently create global outcomes to guide local partners both in program design and evaluation planning. For example, Citi Foundation makes grants in 91 countries globally and employs a results-oriented framework that has identified a theory of change and outcome in each of its distinct focus areas (internationally, those include microfinance, enterprise development, financial capability and asset-building, and youth education and livelihoods). Within each grant, NGO partners indicate how their program strategy and intervention will contribute to those outcomes both in terms of quantity and quality (e.g., how the program advances participants along the Foundation’s “results path” and demonstrates evidence of impact). SRI International coordinates and develops global tools for the multinational evaluation of Intel® Learn, a technology and education program that operates in 14 countries and has reached 1.5 million learners since 2003. Country-level findings can be rolled up to a global level, but can also be used country-by-country. National evaluators typically adapt global measures to some degree to fit their context. Case studies in individual countries also contribute to more country-specific information.
The Financial Education Fund (FEF) supported 15 grantees in 8 African countries (2009 – 2012), several of which were also funded by the Citi Foundation. They included and funded impact evaluation as a core operational component of each project. Originally, FEF hoped that all partners would achieve the “gold standard” of evaluation that generally included an RTC, quantitative and qualitative aspects, and some probing of financial behavior beyond the financial education initiative. However, they realized that it was unrealistic, expensive, and not always appropriate to expect all organizations to achieve a gold standard. By creating evaluation parameters, the FEF gave sufficient guidance to orient partners and offer greater flexibility to adapt to their particular circumstances 1.
- 1. Excerpted from the presentation of Ms. Alyna Wyatt, Senior Associate in the Business and Development Practice at Genesis Analytics and Team Leader, Financial Education Fund, at the 2012 GYEOC.