9.4 Effective M&E Instruments will Ease Implementation and Enhance Data Quality

Theories of change and SMART indicators are critical to overarching M&E design. However, excellent M&E design can be undone by inadequate instruments (surveys, interview protocols, focus group discussions, etc.) or poor implementation.

  • Make all instruments developmentally and culturally appropriate. Young people should understand and find the questions being posed to be relevant. Box 9.4.1 explains how an Operations Management System is helping to build an evidence-based organization and network, and Box 9.4.2 details how to develop effective survey questions.
  • Pay attention to details in implementation. Roles and responsibilities of project team members should be clearly defined. Work plans should be feasible. Box 9.4.3 offers more specifics.
  • New tools can streamline data analysis. New software, including open source software, can facilitate data analysis for both quantitative and qualitative data. Box 9.4.4 includes examples of qualitative data analysis software.

9.4.1 New Tools: Youth Business International's Operations Management System

Youth Business International (YBI) is a global network of independent non-profit initiatives helping young people to start and grow their own business and create employment. YBI was founded in 2000, and HRH The Prince of Wales is YBI’s President. YBI members assist under-served young entrepreneurs with a combination of training, access to capital, mentoring and other business development services. They adapt this common approach to their local context, working in partnership with governments, businesses and multilateral and civil society organizations.

In 2010 YBI helped 6,346 young people to start their own business. YBI estimates that these businesses will go on to create nearly 20,000 additional jobs within three years. Through YBI, members exchange resources and act collectively to increase the efficiency and scale of support to young people seeking to start their own business. A network team coordinates and leads this activity, with additional responsibilities for driving network growth, quality and performance. YBI members have helped over 100,000 young people to create their own business and generate employment.

YBI is committed to growing an evidence-led global youth entrepreneurship network. Developing a robust evidence base will drive efficient operations and impactful policy. YBI is building an Operations Management System (OMS) on the Salesforce.com platform as a tool for YBI members to track and analyze key performance indicators and for the YBI

network team to aggregate and benchmark global trends.

At the local level, members can closely monitor their operational performance. At the global level, this cloud-based system means YBI has cost-effective access to an expanding repository of data that will drive rigorous impact assessment. The OMS went live in March 2011, and it has been adopted in seven countries so far. YBI’s objective is for at least 30 members to be using the system over the next three years. Critical success factors for the project include:

  • In-country management teams who acknowledge that strong systems are crucial for their organizations to achieve their goals and evidence their work;
  • Local operational experts who have the availability to feed system design and ensure business processes are integrated;
  • User adoption champions who support users and provide feedback that influences continuous improvements to the system.

The centrality of the OMS means YBI can take the operational best practices learnt from working with each member and apply them to the system for the benefit of all. With responsive design, this global infrastructure will become a valuable research tool to understand the most effective interventions to support youth entrepreneurship.
 

 

With survey writing, it is important to take an approach that demonstrates an understanding of the audience and the ultimate learning objectives of the initiatives. The following text box shares guidance from YouthSave on how to develop effective surveys.

9.4.2 Practical Tips: YouthSave On How to Write Good Survey Questions

Knowledge tests and/or youth’s self-reported changes in knowledge, skills and behavior are often used to evaluate financial capability. Answers to tests and surveys can act as indicators for our outcome objectives until conducting an impact assessment. Standardized surveys (questionnaires or interview forms that are tested for and found to be reliable and valid) are ideal. However, if no culturally valid (i.e., tested in the same region or country) or developmentally valid (i.e. tested with a similar age group) are available, Despard shared these simple rules for developing survey questions:

  • Test the questions with a group of youth who represent the youth you are targeting (by age, geographic location, gender, etc.). Ask them, “What do you think this question means?” and “Why did you give the response that you did?” Their answers will tell you if you are measuring what you think you are measuring.
  • Make the questions relevant to the project. If your project did not target youth’s knowledge about managing debt, don’t ask debt management questions.
  • Word questions at the reading and comprehension level of the youth you serve. Limit use of or avoid abstract concepts.
  • Use words, terms, and phrasing that is familiar to youth and their day to day experiences, like referring to what Ghanaian youth do with “chop” money they receive from their parents rather than calling it a “food allowance”.
  • Keep questions clear, brief and simple. Fewer words are better.
  • Define important words. Don’t assume that youth know what you mean by “save”. Define it for them – “Putting aside money in a safe place to be used later”.
  • Ask follow-up questions. A youth may say that she usually saves, but you may also want to know: a) what a typical amount saved is; b) the purposes for which they save; and c) how long they typically hold onto savings.
  • Ask open-ended questions too. Like with focus or discussion groups, it’s very important to get responses in youth’s own words by asking questions that require youth to offer description and explanation. For example, “What helps you save? What gets in the way? Why? Can you tell me more?”
  • Avoid yes/no questions. If you ask a youth “Do you save money?” A youth who sometimes saves may find this difficult to answer. Either way, the yes or no response you get is not accurate. It is better to use 3- (“Never/Sometimes/Always”) or 5-point (Poor/Fair/Good/Very Good/Excellent) scales.
  • Avoid “leading the witness.” Don’t ask, “You save money so you can have a better future, right?” Avoid any questions that might elicit a socially desirable response–when youth tell us what they think we want to hear.
  • Avoid “double barreled” questions. This is when you ask more than one question at the same time, like: “How often do you make yourself wait to buy something you want and make a savings deposit in a bank?” (These are two separate questions!)

 

 

There are two basic ways to ask survey questions: using questionnaires that youth complete by themselves OR interviewing youth by asking them the survey questions and recording their responses.

The Population Council’s experience, highlighted below, shares how tools cannot operate in vacuums and organizations need to pay particular attention to who will be using them and who will be managing next steps following the use of them.

9.4.3 Checklist: Population Council's Considerations for M&E Tool

Effective M&E design needs to consider implementation challenges, as well as the roles and responsibilities of team members involved in data collection and analysis. M&E specialists need to work with program managers to ensure that the M&E is feasible and that tools work within a particular operating context. The following questions can help ensure that M&E works on the ground. Does your M&E plan consider the following?

✔Who will create/finalize the registers, tools or surveys & by when?

✔When will these registers be used?

✔Who is responsible for ensuring they get completed?

✔Who is responsible for analysis/reporting?

✔How often/when will the analysis/reporting be done?

✔Do you need a monthly/quarterly summary form?

✔What information will you need to report to donors or other stakeholder?

When conducting M&E with young people, the Population

Council emphasizes that team members need to consider young people’s unique developmental needs. The Population Councilmakes the following suggestions for monitoring and evaluation with youth:

  • Timing: Keep in mind that youth have their own time constraints with school, agricultural seasons, and household chores. Schedule M&E events in such a way that more young people can participate.
  • Attention span: M&E activities, such as surveys and questionnaires, must be engaging to catch and keep their attention. The length of time of surveys must take into consideration young people’s attention span.
  • Interviewer selection: Work with young trainers to match youth and same sex interviewers.
  • Legal context: In both program design and M&E, practitioners should plan to acquire consent from parents for research with youth. Indicators should also reflect an understanding of the country’s legal context; for example, in some countries banking regulations prevent youth from opening up bank accounts, so that would not be an appropriate measurement unless the program aims to change those regulations.

For more information, see: www.popcouncil.org/ projects/242_EnhancingEconHealthSocial.asp.

 

 

 

Qualitative data helps evaluators and program designers understand the story behind the numbers. It can reveal delayed program impact, provide a more relevant understanding of the intervention when utilizing standard field-wide indicators, and draw attention to contextual factors such as cultural barriers that might impact program results. Qualitative data also helps to shed light on confusing results and reveals new program and research issues.

9.4.4 New Tools: Help with Analyzing Qualitative Data

While beneficial, qualitative data can present certain challenges in analysis. The following tools can help:

  • Weft QDA is an open source software for qualitative analysis that also allows for analysis of transcripts and coding. www.pressure.to/qda/.