5.4.2 The Evidence Base is Growing but the Results are Heterogeneous

In order to contribute to the evidence base on life skills programming, the International Youth Foundation (IYF) carried out a randomized control trial (RCT) for its program in Kenya (see box below). The process of going through a robust evaluation provided several valuable lessons that are transferrable to other programs.
 

Lessons on Conducting an Impact Evaluation
  • Be sure to include a “Plan B” for high-levels of dropout, especially for the individuals in the control group. Oversampling a fair amount but even that was not enough.
  • Make sure you have a plan to deal with questions about the perception of “denial of treatment”, which should include a plan to offer the treatment in the near future.
  • Include qualitative data collection. While the quantitative results are important, the nuanced details need to be collected through additional tools.
  • Include concrete steps on how the evaluation is going to affect practice, particularly for the project being evaluated.
  • Have a solid dissemination strategy for publishing the results. 

The impact evaluation itself found that adding a life-skills component to a comprehensive ICT-training program helped strengthen effectiveness for some outcomes of interest. For example, participants of the life skills training were more confident and more likely to work in the technical area of the training (i.e., ICT). However, other outcomes of interest, such as the likelihood to look for a job and income level were not enhanced during the evaluation period (while life-skills recipients did much better than the comparison group, they did not exceed the outcomes of those who had only received the technical training).

IYF’s Ninaweza (“I can”) Program, Kenya

IYF’s Ninaweza program in Kenya, implemented in partnership with The African Center for Women and Information & Communication Technology (ICT), aimed at increasing the employability and income-generating capacity of 700 disadvantaged young women (18-35) in informal settlements surrounding Nairobi by providing them with comprehensive skills training (combining technical skills in ICT, life skills, and internships) and job placement support.

The source of the life skills training was the “I LIFE-WORK Training Manual” implemented by the African Centre for Women, Information and Communications Technology. This consisted of a 40 hours training program in skills for everyday life, such as emotional control, body language and personal planning, as well as skills for the world of work, such as workplace ethics, listening and communicating in the workplace and understanding employers’ needs.

Research questions focused on the following aspects of the program:

1. Does the youth employability program produce technical knowledge and skills that will allow the young women to obtain work in the IT and related sectors?
2. Does the program have a positive effect on income-generating capacity, whether in the formal sector, informal sector, or from self-employment?
3. Does the program lead to improved self-confidence and other attributes of psychosocial and interpersonal empowerment?
 

A randomized control trial was conducted with support from School-to-School International, randomly assigning eligible youth into treatment group 1 receiving the whole package including the life skills, treatment group 2 including all the services but without the life skills, and a comparison group receiving no services. Groups were surveyed before the start of the intervention, at the end of the two-month training component, and at graduation, after the six-month internship and placement support components. This design allowed evaluating the relative effectiveness of the life skills component vis-à-vis the ICT training.

One of the main challenges faced during the evaluation was to follow-up with the respondents during three points in time over the 10-month program. Although the high graduation rate from the M&E data indicates that attrition was not an issue for the program itself, it was an issue for the evaluation, as 31.5 percent of those that took part in the baseline did not take part in the mid-line and 50.2 percent did not take part in the end-line. Comparison of demographic characteristics, however, found that that this attrition did not alter the profile of the respondents. This suggests that attrition was random (based on observable characteristics); thus that the decrease in the sample size reduced statistical power but did not affect the validity of the counterfactual (provided that treatment and control group also remained similar in terms of unobservable characteristics, for which there is no absolute certainty). Some of the evaluation’s main results are listed below:

  • Self-confidence and interpersonal empowerment: Youth who received the life skills training were more confident in their qualifications than their peers. They also perceived finding a job at end line easier than youth who had not received the life skills training and those in the comparison group.
  • Technical knowledge and skills: The training component in ICT and life skills succeeded in providing those treated with technical knowledge in IT and knowledge of life skills that otherwise they would not have acquired.
  • Employment: Participants that received life skills training showed greater gains in their likelihood of obtaining a job than those in the control group.
  • Sector of work: Youth who received the life skills and ICT training were more likely to work in ICT-related jobs than those who received ICT training only. Both groups were more likely to work in this sector than those in the control group.
  • Income: Participants had a significantly higher income than the comparison group, but there were no differences between the youth who had received life skills and those who didn’t.

The detailed methodology and results of the impact evaluation are available at: http://library.iyfnet.org/library/testing-what-works-youth-employment-evaluating-kenya-s-ninaweza-program

Another impact assessment carried out by the University of Chicago Crime Lab suggests that cognitive behavioral therapy (CBT)-based programming that targets judgment and decision-making can improve a range of key life outcomes (see box below). Researchers studied an intervention called “Becoming a Man” (BAM), a CBT-based counseling/mentoring program coupled with non-traditional after-school sports, through a randomized control trial.

In this case, researchers could carry out the evaluation based on already available administrative data from the Chicago school and law enforcement systems. This drastically reduced the cost of the evaluation since no primary data collection was necessary. Notably, the study also included a cost-benefit analysis, which is rare among youth-focused interventions. To this end, the study team compared the cost of the program (around $1,100 per participant) with the estimated savings based on crime reduction and potential increases in high school graduation. Benefits were calculated on the individual level and government level (see box). Because the researchers could show that the program’s benefits outweighed its costs via the most rigorous possible analysis, policymakers have been very supportive of committing additional funding to the program.

Youth who participated in the program showed a 44 percent decrease in violent-crime arrests during the intervention and became more engaged with school. In addition, the BAM program also proved to be cost-effective. The findings emphasize the potential of helping youth to develop their judgment and decision-making skills as a strategy to improve school outcomes and decrease violence.

Becoming a Man (BAM), USA, Chicago

The program, Becoming A Man—Sports Edition, was developed and delivered by Youth Guidance and World Sport Chicago to more than 800 boys in 18 public schools during the 2009-2010 school year. The program consisted of 27 one-hour small group sessions (once a week, over eight months of the school year) and complemented by after-school sports activities, though in practice, each program participant only had about 13 contact hours with the program on average.

The intervention focused on developing skills related to emotional regulation, control of stress response, improved social-information processing, interpersonal problem solving, goal setting and attainment, and personal integrity. The program was based on cognitive behavioral therapy principles – a variety of techniques that help individuals identify, monitor, challenge, and change their thoughts and behavior. The key motivating idea is to teach youth to slow down and make reflective, rather than reflexive, decisions in moments of potential conflict.

Based on available Chicago Public School administrative data the research team identified eligible 2,740 male seventh- to tenth-graders in the program schools. These youth were then assigned via a fair lottery either to a program group that received BAM counseling and sports programming (in-school activities only, after-school activities only, or both) or to a comparison group that received no additional services beyond those that public schools typically offer.

Key results included:

  • Participation reduced violent-crime arrests by 44 percent.
  • Participation reduced arrests for crimes categorized as “other,” including vandalism, trespassing, and weapons possession, by 36 percent.
  • Participation reduced the likelihood of attending a school inside a juvenile justice setting in the year after the program by 53 percent.
  • The intervention improved school performance and engagement, measured by days present in school, grade-point average and school persistence. Importantly, these impacts on schooling outcomes lasted through the year after the program ended. The size of the schooling impacts imply that graduation rates might increase in the future by an additional 10 to 23 percent compared to the control group.
  • Cost-benefit. The program cost around $1,100 per participant, while its impacts on criminal behavior generated benefits to society that were valued on the order of $3,600 to $34,000 per participant, depending on how the costs of crime were calculated. In addition, impacts from the potential increase in graduation were estimated at over $49,000 per participant from increased lifetime earnings, tax payments, and lower public benefit use.

More information on the research methodology and the results are available at: http://crimelab.uchicago.edu/page/becoming-man-bam-sports-edition-findings

While the above evaluation results are encouraging, more rigorous evidence is needed to really understand what socio-emotional and broader life skills matter most, at what time, and how they can be taught in locally appropriate environments.