1.1.6 Making Performance Measurement More Meaningful: Measuring Employment Outcomes in Youth-focused Workforce Programs

Workforce practitioners share common practices, challenges, and opportunities in measuring job outcomes. One of the ultimate tests of workforce programs is how successful they are in helping youth graduates obtain employment. Recognizing the relevance of this topic to the field, representatives from CARANA, Catholic Relief Services (CRS), Education Development Center (EDC), Education For Employment (EFE) Foundation, and the International Youth Foundation (IYF) shared lessons learned in this area at the 2012 GYEOC.

Successful programs follow best practices in measuring job placement. Many workforce programs track the percentage of youth placed in jobs with remuneration at multiple time intervals after training: time of graduation, three months post-graduation, and six months post-graduation. Most follow-up is conducted through individual phone calls, and in some cases, email-based surveys are used.  Also, some programs collect baseline employment information to compare before-and-after outcomes, such as IYF, but this is not standard practice. 

Job Placement Definitions/Indicators used by USAID

Job placement: Found and held job for at least 90 days after completing program 1.

“F” Indicator 4.6.3-2 Number of persons receiving new or better employment (including better self-employment) as a result of participation in USG-funded workforce development programs. Total number of people gaining employment or better employment within six months of participation in U.S. Government- funded workforce development project. Better employment is based on the participant’s perception of whether the employment is better. (It could be better because it is closer to home, has better pay, a better schedule, etc.)2

IYF’s standardized approach to measuring job placement for entra21, a workforce program that ran from 2001 to 2011 and included 59 projects in Latin America and the Caribbean, is notable. Locally-based independent consultants, using IYF’s evaluation guide and standardized questionnaire, conducted evaluations and follow-up studies of a sample of graduated youth cohorts six to nine months after graduation for each project. IYF use of third-party evaluators is a best practice that should be adopted in the field.

Data on job placement rates can be disaggregated in various ways, such as gender, urban/rural youth, educational background, youth with disabilities, and at-risk youth (for example, those involved in crime, have substance abuse challenges, are school drop-outs, etc.). Gender data is the most commonly disaggregated piece of information. Most programs also track job placement into specific sectors or industries.

Sharing information on job placement rates can help address information gaps in the field, but additional impact studies are needed to attribute causality.
Information on the job placement rates of youth workforce programs is not readily available, and as a result, the field faces an information gap on performance benchmarks. The table below presents examples of job placement rates. However, readers should note a number of limitations when making comparisons across programs. Programs work in different country environments with different labor laws, across labor markets with varied performance, and with different target youth populations. Generally, programs that work with the most vulnerable and at-risk youth populations face greater difficulty in securing jobs with remuneration for graduates, as compared to programs that work with skilled graduates and youth that have higher education levels. However, the interactions of numerous factors influence specific job placement rates, making it difficult to draw meaningful comparisons that are not grounded in rigorous impact evaluation design.

Program(s)

Job Placement Rates

Work with remuneration?

Education For Employment (EFE) programs. Established in 2006. Affiliates in Egypt, Jordan, Palestine, Tunisia, Morocco, and Yemen. Funded by multiple donors.

80-85% job placement

Yes

Educational Quality Improvement Program (EQUIP3). 26 projects from 2003-2012. EDC-led with implementing partners and USAID-funded.

Placement rates ranged from 54%- 88% among the 26 Associate Award projects

Yes in many cases – variation across the 26 projects

IYF’s Entra21 Program: 35 projects in Phase 1 (2001-2007) and 24 projects in Phase 2 (2008-2011). Across Latin America and the Caribbean. IYF-implemented and funded by multiple donors.

In Phase 1, the average employment rate ex post was 54% (measurements taken in 2006 and 7). In Phase II, where IYF targeted more vulnerable youth and worked through government-led programs, the average rate was 45% (measured 2009-11).

Yes

 

Improving Access to Employment, 2009-2013, El Salvador. (CARANA-implemented and USAID-funded)

85-90% job placement

Yes

Information Technology Training Program for People with Disabilities, 2007-2014, Vietnam. CRS-implemented and USAID-funded. 

85% job placement

Yes

The youth WfD field faces limitations in addressing the question of what would have happened to youth graduates, in terms of job placement outcomes, if they had not participated in workforce programs. The impact studies of the Jovenes programs 3 in Latin America offer evidence that comprehensive workforce interventions can help disadvantaged youth in developing countries gain employment. However, empirical evidence is still relatively limited for the broader field. More programs are reaching readiness levels for conducting rigorous evaluations to determine causality between a specific program intervention and change in youth outcomes.

Innovations in data collection and use of social media and alumni groups help workforce providers overcome the challenge of long-term tracking of young alumni on-the-move.
Workforce practitioners want to know where youth are five years, or even one year, after graduation in order to have a more meaningful picture of employment success. Typically, most programs only have employment information as a snapshot in time at three months or six months after training. Staying in touch with youth graduates in the long-term of one year or longer and maintaining up-to-date contact information is a struggle. Young people are very mobile, as they tend to move around to pursue new work and educational opportunities. Their mobile phone numbers and email addresses also can change frequently. To address this problem, a number of organizations are experimenting with new and innovative ways of staying in touch with alumni and capturing long-term job information.

  • Collecting data through mobile technology solutions: Mobile phone data collection platforms, such as those developed by leading mobile technology provider Souktel, help workforce implementers save time and money in collecting job information and staying in touch with young alumni. Souktel’s research shows that time savings are significant over the course of one year. For example, to administer a 10-question survey to 500 youth by phone requires close to two months of staff time whereas delivering the same survey by SMS takes less than an hour total per year. To be most effective, it is important to collect multiple contact numbers for young people—for example, phone numbers of family members and friends—to increase chances of staying in touch when youth change their number or move. Also, securing funding for phone credit is important so the cost is not a burden to the young person in responding to SMS surveys.
  • Staying in touch through social media platforms: Young people are active with online social networks through Twitter and Facebook. Even in rural areas, young people are increasingly accessing social media through mobile phones. Social media can be used for staying in touch with youth graduates of workforce programs when training ends, building virtual networks of peer alumni, and maintaining their personal connections to programs. The CARANA-implemented USAID program in El Salvador, Improving Access to Employment, has built an online network of youth followers on Facebook reaching over 32,000 fans as a way of engaging with alumni and giving youth a platform to share their experiences with peers. CRS has also started Facebook group pages for training alumni and keeping in touch with graduates.
  • Connecting through alumni groups: Alumni groups help maintain relationships with graduates and create a sense of community. If youth feel well connected to the program and alumni, they are more willing to stay in touch and share information about their longer-term employment situation. EDC has successfully used ‘youth trackers’ to follow up with graduated youth cohorts. Youth trackers are identified within each graduating class as those individuals who are naturally ‘connectors.’ They are given cell phone credits and asked to follow up with their classmates at designated periods. This method has the added advantage of building an alumni base, as the youth maintain connections with one another. EFE offers ongoing support and professional development to alumni, which is the third pillar of its employment model. It is committed to tracking and staying in touch with alumni one year after graduation. Also, CRS creates alumni clubs and recruits youth leaders to act as stewards responsible for keeping alumni in touch with each other and with CRS.

Measuring job outcomes beyond job placement is important to capturing a more complete picture of workforce programs’ success in terms of job quality, return to schooling, starting a business, and skills attainment.
While most implementers measure job placement rates, as required by USAID and other donors, there is greater variance across programs in tracking job quality. A number of job elements can be measured such as wages, benefits, formality of employment (existence of contract), type of employment (full-time, part-time, or temporary), job satisfaction, and job mobility/promotion, among other indicators.

CARANA offers a good model in this area. It thinks about job impact beyond just wage earnings and is interested in learning about working hours, income from tips where relevant, length of contract, benefits, job satisfaction, and job stability. CARANA tracks job movement and promotion with youth graduates after they get hired and asks employers upfront about job growth and promotion potential. EDC also tracks job quality. EDC asks participants to report whether their employment improved post program as compared to their previous employment (if they had one)—and how.  Youth respond according to categories such as: work hours, proximity to home, wage, type of work, and work environment. As another example, CRS is putting in place a new system to follow graduates for up to 18 months and to track types of jobs, trends in job retention, changes in income, and promotions.

Many workforce implementers are also interested in learning which of their training graduates return to formal schooling and some track and report on this indicator, although this is not standard practice. Moreover, many programs collect qualitative information through employer surveys to understand employer perceptions about the quality of graduates’ job performance, skills, and behaviors and to assess overall satisfaction. This can offer powerful information on the effectiveness of programs in combination with other indicators.

Finally, evaluating changes in skill competencies, knowledge, and behaviors before and after program training is important because this information can be a key predictor of future employment and livelihoods success. A number of workforce programs implement pre- and post-training tests as part of their monitoring and evaluation systems to measure acquisition of “soft skills” related to the workplace such as how to resolve conflicts, communication, appropriate hygiene for work, or punctuality. These tests typically capture self-reported or perceived acquisition of skills. Efforts to measure soft skills behavior changes among youth in a more objective manner remain challenging.  (See text box.) Employer surveys that measure employers’ perceptions of graduates’ performance, as mentioned above, are one way to capture this information but employers are not always available and so this is not always feasible.

New Tools: Assessing Soft Skills

EDC is experimenting with a work readiness assessment tool to try to measure behavior and attitude changes related to “soft skills” among youth job seekers. This assessment tool being piloted in Liberia and Rwanda includes a written test comprised of situational judgment questions that youth answer to gauge their ability to respond to new situations with new skills. The assessment focuses on four key skills clusters: 1) thinking skills and strategies, 2) collaboration skills, 3) interpersonal communication skills, and 4) work habits and conduct 4.

 

  • 1. See: Guide to Cross-Sectoral Youth Assessments, USAID and EQUIP3, 2009.
  • 2. See: FY11 Foreign Assistance Indicators, US Department of State
  • 3. Inter-American Development Bank Jovenes en Accion Program
  • 4. Education Development Center (EDC) Preparing4Work Program Assessment.