Advance Youth Employment with Structured Experiential Learning and Evaluative Thinking (part 1)
This is the second entry of Save the Children’s blog series on the Structured Experiential Learning (SEL) process for youth employment. The first blog talked about how SEL connects data users and data producers (like Mars and Venus hanging out together!). This time, SEL has the opportunity to meet its friend: Evaluative Thinking (ET). SEL and ET introduce themselves to each other, and discuss how they can work together. Check out part 1 of this interesting conversation led by Save the Children, CRS and CORE!
Jane Buckley, Evaluation Facilitator, Cornell Office for Research on Evaluation
Andrea Lozano, Senior Specialist, Research and M&E, Skills to Succeed, Save the Children
Guy Sharrock, Senior Advisor for Learning, Catholic Relief Services
Q1-1: What is Structured Experiential Learning? Why do we use it?
Andrea: In our youth employment program “Skills to Succeed,” as in other programs, we collect data constantly, but frequently its use is limited to reports for donors. We realized that we could use it for more, especially when we have implementation challenges. So we decided to take a systematic approach to solve problems encountered during project implementation using data. The method we use is variously called “Structured Experiential Learning (SEL)” or “adaptive learning.”
SEL is a 4-step process that involves collecting and using routine project monitoring data across alternative project designs, so that we can have real-time information about project performance. This information is fed back directly into the project for mid-course corrections. SEL links data users – who are typically project managers - and producers – M&E (monitoring and evaluation) or research specialists - together in a deliberate, systematic way that can be documented for organizational and program learning.
First, we identify and narrowly state a problem; second, identify root causes of the problem; third, choose alternative project designs aimed at solving the problem; and fourth, test how alternative project designs compare against each other to select the best one. We see these alternative project designs as ‘potential’ solutions, since we don’t really know which one will be the most effective at the design stage. This process is experiential because we’re learning by doing. We monitor our learning progress, adjust strategies or workplans based on tested alternative project designs, and continue to ask new questions as we go forward with the chosen solution (for more information on SEL, click here).
Q1-2: What is Evaluative Thinking? Why do we use it?
Guy & Jane: Evaluative Thinking (ET) is similar to critical thinking. It involves identifying assumptions, posing thoughtful questions, allowing multiple perspectives and making informed decisions. The focus of ET is on paying attention to both anticipated and unanticipated results and making better decisions in contexts of complexity and uncertainty. ET should be embedded in the fabric of every organization having a value-adding role in all aspects of MEAL (monitoring, evaluation, accountability, and learning) and applying equally to both programming and operations decision-making.
Q2: Who should use SEL and ET? Why?
Andrea: We believe SEL could be adopted by data users and data producers, because, although project workplans and logframes look linear, life is not. SEL is a powerful tool to combat the sometimes simplistic assumptions about how interventions should work on the ground. Typically, we have tools like the project’s theory of change and logframe, which are built on linear assumptions. At the same time, there might be other kinds of assumptions that are not explicitly documented in these tools or that we are not consciously aware of. That’s why we need SEL to help us work around this non-linear reality.
Guy & Jane: ET is good for everyone! ET is not just for decision-makers, but also an important tool for field staff. One reason is that field staff can use ET as a basis for contributing their insights arising from direct interactions with the people they serve. By doing so, this can make ET a vehicle for beneficiary accountability, and further drive more demand-led programs with better quality. It needs everyone in the organization to practice ET consciously, like a culture, or a way of doing business.
Q3: How do SEL and ET complement each other?
Guy & Jane: Something that comes immediately to mind concerns using data. We’ve been planning several rounds for ET. First, we set up the theory of change, and focus on generating learning questions and, in the second round, we concentrated more on aligning these with methods for data collection. In the third round, we will get into data utilization, where we can learn from SEL’s rapid feedback and its focus on closing the learning loop. What SEL does is, when people see results and are able to use results, it creates an incentive, like a self-rewarding system, that encourages people to do more evaluative work, and eventually brings us to the point of making decisions. ET can learn a lot from SEL’s focus on date use for decision-making.
Andrea: There is a lot of interest out there in iterative adaptation and adaptive management– The World Bank, USAID Learning Lab, the Center for Global Development, to name a few, are all talking about this. So our motivation for coming up with this 4-step SEL process was to gather insights from all these different sources and tailor it in a more digestible and practical way for practitioners. But we recognize there’s a risk of using SEL like a cookie cutter, going through the steps without really thinking creatively or critically. That’s why I see a value in integrating ET into every step of the SEL process. ET really pushes practitioners at all levels to be more critical and question their assumptions. This is essential to truly have an iterative process all along the SEL ‘pathway’.
Guy & Jane: Agree. A part of ET that very useful for SEL, is to manage conversations with different opinions more constructively. Essentially, ET can be as simple as opening up questions like we mentioned earlier. At some point, you’ll get into the key question: On what basis are we going to make decisions? ET helps us dig deeper into this basis, and lead us towards look into assumptions, validating them and then deciding how best to move forward
Andrea: Adding to this point, I can also see a way for ET to help SEL when data is not necessarily perfect or points to a clear direction. For instance, in a situation where you may have different data sources that contradict each other, ET can help SEL participants discern the priority and relative merit of the different data sources in the iterative learning process.
Guy & Jane: Just to highlight some thoughts about assumptions. There can be a lot of complexity in every project or program. When choosing assumptions, we have to be careful: which assumptions to depend on, which ones to test, and which ones don’t seem to be safe. The insights of field staff can be so important here as they can often remind us of the resource challenges faced by the people we serve that, in turn, affects their willingness to adopt new approaches. Whilst we may all understand the benefits of a particular health intervention, there can be all sorts of legitimate reasons to explain varying levels of adoption – just think of public health attempts to stop people smoking! We really need the knowledge that is ground-truthed. It’s always very valid.
Andrea: Thanks for bringing up ground-truthing. SEL and ET are about mindsets – willingness to continuously improve which may entail adjusting strategies and workplans – and habits – using data systematically to do so. Changing strategies and workplans based on the tested alternative project designs may be a new way of working for many people. The people factor can’t be underestimated. We need to talk more about that.
Stay tuned for part 2 of SEL and ET conversation, where we’ll address the people factor – the change management process and resources needed to apply these tools.
Learn more about:
Let us know if you have any idea on how we can improve SEL! Please contact Andrea Lozano at [email protected]