Advance Youth Employment with Structured Experiential Learning and Evaluative Thinking (part 2)
Following the first and second blog, this is part 2 of the conversation between Structured Experiential Learning (SEL) and Evaluative Thinking (ET), led by Save the Children, CRS and CORE. Let’s see what challenges we could have when applying SEL and ET, how SEL and ET can help in the change management process, and what resources they need!
Jane Buckley, Evaluation Facilitator, Cornell Office for Research on Evaluation
Andrea Lozano, Senior Specialist, Research and M&E, Skills to Succeed, Save the Children
Guy Sharrock, Senior Advisor for Learning, Catholic Relief Services
Q4: Are there any challenges to applying SEL and ET?
Andrea: Applying Structured Experiential Learning (SEL) to a pre-mature M&E system could be a challenge. It is because SEL is built on tight feedback loops between generating and using data to inform project implementation decisions, for adaptive management. Because of this fast-paced iteration, SEL is better suited for projects that already have robust M&E systems in place. Another challenge is about the type of data you use. Initially, SEL relies more heavily on quantitative data than qualitative data. It specifically defines a problem using ‘who,’ ‘what,’ ‘when’ and ‘where’ (‘why’ is the problem itself, and ‘how’ will be the best solution that SEL finds out). An emphasis on qualitative data occurs during steps 2 and 3 – identify root causes and design potential solutions to the problem. In step 4, because we need to compare potential solutions against each other in a standardized and comparable way, more emphasis is again placed in quantitative data. (For more on the 4 step SEL process, see the webinar.)
Guy & Jane: The concept we use at CRS is Evaluative Thinking (ET). One challenge of applying ET is about culture change. It requires not only adapting a habit at the individual level, but for ET to be maintained, you also have to find ways with your peers or colleagues to incorporate it in systems and processes. This requires a mindset open to multiple perspectives among all staff, and that everybody feels free to express ideas grounded in ET. It must be ‘safe’ for staff to talk to their supervisor like, “hey, I’ve been wondering why we’re intervening in this way because my experience suggests there may be an alternative,” for example. For senior staff, it’s ok to say, “I don’t have the best solution, but I’m going to hear what others say.” This may present a new challenge about managing disagreements, but our experience to date is positive – this can be facilitated and managed with appropriate training and, critically, support from leadership. The key point for ET is to understand that nobody has a monopoly on wisdom, and projects can benefit greatly from being open to the different perspectives of its entire staff.
Q5: How do SEL and ET help in the change management process?
Andrea: Fundamentally, we know that no project operates in a vacuum. The fact that we’ll encounter different contextual factors requires us to constantly adapt to changes. Thus, we need SEL to embrace this uncertainty, and to recognize that our assumptions are based on context, and context might change. SEL can help us manage that change. Whatever these changes might be, you can always use SEL to approach a problem, test alternative solutions to the problem and choose the most relevant solution for your project. SEL can be adopted by data users and data producers in development projects, because SEL helps them better communicate with each other.
Guy & Jane: A key challenge for staff in all development projects is to accept that targets simply reflect their best estimates of expected achievement at one point in time. It is essential for high quality programming that they remain open to ‘surprises’ when they find there is variance between what they anticipated and what actually occurs. ET is a habit of mind that seeks out these surprises as well as evidence for why things do and don’t happen as they expect. A key part of ET is to build the habit of pausing, and to surface and question the assumptions that underpin the project’s theory of change, and to align what you’re expecting with realities on the ground. Program staff might also use ET in their regular program and evaluation planning work and ask, “How do we know what we think we know?” even when no ‘surprise’ has occurred.
Two phrases from a well-known peace-building professor, John Paul Lederach, can apply well to ET: the first is about “demystifying theory. “People usually step back when they hear the word ‘theory’. But part of the ET is to get people grounded in the theory of change, and understand it as the basis of a project intervention. At the same time, by surfacing and examining assumptions, ET can help “re-mystify practice” - that is to bring mystery back into our practice, to maintain a sense of wonder and openness, when we’re promoting particular solutions.
Q6: What resources would be needed in adopting SEL and ET?
Andrea: For SEL, it really depends on your starting point, since SEL requires different inputs in each step. In the step of identifying problems, we would expect this conversation is already happening during regular project meetings, though it might take additional time to state the problem in a narrow, actionable way. Next, when we’re going through the root causes and potential solutions, we envision a 3-day workshop where we invite external stakeholders and facilitate fast-paced brainstorming discussions. For the testing step, it will very much depend on how you structure the test. If you’re relying on your routine project monitoring system to track how the solutions work, you can just use data from the system. If instead you are gathering new sources of data, then this will undoubtedly require more time and effort to design monitoring tools, collect the data and analyze it. As in ET, the commitment of staff is critical. Engaging high-level decision-makers early on is really helpful, as we are seeing in Indonesia (see the webinar).
Guy & Jane: Essentially, it requires individuals to introduce ET, and the commitment of agencies to practice ET. Currently, we have introduced ET through a 3-day capacity-building workshop each year for 3 years. We’re now starting to look at ways to scale up. It’s largely a mindset change, but that does not make it any easier! Changes can be small, like asking staff to adapt existing meetings to incorporate ET; for example, taking the last 5 minutes to ask, “Let’s see if we can identify assumptions we’re making.” Or, when you’re in a car leaving a project site, ensuring there is time for a reflective thinking conversation about how things went, what you noticed, and how you would’ve done things better. Again, it’s about a ‘tiny’ but not insignificant change in habit. It shouldn’t be time-consuming. We’re not focusing a mechanism of evaluation, but rather, a conversation with ET. Our ambition is to transform field staff from being mere aid deliverers into “reflective practitioners.” That’s a dream worth pursuing!
Learn more about:
Let us know if you have an idea on how we can improve SEL! Please contact Andrea Lozano at [email protected]