Looking to create impact?

As an organisation that prides itself on being for-impact (creating benefit for society through our work), this is what sets the bar for any of our activities. Yet as we and many other not-for-profit organizations that are now being asked this very same question have found out, measuring impact – real impact – is no easy task. But rest assured, it does get easier - kind of! Over the years of running SERES and thinking through this question, and of training hundreds of other young change-agents to do the same, we feel as though we are getting closer to being able to navigate these waters. So, if you find yourself wondering what your impact is, or what the impact of an organization or initiative that you are involved in is, we hope that these ideas – reflecting some of our own experience in SERES – may help you do that.

1. DIFFERENTIATE IMPACT FROM OUTPUT

Many organisations talk about impact when they really mean output. Output is a measure of activity, and is relatively easy to measure. For example, the number of schools that an organization has built or, as in SERES case, the number of youth that we have trained. SERES publishes our output here in a live By the Numbers dashboard because we believe that this information helps in transparency and is great for providing statistical and quantitative data. But this is not a substitute for impact. Take Guatemala and its public schools as an example: Many schools and classrooms in Guatemala, particularly in rural areas, are very basic and often inadequate for the learning needs of the children. There are also hundreds of charitable organizations and institutions that have made it their mission to build schools in Guatemala as a way of improving education in the country. But does building schools really have an impact on the education levels and standards in the country? A study by the Guatemala’s Ministry of Education in 2013 found that less than 4 percent of the country’s teachers reached minimum standards in math and quantitative skills, and that not even two out of every hundred students reached the minimum level. Guatemala has the lowest level of educational achievement in Central America, and it is clear that deep educational reform is necessary, something you can read more about here. Total number of schools built? High. Overall impact? Unknown. So, without burying ourselves in the political and social depths of Guatemala’s educational challenges, how do we know that our support will create impact? Try this: think back to your most transformative and wonderful memories in school. What stands out most? Was it the slightly eccentric teacher with their unorthodox ways who gave you your passion for geography or your love of Keats and Yates? Or was it the classroom that you sat in each day as you read those books and studied those maps? This reflection I hope has lead you to two places: first, that perhaps investing in things like teacher training, salaries for good teachers or other educational infrastructure like computers or Raspberry Pi’s may have more impact than building a school (yes!) Second, that this is complicated (yes again!) Which brings us to my next point:  

2. SUSTAINABLE IMPACT NEEDS SYSTEM CHANGE

The organisations that I believe are the most effective are working at some level on achieving systemic transformation. They may address some of the more pressing needs presented by the current system (e.g. feeding the hungry) but they will also work on deeper change on a systemic level (e.g. addressing policy issues that allow people to be hungry) or creating new systems that address these issues in a more equitable and sustainable way (e.g. food co-ops, rather than food banks). Our human societies are complex systems and systems by their nature tend to resist change. As leadership coach and organizational consultant Robert Gass explains: “Most change efforts fail because they fail to address the system as a whole.” As Megan Mukuria points out in this article from the Good Life, “to achieve lasting social transformation, a systems-change is needed.” When I am evaluating the impact of an organisation, I try to look at whether their efforts are just serving or responding to the injustices and inequalities of our existing systems (e.g. providing shelters for battered women) or also looking at ways to reform, reimagine, resist or recreate our existing system (such as providing support programs for anger management, or educational programs about domestic violence). This article on Theory of Transformation provides a great overview about how this approach works. At SERES, we use the Social Transformation Project’s Wheel of Change model to help us think through the work that we are doing that uses a three-pronged systems transformation approach:

• We use personal development and provide skills such as community visioning, strategic questioning and non-violent communication techniques to help change the ideas, perceptions and beliefs that our youth have and
• We get youth to start exercising their citizen muscle by implementing community action plans that start to change the structures and systems within their communities
• We use transformative leadership to help young people change their behaviors and become role models for others to do the same

So, all of this brings me back to the original question: how to measure impact?. We can as I mentioned measure our output (number of youth trained, number of programs run), we can even measure the metrics of our youths’ action plans (number of trees planted, number clean up campaigns) and we can even start to measure that much more difficult factor – behavioural change – through surveys and studies (SERES currently has a research student undertaking an analysis of Youth Leadership and Empowerment Towards Sustainable Community Livelihoods in Guatemala and El Salvador, results soon to be published). But none of this is actually a measure of impact.

3. TELL ME A STORY

At the end of the day, when it comes down to measuring impact, I don’t want to know about numbers or metrics or statistics. Yes, they are useful tools for helping to analyse, develop strategy and plan for the future. Yes, they tell a good story to donors and funders. But what I most want to know is: Is my work giving people the opportunity to live peaceful, healthy and happy lives? Is my work helping to transform our current systems and cultures to create a more equitable, peaceful and sustainable future? Is my work contributing towards a world that I wish for my children? They are the questions that really matter to me, and the only thing that I have found that can answer that for me are the people and their stories. I woke up this morning to this note. I could tell the importance of this message by the careful and practiced way in which Esteban (Youth Ambassador Fellow on the SERES Board) had copied it out. Having just returned from taking our research student (Eli) up to the Guatemalan highlands to do her research, Esteban was full to bursting: “I felt so inspired, so emotional watching as Eli started interviewing our youths’ parents this afternoon.” He wrote. “She was speaking with 5 mothers, who told Eli that at first they had doubts about where their children were going and all the time they were spending with SERES. Then they started to feel more secure when they began to see the changes in their children – they noticed them being more respectful, more responsible…One of the mothers told her: ”We saw them starting to plant gardens and orchards, and beginning to teach their friends. Our neighbours started talking about all they were doing. Then, they began speaking up in public whereas before they had been afraid to participate in anything”. Another of the mothers told her:

“they are mature in their thinking, and they communicate with us.” One of the mothers told a story of her son who had stopped studying, but then decided to go back. “Three teachers came to me and said that when he returned, he returned more mature, with clear ideas, leadership and initiative. He is promoting workshops such as values-based leadership and cultural events.”

“What is most valuable”, Esteban wrote at the end of his letter, “is that I can see how this change – emotional and psychological – is having such a positive impact on the participants.” Esteban’s story and all the others like it (I love this student's video of youth talking about their vision) – that is the impact that counts for me. It’s not a number, it’s not a statistic, and you can’t count it or display it on an infographic. It can't be calculated or computed. This impact is deeper, and it takes time - both to create, and to collect. Taking a walk through a village, having a cup of atol with the mother of one of your youth, or an afternoon of play and stories with kids. This impact requires going out and listening. And how do I measure it? By the size of the feeling in my heart when I do just this.