When you scroll through the posts featured by social enterprises and nonprofits in your media feed right now, you'll find that many look similar to the post below, speaking to what was accomplished in 2020 as a result of your support. If they have a measurement system in place, they are using numerical data to demonstrate their good work, perhaps sharing the number of individuals benefitting from their services or the percentage by which their programs have grown.
In doing this, the organization shows they have diligently applied program measurement techniques throughout the past year (see our previous article), and can therefore point toward quantifiable data that reinforces their good work.
While communicating the number of individuals impacted by a program is important, it is critical that we not stop here. If we wish to build a compelling case in support of our valuable mission, we must do more than tell supporters how many have been affected by our programs; we must also share the way in which participants have experienced positive change as a result.
Communicating that an individual has been the recipient of a program or service does not necessarily prove that the intended outcome of that program or service has been realized in that person's life.
In order to measure the overall effectiveness of our well-intentioned efforts, we must commit to the more involved and time-consuming process of program evaluation.
According to www.urban.org:
“Evaluation tells the program’s effect on the people, families, or communities it is serving, that is, whether a program is producing results or having [the desired and sought after] impact.”
Evaluation is by far the more intensive component of the M&E process. When diligently pursued, it allows us to:
Create a compelling case demonstrating the actual NEED for our work
Determine the effectiveness of each program and initiative by evaluating real impact
Allocate our limited resources most effectively
Identify areas for improvement
Justify the addition of new programs, products, and initiatives
Communicate our progress and deficits to both customers and funders
Evaluation is critical if we want to be sure that our social impact company or nonprofit organization is progressing in its mission, generating the desired impact within our community, and stewarding resources in the most efficacious manner.
Ok, so how do we do it?
Hint: Evaluation requires more than a Case Study!
In an effort to demonstrate program efficacy, many grassroots organizations rely on case studies to validate their success and justify their call to action. If you are unfamiliar with a case study, it is most often a heartwarming testimony given by a carefully selected program recipient who has had a positive experience with the organization and achieved some level of life-change as a result. Sounds pretty good right?
It is! In fact, there are several benefits to sharing case studies. They are often:
Qualitative and motivating on an emotional level
Meaningful in peer-to-peer fundraising or individual selling opportunities
Engaging as a component of an impact report or newsletter
Encouraging of our own hearts to keep going
Relying solely on case studies as your method of program evaluation is problematic because:
They are essentially 'cherry picking' and misrepresentative of an entire program
They are not random and therefore considered biased
They are not a broad enough sample to demonstrate overall effectiveness, impact, resource allocation, nor areas for improvement
They will not be accepted as a basis for grant applications and large-scale investments
They do not allow YOU to know if what you are doing is having the desired impact
This is why we must employ a more systematic approach to our program evaluation. While this may seem initially intimidating, it is possible to start small and grow over time!
The key tool you will be utilizing for evaluation purposes is a SURVEY, employing these 3 steps:
First, establish a baseline. Whenever possible, survey your program participants and clients prior to engaging in your programs. OR survey a similar demographic in your community that has never been a recipient of your programs. In some instances, you may ask participants to remember back to their circumstances prior to entering your program to establish baseline, but this is not a best practice.
Second, measure against baseline. Give the SAME survey (or very similar) within 6-12 months after your program is complete (and possibly again at 18-24 months) to the SAME constituents in order to evaluate change.
Third, report accurately. Communicate how many of your original group you were able to follow up with. If your program serves less than 100 at a time, you will need to try and survey ALL of them to reduce your margin of error. The smaller your sample size, the larger statistical margin of error. If you serve thousands, you can survey a much smaller percentage for your sample group and achieve a relatively accurate representation of your larger impact.
When designing your survey:
Consider the outcomes you are hoping to achieve
Keep the questions short, clear, and factual
Err on the side of offering "yes/no," "multiple choice," or “check all that apply” questions
Save the bulk of your open-ended questions for case studies
If we were to create a survey for the purpose of establishing baseline and determining felt need for the homeless population served by our fictitious organization, Blankets for All introduced in Part 1 of this article, we could give the following survey:
Baseline Survey: 200 Homeless Blanket Recipients in December 2018
We would want to conduct a similar survey a few months after the blanket distribution to evaluate impact. In addition to asking the first 7 questions, we would also include a few more. We would do everything in our power to find the original individuals surveyed, understanding this can be tricky with transient populations.
Program Evaluation Survey 100 Homeless Blanket Recipients May of 2018:
We would then track our answers in a spreadsheet and use percentages to quantify our reporting, resulting in the following impact statements:
Of the 200 recipients of Blankets for All blankets, we were able to survey a sample of 100 individuals. Of these100:
75% reported better rest at night because of having a warm blanket
95% reported feeling cared about by their community
This recent evaluation would allow our newest and most compelling media post to read:
Post 3:
The number one request in a survey given to friends experiencing homelessness residing in our community is for clean blankets to keep warm during the winter. With COVID limiting capacity of our shelters by 50%, many of our friends are spending cold nights out of doors. 38 of you volunteered 80+ hours to help us share love and warmth with 580 of our neighbors living on the streets this month. Of the 100 blanket recipients we surveyed, 75% reported getting better rest at night because they stayed warmer. Additionally, 95% of our friends reported feeling cared about by their community and the interaction they had with their Blankets For All Volunteers! As one of our recipients, Dan, told us: “Receiving a warm blanket was the best thing that happened to me this month. Not only did it help to keep me warm outside, it kept me warm inside just knowing that someone cared for me.”
This sample media post demonstrates how conducting a simple baseline and follow up survey verifies the intended impact of our programs and establishes a compelling case for donor and volunteer engagement. Not only that, it affirms that we are in the right line of work; providing an impactful service and using resources effectively. And if by chance it doesn't, well, this is also good information for us to have discovered!
Applied intentionally, negative findings and less-desirable outcomes can allow our organization to redesign programs, reallocate resources, or move in a new direction! All of this helps to ensure that we are operating efficaciously and being good stewards of what has been entrusted to us.
If establishing a measurement and evaluation system on behalf of your social impact company or nonprofit is a goal of yours in this new year, Neema can help! With nearly a decade of practice in M&E, we can consult on survey construction, systems, and data interpretation for use in creating the meaningful content your organization needs! Simply fill out our contact form for a free consultation and together we can explore your organization's evaluation approach.