Charity's Existential Dilemma: Are We Really Making a Difference?

by Jason Saul

Published January 14, 2009 @ 07:59AM PT

The biggest mystery lurking in the depths of the nonprofit sector these days is the murky question of measurement: how do we know if charities have an impact? Frankly, with $1 trillion at stake in the nonprofit sector, measurement is a Loch Ness monster that must be slayed.  And lately, there seem to be a cavalcade of white knights reporting for duty.  Journalists, bloggers, armchair evaluators, foundation CEOs and self-styled philanthropic "analysts" pontificate solipsistically about logic models, theories of change, "Morningstar-like" rating services, sector-wide taxonomies, Zagat-guides and philanthropic "data management systems."

It's all so audacious... Unfortunately, everyone seems to be blindly whacking away at the piñata of measurement without even knowing what's inside.   And that's the bigger problem: it's not that we can't figure out the answer - it's that we can't seem to ask the right questions.  Solving this problem requires a clearer understanding of what we are trying to accomplish with measurement.

First, we need to stop acting like social scientists.  It is the job of social scientists to obsess over causation: "how can we prove that this program works?"  But that's the wrong question for anyone other than social scientists to ask.  Formal program evaluation is a research-based inquiry designed to isolate exogenous variables through a randomized control study in order to demonstrate a statistically significant correlation to the desired outcome.  Sound complicated?  It is.  Most nonprofits are not in the business of proving theory; they are in the business of improving outcomes. For example, social science has proven that students who are more interested in school have higher attendance.  A nonprofit doesn't need to re-prove that theory: it just needs to implement it effectively and measure the increase in student interest.

Second, we need to stop acting like lawyers.  Lawyers worry about risk: "how do we know that our money isn't being wasted?"  In pursuit of effectiveness, many donors, journalists and analysts ask lawyer-type questions.  For example, the BBB Wise Giving Alliance rates nonprofits on four "accountability standards": how they govern; how they spend money; truthfulness; and transparency.  The answers to these questions may help a donor weed out bad apples, but it's not going to provide much information about an organization's positive results.  Take any one of the 700 breast cancer research organizations in this country.  Is one inefficient if it spends more than 50% of its funds on overhead?  Maybe.  But what if that's because the organization has a team of medical researchers working in-house to develop new testing protocols that are brought to market faster and cheaper than outsourcing to academics?  Hmmmm.  Now maybe not so much.

By asking the wrong questions about nonprofit effectiveness, we continue to focus on the wrong data.  Most people interested in measurement aren't really trying to prove a theory or control for risk; what they really want to determine is value. Which organizations will deliver the best results for outcomes that we care about? If we want to prevent breast cancer, is our $25,000 better off with Susan G. Komen, Y-Me or Race for the Cure?  Accountability and evaluation data won't answer that question.  Most often, the real measurement inquiry is not about effectiveness (what works) or accountability (what doesn't), but about performance (what works best).

So how do we generate performance data? We need to accept that we operate within a market: a $1 trillion social capital market that consists of donors, foundations, corporations, governments and consumers who allocate resources to social outcomes.  Measurement is the currency of the social capital market.  Measurement is the proxy for value created: positive social outcomes.  Social investors need to start asking better questions - not about downside risks or efficacy (those should be a given) - but about performance and results.  Second, funders must shift their thinking from financing charitable activities to "purchasing results."  Nonprofits must also change their thinking: from fundraising to "selling outcomes."  To figure out what to measure, nonprofits must engage their stakeholders, research meaningful metrics and experiment with trial and error.  Over time, these measures will norm to what the market finds most compelling.  It's that simple.  There is no Excalibur waiting to be pulled out of a rock.

Take the example of Ronald McDonald House Charities (RMHC).  RMHC used to measure the number of houses, the number of families served and the number of dollars raised.  These seemed logical, and no one really questioned them.  But when RMHC engaged some of its key stakeholders (e.g. hospitals and McDonald's franchisees) they found that the hospitals most valued the impact that RMHC had on patient satisfaction, bed turnover and children's adherence to treatment.  Franchisees valued the impact RMHC had on consumer "trust" and employee turnover.  When RMHC began measuring and communicating these outcomes, higher revenues followed almost immediately.

The time to start is now.  Here are some practical steps that both nonprofits and funders can take to shift from social science to social capital market.

Nonprofits:

• Directly engage your stakeholders and clarify the outcomes they value most
• Align your programs to produce outcomes that the "market" values
• Do some quick research and then start tracking the best "proxy" measures you can; refine as you go
• When funders ask whether your programs are "effective" clarify what they're really looking for
• Build outcomes into your fundraising plans and get your development director into the conversation early

Funders:

• Think through what "success" means and translate your program goals into clear outcomes
• Make your outcomes visible and transparent to peer funders and to nonprofits
• Work with nonprofits to refine their metrics to be as compelling and credible as possible
• Use formal evaluations only when you are testing a new theory or program strategy
• Analyze your grants and investments on a "cost per outcome" basis to determine value

At the end of the day, we must all agree on one thing: we cannot research our way to a better world, and we certainly cannot comply our way there.  We can only perform our way to achieving the outcomes we cherish for society.

(Ed. Note: Were thrilled that Jason Saul and his team at Mission Measurement decided to launch this post here on Change.org first. We're looking forward to the discussion it inspires.)