A Brave New World of Social Impact Data

by Steve Wright

category: Impact Assessment and Measurement

Published July 14, 2009 @ 08:30AM PT

(This post was written by Steve Wright, Director of Innovation at the Salesforce.com Foundation. We're having some technical difficulties but will correct the byline as soon as possible)

It is my intention to provide a practical explication of systems for social impact data collection and reporting. I will try not to make it as dry as it sounds.
The Direction in Which we Must Err

I believe that despite the enormous odds which exist, unflinching, unswerving, fierce intellectual determination, as citizens, to define the real truth of our lives and our societies is a crucial obligation which devolves upon us all. It is in fact mandatory.

If such a determination is not embodied in our political vision we have no hope of restoring what is so nearly lost to us, the dignity of man.

* Harold Pinter Dec 7th, 2005 Nobel Prize for Literature acceptance speech

I love the above quotation from Harold Pinter because, for me, it clears the fog. It makes clear the "why". Why do we do this work? Why does it matter? Why do we care? I believe that it is important for us to make all of our mistakes in the direction of greater social impact, that we share too openly, that we place our humanity in front of our economy, that we transact specifically as our contribution to evolution.

Declaring the change you want to see

Measuring social impact is not like measuring height or counting dollars. I tried to address this in depth in my last Change.org post: Reimagining "Value" For A Post-Crisis Economy. Because there is no standard yardstick, one must be created for every ecosystem. There are a number of ways to do this. Creating a theory of change is one way. Strong alignment around a vision and mission can be effective as well. Without going in to detail, a theory of change creates a vehicle that moves an organization's daily actions towards meaningful outcomes.

Below is an example of the theory of change for the Center For Employment Opportunities. It starts with a declaration of what they want, "the formerly incarcerated maintain freedom and gain entry into the legitimate workforce," and then it details how they will make that happen and what it will look like when they get there. Knowing where you are going makes it more likely that you will recognize the destination.

Theory of change Resources:

The University of Wisconsin has some excellent resources including this page of concrete examples.

The W.K. Kellogg Foundation has created a "Logic Model Model Development Guide (pdf)" which is very useful.

A Brief Case Study
In an effort to ground this in a specific example I have invented a short case study. This is loosely based on a reality.

Sanitation Stations: Imagine an organization in Dakar working to decrease disease by building urban sanitation stations. Their theory of change points at the following outcome: "A healthy Dakar, better able to address its challenges". They have a marketable, revenue producing product with residents and municipalities receiving direct benefit. The business model involves some government capital for construction, individual per use payment and some advertising opportunities on the walls of the facilities. They need start-up capital.

Return on Investment: There are several entities that are interested in investing in this social enterprise. All of whom are looking for a return on their investment (ROI). ROI, simply put, is what an "investor" gets for their money. Major Financial Institution, inc. is investing in order to get a near market rate return as part of their emerging markets fund. They want a financial return for a financial investment. Philanthrocapital Foundation is investing as well and they expect a social (as opposed to financial) return on their investment. Specifically, they are interested in decreasing mortality from preventable disease. Other organizations may invest hoping to receive a blended return, below market rate financial return offset by a measurable social impact, a social return.

So, it is easy to understand how a financial return is provided. The reason that I am writing this post is to attempt to clarify how a social return can be provided.

Proxy Data: Central to the arithmetic of calculating a social return is the concept of a proxy. Going back to our example:

1. A financial investment was made in an organization whose goal is to decrease disease.
2. That organization was able to create enough of a profit to sustain themselves and repay their investors.
3. That organization is likely to be good at decreasing disease.

Sanitation Stations' ability to create a profit is a proxy for their success. Their success is decreased disease. The tighter the connection between the proxy data (the output) and the organization's success story (its outcome), the more effective the proxy will be. Proxy data is necessary because outcomes are rarely quantifiable. There is an art to this however. The social sector can never (and never should) replace our narratives with equations. Our stories make us resonant, out proxy data makes us accountable.
Two Markets
As I see it, markets represent the physics of social change. Understanding their rules is critical to approximating an empirical understanding of social impact. At the moment, in the social sector, we operate in two disparate, conflicting marketplaces. The first is characterized by transactions and competition and the second by ideas and collaboration. It is my belief that we need to find synergies between these two markets if we are to create lasting global solutions. To a large extent, in the same way that an individual organization focuses on the change that it can make, the role of the market place is to focus on the change that can be made globally by leveraging the strengths and weaknesses of the network as a whole, both the nodes and the connections between them.

Competition

This is a huge topic to which much attention has been paid. I don't claim to have any new insight other than to suggest that competition and specifically capitalist competition, while a powerful engine, is a means and not an end. Social sector organizations must compete to earn currency which they use to run their businesses. The more they earn the more they can create breadth or depth. Additionally, in the context of social enterprise, a winner is an organization that has the most social impact, the organization that comes closest to solving the problem. This winner is not necessarily the organization that makes the most money. In this context, the only way to know who wins is with reliable, comparable social impact data. In this frame, all of the importance is placed on the nodes of the network and the connections between them are largely ignored.

Collaboration

It is easy to define a market in a competitive frame. Even in a context where the data is obtuse or unreliable a competitive market defines winners via their individual ability to accumulate through "selling". She with the most, wins. Conversely, a collaborative market focuses on the buyer and creates winners by strengthening connections. This feels foreign to me as I write this. The concepts of marketplace and collaboration feel mutually exclusive. However, if we are to solve the world's most intractable problems we must collaborate. A fundamental assumption here is that the whole is indeed more than the sum of its parts, that collaboration will actually provide value more effectively than competition can alone. Creating High-Impact Nonprofits from the Stanford Social Innovation Review documents this to be true at least sometimes.

Sustainability and Efficacy

I believe that both marketplaces are needed in order to create resilient organizations that can collaborate effectively to create solutions. And we need to measure both the efficacy or strength of individual organizations as well as the quality of the connections between them if we are to understand and optimize our collective progress towards solutions.
The Social Impact Data Cycle
A theory of change defines outcomes. The social sector marketplace defines the physics for success and the context in which we must operate. With the final section of this post I will focus on the details of the data itself. I have borrowed my understanding of the operational dynamics of social impact data from many people. I believe that there is a forming consensus that social impact data must be thought of as an iterative cycle. The purpose of this cycle is to move data from well designed operational environments that provide both efficiencies and introspection, through sector wide aggregations exposed to insiders to define baselines and benchmarks and to external audiences for new crowdsourced insight which is then fed back into the cycle to tweak best practices and improve design.

Design and Collect

In order to produce accurate, complete and insightful results, social impact data collection must be incidental to and deeply integrated in the daily operational context of an organization. Traditionally, the best way to collect good data has been through intrusive and expensive external evaluators. Organizations that didn't have access to these resources collected data either periodically (annually, quarterly,...) or didn't collect social impact data at all. What is becoming understood as the best practice is to bake social impact data collection in to the daily process of running an organization. This is why design is so critical. Good design will both reduce friction and increase visibility.

The Adolescent Pregnancy Prevention Program of the Children's Aid Society provides a great example of how this works. They have defined program attendance by teen mothers as "dosage" and they can equate compliance with recommended dosage with positive outputs that have compelling links to their desired outcomes. This screencast describes their model with very specific examples of their operational environment.

Report and Aggregate

If we are to solve the world's most intractable problems then operational environments must provide us with the ability to be introspective about our organization's ability to create impact. Then, the stories that we tell about the impact that we make must resonate across the sector. This is accomplished with good design and operational practices that provide a fertile data repository coupled with powerful, ad hoc reporting tools that allow individual organizations to mine their own data.

Additionally, if collaboration is important, we also need to find expedient, intelligent ways to publish that data to locations where it can be aggregated with data from similar organizations. This will enable us to create baselines as well as define benchmarks and milestones. This is difficult and primarily political work. The landscape is littered with failed attempts to 1) describe the elements to be measured and 2) sustain a collaborative ecosystem. It is my belief that concentrating on a subset of discrete proxy metrics, as opposed to the super-set, is the key to success. What this means is aggregating only the small set of output data that nearly everyone in a particular field already collects. As a collaborative matures and trust develops, new outputs and compelling narratives can emerge. Additionally, it is worth noting that each aggregation effort is likely only relevant to a specific "industry". Discrete proxy metrics are not transferable across industries.

A great example of this type of collaborative is the PULSE / IRIS project. PULSE won a 2008 Fast Company Social Capitalist award. It was built through a collaboration with the Acumen Fund, Salesforce.com Foundation and Google.org. PULSE is a data management environment for social investors like our imaginary Better World Fund from our social investor case study above. The purpose of PULSE is to facilitate the process of setting specific metrics (financial, operational and social impact metrics) that map to intended outcomes for each investment. IRIS is a set of standards that have been agreed upon across a large group of social investors. These metrics apply to the performance of investees and include things like revenue growth, local compliance, customers served, ... The design is to use PULSE to manage an investment portfolio and to export that data on regular intervals to the IRIS standards aggregation point. Over time the collaborative will be able to benchmark against their peers as well as answer questions about the health of the economy in specific regions or about their impact in specific verticals like education or housing.

Inspire and Design

We need to give up on creating demand and concentrate on discovering value. I believe that, at this particular moment in history, insight is more important that initiative. In an era where philanthropy has a long tail and each of us can find our personal niche or our philanthropic passion, choice has the potential to eclipse impact. We need more accessible, mashable, well documented, semantically defined data. With broadly accessible social impact data we can crowdsource insight and facilitate unpredictable explications and fresh perspectives.

A fantastic example of this is the GapMinder website. My interpretation of the Gapminder theory of change (based on their mission statement) is to create a fact based world view through unveiling the beauty of statistics. Hans Rosling and crew have created dozens of screencasts that explicate various data sets, mostly health related. The video embedded here attempts to shed light on our progress in the human rights arena. It is less precise, muddier, than most GapCasts. It is muddy because that is the nature of the work; the math is really hard; however, he made the screencast anyway. Because it was important.

From the GapMinder site:

In this video, made for the Oslo freedom Forum 2009, Hans Rosling discusses the difficulty in measuring progress in Human Rights in the form of comparable numerical statistics. He also shows the surprisingly weak correlation between existing estimates for democracy and socio-economic progress.

The reason may be that democracy and human rights measurements are badly done. It may also be that democracy and human rights are dimensions of development that are in themselves difficult to assign numerical values. But it also seems as much improvement in health, economy and education can be achieved with modest degrees of human rights and democracy. Hans Roslings concluding remark is that Human rights and Democracy maybe should be mainly regarded as values in themselves rather than means to achieve something else.

We have a lot of mistakes yet to make, but as Harold Pinter said, "I believe that despite the enormous odds which exist, unflinching, unswerving, fierce intellectual determination, as citizens, to define the real truth of our lives and our societies is a crucial obligation which devolves upon us all"