Good stuff.  Much of my work has been focused on HIV-prevention, so I am always interested in how to address the challenges of measuring prevention in the short-term.  My experience with the demographically-driven numbers game and prevention has been that it is often futile.

Jason, thank you for the post. The question that usually arises in quantifying work in arts based nonprofits is, How does one quantify if art therapy has helped a child who is grieving over his dead father? of course, we know art helps, but how do we show it to the funder?
The donors/funders do want to get the bang out of their buck, but i feel like usually it is much easier to measure when one is working under the umbrella of "survival needs, i.e giving lunch, money for school, building hospitals, Vs. psycho/social needs, i.e art therapy, movement arts, and such?
Do you have a different criteria for such organizations?

hi jason ~
i thought this was a really insightful article. i can think of many discussions i've had about how to pitch certain programs where we get bogged down in the theoretical questions. this gives me and easy way to redirect those conversations.

one question, though. if non-profits analyze their results internally without ever asking themselves if their results are actually achieving change, (ie, if their theory of change is right) don't we run the risk of having a bunch of well-funded non-profits toiling away at useless, but "result-acheiving" projects?

Excellent post - I think that the point about *relative* performance is a particularly important one that is all too often overlooked.  As a donor, I want to know not just that my money is doing good, but that it is doing maximal good at a particular goal. Why are there so many studies into the impact/effectiveness of a particular nonprofit, but so few that compare the impact (per dollar) of several? 

As a nonprofit leader, if we're going to pay to have our organization's impact assessed, we would much rather see it assessed in comparison to the impact produced per dollar of other organizations working toward the same goals.  To me, that is the only way it becomes meaningful.  Every social goal (ie curing Breast Cancer) will have a different cost structure per 'unit' of impact, but we should be able to compare the cost effectiveness of different nonprofits working toward the same goals. 

I think that your point 3 under nonprofits
-Do research and then start tracking your proxy measures and refine as you is very important and often overlooked.

How does an nonprofit incorporate this continuous learning and not only to contruct their measures or metrics, but to improve their ongoing results? How do you use both hard data points (the tape measure) and qualitative data together?   Where do pilots, experiments, and testing come in?   And, when is failure okay?

My frame is how you use or adpat these techniques to social media strategy evaluation - so you know your nonprofit is using social media effectively. 

Thoughtful post. Thanks

This is a great post. I agree that donors need to change how they think, and that fundraisers need to change the nature of appeals. I'm sick of getting appeals that the professional in me recognizes as BS, and even sees that it's good BS likely to generate money.

The money you withhold from charities performs a vital social function: weeding out less effective groups, providing feedback, and encouraging higher standards of accountability. For every donation you give, consider telling one other group why you didn't give a donation.

"First, we need to stop acting like social scientists." 
Does this mean, Jason, that there's no role for the sort of clinical "trials" of nonprofit programs? I agree that nonprofits themselves should focus on service provision rather than analysis, but I think that social scientists can probably bring a lot to the table - I think of the dichotomy as analogous to medical researchers and practitioners. 
"Directly engage your stakeholders and clarify the outcomes they value most"
I think this is the rub. In a normal market transaction, the person purchasing and using a service are generally the same. In the nonprofit world, our model is often to have the consumer (the target audience of the nonprofit) and the buyer (the donor) separate. This bothers me often as a nonprofit manager - we could be doing a great job on the ground, but be marketing to donors poorly, and go quickly out of business. On the other hand, we could be doing poor work but have a snazzy marketing team, and be flush with cash indefinitely. I think centering measurement around the desires of our "consumers," and expending the effort to our "buyers" to the latter why that it important, is the most effective way to address that gap. 
"Analyze your grants and investments on a "cost per outcome" basis to determine value."
Like Harish, I'm a little dubious about the application of quantitative measurements to nonprofit work. I think they are essential - even in child therapy, you can come up with clever ways to measure both impact and outcome - but my growing feeling is that such measures are a necessary, but not sufficient, way for us to determine whether social enterprises are effective. 

Don't get me wrong. I believe in the work of non-profits. In fact all the groups I work with as an activist are non-profits. It is a fact known world-wide that Americans are the most charitable people in the world. That is nice! But I think it tells something else about America that most people would care to realize. First of all, the large number of charity organizations here is probably more because we have so little of people's basic needs met by our society. And having charities provide these things is NOT the way to meet basic needs! And it gives the government more ammunition to claim that the services these organizations offer is all that is needed. So we get encouragement for charities to take care of things and we have to put up with things like churches proselytizing along with the charity. I think the situation is deplorable. In so many ways I wish I had the money to donate to these charities to help the people who so much need help, but I am broke and needy too, just like so many other people I know right now. I TOO HAVE BASIC UNMET NEEDS! For me it is healthcare! These are BASIC NEEDS that government is responsible for NOT CHARITIES!

An excellent thought-provoking article. While we do our best to ensure the NGO's we work with use the funds we generate responsibly and efficiently, all we actually do is raise money and let other organisations do the actual work. What is interesting though is relevancy and customer engagement in the process. We find (not surprisingly) that the more relevant we make projects to the customer donating the higher the contribution rate, and the more engaged the customer is in the project the greater brand loyalty at the point of donation. 
If anyone is interested in firm numbers: at WorldNomads.com they have a 91% donation rate (partly because they make it opt-out not opt-in, which was a philosophical decision in the early days) and more than 60% of their customers say it was the number one reason they chose that brand over the other choices in the market.
The three primary means of measurement suggested here, accountability, effectiveness and performance resonate with our own values. We'd also add relevance to that, just because we know it works.

Hi Jason.Thanks for the post. In reading, a few thoughts came to mind:First, I was definitely caught by your use of the word "solipsistically" and agree that there is a real problem in the source and method of many forms of evaluation today. Programs tend to be evaluated from the perspective of both funders and implementers (non-profit staff and management) rather than the customers at the end of the chain. I work in global health programming for a non-profit, and we are consistently challenged to develop evaluation mechanisms that sincerely and effectively engage populations of impact while still fulfilling the mission and vision of foundations (our other clients). In many ways, non-profits are the negotiators between funders and target audiences so programs perform not at the highest levels of efficacy for either party. Rather, non-profits negotiate settlements (of impact and assessment) that are not unacceptable to either party - bridging cultural divides in the process. (In many ways, this point echoes Mark's comment above)Second, I challenge that there are times when non-profits need to prove theory. Excising organizations from measurement does not solve the issue. Those conducting programs should work in concert with evaluators or 'hard scientists' to influence evaluation priorities and mechanisms. One evaluation program that blends quantitative and qualitative assessments for leadership development programs that does a very nice job of this is EvaluLead. Blended approaches help us avoid the conundrums created when we assume that peer reviewed literature or community proven best practices are in competition with each other, rather than in concert. Thirdly, in response to 'determining value' of non-profit work, I would challenge that many of the softer roles of non-profits are not valuable on a cost-per-deliverable basis. Return to the idea of non-profit programs as the negotiators or middle men between a foundation's ideals and a community's needs. Value comes in the completion of a program that maintains and strengthens relationships and buy in from both parties. How do you measure that? My final comment is a challenge to the current satisfaction with the status quo that the food chain goes Foundations/Govt/Funders-->Non-profits-->Stakeholders. Why not challenge foundations to engage more directly with stakeholders before advising non-profits? Perhaps we could all avoid some of the monkey-in-the-middle game that reduces impact.

Is the charity The American Red Cross really making a difference? This is how I would suggest phrasing the question. Let me see recent positive results.... Hurricane Katrina.... Hurricane Hugo... Hurricane Gustave... The apartment fire displacing 20 families up the street... The Service to Military Families helping those left at home to communicate with their loved one, and getting through the military BS... The Millions of lives saved through their water safety / swimming programs... Teaching me how to render First Aid, which I paid the class fee, bought the material, but the instructor was volunteer... I then saved the lives of three people with the training, became a volunteer instructor to a Boy Scout that then without doubt saved the life of a 2 year old girl... who may one day grow up to be president(?)....Mr. Saul, how can you measure something that may well take decades to prove out? What is a successful / not successful outcome? Another example is my sister operates a charity thrift store... Her profits go to a school in Kenya.. We can measure the funds to graduation of students, but does that give us a true and fair evaluation of the success of her charity? The Kenyan Government takes a cut of the money and supplies sent... The qualifications of the teachers are bare... The variables are beyond her control, does that make her funding inefficient and the school should be shut down? However, back at home, being a non-profit she qualifies for state supported funds to pay women who are on probation / released from jail to work in her store... This may / may not be changing these ex-cons lives... What do you propose is the measurement of success? How many do not return to jail? What about measuring those that do commit crime again, can I measure the severity of the crime... Say they go from man-slaughter to petty theft? is that an improvement in their life?...
I don't get the drive to measure the success of a charity... I do fully understand the need to keep an eye on their finance books and report these numbers to the public that they may decide which one to support for the biggest bang for the buck..i.e. one dime of each dollar sent to the Salvation Army goes into their charity programs. One dime of each dollar sent to the local church goes into their charity program. ( I have been in church working behind the scenes to believe this is true across the board. They just don't post the numbers for the public.)I think there is a basic greed in people that prohibits them from understanding the true meaning of "Gift or Charity"... If you give to get something back, then you are doing so out of your own self interest, greed. IF you can give and let it totally go and fly away, then you know true charity.That is my short answer to the question.

When I think about organizations to address breast cancer, I want to support those who identify the causes, so that people, governments, other organizations can eliminate the causes.  It would be the same for organizations that deal with poverty, and other diseases and problems.  
Jason Saul is right about what are the questions to ask to research the effectiveness of an organization.

Great article. We always have to consider the ripples of the stones we throw. One person, one group, one city is not the same as the next. If a program, say, to prevent drug abuse fails in one school, it may be a smashing success in another. Instead of abandoning the program because it didn't work or continuing to present it at the failing school, we should evaluate the outcome and modify the program to fit that particular school. At the same time, we should continue the program in the location where it succeeded. Just an example...

my post katrina view
faith based charities shunned secular needs. used tragedy as opportunity to establish churches from the ruins.
red cross gave away plastic food from the parking lot of a super market food of much more useable items, but guarded by local police. maybe they should have just given away the useful food, water, etc.
the system is very screwed up

This is something I have been giving a great deal of thought to recently.  As someone who works for a NPO and is being asked only for quantitative information I think you are on to something big here.  The real problem is that the donors themselves want to hear the qualitative pice, but the gatekeeprs don't.  This article and idea, if it gewts enough traction, could have fundamental implications in the non-profit world.  Way to go!

I work for a charity and each day I see how it is working. My program helps young mothers who are often single and living below the poverty guidelines. I pair them up with mentors and provide educational programs to help them become better parents. We are also working to provide scholarships so that the women can get an education and training to find jobs and be self-supporting. Although we are a faith-based program, we work with women of all faiths as well as non-believers.

I'm writing as someone who has been very involved with our local residential recovery center for nearly 20 years, was a public defender type for many years, who has a substantial background in the social sciences and was raised by a physics teacher.  I've also been through two of the prominent rehab centers:  once for the regular stuff, and once for chronic pain.  My AA date is 8/24/87.
Something I would observe in this process, very closely, is No Child Left Behind:  it is acknowledged by many to impose a ridiculous testing regimine focussed on performance, and a means for educational materials producers to sell lots of product and "teacher training" programs.
Part of the problem is that so many people think they know so much about recovery.  There's 12 Steps, the Medical Modle, and now someone who disavows both, but is selling a book.
The prominent treatment centers, for the most part, having been through them, are obscenely posh resorts to entice patients and corporate referrals, employing a full range of medical services (effectively required by insurance companies) and serve a clientel which doesn't want much to be there.  The daily cost for each client was about $1500 when I last attended in 1997. 
There are also places which will not accept "insurance patients".  I've never been in one of those, but think they probably work better than those who only admit insurance patients.  There is another level which involves local or regional clients, have lots of paperwork to do and must hire a lot of people to fill state designed positions.  Still another that may not be licensed at all, are very local, and which can meet its monthly budget with the daily charge at the upper scale treatment centers.  They survive on local donations, an occassional state-referred client, and whatever else they can conjur up to stay afloat.  It is at this level that I have the most experience.
I think something which is almost universally overlooked or discounted is the amount of time it takes for some sort of stability to exist.  Secondly, it is largely a subjective process which the client must proceed through in his or her own head.  Being the object doesn't work.  I think these alone do not bode well for testing, especially effectiveness (meaning as cheaply as possible for the largest number possible) testing.  It takes longer for some, and for some it comes as a eureka moment, but still requires practice at living life differently -- letting things "settle in" so to speak.
I've got to break now to make sure I'm not violating some word count, and to cook dinner.  I will return, I hope.

It was my idea, as the then president, to become licensed and develop a more regular flow of clients.  Part of my concern that our Program Director was nearing retirement age, after some 25 years at the helm, and no provision had ever been made to be responsible about some sort of retirement for him.
It worked pretty well for about a year.  We spent nearly all our reserves (as I recall, we had almost 25K in the bank) getting our building up to licensing requirements, including a fire alarm system that notified the local Fire Department, which was line of site a few blocks away, and required our confirmation of the alarm signal with a phone reserved for that purpose.  That's for 10 clients in 5 rooms.  We were also required to staff 24/7, which we'd never done in our previous 20 years of existence, and which we were able to accomplish by dipping into the balance of our reserves.  They also required us to cut short our standard 6 month program to 3, then 2 months.  We balked at 30 days, they began their own program, and we stopped subsidizing theirs.  In less than 2 years, we were broke, our Director had a heart attack requiring surgery and implants, and we survived because a former resident ran the place without pay, until the community held some fund raisers and collected nearly 20K in a few short weeks.  Our town has a population of about 12K.  I don't think there were any one-time gifts in excess of $100.
So what, in that scenario, counts how, in this evaluation testing being discussed?  Is it fair for me to consider that the licensing was a means whereby competition was eliminated for the benefit of state run agencies?  How does one account the incredible community support we enjoy?  Is it, or should it be, some basis for yielding credibility to our "unlicensed" program?  Or is there something in this which indicates that 30 day wonder programs are the product of cynical governments who don't want the programs and they are only being implemented to provide verbage to the effect that they tried and had done wonderful, benevolent and humanitarian things which failed?  Just what is this "evaluation" supposed to be, really?  Evaluate what?  Does the feeling of contribution by volunteers count?  If some guy changes from being the operator of a chop-shop (and going to prison for it) to running a whole residential recovery operation does it come into any prospective count of what in this evaluation.
I'm taking another break, to cool out a bit, but, like Jack Nicholson, I'll be back.  I do have some positive ideas about what might be the subject of evaluation.

There are lots of things social scientists might do to help recovery/treatment programs.  First, figure out what they do,  develop some typologies and categories for enumeration.  Places like the one I try to help are numerous and serve as models and places where trial and error has been in effect for years.  Most are simply beyond the ambit of academia.  Since they aren't on the screen, they get categorized as fly-by-night and delegitimized.  I have long thought that they should be able to earn some money by furnishing testing subjects, but I don't know how to do that.  I believe that something measureable happens over time, but I don't know what to call it or how to measure it. They are also a place which would welcome the broad category of guest speakers, and maybe even provide a few.  It would be great if they could somehow be networked, but most are so resource poor, and their staffs are so overworked, that it will require someone from outside to do it.  It's like we provide a context to flop around in which is not constantly oriented towards obtaining the next drink or fix, and involves teaching young lads how to work and what to do in that kind of life.  This effectiveness analysis needs to figure out what they're looking for first, instead of measuring things against an unknown norm.  How to help instead of administer and require.
More later.  Woke up in middle of the night.  Back later.

Thanks to Saul and everyone posting a comment.  You have given me much to chew on.  These issues are extremely important and timely with the organizations I work with.  Please keep this conversation going.  Thanks

It seemed to me the article was more about how to convince people to send more money to organizations, instead of asking, "What is the best way for society to address this particular problem?"For example, if we could provide government-funded universal health care, we could take care of the needs of millions of people, now served by thousands of small non-profit organizations and volunteers, all funded by individual and corporate grants. It's not that I don't support non-profits, but I think in many cases they exist because the service they provide is VERY valued by a small percentage of the population. If a large percentage of the population valued it, then we should communally support it, as a part of our national social services.The fact that we communally spend so much money on our military, despite the fact that a majority of our population opposes the particular methods and venues, is so frustrating, when we look at all the valuable services for which we have to beg for funding.

These are great comments. Here are a few thoughts in response to the issues you raise:First, the notion of the “customer” has often been lost or ignored in nonprofit work.   One of the primary reasons why the sector is not functioning efficiently is because supply and demand are disconnected.  What I mean is that in the “real world” capital flows to those goods and services that consumers, who carry purchasing power, demand.  Why have there been 100 million Ipods sold but only 2 million Zunes? The rational market prevails when consumers make choices about value, and reward the firms that produce the best products or services.  But in the nonprofit sector, it’s different: “demand” for services like homeless shelters, food pantries, domestic violence counseling, etc. comes from donors, foundations and government, not the consumers or beneficiaries of the services themselves.  So the problem is that one of the most powerful metrics we can use – measuring demand – is artificially skewed.  Ideally, the money would flow to those service providers that are most highly valued by those who use them.  Instead, “demand” is proxied by funders who step into the shoes of the needy to determine which nonprofits should be rewarded. Because of this asymmetry, funders (and nonprofits) need to develop better proxies for demand. Many organizations measure customer satisfaction, but frankly, most report customer sat at 80%, so this is somewhat effete.  A better proxy for demand would be earned income revenues, % of market demand satisfied or referral rates. Easter Seals does a great job tracking these types of metrics – even going so far as to estimate what percentage of the need for disability services in a particular community that each Easter Seals affiliate meets.  Bottom line: it comes back to the fact that we are operating in a social capital market where measurement is the currency.    Second, a number of you have asked about the value of qualitative information, suggesting (correctly) that quantitative data is not dispositive.  Remember, performance data is only valuable insofar as it prompts us to ask better questions: Why did so few people complete the course? Why did our retention rate suffer last month?  Why was our cost per person served so high?  Qualitative explanations give us the answers.  So it is both that count.  But the point is that we need to put our explanations, anecdotes and stories into some context – and that’s what performance data provides.  Otherwise, we’re stuck with a bunch of random, open-ended descriptions about one organization or another with no comparative context.  Imagine if one automotive website bragged about a car – its high-performance engine, fancy on-board computer, seat heaters, etc. but had no pricing information.  The qualitative information is nice to know, but useless without some quantitative measures to help us determine value.    Finally, a number of comments inquired about the value of social science.  Don’t get me wrong: there is tremendous value to this research.  In fact, we‘d be spinning wheels (or reinventing them) daily without it.  My point is that there is a time and place for researching and proving theories of change – and this, as Paul Brest aptly points out, is primarily for think tanks, universities and research outfits like MDRC, the Consortium on Chicago School Research, and others.  When nonprofits (or foundations) hire program evaluators it rarely advances the field of understanding in social science.  It’s more what we call “CYA” (proving that the money wasn’t wasted).  Remember, proving the theory is just half the battle – the other (harder) half is effectively executing it!  That’s where measurement has its greatest value to the sector. 

It is all about getting involved. As the saying goes one person works while two help and nine watch.

Happy Inaugaration Day!

Jason, Wonderful stuff.  I understand and appreciate what you are saying.  For me putting theory into practice, effectively executing that practice, and measuring the effectiveness of that practice is where things become very interesting and valuable.  However, there is always a however, I'm not so sure that markets are ever as rational as you describe them.  David Brooks in a 1/15/09 column in the NYT discussed this over belief in rational markets.  http://www.nytimes.com/2009/01/16/opinion/16brooks.html?_r=2&th&emc=th I understand what you are saying about markets and nonprofits, but I do wonder if for-profit markets provide one with a great model of what ought to be. 

The whole idea of "profit" absolutely drives me crazy. Why does everyone  go through life trying to profit? And why do the nonprofits NOT want to make profit? Wouldn't making a profit help them to further their cause. Are they not trying to help others in some way? Why is it that greedy individuals who don't really care about the misfortunes of others are making profit, while the (seemingly) caring and thoughtful individuals who's motivation is to help others do NOT want or try to make profit?  It seems to me that if people really understood how things are inter-related they would realize that you can't profit from yourself. It is obvious that people do not understand the meaning and the scope of the word "self". Can you profit from your wife while she profits from you? No. Can you profit from your neighbor while your neighbor profits from you? No. Can you profit from anyone while they profit from you? No. So who CAN you profit from?  YOU CAN'T!!
 The only reason nonprofit organizations even exist is because for-profit organizations exist. Because some have more than they need, others have less than they need. If no one made profit, no one would need the assistance of any nonprofit. The two are interdependent.

One of the inherent dilemmas for much of the non-profit world (esp. when they are condition-driven, such as disease-related, homelessness, hunger, etc) is that underlying their mission should be the goal of going out of business because they have been successful.  Rarely at board meetings is the question asked "what do we need to do so we can go out of business?"  I have learned from my years of working in HIV/AIDS that one of the reasons we have not throttled the spread of HIV is that too many institutions are dependent on HIV for economic survival as well as cultural purpose and acceptance.  HIV is not the only condition like this.  Many are, and what they do often is try to address the symptom but not the cause. Pat Buchanan, of all people, said it best when he said "a cause becomes a movement; a movement becomes an institution, and an institution becomes a racket" (he was referring to the current state of the GOP when I most recently heard this). 

Exactly Brad Ogilvie!!! Ive been saying this for years. Its so obvious. I will even apply this to government. If the purpose of the government is anything other than trying to make it so that the government itself is no longer required ( whether that means world peace or just some form of sustainable, peaceful, civilization), I would like to know exactly what that purpose is. Wouldn't you? I will tell you this... The Cancer Society has NO interest in a cure for cancer. Just like your doctor does NOT want you to be healthy. Just like your mechanic will NEVER tell you what to do to avoid needing his services. The government will do anything and everything to keep you dependant on them. They keep us ignorant, weak, medicated, & unhealthy. And we let them do this. Most of us actually celebrate this whole process, calling it democracy and freedom. It makes me sick!

Harish, I work with Jason Saul at Mission Measurement.  Here are some thoughts drawn from client work trying to address a similar question.

The concern I hear most often from nonprofits is some version of: “How can we communicate intangible impact?”  These “soft” impacts – the confidence a child gains by finishing an art project, the community connections built by creating a mural, or (as you say) the comfort art provides to a grieving child – all seem too ethereal to measure.  Common to all of these examples, however, is attitude change.  So, the real question becomes: “How do you measure attitudes?”   
We all know that opinions are affected by dozens of inputs: parents, peers, radio, the guy collecting signatures on the street corner.  Rarely can an organization claim full ownership over an attitude change, but they certainly can contribute to it.  Ultimately, you (and your funders) want to know two things:  
1. Has the participant’s attitude changed? 
2. How much of an influence did XYZ organization have in creating that change? 
The easiest and quickest way to answer the first question is through a survey.  Ask the participants.  In the long run, attitude change will lead to more easily measurable behavior change.  It is rare, though, for an organization to remain in touch with its participants long enough to capture this secondary effect.  So, a survey will provide a sufficient proxy.  
The second question should be addressed in two parts.  First, by measuring participant behavior:  how often did they participate in the program?  for how long?  at what intensity?  Second, by research.  Many research studies and evaluations exist to prove the impact of the arts.  You don’t need to re-prove those conclusions.  Rather, you need to show your contribution to the intermediate outcomes that lead to those conclusions.  In other words, if we know that exposure to the arts gets kids more engaged in school, let’s measure how effective we are at bringing the arts into schools.  
Organizations that have less tangible missions (like arts-based nonprofits) are challenged to measure their impact.  However, for that very reason, they have an even greater need to do so.

In response, I can tell you this: A bit over a year ago I realized that circumstances were such that I was realistically facing becoming homeless, and through no fault of my own (other than merely not having sufficient resources) due to an illegal fraud perpetrated against me along with some unlawful adversarial moves by some opponents (see URL below).
I began to "seek help" and was even falsely comforted by how many non-profit agencies I found that claimed to be missioned with exactly the kind of help I needed -- that is, until I began to try to contact and avail these.
What I found, in act and fact, was that almost ALL of these wouldn't or "couldn't" actually DO anything to manifestly help me at all. In too many cases, even my contacts were ignored or dodged. Mostly, these all turned out to actually be a ring of "referrals" eventually going circular.
I'm left with the pungent impression that the large majority of these "organizations" actually are funding mechanisms for the staffs' personal lives than anything else, fueling these with "donations" ostensibly to "help" others.
And THAT is as unfortunate as it gets.