After 10 years of wear, I finally had to admit that my favorite Merrell clogs needed replacing. So I went online to Zappos.com to find a new pair. I am a big fan of Zappos, and not just because they have flattered me with a VIP membership. I can sort by size, color, brand, even heel height, to narrow down the 1,601 clog choices to a few.
But my favorite feature is sorting by customer rating to find the shoes other shoppers liked the best.
After about 15 minutes of shopping, I decided to go with a pair of Clarks because, in addition to the 5-star rating they received, one of the customers is a nurse and said they are comfortable all day long.
I have all that information on an $80 purchase.
But what if, instead of shoes, I wanted to invest $80, or $80,000, in a charity working in international aid? What information is available to me? Not nearly enough, in my opinion.
A lot of donors rely on the information provided on sites like Charity Navigator. Charity Navigator’s main goal is to inform potential donors about the financial health and “efficiency” of a charity. The efficiency rating comes from financial data in the annual tax forms most US nonprofits have to file every year with the federal government.
While the Charity Navigator ratings may tell you what percent of your donation goes to fundraising or how much the charity’s CEO gets paid, it will not tell you anything about what the charity actually accomplishes in the field. That’s why the Central Asia Institute, co- founded by Three Cups of Tea author, Greg Mortenson, had the highest (4-star) rating from Charity Navigator until media attention exposed questionable spending and alleged exaggerated accomplishments.
Hundreds of stories were written about this particular scandal, but there are many more less extreme examples of organizations out there who may look financially healthy and efficient, but actually have poor to mediocre programs in the field.
I was recently asked by Philanthropedia (owned by Guidestar) to be part of an expert opinion poll on which water and sanitation organizations are the best. I declined to participate in this crowd-sourcing exercise, because I don’t think it’s fair or accurate to evaluate an organization based on what I’ve heard or read about them. I base my opinions on actual field analysis. While I appreciate that they are trying to provide a more complete picture of a nonprofit’s activities than Charity Navigator, the result is what I would characterize as a popularity contest, not an independent assessment of a charity’s work.
Even Zappos reviews come from people who have actually worn the shoes.
Most donors ignore these rating systems and base their donations on information provided by the charity themselves. That’s the case for the majority of the supporters of the organization I work for, Water 1st. And while I believe Water 1st’s self-monitoring program works wells and fosters an environment of continuous program improvement, I realize that we are essentially filling out our own report card.
So, why shouldn’t you trust what charities report themselves? Well, I certainly can’t speak for all international aid, but I do have over 20 years experience evaluating global water and sanitation projects and here is what all donors to this sector should know:
It’s really hard to do a water and toilet project. It is estimated that 35-50% of projects stop functioning only two to five years after completion. There are many reasons that water projects fail, and none of them are that organizations who implement projects are not well-intentioned. Usually it’s because communities have not or cannot assume responsibility for maintenance and repairs.
Project follow-up is almost non-existent. I believe the underlying reason for the failure of water projects is that organizations that implement them are not investing in their own learning. While there is little information out there on how many organizations actually visit their projects after the ribbon-cutting ceremonies, expert opinion is that less than 5% of projects are visited post-construction and less than 1% receive any long-term monitoring. Follow-up allows organizations to improve the planning and execution of future projects, take any corrective action necessary to improve the functioning and impact of existing projects, and build a knowledge base of experience. While GPS technology and Google maps are very cool fundraising tools to connect donors with specific projects that they cannot visit themselves, placing a GPS point on a map at project completion does not count as monitoring and evaluation.
These failures mean that billions of dollars are wasted each year, forcing women and girls to return to the drudgery of carrying water from distant, contaminated sources. If my new shoes aren’t comfortable, I will simply send them back. But, poor people don’t have the luxury of the Zappos return policy – they rarely get a second chance at a major development project.
Nothing I’ve written here is news to those of us in the international water and sanitation sector. We have been quoting these same statistics and challenges for the past 30 years. Intellectually we all understand the need for better follow-up, but other activities – like fundraising – take priority.
I believe that until demonstrating projects are long-term successes impacts an organization’s bottom line, implementers of water and sanitation projects will continue business as usual.
In December, Water 1st piloted this idea of an independent audit of water and sanitation project outcomes in Honduras. We hope it will capture the attention of donors who are interested in having more information on which to make their philanthropic decisions.
About $10 billion is spent annually on water and sanitation projects. If half of these projects are failing, the potential of independent audits in this sector alone is enormous, in terms of steering donations to the best programs, and also incentivizing organizations to improve their performance.
The rating system Water 1st envisions for these independent audits is a little more complicated than the one provided by Zappos. But we would like the results to be as easy to find and interpret by donors. We can also imagine independent outcome-based rating systems being useful in other sectors, helping compassionate donors to make informed decisions about their support of programs to improve education or healthcare (and preventing organizations from making false or exaggerated claims).
Ultimately, we want all the money spent on international aid to be spent wisely. After all, shouldn’t development projects for the world’s poorest last at least as long as my Merrells?
Marla Smith-Nilson is the founder and Executive Director of Water 1st, a Seattle-based non-profit devoted to helping the world’s poor meet their needs for safe water. She has overseen the implementation of over 600 community water projects in Latin America, Asia and Africa benefiting 250,000 people. Water 1st has not been rated by Charity Navigator, which only rates charities with more than $1,000,000 in total revenue.
Related post on Humanosphere re: “Day Without Dignity”
http://humanosphere.kplu.org/2012/04/its-day-without-dignity-day-skewering-kony-2012-toms-shoes-and-other-bad-aid/
Related post on Humanosphere re: “Day Without Dignity”
http://humanosphere.kplu.org/2012/04/its-day-without-dignity-day-skewering-kony-2012-toms-shoes-and-other-bad-aid/
Tremendous article, Marla. I look forward to seeing if this type of evaluation becomes a standard in your field. I know Water 1st will score highly on it!
Thanks, Jim!
I agree with the theme of this article, but their proposed alternative (from above link “this idea” http://water1st.org/waterlog/?p=2449&cpage=1#comment-4774) is not sustainable in itself. They propose Accountability Forums, which the first one they tried (low hanging fruit, their own partner organization where data is extremely easy to gather) took at the least 12 people’s time for one week just to evaluate one organization. This could not be done on the long-term for as many water aid organizations that are out there, or at least not without monumental effort.
However, honestly, projects can be completed overseas, several other projects can be followed-up on in a week, and without 12 people’s time included. I propose that a more efficient system be developed that measures long-lasting deliverables as well as financial decisions.
John – You raise a lot of good points and not ones that we could cover in the blogs you read, so I’m now hopefully not going to overwhelm you with a response :).
I agree with you – the Forum itself will not be sustainable in the long-term if it is run like the pilot. The questions we wanted to answer with the pilot Forum were the following:
1) Will people come? Are organizations were interested in this idea of independent evaluations?
2) As a group of Forum participants, can we actually agree upon a set of outcome criteria for water and sanitation projects?
3) Can the 3rd party evaluators collect sufficient data in 3-4 field days to make an assessment based on the outcome criteria proposed by the group of organizations participating in the Forum?
We were pleased with the results of the pilot Forum because the answer was “yes” to all 3 questions.
In 2012, we are working on the long-term sustainability questions which you mention. There are other models of voluntary regulation programs or “Accountability Clubs” out there. We are continuing our research of those models to see what might make sense for the Water and Sanitation Accountability Forum in the long-term. We want to address issues like rules for membership (membership fees, how often are organizations assessed, etc.), how are standards proposed or changed, and what information is published about an assessment.
We certainly do not expect that future Forums would require the participation of 11 organizations in one assessment. That would definitely be an unreasonable expectation, especially given the field visit format of these evaluations. Personally, I think representatives from 2 peer organizations per assessment would be reasonable, and that could be part of an organization’s responsibility as a member – to commit a certain amount of time each year to peer evaluation. For the December 2011 evaluation of our partner organization in Honduras, there were simply many organizations interested in seeing how the pilot Forum worked, and we turned no one away. There is a benefit to the peer organizations – most of them sent field staff who rarely (or never) get an opportunity to see another organization’s work in person. That kind of experience-sharing was very valuable to all of us who participated.
Currently organizations like Water 1st pay about 1% of our budget for independent financial audits. Water 1st is not required by law in Washington state to have an independent audit. We do it because we want to have that information to share with our donors (and they expect it). Thus we think the funding for future assessments (mainly to pay consultants to do the independent evaluations) could come from “membership fees” based on organizations wanting to provide more independent information to donors and potential donors. I would certainly rather spend money on an outcome evaluation of our projects (which is where over 80% of our money goes) rather than the current system of financial audits which really only tells donors about 20% of our expenditures – the ones use for our overhead and fundraising expenses.
For the pilot, you are right – our own partner organization was the first evaluated. We believe in this idea enough that we were willing to put our own work on the line for evaluation to show how the process would work.
In planning our work for the Forum in 2012 and beyond, we have made lists of success points and failure points in this process. Certainly a failure point will be if we can’t convince another organization to be evaluated. Although we haven’t scheduled the next Forum yet, the interest in this idea is very strong, especially from organizations that witnessed the pilot. Therefore, I feel very confident that another organization will step up to the plate for Forum #2 and beyond. So stay tuned.
Also, while I’m going on and on here, I should say that our intention is not for this to replace self-monitoring of programs. Although my experience, and the experience of many others, is that very little self-monitoring actually occurs in this field. Self-monitoring and creating an environment of constant program improvement is a very important piece to an organization’s whole monitoring and evaluation program. And, like you mentioned, it’s pretty easy to evaluate completed projects at the same time you are looking at new ones. Many organizations say they don’t monitor because it’s too expensive, and I just don’t buy that argument. At Water 1st, we have a relatively small budget but we are still able to comprehensively monitor our programs by consistently funding only a handful of in-country partners, rather than spreading ourselves too thin or only participating in short-term contracts, so that we can have an impact in a region and be cost-effective.
The training of locals at various age levels (teenage, mid 20’s, mid 30’s, ) to make sure the system stays viable seems more economical. If the locals don’t have a stake in the project, it will get broken.