American Institute of Philanthropy - Criticism of AIP and Other Charity Raters' Methodology

Criticism of AIP and Other Charity Raters' Methodology

The major criticism from low rated charities is the claim that AIP’s rating system does not follow Generally Accepted Accounting Principles (GAAP) or rules for reporting financial information on the IRS tax Form 990. These groups posit that if AIP took the figures as reported in these financial documents, their ratings would be outstanding. While GAAP reporting rules provide guidelines for a charity to report its financial activities, these reporting rules do not measure or claim to measure how efficiently an organization is raising and spending donated dollars.

Charities have wide latitude in how they choose to report activities even within IRS and GAAP standards. In addition, a charity can spend as little as 1% of its budget on its programs and still be in compliance with GAAP and IRS reporting requirements. Direct mail and telemarketing solicitations that contain educational messages and other income-generating activities that accounting rules allow charities to report as program costs, are not considered to be program services by many donors. For these reasons AIP analyzes and makes adjustments to the audits and tax forms of some charities for consistency and to better reflect the goals of many donors who want their donations to be spent on bona fide programs.

Charities poorly rated by AIP for financial efficiency often cite favorable reviews or ratios from other sources of charity information. These other sources typically do not perform AIP’s in-depth level of financial analysis and may accept a charity’s own reporting without question. A study by the National Council of Nonprofit Associations, titled Rating the Raters: An Assessment of Organizations and Publications That Rate/Rank Charitable Nonprofit Organizations, states of AIP's ratings, "Rigorous and fair analysis of objective criteria. Does not simply repeat self-reported analysis from ."

Another criticism is that national charity rating organizations like Charity Navigator, BBB, and AIP do not rate the quality of charitable programs. AIP encourages donors to consider a charity’s program accomplishments in relationship to the resources it receives.

Studies of charity watchdogs’ methods have raised concerns about the validity of their ratings and suggest they may not be reliable sources for charity ratings. However, such studies frequently lack independence because they are often published by nonprofit consultants who are paid by the charities that watchdogs oversee or are critical of. AIP rates nearly 550 charities, whereas Charity Navigator rates over 5,400. AIP states on its website that the quality and in-depth level of analysis it performs is time-intensive and limits the number of groups that it is able to review.

Charity rating organizations have been criticized by philanthropy experts for the validity of their evaluation methods and their conclusions. A study developed by nonprofit consultants reported in the Stanford Social Innovation Review—an award-winning magazine covering successful strategies of nonprofits, foundations and socially responsible businesses—gave the opinion that some watchdog groups:

• Rely too heavily on simple analyses and ratios derived from poor-quality financial data;
• Overemphasize financial efficiency while ignoring program effectiveness; and
• Do a poor job of conducting analyses in important qualitative areas, such as management strength, governance quality and organizational transparency.

Specifically, this study gave the opinion that AIP’s shortcomings include a “gotcha” mentality and lack of transparency, saying that “AIP is not afraid to fail an organization; in fact, they specifically aim to review nonprofits they feel aren’t spending wisely or performing ethically, to help educate the public.” The study quotes AIP president Daniel Borochoff saying, “We’re really looking at the numbers and what they mean, not just running 990 inputs through an equation…At times we actually find that a nonprofit is selling itself short in the way they report numbers, and help them fill out their 990 more accurately, but more often we see nonprofits misleading potential donors with the way they report their financials. You have to ask yourself why the other rating organizations aren't seeing the same really bad things going on with the numbers at some of these other charities.” With respect to AIP’s transparency, the report states that “A donor sees the score but only limited explanation, which can cause more harm than good.” The study goes on to say that “While it is true that nonprofits have wide latitude in completing their 990’s (and many go to great lengths to misrepresent their financial information), it is difficult for a donor to understand what specific adjustments AIP made to a given nonprofit’s rating and why.”

On the positive side, the report states that AIP “recognizes the limitations of the 990 and thus develops its financial health ratios by analyzing a charity’s audited financial statements. AIP’s small staff of analysts looks closely at specific calculations, including how nonprofits allocate telemarketing costs, which are often labeled 'education and outreach,' and in-kind contributions which they assert are overvalued, among other practices they think nonprofits use when preparing their 990 to cast a more positive light on their financial position."

The study suggested:

A more effective nonprofit rating system should have at least four main components: improved financial data that is reviewed over three to five years and put in the context of narrowly defined peer cohorts; qualitative evaluation of the organization's intangibles in areas like brand, management quality, governance, and transparency; some review of the organization's program effectiveness, including both qualitative critique by objective experts in the field, and, where appropriate, "customer" feedback from either the donor or the aid recipient's perspective; and an opportunity for comment or response by the organization being rated.

A second study, Rating the Raters: An Assessment of Organizations and Publications that Rank/Rate Charitable Nonprofit Organizations, provides a separate assessment of AIP, Charity Navigator, Better Business Bureau Wise Giving Alliance, and other charity information services. The major findings are:

• Approaches and criteria are not the same. The methodologies and criteria used vary significantly among the various rating and ranking organizations.
• Evaluation criteria may not be readily apparent. Not all nonprofit rating and ranking groups make it easy for the donor to determine the evaluation method and criteria used.
• Evaluators may use criteria that are overly simplistic. Simple financial ratios and/or measurements that apply in some circumstances may not apply in others.
• Evaluators focus on financial measurements and overlook program effectiveness. Financial "efficiency" is assessed by most third-party ratings groups as a percentage of contributions received. This tends to be their primary focus.
• Competence of the evaluator is critical and difficult to determine. It is virtually impossible for donors to determine the relevant credentials, expertise and experience of the rating organization's staff.
• Evaluators often derive revenue as a result of their rating reports, creating a potential conflict of interest and questioning whether these groups are motivated by the desire to inform potential donors or by the media attention that improves their revenue stream. AIP, for instance, charges a $3 fee for a sample copy and requires $40 membership, which it uses to fund research, as a condition for receiving its annual rating reports. The BBB charges charities it reviews up to $15,000 to use its charity seal to publicize their ratings.

Some groups criticized by AIP, such as Paralyzed Veterans of America, have pointed out that they meet "all 20 criteria that the Better Business Bureau Wise Giving Alliance establishes for charities, including that a charity's fundraising costs not exceed 35 percent of contributions, a common standard." The Better Business Bureau Wise Giving Alliance charges charities to use its seal of approval.

AIP appears to agree with many of these criticisms, cautioning donors on its website to be wary of highly automated or overly simplistic ratings that do not provide an adequate analysis of a charity's activities. According to AIP, its analysts "dig deep, carefully scrutinizing the individual finances of charities to give donors a clearer understanding of how their cash donations are being spent." AIP also claims that its ratings are the most stringent in the sector; that it is fiercely independent because over 95% of its support comes from small, individual donations; that it uses reliable information and treats charities consistently and fairly; and that it rates charities that other raters won't.