Simon Scott: Statistical No-Nos, May 2023
Updated: Oct 23
About this paper
In this May 2023 guest paper, Simon Scott examines the statistical principles behind Official Development Assistance (ODA) accounting. He groups his criticisms of the OECD Development Assistance Committee’s rules into two areas: logical errors and breaches of statistical ethics.
The analysis considers grant equivalents, discount rates, double counting of loan risk, debt relief, development finance institutions and equity investment. It also compares the DAC’s rule-setting with international principles of professional independence, impartiality and consistent statistical measurement.
Editorial note: The full original paper follows. Its arguments, judgments and references to rules and proposals are retained in their May 2023 historical context.
Related reading: https://www.odareform.org/post/governance-of-oda-accounting-no-substantial-oda-reform-without-governance-reform
Statistical no-nos
by Simon Scott
Scores of articles over recent years have documented “ODA inflation” as donors have repeatedly altered ODA rules to make their aid programmes look more generous than they really are. But what has passed largely unnoticed are the infringements of sound statistical principles that this process has involved. In this note, I try to identify the main “statistical no-nos” committed by the DAC over recent years, in the hope that this may be useful in reconstructing aid statistics on a firmer basis.
The DAC’s statistical “no-nos” are of two kinds: logical errors, and breaches of ethical principles.
1. Logical Errors
The logical errors are mismatches or contradictions in how the DAC produces statistics.
Sound statistics arise from a sound process. They can be traced back, through the rules used to compile them, to the definition of a target variable, which is a known and understood phenomenon in the real world. If the statistical machinery is wired correctly, the number it produces quantifies some real and clearly identified thing.
Logical flaws mean the wiring has gone awry somewhere, so that the statistical apparatus produces figures that are wrong, misleading, or meaningless.
Most of the flaws now present in ODA accounting stem from DAC rule changes that contain a logical contradiction. However, many of these can be difficult to spot because they involve technicalities unfamiliar to the layman.
In particular, one needs to grasp the significance of the DAC’s switch from measuring the actual outflow and reflow on a loan to measuring loans’ grant equivalents – their estimated value as gift or subsidy. It is easy to produce exaggerated “grant equivalents” by deflating the reflows with an excessive discount rate, and this is key to understanding the problems in the new ODA measurement rules. Those problems are fully explained elsewhere on this website, so that for now we can focus only on the logical flaws involved, which include:
- Adding incommensurable quantities. In 2018, the DAC decided to keep measuring loans to the private sector on a “capital flow” basis, while switching to “grant equivalents” for loans to governments. This means that, for 2018-22, its “total ODA loans” figures mix flows and grant equivalents, and have no real meaning.
- Scoring gifts where they can’t exist. Grant equivalents are measures of concessionality, or gift-value. Non-concessional transactions have no gift-value. So it is contradictory for the DAC to seek out “grant-equivalents” in instruments that it admits are “non-concessional in nature”, as it now does.
- Double counting. In 2014, the DAC agreed to evaluate loans’ grant equivalents including the risk of non-repayment. This means that the ODA counted on every loan scores in advance the risk of eventual relief on any of them. Recognising this, the DAC promised to stop counting actual relief as a new aid effort. It later broke this promise, committing the mathematical error of double-counting -- for if you score prospective risk on all loans you may not score realised risk on any loan.
- Scoring extra ODA for expired risks. Financial risk derives from the unknowability of the future; it is the probability of (not) receiving payments. Once the future arrives, the risk expires; the payment record shows what happened. So one cannot include a risk margin when updating known past payments to today’s money. Yet the DAC does so when scoring debt relief, and now proposes to do it with equity investment.
Not all the DAC’s logical contradictions relate to its misuse of grant equivalent methodology. For example, it also now allows donors to report either:
i. their net subsidies to their “development finance institutions” (DFIs) or
ii. the amounts these institutions lend or invest in developing countries. In scoring (ii) the DAC does not deduct the interest, profits and dividends that the DFIs make. But such earnings drastically reduce their need for subsidies, so over the long run (ii) is likely to be much higher than (i) for any given DFI programme. Yet the DAC illogically asserts that “The two approaches are implemented with the joint understanding that they should generate, over time, comparable ODA figures for comparable donor efforts”.
In other cases, arbitrary caps are applied, so that once a value hits the cap, nothing more is scored. This applies both to relief on ODA loans (capped at the value of the loan minus ODA already reported) and to the proceeds from selling equity investments (capped at the value initially invested). The only logical justification for such caps would be that the underlying methods producing “cappable” figures are themselves defective.
Sometimes a single figure incorporates multiple logical errors. The DAC’s example here double-counts monies already scored, scores extra ODA for an expired risk, and finally caps a present-value figure using nominal amounts from 33-48 years earlier. And there are other similar statistical “shockers”.
The DAC is aware of some of its incoherencies, and has advanced various justifications, excuses, or promises of reform. It has justified claiming non-concessional flows if they are “additional”, which is, however, not at all the same thing. It has excused double-counting loan risk by stressing that it removes its prospective-risk scoring when it does its realised-risk scoring – yet it still retains prospective-risk scoring on every loan where risk does not materialise. It has promised to review its inflated discount rates, but not to tie them to market rates, or to remove their risk component when applying them to the past.
Meanwhile, though, the former DAC Chair recently admitted that the Committee’s rule-setting process is “politics, not statistics”. And the DAC is still trying to add more items to ODA – instruments it has never previously included in its more than 50-year history – on the main ground that not scoring them as ODA “disincentivises” their use. It has also been justifying its rule changes with “narratives”, a word which raises the question whether they are really intended to be believed, and touting “safeguards” and “monitoring” which would be redundant if ODA rules just measured reality in a neutral and unbiased fashion.
Whatever one thinks of the DAC’s line, it is not statistical. It never gets to grips with the fundamental issue of whether ODA rules ensure that numbers reported consistently record real identifiable things. The DAC’s justifications instead focus on the claimed consequences of statistical measurement; on how it might affect donor behaviour.
This is an ethical argument, but one which ignores the fact that statistics have their own ethics, which are almost the polar opposite of the DAC’s.
2. Breaches of Ethical Principles
There has been a boom in codifying statistical ethics since the fall of the Soviet empire. In reaction to the politically-motivated distortions of statistics in the old Eastern bloc countries, Fundamental Principles first developed by the United Nations Economic Commission for Europe in 1991 recognised that since “official statistics provide an indispensable element in the information system of a democratic society”, agencies needed “to honour citizens’ entitlement to public information” and
“to decide according to strictly professional considerations, including scientific principles and professional ethics, on the methods and procedures for the collection, processing, storage and presentation of statistical data”.
The DAC fails to meet these ethical standards, and this is its second big “statistical no-no”.
For a start, it clearly does not set ODA rules based on “strictly professional considerations”. The logical flaws discussed above also breach “scientific principles”, specifically the need to objectively measure observable reality. And “professional ethics” require statistics to be developed without political pressure and neutrally with regard to what is measured.
Ironically, it is the DAC’s parent body, the OECD, that has produced, in its Recommendation on Good Statistical Practice, probably the best compendium to date of the statistical ethics elaborated by numerous global and regional organisations over the last 30 years. The Recommendation was primarily designed for use at national level, but the OECD undertook that “the good practices contained in the Recommendation will also serve as a reference for the statistical work done by the OECD”.
Among the many good practices enumerated in the Recommendation that the DAC has failed to observe are:
“The professional independence of the Statistical Authorities from other policy, regulatory or administrative departments and bodies…is…ensured in practice…
Official statistics are collected, compiled and disseminated on an impartial and objective basis and determined by statistical considerations only…
Statistical release[s] and statements made in press conferences are objective and non-partisan…
There are regular and thorough reviews of key statistical outputs involving external experts where appropriate…
Official statistics are consistent within datasets (i.e. elementary data are based on comparable concepts, definitions and classifications and can be meaningfully combined).”
I could give dozens of examples where the DAC has flouted these good practices, including DAC meetings on statistics where finance ministries or DFIs effectively dictated how to score their own ODA performances; DAC members insisting on vetting statistical releases so as to insert their own political messages; twisting data to fit predetermined narratives; avoiding external review of ODA rule changes even though the UN “owns” the ODA targets being measured; and any number of heterogenous concepts, alternative methodologies, and last-minute provisional compromise decisions. But such examples would be otiose: it is clear that the DAC is a “donors’ club” with a deeply-entrenched non-statistical culture.
This culture also extends to disregarding the standards set by an international agreement to which the OECD is party: the Principles Governing International Statistical Activities. This document lays particular emphasis on “ensuring that decisions on…standards are free from conflicts of interest” and “using strictly professional considerations for decisions on methodology”. The excessive influence of finance ministries on DAC statistical discussions, and the DAC’s repeated appeals to the need for statistical methodologies to “encourage”, “incentivise” or “reward” donors, fly in the face of these principles.
It is high time that the OECD’s Council, its supreme governing body, stepped in to put an end to the DAC’s “statistical no-nos”. After all, the Recommendation on Good Statistical Practice is the Council’s own document, as well as a legal instrument of the Organisation. And the main reason for creating it was that “quality statistics are an indispensable tool for good analysis, transparency, accountability and ultimately for the functioning of democracies”. If Council meant this seriously, it must act to restore the credibility of ODA, which the OECD continues to present as one of its major statistical products.
The only alternative is to admit that ODA is no longer a statistic. This has already been suggested. A recent study advocated dropping ODA’s “neutral approach” to instead score financial instruments by their “mobilisation potential”, adding that “This would of course further transform the DAC statistical system from a measurement tool to an incentives system, but it seems clear that this has been a foregone conclusion for some time.”
The trouble with this approach is that turning ODA into an “incentives system” that eschews neutrality and that no longer pretends to be a “measurement tool” will belie its claim to reflect reality. This will violate citizens’ right to accurate public information on foreign aid, and undermine global confidence in the OECD as a source of reliable official statistics.
Original document image
