TOSSD: A statistical measure? Or an anecdotal quantitative narrative?
Updated: 6 days ago
About this paper
Hedwig Riegler asks whether TOSSD provides a coherent statistical measure or a collection of numbers supporting political narratives. Her paper examines definitions, the relationship between its pillars, overlap with ODA and the need for independent statistical methods. It discusses the 2024 reporting instructions.
Editorial note: The original text follows in its historical context. References to rules, proposals, figures and future events reflect the source document, rather than an update to the present day.
Click twice on the link below to open the original pdf:
TOSSD: A statistical measure? Or rather an anecdotal quantitative narrative?
by Hedwig Riegler
“The public doesn’t want figures, the public wants a story – a good story.”[1] And the public especially likes true stories. Underpinning a story with numbers gives it the appearance of truth, of facts, which is particularly important when the story is about whether and how quantitative political promises are being honoured. Calling numbers “statistical data” and the collection systems “measurement” effectively warrants that the story being told is true. But is TOSSD really a “true story” or just a plausible tale promoted with the well-known exaggerations and embellishments of advertising?
Already the DAC, with its anachronistic set-up of having political actors and considerations determine statistical methodology – a practice banned for years under UN and OECD good practice principles[2] – embarked on this route of “telling stories” instead of doing sound statistics, which has ruined the DAC ODA measure. And this has been carried over to TOSSD. Much of the DAC system, especially text from the DAC reporting directives, has been transferred to TOSSD, often watering down further what had already been identified as “weak” or overly permissive methodology in the DAC system (e.g. in-donor refugee costs). The DAC has demolished the credibility and undermined the usefulness of its own ODA figures through its politically motivated rule changes, mainly since 2016, but the numbers produced by the TOSSD system are even worse. They are not statistical figures at all but just numbers used in support of desired narratives and political aspirations.
TOSSD is characterised by conceptual and definitional deficiencies that take it far from being the “metric” claimed by its promoters. The difference? Statistical data are based on a system of clear definitions, a clearly defined universal set and consistent subsets. The data should originate from a stringent collection concept where the overall set is not so broad as to lose a clear and definite meaning, and where each subset clearly fits within the overall set, and is clearly measurable. Trying to measure “the universe” would be tackling an undefinable, overly broad concept. It has been done successfully only once, by Douglas Adams[3], and he managed to give a clear quantitative answer: 42. But that was a satire – not an answer to a serious quantitative question.
Statistical measurement is somewhat similar to physical measurement in the sense that different dimensions require different instruments. For example, the “weather” cannot be measured with just one instrument; instead, temperature, humidity, air pressure etc. need to be captured separately. What can then be legitimately considered is combining the individual results suitably to form a “weather index”.
By contrast, the so-called “metric” of “total official support to sustainable development” represents an overly broad concept whose “measurement” consists of combining the numbers within inconsistent, ill-defined subsets. For the definition of sustainable development, reporters are referred to UN political agreements. But a sound statistical system is based on its own clear definitions, especially of its main categories – and, most of all, of its quantitative variable(s). A weakness both in the TOSSD and DAC-ODA reporting instructions is that they go into details before providing a clear picture of the underlying rationale and basic concept of measurement, which requires unambiguous definitions of the core elements and not politically driven diplomatic text that leaves ample room for different interpretations.
TOSSD also combines items of such different natures that they cannot legitimately be added together into a single “metric”. Moreover, while it claims to be measuring “resource flows”, it includes contingent liabilities such as guarantees, which do not constitute a resource flow at the point of measurement and in most cases will never trigger a flow. And even if a guarantee is called, it may not involve any official resources, as the costs may be covered by premium fees collected from private sector actors[4].
The subsets of TOSSD are inadequately defined and are inconsistent with the overall TOSSD set. For example, the subset “resources mobilised from the private sector through official interventions” does not belong. The OECD was collecting these data from 2012, years before TOSSD existed, and it is clearly not “official support”. While the TOSSD site does not actually add “mobilised private” numbers to the two TOSSD “pillars”, it effectively invites users to do so by presenting “mobilisation” as a co-equal variable with the pillars.[5]
The two main “pillars” of TOSSD should also not be combined into a total. They are far too different in nature to be treated as components of a single statistical quantity. While “cross-border resource flows” can be properly measured given appropriate definitions, the second subset – “regional and global expenditures in support of international public goods, development enablers and global challenges” – is far too broad and unclear to constitute a credible statistical measure, which TOSSD purports to be. Published TOSSD figures show such a vast diversity of activities scored as contributions to international public goods[6] as to suggest that “anything goes”. This makes this part of the TOSSD dataset useless as an informative objective measure.
Probably the most problematic conceptual flaw is the almost total subsuming of ODA in TOSSD, but with definitions that diverge from DAC-ODA. There is a lack of transparency about this, with ANNEX G of the TOSSD reporting instructions only hinting at the overlap, and giving a cursory indication of the differences.[7] This is highly problematic as the largest share of data reported to the TOSSD system are the ODA activities reported by DAC member countries to the DAC-ODA system[8]. The problem is not so much that the quantitative variable used for producing “headline” TOSSD aggregates (gross disbursements) is different from that used for “headline” ODA figures (so-called grant equivalents), because gross disbursements are part of the DAC ODA collection anyway, and are used for appropriate purposes there. The real problem is that TOSSD effectively reclassifies ODA data using its own criteria. The most important difference is TOSSD’s definition of concessionality, which uses much stricter criteria than the DAC ODA criteria. This means that the same transaction ruled as concessional and therefore as ODA in the DAC system can be ruled as non-concessional in TOSSD – undermining its ODA classification. There are differences in other areas too, such as in-donor-country refugee costs, where for ODA the DAC counts only the costs in the first 12 months of stay and excludes costs for integration of refugees in the host country, whereas TOSSD includes costs beyond the 12 months and for integration.
This leads to different aid data sets in each system. While defenders of this set-up may say this is of no importance because TOSSD does not produce any “ODA” aggregates, the fact remains that when data users and analysts look for quantitative answers to highly relevant questions like “What is the level of lending at concessional terms to developing countries?”, they will get very different answers from the TOSSD and ODA systems. The same applies to the question “How much support is given to refugees in donor countries?”. This is confusing, because both systems cover a very similar topic (support for developing countries’ development vs. their sustainable development), and it is especially confusing for outside users not well-acquainted with these complex and overly complicated systems. It also heavily affects usability and mutually damages the credibility of the figures that each system produces.
TOSSD was originally intended to be a complementary measure to ODA, and this was stressed many times over by its promotors to pacify critics concerned that DAC-ODA was quietly being abolished (along with its 0.7 % of GNI target). However, instead, the TOSSD collection system has a large, but still partial overlap with DAC-ODA. If it had been designed differently, as a truly complementary set in the mathematical sense, it could have become a valuable measure to capture non-ODA finance for development. But by turning it into a measure of “official support to sustainable development” – presumably for political reasons so as to “catch the wave” of the Sustainable Development Goals – it has become so muddled that it cannot be considered a “statistical measure”. Just like today’s ODA, it is based on political, not statistical considerations.
Without going into further details of TOSSD Reporting Instructions and their deficiencies, my conclusion is that the two systems – DAC-ODA and TOSSD – cannot coexist for a prolonged period of time. For one thing, they overlap too much – thematically and in terms of the financed or “supported” activities that they cover. And since they quantitatively describe almost the same thing, they will not be taken seriously in the long run as equally valid measures of development finance: either one will prevail, or they both will crumble.
The fundamental flaw in both the DAC’s current “decadent” ODA concept and TOSSD is that they represent politically negotiated reporting standards and systems. They have been bent to meet political interests and haggled out through diplomatic negotiations. As such, they fail elementary principles of good statistical practice. And while TOSSD made a move in the right direction by including developing countries in the task force that decided on its reporting rules, this improvement in governance has proved insufficient to ensure robust definitions and a credible data set. Genuine “statistical measurement systems” cannot emerge from bodies dominated, or even significantly influenced, by political delegates, whether from donor or developing countries. While politics has a legitimate role in determining WHAT needs to be measured, decisions on the HOW of measurement must be left to impartial experts – provided that the objective really is to produce credible and robust statistics, and not just “illustrative numbers” or “data stories”.
Notes
1 Quote from a 2014 German TV Mini Series entitled “Alles muss raus - eine Familie rechnet ab”, re-broadcast by 3sat in October 2016.
2 See https://unstats.un.org/unsd/dnss/gp/FP-Rev2013-E.pdf and https://unstats.un.org/unsd/methods/statorg/Principles_stat_activities/principles_stat_activities.pdf and https://legalinstruments.oecd.org/public/doc/331/331.en.pdf.
3 In his satirical science-fiction novel “The Hitchhiker’s Guide to the Galaxy”.
4 The brief definition of “resource flows” in the TOSSD instructions does not even say that contingent liabilities are included; this fact is hidden in a side remark in a section on “financial instruments” (https://www.tossd.org/docs/2024_Reporting_Instructions.pdf, page 10).
5 See https://tossd.online and https://www.tossd.org/docs/2024_Reporting_Instructions.pdf, pages 6-7, where TOSSD is defined as “all officially-supported resources“ (para. 8) and “officially supported” is defined as including “private resources mobilised through official interventions” (para. 13 b).
8 However, the reporting overlap is not limited to ODA (as discussed here), it also concerns other developmental flows reportable to the DAC system including Other Official Flows, Officially-Supported Export Credits and Private Flows Mobilised by Official Interventions. All these are covered in both systems, with TOSSD effectively cannibalising DAC reporting. In 2022, 82% of total TOSSD was derived from DAC reporting ($360 billion out of the total of $441 billion). Most of the rest was the overly-broad “regional and global expenditures in support of international public goods, development enablers and global challenges” already discussed.
Original post image
Comments