Policy-making, the challenges of reductionism and the role of evidence

In the context of a discussion on kitemarks for policy, Claire Melamed of ODI mentioned the kitemark, tweeting “national resources still have to be allocated [between] diff[erent] things. some way of comparing impact v imp[ortnant]” (helpful filling in, my own). This I thought was interesting and triggered this rather long post. Reading the tweet again, it struck me that it could be taken in two ways: the less extreme interpretation says that evidence is important to understand impact, and inform judgements; the more extreme argues for boiling down the policy-process into a comparison of evidence, one predicted intervention against another.

The latter interpretation would be based on a very bullish view of the value and capabilities of research. It reminded me of a hypothetical decision-making process described in Charles Lindblom’s article back in 1959 (Public Administration Review, 19:2, 79-88), the article which  as far as I know really started the examination of the use of evidence in policy (it’s worth replicating ):

“[A policy-maker] might start by trying to list all related values in order of importance … Then all possible policy outcomes could be rated as more or less efficient in attaining a maximum of these values. This would of course require a prodigious inquiry into values held by members of society and an equally prodigious set of calculations on how much of each value is equal to how much of each other value. He could then proceed to outline all possible policy alternatives. In a third step, he would undertake systematic comparison of his multitude of alternatives to determine which attains the greatest amount of values. In comparing policies, he would take advantage of any theory available that generalized about classes of policies… Finally, he would try to make the choice that would in fact maximize his values.”

Lindblom makes clear his attitude to the feasibility of this approach clear:

“For complex problems, [this approach] is of course impossible. Although such an approach can be described, it cannot be practiced except for relatively simple problems and even then only in a somewhat modified form. It assumes intellectual capacities and sources of information that men simply do not possess, and it is even more absurd as an approach to policy when the time and money that can be allocated to a policy problem is limited, as is always the case.”

It is worth noting that it is precisely this Herculean task that the Public Accounts Committee expect DfID to engage in (para 18), as translated through value-for- money indicators as common currency for our values (itself something of a reach, since even for myself I’m not sure in these straitened times whether my money translates better into, say a first rate rye-loaf for lunch today or a pint of stout in my local at the end of the week).

Indeed, Lindblom’s article misses out some key additional elements of the literature which have subsequently emerged, which have gone to further qualify the validity of the bullish interpretation of the quote:

1. Complex projects are not predictable and are therefore not replicable – previous experience does not help, so previous evidence is no guide to whether the intervention will work again – while complicated projects may be evaluated, but have so many factors that they are prohibitively difficult to measure reliably; and
2. If the question is how you allocate resources to achieve an impact, then there’s an institutional complexity – as the donor side fragments, and becomes more difficult to coordinate, resources are going to be increasingly allocated through different channels, and with different controls, and through different agents in ways that are difficult to predict – what are others doing, what if they change the game, what if they make your aid dollar less valuable through prior actions?

All very difficult.

Evidence plays a vital role for informing decision-making. However, for me, that role is to inform the decisions of the policy-makers who are (of course) competent and humble, willing to learn, deeply embedded in their context, as well as possessed of strong relationships to partners and a good dose of political nous. They will make decisions, based on their knowledge and intuition. No leader working in complex and dynamic environments – whether an officer in charge of a FOB in Helmand or an executive in the private sector – is expected to boil their decision-making down to numbers. It seemed strange to me to want to reduce this to a reductionist formula of compared values – that’s why we have experts – but I suspect it’s because the wider development industry isn’t trusted in the public, and therefore needs to be scrutinised. Again, for me accountability for these decisions could be based on a review that the decision was rational, based on the inputs and stated criteria for justification (thinking about it, not unlike an appeal in our law courts from a lower to a higher court) – i.e. a process assurance, plus a review of the outcomes measured insofar as possible, for the sake of learning.

It’s been interesting to see how DfID has wrestled with these challenges. Writing this recollected two recent evaluations of programme areas that I came across while scutting about the web – one is of public awareness campaigns in the UK, the other transparency and accountability initiatives. In the former the evaluators said:

“We are confident that … raising awareness of development issues in the UK has contributed to reducing poverty overseas. However, the evidence is circumstantial and consequently we have been unable to prove conclusively that this is the case. We can make the argument that it does, but there are simply too many causal connections to be able to prove it.”

Without this, the team concluded that DfID should reconfigure its spending. For the second field, that of TAIs, the evaluators said  “[t]he evidence base is not large enough – there are simply not enough good impact studies – to begin to assess overall trends” (PDF p. 19). Investment in transparency and accountability remains (appropriately in my view) a core area of DfID spending. The cases are not identical, since there are differences in the volume and nature of the literature, and in the degree to which theories of change for the interventions have been articulated and tested, but the differences in response to a lack of evidence is illuminating. Another example I came across this morning, from the recent Africa All Party Parliamentary Group report on the Bilateral Aid Review is also interesting:

“Our concerns relate to the lack of objective criteria used to select focus countries… The Needs-Effectiveness Index (NEI) appears to have been used to justify the subjective decisions of officials, rather than to make objective decisions. DFID did not provide a clear and convincing explanation of why its Needs-Effectiveness Index was constructed in the way it was, and our study shows that alternative and equally credible assumptions would have pointed to significantly different results… In particular we disagree with the decision to close DFID’s bilateral programme in Burundi, a small, extremely poor, fragile country recovering from decades of civil war, which is highly dependent on aid and whose stability has deep consequences for the wider region. When our alternative assumptions are used in the Needs-Effectiveness Index, Burundi ranks as the number one country, and, had these criteria been used to select focus countries, DFID’s programme in Burundi would not have been selected for closure.”

The critical conclusion from this work I think is an inevitable consequence of the vast task DfID is being asked to do, through rolling out across the board a comparable and aggregatable VfM approach.  For me, by making the decision-making a technical process of reducing down complex programmes to a comparison of aggregated numbers, the actual, real decision is inevitably and inadvertently submerged somewhere in the avalanche of assumptions that are required to make the exercise in any way feasible. In the end, by obscuring the reasons for decisions, I think this makes for less rather than more accountability.

Why I think a social policy kitemark won’t work

This post came courtesy of a Guardian article, which then floated around the ether for a bit. It comes from the Cabinet Secretary, Sir Jeremy Haywood, who tentatively puts forth the idea for a ‘kite mark’ – or badge of certified quality – for a social policy, such as the rehab of prisoners. I had several consecutive reactions to this (not in a Sherlock-fast deduction kind of a way, but more over a couple of days, in a single-cylindered-Soviet-tractor kind of a way) which I replicate more or less in the order they came in:

…ooo, this is a cool idea…

…but soft, policies are only valid, effective and therefore ‘of quality’ depending on their context, and therefore a kitemark probably doesn’t make sense since by its nature it claims universal applicability and quality?

…hmmm, but what if the kitemark is not interpreted as a universal claim that ‘this is a social policy which works’, but instead simply a claim that ‘this intervention works like this, through this causal theory of change, and therefore if these circumstances pertain, then the intervention will have the following effects’ i.e. the quality assurance includes statements of the limitations and applicability of social policy…

…this would mean that it only applies for ‘simple’ replicable interventions in some fields, and not complex or complicated interventions (using Glouberman and Zimmerman’s distinctions) where impact is unpredictable or cannot practicably be measured, since in such interventions, we just don’t have a solid theory of change or know for sure what’s going to happen…

…in which case, the idea doesn’t make sense, because a kitemark of a social policy implies (1) that any social policy can get a kitemark and (2) either that all social policies are universally applicable once they have one (which would be very limiting in applicability), or that the kitemark’s quality is partially based and on applicability limitations, expressions of causal links etc. Combined, for me, they scupper the notion – (1) because some perfectly good  ones wouldn’t earn a kitemark because they work through too complex causal chains; and (2) because a useful kitemark in fact makes no claim more than a good research project so there’s no added value.

Ultimately, therefore, I think it’s a cool idea  but I don’t see the value in it over existing ways of ‘stamping’ a social policy’s quality, and I think it might actively mislead when it is applied to really complex interventions.

Democratic Ownership and the Busan Partnership

The debate about the qualifying adjective for ownership was vibrant and intense during the lead-up to Busan. Contenders included ‘country’, ‘democratic’ and ‘inclusive ‘ ownership? In the end, ‘democratic’ won: the Busan partnership document commits to actions which seek to “deepen, extend and operationalise the democratic ownership of development policies and processes”.

Besides the choice of qualifier, I think another interesting matter is on how to understand the operation of the qualifier. This can be interpreted in two very different ways:

1. In the first interpretation, ’democratic’ describes the process of formulating a policy or commitment. Read in this way, the commitment exhorts development actors and governments to engage ‘democratic stakeholders’, be they civil society or parliamentarians or whomsoever else, either inclusively in the process of programming design, or in subsequent evaluations or both.
2. The second is very different. Rather than an exhortation to a particular type of policy formulation process, it treats ‘democratic’ as a prior requirement for ownership and cooperation between government and development partners; i.e. that ‘ownership’ will only be accorded to a commitment if the authority making the commitment is democratic and has the legitimate authority that (depending on who you talk to) democracy confers.

It seems clear from a reading of the document that the first meaning is intended: the process of formulating programming should democratic but should happen alongside processes of democracy-building – the latter, democracy-building, is clearly not some form of prior condition.

Accra adopted similar exhortations, with a view to bringing in a wider set of actors into the notion of country ownership enshrined in Paris. Although the Accra commitments were not systematically tracked in the Paris Declaration evaluations, personal observations in Sierra Leone and Malawi and elsewhere suggest that the political governance context in some countries means that this does not happen. The executives of many developing countries are frequently highly centralised and wider inclusion has often not come naturally – although there are signs that this is changing. Parties have often been subordinated to senior party figures, so parliamentarians’ role rarely includes policy-discourse. Studies argue that in many countries, civil society’s space for engagement is being restricted. For their part, development partners cannot afford to sacrifice their working relationship with government officials on whom they rely for effective execution of an aid agenda. Sovereignty is a veil which cannot easily be looked through. Inclusive formulation of aid policy is therefore often an illusive creature. While it is to be hoped that the Busan focus renders it less so, the creation of parallel processes (deal with the executive, while deepening democracy on the side) suggests that progress is by no means guaranteed.

What of the second interpretation? Its adoption would suffer from two challenges: one of these is surmountable, I think, while the other seems crippling. The surmountable one is that making support on the basis of democratic ownership creates a normative frame which will, inevitably, be contested, particularly if large volumes of aid funds were at stake. What would count as sufficient democratic ownership to render aid justified? Making that frame a condition for international engagement with the US or UK would raise some interesting questions. While this presents problems, there are and have been for some time a range of governance indices that have common currency which are already used, including democratic values, institutional assessments and public financial management: some aggregate of these could offer a model. The crippling challenge is that developing country governments, finding their right to ownership and thus sovereignty challenged, would be expected to resist such an interpretation strongly. Such a definition would raise questions of their claim to legitimate government in a manner that would be unacceptable.

Yet despite the political challenges, the second interpretation focuses usefully on the quality of the ownership. While ‘developmental’ and ‘democratic’ ownership are by no means synonymous, this interpretation emphasises the need to work towards legitimate ownership, for the benefit of the people. It therefore brings us back to the key idea for development effectiveness – the need for ‘developmental ownership’ i.e. that ‘country’ ownership of aid is not a matter of right simply by virtue of sovereignty, but as a matter of the qualities and aims of the leadership who claim ownership (see for the argument this paper by David Booth).  It usefully focuses on the construction of a development relationship between partner governments for the benefit of the people affected, and not the citizens of the development partner countries or the country government. It implies that the process of constructing relationships and legitimacy is central to the provision of aid and, even if is not a condition of aid, certainly requires considerably greater focus from the beginning rather than simply taking place as a parallel process.

Ownership and results in Busan: the risks of policy conditionality

In the Busan High Level Forum, the attention has focused on transparency and the BRICS inclusion. However, this has distracted from what is for me the crux of the whole document: the marriage of democratic ownership with the results agenda. This ‘marriage’ is articulated in the outcomes document [pdf] as the adoption of “country-led results frameworks and platforms … based on a manageable number of indicators drawn from the development priorities and goals articulated by developing countries.”

In other words, the Busan outcomes document takes country ownership and backs it up with a focus on the delivery of results. Both are laudable goals, where there is a country-owned development vision shared by the donors – however, the outcome document’s reliance on this as a precondition for success fails to address two challenges from the literature.

First, manifestations of ‘country ownership’ have been afflicted by two key syndromes (identified in this paper by Laura Collins): some cases, as Van de Walle suggests, are exercises in ‘ventriloquism’ where governments say what they think donors want to hear, in order to secure more funds, without political interest in implementing the reforms; in others, they are a matter of resistance to a dominant donor-led narrative (see the case studies in Whitfield and Fraser 2009). Of course, there are examples where there is genuine shared ownership afflicted by neither syndrome: however, many cases are characterised by one or the other, and in neither is there shared ownership by government and donor of development results. Under such circumstances, the adoption of results frameworks will be resisted by the government or not accepted by the donor.

A second challenge is addressed to the idea of ownership itself. Booth argues that country-owned development is an outcome to be constructed, rather than an established fact: it is an end to be accomplished, and to do so will require more complex means than the mere issuance by a government of a sector plan or strategy. Country-owned development requires utterances to be development-focused and ‘owned’ by the government to the point where there is sufficient political support to generate buy-in among line-ministries and local government implementers: in some cases, this may require a long process of negotiation and support with donors. Booth argues that the means prescribed by the Paris principles – harmonisation, mutual accountability and management for results – are insufficient to achieve these ends.

The fifth outcome document for Busan does not address these challenges. Although it articulates the need to “[d]eepen, extend and operationalise the democratic ownership of development policies and processes” it fails to offer solutions to what happens when there is no shared development ownership. In such circumstances – and the literature suggests they are frequent – results frameworks are likely to become vehicles of donor-driven policy conditionality, since it is donors who have the power.

Seeking to force through results which are not shared by government risks three impacts:

1. they will be ineffective in delivering the results, particularly when they are politically unpalatable (as many governance reforms are): this was the conclusion from the failed efforts of policy conditionality in the eighties and nineties, and experiences in performance assessment frameworks in many budget support mechanisms appears to confirm it;
2. in their exaction of heavy consequences for failing to meet results (such as delayed disbursement), they risk damaging the long-term aid relationships between donors and governments, and thus actively undermining the necessary process Booth prescribes – that of moving towards country-owned development; and
3. the use of simple indicators is subject to the wider critique of the results agenda, in that it risks the creation of perverse incentives, unforeseen impacts, and reduces opportunities to learn and improve aid delivery, two processes that are absolutely vital to effective aid.

I’m with Alison Evans @ODI on the enthusiasm for the continued commitment to ownership. Ownership is fundamental. However, coupling ownership to the results agenda without explicitly addressing some of the underlying political issues and ensuring first of all a genuine shared ownership is likely not only to be ineffective but to be actively damaging to effective aid.

Thin gruel at Busan

The biggest ball in aid effectiveness, the fourth High Level Forum, is being held next week in Busan, Korea, and sadly I shall not be going to it. Instead, I shall be reading the documents while sweeping up ash and so forth. The latest version of the outcomes document came out earlier in the week. Analysing it as a guest on Duncan Green’s blog, Gideon Rabinowitz of UKAN today identified aid transparency, tied aid and monitoring its implementation as the three key points. To this, I would add the importance of ‘management for results’, whose conceptual imprint can be seen throughout the document.

If that’s it, it’s some pretty thin gruel. Paris had a clear and ambitious vision, rooted in country ownership. It was based on the insight that recipients’ buy-in was vital to aid programming, and that to secure this, donors should therefore follow country agendas. As it turned out, this vision was flawed; the implementation of Paris failed. Only one of the thirteen commitments been achieved since 2005. However, most analysts conclude that Paris’ core insight remains valid: country commitment to development agendas is vital for effectiveness. The flaw is that the principle of country ownership presents a great goal, but is not a means. As David Booth argues, drawing on several political economic analyses, country leadership in a development-oriented agenda is in fact relatively rare [pdf]. Donors have a role in supporting that process, as political actors. Projects like the APPP, Developmental Leadership Program and Tony Blair’s AGI are exploring opportunities for this.

For Busan, however, this vision is obscured. Reading the outcomes document, it is unclear whether Busan marks Paris 2.1 or something new. Certainly, the Paris principles are there, but they are so watered down as to be almost unrecognisable. Ownership, insofar as it is referred to, is there to define what governments can be held to account for by their partner donors. Alignment (the use of systems) is reduced to a vague presumption, with the African states withholding endorsement. Commitments to transparency and tied aid are appropriate and useful, but are not new and do not pertain to country ownership (taking nothing away from the importance of work by IATI, which is absolutely essential to good, accountable, well-governed aid programming).

As a document, it reads to me like an effort in realpolitik: bring in the emerging donors and keep increasingly sceptical donors happy by including results agenda references (who are themselves under pressure from their domestic stakeholders). One part of me thinks that, as someone who’s never been involved in such a durbar, it’s probably a good thing and a necessary part of engaging with the emerging donors. Another part of me thinks that Paris was always asymmetrically implemented, with some donors way more committed than others, so what does it matter what Busan actually says? Maybe a more modest focus on transparency and untied aid will actually achieve something. However, what I’m really concerned about is the way that it obscures the importance of ownership.