OBA Population Accountability: A critical perspective
In the first blog article in this series, we explained the key basic concepts and theory of Outcomes Based Accountability (OBA). In this post, we look in more detail at one of the two levels of OBA – namely, ‘population accountability’.
Population accountability
As detailed previously, ‘population accountability’ is a process for holding people in a geographic area – government, the private and voluntary sectors and even individual citizens – responsible for the wellbeing of the whole population in that area.
The Executive has based its Programme for Government at the population accountability level of OBA. The Executive first defines and agrees a set of outcomes for the Northern Ireland population. Indicators for these outcomes are then developed and monitored to measure progress. In theory, all publicly funded initiatives should contribute to progressing the outcomes and their indicators in some way. (A full definition of population accountability can be found in the first post in this series.)
Population accountability can possibly serve as a useful medium- to long-term ‘North Star’ for Executive priorities and decision-making. However, there are a number of potential issues with putting this level of OBA into practice, in the context of government and legislative oversight. We consider several of these below.
Indicators can never precisely represent the outcome
Population level outcomes are deliberately broad, aspirational and long-term. For example, take Outcome 12 of the Outcomes Delivery Plan:
We give our children and young people the best start in life.
If we try to unpack this statement: what does it actually mean? For instance, what is the upper age limit on ‘young people’? What, precisely, constitutes the ‘best start’ for this population?
Any outcome is given more definition by the indicators chosen to measure progress towards it. There are six indicators for this outcome:
- % babies born at low birth weight
- % children at appropriate stage of development in immediate preschool year
- % schools found to be good or better
- Gap between the percentage of Free School Meal eligible school leavers, and non-eligible school leavers, achieving at Level 2+ including English and Maths
- % school leavers achieving at Level 2+ including English and Maths
- % care leavers who, aged 19, were in education, training or employment
Each of these indicators is relevant and valuable. However, taken together, they do not comprehensively capture progress towards the whole outcome.
Many other existing or potential indicators could add to this picture. For example, the percentage of children in absolute or relative poverty; the percentage of young people suffering from mental health issues; or an index of social mobility over time.
This fact indicates a broader inevitability with the population accountability level of OBA. The indicators chosen for each outcome, as valuable as they may be, will never precisely represent the full complexity and scope of the outcome. There will always be a gap between an outcome, and the indicators which are chosen to represent it.
This gap – between a broad and general outcome statement, and a small number of indicator data points – presents the theoretical possibility that an outcome could be progressed, while data for some or all indicators is not improved. Conversely, it is possible that indicators could be improved, without actual progress towards the overall outcome. This possibility leads us neatly to consideration of ‘perverse incentives.’
Indicators can create perverse incentives
The Northern Ireland Audit Office defines a perverse incentive as ‘an incentive or target that has unintended and adverse consequences due to the actions undertaken to meet’ it.
When indicators are the only measure of progress at the population accountability level of OBA, they can create these perverse incentives – and an environment which encourages production of the right data, over delivering the right services for the outcome.
The structure of OBA – where vague and aspirational outcomes are represented by objective and tangible indicator data – naturally focuses the work of responsible public bodies on improving the indicator data. However, as discussed above, improving on the indicators does not necessarily mean actually progressing the overall outcome. The OBA structure simplifies the complexities of real-life interventions to an artificial and basic action-to-indicator process.
In the Executive’s 2016 draft Programme for Government, each indicator was allocated to a senior responsible official. In the Outcomes Delivery Plan, maintained by the civil service during the Executive’s absence between 2017 and 2020, each outcome had an Outcome Team led by the relevant permanent secretary. Within the structure of OBA, the rational way for these teams to apparently deliver progress would have been to improve performance on indicator data – even where that did not necessarily equal progress towards the outcome.
For example, take the indicator ‘number of households in housing stress’ within Outcome 8 of the Outcomes Delivery Plan: ‘We care for others and we help those in need’. This indicator is based on the number of points a household has on the social housing waiting list: a household must have 30+ points to be in ‘housing stress’.
Within the context of OBA, it appears rational to change housing allocations policy to make it harder for households to be deemed ‘in housing stress’. For instance, the threshold for ‘housing stress’ could simply be raised from 30 to 50 points. This would improve performance against this indicator – whilst doing nothing to reduce housing stress, or actually improve the circumstances of affected households.
This is a fundamental challenge for the population accountability level of OBA. Multiple studies of outcomes-based management have found that its structure encourages improving indicator data, without necessarily improving services or progress towards overall outcomes.
It is possible to improve indicator data and progress towards an outcome at the same time. However, OBA clearly has the potential to create perverse incentives at the population accountability level, where indicator data can be improved, at the expense of progress towards the actual outcome.
Attribution and accountability
A third issue with OBA is the difficulty in establishing who is responsible for positive or negative change, on any outcome or indicator, at the population level.
As an example, take the NI Composite Economic Index, a measure of private sector output under Outcome 1 of the Outcomes Delivery Plan.
Between 2015 and 2020, this indicator has substantially improved. But who is responsible for this, and to what extent? Is it Department of Finance policies? Invest NI interventions? Actions by the Department for the Economy? If it is some combination, how do we meaningfully divide and attribute responsibility? And to what extent is this improvement down to, for example, Westminster policies, or global economic conditions, which are beyond the Executive’s control entirely?
Conversely, the COVID-19 pandemic has necessitated global restrictions on trading and travel, which have and may continue to significantly harm this indicator. The outbreak was an external event, not within the control of the Executive or the population of Northern Ireland. How can these external causes of the downturn in this indicator be accurately accounted for within OBA?
Mark Friedman has acknowledged that real-world interventions do not function in the linear cause-and-effect way that OBA presumes. However, his response effectively dismisses this:
Don’t accept lack of control as an excuse … If control were the overriding criteria for performance measures then there would be no performance measures at all.
Friedman has further suggested that if an indicator improves, it would be fair for an individual agency or department to say that it contributed to this. However, OBA suggests that it would be ‘a fruitless exercise’ to attempt to calculate precisely what proportion of the success is down to that agency or department.
This highlights a vital point on attribution and accountability, at the population accountability level of OBA.
It is not fair to expect to be able to isolate and measure the multiple factors – whether within or outside Executive control – impacting a population-level indicator. No individual Executive department, agency, or funded stakeholder can be held solely responsible for any population-level outcomes and indicator. However, if there is no way of clearly specifying and dividing responsibility for population-level outcomes and indicators, then accountability at the Northern Ireland-wide level is significantly reduced.
This means that population accountability is of very limited value in assessing government performance. Any responsible body could theoretically claim as much credit as possible for improvements on population-level outcomes and indicators, and potentially blame negative performance on external factors beyond their control.
Outcomes contradict and compete with one another
In an OBA framework with multiple population-level outcomes, it is inevitable that these outcomes will at some point contradict or compete with one another. Progressing Outcome A may mean either sacrificing progress – or worse, actively damaging – progress on Outcome B.
Consider Outcomes 1 and 2 of the Outcomes Delivery Plan:
- Outcome 1: We prosper through a strong, competitive, regionally balanced economy
- Outcome 2: We live and work sustainably – protecting the environment
We can imagine a decision facing the Executive, or one or more of its departments, in which either option will favour one of these outcomes over the other. A significant inward investment opportunity could damage the environment. Conversely, stronger environmental protections could deter such inward investment.
This could even be reflected in the indicators for Outcomes 1 and 2: a single Executive decision could improve Outcome 1’s Private Sector NI Composite Economic Index, whilst also negatively affecting greenhouse gas emissions under Outcome 2.
In a Programme for Government which aims to encompass and direct all governmental activity, competing outcomes like these are inevitable. Decision-making in such an environment requires judgements and influences beyond that which OBA can offer, which may limit its value as an overall framework.
Conclusion
This post in our series on OBA and the Programme for Government has focused on population accountability.
While the population accountability level of OBA can usefully set a long-term vision for Executive activity, there are several significant weaknesses in putting it into practice. The indicators for each outcome can never precisely represent it. These indicators can encourage perverse incentives, and an environment which encourages producing the right data over delivering the right services. The broad, aspirational and cross-cutting nature of outcomes and indicators makes it incredibly difficult to attribute responsibility for performance. Finally, outcomes will sometimes contradict and compete with one another, requiring judgements and decision-making outside of the OBA framework altogether.
In our next post, we will take a closer look at performance accountability, and the possibilities of integrating this level of OBA into Executive performance management and Assembly scrutiny.
