Making Sense of the Metrics:
Challenges of Measuring Progress towards Universal Health Coverage and the Sustainable Development Goals:
The recently concluded Regional Meeting of the South East Asia Regional of World Health Organization (October 7th to 9th, 2024) , saw the release of a Report titled: “Monitoring progress on universal health coverage and the health-related Sustainable Development Goals in the South-East Asia Region – 2024 update”. This annual update features a different theme each year. This year’s focus is on Human Resources for Health, which forms the second part of the Report. The first part measures regional progress using a set of indicators, while the third part provides country-specific reports based on those indicators.
This paper enables a critical reading of this report, which we suggest is essential for those in engaged in practice or research into health systems and health policy. This paper is in two parts. Part I is a conversation with Dr Siyam Amani. (SA), Regional Advisor for Health Information Systems, WHO-SEAR, to understand better the WHOs recommendations on how to understand and improve the two main indicators used to measure progress towards UHC. These are the UHC Health Services Coverage Index (indicator 3.8.1) and the measurement of financial protection (3.8.2). Of this the UHC Health Services Coverage Index is a composite of 14 indicators- and the list of these indicators and approach to computing the index is given in annexures of this Progress Report. (pg. 130,131).
Part II will focus on the approach to measuring progress towards the other SDG goals. There are 59 indicators linked to other health-related SDG targets that this report presents. Many of these indicators vary in validity and feasibility. We need to assess which indicators are most useful and feasible for measurement at sub-national levels—province and district—to better inform policy and implementation. We also discuss other approaches to measurement that are being considered and that may be more useful
This paper therefore is on health systems metrics and not on why progress is limited or how to improve it.
If you are not the type who can read through such a detailed progress report, perhaps you could begin by just reading your five-page country score card. For Indian readers that is pages 64 to 68 of the Report and for Human Resources for Health pages 113 and 114.
PART I
Conversation with Dr Siyam Amani on the two UHC indicators
TS. Dr Amani, could we begin by your introduction to UHC and the main indicators used by United Nations and WHO to report on progress towards UHC?
SA. The goal of Universal Health Coverage (UHC) is to ensure that every individual and community, irrespective of their circumstances, receives the quality health services they need without risking financial hardship. In the UHC monitoring framework, two indicators were adopted by the United Nations Statistical Commission for tracking UHC (SDG target 3.8): the coverage of essential health services (SDG 3.8.1) and the proportion of a country’s population having catastrophic health-care spending (SDG 3.8.2).
TS. How is the first of these Indicators viz. “3.8.1” defined? And how is it measured?
SA. Indicator 3.8.1 i.e. “Coverage of essential health services” is defined as “the average coverage of essential services based on tracer interventions that include reproductive, maternal, newborn, and child health, infectious diseases, non-communicable diseases, and service capacity and access, among the general and the most disadvantaged population”. This definition acknowledges that countries provide a wide range of services for health promotion, prevention, treatment, and care, including rehabilitation and palliation, and that tracer indicators should be selected to represent overall coverage of essential services.
In each of the four categories described above, four tracer indicators were selected, based on four criteria and ensuring that within each category, the indicators reflected a range of programme service delivery strategies. The four criteria were relevance based on epidemiological burden and cost effectiveness, conceptually sound, feasible to measure, and usable.
Identifying indicators that fulfil these criteria is challenging, and few of the selected indicators fulfil all criteria. Given data availability constraints, proxy indicators were used to track effective service coverage. Proxy indicators are not direct measures of service coverage, effective or otherwise, rather their used ensured that other criteria, especially on relevance and feasibility were met for all indicators.
These tracer indicators are then summarised in an index, called the UHC service coverage index (UHC SCI) which is a single numerical value. These 14 tracer indicators in use are listed in the table below, but for an understanding of how the composite index is computed, refer annexure 1:
Indicators used for computing UHC service coverage index:
Reproductive, Maternal and Child Health: 1. Demand satisfied with modern methods (FP) 2. Antenatal care, 4+ visits (ANC) 3. DTP immunization, three doses (DTP3) 4. Care-seeking behaviour for suspected acute respiratory infection
Infectious diseases (ID) 1. TB treatment coverage (TB) 2. HIV antiretroviral treatment (ART) coverage 3. Population with access to at least basic sanitation (WASH) 4. Insecticide treated bed nets- not use in SEAR region
Noncommunicable diseases (NCDs) 1. Prevalence of treatment for hypertension (BP) 2. Mean fasting plasma glucose (FPG) 3. Tobacco non-use (tobacco)
Service capacity and access (capacity) 1. Hospital bed density (hospital) 2. Health worker density (HWD) 3. IHR (2005) Core Capacity Index [IHR (2005)]
For most indicators included in the UHC SCI, the preferred primary data sources were nationally representative, population-based surveys, which enabled the measurement. Such surveys usually allow for disaggregation of service coverage by different subpopulations, and this allows for equity analysis. They are however not powered for measuring most of these indicators at district/block level. The standard HMIS does not have such disaggregation. (Peer reviewer comment: We note that if reformed and strengthened further, HMIS has better potential given its total population coverage. And, although HMIS only measures government service coverage; we could apply a multiplier to account for private health services. These are important areas for further discussion and hopefully national and state level epidemiologists could begin to address these issues)
The UHC SCI is not useful for measuring quality of health services.
TS. How is this UHC Service Coverage Index (UHC-SCI), which is used to measure performance over time and between countries or states, computed? achievement score? And how is it computed?
SA. It is constructed from the “geometric means” of the 14 tracer indicators, first within each of the four categories, and then across the four category-specific means to obtain the final summary index. “Geometric means” are used instead of “arithmetic means” as they are less affected by higher coverage for some services at the expense of others. (in arithmetic means one adds the values of all indicators and divides by the number of indicators. In geometric means one multiplies the values of each indicator and then take the nth root of the final product. [Thus the service coverage index of RMNCH coverage is quadrant root of (FP*ANC*DPT3*ARI) and so on]
TS. If we look at the 14 tracer indicators, we note that 8 of these tracer services, are part of the earlier selective package- 4 from RMNCH package and 4 from the major communicable diseases. The four “service capacity” indicators are input indicators and of these, only the four NCD indicators were new. Of these cervical cancer screening was excluded because of low data availability and the other two are population prevalence of two biomarkers of diabetes and hypertension respectively and not necessarily representing service delivery. Today most states and countries have introduced some interventions against these two diseases, but almost all other NCDs including chronic respiratory disease, chronic kidney disease and mental health are not addressed. The requirement for an intervention to be ongoing and measurable has likely contributed to our continued focus on selective care, rather than measuring progress toward universal care.
SA.. In 2017, when the first UHC SCI was produced, for indicators of cardiovascular disease prevention and diabetes management, no standardised datasets for indicators of effective coverage of cardiovascular disease and diabetes treatment existed, nor do datasets for indicators of treatment of increased cardiovascular risk. Therefore, the prevalence of non-raised blood pressure (including those whose blood pressure is controlled by medication) and mean fasting plasma glucose (an indicator for diabetes) were used as proxy measures. At best, these reflect the success of both effective health promotion programmes and effective screening and treatment programmes. Non-smoking of tobacco is also included as a proxy for effective coverage of measures to reduce tobacco use.
Today, WHO is in the process of an open country consultation to update the UHC SCI index (2000-2023) for the Global monitoring report 2025 which will be published in September 2025. Some of the changes that are being proposed are given in this annexure 2..
TS. Now let us move to the next core indicator- the measure of financial protection. Could you elaborate this?
SA. The SDG 3.8.2 indicator is defined as the “proportion of the population with large household expenditure on health as a share of total household expenditure or income”. Two thresholds are used to identify “large household expenditure on health”: greater than 10% and greater than 25% of total household expenditure or income.” These are also known in the field as “catastrophic out-of-pocket (OOP) health spending at 10% and 25% of a household budget” respectively.
These indicators were adopted in 2017 as against a proposal to monitor the “Number of people covered by health insurance or a public health system, per 1,000 population”. The population coverage indicator was not supported by WHO and the World Bank because it is a measure of affiliation or entitlement, not actual experience – people may be insured or entitled but still face financial hardship due to OOP health spending. Similarly, “public health system coverage” is a vague concept, and health insurance programs vary widely, making comparisons hard to interpret. Also, it is not only the proportion of population covered, but what proportion of OOP spending remains and what disease conditions are included in the package.
However, this indicator does not adequately capture better the financial hardship experienced by poorer people. It is estimated that about 65.3% of the 344 million people pushed or further pushed into extreme poverty by OOP health spending in 2019, and 77.3% of the 1.3 billion people pushed or further pushed into relative poverty that same year spent less than 10% of their households’ budgets on health. The WHO and the World Bank have requested a revision and the new definition proposed is “the population with positive out-of-pocket household expenditure on health exceeding 40% of household discretionary budget”. This is also sometimes called “the capacity to pay for health care.” This is the household total consumption expenditure minus the expenditure that goes to paying for the cost of basic needs, the latter also being known as the Societal Poverty Line (SPL) (see annexure 3) . Similar to the current indicator, households will be considered to incur financial hardship in health if their OOP spending on health is large as relative to the household discretionary budget. This approach defines catastrophic health spending based on what people can afford to pay. It recognizes that poorer people mostly spend their money on basic needs, so it calculates health spending based on the budget left after covering a minimum standard of living. As a result, anyone living in poverty who has any out-of-pocket health expenses, or whose remaining income after health costs falls below the poverty line, is considered to be facing financial hardship (or impoverishing health spending).
The exact details of calculating this are provided in this annexure 4
TS. What are the main limitations of the SDG 3.8.2 indicator, even after revision?
SA. The current and proposed indicators fail to identify those funding OOP health spending through distress financing (selling or mortgaging household assets/lands, borrowing money from lenders/banks/friends/relatives, and receiving assistance from friends/relatives) because most data sources do not provide such information. Further it does not track the ‘care foregone” due to lack of funds or other reasons.
TS. One follow- up question is regarding the data requirements and collection methods? What are these and is any change likely with revision.
SA. The data source for computing indicator 3.8.2. are household surveys which includes household budget surveys (HBS), household income and expenditure surveys (HIES), socio-economic (SES) or living standards surveys (LSMS). The survey must collect information on both total household consumption expenditure on health and total household consumption expenditures or income. The revised definition requires the computation of two additional variables, the Societal Poverty Line (SPL) and the discretionary budget, but both of these can be computed from the same source as for measuring CHE. The consumer price index (CPI) data and purchasing power parities (PPP) that are needed to convert the SPL into local nominal currency values and the societal poverty estimates are available on different web-platforms. (https://pip.worldbank.org/poverty-calculator).
TS. What are the current trends for these indicators, and how are these likely to change with proposed revision?
SA.The level of the proposed revised indicator for catastrophic health expenditure will be higher than what it is now, globally, regionally and for almost all countries as it will be combining the rate of the population with both catastrophic health expenditure as measured earlier and impoverishing health spending, The global level of catastrophic health expenditure has been continuously increasing between 2000 and 2019. The rate of impoverishing OOP health spending followed a different trend. If the fixed international poverty line is the benchmark for impoverishment, it would show a sharp decrease in the rate of people experiencing impoverishing OOP health spending since 2000, largely driven by reduction in the global headcount. When considering a relative poverty line, however, there would be increase in the rate of population incurring impoverishing OOP health spending. Country specific trends in large and impoverishing health spending differ so the change in trend will differ from country to country.
In conclusion, I note that the SDG 3.8.2 indicator of financial protection has been worsening continuously since 2000, and conveys a picture of entrapment if not a reversal of direction towards achieving UHC by 2030. The proposed revision acknowledges that even small out-of-pocket household spending on health can cause financial hardship for people living in poverty and in near poverty and improves coherence in the way the measurement is made across countries, and simplifies the communication of the concepts to member states and civil society.
TS. Thank you for your contribution. We will get back to you if further clarification is sought. Our readers through could contact you. Let me place on record our appreciation of the personal and professional journey you have made and the difficult role you are playing. (see note on Dr Siyam Amani)
PART II
The SDG indicators for Health.
The limitations of UHC indicators 3.8.1 and 3.8.2
The discussion with Dr Siyam Amani, brings out the complexity of measuring these two core UHC indicators. With 3.8.1, the main problem is that it remains largely a measure of the earlier package of services, except for the addition of diabetes and hypertension. Even for these it is only proxy indicators with poor precision and validity are in use.
A much larger problem is that though officially these are “tracer” indicators, they come to define the services made available, and those services not being measured get left out. Thus, after adoption of UHC, most countries add on diabetes and hypertension and two or three common cancers, but equally prevalent concerns like chronic respiratory disease or chronic kidney disease, or musculo-skeletal disorders or mental health remain excluded from the measurement. Not measuring for this wider set of services is sought to be justified by the lack of data and the lack of interventions. But, the concern on such an approach to health metrics that health activists in the public health community have been raising is that when it comes to implementation, the lack of its inclusion becomes a reason for such services to lose priority.
These two indicators play almost no role in guiding improvement. While scoring and ranking based on composite indicators could encourage the political and administrative leaderships to take UHC more seriously, in practice they seldom do so- and any indicator including IMR and life expectancy can yield similar ranks. Composite indicators are of little value for understanding where the gaps are or even the trends, especially when like in the UHC coverage indicator, they are showing a misleadingly high coverage. The worsening of financial protection indicators is as likely to due to selective care packages in public services and not only to be attributed to inadequate insurance coverage. Nor can these indicators be measured or made use of at provincial and district levels. Ability to measure progress at disaggregated levels is essential to guide corrective action as also to fix responsibility and accountability- for which again these indicators are no help.
When SDG-3, the SDG on health was being framed, there was a strong push by some global institutions/experts to propose UHC as the umbrella strategy that is inclusive of all the SDG goals. In hindsight, given the problems of measurement, it is good that UHC was limited to being only one of the 13 targets of SDG 3. This UHC target 3.8 has three indicators and the other 12 targets have over 40 indicators. We can make sense of where we are in respect to achievement of UHC only if we give importance to many of the other SDG-3 indicators.
A brief overview of the SDG health indicators.
SDG-3, the health SDG has 9 main targets listed as 3.1 to 3.9 and four supplementary targets listed as 3.a to 3.d. (see annexure of the SDG Progress Report for 2024)
This Report presents measurements for 25 indicators for the nine main targets of which three are for target 8 which is Universal Health Coverage (UHC) and 22 are for the rest. There are then 14 indicators for the four supplementary targets and 17 indicators for health related targets which are part of the other SDG goals. These other SDG goals cover such vital areas as malnutrition and anaemia, water and sanitation services, gender based violence, crime, civil registration system systems and government expenditure. All of these are presented in this Report across all the nine countries and for each of the nine, and make for interesting reading.
Indicators for the main nine SDG targets: measuring outcomes and service delivery.
Of the 22 indicators that relate to targets the eight main SDG-3 targets (excepting target 8 on UHC) seven are related to RCH, seven related to communicable disease, and five related to non-communicable disease, and three related to environment health. Of these 22 indicators 11 indicators are related to births and mortality and can therefore be computed in any district or state that has a robust birth and death registration system – the possibilities of which we discussed in an earlier conversation. This is now possible for almost any state in India. The 2024 Progress Report however uses figures attributed to Global Health Observatory, which are highly modelled estimates often from IHME. These estimates have unquantifiable margins of uncertainty, and hence could be well off the mark.(Chalapathi Rao, 2021) Therefore, this Report should be read with extreme caution. But the positive side is that since nearly all states have good birth and death reporting systems, most states should be able to track these indicators, with some close local attention to strengthening data compilation and quality. Of the remaining 11 indicators 5 are related to prevalence of the health condition and 6 to service delivery access and most or all of these are also possible to measure, but many of these are also modelled estimates which are just not good enough, for their wide margins of uncertainty.
In summary, while these indicators are useful and feasible, they should rely on national data from valid sources rather than solely on modelled estimates from the observatory, which are insufficient. Although the Progress Report acknowledges these limitations, it proceeds with the estimates, suggesting that having some data is better than none. However, in our view, if we cannot trace the final figures back to sub-national sources, as they should be derived, the data does not adequately fulfil its intended purpose especially in large and diverse settings like India.
It is important for any system of population health metrics to establish the data audit trail and unpack computation methodology so as to subject epidemiological and statistical assumptions applied in the estimation processes to scientific scrutiny. Our concern is that these metrics, endorsed passively by experts and institutions, come into regular use and such use lends an undeserved air of credibility to the same. Data from empirical observations on these essential components of health service delivery and health outcomes is the need of the hour, for evidence-based guidance to make progress towards UNSDG 2030.
One important new indicator of particular value is to measure “a one thirds reduction of premature mortality due to non-communicable disease”. The most preferred is a proxy indicator “all-cause mortality rate in the 30 to 70 age group disaggregated by gender”. This indicator is easy to measure in all states where there is good death registration, and it need not wait for universalization of medical cause of death reporting. A recent Lancet Commission’s recommendation is for Probability of Premature Death (PPD), defined as the probability of dying before age 70 years under the current age-specific mortality rates as the main metric for measurement of outcomes of a universal health care strategy. PPD is related to life expectancy at birth, which could also be used, but once under five mortality has decreased, this could be the better measure. A simple equivalent that one can work up within a district, is the “proportion of all-cause mortality that occurs before the age of 70”.
Measurement of mortality due to Road Traffic injuries is another so-to-speak low hanging fruit where immediate action is possible- and which can trigger immediate public health action. Tamil Nadu for example has developed a very effective programme to address this.
One important suggestion in this regard is to measure the unmet need for appropriate health care services using all cause morbidity as well as tracer indicators for certain key health care services and the proportion of such access that is received with financial protection. This can be based on population-based sample surveys which have some elements of biologic measurements- like is done for the National Family Health Surveys and India’s National Sample Surveys on Health morbidity and out of pocket expenditure. Such surveys would do better for equity in access. They would be able to measure “proportion of persons in need of service X, who were able to access the required service with effective quality” as a measure of service coverage”. In a recent study on unmet needs we show that utilization of services and therefore presumably prevalence of overt disease is higher in higher income quintiles. But when you factor in latent disease and care foregone, the prevalence of disease is much higher in the lower income quintiles.
This Progress Report does not measure unmet needs, but it acknowledges that such an indicator is very much required. We also note that a number of countries have moved a resolution in the 77th World Health Assembly asking for measurement of unmet needs to be added in. This proposal was considered in the WHA, but deferred because all member countries were not sure of the capacity to undertake such a survey. We also need greater clarity on the definitions- and this is work in progress.
Another important and promising metric is the measurement of what is termed “effective coverage”. First proposed by Tanahashi as early as 1978, it was not till 2006 that this term came into regular use and it took till 2012, post the UHC launch, for publications in this topic to rapidly multiply. Studies on effective coverage also address it as the cascade of care or systems effectiveness studies. It is based on the concept, that even if an intervention is launched universally viz universal coverage, due to barriers in detecting the disease, access, provider and user compliance and efficacy of the intervention the universal effective coverage is less. In this context effective coverage would mean the proportion of potential health from that which the health system could deliver to what is effectively delivered. (for a detailed review see Karim A, 2023). These studies also illustrate the hazard of reliance on isolated indicators as benchmarks of performance, and draw attention to the importance of the denominators we use to measure them.
We also note that to use measurement to guide action, health metrics that measure “equity effectiveness” are essential. Most measures of service coverage or unmet needs based on household surveys are able to provide such data. But unfortunately, both as regards mortality data and HMIS, this remains a challenge.
Indicators for the supplementary health system targets ( 3 a to 3d)
The first of these four supplementary targets of SDG-3 is for tobacco control and there is one indicator for it. That is the easy one. The second, 3b, is for access to essential medicines and vaccines and includes a highly aspirational statement such as implementation of TRIPS flexibilities. Yet of the six indicators four have been used up on immunization coverage which is hardly the spirit of this indicator and the other two though somewhat more valid have not been measurable. The next target, 3c, on human resources for health, has four indicators which measure density of four important categories of health professionals, but measuring density without knowing the distribution across geographies and across public and private sectors gives us little understanding of the gaps. And finally on target 3d, emergency preparedness, there are three indicators – the first of which is the complex IHR core capacity indicator which is itself a composite of 15 indicators and two very interesting indicators for measuring AMR. A third more useful indicator has also been added on as the last and final 61st indicator and this is “Percentage of total antibiotic consumption, being from the AWaRe “Access” antibiotics category (%)”. A good indicator but as of now no measurements available.
Overall, most of these 14 indicators give us very little understanding of the progress being made on health systems strengthening. But then that is the problem with most indicators. Indicators have been likened to watching a football match through a chink in the fence. You get an idea of what is going on, but far from the whole picture. The Thais have an equivalent saying- its like serving soup with a fork. You get a taste of what it is about, but that is about it.
We have not addressed the indicators related to the social, economic and political indicators of health- but we note that the Report has made a start on including this and relating it to outcomes.
In conclusion:
The indicators related to the 9 main targets of SDG3 are potentially feasible and useful. However, as of now, many of these are dependent on the estimates and analysis of global institutions and not measurable independently at the national or sub-national level. The indicators of the supplementary health systems targets are entirely based on national sources of information, but many give only a very incomplete picture of the objective.
To look at it positively the very incomplete nature of solutions offered for the “measurement problematic” opens up a huge area for researchers in health systems and health policy. For those in public health practice, the need is to find that sub-set of indicators that would guide their action at sub-national levels and then build reliable data sources for this. This is ambitious but not impossible. There is a considerable body of work on measurement of service coverage and unmet needs that is promising.
But whatever the indicators and the conceptual framework, some of the important preconditions for progress in measurement are a universal reliable civil registration of births and deaths, a better health management information system, two or at best three well planned household surveys- and above all capacity building for the use of information to guide action.
Going through this Progress Report on UHC and the SDGs could be a good starting point to engage with this problematic.
Acknowledgements:
Gratefully acknowledge Dr Amani’s participation to elaborate on the challenges related to the UHC indicators. Also gratefully acknowledge Dr. Chalapathi Rao, and Dr. Shalini Singh for their peer review and comments and Ms Roubitha David for her editorial support.
References:
- Monitoring progress on universal health coverage and the health-related Sustainable Development Goals in the South-East Asia Region: 2024 update. New Delhi: World Health Organization, Regional Office for South-East Asia; 2024.
- Rao C, Gupta A,Gupta M, et al. Premature adult mortality in India: what is the size of the matter? BMJ Global Health 2021;6:e004451. doi:10.1136/ bmjgh-2020-004451
- Jamison, Dean T et al. Global health 2050: the path to halving premature death by mid-century, The Lancet, Volume 404, Issue 10462, 1561 – 1614
- Ranjan et al. Measurement of unmet healthcare needs to assess progress on universal health coverage – exploring a novel approach based on household surveys: BMC Health Services Research (2023) ://doi.org/10.1186/s19-023-09542-0
- Karim, A.; de Savigny, D. Effective Coverage in Health Systems: Evolution of a Concept. Diseases 2023, https://doi.org/10.1186.s12913-02-09542 -0
About the Contributors
Dr Ms Siyam Amani, began her career as a Data Analyst working for the Federal Ministry of Health, in Khartoum, Sudan which is her home country. She then completed her masters in biostatistics from the London Schook of Hygiene and Tropical Medicine (1995) and her doctorate from the London School of Hygiene and Political Science (2001). For the next 19 years she worked in as a statistician in the Health Systems Metrics Division of WHO, Geneva, and much of it was under the mentorship of the renowned Prof. Ties Boerma. From October 2022, she serves as the regional advisor for health information systems in the WHO Office for South-East Asia which supports eleven WHO member states
