Assessing PEPFAR’s Impact: Analysis of Mortality in PEPFAR Countries

We used a difference-in-difference¹, quasi-experimental design to estimate a “treatment effect” (PEPFAR), based on comparison to a control group (the counterfactual). The difference-in-difference design compares the before and after change in outcomes for the treatment group to the before and after change in outcomes for the control group. Our outcome of interest was the crude death rate, all causes (per 1,000). We chose this outcome, instead of the HIV mortality rate, because available HIV mortality estimates are derived using assumptions that include the role of HIV treatment, which is itself one of PEPFAR’s interventions. We included data on the mortality rate starting in 1990, to assess patterns before and after PEPFAR.

We constructed a panel data set for 157 low- and middle- income countries between 1990 and 2018. Our PEPFAR group included 90 countries that had received PEPFAR support over the period. Our control group included 67 low and middle income countries that had not received any PEPFAR support or had received minimal PEPFAR support (<$1M over the period or <$.05 per capita) between 2004 and 2018.  Data on PEPFAR spending by country were obtained from the U.S. government’s https://foreignassistance.gov/ database and represent U.S. fiscal year disbursements. Data for the mortality rate were obtained from the World Bank’s World Development Indicators (WDI) (https://datatopics.worldbank.org/world-development-indicators/.  We explored several difference-in-difference model specifications. Each specification controlled for numerous baseline variables, compared to an unadjusted model, variables which may be expected to influence mortality outcomes and which help make the control group more comparable to the PEPFAR group.

Our baseline variables and model specifications were as follows:

Table 1: Baseline Variables
Variable	Data Source
1. GDP per capita (current USD)	WDI, https://datatopics.worldbank.org/world-development-indicators/
2. Recipient of U.S. HIV funding prior to 2004 (dummy variable)	USAID, https://foreignassistance.gov/
3. Total population	United Nations, Department of Economic and Social Affairs, Population Division (2019). World Population Prospects 2019, Online Edition. Rev, https://population.un.org/wpp/
4. Life expectancy at birth (years)	WDI, https://datatopics.worldbank.org/world-development-indicators/
5. Total fertility rate (births per woman)	WDI, https://datatopics.worldbank.org/world-development-indicators/
6. Percent urban population (of total population)	WDI, https://datatopics.worldbank.org/world-development-indicators/
7. School enrollment, secondary (% gross)	WDI, https://datatopics.worldbank.org/world-development-indicators/
8. WB country income classification	World Bank, https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups
9. HIV prevalence (% of population ages 15-49)	WDI, https://datatopics.worldbank.org/world-development-indicators/ (from UNAIDS). To address missing values in some cases, additional data were obtained from the Global Burden of Disease Collaborative Network, Global Burden of Disease Study 2019 (GBD 2019) Results. Seattle, United States: Institute for Health Metrics and Evaluation (IHME), 2020, http://ghdx.healthdata.org/gbd-results-tool.
10. Per capita donor spending on health (non-PEPFAR)	OECD Creditor Reporting System database, https://stats.oecd.org/Index.aspx?DataSetCode=crs1
11. Per capita domestic health spending, government and private, PPP (current $)	WDI, https://datatopics.worldbank.org/world-development-indicators/

Table 2: Model Specifications
Model	Difference-in Difference Specification
1	Unadjusted model
2	Includes baseline variables 1-9
3	Includes baseline variables 1-11
4	Includes baseline variables 1-9, and yearly per capita donor spending on health (non-PEPFAR) by all donors.

Our final model for main reported results is model 4 which, in addition to baseline variables, includes a yearly estimate of donor health spending from all sources other than PEPFAR (including, for example, U.S. spending on other health areas as well as spending by other bilateral and multilateral donors on health) to adjust for potential confounding influences of these other health investments on all-cause mortality. We did not include domestic health spending as a baseline variable in this model due to the potential confounding with donor health spending. The pre-intervention period for this model started in 2002.

Each of our model specifications produced similar, statistically significant results. In our final model, almost all results were significant at the p<0.001 level; one result was significant at the p<0.01 and three were significant at p<0.05. We also ran all models with and without China and India, the two most populous countries in the world, to assess whether they were influencing the results. In both cases, PEPFAR’s impact was still significant and results were similar.

Despite the strengths of the difference-in-difference design, there are limitations to this approach. While we adjusted for numerous baseline factors that could be correlated with mortality outcomes, there may be other, unobservable factors that are not captured here. Similarly, while our baseline factors are also intended to adjust for selection bias, and make the PEPFAR and control groups more comparable, there may be other ways in which control countries differed from PEPFAR countries (and factors which influenced which countries received PEPFAR support), which could bias the estimates.

Table 3: Baseline Mean Mortality Rate, All Causes, 2004 (crude deaths per 1,000)
All PEPFAR countries	10.5
COP countries	12.6
Non-COP countries	9.4
PEPFAR Spending Intensity
High	12.3
Medium	9.7
Low	9.5

Table 4: Estimates of PEPFAR’s Impact on Mortality, 2004-2018 (Percent change in all-cause mortality rate)
Model Specification	Model 1	Model 2	Model 3	Model 4
All PEPFAR countries	-19.9%	-22.5%	-27.4%	-20.4%
COP countries	-22.8%	-26.8%	-29.6%	-25.7%
Non-COP countries	-17.9%	-19.7%	-25.0%	-16.6%
PEPFAR Spending Intensity
High	-25.0%	-29.3%	-30.6%	-26.6%
Medium	-20.0%	-21.4%	-28.3%	-14.0%
Low	-13.3%	-15.3%	-19.5%	-15.7%
Time Period: All PEPFAR countries
2004-2008	-9.0%	-11.2%	-13.9%	-7.9%
2004-2013	-15.0%	-17.3%	-21.2%	-15.0%
2004-2018	-19.9%	-22.5%	-27.4%	-20.4%
Time Period: PEPFAR COP countries
2004-2008	-7.8%	-11.0%	-12.3%	-8.8%
2004-2013	-15.9%	-19.5%	-21.5%	-18.2%
2004-2018	-22.8%	-26.8%	-29.6%	-25.7%
NOTE: Refer to Table 2 for model specifications.

Table 5: Estimates of PEPFAR’s Impact on Mortality, 2004-2018 (Percentage point difference-in-difference and standard errors)
Model Specification	Model 1	Model 2	Model 3	Model 4
All PEPFAR countries	-2.095*** (0.232)	-2.364*** (0.190)	-2.879*** (0.265)	-2.139*** (0.435)
COP countries	-2.875*** (0.294)	-3.380*** (0.262)	-3.726*** (0.341)	-3.232*** (0.541)
Non-COP countries	-1.682*** (0.236)	-1.847*** (0.179)	-2.346*** (0.218)	-1.565*** (0.423)
PEPFAR Spending Intensity
High	-3.081*** (0.304)	-3.608*** (0.247)	-3.770*** (0.299)	-3.271*** (0.495)
Medium	-1.942*** (0.304)	-2.080*** (0.244)	-2.744*** (0.326)	-1.357* (0.542)
Low	-1.263*** (0.304)	-1.451*** (0.244)	-1.850*** (0.315)	-1.494** (0.515)
Time Period: All PEPFAR countries
2004-2008	-0.949** (0.355)	-1.172*** (0.261)	-1.463*** (0.357)	-0.830* (0.401)
2004-2013	-1.571*** (0.269)	-1.813*** (0.208)	-2.224*** (0.287)	-1.578*** (0.413)
2004-2018	-2.095*** (0.232)	-2.364*** (0.190)	-2.879*** (0.265)	-2.139*** (0.435)
Time Period: PEPFAR COP countries
2004-2008	-0.988* (0.434)	-1.385*** (0.372)	-1.547** (0.473)	-1.114* (0.502)
2004-2013	-2.008*** (0.335)	-2.457*** (0.292)	-2.713*** (0.376)	-2.298*** (0.513)
2004-2018	-2.875*** (0.294)	-3.380*** (0.262)	-3.726*** (0.341)	-3.232*** (0.541)
NOTES: Refer to Table 2 for model specifications. Standard errors are shown in parentheses. ***p < 0.001   **p < 0.01 *p < 0.05