Introduction: Citizen report cards on health care providers have been identified as a potential means to increase citizen engagement, provider accountability and health systems performance.  Research in high-income settings indicates that the wording, presentation and display of performance information are critical to achieve these goals. However, there are limited insights on developing effective report card designs for middle- and low-income settings.  We conducted cognitive interviews to assess consumers’ understanding, interpretation of and preferences for displaying information for a health care report card in rural Tajikistan.

Materials and Methods: We recruited a convenience sample of 40 citizens (20 women and 20 men aged 18-45) from rural areas of two provinces of Tajikistan (Soghd and Khatlon oblasts).  The interview protocol was adapted from the model of cognitive interviews used in social science research to improve survey questionnaires.  We used multivariate regression to assess understanding and interpretation of the report card; chi2 tests to assess differences in preferences for displaying information; and tests of proportions to assess the preferred comparison group.

Results: Respondents understood the main idea of the report card and are not confused by the indicators or display.  However, many respondents had difficulties making comparisons, and when asked to identify worst-performing services.  Respondents preferred detailed rankings using school grades, comparisons of their local clinic with the regional or national average performance, and the use of color in the report card.  We found some heterogeneity across the two provinces.

Conclusions: Overall, our findings are promising regarding the citizens’ comprehension of health care report cards in rural Tajikistan, while underscoring the challenges of effectively providing health care performance information to communities.  Cognitive interviews and iterative testing can support an effective implementation of reporting initiatives.

Josephson, Erik, Jessica Gergen, Martha Coe, Samantha Ski, Supriya Madhavan, and Sebastian Bauhoff. 2017. “How do Performance-Based Financing Programs Measure Quality of Care? A Descriptive Analysis of 68 Quality Checklists from 28 Low and Middle-Income Countries.” Health Policy and Planning 32 (8): 1120-1126. Published paper (open access) Abstract

Objective: To systematically describe the length and content of quality checklists used in performance-based financing programs, their similarities and differences, and how checklists have evolved over time.

Methods: We compiled a list of supply-side, health facility-based performance-based financing (PBF) programs in low- and lower middle-income countries based on a document review. We then solicited PBF manuals and quality checklists from implementers and donors of these PBF mechanisms. We entered each indicator from each quality checklist into a database verbatim in English, and translated into English from French where appropriate, and categorized each indicator according to the Donabedian framework and an author-derived categorization.

Findings: We extracted 8,490 quality indicators from 68 quality checklists across 32 PBF implementations in 28 countries. On average, checklists contained 125 indicators; within the same program, checklists tend to grow as they are updated. Using the Donabedian framework, 80% of indicators were structure-type, 19% process-type, and less than 1% outcome-type. The author-derived categorization showed that 57% of indicators relate to availability of resources, 24% to managing the facility and 17% assess knowledge and effort. There is a high degree of similarity in a narrow set of indicators used in checklists for common service types such as maternal, neonatal and child health.

Conclusion:  Performance-based financing offers an appealing approach to targeting specific quality shortfalls and advancing toward the Sustainable Development Goals of high quality coverage.  Currently most indicators focus on structural issues and resource availability. There is scope to rationalize and evolve the quality checklists of these programs to help achieve national and global goals to improve quality of care.

Bauhoff, Sebastian, Dirk Göpffarth, Lisa Fischer, and Amelie C. Wuppermann. 2017. “Plan Responses to Diagnosis-Based Payment: Evidence from Germany's Morbidity-Based Risk Adjustment.” Journal of Health Economics 56: 397-413. Published paper (gated) Abstract

Many competitive health insurance markets adjust payments to participating health plans according to their enrollees’ risk – including based on diagnostic information.  We investigate responses of German health plans to the introduction of morbidity-based risk adjustment in the Statutory Health Insurance in 2009, which triggers payments based on “validated” diagnoses by providers.  Using the regulator’s data from office-based physicians, we estimate a difference-in-difference analysis of the change in the share and number of validated diagnoses for ICD codes that are inside or outside the risk adjustment but are otherwise similar.  We find a differential increase in the share of validated diagnoses of 2.6 and 3.6 percentage points (3-4%) between 2008 and 2013.  This increase appears to originate from both a shift from not-validated toward validated diagnoses and an increase in the number of such diagnoses.  Overall, our results indicate that plans were successful in influencing physicians’ coding practices in a way that could lead to higher payments.

Also available as CESIfo Working Paper 6507

Gergen, Jessica, Erik Josephson, Martha Coe, Samantha Ski, Supriya Madhavan, and Sebastian Bauhoff. 2017. “Quality of Care in Performance-Based Financing: How It Is Incorporated in 32 Programs Across 28 Countries.” Global Health: Science and Practice 5 (1): 90-107. Published paper (open access) Abstract

Objective: To describe how quality of care is incorporated into performance-based financing (PBF) programs, what quality indicators are being used, and how these indicators are measured and verified.

Methods: An exploratory scoping methodology was used to characterize the full range of quality components in 32 PBF programs, initiated between 2008 and 2015 in 28 low- and middle-income countries, totaling 68 quality tools and 8,490 quality indicators. The programs were identified through a review of the peer-reviewed and gray literature as well as through expert consultation with key donor representatives.

Findings: Most of the PBF programs were implemented in sub-Saharan Africa and most were funded primarily by the World Bank. On average, PBF quality tools contained 125 indicators predominately assessing maternal, newborn, and child health and facility management and infrastructure. Indicators were primarily measured via checklists (78%, or 6,656 of 8,490 indicators), which largely (over 90%) measured structural aspects of quality, such as equipment, beds, and infrastructure. Of the most common indicators across checklists, 74% measured structural aspects and 24% measured processes of clinical care. The quality portion of the payment formulas were in the form of bonuses (59%), penalties (27%), or both (hybrid) (14%). The median percentage (of a performance payment) allocated to health facilities was 60%, ranging from 10% to 100%, while the median percentage allocated to health care providers was 55%, ranging from 20% to 80%. Nearly all of the programs included in the analysis (91%, n=29) verified quality scores quarterly (every 3 months), typically by regional government teams.

Conclusion: PBF is a potentially appealing instrument to address shortfalls in quality of care by linking verified performance measurement with strategic incentives and could ultimately help meet policy priorities at the country and global levels, including the ambitious Sustainable Development Goals. The substantial variation and complexity in how PBF programs incorporate quality of care considerations suggests a need to further examine whether differences in design are associated with differential program impacts.


Hoerl, Maximiliane, Amelie Wuppermann, Silvia Helena Barcellos, Sebastian Bauhoff, and Joachim Winter. 2017. “Knowledge as a Predictor of Insurance Coverage under the Affordable Care Act.” Medical Care 55 (4): 428-435. Published paper (gated) Abstract


Background: The Affordable Care Act established policy mechanisms to increase health insurance coverage in the United States. While insurance coverage has increased, 10%-15% of the US population remains uninsured.

Objectives: To assess whether health insurance literacy and financial literacy predict being uninsured, covered by Medicaid, or covered by Marketplace insurance, holding demographic characteristics, attitudes toward risk, and political affiliation constant.

Research Design: Analysis of longitudinal data from fall 2013 and spring 2015 including financial and health insurance literacy and key covariates collected in 2013.

Subjects: A total of 2742 US residents ages 18-64, 525 uninsured in fall 2013, participating in the RAND American Life Panel, a nationally representative internet panel.

Measures: Self-reported health insurance status and type as of spring 2015.

Results: Among the uninsured in 2013, higher financial and health insurance literacy were associated with greater probability of being insured in 2015. For a typical uninsured individual in 2013, the probability of being insured in 2015 was 8.3 percentage points higher with high compared with low financial literacy, and 9.2 percentage points higher with high compared with low health insurance literacy. For the general population, those with high financial and health insurance literacy were more likely to obtain insurance through Medicaid or the Marketplaces compared with being uninsured. The magnitude of coefficients for these predictors was similar to that of commonly used demographic covariates.

Conclusions: A lack of understanding about health insurance concepts and financial illiteracy predict who remains uninsured. Outreach and consumer-education programs should consider these characteristics.


Ngo, Diana, Tisamarie Sherry, and Sebastian Bauhoff. 2017. “Health System Changes under Pay-For-Performance: the Effects of Rwanda’s National Programme on Facility Inputs.” Health Policy and Planning 32 (1): 11-20. Published paper (gated) Abstract

Pay-for-performance (P4P) programs have been introduced in numerous developing countries with the goal of increasing the provision and quality of health services through financial incentives. Despite the popularity of P4P, there is limited evidence on how providers achieve performance gains and how P4P affects health system quality by changing structural inputs. We explore these two questions in the context of Rwanda's 2006 national P4P program by examining the program's impact on structural quality measures drawn from international and national guidelines. Given the program's previously documented success at increasing institutional delivery rates, we focus on a set of delivery-specific and more general structural inputs. Using the program's quasi-randomized rollout, we apply multivariate regression analysis to short-run facility data from the 2007 Service Provision Assessment. We find positive program effects on the presence of maternity-related staff, the presence of covered waiting areas, and a management indicator and a negative program effect on delivery statistics monitoring. We find no effects on a set of other delivery-specific physical resources, delivery-specific human resources, delivery-specific operations, general physical resources, and general human resources. Using mediation analysis, we find that the positive input differences explain a small and insignificant fraction of P4P's impact on institutional delivery rates. The results suggest that P4P increases provider availability and facility operations but is only weakly linked with short-run structural health system improvements overall.

Bauhoff, Sebastian, Adrian Montero, and Deborah Scharf. 2017. “Perceptions of E-Cigarettes: A Comparison of Adult Smokers and Non-Smokers in a Mechanical Turk Sample.” American Journal of Drug and Alcohol Abuse 43 (3): 311-323. Published paper (gated) Abstract

Background: Given plans to extend its regulatory authority to e-cigarettes, the Food and Drug Administration (FDA) urgently needs to understand how e-cigarettes are perceived by the public.

Objectives: To examine how smoking status impacts adult perceptions and expectations of e-cigarettes.

Methods: We used Mechanical Turk (MTurk), a “crowdsourcing” platform, to rapidly survey a large (n=796; female=381; male=415), diverse sample of adult ever (44%) and never smokers (56%), with ever (28%) and never (72%) users of e-cigarettes.  

Results: Smokers and non-smokers learned about e-cigarettes primarily through the internet and conversations with others.  Ever smokers were more likely than never smokers, and female current smokers were more likely than female former smokers, to have learned about e-cigarettes from point of sale advertising (p’s<0.05) and to believe that e-cigarettes help smokers quit (p’s<0.05).  Among never-users of e-cigarettes, current smokers were more likely than never smokers and former smokers to report that they would try e-cigarettes in the future (p’s<0.01). Current smokers’ top reason for wanting to try e-cigarettes was to quit or reduce smoking (56%), while never and former smokers listed curiosity. In contrast, female current smokers’ top reason for not trying e-cigarettes was health and safety concerns (44%) while males indicated expense (44%).

Conclusions: Adult smokers and non-smokers have different perceptions and expectations of e-cigarettes.  Public health messages regarding e-cigarettes may need to be tailored separately for persons with and without a history of using conventional cigarettes. Tailoring messages by gender within smoker groups may also improve their impact.

Sherry, Tisamarie, Sebastian Bauhoff, and Manoj Mohanan. 2017. “Multitasking and Heterogeneous Treatment Effects in Pay-for-Performance in Health Care: Evidence from Rwanda.” American Journal of Health Economics 3 (2): 192-226. Published paper (gated) Abstract

Performance-based contracting is particularly challenging in health care, where multiple agents, information asymmetries and other market failures compound the critical contracting concern of multitasking. As performance-based contracting grows in developing countries, it is critical to better understand not only intended program impacts on rewarded outcomes, but also unintended program impacts such as multitasking and heterogeneous program effects in order to guide program design and scale-up. We use two waves of data from the Rwanda Demographic and Health Surveys collected before and after the quasi-experimental roll- out of Rwanda’s national pay-for-performance (P4P) program to analyze impacts on utilization of healthcare services, health outcomes and unintended consequences of P4P. We find that P4P improved some rewarded services, as well as some services that were not directly rewarded, but had no statistically significant impact on health outcomes. We do not find evidence that clearly suggests multitasking. We find that program effects vary by baseline levels of facility quality, with most improvements seen in the medium quality tier.

Bauhoff, Sebastian, Olesya Tkacheva, Lila Rabinobich, and Olena Bogdan. 2016. “Opportunities and Considerations for Citizen Report Cards for Primary Care: Qualitative evidence from Rural Tajikistan.” Health Policy and Planning 31 (2): 259-266. Published paper (gated) Abstract

Transparency interventions, such as public reporting, have emerged as a potential policy approach to improving the performance of health care providers in resource-constrained settings.  We report on results from focus groups and key informant interviews in rural areas of two Tajik provinces, Soghd and Khatlon, with regards to three important initial considerations for developing a report card initiative for primary health care in this setting: selecting indicators for the report card, collecting data, and working with existing institutions and stakeholders.  The findings suggest that citizens are able to articulate and prioritize concerns with respect to local health care services.  Participants indicated a preference for arms-length collection of sensitive feedback on local providers.  Since citizens and local institutions have close and important relations with their local health care providers, there may be scope for a trusted external actor, such as a non-governmental organization, to facilitate the report card process.

Wuppermann, Amelie C., Sebastian Bauhoff, Andreas Filser, and Manfred Antoni. 2016. “Krankenkassen im Regionalen Vergleich.” Krankenversicherung im Rating: Leistungsbewertungen und Management als Schlüsselfaktoren, edited by Thomas Adolph, Oliver Everling, and Marco Metzler, 3rd ed., 97-121. Heidelberg: Springer Gabler. Published chapter (gated)
Morton, Matthew, Somil Nagpal, Rajeev Sadanandan, and Sebastian Bauhoff. 2016. “India’s Largest Hospital Insurance Program Faces Challenges in Using Claims Data to Measure Quality.” Health Affairs 35 (10): 1792-1799. Published paper (gated) Abstract

The routine data generated by India’s universal coverage programs offer an important opportunity to evaluate and track the quality of health care systematically and on a large scale. We examined the potential and challenges of measuring the quality of hospital care through claims data from India’s hospital insurance program for the poor, Rashtriya Swasthya Bima Yojana (RSBY). Using data from one district in India, we illustrate how these data already provide useful insights and show that simple efforts to enhance data quality and an effort to expand the data captured could facilitate RSBY’s ability to track quality of care. The data collected by RSBY has significant potential to characterize and uncover the provision of low-quality care and help inform much-needed efforts to raise the quality of hospital care.

We study how the announcement by CVS Health, a large US-based pharmacy chain, to stop selling tobacco products affected its share price and that of its close competitors, as well as major tobacco companies. Combining event study and synthetic control methodologies we compare measures of CVS’s stock market valuation with those of a peer group consisting of large publicly listed firms that are part of Standard & Poor’s S&P 500 stock market index. CVS’s announcement is associated with a short-term decrease in its share price, whereas close competitors have benefitted from CVS’ decision. We also find a negative share price effect for Altria, the largest US domestic tobacco firm. Overall our findings are consistent with markets expecting consumers to shift from CVS to alternative outlets in the short-run, and interpreting CVS’ decision to drop tobacco products as signal that other firms may follow suit.


Lee, Elizabeth, Supriya Madhavan, and Sebastian Bauhoff. 2016. “Levels and Variations in the Quality of Facility-Based Antenatal Care in Kenya: Evidence from the 2010 Service Provision Assessment.” Health Policy and Planning 31 (6): 777-784. Published paper (open access) Abstract

Quality of care is emerging as an important concern for low- and middle-income countries working to expand and improve coverage. However, there is limited systematic, large-scale empirical guidance to inform policy design. Our study operationalized indicators for six dimensions of quality of care that are captured in currently available, standardized Service Provision Assessments. We implemented these measures to assess the levels and heterogeneity of antenatal care in Kenya. Using our indicator mix, we find that performance is low overall and that there is substantial variation across provinces, management authority and facility type. Overall, facilities performed highest in the dimensions of efficiency and acceptability/patient-centeredness, and lowest on effectiveness and accessibility. Public facilities generally performed worse or similarly to private or faith-based facilities. We illustrate how these data and methods can provide readily-available, low-cost decision support for policy.

Burns, Rachel M, Rosalie Pacula, Sebastian Bauhoff, Adam J Gordon, Hollie Hendrikson, Douglas L Leslie, and Bradley D Stein. 2016. “Policies Related to Opioid Agonist Therapy for Opioid Use Disorders: The Evolution of State Policies from 2004 to 2013.” Substance Abuse 37 (1): 63-69. Published paper (gated) Abstract

Background: State Medicaid policies play an important role in Medicaid-enrollees' access to and use of opioid agonists, such as methadone and buprenorphine, in the treatment of opioid use disorders. Little information is available, however, regarding the evolution of state policies facilitating or hindering access to opioid agonists among Medicaid-enrollees.

Methods: During 2013–14, we surveyed state Medicaid officials and other designated state substance abuse treatment specialists about their state's recent history of Medicaid coverage and policies pertaining to methadone and buprenorphine. We describe the evolution of such coverage and policies and present an overview of the Medicaid policy environment with respect to opioid agonist therapy from 2004 to 2013.

Results: Among our sample of 45 states with information on buprenorphine and methadone coverage, we found a gradual trend toward adoption of coverage for opioid agonist therapies in state Medicaid agencies. In 2013, only 11% of states in our sample (n = 5) had Medicaid policies that excluded coverage for methadone and buprenorphine, while 71% (n = 32) had adopted or maintained policies to cover both buprenorphine and methadone among Medicaid-enrollees. We also noted an increase in policies over the time period that may have hindered access to buprenorphine and/or methadone.

Conclusions: There appears to be a trend for states to enact policies increasing Medicaid coverage of opioid agonist therapies, while in recent years also enacting policies, such as prior authorization requirements, that potentially serve as barriers to opioid agonist therapy utilization. Greater empirical information about the potential benefits and potential unintended consequences of such policies can provide policymakers and others with a more informed understanding of their policy decisions.

Stein, Bradley D, Rosalie Pacula, Adam J Gordon, Rachel M Burns, Douglas L Leslie, Mark J Sorbero, Sebastian Bauhoff, Todd W Mandell, and Andrew W Dick. 2015. “Where Is Buprenorphine Dispensed to Treat Opioid Use Disorders? The Role of Private Offices, Opioid Treatment Programs, and Substance Abuse Treatment Facilities in Urban and Rural Counties.” Milbank Quarterly 93 (3): 561–583. Published paper (gated) Abstract

Context: Opioid use disorders are a significant public health problem. In 2002, the FDA approved buprenorphine as an opioid use disorder treatment when prescribed by waivered physicians who were limited to treating 30 patients at a time. In 2006, federal legislation raised this number to 100 patients. Although federal legislators are considering increasing these limits further and expanding prescribing privileges to nonphysicians, little information is available regarding the impact of such changes on buprenorphine use. We therefore examined the impact of the 2006 legislation—as well as the association between urban and rural waivered physicians, opioid treatment programs, and substance abuse treatment facilities—on buprenorphine distributed per capita over the past decade.

Methods: Using 2004-2011 state-level data on buprenorphine dispensed and county-level data on the number of buprenorphine-waivered physicians and substance abuse treatment facilities using buprenorphine, we estimated a multivariate ordinary least squares regression model with state fixed effects of a state’s annual total buprenorphine dispensed per capita as a function of the state’s number of buprenorphine providers.

Findings: The amount of buprenorphine dispensed has been increasing at a greater rate than the number of buprenorphine providers. The number of physicians waivered to treat 100 patients with buprenorphine in both rural and urban settings was significantly associated with increased amounts of buprenorphine dispensed per capita. There was no significant association in the growth of buprenorphine distributed and the number of physicians with 30-patient waivers.

Conclusions: The greater amounts of buprenorphine dispensed are consistent with the potentially greater use of opioid agonists for opioid use disorder treatment, though they also make their misuse more likely. The changes after the 2006 legislation suggest that policies focused on increasing the number of patients that a single waivered physician could safely and effectively treat could be more effective in increasing buprenorphine use than would alternatives such as opening new substance abuse treatment facilities or raising the overall number of waivered physicians.

Taylor, Erin, Evan Saltzman, Sebastian Bauhoff, Rosalie Pacula, and Christine Eibner. 2015. “More Choice in Health Insurance Marketplaces May Reduce the Value of Subsidies Available to Low-Income Enrollees.” Health Affairs 34 (1): 104-110. Published paper (gated) Abstract

Federal subsidies available to enrollees in health insurance Marketplaces are pegged to the premium of the second-lowest-cost silver plan available in each rating area (as defined by each state). People who qualify for the subsidy contribute a percentage of their income to purchase coverage, and the federal government covers the remaining cost up to the price of that premium. Because the number of plans offered and plan premiums vary substantially across rating areas, the effective value of the subsidy may vary geographically. We found that the availability of more plans in a rating area was associated with lower premiums but higher deductibles for enrollees in the second-lowest-cost silver plan. In rating areas with more than twenty plans, the average deductible in the second-lowest-cost silver plan was nearly $1,000 higher than it was in rating areas with fewer than thirteen plans. Because premium costs for second-lowest-cost silver plans are capped, deductibles may be a more salient measure of plan value for enrollees than premiums are. Greater standardization of plans or an alternative approach to calculating the subsidy could provide a more consistent benefit to enrollees across various rating areas.

Bauhoff, Sebastian. 2014. “Self-Report Bias in Estimating Cross-Sectional and Treatment Effects.” Encyclopedia of Quality of Life Research , edited by Alex C. Michalos, 5798-5801. Berlin: Springer. Published chapter (gated)
Barcellos, Silvia, Amelie Wuppermann, Katherine Carman, Sebastian Bauhoff, Daniel McFadden, Arie Kapteyn, Joachim Winter, and Dana Goldman. 2014. “Preparedness of Americans for the Affordable Care Act.” Proceedings of the National Academy of Sciences 111 (15): 5497–5502. Published paper (open access) Abstract
This paper investigates whether individuals are sufficiently informed to make reasonable choices in the health insurance exchanges established by the Affordable Care Act (ACA). We document knowledge of health reform, health insurance literacy, and expected changes in healthcare using a nationally representative survey of the US population in the 5 wk before the introduction of the exchanges, with special attention to subgroups most likely to be affected by the ACA. Results suggest that a substantial share of the population is unprepared to navigate the new exchanges. One-half of the respondents did not know about the exchanges, and 42% could not correctly describe a deductible. Those earning 100–250% of federal poverty level (FPL) correctly answered, on average, 4 out of 11 questions about health reform and 4.6 out of 7 questions about health insurance. This compares with 6.1 and 5.9 correct answers, respectively, for those in the top income category (400% of FPL or more). Even after controlling for potential confounders, a low-income person is 31% less likely to score above the median on ACA knowledge questions, and 54% less likely to score above the median on health insurance knowledge than a person in the top income category. Uninsured respondents scored lower on health insurance knowledge, but their knowledge of ACA is similar to the overall population. We propose that simplified options, decision aids, and health insurance product design to address the limited understanding of health insurance contracts will be crucial for ACA’s success.
Mohanan, Manoj, Sebastian Bauhoff, Gerald La Forgia, Kimberly Singer Babiarz, Kultar Singh, and Grant Miller. 2014. “Effect of Chiranjeevi Yojana on Institutional Deliveries and Neonatal and Maternal Outcomes in Gujarat, India: A Difference-in-Differences Analysis.” Bulletin of the World Health Organization 92 (3): 187-194. Published paper (open access) Abstract

Objective: To evaluate the effect of the Chiranjeevi Yojana programme, a public–private partnership to improve maternal and neonatal health in Gujarat, India.

Methods: A household survey (n = 5597 households) was conducted in Gujarat to collect retrospective data on births within the preceding 5 years. In an observational study using a difference-in-differences design, the relationship between the Chiranjeevi Yojana programme and the probability of delivery in health-care institutions, the probability of obstetric complications and mean household expenditure for deliveries was subsequently examined. In multivariate regressions, individual and household characteristics as well as district and year fixed effects were controlled for. Data from the most recent District Level Household and Facility Survey (DLHS-3) wave conducted in Gujarat (n = 6484 households) were used in parallel analyses.

Findings: Between 2005 and 2010, the Chiranjeevi Yojana programme was not associated with a statistically significant change in the probability of institutional delivery (2.42 percentage points; 95% confidence interval, CI: −5.90 to 10.74) or of birth-related complications (6.16 percentage points; 95% CI: −2.63 to 14.95). Estimates using DLHS-3 data were similar. Analyses of household expenditures indicated that mean household expenditure for private-sector deliveries had either not fallen or had fallen very little under the Chiranjeevi Yojana programme.

Conclusion: The Chiranjeevi Yojana programme appears to have had no significant impact on institutional delivery rates or maternal health outcomes. The absence of estimated reductions in household spending for private-sector deliveries deserves further study.