To provide an overview of applications of simulation models in health policy, analyze the use of best reporting practices, and assess the reproducibility and quality of studies.
Systematic review of studies that used simulation modeling to inform health policy.
Web of Science and PubMed until March 2016.
ELIGIBILITY CRITERIA FOR STUDY SELECTION
Studies that used simulation modeling as the core method to address any health policy questions.
MAIN OUTCOME MEASURES
Reproducibility was assessed using predefined, categorical criteria. Health policy domain distribution and changes in quality over time were well-characterized using MeSH terms and model characteristics, respectively.
This systematic search identified 1,613 eligible studies. A subset of 100 articles (50 top-cited and 50 randomly selected) were selected to analyze in-depth criteria for reproducibility and quality. We found an exponential growth in simulation-based studies of health policy over the past half century, with the highest growth in dynamic modeling approaches. The largest subset of studies is focused on disease policy models (70%), within which pathological conditions, viral diseases, neoplasms, and cardiovascular diseases account for about one-third of the articles. Nearly half of the 1,613 articles do not report the details of their models. Significant gaps against modeling best practices could be found in both random and highly-cited samples; only seven of 26 in-depth evaluation criteria were satisfied by more than 80% of samples. We found no evidence that the highly-cited sample of articles is better at following modeling best practices, defined as reporting model equations, specifying modeling assumptions, and discussing limitations, among others.
Simulation models are increasingly used to inform health policy, yet there is great variation in their adherence to best research practices. Our results suggest areas ripe for increased application of simulation modeling, as well as opportunities to enhance the rigor and documentation in the conduct and reporting of simulation studies in health policy.
WHAT IS ALREADY KNOWN ON THIS TOPIC
Simulation models help to better understand the impacts of health policies and minimize unintended consequences within a wide variety of complex health issues.
Given the popularity and impact of these methods in health policy and decision-making, rigor and quality of design, conduction, validation, reporting, and disseminating are essential.
The status of recent efforts to increase the reproducibility and quality of simulation modeling remains unknown.
WHAT THIS STUDY ADDS
Simulation models have become more popular and advanced over the last half century to better encompass and represent the real-world complexity of health systems.
Our review highlights the imbalance in the applications of simulation modeling in health-policy topics; >70% of the studies’ topics were focused on biomedical disease, meanwhile, health policies are contained within a much broader public health domain.
The reproducibility and quality of simulation models need to be improved; many studies do not report elements necessary to evaluate and reproduce their work, and top-cited studies exhibit similar weaknesses to studies with fewer citations.
Objectives: To investigate the impact of insurance coverage on the adoption of customized individually made (CIM) knee implants, and to compare patient outcomes and cost-effectiveness of off-the-shelf (OTS) and CIM implants.
Study Design: A system dynamics simulation model is developed to study adoption dynamics of CIM and meet the research objectives.
Methods: The model reproduced the historical data on primary and revision knee replacement implants obtained from the literature and the Nationwide Inpatient Sample. Then, the dynamics of adoption of CIM implants were simulated from 2018 to 2026. The rate of 90-day readmission, 3-year revision surgery, recovery period, time savings in operating rooms, and the associated cost within three years of primary knee replacement implants were used as performance metrics.
Results: The simulation results indicate that, by 2026, an adoption rate of 90% for CIM implants can reduce the number of readmissions and revision surgeries by 62% and 39%, respectively, and can save hospitals and surgeons 6% on procedure time, and cut down cumulative healthcare costs by approximately $38 billion.
Conclusions: CIM implants have the potential to deliver high-quality care while decreasing overall healthcare costs, but their adoption requires the expansion of current insurance coverage. This work presents a first systematic study to understand the dynamics of adoption of CIM knee implants and instrumentation. More broadly, the current modeling approach and systems thinking perspective could be utilized to consider the adoption of any emerging customized therapies for personalized medicine.
Overall impact of public health prevention interventions relies not only on the average efficacy of an intervention, but also on the successful adoption, implementation, and maintenance (AIM) of that intervention. In this study, we aim to understand the dynamics that regulate AIM of organizational level intervention programs. We focus on two well-documented obesity prevention interventions, implemented in food carry-outs and stores in low-income urban areas of Baltimore, Maryland, which aimed to improve dietary behaviour for adults by providing access to healthier foods and point-of-purchase promotions. Building on data from field observations, in-depth interviews, and data discussed in previous publications, as well as the strategy and organizational behaviour literature, we developed a system dynamics model of the key processes of AIM. With simulation analysis, we show several reinforcing mechanisms that span stakeholder motivation, communications, and implementation quality and costs can turn small changes in the process of AIM into big difference in the overall impact of the intervention. Specifically, small changes in the allocation of resources to communication with stakeholders of intervention could have a nonlinear long-term impact if those additional resources can turn stakeholders into allies of the intervention, reducing the erosion rates and enhancing sustainability. We present how the dynamics surrounding communication, motivation, and erosion can create significant heterogeneity in the overall impact of otherwise similar interventions. Therefore, careful monitoring of how those dynamics unfold, and timely adjustments to keep the intervention on track are critical for successful implementation and maintenance.
Background: Over the past decade, clinical care has become globally dependent on information technology. The cybersecurity of health care information systems is now an essential component of safe, reliable, and effective health care delivery. Objective: The objective of this study was to provide an overview of the literature at the intersection of cybersecurity and health care delivery. Methods: A comprehensive search was conducted using PubMed and Web of Science for English-language peer-reviewed articles. We carried out chronological analysis, domain clustering analysis, and text analysis of the included articles to generate a high-level concept map composed of specific words and the connections between them. Results: Our final sample included 472 English-language journal articles. Our review results revealed that majority of the articles were focused on technology: Technology–focused articles made up more than half of all the clusters, whereas managerial articles accounted for only 32% of all clusters. This finding suggests that nontechnological variables (human–based and organizational aspects, strategy, and management) may be understudied. In addition, Software Development Security, Business Continuity, and Disaster Recovery Planning each accounted for 3% of the studied articles. Our results also showed that publications on Physical Security account for only 1% of the literature, and research in this area is lacking. Cyber vulnerabilities are not all digital; many physical threats contribute to breaches and potentially affect the physical safety of patients. Conclusions: Our results revealed an overall increase in research on cybersecurity and identified major gaps and opportunities for future work.
Background: Cybersecurity incidents are a growing threat to the health care industry in general and hospitals in particular. The health care industry has lagged behind other industries in protecting its main stakeholder (ie, patients), and now hospitals must invest considerable capital and effort in protecting their systems. However, this is easier said than done because hospitals are extraordinarily technology-saturated, complex organizations with high end point complexity, internal politics, and regulatory pressures.
Objective: The purpose of this study was to develop a systematic and organizational perspective for studying (1) the dynamics of cybersecurity capability development at hospitals and (2) how these internal organizational dynamics interact to form a system of hospital cybersecurity in the United States.
Methods: We conducted interviews with hospital chief information officers, chief information security officers, and health care cybersecurity experts; analyzed the interview data; and developed a system dynamics model that unravels the mechanisms by which hospitals build cybersecurity capabilities. We then use simulation analysis to examine how changes to variables within the model affect the likelihood of cyberattacks across both individual hospitals and a system of hospitals.
Results: We discuss several key mechanisms that hospitals use to reduce the likelihood of cybercriminal activity. The variable that most influences the risk of cyberattack in a hospital is end point complexity, followed by internal stakeholder alignment. Although resource availability is important in fueling efforts to close cybersecurity capability gaps, low levels of resources could be compensated for by setting a high target level of cybersecurity.
Conclusions: To enhance cybersecurity capabilities at hospitals, the main focus of chief information officers and chief information security officers should be on reducing end point complexity and improving internal stakeholder alignment. These strategies can solve cybersecurity problems more effectively than blindly pursuing more resources. On a macro level, the cyber vulnerability of a country’s hospital infrastructure is affected by the vulnerabilities of all individual hospitals. In this large system, reducing variation in resource availability makes the whole system less vulnerable—a few hospitals with low resources for cybersecurity threaten the entire infrastructure of health care. In other words, hospitals need to move forward together to make the industry less attractive to cybercriminals. Moreover, although compliance is essential, it does not equal security. Hospitals should set their target level of cybersecurity beyond the requirements of current regulations and policies. As of today, policies mostly address data privacy, not data security. Thus, policy makers need to introduce policies that not only raise the target level of cybersecurity capabilities but also reduce the variability in resource availability across the entire health care system.
We developed a simulation game to study the effectiveness of decision-makers in overcoming two complexities in building cybersecurity capabilities: potential delays in capability development; and uncertainties in predicting cyber incidents. Analyzing 1479 simulation runs, we compared the performances of a group of experienced professionals with those of an inexperienced control group. Experienced subjects did not understand the mechanisms of delays any better than inexperienced subjects; however, experienced subjects were better able to learn the need for proactive decision-making through an iterative process. Both groups exhibited similar errors when dealing with the uncertainty of cyber incidents. Our findings highlight the importance of training for decision-makers with a focus on systems thinking skills, and lay the groundwork for future research on uncovering mental biases about the complexities of cybersecurity.
Background: Connected medical devices and electronic health records have added important functionality to patient care, but have also introduced a range of cybersecurity concerns. When a healthcare organization suffers from a cybersecurity incident, its incident response strategies are critical to the success of its recovery.
Objective: In this article, we identify gaps in research concerning cybersecurity response plans in healthcare. Through a systematic literature review, we develop aggregated strategies that professionals can use to construct better response strategies in their organizations.
Methods: We reviewed journal articles on cyber incident response plans in healthcare published in PubMed and Web of Science. We sought to collect articles on the intersection of cybersecurity and healthcare that focused on incident response strategies.
Results: We identified and reviewed 13 articles for cybersecurity response recommendations. We then extracted information such as research methods, findings, and implications. Finally, we synthesized the recommendations into a framework of eight aggregated response strategies (EARS) that fall under managerial and technological categories.
Conclusions: We conducted a systematic review of the literature on cybersecurity response plans in healthcare and developed a novel framework for response strategies that could be deployed by healthcare organizations. More work is needed to evaluate incident response strategies in healthcare.
The drug shortage crisis in the last decade not only increased health care costs but also jeopardized patients’ health across the United States. Ensuring that any drug is available to patients at health care centers is a problem that official health care administrators and other stakeholders of supply chains continue to face. Furthermore, managing pharmaceutical supply chains is very complex, as inevitable disruptions occur in these supply chains (exogenous factors), which are then followed by decisions members make after such disruptions (internal factors). Disruptions may occur due to increased demand, a product recall, or a manufacturer disruption, among which product recalls—which happens frequently in pharmaceutical supply chains—are least studied. We employ a mathematical simulation model to examine the effects of product recalls considering different disruption profiles, e.g., the propagation in time and space, and the interactions of decision makers on drug shortages to ascertain how these shortages can be mitigated by changing inventory policy decisions. We also measure the effects of different policy approaches on supply chain disruptions, using two performance measures: inventory levels and shortages of products at health care centers. We then analyze the results using an approach similar to data envelopment analysis to characterize the efficient frontier (best inventory policies) for varying cost ratios of the two performance measures as they correspond to the different disruption patterns. This analysis provides insights into the consequences of choosing an inappropriate inventory policy when disruptions take place.
A wide range of modeling methods have been used to inform health policies. In this chapter, we describe three models for understanding the complexities of post-traumatic stress disorder (PTSD), a major mental disorder. The models are: (1) a qualitative model describing the social and psychological complexities of PTSD treatment; (2) a system dynamics model of a population of PTSD patients in the military and the Department of Veterans Affairs (VA); and (3) a Monte Carlo simulation model of PTSD prevalence and clinical demand over time among the OEF/OIF population. These models have two characteristics in common. First, they take systems approaches. In all models, we set a large boundary and look at the whole system, incorporating both military personnel and veterans. Second, the models are informed by a wide range of qualitative and quantitative data. Model I is rooted in qualitative data, and models II and III are calibrated to several data sources. These models are used to analyze the effects of different policy alternatives, such as more screening, more resiliency, and better recruitment procedures, on PTSD prevalence. They also provide analysis of healthcare costs in the military and the VA for each policy. Overall, the developed models offer examples of modeling techniques that incorporate a wide range of data sources and inform policy makers in developing programs for mitigating PTSD, a major premise of policy informatics.
The systemic interactions among depressive symptoms, rumination, and stress are important to understanding depression but have not yet been quantified. In this article, we present a system dynamics simulation model of depression that captures the reciprocal relationships among stressors, rumination, and depression. Building on the response styles theory, this model formalizes three interdependent mechanisms: 1) Rumination contributes to ‘keeping stressors alive’; 2) Rumination has a direct impact on depressive symptoms; and 3) Both ‘stressors kept alive’ and current depressive symptoms contribute to rumination. The strength of these mechanisms is estimated using data from 661 adolescents (353 girls and 308 boys) from two middle schools (grades 6–8). These estimates indicate that rumination contributes to depression by keeping stressors ‘alive’—and the individual activated—even after the stressor has ended. This mechanism is stronger among girls than boys, increasing their vulnerability to a rumination reinforcing loop. Different profiles of depression emerge over time depending on initial levels of depressive symptoms, rumination, and stressors as well as the occurrence rate for stressors; levels of rumination and occurrence of stressors are stronger contributors to long-term depression. Our systems model is a steppingstone towards a more comprehensive understanding of depression in which reinforcing feedback mechanisms play a significant role. Future research is needed to expand this simulation model to incorporate other drivers of depression and provide a more holistic tool for studying depression.
In this study, we present case studies to explore the dynamics of implementation and maintenance of health interventions. We analyze how specific interventions are built and eroded, how the building and erosion mechanisms are interconnected, and why we can see significantly different erosion rates across otherwise similar organizations. We use multiple comparative obesity prevention case studies to provide empirical information on the mechanisms of interest, and use qualitative systems modeling to integrate our evolving understanding into an internally consistent and transparent theory of the phenomenon. Our preliminary results identify reinforcing feedback mechanisms, including design of organizational processes, motivation of stakeholders, and communication among stakeholders, which influence implementation and maintenance of intervention components. Over time, these feedback mechanisms may drive a wedge between otherwise similar organizations, leading to distinct configurations of implementation and maintenance processes.
Rapid growth in scientific output requires methods for quantitative synthesis of prior research, yet current meta-analysis methods limit aggregation to studies with similar designs. Here we describe and validate Generalized Model Aggregation (GMA), which allows researchers to combine prior estimated models of a phenomenon into a quantitative meta-model, while imposing few restrictions on the structure of prior models or on the meta-model. In an empirical validation, building on 27 published equations from 16 studies, GMA provides a predictive equation for Basal Metabolic Rate that outperforms existing models, identifies novel nonlinearities, and estimates biases in various measurement methods. Additional numerical examples demonstrate the ability of GMA to obtain unbiased estimates from potentially mis-specified prior studies. Thus, in various domains, GMA can leverage previous findings to compare alternative theories, advance new models, and assess the reliability of prior studies, extending meta-analysis toolbox to many new problems.
As a lot of communication and media consumption moves online, people may be exposed to a wider population and more diverse opinions. However, individuals may act differently when faced with opinions far removed from their own. Moreover, changes in the frequency of visits, posting, and other forms of expression could lead to narrowing of the opinions that each person observes, as well as changes in the customer base for online platforms. Despite increasing research on the rise and fall of online social media outlets, user activity in response to exposure to others’ opinions has received little attention. In this study, we first introduce a method that maps opinions of individuals and their generated content on a multi-dimensional space by factorizing an individual–object interaction (e.g., user-news rating) matrix. Using data on 6151 users interacting with 287,327 pieces of content over 21 months on a social media platform we estimate changes in individuals’ activities in response to interaction with content expressing a variety of opinions. We find that individuals increase their online activities when interacting with content close to their own opinions, and interacting with extreme opinions may decrease their activities. Finally, developing an agent-based simulation model, we study the effect of the estimated mechanisms on the future success of a simulated platform.
Post-traumatic stress disorder (PTSD) stands out as a major mental illness; however, little is known about effective policies for mitigating the problem. The importance and complexity of PTSD raise critical questions: What are the trends in the population of PTSD patients among military personnel and veterans in the postwar era? What policies can help mitigate PTSD? To address these questions, we developed a system dynamics simulation model of the population of military personnel and veterans affected by PTSD. The model includes both military personnel and veterans in a “system of systems.” This is a novel aspect of our model, since many policies implemented at the military level will potentially influence (and may have side effects on) veterans and the Department of Veterans Affairs. The model is first validated by replicating the historical data on PTSD prevalence among military personnel and veterans from 2000 to 2014 (datasets from the Department of Defense, the Institute of Medicine, the Department of Veterans Affairs, and other sources). The model is then used for health policy analysis. Our results show that, in an optimistic scenario based on the status quo of deployment to intense/combat zones, estimated PTSD prevalence among veterans will be at least 10% during the next decade. The model postulates that during wars, resiliency-related policies are the most effective for decreasing PTSD. In a postwar period, current health policy interventions (e.g., screening and treatment) have marginal effects on mitigating the problem of PTSD, that is, the current screening and treatment policies must be revolutionized to have any noticeable effect. Furthermore, the simulation results show that it takes a long time, on the order of 40 years, to mitigate the psychiatric consequences of a war. Policy and financial implications of the findings are discussed.
There is limited methodological guidance for estimating system dynamics (SD) models using datasets common to social sciences that include few data points over time for many units under analysis. Here, we introduce indirect inference, a simulation‐based estimation method that can be applied to common datasets and is applicable to SD models that often include intractable likelihood functions. In this method, the model parameters are found by ensuring that simulated data from the model and available empirical data produce similar auxiliary statistics. The method requires few assumptions about the structure of the model and error‐generating processes and thus can be used in a variety of applications. We demonstrate the method in estimating an SD model of depression and rumination using a panel dataset. The overall results suggest that indirect inference can extend the application of SD models to new topics and leverage common panel datasets to provide unique insights.