Receiver operating characteristic (ROC) analysis is used for comparing predictive models in both model selection and model evaluation. ROC analysis is often applied in clinical medicine and social science to assess the tradeoff between model sensitivity and specificity. After fitting a binary logistic or probit regression model with a set of independent variables, the predictive performance of this set of variables can be assessed by the area under the curve (AUC) from an ROC curve. An important aspect of predictive modeling (regardless of model type) is the ability of a model to generalize to new cases. Evaluating the predictive performance (AUC) of a set of independent variables using all cases from the original analysis sample often results in an overly optimistic estimate of predictive performance. One can use K-fold cross-validation to generate a more realistic estimate of predictive performance in situations with a small number of observations. AUC is estimated iteratively for k samples (the “test” samples) that are independent of the sample used to predict the dependent variable (the “training” sample). cvauroc implements k-fold cross-validation for the AUC for a binary outcome after fitting a logit or probit regression model, averaging the AUCs corresponding to each fold, and bootstrapping the cross-validated AUC to obtain statistical inference and 95% confidence intervals. Furthermore, cvauroc optionally provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named \_fit; the sensitivity and specificity for each of the levels of the predicted outcome, contained in two new variables named \_sen and \_spe; and the plot of the mean cross-validated AUC and k-fold ROC curves.
Longitudinal targeted maximum likelihood estimation (LTMLE) has very rarely been used to estimate dynamic treatment effects in the context of time-dependent confounding affected by prior treatment when faced with long follow-up times, multiple time-varying confounders, and complex associational relationships simultaneously. Reasons for this include the potential computational burden, technical challenges, restricted modeling options for long follow-up times, and limited practical guidance in the literature. However, LTMLE has desirable asymptotic properties, ie, it is doubly robust, and can yield valid inference when used in conjunction with machine learning. It also has the advantage of easy-to-calculate analytic standard errors in contrast to the g-formula, which requires bootstrapping. We use a topical and sophisticated question from HIV treatment research to show that LTMLE can be used successfully in complex realistic settings, and we compare results to competing estimators. Our example illustrates the following practical challenges common to many epidemiological studies: (1) long follow-up time (30 months); (2) gradually declining sample size; (3) limited support for some intervention rules of interest; (4) a high-dimensional set of potential adjustment variables, increasing both the need and the challenge of integrating appropriate machine learning methods; and (5) consideration of collider bias. Our analyses, as well as simulations, shed new light on the application of LTMLE in complex and realistic settings: We show that (1) LTMLE can yield stable and good estimates, even when confronted with small samples and limited modeling options; (2) machine learning utilized with a small set of simple learners (if more complex ones cannot be fitted) can outperform a single, complex model, which is tailored to incorporate prior clinical knowledge; and (3) performance can vary considerably depending on interventions and their support in the data, and therefore critical quality checks should accompany every LTMLE analysis. We provide guidance for the practical application of LTMLE.
Stillbirth, one of the urgent concerns of preventable perinatal deaths, has wide-reaching consequences for society. We studied secular stillbirth trends by maternal socioeconomic status (SES) in Spain.We developed a population-based observational study, including 4 083 919 births during 2007–15. We estimate stillbirth rates and secular trends by maternal SES. We also evaluated the joint effect of maternal educational attainment and the Human Development Index (HDI) of women’s country of origin on the risk of stillbirth. The data and statistical analysis can be accessed for reproducibility in a GitHub repository: https://github.com/migariane/Stillbirth
We found a consistent pattern of socioeconomic inequalities in the risk of delivering a stillborn, mainly characterized by a persistently higher risk, over time, among women with lower SES. Overall, women from countries with low HDIs and low educational attainments had approximately a four times higher risk of stillbirth (RR: 4.44; 95%CI: 3.71–5.32). Furthermore, we found a paradoxical reduction of the stillbirth gap over time between the highest and the lowest SESs, which is mostly due to the significant and increasing trend of stillbirth risk among highly educated women of advanced maternal age.
Our findings highlight no improvement in stillbirth rates among women of lower SES and an increasing trend among highly educated women of advanced maternal age over recent years. Public health policies developing preventive programmes to reduce stillbirth rates among women with lower SES are needed as well as the necessity of further study to understand the growing trend of age-related stillbirths among highly educated women in Spain.
We investigated the temporal pattern of death in cancer patients using a large sample size and robust statistical methods to account for chronobiological periodicity.
We did not detect a circadian pattern of cancer death. The present study evaluated the temporal pattern of death among cancer patients using trigonometric functions and with time modeled in a circular scale.
To conclude, we found no evidence of a chronobiological circadian pattern in death among cancer patients by using robust statistical methods and data from a large population in a hospital setting. Increased understanding of the temporal pattern of deaths may yield important insights toward understanding external factors associated with cancer death.
In the last ten years, there have been intense debates to boost the discipline, make it relevant to the genomic revolution, and place it at the forefront of the digital era. As a result, training in public health and epidemiology has been renewed, with marked statistical and methodological reinforcement, such as the current emphasis on causal inference, or the inclusion of master's in data science for health as a new academic degree in some schools of public health https://www.hsph.harvard.edu/health-data-science/.
The Evaluating Public Health Interventions AJPH series offers excellent practical guidance to public health researchers. The eighth part of the series provides a valuable introduction to effect estimations of time-invariant public health interventions. In their commentary Spiegelman and Zhou suggest that, in terms of bias and efficiency, there is no advantage to using modern causal inference methods over classical multivariable modeling. However, this statement is not always true. Most important, both effect modification and collapsibility are critical concepts when assessing the validity of using regression for causal effect estimation.
Suppose that one is interested in the effect of combined radiotherapy and chemotherapy versus chemotherapy only on one-year mortality among patients diagnosed with colorectal cancer. A clinician may ask: how different would the risk of death have been had everyone received dual therapy as compared with if everyone had experienced monotherapy? The causal marginal odds ratio (MOR) offers an answer to this question. Each individual has a pair of potential outcomes: the outcome he or she would have experienced had he or she been exposed to dual treatment (A = 1), denoted Y(1), and the outcome had he or she been unexposed, Y(0). The MOR is defined as
A common approach would be to use logistic regression to model the odds of mortality given the intervention and adjust for confounders (W) such as clinical stage and comorbidities. Note that this regression will provide an estimate of the conditional odds ratio (COR), which is
[P(Y = 1 A = 1,W) / (1 − P(Y = 1 A = 1,W))] / [P(Y = 1 A = 0,W) / (1 − P(Y = 1 A = 0,W))].
The MOR and COR are typically not identical. First, if there is effect modification (e.g., if the effect of dual therapy is different between patients with no comorbidities and those who have hypertension), logistic regression including an interaction term will not provide a marginal effect estimate but only the conditional effect of the interaction term between dual therapy and hypertension. Second, the odds ratio is noncollapsible, which means that the MOR is not necessarily equal to the stratum-specific odds ratio (i.e., the COR). This holds even when a covariate is related to the outcome but not the intervention and is thus not a confounder.
Colorectal cancer is the second most frequently diagnosed cancer in Spain. Cancer treatment and outcomes can be influenced by tumor characteristics, patient general health status and comorbidities. Numerous studies have analyzed the influence of comorbidity on cancer outcomes, but limited information is available regarding the frequency and distribution of comorbidities in colorectal cancer patients, particularly elderly ones, in the Spanish population. We developed a population-based study of all incident colorectal cases diagnosed in Spain in 2011 to describe the frequency and distribution of comorbidities, as well as tumor and healthcare factors. Data were obtained from two population-based cancer registries and the complete version of patients' digitalized clinical records history. We then characterized the most prevalent comorbidities, as well as dementia and multimorbidity, and developed an interactive web application to visualize our findings (http://watzilei.com/shiny/CoMCoR/). The most common comorbidities were diabetes (23.6%), chronic obstructive pulmonary disease (17.2%), and congestive heart failure (14.5%). Dementia was the most common comorbidity among patients aged >=75 years. Patients with dementia had a 30% higher prevalence of being diagnosed at stage IV and the highest prevalence of emergency hospital admission after colorectal cancer diagnosis (33%). Colorectal cancer patients with dementia were nearly three times more likely to not be offered surgical treatment. Age >=75 years, obesity, male sex, being a current smoker, having surgery more than 60 days after cancer diagnosis, and not receiving surgical treatment were associated with a higher risk of multimorbidity. Patients with multimorbidity aged >=75 years showed a higher prevalence of hospital emergency admission followed by surgery the same day of the admission (37%). We found a consistent pattern in the distribution and frequency of comorbidities and multimorbidity among colorectal cancer patients. The high frequency of stage IV diagnosis among patients with dementia and the high proportion of older patients not receiving surgical treatment are significant findings that require policy actions.
Survival data analysis results are usually communicated through the overall survival probability. Alternative measures provide additional insights and may help in communicating the results to a wider audience. We describe these alternative measures in two data settings, the overall survival setting and the relative survival setting, the latter corresponding to the particular competing risk setting in which the cause of death is unavailable or unreliable. In the overall survival setting, we describe the overall survival probability, the conditional survival probability and the restricted mean survival time (restricted to a prespecified time window). In the relative survival setting, we describe the net survival probability, the conditional net survival probability, the restricted mean net survival time, the crude probability of death due to each cause and the number of life years lost due to each cause over a prespecified time window. These measures describe survival data either on a probability scale or on a timescale. The clinical or population health purpose of each measure is detailed, and their advantages and drawbacks are discussed. We then illustrate their use analyzing England population-based registry data of men 15–80 years old diagnosed with colon cancer in 2001–2003, aiming to describe the deprivation disparities in survival. We believe that both the provision of a detailed example of the interpretation of each measure and the software implementation will help in generalizing their use.
Classical epidemiology has focused on the control of confounding, but it is only recently that epidemiologists have started to focus on the bias produced by colliders. A collider for a certain pair of variables (e.g. an outcome Y and an exposure A) is a third variable (C) that is caused by both. In a directed acyclic graph (DAG), a collider is the variable in the middle of an inverted fork (i.e. the variable C in A → C ← Y). Controlling for, or conditioning an analysis on a collider (i.e. through stratification or regression) can introduce a spurious association between its causes. This potentially explains many paradoxical findings in the medical literature, where established risk factors for a particular outcome appear protective. We use an example from non-communicable disease epidemiology to contextualize and explain the effect of conditioning on a collider. We generate a dataset with 1000 observations, and run Monte-Carlo simulations to estimate the effect of 24-h dietary sodium intake on systolic blood pressure, controlling for age, which acts as a confounder, and 24-h urinary protein excretion, which acts as a collider. We illustrate how adding a collider to a regression model introduces bias. Thus, to prevent paradoxical associations, epidemiologists estimating causal effects should be wary of conditioning on colliders. We provide R code in easy-to-read boxes throughout the manuscript, and a GitHub repository [https://github.com/migariane/ColliderApp] for the reader to reproduce our example. We also provide an educational web application allowing real-time interaction to visualize the paradoxical effect of conditioning on a collider [http://watzilei.com/shiny/collider/].
The aim of this study was to analyse the trends in socioeconomic inequalities in induced abortion during the pre-crisis and crisis periods in the postcodes of two major cities of Spain. Ecological regression model showed that rates of induced abortion tended to increase between the two pre-crisis periods, but remained stable between the second pre-crisis period and the crisis period. In addition, we observed socioeconomic inequalities in induced abortion in both cities and in all age groups, and that these inequalities persisted across the three study periods.
Classical epidemiology has focused on the control of confounding but it is only recently that epidemiologists have started to focus on the bias produced by colliders. A collider for a certain pair of variables (e.g., an outcome Y and an exposure A) is a third variable (C) that is caused by both. In a directed acyclic graph (DAG), a collider is the variable in the middle of an inverted fork (i.e., the variable C in A -> C <- Y). Controlling for, or conditioning an analysis on a collider (i.e., through stratification or regression) can introduce a spurious association between its causes. This potentially explains many paradoxical findings in the medical literature, where established risk factors for a particular outcome appear protective. We use an example from non-communicable disease epidemiology to contextualize and explain the effect of conditioning on a collider. We generate a dataset with 1,000 observations and run Monte-Carlo simulations to estimate the effect of 24-hour dietary sodium intake on systolic blood pressure, controlling for age, which acts as a confounder, and 24-hour urinary protein excretion, which acts as a collider. We illustrate how adding a collider to a regression model introduces bias. Thus, to prevent paradoxical associations, epidemiologists estimating causal effects should be wary of conditioning on colliders. We provide R-code in easy-to-read boxes throughout the manuscript and a GitHub repository (this https URL) for the reader to reproduce our example. We also provide an educational web application allowing real-time interaction to visualize the paradoxical effect of conditioning on a collider this http URL
The third most frequently diagnosed cancer in Europe in 2018 was lung cancer; it is also the leading cause of cancer death in Europe. We studied patient and tumor characteristics, and patterns of healthcare provision explaining regional variability in lung cancer survival in southern Spain.
A population-based cohort study included all 1196 incident first invasive primary lung cancer (C33-C34 according to ICD-10) cases diagnosed between 2010 and 2011 with follow-up until April 2015. Data were drawn from local population-based cancer registries and patients' hospital medical records from all public and private hospitals from two regions in southern Spain.
There was evidence of regional differences in lung cancer late diagnosis (58% stage IV in Granada vs. 65% in Huelva, p value < 0.001). Among patients with stage I, only 67% received surgery compared with 0.6% of patients with stage IV. Patients treated with a combination of radiotherapy, chemotherapy, and surgery had a 2-year mortality risk reduction of 94% compared with patients who did not receive any treatment (excess mortality risk 0.06; 95% CI 0.02-0.16). Geographical differences in survival were observed between the two regions: 35% vs. 26% at 1-year since diagnosis.
The observed geographic differences in survival between regions are due in part to the late cancer diagnosis which determines the use of less effective therapeutic options. Results from our study justify the need for promoting lung cancer early detection strategies and the harmonization of the best practice in lung cancer management and treatment.
Introduction We investigated socioeconomic disparities and the role of the main prognostic factors in receiving major surgical treatment in patients with lung cancer in England.Methods Our study comprised 31 351 patients diagnosed with non-small cell lung cancer in England in 2012. Data from the national population-based cancer registry were linked to Hospital Episode Statistics and National Lung Cancer Audit data to obtain information on stage, performance status and comorbidities, and to identify patients receiving major surgical treatment. To describe the association between prognostic factors and surgery, we performed two different analyses: one using multivariable logistic regression and one estimating cause-specific hazards for death and surgery. In both analyses, we used multiple imputation to deal with missing data.Results We showed strong evidence that the comorbidities ‘congestive heart failure’, ‘cerebrovascular disease’ and ‘chronic obstructive pulmonary disease’ reduced the receipt of surgery in early stage patients. We also observed gender differences and substantial age differences in the receipt of surgery. Despite accounting for sex, age at diagnosis, comorbidities, stage at diagnosis, performance status and indication of having had a PET-CT scan, the socioeconomic differences persisted in both analyses: more deprived people had lower odds and lower rates of receiving surgery in early stage lung cancer.Discussion Comorbidities play an important role in whether patients undergo surgery, but do not completely explain the socioeconomic difference observed in early stage patients. Future work investigating access to and distance from specialist hospitals, as well as patient perceptions and patient choice in receiving surgery, could help disentangle these persistent socioeconomic inequalities.
Gestational diabetes mellitus (GDM) is a global public health concern with potential implications for the health of a mother and her offspring. However, data on the prevalence and risk factors of GDM in Latin America are scarce.
When estimating the average effect of a binary treatment (or exposure) on an outcome, methods that incorporate propensity scores, the G‐formula, or targeted maximum likelihood estimation (TMLE) are preferred over naïve regression approaches, which are biased under misspecification of a parametric outcome model. In contrast propensity score methods require the correct specification of an exposure model. Double‐robust methods only require correct specification of either the outcome or the exposure model. Targeted maximum likelihood estimation is a semiparametric double‐robust method that improves the chances of correct model specification by allowing for flexible estimation using (nonparametric) machine‐learning methods. It therefore requires weaker assumptions than its competitors. We provide a step‐by‐step guided implementation of TMLE and illustrate it in a realistic scenario based on cancer epidemiology where assumptions about correct model specification and positivity (ie, when a study participant had 0 probability of receiving the treatment) are nearly violated. This article provides a concise and reproducible educational introduction to TMLE for a binary outcome and exposure. The reader should gain sufficient understanding of TMLE from this introductory tutorial to be able to apply the method in practice. Extensive R‐code is provided in easy‐to‐read boxes throughout the article for replicability. Stata users will find a testing implementation of TMLE and additional material in the Appendix S1 and at the following GitHub repository:
The aim of this study was to evaluate trends in small-for-gestational age covering the period before and after the Spanish economic crisis, taking into account mother’s age, nationality and the autonomous community where she resides. We performed a trend study including children born to fertile women in Spain between 2002 and 2013. Poisson mixed models showed that the prevalence of small-for-gestational age increased following the onset of the crisis, and that a previous downward trend was interrupted.
Longitudinal targeted maximum likelihood estimation (LTMLE) has hardly ever been used to estimate dynamic treatment effects in the context of time-dependent confounding affected by prior treatment when faced with long follow-up times, multiple time-varying confounders, and complex associational relationships simultaneously. Reasons for this include the potential computational burden, technical challenges, restricted modeling options for long follow-up times, and limited practical guidance in the literature. However, LTMLE has desirable asymptotic properties, i.e. it is doubly robust, and can yield valid inference when used in conjunction with machine learning. We use a topical and sophisticated question from HIV treatment research to show that LTMLE can be used successfully in complex realistic settings and compare results to competing estimators. Our example illustrates the following practical challenges common to many epidemiological studies 1) long follow-up time (30 months), 2) gradually declining sample size 3) limited support for some intervention rules of interest 4) a high-dimensional set of potential adjustment variables, increasing both the need and the challenge of integrating appropriate machine learning methods. Our analyses, as well as simulations, shed new light on the application of LTMLE in complex and realistic settings: we show that (i) LTMLE can yield stable and good estimates, even when confronted with small samples and limited modeling options; (ii) machine learning utilized with a small set of simple learners (if more complex ones can't be fitted) can outperform a single, complex model, which is tailored to incorporate prior clinical knowledge; (iii) performance can vary considerably depending on interventions and their support in the data, and therefore critical quality checks should accompany every LTMLE analysis.
Plasma glycated CD59 (pGCD59) is an emerging biomarker in diabetes. We assessed whether pGCD59 could predict the following: the results of the glucose challenge test (GCT) for screening of gestational diabetes mellitus (GDM) (primary analysis); and the diagnosis of GDM and prevalence of large for gestational age (LGA) newborns (secondary analyses).
RESEARCH DESIGN AND METHODS:
Case-control study of 1,000 plasma samples from women receiving standard prenatal care, 500 women having a normal GCT (control subjects) and 500 women with a failed GCT and a subsequent oral glucose tolerance test (case patients).
Compared with control subjects, the median (interquartile range) pGCD59 value was 8.5-fold higher in case patients and 10-fold higher in GDM patients, as follows: control subjects 0.33 (0.19); case patients 2.79 (1.4); GDM patients 3.23 (1.43) (P < 0.001); area under the receiver operating characteristic curve 0.92. LGA prevalence was 4.3% in the lowest quartile and 13.5% in the highest quartile of pGCD59.
One pGCD59 measurement during weeks 24-28 identifies pregnancy-induced glucose intolerance with high sensitivity and specificity and can potentially identify the risk for LGA.
Objectives The clinical course and prognosis of follicular lymphoma (FL) are diverse and associated with the patient’s immune response. We investigated the lymphocyte-to-monocyte ratio (LMR) and neutrophil-to-lymphocyte ratio (NLR) as prognostic factors in patients with FL, including those receiving radiotherapy.Design A retrospective cohort study.Setting Regional cancer centre in Hong Kong.Participants 88 patients with histologically proven FL diagnosed between 2000 and 2014.Materials and methods The best LMR and NLR cut-off values were determined using cross-validated areas under the receiver operating characteristic curves. The extent to which progression-free survival (PFS) and overall survival differed by NLR and LMR cut-off values was assessed using Kaplan-Meier analysis and log-rank tests. A Cox proportional hazards model was fitted to adjust for confounders.Results The best cut-off values for LMR and NLR were 3.20 and 2.18, respectively. The 5-year PFS was 73.6%. After multivariate adjustment, high LMR (>3.20) at diagnosis was associated with superior PFS, with a HR of 0.31 (95% CI 0.13 to 0.71), whereas high NLR at relapse was associated with poorer postprogression survival (HR 1.24, 95% CI 1.04 to 1.49).Conclusions Baseline LMR and NLR at relapse were shown to be independent prognostic factors in FL. LMR and NLR are cheap and widely available biomarkers that could be used in combination with the Follicular Lymphoma International Prognostic Index by clinicians to better predict prognosis.
Background: Primary aldosteronism is recognized as a severe form of renin-independent aldosteronism that results in excessive mineralocorticoid receptor (MR) activation. Objective: To investigate whether a spectrum of subclinical renin-independent aldosteronism that increases risk for hypertension exists among normotensive persons. Design: Cohort study. Setting: National community-based study. Participants: 850 untreated normotensive participants in MESA (Multi-Ethnic Study of Atherosclerosis) with measurements of serum aldosterone and plasma renin activity (PRA). Measurements: Longitudinal analyses investigated whether aldosterone concentrations, in the context of physiologic PRA phenotypes (suppressed, ≤0.50 μg/L per hour; indeterminate, 0.51 to 0.99 μg/L per hour; unsuppressed, ≥1.0 μg/L per hour), were associated with incident hypertension (defined as systolic blood pressure ≥140 mm Hg, diastolic blood pressure ≥90 mm Hg, or initiation of antihypertensive medications). Cross-sectional analyses investigated associations between aldosterone and MR activity, assessed via serum potassium and urinary fractional excretion of potassium. Results: A suppressed renin phenotype was associated with a higher rate of incident hypertension than other PRA phenotypes (incidence rates per 1000 person-years of follow-up: suppressed renin phenotype, 85.4 events [95% CI, 73.4 to 99.3 events]; indeterminate renin phenotype, 53.3 events [CI, 42.8 to 66.4 events]; unsuppressed renin phenotype, 54.5 events [CI, 41.8 to 71.0 events]). With renin suppression, higher aldosterone concentrations were independently associated with an increased risk for incident hypertension, whereas no association between aldosterone and hypertension was seen when renin was not suppressed. Higher aldosterone concentrations were associated with lower serum potassium and higher urinary excretion of potassium, but only when renin was suppressed. Limitation: Sodium and potassium were measured several years before renin and aldosterone. Conclusion: Suppression of renin and higher aldosterone concentrations in the context of this renin suppression are associated with an increased risk for hypertension and possibly also with increased MR activity. These findings suggest a clinically relevant spectrum of subclinical primary aldosteronism (reninindependent aldosteronism) in normotension.