Evaluating New Cardiovascular Risk Factors for Risk Stratification

Abstract

There is an active debate on the best approach to evaluate new cardiovascular risk factors. The divergent views have been most clearly laid out in the case of C-reactive protein (CRP) and its incremental value over conventional cardiovascular risk factors. Advocates of CRP argue that its addition improves global measures of fit,1 while others point out that it does not improve measures of discrimination.2,3 Discussions surrounding this debate have largely been technical, focusing on unique data sources and specific analytic approaches. The underlying principles, in contrast, have not been sufficiently highlighted. This paper presents a clinical perspective on the potential value of adding new cardiovascular risk factors to current methods for risk stratification. It proposes that novel risk factors should be evaluated by their effects on the population risk distribution curve, which presents the probability of occurrence of different levels of risk in a population as determined by a predictive model. Broader population risk distribution curves represent superior risk stratification. Last, it should be recognized that there are many correct ways to risk-stratify a population and that the best approach ultimately depends on the clinical goals. The goal of risk stratification is to identify subpopulations of differing cardiovascular risk within a larger general population to allow interventions to be employed selectively. Mathematic risk models must be applied to populations to determine their clinical utility. New cardiovascular risk factors that do not improve risk stratification within these models are not of clinical value, although they may increase scientific understanding or reveal new opportunities to develop preventive measures. Applying a predictive model to an entire population generates a population risk distribution curve. These are rarely presented in the cardiovascular literature but have important implications for assessing the value of risk factors. We have been unable to locate a graph depicting the population distribution of cardiovascular risk determined by a Framingham risk score, despite the widespread use of this risk score in the literature. An example of this is depicted in Figure 1. This population risk distribution curve shows how frequently patients with different levels of cardiovascular risk occur in the population. As expected, the higher their cardiovascular risk, the less often these patients are encountered. It is uncommon to find healthy patients with a 10-year risk >20%, the range expected for patients with coronary artery disease. Graphs of this type represent a valuable way to evaluate the clinical importance of risk factors. Neither global measures of fit nor the receiver operating characteristic (ROC) curve so directly communicate the results of the risk stratification process. Approximation to the frequency distribution of 10-year cardiovascular risk in US adults without cardiovascular disease or cardiovascular disease risk equivalents. This distribution (an exponential distribution with a lambda of 17) assigns patients to low, intermediate, and high risk in the same proportions reported in the literature.12 Absent any information about risk factors, all members of the population would be assigned the same risk. The risk distribution "curve" would be a spike at the mean population risk. As risk factors are added to a mathematic risk model, the risk distribution curve should progressively widen, so long as the risk factors contribute additional information. Figure 2 provides a hypothetical example of risk distribution curves of increasing width, demonstrating improved risk stratification. The increasing width shows an increasing ability to separate the population into subgroups of higher and lower risk. Sinusoidal population risk distribution curves with the same mean risk but increasing width, indicating improved risk stratification. The broader the risk distribution curve, the better the separation of patients of differing risks. If two mathematic risk models produce identical risk distribution curves when applied to the same population, then the risk stratification of the two models is equivalent. Real-world data are unlikely to generate mathematically identical risk distribution curves; however, similar risk distribution curves should be considered clinically equivalent, even if there are mathematic differences. The ROC curve, which depicts the relationship between sensitivity and specificity, has been an important tool for assessing diagnostic methods and predictive models.4 Diamond5 showed that the ROC curve for a risk model can be calculated from the population risk distribution. Since the ROC curve can be derived from the risk distribution curve (and vice versa), these curves contain identical information. This means that the mathematic analyses of the ROC curve in the risk factor research literature can be better understood simply by evaluating the corresponding risk distribution curve. Because the ROC curve and the risk distribution curve contain identical information, properties of one curve must correspond to properties of the other. The area under the ROC curve (the ROC curve AUC), used to compare both diagnostic methods and predictive models, is an example. Although the ROC curve AUC is commonly understood to measure the overlap of the risk distribution curves of patients with and without events, also known as discrimination, it is equally valid to consider the ROC curve AUC as a measure of the width of the population risk distribution curve. For example, the 3 population risk distribution curves with increasing width in Figure 2 have ROC curve AUCs of 0.55, 0.65, and 0.75. Cook6 provides a similar example using the β distribution. For diagnostic methods, there is only one way to be correct: perfectly discriminate between patients with and without a disease. This is the definition of a gold standard or perfect test. For a predictive model to match this performance, it would need to identify patients who will or will not have an event in the future. Unfortunately, this is not possible because the occurrence of events is random or stochastic. An alternative and more realistic criterion is that a predictive model is correct if it accurately assigns risk to different subpopulations. This property is referred to as calibration. It is commonly evaluated by comparing the observed to predicted risk for each decile of risk.7 For a given population, there will be a multitude of risk stratification methods, all of which are correct or calibrated. However, because they utilize different risk factors, these methods will assign different risks to the same individual. Consider risk-stratifying the same population twice, once by systolic pressure and once by low-density lipoprotein cholesterol. Assume the corresponding cardiovascular risk for each decile was the same for the 2 risk factors. Then the risk stratification of the population by the 2 risk factors is identical. But patients with low blood pressure may have high low-density lipoprotein cholesterol, and vice versa, so individuals may be assigned very different risks depending on which risk factor is used to risk stratify the population. Because the goal of risk stratification is to identify subpopulations of differing cardiovascular risk, the population risk distribution curve is the most useful way to assess a predictive method's risk stratification in a given population. Neither global measures of fit nor ROC curves show this directly. Framingham risk models are currently able to identify adult patients without cardiovascular disease with 10-year risks ranging from <1% to >30%. Figure 1 (an approximation) illustrates the corresponding population risk distribution. It is unknown whether this risk stratification can be improved. Newer risk factors8 or even measures of carotid atherosclerosis combined with CRP9 have not improved risk stratification using conventional risk factors. Diamond5 showed that risk models have limited discriminatory ability. As discussed above, perfect discrimination would mean that a predictive method could identify patients who will or will not have an event in the future. But making accurate predictions over a 10-year period is only possible in deterministic systems (eg, prediction of eclipses in the solar system). The occurrence of clinical events in low-risk individuals is often thought to indicate that improved predictive methods are possible, but as long as the predicted number of events match the observed number of events in a low-risk subpopulation, this is not the case. In comparing risk models, one should recognize that the predicted risk for individual patients can vary between calibrated models. An example of this for univariate risk models using systolic blood pressure and low-density lipoprotein cholesterol was given previously. If additional risk factors were added to such univariate models, patients with high levels of the additional risk factor would be assigned higher risks, while those with low levels of the additional risk factor would be assigned lower risks. This would occur throughout the risk distribution curve, ensuring extensive scrambling or reassortment of individual patients with the addition of each new risk factor. This would also occur even if the population risk distribution curve is not changed by the additional risk factors. Ridker and colleagues10 described 2 calibrated multivariate models that differ in the number of risk factors. In spite of having the same discrimination, these models classified many patients into different risk strata. In this situation, the risk distribution curves should be the same, and thus the number of patients moved out of a risk stratum will be balanced by the number of patients moved into that risk stratum when additional risk factors are added. Reynolds and associates11 compared 3 cardiovascular risk models that predict simil

Keywords

Affiliated Institutions

• University of Michigan US

Related Publications