Cover Story: SCORE! Tallying the Risk and Rewards of Risk Scoring
CardioSource WorldNews | If anyone is keeping score of cardiology risk scores, the count has soared into triple digits. But how many truly see regular use? Which ones tally up as “can’t miss” scores and which are lookin’ for some love? Are we besieged with too many scores for our own good such that some physicians avoid nearly all of them based on lack of certainty?
An important component of public health and medical care, risk assessment is also heavily relied upon to facilitate the shared decision-making process. But risk prediction can be a tricky business and a high-stakes one, too. Just ask the gambler at the roulette table or the credit ratings agencies during the 2008-09 market meltdown. The big difference: when risk prediction fails in medicine, the results can be deadly.
By one count, there are well in excess of 100 different cardiovascular risk scores developed for use in the general population.1 Seems like every other week you’ll read a journal article proposing new additions to established scores – new biomarkers, genetic information, findings from advanced imaging, etc. – as well as proposals for altogether new scores. Yet, the number of risk scores whose use is actually mandated by performance standards or guidelines remains quite small. Are the new components really adding value or just amping up a score? How should a clinician truly separate the wheat from the chaff? Are risk scores being used as often as they should be? And, perhaps, most importantly, are they being used appropriately?
“When you’re using something to decide if someone should get open-heart surgery or hospice care, these life and death decisions, I think we justifiably want to be really certain were getting accurate information,” said Thomas M. Maddox, MD, from the University of Colorado School of Medicine in Denver, in an interview.
Dr. Maddox is a cardiologist at the Department of Veterans Affairs (VA) Eastern Colorado Health Care System and the national director of the VA Clinical Assessment, Reporting, and Tracking (CART) program. His research is focused on the use of real-time clinical data to inform high-value cardiology practice and research.
ASCVD Score Vindicated
For a risk score that stirred up a lot of fuss when first introduced, the ASCVD (atherosclerotic cardiovascular disease) Pooled Cohort Equations (PCEs) appears to have made the cut and is very possibly the most commonly used tool to predict coronary heart disease (CHD) risk in the United States. (It is not validated for use in non-U.S. populations.)
The 2013 American College of Cardiology/American Heart Association updated cholesterol guidelines recommend the use of the PCEs to estimate 10-year absolute risk for ASCVD in primary prevention.2 Unlike the Framingham score that is used to determine CHD risk in the National Cholesterol Education Program Adult Treatment Panel III guidelines, the PCEs focus on estimating risk of CHD and stroke and additionally provide specific risk assessment for African-American individuals.
The PCEs were derived from 5 large racially and geographically diverse National Heart, Lung, and Blood Institute–sponsored cohort studies designed to determine the natural history of ASCVD.
“There are many risk scores, but certainly one that is used very often is the ASCVD Risk Calculator, which I think for many of us is extremely useful for deciding in our clinics who should be placed on a statin,” said Kim A. Eagle, MD, the director of the Frankel Cardiovascular Center at the University of Michigan Health System and the editor of ACC.org. Dr. Eagle was instrumental in the development of the GRACE (Global Registry of Acute Coronary Events) risk score for predicting 6-month mortality in patients with acute coronary syndromes (ACS).
Dr. Eagle’s a big fan of the MESA Risk Score for primary prevention because it includes a calcium score. “I saw someone last week where before we did the calcium score on him, if the calcium score had been 0 his risk was going to be 1.8%, and if the calcium score was 300, the risk was 10.5%. That’s a 10-fold difference based on a single score and an example of how a risk score can really help you in your clinical practice.”
There has been controversy surrounding the PCEs since their introduction in late 2013. Some studies have shown that they appear to overestimate risk.3,4 However, this lack of discrimination (see the sidebar “A (Brief) Primer on Risk Statistics” for an explanation of ‘discrimination’), has been attributed to differences in the clinical trial cohorts used to test the PCEs and the natural history studies used to develop them.5
Several investigators proffer reassuring evidence regarding the calibration and discrimination of the PCEs in the broad population in which they were designed to be used. Moreover, the authors of the original guidelines have taken pains to note that the 7.5% cutoff is not a “mandate” to prescribe statins, but rather a conversation starter.
Risk Scoring by App
Ty J. Gluckman, MD, the medical director of the Providence Heart and Vascular Institute, in Portland, Oregon, has been deeply involved in bringing risk scores to point of care, via your pocket. He helped develop both the ASCVD Risk Estimator and the newest version of the AnticoagEvaluator.
The ACC’s ASCVD Risk Estimator app helps providers and patients calculate risk and also provides easy access to recommendations specific to calculated risk estimates. Additionally, the app (developed jointly with the AHA) includes guideline reference information for both providers and patients related to therapy, monitoring, and lifestyle.
“My biggest thing is that risk scores can be very simple or very complicated, but a big barrier that we have long recognized is that people are not using risks scores appropriately and thus there is either an under-appreciation or an over-appreciation of risk,” said Dr. Gluckman in an interview with CardioSource WorldNews. Since we tend “not to do a great of a job of estimating risk and having that correlate with what the true risk is, having these aids at the point of care can better inform decision making.”
But like pretty much anything having to do with the technological translation of algorithms to apps, it is a challenge to make the apps optimally usable and relevant. Dr. Gluckman and the ACC are now working on how to make the app give feedback about patients for whom the calculator is not appropriate, such as someone already taking a statin. They’re also working on making the tools require minimal inputting of data, possibly through mining electronic health records (EHRs) for discrete data that may exist therein.
The ASCVD Risk Estimator is far and away the most downloaded app in the ACC library of mobile clinical apps (TABLE). With >328,000 downloads and >4.8 million sessions (meaning the app was opened and used), we’re taking no risk in saying that this score estimator has achieved widespread acceptance.
The ACC’s AnticoagEvaluator calculates a patient’s CHA2DS2-VASc score for stroke risk, the HAS-BLED score for major bleeding, and renal function (Cockroft-Gault Equation), if necessary, to assist decision making in patients with nonvalvular atrial fibrillation (AF).
Both apps are available on both iTunes (iPhones, iPads) and Google Play (Galaxy, Nexus, other Android devices).
Pick a Score, Any Score
While there are scores of risk scores out there, precious few are actually recommended in the clinical practice guidelines or performance measures. Obviously, the ASCVD PCEs have a recommendation, as does the CHA2DS2-VASc stroke risk score for patients with atrial fibrillation. Beyond those, the new DAPT score, derived from the Dual Antiplatelet Therapy study, is mentioned in the 2016 ACC/AHA duration of DAPT guideline update as something that “may be useful for decisions about whether to continue (prolong or extend) DAPT” in patients who have received a stent.6 In the ACS arena, the GRACE and TIMI scores are both widely used, as is the Seattle Heart Failure Score in the HF world.
“Of course, when a national guideline comes out and says you should be using this risk score, most people pay attention to that,” said Dr. Eagle. “For virtually everything we do, somebody has published a multivariable model and established some sort of a risk score, but the primary prevention, secondary prevention, and AF stroke risk, these are the big ones.”
Predictably, the most validated risk scores are the ones included in the guidelines and they are being used “prevalently,” said Dr. Maddox, but “it’s difficult to out get your arms around all the various ways clinicians are accessing risk calculators and we are definitely undercounting.
“I can do a CHA2DS2-VASc in my head, as can others, and I may or may not record the actual result,” said Dr. Maddox. “I may just calculate the score and realize my patient needs anticoagulation and put him on it.
Also, the apps measure sessions—each time the app is opened and used—but not multiple uses per session. Other docs are still using paper pads to calculate risk scores or interactive website-based calculators, so it’s impossible to truly track how often calculators and/scores are used in clinical practice. Even when a score is calculated, Dr. Maddox said, there’s no surety that it’s documented. A free-text score is difficult to abstract and may not make it into the medical record.
Not Much New
Newer risk factors and biomarkers for heart disease, including genetic data, can be evaluated in the context of existing risk estimation approaches—by adding the new component and then assessing the change in C statistic. But at the end of the day, clinicians just want to know whether an added predictor will change risk such that they should change their treatment.
“The more evidence and validation you have behind it -- and that’s what going into vetting them for the guidelines -- the more likely it is to be taken up,” said Dr. Maddox. “The brand new ones are not validated.”
As for the research on new risk factors and biomarkers, Dr. Maddox said that, in reality, most of them add “very little” to the prior way that we would calculate risk.
Or, to put it more bluntly, “If the new data take the C statistic from 0.65 to 0.66, so instead of misclassifying 35% were going to misclassify 34%, well, so what. And it’s usually a score that’s been derived in a small cohort and whether or not it applies to the person in front of me, I have no idea.”
He suggested that there is some “naiveté” that says if the number goes up at all, it matters. “Is it applicable to other populations? How much lifting does it take to get that additional information? Just showing a little bit of improvement is just one step of 20 before it’s ready for clinical use,” he said.
Big Data ≠ Good Data
Information technology has been touted as the means to support and improve clinical practice. However, moving from promise to reality has been a much slower process than imagined. Difficulties in implementation have dogged many technologies, including EHRs.
EHR use has increased dramatically in the last 5 years. In 2009, 12% of US hospitals were using basic EHR systems.7 Fast forward to 2016 and the technology has reached the point of “near universal adoption,” according to HIMSS Analytics, a global health care advisor.8
The fact that EHR use was mandated in the Affordable Care Act (ACA) and the Health Information Technology for Economic and Clinical Health (HITECH) Act hasn’t hurt the adoption curve any, nor have the incentive payments from Medicare and Medicaid. But adoption is different from satisfaction, and users continue to grumble about EHR usability, reliability, and accuracy: The systems don’t work properly or integrate efficiently. They are a time-wasting imposition. The data inputs are notoriously inaccurate or of poor quality and coding is pretty much a crapshoot.
“Medical records are not necessarily coded consistently from one provider to another, or one institution to another,” explained Dr. Eagle. “If I have a patient who comes in with pneumonia, A-fib, and a small MI, what’s my principal diagnosis? One of the big challenges with using this kind of administrative data to drive clinical decision making revolves around the inconsistencies with how we code.”
Despite their limitations, EHRs offer exciting opportunities to explore risk prediction using terabytes or even petabytes of qualitative and quantitative data. But having a ton of data and knowing what to do with it are very different things.
In some cases, EHRs go to the head of the class by being “smart” about things and will automatically derive a risk score when appropriate and present it to the provider. More often, at this point, they require some prompting, but are still useful in assisting in the derivation of a risk prediction.
“We want to make sure we’re asking a relevant question instead of just blindly calculating a score for everybody or re-calculating it even though an action has already been taken,” said Dr. Maddox. With greater automation of risk scores, he is also concerned they may impede clinical flow, unnecessarily adding time to a clinic visit.
“With the amount of automation that is available at the bedside, through your smartphone, through the EHR at the hospital or the clinic, it’s not longer an issue that physicians don’t have the time to start mentally doing the math to calculate a risk score,” said Dr. Maddox.
In May, Benjamin A. Goldstein, PhD, (see sidebar on “Machine Learning”) and colleagues published in the Journal of the American Medical Informatics Association a systematic review of studies that have utilized EHR data as the primary source to build and validate risk prediction models.9 Searching over a 6-year period, they found more than 100 papers from 15 different countries. Not surprisingly, most of the studies had large sample sizes (median n = 26,100); 39 studies had sample sizes exceeding 100,000 patients.
However, despite sample sizes that are the stuff of statistical dreams, the authors found that most of the studies failed to fully leverage the breadth of EHR data and employed relatively few predictor variables (a median of 27). Also, even though the data were conveniently electronic, less than half of the studies were multicenter and less than one-quarter of the studies performed validation across multiple sites.
“Overall, we found room for improvement in maximizing the advantages of EHR-data for risk modeling and addressing inherent challenges,” wrote Goldstein et al.
Many of the presumed advantages of using EHR data—the sample sizes, the large number of variables available, the fact that the data are not disease-specific (like registry data), and the opportunity to create external validation sets—were all largely under-exploited, according to Dr. Goldstein. Add that to some needed evolution in study design, the issues of missing data and loss of follow-up, and the difficulty of quantifying the impact of informed presence, and the field remains a work in progress.
Follow the money and we see that in the first half of 2016 alone, according to recent research from Rock Health, (a venture fund dedicated to digital health), that digital health and data management vendors amassed $2 billion in funding.10 Almost 25% of this money went to companies dedicated to data aggregation and/or analysis and to population health management, or “comprehensive delivery system tools to manage the health of populations under the shift to ACO models,” said Rock Health.
Indeed, according to Jennifer Bresnick, lead editor for HealthITAnalytics, it’s been predicted that the population health management market will grow from $14 billion to $31.9 billion by 2020, “driven largely by risk stratification technologies and predictive analytics services.”11 In other words, using risk prediction to help prevent adverse events is big money.
Shared Decision-Making Aids
For many clinicians, risk scores are primarily a means of involving patients in their own care and motivating adherence and lifestyle change.
“If you just say, ‘your risk is high,’ that doesn’t mean much to a patient, but if you say your risk for having a heart attack or stroke in the next 10 years is 15% and the national guidelines say anything above 7.5% is high enough that we should consider putting you on a statin because is lowers risk by at least one third, then they say, ‘yeah, OK, I get it. I want that benefit.”
Perhaps more helpful is showing a patient how they can reduce their risk with behavioral change--initiating exercise, losing weight, tobacco cessation.
“The rate-limiting step in cardiovascular disease prevention is the implementation and maintenance of healthy lifestyle behaviors,” wrote Nicole D. White, PharmD, and colleagues, from the Creighton University School of Pharmacy and Health Professions (Omaha, NE).12 White and colleagues demonstrated that well-designed risk assessment education—showing a patient, for example, that she can lower her risk score by three points by quitting smoking—can motivate behavioral change.
“I think the more patients understand what we’re thinking and the risk we’re trying to estimate, the more on board they are with out strategies to lower risk,” said Dr. Eagle.
Finding The Sweet Spot
New scores, old scores with new components, EHR-based automatic scoring, convenience, inconvenience. In the end, even the best risk score will not replace experience and intuition. Risk scores are only estimates of risk based on population statistics. They are not fate, nor can they stand alone or replace physician experience and intuition.
As with any clinical tool or new technology, there is big hype and then pull back, and then the trick is to find “the sweet spot of its use,” said Dr. Goldstein.
And certainly there is little risk in saying that technology—mobile devices, apps, EHRs—is helping physicians find a comfort zone where they can utilize risk prediction tools to their best avail.
- Pell JP. Heart. 2012;98:1272-7.
- Goff DC Jr, et al. J Am Coll Cardiol. 2014;63:2935-59.
- DeFilippis AP, et al. Ann Intern Med. 2015;162:266-75.
- Ridker PM, Cook NR. Lancet. 2013;382:1762-5.
- Stone NJ, Lloyd-Jones DM. Mayo Clin Proc. 2016;91:692-4.
- Levine GN, et al. J Am Coll Cardiol. 2016; doi: 10.1016/j.jacc.2016.03.513.
- Goldstein BA, et al. Eur Heart J. 2016 July 19. [Epub before print]
- 2016 Outpatient Practice Management (PM) and Electronic Health Record (EHR) Essentials Brief. Available at www.himssanalytics.org/news/2016-study-adoption-of-outpatient-solutions?utm_source=pr&utm_medium=news&utm_campaign=2016outpatient. Accessed August 8, 2016.
- Goldstein BA, et al. J Am Med Inform Assoc. 2016 May 17. [Epub ahead of print]
- Digital Health Funding 2016 Midyear Review. Available at https://rockhealth.com/reports/digital-health-funding-2016-midyear-review/. Accessed August 9, 2016.
- Bresnick J. “How the search for smart data drives healthcare IT investment.” Available at healthitanalytics.com/news/how-the-search-for-smart-data-drives-healthcare-it-investment. Accessed August 9, 2016.
- White ND, et al. Psychol Res Behav Manag. 2013;6:55-63.
|Read the full September issue of CardioSource WorldNews at ACC.org/CSWN|
< Back to Listings