The Need for Improved ICU Severity Scoring

How do we know we’re doing a good job taking care of critically ill patients? This question is at the heart of the paper recently published in this journal by Raschke and colleagues (1). Currently, one key method we use to assess the quality of patient care is to calculate the ratio of observed to predicted hospital mortality, or the standardized mortality ratio (SMR). Predicted hospital mortality is estimated with prognostic indices that use patient data to approximate their severity of illness (2). Examples of these indices include the Acute Physiology and Chronic Health Evaluation (APACHE) score, the Simplified Acute Physiology Score (SAPS), the Mortality Prediction Model (MPM), the Multiple Organ Dysfunction Score (MODS), and the Sequential Organ Failure Assessment (SOFA) (3).

Raschke et al. (1) evaluated the performance of the APACHE IVa score in subgroups of ICU patients. APACHE is a severity-of-illness score initially created in the 1980s and subsequently updated in 2006 (4,5). This index was developed using data from 110,558 patients from 45 hospitals located throughout the United States, and encompassed 104 intensive care units (ICUs) including mixed medical-surgical, coronary, surgical, cardiothoracic, medical, neurologic, and trauma units. The final model used 142 variables including information from the patient’s medical history, the admission diagnosis, and physiologic data obtained during the first day of ICU admission (4). Although it subsequently has been validated using other large general ICU patient cohorts, its accuracy in subgroups of ICU patients is less clear (6).

To benchmark whether the APACHE IVa performed sufficiently, Raschke et al. (1) employed an interesting and logical strategy. They created a two-variable severity score (2VSS) to define a lower limit of acceptable performance.  As opposed to the 142 variables used in APACHE IVa, the 2VSS used only two variables: patient age and need for mechanical ventilation. They included 66,821 patients in their analysis, encompassing patients from a variety of ICUs located in the southwest United States. The APACHE IVa and 2VSS was calculated for all patients. Although the APACHE IVa outperformed the 2VSS in the general cohort of ICU patients, when patients were divided into subgroups based on admission diagnosis the APACHE IVa showed surprising deficiencies. In patients admitted for coronary artery bypass grafting (CABG), the APACHE IVa did no better in predicting mortality than the 2VSS. The ability of APACHE IVa to predict mortality was significantly reduced in patients admitted for gastrointestinal bleed, sepsis, and respiratory failure as compared to its ability to predict mortality in the general cohort (1).

The work by Raschke et al. (1) convincingly shows that APACHE IVa underperforms when evaluating outcomes in subgroups of patients. In some instances, it did no better than a metric that used only two input variables. But why does this matter? One might argue that the APACHE system was not created to function in this capacity. It was designed and validated using aggregate data. It was not designed to determine prognosis on individual-level patients, or even on subsets of patients. However, in real-world practice it is used to estimate performance in individual ICUs, which have unique cases mixes of patients that may not approximate the populations used to create and validate APACHE IVa. Indeed, other studies have shown that the APACHE IVa yields different performance assessments in different ICUs depending on varying case mixes (2).

So where do we go from here? The work by Raschke et al. (1) is helpful because it offers the 2VSS as an objective method of defining a lower limit of acceptable performance. In the future, more sophisticated and personalized tools will need to be developed to more accurately benchmark ICU quality and performance.  Interesting work is being done using local data to customize outcome prediction (7,8). Other researchers have employed machine learning techniques to iteratively improve predictive capabilities of outcome measures (9,10). As with many aspects of modern medicine, the complexity of severity scoring will likely increase as computational methods allow for increased personalization. Given the importance of accurately assessing quality of care, improving severity scoring will be critical to providing optimal patient care.

Sarah K. Medrek, MD

University of New Mexico

Albuquerque, NM USA

References

1. Raschke RA GR, Ramos KS, Fallon M, Curry SC. The explained variance and discriminant accuracy of APACHE IVa severity scoring in specific subgroups of ICU patients. Southwest J Pulm Crit Care. 2018;17:153-64. [CrossRef]
2. Kramer AA, Higgins TL, Zimmerman JE. Comparing observed and predicted mortality among ICUs using different prognostic systems: why do performance assessments differ? Crit Care Med. 2015;43:261-9. [CrossRef] [PubMed]
3. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207. [CrossRef] [PubMed]
4. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34:1297-1310. [CrossRef] [PubMed]
5. Zimmerman JE, Kramer AA, McNair DS, Malila FM, Shaffer VL. Intensive care unit length of stay: Benchmarking based on Acute Physiology and Chronic Health Evaluation (APACHE) IV. Crit Care Med. 2006;34:2517-29. [CrossRef] [PubMed]
6. Salluh JI, Soares M. ICU severity of illness scores: APACHE, SAPS and MPM. Curr Opin Crit Care. 2014;20:557-65. [CrossRef] [PubMed]
7. Lee J, Maslove DM. Customization of a Severity of Illness Score Using Local Electronic Medical Record Data. J Intensive Care Med. 2017;32:38-47. [CrossRef] [PubMed]
8. Lee J, Maslove DM, Dubin JA. Personalized mortality prediction driven by electronic medical data and a patient similarity metric. PLoS One. 2015;10:e0127428. [CrossRef] [PubMed]
9. Awad A, Bader-El-Den M, McNicholas J, Briggs J. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform. 2017;108:185-95. [CrossRef] [PubMed]
10. Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ. Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med. 2015;3:42-52. [CrossRef] [PubMed]

Cite as: Medrek SK. The need for improved ICU severity scoring. Southwest J Pulm Crit Care. 2019;18:26-8. doi: https://doi.org/10.13175/swjpcc004-19 PDF

Breaking the Guidelines for Better Care

Two events happened this past week that inspired this editorial. First, on Wednesday morning I read the editorial titled “Breaking the Rules for Better Care” by Don Berwick et al. in JAMA (1). Berwick reports a survey of about 40 hospitals done by The Institute of Healthcare Improvement (IHI). The survey asked the question “If you could break or change any rule in service of a better care experience for patients or staff, what would it be?”. The answers were not surprising. Most centered on annoying hospital rules such as visiting hours, not waking patients, correct HIPPA interpretation, and eliminating the 3-day rule. Although these are correct, in the whole they have minimal effect on healthcare. Other suggestions more likely to improve patient care included improving access, reducing wait times and earlier patient mobility. From the suggestions, it seems likely that most were from administrators. In the editorial Berwick decried, “Habits embedded in organizational behaviors, based on misinterpretations and with little to no actual foundation in legal, regulatory, or administrative requirements”. He goes on to say, “Health care leaders may be well advised to ask their clinicians, staffs, and patients which habits and rules appear to be harming care without commensurate benefits and, with prudence and circumspection, to change them.” As a clinician, I thoroughly agree with both of Berwick’s points.

Later that afternoon, I listened to a lecture by Clement Singarajah on sepsis guidelines. He reviewed the severe sepsis bundles promoted by the Surviving Sepsis Campaign and IHI, the latter being Berwick’s organization who wrote the editorial noted above (Table 1) (2,3).

Table 1.  Severe Sepsis Bundles.

The Severe Sepsis 3-Hour Resuscitation Bundle contains the following elements, to be completed within 3 hours of the time of presentation with severe sepsis:

• Measure Lactate Level
• Obtain Blood Cultures Prior to Administration of Antibiotics
• Administer Broad Spectrum Antibiotics
• Administer 30 mL/kg Crystalloid for Hypotension or Lactate ≥4 mmol/L

The 6-Hour Septic Shock Bundle contains the following elements, to be completed within 6 hours of the time of presentation with severe sepsis:

• Apply Vasopressors (for Hypotension That Does Not Respond to Initial Fluid Resuscitation to Maintain a Mean Arterial Pressure (MAP) ≥65 mm Hg)
• In the Event of Persistent Arterial Hypotension Despite Volume Resuscitation (Septic Shock) or Initial Lactate ≥4 mmol/L (36 mg/dL):
  • Measure Central Venous Pressure (CVP)
  • Measure Central Venous Oxygen Saturation (ScvO2)
• Remeasure Lactate If Initial Lactate Was Elevated

We carefully reviewed each of the metrics, and concluded most were non-evidence based, outdated, or contradicted by more recent and better trials. The only exception was early antibiotic administration. Most of us reaffirmed our belief in the germ theory and felt that early administration of the correct antibiotics was probably mostly evidence-based and reasonable (4).

Is it possible that most of the metrics in the bundle are merely a waste of time as we concluded or could some be harmful? First, a recent meta-analysis examined a conservative fluid strategy in sepsis compared with a liberal strategy (the goal-directed therapy as advocated by the sepsis bundles) (5). Although there was no change in mortality, a conservative strategy resulted in increased ventilator-free days and reduced length of ICU stay. The meta-analysis concluded that the studies were underpowered to show a mortality benefit. Second, most of us had experienced delays in initiating antibiotics, the only guideline that makes a difference, while waiting for blood cultures to be drawn. None of us knew data that drawing blood cultures makes a difference in patient outcomes.

Berwick recommended asking clinicians which rules may be harming care. Rather than chiding others to do something, a good place to start might be IHI’s sepsis guidelines. The issue of continued support for non-evidence based or outdated guidelines points out the rigid dichotomy between self-delusional beliefs and science. Many (some would say most) guidelines are based on opinions and not science (6). Healthcare would be better if groups such as the Surviving Sepsis Campaign, IHI and the Centers for Medicare and Medicaid Services would follow their own advice and not burden healthcare providers with non-evidence based guidelines. Instead, they should only issue guidelines after carefully conducted, randomized, controlled trials establish a guideline rather than mandating the self-delusional beliefs of a few.

Richard A. Robbins, MD

Editor, SWJPCC

References

1. Berwick DM, Loehrer S, Gunther-Murphy C. Breaking the rules for better care. JAMA. 2017 Jun 6;317(21):2161-2. [CrossRef] [PubMed]
2. Surviving Sepsis Campaign. Updated bundles in response to new evidence. Available at: http://www.survivingsepsis.org/SiteCollectionDocuments/SSC_Bundle.pdf (accessed 6/9/17).
3. Institute for Healthcare Improvement. Severe sepsis bundles. Available at: http://www.ihi.org/resources/Pages/Tools/SevereSepsisBundles.aspx (accessed 6/9/17).
4. Seymour CW, Gesten F, Prescott HC, et al. Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med. 2017 Jun 8;376(23):2235-44. [CrossRef] [PubMed]
5. Silversides JA, Major E, Ferguson AJ, et al. Conservative fluid management or deresuscitation for patients with sepsis or acute respiratory distress syndrome following the resuscitation phase of critical illness: a systematic review and meta-analysis. Intensive Care Med. 2017 Feb;43(2):155-170. [CrossRef] [PubMed]
6. Lee DH, Vielemeyer O. Analysis of overall level of evidence behind Infectious Diseases Society of America practice guidelines. Arch Intern Med. 2011;171:18-22. [CrossRef] [PubMed]

Cite as: Robbins RA. Breaking the guidelines for better care. Southwest J Pulm Crit Care. 2017;14(6):285-7. doi: https://doi.org/10.13175/swjpcc072-17 PDF 

Using the EMR for Better Patient Care

The medical record was developed in the US in major teaching hospitals in the 19th century and widely adopted when it was realized the records benefited patients, nurses and doctors (1). These paper records continued (although with many alterations) until the early 21st century when electronic medical or healthcare records (EMR) were mandated by the Federal government. EMRs offer great promise by handling the enormous amounts of data generated in healthcare. Furthermore, in those instances where early identification of disease process seems to make a difference, EMRs would seem an ideal tool to alert nurses and doctors. Sepsis is a disease process which would seem appropriate for early identification by EMR since early recognition can be difficult but early intervention improves outcomes (2). However, previous attempts to use the EMR to identify septic patients have been disappointing (3,4). In this issue of the SWJPCC Fountain and her colleagues (5) used clinical decision support systems (CDSSs) incorporated into EMRs to successfully identified septic patients with reasonable sensitivity and positive predictive value.

Why did Fountain et al. succeed while others failed? The 20 year old definition of sepsis that required two or more systemic inflammatory response syndrome criteria to define sepsis did not identify the sickest patients at the greatest risk for death (6). Realizing this weakness, Fountain and colleagues shifted their diagnostic focus from systemic inflammation to infection-triggered organ failure consistent with the new definition of sepsis proposed by the international Sepsis Definitions Task Force (7). This insight would seem most likely to account for their success.

Fountain's success also raises the question of why so many EMR interventions for sepsis and other disease processes have failed to improve patient care. In order to be successful, CDSSs need to pick diseases with well grounded criteria and interventions. This requires extensive expertise in reading and evaluating the medical literature. It seems too often a quick internet search by a non-expert committee chooses poorly. For example, ventilator-associated pneumonia is a disease with no well established criteria or accepted prevention other than extubation. Too often EMRs have increased workload and inefficiency without apparent patient benefit, even potential patient harm as suggested by some.

If Fountain's criteria is replicated in randomized trials and early identification improves outcomes, it may represent a major step forward in sepsis care. However, perhaps more importantly it could represent a major step forward in how CDSSs are conceived and developed.

Richard A. Robbins, MD

Editor, SWJPCC

References

1. Gillum RF. From papyrus to the electronic tablet: a brief history of the clinical medical record with lessons for the digital age. Am J Med. 2013 Oct;126(10):853-7. [CrossRef] [PubMed]
2. Miller RR 3rd, Dong L, Nelson NC, Brown SM, Kuttler KG, Probst DR, Allen TL, Clemmer TP; Intermountain Healthcare Intensive Medicine Clinical Program. Multicenter implementation of a severe sepsis and septic shock treatment bundle. Am J Respir Crit Care Med. 2013 Jul 1;188(1):77-82. [CrossRef] [PubMed]
3. Tafelski S, Nachtigall I, Deja M, Tamarkin A, Trefzer T, Halle E, Wernecke KD, Spies C. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010 Sep-Oct;38(5):1605-16. [CrossRef] [PubMed]
4. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015 Jan;10(1):26-31. [CrossRef] [PubMed]
5. Fountain S, Perry J III, Stoffer B, Raschke RA. Design of an electronic medical record (EMR)-based clinical decision support system to alert clinicians to the onset of severe sepsis. Southwest J Pulm Crit Care. 2016 Apr;12(4):153-60. [CrossRef]
6. Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015 Apr 23;372(17):1629-38. [CrossRef] [PubMed]
7. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM, Hotchkiss RS, Levy MM, Marshall JC, Martin GS, Opal SM, Rubenfeld GD, van der Poll T, Vincent JL, Angus DC. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016 Feb 23;315(8):801-10. [CrossRef] [PubMed] 

Cite as Robbins RA. Using the EMR for better patient care. Southwest J Pulm Crit Care. 2016 Apr;12(4):161-2. doi: http://dx.doi.org/10.13175/swjpcc034-16 PDF