Accessing HSMR results

  1.  Where can I access private HSMR results?
  2.  What happened to the HSMR results disseminated through other web tools?
  3.  How can I register for Insight?

Methodology

  1.  What are the most recent methodology changes?

Calculation

  1.  Are MAID cases excluded from the analysis?
  2.  Does the HSMR include palliative care cases?
  3.  Is the logistic regression model based on only publicly reportable hospitals?
  4.  What are the reference groups for the logistic regression that determines expected deaths?
        When I look at the coefficient file, why don’t some variable levels have a coefficient value?
  5.  My hospital’s patients are older. Does this get taken into account?
  6.  Our hospital treats more complicated patients. How has this been taken into account in the HSMR methodology?
  7.  How does the analysis handle transfers between hospitals?
  8.  Are alternate level of care patients excluded from the analysis?
  9.  How are HSMR subgroups (medical, surgical, ICU and excluding transfers) calculated?
  10.  What is included in regional-level HSMR results?
  11.  Are provincial- and national-level HSMRs provided?
  12.  Are there other factors that affect mortality?

Data quality

  1.  Can under-coding or over-coding of comorbid conditions affect the HSMR results?
  2.  How reliable is the data being used to calculate HSMRs?

Interpretation

  1.  How should I interpret HSMR subgroup results (medical, surgical and ICU-related HSMRs)?
  2.  How is the 95% confidence interval interpreted?
  3.  Why do my results differ between reports?
  4.  How do our results compare with those of other, similar facilities?

How to use the HSMR results

  1.  What are the criteria for reporting on the Hospital Deaths (HSMR) indicator in CIHI’s Your Health System public web tool?
  2.  How have organizations in Canada used the HSMR?
  3.  How can we identify areas that need improvement?
  4.  Can the HSMR tell me the number of unexpected deaths that happened at my hospital?

Additional information

  1.  Do you provide the HSMR SAS code and SAS assistance?
  2.  Where can I get the HSMR coefficient files?
  3.  Where can I get more information on the HSMR?
  4.  How can I provide feedback or submit other questions to CIHI?

 

Accessing HSMR results

1. Where can I access private HSMR results?

As of April 2016, the Hospital Standardized Mortality Ratio (HSMR) indicator has been added to Your Health System: Insight, CIHI’s new interactive, secure web tool.

Your Health System: Insight currently houses a number of CIHI indicators. The tool is updated monthly with open-year data so users can monitor their performance on key indicators throughout the year. In addition to seeing province-, region- and hospital-level results, approved users can drill down to their own chart-level information to identify factors that may be driving their organization’s results.

2. What happened to the HSMR results disseminated through other web tools?

When HSMR results were added to Insight, the following private web tools were decommissioned:

  • eHSMR – located in DAD Operational Reports
  • HSMR eReporting Service

3. How can I register for Insight?

If you do not have access to Your Health System: Insight, please log in to your CIHI profile to request access. Should you have any questions, please contact us at help@cihi.ca.

 

Methodology

4. What are the most recent methodology changes?

  • Instead of using separate models for each HSMR subgroup population, HSMR subgroups are based on the risk-adjusted model for the All Cases HSMR.
  • National HSMR: Starting with the 2013–2014 data year, the national HSMR is calculated as a blended average; records from the current fiscal year from all jurisdictions outside of Quebec and records from the previous fiscal year from Quebec are blended to calculate Canada and peer group results. Please refer to the General Methodology Notes on the Resources page of the Indicator Library for more details.
  • Cases from specialty hospitals are included in region-/province-/national-level results; however, HSMR results for specialty hospitals are suppressed. The list of specialty hospitals is available upon request.
  • HSMR results are calculated with an updated baseline using 2012–2013 data. The previous baseline was calculated using 2009–2010 data.
  • The peer groups developed in 2013 for facility-based indicators are used. Details on the peer group methodology are available in CIHI’s Indicator Library under Resources.
  • Instead of comparing HSMR results with 100, HSMRs are compared with the national HSMR. This strengthens the credibility of the measure, as more meaningful comparisons can be made between hospitals and the national result in the years beyond the baseline year.

    An HSMR above the national average indicates that the hospital’s mortality rate is higher than the average rate. An HSMR below the national average indicates that the hospital’s mortality rate is lower than the average rate. An HSMR is scaled such that a score of 100 represents the national average for the baseline year of 2012–2013.
  • The ICD-10-CA codes for stroke have been revised so they are more clinically accurate; therefore, the previous diagnosis group “stroke” has been relabelled “cerebrovascular disease (CVD).” Please refer to the Technical Notes for details.

 

Calculation

5. Are MAID cases excluded from the analysis?

Medical assistance in dying (MAID) was decriminalized in Canada with the enactment of Bill C-14 in June 2016. Information on MAID that is performed in acute care hospitals is being submitted to the Discharge Abstract Database (DAD). Please note that MAID cases are currently included in HSMR calculations. Work is under way to adjust the methodology to remove these cases.

6. Does the HSMR include palliative care cases?

No, the HSMR does not include palliative care patients. For the purposes of HSMR calculation, palliative care cases are defined as those with a most responsible diagnosis of palliative care (patients whose hospitalization was for the purpose of palliative care or patients who received palliative care for the largest portion of their hospital stay). Note that in Quebec, due to different palliative care coding standards, palliative care patients who have cancer are identified as having cancer as the most responsible diagnosis and palliative care as an other diagnosis.

The HSMR calculation does include acute care inpatients who received some palliative care (not representing the largest portion of their length of stay). An example of the type of case that would be included would be a patient who was admitted to an acute care hospital with a hip fracture and who had surgery but, at some point post-operatively, whose condition became progressively worse. The patient, family and care team then determined that the patient’s treatment program would consist of comfort care or palliative care. The patient subsequently died in hospital shortly after being switched over to palliative care.

The number of palliative care cases in a facility is available, along with other descriptive/summary analyses, in Your Health System: Insight, CIHI’s secure web tool. Please note that if a facility does not have eligible HSMR cases, the number of palliative care cases is not shown in Insight.

7. Is the logistic regression model based on only publicly reportable hospitals?

No, the coefficients are derived using all records meeting the inclusion and exclusion criteria from all acute care hospitals in the 2012–2013 Hospital Morbidity Database.

8. What are the reference groups for the logistic regression that determines expected deaths?
      When I look at the coefficient file, why don’t some variable levels have a coefficient value?

In all logistic regression analyses, a reference category must be specified. For all of the HSMR logistic regression models, the following are the reference variables: type of admission = elective, sex = female, LOS group = 3, transfer = 0 and Charlson group = 0. These reference variable levels do not have coefficients in the coefficient files because reference groups have a coefficient of 0. Please refer to the HSMR Technical Notes on the HSMR web page for more details about the risk factors and coefficients.

9. My hospital’s patients are older. Does this get taken into account?

Yes, the HSMR is adjusted for a patient’s age. All else being equal, older patients have a higher risk of dying in hospital than their younger counterparts and are adjusted for accordingly in the calculation.

10. Our hospital treats more complicated patients. How has this been taken into account in the HSMR methodology?

A number of factors contribute to in-hospital mortality. The HSMR methodology adjusts for several of them. Complicated patients tend to be those who are older, admitted under the urgent or emergent category and stay longer in the hospital. The methodology has taken these factors into account, which is consistent with the HSMR methods used in different countries.

In addition, the methodology adjusts for a patient’s Charlson Index score, which reflects preadmission diagnoses recorded on a patient’s discharge abstract. It provides a weighted score for each patient depending on the number and type of diagnoses on the discharge abstract. A higher score generally indicates a more complex case. The Charlson Index is an overall comorbidity score that has been shown to be highly associated with mortality and has been widely used in clinical research. For more information about the Charlson Index, refer to the HSMR Technical Notes.

Note that the HSMR provides a measure of overall mortality and is intended primarily as a tool to track changes over time within a facility. If the patient mix within a facility is relatively stable over time, then changes in outcomes may be identified.

11. How does the analysis handle transfers between hospitals?

Transfers between hospitals are treated as separate admissions. For example, if a patient was transferred from hospital A (acute) to hospital B (acute) and then to hospital C (acute), he or she would be considered a transfer in for hospitals B and C and would be counted in the HSMR for all 3 hospitals (if inclusion and exclusion criteria are otherwise satisfied). For hospitals A and B, this patient would also be considered a transfer out. This is consistent with the approach taken in other countries’ HSMR calculations. The current methodology adjusts for transfers in, which are patients transferred from an acute care institution. Note that transfers from one hospital’s emergency department to another hospital are not adjusted for.

HSMRs excluding all acute transfers (in and out) are provided to help assess how transfers affect patient care in your organization.

12. Are alternate level of care patients excluded from the analysis?

Patients with alternate level of care (ALC) days are not automatically excluded from the HSMR calculations. All patients who meet the inclusion criteria (see the HSMR Technical Notes) for the HSMR are included in our analysis; these may be patients with or without ALC days.

If a patient with ALC days meets the inclusion criteria for the HSMR, the total number of days spent in hospital, including ALC days, is used to calculate the total length of stay. Length of stay is one of the variables adjusted for in the regression model.

13. How are HSMR subgroups (medical, surgical, ICU and excluding transfers) calculated?

As of April 2016, instead of using separate models for each HSMR subgroup population, HSMR subgroups are based on the risk-adjusted model for the All Cases HSMR.

14. What is included in regional-level HSMR results?

Regional-level reports include HSMR cases and deaths from all acute hospitals (including specialty hospitals) in the region. This provides a comprehensive picture of the quality of care at the regional level. Note that if a site did not have cases in the HSMR top 80% list, it was not included in the roll-up.

15. Are provincial- and national-level HSMRs provided?

The HSMR is calculated and provided at the facility, health region, provincial and national levels, provided that the inclusion criteria are met. Please refer to question 4 above on recent methodology changes for more details about the national HSMR.

16. Are there other factors that affect mortality?

Research has suggested that a variety of factors both within and outside the health system may affect in-hospital mortality. Other factors present on admission may also matter (e.g., underlying health status of the population, severity of illness, organization and delivery of care). While the HSMR is adjusted for a number of factors known to affect the risk of in-hospital mortality, we were not able to control for everything.

 

Data quality

17. Can under-coding or over-coding of comorbid conditions affect the HSMR results?

Systematic under- or over-coding of comorbidities may affect HSMR estimates, although the most responsible diagnosis continues to be the most important predictor of in-hospital mortality in most cases. During the HSMR validation process, some organizations identified coding inconsistencies and have implemented processes to make improvements in this area. Further review will hopefully lead to more consistent coding and better data quality across the country.

CIHI advises all facilities to code according to the nationally mandated Canadian Coding Standards. Complying with the standards is essential to ensure the national consistency and quality of the data, leading to the most accurate results on all indicators (including but not limited to the HSMR). Should coders have questions about implementing these standards, they may contact the CIHI Classifications team through the eQuery service.

18. How reliable is the data being used to calculate HSMRs?

A variety of approaches are continually used to improve the quality of the data, including establishment of coding and abstracting standards, automated edits on data submission and database closure, and ongoing education for hospital staff and others involved in the data submission process.

 

Interpretation

19. How should I interpret HSMR subgroup results (medical, surgical and ICU-related HSMRs)?

In addition to the All Cases HSMR, HSMR subgroups have been developed to enable clinical teams within organizations to compare results with those of their peers. As of April 2016, each group is compared with its national average. For example, the medical HSMR is compared with the medical national average; the surgical HSMR is compared with the surgical national average.

20. How is the 95% confidence interval interpreted?

The 95% confidence interval can help to evaluate the precision of the calculated HSMR. The upper and lower confidence intervals are estimated to contain the true value of the HSMR 19 times out of 20 (95% confidence interval). The width of the confidence interval is an indication of the degree of variability associated with the HSMR point estimate: the wider the interval, the greater the variability.

21. Why do my results differ between reports?

HSMR results are available in both Your Health System: In Depth/In Brief (publicly available) and in Your Health System: Insight (secure). In Insight, HSMR results are based on available data as of the corresponding data submission cut-off dates. Data may change significantly throughout the course of the year due to continuous updates to the database. Thus HSMR results reported during the year in Insight may vary until the database is closed and closed-year HSMR results are calculated.

22. How do our results compare with those of other, similar facilities?

As of November 2016, HSMR results are shown using funnel plots in the Your Health System web tool. The funnel plots provide a visual representation of performance relative to the national average, while considering the number of HSMR cases in various organizations. Indicator values outside of the funnel indicate out-of-the-ordinary results. For more details, consult the Methodology document in the tool.

Peer ranges and peer quartiles (QRs) are also provided to allow for additional interpretation of the results in Your Health System: Insight, CIHI’s secure web tool.

 

How to use the HSMR results

23. What are the criteria for reporting on the Hospital Deaths (HSMR) indicator in CIHI’s Your Health System public web tool?

Only those facilities that have 2,500 eligible HSMR cases for each of the most recent 3 consecutive years will be publicly reported in Your Health System: In Depth/In Brief. This ensures stability of the indicator results.

HSMR results for smaller facilities with fewer cases are not publicly reported; however, the results are available in Your Health System: Insight, CIHI’s secure web tool.

24. How have organizations in Canada used the HSMR?

Since the beginning of this project, organizations have been monitoring HSMR results and interpreting their HSMR trends. Some organizations have added this measure to their balanced scorecards or quality monitoring/improvement programs, and they have indicated that the HSMR is reviewed regularly by their boards of directors. In addition, organizations have conducted further investigation and drill-down analysis using various resources and tools. They have identified areas for improvement and developed action plans to focus improvement efforts to reduce hospital mortality.

25. How can we identify areas that need improvement?

As a “big-dot” measure, the purpose of the HSMR is to provide a reflection of in-hospital mortality changes over time for a broad range of disease groups for an organization. CIHI believes that the HSMR should be used along with other indicators to help assess quality of care in hospitals. While a single indicator offers useful information, it should be considered a starting point for further analysis. For example, medical, surgical and ICU HSMRs, where applicable, are provided to help further understand your hospital’s results. Potential starting points for identifying areas for improvement include reviewing mortality rates for meaningful groups (e.g., programs, diagnosis groups). The Institute for Healthcare Improvement's website also contains tools that may help you identify areas for improvement.

26. Can the HSMR tell me the number of unexpected deaths that happened at my hospital?

The HSMR is a broad system-level measure comparing observed to expected deaths. The expected is based on the national experience. Following your organization’s HSMR over time and further analysis of the source data may help to identify and target areas for improvement. At the individual patient level, there might be no obvious issue identified, but statistical measurement and the overall picture may provide a compelling clue to prompt further investigation into clinical processes and lead to improvement in care.

 

Additional information

27. Do you provide the HSMR SAS code and SAS assistance?

We do provide our clients with SAS code to calculate the HSMR. However, the code is provided as is, which means the way we use it for calculating the HSMR using CIHI data. It has to be modified to suit the data structure and the data elements of the hospital’s data. The whole process requires intermediate (including knowledge of SAS macros) SAS skills.

28. Where can I get the HSMR coefficient files?

The All Cases Coefficients are available in the coefficients Excel file. For further questions on coefficients, send an email to hsmr@cihi.ca.

29. Where can I get more information on the HSMR?

Background information and additional reference material are available on CIHI’s website and in the Indicator Library.

30. How can I provide feedback or submit other questions to CIHI?

We welcome your comments/questions about your facility’s results, how you are using the HSMR for quality improvement or the HSMR methodology. Please send an email to hsmr@cihi.ca.