Indicator development cycle
< Back to CIHI Health Indicators
At CIHI, indicators are developed in 4 stages after an information need or knowledge gap has been identified. Each stage has multiple activities. Following is an overview of each stage and some of the activities included.
Stage 1: Initiate/evaluate
An idea undergoes initial investigations and scoping. During this stage, we conduct literature reviews and environmental scans to gain an understanding about what information already exists about the identified gap. We also consult with potential experts and users to clarify what needs to be measured, why and how often.
Stage 2: Develop
We develop and iteratively refine the indicator’s methodology based on input from experts who have a variety of perspectives. This is an essential stage that ensures we develop reliable and comparable indicators that support sound decision-making. Activities during this stage include
- Developing the indicator definition
- Establishing the inclusion/exclusion criteria for case and population selection
- Assessing/confirming the quality of the data
- Seeking expert advisory input (including clinical, research, policy and user perspectives)
- Developing a method to risk-adjust the indicator to make it comparable
- Sharing the overall methodology and preliminary results with experts and stakeholders, including conducting validation exercises with interested parties
- Refining the methodology based on expert and stakeholder feedback after review of preliminary results and (possibly) validation exercises
We also conduct in-depth analyses, including those focused on data quality assessments, to gain a fuller understanding of results (e.g., differences across age groups, males versus females and rural versus urban; outlier anomalies; large percentage changes across data years).
Stage 3: Calculate
This is the number-crunching stage once the methodology has been refined and finalized. We apply the methodology to the full scope of data to be included and prepare the results for release according to the agreed-upon specification and in the agreed-upon format (e.g., data tables, visuals). To ensure that no errors are introduced, we develop standard and repeatable approaches to calculations and complete specific checks (e.g., double runs of results, external auditing of calculations).
Stage 4: Release
Indicator results are released according to the agreed-upon specifications, either privately or publicly. For public release, a preview period in which stakeholders can review their results a final time before they are made public is often used to help stakeholders prepare. During this stage, we conduct final rounds of engagement to gain support for the release, which could include outreach to indicator-relevant stakeholders, system leaders and advocacy groups. Ensuring there are no surprises is CIHI’s goal when releasing new information. An embargo period precedes a public release to further ensure that stakeholders are prepared.
After we’ve released a new indicator, we return to Stage 1 and evaluate it, beginning the development cycle again. Evaluating an indicator follows a similar process to initiating a new one: we re-confirm what needs to be measured, why and how often. We evaluate our indicators on a regular basis to determine what we should keep reporting, what we should redevelop and what we should retire.
The time it takes to go from Stage 1 to Stage 4 in the development cycle of an indicator depends on many factors. A driving factor is the availability of high-quality data.
Timelines are also affected by complexity. An indicator that crosses several sectors (e.g., primary care, home care, emergency department, hospital, rehabilitation, long-term care) and involves linking data across these sectors will take longer to develop than an indicator that is based on 1 sector (e.g., hospital) and 1 outcome (e.g., death). Depending on these factors, 18 months is a reasonable time frame to complete the development cycle for a given indicator.