Introduction

This concept contains a description of a method that investigates the diagnostic validity / level of agreement of cancer diagnoses codes that are recorded in different administrative health data sets, and the findings of this investigation. The information in this concept comes from the MCHP deliverable, Cancer Data Linkage in Manitoba: Expanding the Infrastructure for Research, by Lix et al. (2016).

The investigation examined the validity (accuracy and timing) of cancer diagnoses in administrative health data sets using the Cancer Registry data as the reference data. Knowledge of this type of information is important for investigators who may not have access to the Cancer Registry data and want to develop cancer cohorts for their research.

Methodology

As one aspect of data quality, Lix et al. (2016) examined the diagnostic validity / level of agreement between specific cancer diagnosis codes recorded in the Manitoba Cancer Registry and Treatment data and the corresponding information in the Hospital Abstracts data and the Medical Services (Physician Claims) data. The following five cancer sites were investigated: bladder, breast, colorectal, lung, and prostate.

Different International Classification of Diseases (ICD) coding systems are used to record diagnoses information in each of these data sets, including: ICD-O-3 in the Cancer Registry data; ICD-9-CM and ICD-10-CA in the Hospital Abstracts data, and ICD-9-CM in the Medical Services (Physician Claims) data. The following table identifies each of the five cancer sites investigated and the corresponding ICD codes used to identify each type of cancer.



Cancer Site	ICD-0-3	ICD-9-CM	ICD-10-CA
Bladder	C670–C679	188	C67
Breast	C500–C509	174	C50
Colorectal (colon, rectum and rectosigmoid)	C180–C189, C199, C209, C260	153, 1540, 1541	C18, C19, C20
Lung and bronchus	C340–C349	162	C34
Prostate	C619	185	C61


This analysis was undertaken for the period from April 1, 1997 to March 31, 2011. Individuals were identified as cancer cases in administrative health data if they had at least one hospital abstract or one physician claim with the relevant cancer diagnosis within one month, six months, and one year (before or after) the cancer diagnosis date in the Manitoba Cancer Registry.

Measuring the Diagnostic Validity / Level of Agreement

The following statistics were used to estimate the diagnostic validity for each cancer site:


• sensitivity,
  
• specificity,
  
• positive predictive value (PPV), and
  
• negative predictive value (NPV),

The Wilson score interval method was used to calculate the 95% confidence interval for these statistics.

The kappa (κ) statistic was used to measure the level of agreement between the data sources. The interpretation of kappa (κ) adopted in this analysis was:


• κ < 0.20 is poor agreement,
  
• 0.20 <= κ <= 0.39 is fair agreement,
  
• 0.40 <= κ <= 0.59 is moderate agreement,
  
• 0.60 <= κ <= 0.79 is good agreement, and
  
• κ >= 0.80 is very good agreement (Altman, 1990).

Findings

The results of this investigation are reported below in Table 1. "When we used a one–month observation window, that is, when we considered the period from one month before to one month after the cancer diagnosis to estimate diagnostic validity, sensitivity estimates ranged from 0.69 for prostate cancer to 0.89 for lung cancer. Specificity attained its upper bound of 1.00 for all cancers. PPV values were below 0.99 for bladder cancer only. Kappa estimates ranged from 0.77 (good agreement) for prostate cancer to 0.92 (very good agreement) for lung cancer.

Diagnostic validity improved when a six–month observation window was used, that is, when we considered the period from six months before to six months after the cancer diagnosis to estimate diagnostic validity. For prostate cancer, sensitivity increased to 0.95. Sensitivity was lowest for bladder cancer (0.92). PPV was also lowest for bladder cancer (0.84), and was highest for breast cancer (0.98). Kappa estimates showed very good agreement for all cancers, and ranged from 0.87 for bladder cancer to 0.97 for breast cancer.

There was almost no change in the estimates when a 12–month observation window was used, that is, when we considered the period from 365 days before to 365 days after the cancer diagnosis. Furthermore, in a sensitivity analysis we found that the validity estimates did not vary by cancer stage (data not shown)."

Table 1: Validation Measures for Cancer Diagnoses in Administrative Data
By time period, cancer site and agreement measure (95% Confidence Interval)

Time Period: 1 month before and after cancer diagnosis



Cancer Site	Sensitivity	Specificity	Positive Predictive Value (PPV)	Negative Predictive Value (NPV)	Kappa
Bladder	0.72 (0.69, 0.74)	1.00 (1.00, 1.00)	0.89 (0.87, 0.91)	0.99 (0.99, 0.99)	0.79 (0.77, 0.81)
Breast	0.87 (0.86, 0.87)	1.00 (1.00, 1.00)	0.99 (0.99, 0.99)	0.96 (0.96, 0.96)	0.90 (0.90, 0.91)
Colorectal	0.86 (0.86, 0.87)	1.00 (1.00, 1.00)	0.99 (0.99, 0.99)	0.96 (0.96, 0.96)	0.90 (0.89, 0.91)
Lung	0.89 (0.89, 0.90)	1.00 (0.99, 1.00)	0.99 (0.98, 0.99)	0.96 (0.96, 0.97)	0.92 (0.91, 0.92)
Prostate	0.69 (0.68, 0.70)	1.00 (1.00, 1.00)	0.99 (0.98, 0.99)	0.92 (0.92, 0.92)	0.77 (0.77, 0.78)

Time Period: 6 months before and after cancer diagnosis



Cancer Site	Sensitivity	Specificity	Positive Predictive Value (PPV)	Negative Predictive Value (NPV)	Kappa
Bladder	0.92 (0.90, 0.93)	0.99 (0.99, 0.99)	0.84 (0.82, 0.86)	1.00 (1.00, 1.00)	0.87 (0.86, 0.89)
Breast	0.98 (0.98, 0.98)	0.99 (0.99, 0.99)	0.98 (0.97, 0.98)	0.99 (0.99, 0.99)	0.97 (0.97, 0.97)
Colorectal	0.96 (0.95, 0.96)	0.99 (0.99, 0.99)	0.97 (0.97, 0.98)	0.99 (0.99, 0.99)	0.96 (0.95, 0.96)
Lung	0.96 (0.96, 0.97)	0.99 (0.99, 0.99)	0.97 (0.97, 0.97)	0.99 (0.99, 0.99)	0.96 (0.95, 0.96)
Prostate	0.95 (0.95, 0.96)	0.99 (0.99, 0.99)	0.97 (0.97, 0.98)	0.99 (0.99, 0.99)	0.95 (0.95, 0.95)

Time Period: 1 year before and after cancer diagnosis



Cancer Site	Sensitivity	Specificity	Positive Predictive Value (PPV)	Negative Predictive Value (NPV)	Kappa
Bladder	0.93 (0.92, 0.95)	0.99 (0.99, 0.99)	0.81 (0.79, 0.83)	1.00 (1.00, 1.00)	0.86 (0.85, 0.87)
Breast	0.98 (0.98, 0.99)	0.99 (0.99, 0.99)	0.97 (0.96, 0.97)	0.99 (0.99, 1.00)	0.97 (0.96, 0.97)
Colorectal	0.96 (0.96, 0.97)	0.99 (0.99, 0.99)	0.96 (0.96, 0.97)	0.99 (0.99, 0.99)	0.95 (0.95, 0.95)
Lung	0.97 (0.97, 0.97)	0.99 (0.98, 0.99)	0.96 (0.96, 0.96)	0.99 (0.99, 0.99)	0.95 (0.95, 0.96)
Prostate	0.97 (0.96, 0.97)	0.99 (0.99, 0.99)	0.96 (0.96, 0.97)	0.99 (0.99, 0.99)	0.95 (0.95, 0.96)

This study demonstrates that investigators who do not have access to the Manitoba Cancer Registry can accurately ascertain cancer cases from hospital abstract records and medical services / physician billing claims for the specific cancers investigated in this research.

NOTE:

The information documented in this concept is available in the section of the deliverable titled Validity of Cancer Diagnoses in Administrative Health Data.