Concept: Bootstrapping Methods

Introduction

This concept describes a statistical method called bootstrapping and provides information on how it has been used in MCHP research. This includes a definition of the term, and a description of different methods, including SAS code examples for specific MCHP research projects.

Bootstrapping Defined

Bootstrapping is a statistical "... technique for estimating the variance and the bias of an estimator by repeatedly drawing random samples with replacement from the observations at hand. One applies the estimator to each sample drawn, thus obtaining a set of estimates. The observed variance of this set is the bootstrap estimate of variance. The difference between the average of the set of estimates and the original estimate is the bootstrap estimate of bias." (Last, 1995, pg. 18)

Bootstrapping Methods Used in MCHP Research

This section describes how bootstrapping has been used in MCHP research, and provides specific examples and SAS code that have been developed at MCHP.

1. MCHP Bootstrapping SAS Macro

In 2000, MCHP developed a quick and simple SAS macro bootstrap code for calculating confidence intervals around the mean and median. This macro has the following parameters: the data set being investigated, the number of iterations to sample (default is 50), the variable of interest for calculating the median/mean, and the desired confidence interval level (default is 95%). The output from the macro includes:


• the standard mean and confidence interval for the variable;
  
• the standard method bootstrap mean, median and confidence interval for the variable; and
  
• the bias reduced method bootstrap mean, median and confidence interval for the variable.

For more information, please read the Bootstrap macro description and documentation in the MCHP SAS® Macros concept.

2. Bootstrapping and Ambulatory Care Sensitive Conditions (ACS)

In Roos et al. (2005), they used a bootstrapping method to generate confidence intervals. The input data for the program is generated by a bootstrap resampling process. For each Ambulatory Care Sensitive (ACS) condition, all cases were randomly resampled 1000 times, and direct-adjusted rates by income quintile were generated for each replicate.

The result was two files of 72,000 observations: 12 ACS conditions, times six income levels, times 1000 replications. One file included rates of ambulatory visits - physician and the other file included hospital discharge abstract data.

The 95% confidence intervals were generated by taking the 2.5%ile and the 97.5%ile of the distribution of bootstrapped rates. Confidence intervals for other quantities, such as the income quintile Q1/Q5 ratios or the Visit/Hospitalization ratios, were generated by deriving the appropriate quantity for each replicate, and then finding the above percentiles of the distribution of the new quantities.

The reason for bootstrapping was not so much a concern about low N, but we wanted to be sure what the appropriate variance was for the different ratios.

For more information, please see:


• bootstrapping_confidence_intervals.sas.txt - this SAS code example provides the code used in the ACS project.
  
• listing_confidence_intervals.pdf - the output listing file produced when the SAS code program described above was run.

3. Bootstrapping in the Birth Cohort Research

In the Birth Cohort Registry research, bootstrapping methods were used. The following list identifies and describes these situations:


• bootstrapping methods were used to produce estimates of multilevel modeling output in order to derive new standard errors based on the actual distribution of the data, as opposed to a theoretical distribution. Please read the Bootstrapping Documentation from the Birth Cohort Project for more information on the bootstrapping method applied for this project.
  
  
• bootstrapping methods were used to determine the difference in effects. For more information, please read the Using Bootstrapping Methods to Determine Difference in Effects documentation that contains discussion and SAS code programs developed for different situations. The discussion provides background information on two questions posed:
  
  
  1. When using a uniform set of predictors, are the effects larger when using the full population than the effect when using a sibling subsample?; and
     
  2. When using two distinct cohorts to model two distinct (but related) outcomes using a common set of predictors, are there differences in the effects?
  
  and the documentation also provides a methodological approach and SAS code example for tackling each of these situations.

4. Bootstrapping Methods in Other MCHP Research

Bootstrapping has also been used in other MCHP research projects. The following identifies individual research projects and briefly describes how bootstrapping was used in each project. More information can be found in the on-line publication.


• In the Composite Measures/Indices of Health and Health System Performance deliverable by Metge et al. (2009), they age- and sex-adjust rates calculated from Canadian Community Health Survey (CCHS) data using a direct standardization method as opposed to age- and sex-adjustment within a modelling framework. Rates were calculated from the CCHS sample and then weighted to the entire Manitoba population aged 12 and older using the full sample weights provided by Statistics Canada. Confidence intervals were calculated for rates from the standard errors estimated using the 500 bootstrap weights. Comparisons between rates were performed by first calculating the difference between two rates using the full sample weights, then bootstrapping that difference using the 500 bootstrap weights to obtain an estimate of the error of the difference. Then, the 99% confidence interval of the difference was calculated using the bootstrapped standard error. If the confidence interval of the difference did not contain zero, then there was a significant difference between the rates for the indicator under study.
  
  
• In The Additional Cost of Chronic Disease in Manitoba deliverable by Finlayson et al. (2010), testing for the statistical significance of the ratio of the means of the adjusted cost for each RHA against the provincial ratio was not straightforward because the statistic (ratio of the means) does not have any theoretical variance or standard error. Bootstrapping was used to calculate the 95 percentile confidence intervals because of the small number of observations.
  
  
• In the Adult Obesity in Manitoba: Prevalence, Associations, and Outcomes deliverable by Fransoo et al. (2011), they relied on data collected in surveys (e.g.: body mass index (BMI), hypertension, diabetes) of randomly selected Manitobans (not the entire population), so the calculations could only provide estimates of the true values for the population they represent (all Manitobans). Statistically, this uncertainty is expressed by the use of confidence intervals (CIs) (usually at the 95% level with corrections for multiple testing as required). Confidence intervals will be wide for analyses containing a smaller number of survey participants or groups which contain a large amount of variation (or both). Confidence intervals were calculated via the bootstrapping method, using 500 sub–samples of the data. When using the Statistics Canada surveys (National Population Health survey (NPH) and CCHS), the bootstrap weights provided by Statistics Canada were used.
  
  
• In the How are Manitoba's Children Doing? deliverable by Brownell et al. (2012), they used Lorenz curves and Gini coefficients to indicate inequities in the indicators. A Gini coefficient has a value between 0 and 1; zero indicates no inequity and 1 indicates maximum inequity. The Gini coefficient represents the fraction of the area between the Lorenz curve and the line of equity. Confidence intervals (CI) of the Gini coefficients were derived using bootstrapping techniques with 200 replacements.
  
  
• In the Health and Healthcare Utilization of Francophones in Manitoba deliverable by Chartier et al. (2012), statistical testing, using bootstrapping, was conducted to determine if the rates obtained from the survey sample were different than the rates obtained from the Francophone and Matched Cohorts. See the Statistical Testing glossary term in this report for a more detailed explanation.
  
  
• In The Cost of Smoking: A Manitoba Study deliverable by Martens et al. (2015), they used bootstrapping to calculate variances. The report explains that when analytical expressions are complicated or when the distribution of an estimate is unknown (as can sometimes be the case with rate ratios) it can be difficult to obtain a standard-error estimate that can be used in statistical tests. In these situations, researchers need to use an empirical distribution of the data in order to perform significance tests. Bootstrapping is one approach for using the empirical distribution of the data when conducting statistical tests. This project used bootstrapping to develop estimates and obtain the standard deviation of the bootstrapped parameters. They used the bootstrapped standard errors to calculate the Z statistics for differences in proportions, and two-tailed tests with a priori p-values < 0.05 to determine statistical significance. In this research, a rate difference was simply the difference between the rate of an outcome between smokers and non-smokers. This was calculated by subtracting the rate among non-smokers from the rate among smokers. They tested whether the rate differences were statistically significantly different from zero by bootstrapping the estimates and obtaining the standard deviation of the bootstrapped parameters. If the rate difference was statistically significantly different from zero, then smokers had a different rate of healthcare use compared with non-smokers.