Concept: Population Data and Significance Testing: A Brief Discussion

Introduction

This concept contains a question and responses from two MCHP researchers related to the use of significance testing when using population-based data.

Question

The question asked is .. "With respect to the report Indicators of Health Status and Health Service Use for the Winnipeg Regional Health Authority, were sample data used and, if so, was the sample more or less random? I am a bit puzzled as to why the report makes use of inferential statistics if the findings are based on population data, as it seems to be. Unless I'm mistaken, significance is essentially a measure of risk in making an inference on the basis of a sample pattern to the population from which the sample was randomly drawn/assigned. If population data are used, however, then significance is not meaningful since there is no inferential risk. In such a case it might be misleading to report p levels. Specifically, some substantial findings might be inappropriately discarded by the reader as being 'non-significant', while trivial findings might be unduly emphasized by the reader on the basis of being 'significant'".

Response 1 - (from Norm Frohlich)

No, there was no sample. The total population, as captured in the administrative data, form the basis of the report.

The question you raise is one that was debated quite intensely a number of years ago when we first started issuing reports and papers. Yes, indeed, significance tests are usually used to indicate whether or not the variation across units is due to chance as a function of sampling. BUT, we (at least a majority of us) reached the conclusion that even when one has data on the full population, one only has that data cross-sectionally in time. In a sense, the data can be viewed as a sample from possible states in the Province as they unfold over time. Therefore, it made sense to us to try to indicate whether differences which are certainly real across units are statistically significant when one considers the data to be a one-time sample of the unfolding of the universe.

I believe that it helps to highlight what may be important differences which may have theoretical importance across units (as opposed to differences possibly due to a number of stochastic factors). Some folks still find it inappropriate to report these tests but we are convinced it's a good idea.

Response 2 - (from KC Carriere)

When I was first involved with these studies as a statistician who was used to small sample data only, I too was concerned with what could be done with population data. The main goal is that we want to do the study objectively, not subjectively. Population data means that all quantities are fixed and there is nothing reasonable that can tell us, 'What is too much variation across the areas?'

For example, the two areas are different if their procedure rates are, say, .455 and .456, because .455 and .456 are NOT EXACTLY the SAME! This is of course unreasonable, making any such studies pointless, as all will be significant, distinct, and different.

Although your interpretation (Answer 1) is a good one and is defensible, it requires longitudinal data to confirm the assumption. My reasoning is that we assume that there is a super-population. This inferential approach is born out of a survey sampling context. Under this assumption, the data is just a repeated selection from that super-population.

The super-population is made up of all individuals from all areas, similar in key characteristics as in Manitobans (or small areas within) and thus the inference made is applicable to all those who see themselves being included in that super-population, thereby extending the scope of generalizability of the findings to those other areas.