Concept: Education Indices

Introduction

Three education indices have been created at MCHP to help describe an individual's educational attainment. These include:


1. The Language Arts Achievement Index is based on scores from the Manitoba Grade 12 Language Arts standards test, as well as educational outcomes for 18-year-olds not taking the test.
   
   
2. The Mathematics Achievement Index is based on scores from the Manitoba Grade 12 Mathematics standards test / achievement test and on educational outcomes for those not taking the test.
   
   
3. The Grade 9 Achievement Index is based on grade scores (marks) and the number of credits earned in grade 9. Please see the Grade 9 Achievement Index concept for more information on this index.
   
   NOTE: Detailed information on the Language Arts and Mathematics Achievement Indices are provided in this concept. Click on the links above to go to this information.

Methodology

Different birth cohorts can be merged with data from the Manitoba Health Insurance Registry to determine whether cohort members were living in Manitoba. If they were, enrollment, provincial exam results, and data on absenteeism can be used to determine if they were still in school and if they wrote either of the provincial exams in Language Arts or Mathematics. The achievement indices summarize this information for individuals living in Manitoba in their 18th year.

Merging with Registry Data

This record linkage uses one or more files (i.e.: provincial test scores as well as enrollment and course-related high school files) to ascertain the status of each member of each birth cohort. For example, members of the 1984 cohort noted on the registry as Manitoba residents but not linked to anyone on the 2001-2002 school enrollment file were judged as not enrolled. A high linkage rate is necessary to accurately account for all those born in 1984. The more unlinked, enrolled students, the more likely that a substantial number should have been included in the provincial registry but were not.

The individual identifiers on the education files were run separately against the provincial registry at Manitoba Health to provide an encrypted PHIN for each individual. Manitoba Health supplied MCHP with files containing only the encrypted PHIN and appropriate case identifiers. All such linkage work was performed at Manitoba Health.

Between 93 and 94 percent of those born in 1978 through 1985 (as defined in the education data and including births inside and outside Manitoba) and present in these files were linkable to the provincial Health registry for 2003. Linkage rates for those born in Manitoba are probably somewhat higher than this. Some degree of non-linkage is to be expected because the files are designed independently and for different purposes.

In the school year 2001-2002, 42,727 individuals were noted as attending Grades 11 and 12 in Manitoba. The 1,774 students (4.1%) not linked to the health care registry tended to attend secondary schools known for enrolling out-of-province students. Among the 219,026 students enrolled in Kindergarten through Grade 12, only 4,541 (2.1%) could not be linked to the registration file. Relatively few cohort members appear to have been misclassified as "not in school" while actually enrolled.

Statistical Modeling

Statistical modeling benefits from techniques appropriate for studying an entire birth cohort using one or more outcome measures. Our approach to scaling educational achievement combines "academic achievement and the propensity to remain at school beyond the period of compulsory attendance" (Willms, 1986) . These methods handle heterogeneous files (exam results; grades; information on graduation, retention, or withdrawal); particular items vary somewhat in their availability or coding from year to year. Scaled scores are assigned according to the formula presented in Willms (1986). Each birth cohort (1978-1985 with 1983 absent) was scaled separately because the relevant school years varied somewhat in the detail available for those not tested. Both the Language Arts achievement index and the similar Mathematics index merge information from Grade 12 testing with several alternatives experienced by cohort members who did not take the test: absent (about 1 percent of each birth cohort sample), enrolled in Grade 12 but not tested (about 8 percent), enrolled in Grade 11 or lower (about 19 percent), not enrolled (about 2 percent), and withdrawn from school (about 10 percent). For these categories who did not write the test, test scores were imputed by defining additional categories and ranking them below the lowest score attained by the group writing the test (Oreopoulos et al., 2008) .

Language Arts Achievement Index

Following Mosteller and Tukey (1977) and Willms (1986) , the technique assumes that each category represents a score on an underlying logit distribution; the distribution is divided into pieces according to the percentage of cohort members in each category. The fourth column in the table below presents a logit transform used for the Language Arts index. The highest test scorers were given an index score of +2.96, while those withdrawn from school received the lowest score of -1.84. Used over the seven years of data, the logit transform produces an index with an overall mean close to zero and a standard deviation close to one. The ordering on this index corresponds closely with the student's probability of graduation (Oreopoulos et al.(2008), Currie et al. (2010)) .

The Scaled Logit Variable

The Scaled Logit variable is (essentially) a transformation of the Grade 12 Language Arts standards test score. The standards test scores were grouped and placed in a rank order such that each subsequent grouped test score, from the lowest to the highest, represents an improved educational achievement.

The Logit values (Y) were obtained by using the following formula :

Y = ([Q log(Q) + (1-Q)log(1-Q)] - [P log(P) + (1-P)log(1-P)]) / (Q - P)

Where Q = is the fraction of students scoring beyond the given ranked outcome, and P = the fraction of students scoring beyond and including the given ranked outcome.

The logit values were then divided by the standard deviation for the entire cohort. This gives a scaled logit score with a mean of zero and a standard deviation of one. The result is a numeric variable with values (depending upon the cohort year) ranging from -1.84 to 2.96. A larger value of this Scaled Logit indicates a higher grade on the Grade 12 standards test (and a higher likelihood of having completed high school).

Using the Language Arts achievement index allows each person in the birth cohort remaining in the province to be placed on a distribution regarding educational achievement. The complementary Mathematics achievement index can also be used as an outcome for correlational analyses in cross-sectional and longitudinal work. This will permit regression and similar analyses which cannot be done as well using dichotomous (logistic regression) or rank-order techniques.

Rerunning the transformation to produce the Scaled Logit variable for each year is not only necessary because of year-to-year differences in the categories used but is, generally, a good idea. Differences over time in the scoring of achievement tests have been noted in other contexts (i.e., in analyses of the Panel Study of Income Dynamics) (Hofferth and Reid, 2002) . Transforming the data each year controls for any trends in scoring.

Categories

For the current Language Arts achievement index, outcomes were grouped (from low to high) as follows:


19: Withdrawn
18: Not Enrolled
17: S3 (Grade 11) or lower
16: in S4 (Grade 12) - not tested
15: Absent/Dropped
14: 00 - 35 %
13: 35 - 40 %
12: 40 - 45 %
11: 45 - 50 %
10: 50 - 55 %
09: 55 - 60 %
08: 60 - 65 %
07: 65 - 70 %
06: 70 - 75 %
05: 75 - 80 %
04: 80 - 85 %
03: 85 - 90 %
02: 90 - 95 %
01: 95 - 100%

Definitions of Categories:



• Absent/Dropped: for a student who was supposed to write the exam, an outcome of "absent" was assigned when the student was absent on the day or days of the test (for example, due to illness or medical condition). This category also includes students who dropped the course before the exam was administered.
  
  
• in S4 (Grade 12) - not tested: students who were in Grade 12 (S4) but could not be matched to an exam score nor any explanation on the absences file (see above).
  
  
• S3 (Grade 11) or lower: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in Grade 12 (S4) and thus should have written a Grade 12 provincial exam but who instead were enrolled (determined from the enrollment database) in Grade 11 (S3) or lower.
  
  
• Not Enrolled: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in school, but who were not enrolled in school for 1 year.
  
  
• Withdrawn: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in school, but who were not graduated nor enrolled in school for any 2 consecutive years at any time after entering Grade 10 (S2). A few students may have transferred to a First Nations school (formerly called band-operated school); a thorough analysis would also check the enrollment data to determine if this were the case.

Population-Based Categories for Language Arts Achievement Index

Index Category	Frequency	Percent	Scaled Logit	Percent Graduated in 4 years
95-100%	352	1.43	2.96	87.22
90-95%	650	2.63	2.04	86.62
85-90%	1,213	4.92	1.52	85.90
80-85%	1,510	6.12	1.13	84.70
75-80%	1,882	7.63	0.83	83.42
70-75%	2,075	8.41	0.57	80.87
65-70%	1,882	7.63	0.35	80.39
60-65%	1,517	6.15	0.19	80.22
55-60%	1,749	7.09	0.04	80.16
50-55%	846	3.43	-0.08	77.78
45-50%	550	2.23	-0.15	75.64
40-45%	329	1.33	-0.19	73.25
35-40%	272	1.10	-0.22	70.59
00-35%	114	0.46	-0.24	68.42
Absent/Dropped Class	382	1.55	-0.26	44.50
Grade 12 - not tested	1,866	7.56	-0.37	36.60
Grade 11 or lower	4,633	18.78	-0.77	27.46
Not enrolled	395	1.60	-1.19	6.33
Withdrawn	2,459	9.97	-1.84	1.38
N (two birth cohorts)	24,676

* The first 14 categories of the Language Arts achievement index are based on scores on the Grade 12 LA standards test. Grade in high school refers to the grade of the cohort member at the age-appropriate year (1999/2000 for the 1982 Manitoba birth cohort and 2001/2002 for the Manitoba 1984 birth cohort).

Mathematics Achievement Index

As noted with regard to Language Arts, we wanted to develop variables to measure achievement in Mathematics across as many birth cohorts as possible. This task was made more difficult by changes in the curriculum and in the testing procedures used for the birth cohorts studied. As described below, variable development was complicated. Because both the literature and previous work using the Language Arts index suggested that scores between siblings should be reasonably correlated, we used this criterion to judge the success of our efforts.

Index Development

The first phase of index development is summarized in Table A.

Table A. Summary of Mathematics Achievement Measures



Mathematics Achievement	1979-1981 Birth Cohort	1984-1989 Birth Cohort
Course-based Measure (Dichotomous)	Grade 12 high school mathematics (with a grade of 80%) or passing precalculus course	Passing precalculus course
Test-based Measure (Logit Scale)	Single provincial Grade 12 Mathematics exam	Three provincial Grade 12 Mathematics exams

Thus, the course-based measure of mathematics achievement used either a passing mark in the precalculus course or 80% or higher on the Grade 12 high school mathematics course given in the late 1990s. Mathematics curricula changed gradually, with a precalculus course (and other advanced courses) gradually spreading throughout the province's high schools. This algorithm assured that about 25% of each birth cohort was defined as passing "university preparatory" mathematics. By 2001 (the 1984 birth cohort was in Grade 12), the precalculus exam was installed throughout the curriculum in provincial high schools.

The reliability of the indices was checked by dividing the sibling pairs into three groups:

1. that in which both siblings were from the younger birth cohort (born 1979-1981),
   
2. that in which both were from the older birth cohort (born 1984-1989), and
   
3. with one sibling from the younger cohort and one from the older cohort.

Grade 12 mathematics provincial exam data were not available for the 1982 and 1983 birth cohorts; 1999 was a transition year, while provincial Grade 12 exams were not given in 2000.

Correlation

Thus, comparisons involving both siblings in group a (the 1979-1981 birth cohort) and both siblings in group b (the 1984-1989 birth cohort) use the same version of the course-based and of the test-based measures. Reasonable sibling correlations were expected for groups a) and b). Somewhat smaller correlations were expected for group c), since the method of measurement changed slightly between birth cohorts.

Table B. Summary of Correlations



Group	N	Course-based Measure	Test-based Measure	Age Difference Mean (Years)
1 - 1979-1981	1,994	.22	.45	1.64
2 - 1984-1989	17,008	.39	.49	2.44
3 - one in each period	7,623	.32	.44	5.19


The Course-based Measure was dichotomous, with the phi coefficient reported. The (unadjusted) Test-based Measure uses a logit scale standardized for each year, with the product-moment correlation reported. Siblings were chosen randomly from the relevant children in each family (excluding twins and half-siblings). Age differences between siblings affected the size of the correlations reported for the Language Arts index, but did not significantly modify the Mathematics correlations.

Adjustments

Of course, students are not randomly assigned to each test; better students tend to take the harder courses and associated tests. The goal was to adjust for the ability of the students, so that an estimate of the relative difficulty of each test could be determined. Once we have such an estimate, the scores for the math exams can be adjusted to reflect what each student would have received if all students had taken the same test.

The initial Test-based measure was adjusted using propensity scores to better account for the difficulty of each test. This technique matches students taking pre-calculus with students of similar ability who took one of the other (not university-preparatory) math courses. "Similar ability" is determined by estimating the probability of taking pre-calculus based on marks in other courses. Students with high marks are matched together and students with low marks are matched together. Three groups of students were matched on the probability of taking pre-calculus (a basic propensity score matching). The variables used as predictors were sex, Winnipeg/Non-Winnipeg, marks in other courses, and binary variables indicating whether they took computer science, music, industrial arts, or drama (the kitchen sink approach). The difference in their S4 math test scores is somewhat adjusted for student ability (better reflecting the true difference in S4 test difficulty).

The mean score differences for matched students between Pre-calculus and the other two tests are 20 and 9 points for Consumer Math and Applied Math respectively; the differences are about 8 and 2 points for non-matched groups. The tests are designed to be used by students taking Grade 12 Math courses with the same names.

The scaled logit math score on the Test-based measure was recalculated by adjusting the observed S4 math score. Consumer Math scores were reduced by 20 points, and Applied Math scores by 9 points.

The efficacy of this technique was tested by examining the sibling correlations with the adjusted scaled logit scores. Correlations were higher than those generated using the unadjusted Test-based score.

Group	Adjusted Test-Based Measure Sibling Correlation
1 - 1979-1981	0.45
2 - 1984-1989	0.53
3 - one in each period	0.45

This pattern of higher correlations suggests that adjusted scores should be used in further work using the Test-based measure.

Correlation Between the Language Arts and Mathematics Indices

As part of a large Birth Cohort study by Roos et al. (2013), a correlation analysis between the Language Arts Standards Test Index and the Mathematics Achievement Tests Index was performed. The Language Arts Index and the Mathematics Index are not independent of each other. The lower scores on the index (not enrolled in school, withdrawn, not at the age-appropriate grade level) are identical on each index because of its population base and method of construction from multiple files. Examining the two indices over the entire nine birth cohorts results in a correlation of 0.75.

The two indices seem to be examining related, but somewhat different, aspects of high school achievement. Parallel analyses of both indices would seem to be appropriate. When we look at the scores for only those cohort members taking both tests, the correlation is much lower. The results of this analysis are available in a separate document titled Correlation Between Language Arts Indices and Mathematics Indices .

Validation of the Education Indicies

To ensure that the indices are a valid measure of educational achievement, different studies have validated the indices using different measures.


• In preliminary work with the Birth Cohort Study, the Language Arts achievement index was validated against another measure of educational achievement; high school completion (see right hand column - Percent Graduated in 4 years - of the table above). High school graduation could not be determined for all of the cohorts due to a lack of available data for some of the cohorts (i.e., cohorts could not be followed for 2 or 3 years beyond when they should have completed high school). Validation for one or two cohorts, however, would permit use of the index for all available cohorts. The rank ordering of both those taking the Grade 12 Language Arts standards test and those not taking the test was validated against high school graduation for two cohorts.
  
  Students scoring between 95 and 100% on the test were more likely to graduate in four years than were those scoring between 85 and 90%, and so on. Students enrolled, but performing poorly on the test, were more likely to graduate than those absent on test day or dropping the Language Arts course. Few cohort members at the bottom of the achievement scale (those not enrolled or withdrawn) graduated on time; such graduation was possible if students returned to school after high school grade and enrollment status were measured early in the school year.
  
  From the analyses on the individual data, with 19 categories on the Language Arts achievement index and 2 categories (Yes/No) for the "Graduated in 4 Years" measure, the correlation (a point biserial measure in this case) was 0.54 (p<.0001 level).
  
  
• In the publication How Are Manitoba's Children Doing by Brownell et al. (2012), they compared the indices to "educational attainment" reported in the Canadian Community Health Survey (CCHS). They found a strong relationship between the CCHS results and scores on the LA Index and on the Mathematics Index. Discussion of the validation methods and validation tables are available in Appendix 7 of this report.
  
  
• In the journal publication What is Most Important: Social Factors, Health Selection, and Adolescent Educational Achievement by Roos et al. (2013), they approached validation in two ways: comparing grade 9 achievement scores with grade 12 achievement scores, and comparing achievement scores with a student's probability of high school graduation within 5 years. Discussion of the validation methods and validation tables are available in Appendix 3 of this publication.

Twins

Although the Manitoba data do not permit separating monozygotic (identical) and dizygotic (fraternal) twins, our research, like that of others using somewhat different measures, finds twin correlations to be substantially higher than those of non-twin siblings. The overall magnitude of Manitoba same-sex twin correlations on the Language Arts achievement index (.738) proved similar to those reported in sophisticated reviews of IQ correlation studies. (Devlin et al., 1997; Duncan et al., 2001; Posthuma et al., 2002). Pearson correlations were slightly lower for boys than for girls and somewhat higher for adolescents residing outside Winnipeg than for those inside.

Limitations

The criteria used for calculating the "withdrawn" category is: not in school for 2 consecutive years. This means the "withdrawn" category will always be incomplete for the cohort who are expected to be in grade 12, and thereby writing the provincial exams, for the two most current years of high school provincial exam data. For example, as of December 2009, enrollment data were unavailable for 2008/09. Thus, for members of the 1989 cohort who did not take the Language Arts standards test in 2006/07 and were not enrolled in the 2007/08 school year, we do not at present know whether or not they were enrolled in school in the following year (2008/09) and are unable to be certain if they are withdrawn. Our withdrawn category will also be incomplete for members of the 1990 cohort who did not write the test in 2007/08.

In Manitoba, provincial exams are not centrally marked but are marked within the school divisions. This could influence results of analyses if those marking the tests are likely to give lower scores to students from lower socioeconomic backgrounds (Brownell et al., 2006). There is, however, a marking feedback process in which centrally trained markers look at a sample of test booklets from every district.