Concept: Education Indices
Last Updated: 2015-04-30
The Scaled Logit Variable
The Scaled Logit variable is (essentially) a transformation of the Grade 12 Language Arts standards test score. The standards test scores were grouped and placed in a rank order such that each subsequent grouped test score, from the lowest to the highest, represents an improved educational achievement.
The Logit values (Y) were obtained by using the following formula :Y = ([Q log(Q) + (1-Q)log(1-Q)] - [P log(P) + (1-P)log(1-P)]) / (Q - P)Where Q = is the fraction of students scoring beyond the given ranked outcome, and P = the fraction of students scoring beyond and including the given ranked outcome.
The logit values were then divided by the standard deviation for the entire cohort. This gives a scaled logit score with a mean of zero and a standard deviation of one. The result is a numeric variable with values (depending upon the cohort year) ranging from -1.84 to 2.96. A larger value of this Scaled Logit indicates a higher grade on the Grade 12 standards test (and a higher likelihood of having completed high school).
Using the Language Arts achievement index allows each person in the birth cohort remaining in the province to be placed on a distribution regarding educational achievement. The complementary Mathematics achievement index can also be used as an outcome for correlational analyses in cross-sectional and longitudinal work. This will permit regression and similar analyses which cannot be done as well using dichotomous (logistic regression) or rank-order techniques.
Rerunning the transformation to produce the Scaled Logit variable for each year is not only necessary because of year-to-year differences in the categories used but is, generally, a good idea. Differences over time in the scoring of achievement tests have been noted in other contexts (i.e., in analyses of the Panel Study of Income Dynamics) (Hofferth and Reid, 2002) . Transforming the data each year controls for any trends in scoring.Categories
For the current Language Arts achievement index, outcomes were grouped (from low to high) as follows:
19: Withdrawn
18: Not Enrolled
17: S3 (Grade 11) or lower
16: in S4 (Grade 12) - not tested
15: Absent/Dropped
14: 00 - 35 %
13: 35 - 40 %
12: 40 - 45 %
11: 45 - 50 %
10: 50 - 55 %
09: 55 - 60 %
08: 60 - 65 %
07: 65 - 70 %
06: 70 - 75 %
05: 75 - 80 %
04: 80 - 85 %
03: 85 - 90 %
02: 90 - 95 %
01: 95 - 100%Definitions of Categories:
- Absent/Dropped: for a student who was supposed to write the exam, an outcome of "absent" was assigned when the student was absent on the day or days of the test (for example, due to illness or medical condition). This category also includes students who dropped the course before the exam was administered.
- in S4 (Grade 12) - not tested: students who were in Grade 12 (S4) but could not be matched to an exam score nor any explanation on the absences file (see above).
- S3 (Grade 11) or lower: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in Grade 12 (S4) and thus should have written a Grade 12 provincial exam but who instead were enrolled (determined from the enrollment database) in Grade 11 (S3) or lower.
- Not Enrolled: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in school, but who were not enrolled in school for 1 year.
- Withdrawn: students who, based on their age (i.e. will turn 17 by December 31 of the school year) and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in school, but who were not graduated nor enrolled in school for any 2 consecutive years at any time after entering Grade 10 (S2). A few students may have transferred to a First Nations school (formerly called band-operated school); a thorough analysis would also check the enrollment data to determine if this were the case.
Population-Based Categories for Language Arts Achievement Index
Index Category Frequency Percent Scaled Logit Percent Graduated
in 4 years95-100% 352 1.43 2.96 87.22
90-95% 650 2.63 2.04 86.62
85-90% 1,213 4.92 1.52 85.90
80-85% 1,510 6.12 1.13 84.70
75-80% 1,882 7.63 0.83 83.42
70-75% 2,075 8.41 0.57 80.87
65-70% 1,882 7.63 0.35 80.39
60-65% 1,517 6.15 0.19 80.22
55-60% 1,749 7.09 0.04 80.16
50-55% 846 3.43 -0.08 77.78
45-50% 550 2.23 -0.15 75.64
40-45% 329 1.33 -0.19 73.25
35-40% 272 1.10 -0.22 70.59
00-35% 114 0.46 -0.24 68.42
Absent/Dropped Class 382 1.55 -0.26 44.50
Grade 12 - not tested 1,866 7.56 -0.37 36.60
Grade 11 or lower 4,633 18.78 -0.77 27.46
Not enrolled 395 1.60 -1.19 6.33
Withdrawn 2,459 9.97 -1.84 1.38
N (two birth cohorts) 24,676 * The first 14 categories of the Language Arts achievement index are based on scores on the Grade 12 LA standards test. Grade in high school refers to the grade of the cohort member at the age-appropriate year (1999/2000 for the 1982 Manitoba birth cohort and 2001/2002 for the Manitoba 1984 birth cohort).
Index Development
The first phase of index development is summarized in Table A.
Table A. Summary of Mathematics Achievement Measures
Mathematics Achievement 1979-1981 Birth Cohort 1984-1989 Birth CohortCourse-based Measure
(Dichotomous)Grade 12 high school mathematics (with a grade of 80%) or passing precalculus course Passing precalculus course Test-based Measure
(Logit Scale)Single provincial Grade 12 Mathematics exam Three provincial Grade 12 Mathematics exams
Thus, the course-based measure of mathematics achievement used either a passing mark in the precalculus course or 80% or higher on the Grade 12 high school mathematics course given in the late 1990s. Mathematics curricula changed gradually, with a precalculus course (and other advanced courses) gradually spreading throughout the province's high schools. This algorithm assured that about 25% of each birth cohort was defined as passing "university preparatory" mathematics. By 2001 (the 1984 birth cohort was in Grade 12), the precalculus exam was installed throughout the curriculum in provincial high schools.
The reliability of the indices was checked by dividing the sibling pairs into three groups:
- that in which both siblings were from the younger birth cohort (born 1979-1981),
- that in which both were from the older birth cohort (born 1984-1989), and
- with one sibling from the younger cohort and one from the older cohort.
Grade 12 mathematics provincial exam data were not available for the 1982 and 1983 birth cohorts; 1999 was a transition year, while provincial Grade 12 exams were not given in 2000.
Correlation
Thus, comparisons involving both siblings in group a (the 1979-1981 birth cohort) and both siblings in group b (the 1984-1989 birth cohort) use the same version of the course-based and of the test-based measures. Reasonable sibling correlations were expected for groups a) and b). Somewhat smaller correlations were expected for group c), since the method of measurement changed slightly between birth cohorts.
Table B. Summary of Correlations
Group N Course-based Measure Test-based Measure Age Difference
Mean (Years)1 - 1979-1981 1,994 .22 .45 1.642 - 1984-1989 17,008 .39 .49 2.443 - one in each period 7,623 .32 .44 5.19
The Course-based Measure was dichotomous, with the phi coefficient reported. The (unadjusted) Test-based Measure uses a logit scale standardized for each year, with the product-moment correlation reported. Siblings were chosen randomly from the relevant children in each family (excluding twins and half-siblings). Age differences between siblings affected the size of the correlations reported for the Language Arts index, but did not significantly modify the Mathematics correlations.Adjustments
Of course, students are not randomly assigned to each test; better students tend to take the harder courses and associated tests. The goal was to adjust for the ability of the students, so that an estimate of the relative difficulty of each test could be determined. Once we have such an estimate, the scores for the math exams can be adjusted to reflect what each student would have received if all students had taken the same test.
The initial Test-based measure was adjusted using propensity scores to better account for the difficulty of each test. This technique matches students taking pre-calculus with students of similar ability who took one of the other (not university-preparatory) math courses. "Similar ability" is determined by estimating the probability of taking pre-calculus based on marks in other courses. Students with high marks are matched together and students with low marks are matched together. Three groups of students were matched on the probability of taking pre-calculus (a basic propensity score matching). The variables used as predictors were sex, Winnipeg/Non-Winnipeg, marks in other courses, and binary variables indicating whether they took computer science, music, industrial arts, or drama (the kitchen sink approach). The difference in their S4 math test scores is somewhat adjusted for student ability (better reflecting the true difference in S4 test difficulty).
The mean score differences for matched students between Pre-calculus and the other two tests are 20 and 9 points for Consumer Math and Applied Math respectively; the differences are about 8 and 2 points for non-matched groups. The tests are designed to be used by students taking Grade 12 Math courses with the same names.
The scaled logit math score on the Test-based measure was recalculated by adjusting the observed S4 math score. Consumer Math scores were reduced by 20 points, and Applied Math scores by 9 points.
The efficacy of this technique was tested by examining the sibling correlations with the adjusted scaled logit scores. Correlations were higher than those generated using the unadjusted Test-based score.This pattern of higher correlations suggests that adjusted scores should be used in further work using the Test-based measure.
Group Adjusted Test-Based Measure
Sibling Correlation1 - 1979-1981 0.452 - 1984-1989 0.533 - one in each period 0.45