Concept: Standards Tests / Achievement Tests - Test Performance and Outcomes

Introduction

This concept provides information on Standards Tests / Achievement Tests that are provincial exams written by students in Manitoba. The concept includes background information on the types of tests, identifies the data sources in the MCHP Data Repository where the test information is located, describes the methodology and outcome measures developed for analyzing this type of data at MCHP, and provides links to MCHP research that reports on findings for this type of educational testing.

NOTE: In MCHP publications, we have referred to both Language Arts and Mathematics exams as "Standards Tests". For the 2013 academic year (2013/14 school year), we noticed that the mathematics tests are now referred to as "Achievement Tests" in Manitoba Education documents, publications and discussions.

Information in the MCHP Concept Dictionary has been updated to reflect this change as follows: in describing a specific type of test either "Standards" or "Achievement" is used, titles used in relevant glossary terms and concepts contain both descriptions, and in most instances they are now referred to simply as "tests".

Background Information

Standards Tests / Achievement Tests are common provincial exams written by students in Manitoba in certain grades. For Grade 12 students, Manitoba has had a provincial testing system in place since 1993, with the tests counting for 30% of students' final course mark. The current tests are curriculum-based and mandatory for all students, with adaptations available for many special needs students (and exemptions for individual students as required). The annual tests are locally marked by the school divisions, and assess Language Arts and Mathematics in separate tests (Brownell et al., 2004).

There were no Grade 12 tests administered during the 2000 academic year (2000/01 school year) and the test data for grade 12 Math in 1999/00 is not complete. Other than these two years, we have grade 12 test data for Language Arts and Math dating back to 1995 academic year (1995/96 school year), although we do not have complete high school enrollment data for 1995/96. For the current years of education data available, please see the Enrollment, Marks, and Assessments Data Description on the MCHP Data Repository web site.

NOTES:


1. Standardized Grade 3 testing was only administered in 1997/98 and 1998/99 for Math and only in 1998/99 for Language Arts (LA). Standardized grade 3 testing was discontinued after 1998/99. Although the main focus of this concept is on grade 12 test data, this concept also describes MCHP's methods to develop and analyze grade 3 test data.

Data Sources, Methodological Approach and Outcome Measures

To determine outcome measures for students who did write the test and those who should have written the test (those turning 17 years old by December 31 of the school year), we use four databases that are available in the MCHP Repository. The first of these is the Manitoba Health Insurance Registry data and the other three are part of the Enrollment, Marks and Assessments data . Before discussing the data sources and describing how we arrive at the outcome measures, here is a list of the possible test outcome categories developed at MCHP:


• Passed, Have Credit
  
• Failed
  
• Absent, Exempt, Dropped, Incomplete, & FinalMark = -1
  
• In grade 11 (S3) / Grade 2 or lower
  
• In grade 12 (S4), No Test
  
• Withdrawn

Another common outcome we measure is called "on-time pass". For grade 12, this includes students who pass (a mark of 50% or higher) the test and who turned 17 years old by December 31 of the school year. For grade 3, this includes students who pass (a mark of 50% or higher) the test and who turned 8 years old by December 31 of the school year.

Here is an explanation of how we derive these outcomes from the four databases that are available to us:


1. Manitoba Health Insurance Registry Data - this data is used to identify the entire student population who are eligible to write the provincial exams;
   
   
2. Test Results data - this data is used to identify the provincial exam scores/marks achieved by the students who are in school and who wrote the test. The possible scores are -1 to 100%. For analysis purposes at MCHP, the scores are grouped with the following outcomes:
   
   
   • Passed: For students who wrote a test, an outcome of "passed" was assigned for a mark of 50% or higher.
     
   • Failed: For students who wrote a test, an outcome of "failed" was assigned for a mark between 0 and 49.9%.
     
     NOTES
     
     
     • Where the FinalMark = -1, this indicates that the student may have only written part of the test and for some reason (e.g., illness) was not able to complete the test. These students were assigned a mark of -1 rather than a failing mark. Few students fell into this category in completed analyses, and for most analyses these students were grouped with Absent, Exempt, Dropped, Incomplete & FinalMark = -1 (see the Outcome categories listed above).
   
   
3. Absences data - this data is used to identify the students who for one reason or another did not write the test. The reason can be determined by the value in the AbsentCode variable, grouped into one of the following categories:
   
   
   • Exempt: these are students exempted from writing a test for reasons such as: substantial modifications had to be made to the test in order for the student to write it, or the student was classified as an "English as a Second Language" student. (see AbsentCode = 60, 61, 62, 63, 69) Although the exempt do behave most like those students who took the test but scored low, most of these kids (three quarters) do graduate on time (see summary output for 1984 cohort). The exempts are not a large group of students, and in past analyses have been included with the Pass group or with the Absent/Dropped/Incomplete group; however, this distinction has not made a huge difference. Thus, it is up to individual researchers as to the group in which to include these students.
     
   • Dropped: this code indicates that the student dropped the course or the test. (see AbsentCode = 22, 27)
     
   • Incomplete: this is for students who wrote a portion of the test and the test was not completed. (see AbsentCode = 40, 41, 49, 90, 91 ,99, 301, 312) Note: this appears similar to a score of -1 (see above), however, some students do not have a final mark recorded as -1 and are found on the Absences dataset to have a value indicating "Incomplete" and vice versa (i.e. some students will have a score of -1 recorded and we cannot find them in the "Absences" dataset so we will leave the test score as -1)
     
   • Absent: this is for students who were supposed to write a test, but were absent on the day or days of the test (for example, due to illness or medical condition). (this includes all other AbsentCode values that are not included above).
   
   
4. Student Marks data - this data is used, along with linkage to the Registry data, to identify other eligible students who for reasons other than those listed above did not write the exam. Students who do not have a test score can be investigated and grouped into one of the following categories:
   
   
   • Have Credit: In searching through the "Student Marks" data by subject, it is possible to identify a few Grade 12 students who did not have a Language Arts or Math test score, but who seemed to already have credit for the course. These students likely took the course and the test while enrolled in Grades 9-11 (S1-S3), because students in these grades are eligible to write the grade 12 (S4) Math/LA test if they have the appropriate prerequisites; thus, some students have completed these courses prior to entry into grade 12. These students were included with the passed group. It is possible to search for these students in previous test years to find their actual scores, however, if we still cannot find a test score for them we will code them as "Have Credit".
     
   • Grade 2 or lower: Students who, based on their age (as determined from the Health Insurance Registry) will turn 8 years old by December 31 of the school year and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in grade 3 and thus should have written a test, but instead were enrolled in grade 2 or lower. NOTE: This is the same as "enrolled other", although we have attempted to stop using the term "enrolled other".
     
   • S3 (grade 11) or lower: Students who, based on their age (as determined from the Health Insurance Registry) will turn 17 years old by December 31 of the school year and on the assumptions that they entered school on time and were promoted to the next grade each year, should have been enrolled in grade 12 (S4) and thus should have written a test, but instead were enrolled in grade 11 (S3) or lower.
     
   • In Grade 3, No Test: Indicates the student was in Grade 3 but we could not find a test score nor any explanation from the Absences data (see above).
     
   • In S4 (grade 12), No Test: Indicates the student was in Grade 12 (S4) but we could not find a test score nor any explanation from the Absences data (see above).
     
   • Not Enrolled: Students who, based on information from the Manitoba Health Insurance Registry, were residents of Manitoba and had not completed secondary school, should have been enrolled in school but who were not enrolled in school for 1 year (from our enrollment dataset). If 2 years of data are available and the student is not enrolled both years, then the categorization changes to "Withdrawn" (see next category). Note that if a "Not Enrolled" student is enrolled in school the following year, this would not meet our "Withdrawn" criteria.
     
   • Withdrawn: Students who, based on the Manitoba Health Insurance Registry, were residents of Manitoba and who have not graduated (see high school completion / graduation glossary term) and had not been enrolled in school for any 2 consecutive years at any time after entering Grade 10 (S2). It is possible that these students transferred to a First Nations school, so a thorough analysis would also check the enrollment data for First Nations schools - if we had such data - to determine if these students transferred to a First Nations school.

MCHP Research Into Standards Tests / Achievement Tests

MCHP research into Standards Tests / Achievement Tests has investigated two perspectives; the first looking at the performance of those who wrote the tests, and the second looking at the performance of those who were eligible to take the tests (turning 17 years old by December 31 of the school year). When reviewing the performance of students who wrote the test, only "pass" and "fail" outcomes are reported. However, when looking at those who are eligible to take the tests (the entire population), and using the six outcomes measures listed above, we see quite a different story emerge from the research. The following sections identify and describe four MCHP research projects that investigate test results, and links to the detailed findings of each project are provided.

1. Brownell et al. (2004)

In the publication How Do Educational Outcomes Vary With Socioeconomic Status? Key Findings from the Manitoba Child Health Atlas 2004 by Brownell et al. (2004), they investigate the grade 12 and grade 3 Language Arts test performance by Winnipeg SES (socioeconomic status) groups from two perspectives: those who wrote the test and those who should have written the test. The two figures below illustrate the results:


• Figure 3 - Grade 12 (S4) Performance, by Winnipeg SES Group - the left side of the figure shows a graph illustrating the "pass / fail" rates for those who wrote the test, and the right side of the figure shows a graph illustrating the six outcomes developed to provide a population perspective on those who should have written the test.
  
  
• Figure 6 - Grade 3 Performance, by Winnipeg SES Group - the left side of the figure shows a graph illustrating the "pass / fail" rates for those who wrote the test, and the right side of the figure shows a graph illustrating the six outcomes developed to provide a population perspective on those who should have written the test.

2. Brownell et al. (2008)

In the publication Manitoba Child Health Atlas Update by Brownell et al. (2008), they investigate student performance on grade 12 Language Arts and Mathematics tests. The findings compare two cohorts; the first cohort includes all children born in 1984 and remaining in Manitoba in the 2001 academic year (2001/02 school year), and the second cohort includes all children born in 1988 and remaining in Manitoba in the 2005 academic year (2005/06 school year).


• For more information related to the Language Arts tests, please read section 8.3 - Grade 12 Exam Performance - this includes discussion and illustrations related to: on-time pass rates by RHA, by Winnipeg Community Area, and by Income Quintile; and test performance by RHA and by Winnipeg Community Area.
  
  
• For more information related to the Mathematics tests, please see the illustrations available in the report's Data Extras Section available on our external website - Chapter 8: Education - this includes illustrations on: on-time pass rates by RHA, by RHA District, by Winnipeg Community Area, by Winnipeg Neighbourhood Cluster, and by Income Quintiles; and test performance by RHA and by Winnipeg Community Area.

3. Martens et al. (2010)

In the publication Profile of Metis Health Status and Healthcare Utilization in Manitoba: A Population-Based Study by Martens et al. (2010), they investigate student performance in the 2005 academic year (2005/06 school year) on the grade 12 tests for Language Arts and Mathematics. The findings compare two cohorts: Metis and All Other Manitobans, both groups 17 years old by the end of December 31 of the school year who are eligible to write the tests.


• For more information related to Language Arts tests, please read section 16.3 - On-Time Pass Rates for the Grade 12 Language Arts Standards Test - this includes discussion and illustrations related to: on-time pass rates by RHA, by Metis Region, and by Winnipeg Community Area; and test performance by RHA, by Metis Region, and by Winnipeg Community Area.
  
  
• For more information related to Mathematics tests, please read section 16.4 - On-Time Pass Rates for the Grade 12 Mathematics Standards Test - this includes discussion and illustrations related to: on-time pass rates by RHA, by Metis Region and by Winnipeg Community Area; and test performance by RHA, by Metis Region, and by Winnipeg Community Area.

4. Brownell et al. (2012)

In the publication How are Manitoba's Children Doing? by Brownell et al. (2012), they investigate student performance on the tests each year over a nine year period, starting with children born in 1984 and living in Manitoba in the 2001 academic year (2001/02 school year), and ending with children born in 1992 and living in Manitoba in the 2009 academic year (2009/10 school year).


• For more information related to Language Arts tests, please read section Grade 12 Language Arts Standards Tests - this includes discussion and illustrations related to: overall test performance, on-time pass rates by aggregate regions, on-time pass rates by rural and urban income quintiles, and changes in inequities over time in rural and urban areas.
  
  
• For more information related to Mathematics tests, please read section Grade 12 Mathematics Standards Tests - this includes discussion and illustrations related to: overall test performance, on-time pass rates by aggregate regions, on-time pass rates by rural and urban income quintiles, and changes in inequities over time in rural and urban areas.

Cautions / Limitations

Some of the limitations / cautions related to this data include:


• We appear to have marks data for all students who were enrolled in First Nations schools but we do not know for sure what proportion this is of all students enrolled in First Nations schools. We estimate that it is approximately 60%.
  
  
• The education data for students in First Nations schools (formerly called band-operated schools) is incomplete and therefore is usually excluded from analysis.
  
  
• Although Standards Tests / Achievement Tests are common provincial exams written annually by Manitoba students, the test content is usually re-set each year so students may not be tested on the same things each year. Therefore, a change in overall performance may reflect a change in student skills as well as what is being measured in the tests.