Various efforts are underway to improve achievement in high-stakes examinations in school mathematics. This article reports on one such initiative which focuses on the development of quality teaching of school mathematics by embedding it within an examination-driven emphasis. A quantitative approach was used to analyse the performance of Grade 10 learners in three consecutive end-of-year school-based examinations set by the initiative. Results indicate a trend in a positive direction over the three-year period. Nevertheless, there was a discernible decrease between the first and second administration of the examinations. It is concluded that examination-driven teaching holds a promise for enhancing achievement in high-stakes school mathematics examinations if sensibly and sensitively implemented.

Underachievement in school mathematics is a concern in most countries in the world. Watson and De Geest (

By stating that ‘methods of approaching this issue range from macro-changes in policy, curriculum and assessment to institutional change, provision of extra teaching and micro-advice about inclusive teaching in classrooms’, Watson and De Geest (

The seriousness with which countries take the improvement of achievement in mathematics of learners from low socio-economic and historically disadvantaged sectors of a country’s demographic makeup is evident in current reform initiatives in school mathematics. For example, in Australia a desktop review was conducted to identify ‘gaps in current pedagogical approaches and learning resources for the teaching of mathematics to inform the Mathematics by Inquiry initiative’ (Australian Academy of Science,

One issue that had to be addressed in this initiative in Australia was linked to the teaching of socio-economic and historically disadvantaged groups. The commission given to the Australian Academy of Science by the Australian Government’s Department of Education and Training was specifically stated as ‘which pedagogical approaches have been shown to work with specific groups under-represented in advanced mathematics at senior secondary level (girls, Indigenous, disadvantaged students)?’ (Australian Academy of Science,

The popularisation of a programme of teaching adopted to enhance achievement in marginalised groups in a high-stakes mathematics examinations is vividly portrayed in the 1988 film

In Southern Africa there is a paucity of research related to efforts to enhance the achievement in mathematics in high-stakes examinations of students from low socio-economic environments. This does not imply that such projects and efforts do not exist. Many projects report on the impact of their initiatives to improve achievement in high-stakes school mathematics (see Reddy et al.,

This article reports on a classroom-based project to improve achievement in high-stakes examinations. The mentioned underpinnings of the project and results of learner achievement over three years are presented.

The project, the Local Evidence-Driven Improvement of Mathematics Teaching and Learning Initiative, has as part of its aims the increase in the number of learners taking Mathematics as an examination subject for the NSC examination, an increase in the pass rates and an improvement in the quality of the passes in the participating schools. The project developed an intentional teaching model (Julie,

Generally the project operates by offering workshops and institutes attended by participating teachers. Workshops are conducted after school and are usually of approximately two hours duration. Two to three workshops are held per term for the first three terms of the school year.

Institutes are extended and residential gatherings held normally from a Friday afternoon to Sunday lunchtime. Two institutes per year were held for the three years, 2012 to 2014. Overall the teachers were engaged in 64 hours of Continuing Professional Development activities for the three years for which results in the high-stakes school-based mathematics examination were tracked.

The content of the professional development activities focused on pedagogical issues such as analysis of lesson excerpts, discussions around dilemmas teachers face in their teaching, searching for ways to address these dilemmas and the design of lessons. Another feature of the content of the Continuing Professional Development is that in most of the meetings teachers worked on mathematical problems with the aim of developing their mathematicalness – flexible ways of dealing with mathematics. The mathematics of the tasks is explored and discussed. The ways teachers worked with the tasks and the facilitation are then discussed in relation to how teachers can engage learners in doing mathematics.

An example of a dilemma teachers face that was raised by teachers is that of learners not really doing homework. The purposes of homework were then discussed. One of the purposes offered was consolidation of completed work. This was connected in the discussions to the issue of forgetting. The outcome of the deliberations around the issue led to the development of a strategy for which the term ‘spiral revision’ was coined. Basically this consists of learners being presented with two to three exercises of previously covered work which they have to complete in class. This has to be done 3–4 periods per week in about 7–10 minutes before dealing with the lesson for the day. ‘Spiral revision’ is the project’s version of ‘distributed practice’ (see, for example, Johnson & Smith,

Other pedagogical aspects engaged with during the workshops and institutes were clarity to both teachers and learners of the intentions or goals of a lesson, the use of feedback and provision of opportunities to work with different problem types. These are also aspects which Hattie’s (

The objective of the project, as stated above, is the improvement of achievement in high-stakes examinations. High-stakes examinations, which are discussed below, thus played a structuring role within which the above pedagogical aspects were dealt with as shown by using the example of quadratic inequalities. This brought the issue of examination-driven teaching into the picture. The research question being reported on in this article is:

As is evident from the research question, the notion of a high-stakes examination is one of the constructs of importance in this article. Various notions of high-stakes examinations exist. These are normally linked to the purposes of the examinations.

Howie (

In this article, a high-stakes examination is one that has direct consequences, positive or negative, for the examinees. Particularly for Grade 10 learners, the school-based end-of-year mathematics examination has consequences such as promotion to Grade 11 or not and the right to continue taking Mathematics as an examination subject for the NSC examination. Non-continuation with Mathematics up to Grade 12 is a major issue. For the research reported here, of the 403 learners in the five participating schools who wrote the 2012 project-designed examination in Grade 10 only 280 proceeded to write the 2014 NSC Mathematics. This is an instance of the decrease in taking Mathematics from Grade 10 to 12 in a cohort of learners. Adler and Pillay (

For school-based end-of-year summative assessments the focus is on learners. As indicated above, success or not in these examinations has consequences for them. At a very basic level success (and the level of the success) or failure on these assessments decides whether or not learners will be able to proceed from Grade 10 onwards to be awarded a certificate of worth that they can use after their completion of schooling. The NSC in South Africa is such a certificate. The consequence of having at least this certificate is that it greatly enhances the chances of school-leavers to obtain employment and access to further studies. In this regard Statistics South Africa (

In this article, high-stakes examinations are viewed as those that allow learners to progress from one grade to another and in particular to the exit level, Grade 12. As mentioned, the high-stakes examination is the school-based summative examination learners write at the end of the school year, Grade 10 for the purposes of this study. This examination is normally internally constructed and marked. To ensure quality, the head of the mathematics department of the school normally moderates both the construction and the marking of the examination. Further quality assurance and consistency across schools are ensured through a process of external moderation by the mathematics curriculum advisors of the Department of Education (Jacobs, Mhakure, Fray, Holtman, & Julie,

According to Julie (

Proponents of examination-driven teaching, on the other hand, draw attention to its advantages for improving achievement outcomes. These include clarity of instructional goals, cost-effectiveness, motivation and examination assistance for learners by providing clarity on the kinds of problems they can expect to encounter in a high-stakes examinations and the feedback that examinations-driven teaching provides to teachers for instructional decision-making (Popham,

Notwithstanding the debates about examination-driven teaching, there are considered positions about the structuring roles examinations exert on instructional practices. One such position is that examinations play a major role in the constitution of legitimate and valued school mathematics knowledge. Bishop, Hart, Lerman and Nunes (

The recognition of the structuring effects of the examined curriculum provides a strong argument that in order for teaching to comply with meaningful learning, examinations must be changed (Burkhardt & Pollak,

In South Africa there is currently an emergence of the use of large-scale systemic assessments to structure continuing professional development initiatives for mathematics teachers. Shalem, Sapire, and Huntley (

In the project of interest in this article, examinations are used in a similar way to those described in the foregoing paragraph. Learners’ responses in examinations are used to reflect on difficulties learners display in examinations, design of activities to address such difficulties and backward mapping from the high-stakes NSC Mathematics examination to provide focus for teaching in lower grades. For example, in the Curriculum and Assessment Policy Statement document (Department of Basic Education, ^{2}, should be dealt with but the solution of quadratic inequalities is not. This is understandable given the restriction, as stated in the aforementioned sentence, for graphs of quadratic functions in Grade 10. In the project, learners are exposed to solving quadratic inequalities under the topics dealing with the real number system. The graphs of quadratic functions, without specifying the defining expression, are given and learners have to solve quadratic inequalities with a generically specified defining expression as given

Task on quadratic inequalities when dealing with the real number system.

Tasks of the nature given in

Examination item on the quadratic function (the quadratic inequality item is 6.1.4).

It is the contention of the project that if learners start engaging with questions that they will encounter in the ultimate NSC Mathematics examination as early as Grade 10 then they will have high levels of fluency to deal with the cognate problems in the NSC Mathematics examination.

In this section we presented an indication of how examination-driven teaching is conceived and implemented as underpinning in the project. The next section discusses the research design.

A quantitative design was adopted in this study because learners’ scores are used to describe the phenomenon being investigated. The study is a trend study where results of the same phenomenon are tracked over a period. It is different from a tracer study which follows the results of the same cohort over a period. The trend of the overall mathematics scores in the end-of-year summative school-based Grade 10 examination over three years – 2012 to 2014 – was thus investigated.

Trend studies are appropriate in situations of curriculum stability. The Trends in International Mathematics and Science Study (TIMSS) project does trend studies (Martin, Mullis, & Chrostowski,

The sample was an opportunistic sample of five schools whose teachers were involved in the Continuing Professional Development initiative. Ten schools were initially involved in the project. After the first year of implementation, the participation of one of the schools was terminated due to unsatisfactory participation in project activities. Not all the schools wrote the project-set common examinations for the reporting period. The reasons for this are: (1) the timing and availability of the common question papers in that some schools had their examination timetables ready before common agreed examination dates could be negotiated, (2) the standard of question papers was deemed too high in terms of their cognitive demand according the judgement of the teachers of their learners’ cognitive levels and (3) the Grade 10 learners of one school were not available in 2013 and 2014 because they had gone to another school following a prior arrangement. It needs to be borne in mind that teachers’ participation is voluntary and so the decision to write the common examination or not rests with the schools. Voluntary participation and the right to withdraw from research activities or part of it are important ethical principles in a research project involving human participants. This was made clear to teachers at the start of the Continuing Professional Development initiative. This resulted in five schools who wrote the project common examination for the three years.

A possible threat emanating from working with samples over different years is that the characteristics of the cohorts of participants might change and have a confounding effect. Major confounders are normally race, gender, age, class size and school type. Regarding gender, although the names of the learners appeared on the scripts, the difficulty of using names as a signifier for gender is highly problematic. ‘Cyril’, for example, can either be male or female. Furthermore, in a school-based examination learners do not indicate their gender. Gender dimensions were thus not included. Other confounding factors that might be linked to the contexts of the schools might have changed. However, the nature of schools in South Africa is such that the enrolments are reasonably stable with regard to socio-economic status and demographic composition. Our own observations during classroom support visits revealed no observable change along these lines.

In line with common practice for school-based end-of-year summative Mathematics examinations, the examinations are governed by the assessment guidelines as described in the Curriculum and Assessment Policy Statement. The Curriculum and Assessment Policy Statement document describes modalities such as the topics and their weightings to be covered and percentage of marks to be allocated to the different levels of cognitive demand. The school-based end-of-year summative Mathematics examination comprises two papers of 2 hours duration each. The first paper deals with the topics (their weightings given between brackets): algebra and equations (and inequalities) (30 ± 3), patterns and sequences (15 ± 3), finance and growth (10 ± 3), functions and graphs (30 ± 3) and probability (15 ± 3). The topics dealt with in the second paper are: statistics (15 ± 3), analytical geometry (15 ± 3), trigonometry (40 ± 3) and Euclidean geometry and measurement (30 ± 3). The examinations adhered to these guidelines and were thus similar in kind and degree.

To protect anonymity the schools are named A, B, C, D and E.

Number of learners per school, per year and per paper.

School | 2012 |
2013 |
2014 |
|||
---|---|---|---|---|---|---|

Paper 1 | Paper 2 | Paper 1 | Paper 2 | Paper 1 | Paper 2 | |

A | 64 | 64 | 124 | 124 | 91 | 91 |

B | 94 | 93 | 63 | 63 | 90 | 90 |

C | 68 | 68 | 36 | 35 | 22 | 22 |

D | 78 | 78 | 42 | 39 | 109 | 108 |

E | 101 | 100 | 116 | 113 | 95 | 95 |

It can be observed from

In this study, the scores for learners who missed a paper were treated as missing data. Thus the total number of learners was taken as 403 for 2012, 381 for 2013 and 406 for 2014.

The presented results are thus not representative of all the schools participating in the Local Evidence-Driven Improvement of Mathematics Teaching and Learning Initiative project or of Grade 10s in the Western Cape province or in South Africa. Therefore, to generalise about the outcomes for the entire province or for the country requires careful consideration if the results are to be more broadly applied. It also needs to be borne in mind that there are many interventions, for which information was not gathered for the participating schools, addressing the low performance of learners in school mathematics. Thus, the results may be confounded by influences of such interventions. However, it is well known that many interventions at Grade 10 level focus on selected learners with potential and short-term teacher initiatives focus on the enhancement of subject matter knowledge. The focus of the underlying project as referred to above was on the development of quality teaching.

The data were the scores learners obtained in the end-of-year Grade 10 Mathematics examinations. These scores comprise 75% of the total mark of 200 that is awarded for Mathematics. The other 25% is compiled from tasks, tests and the mid-year examination.

Participating mathematics educators, mathematicians, mathematics teachers and mathematics curriculum advisors set the examinations. The mathematics educators and the mathematics curriculum advisors firstly designed draft items. These were discussed with teachers at workshops to ensure that there was fairness with regard to the topics that were covered in their teaching. Upon reaching consensus, the examination papers and the memoranda of marking were moderated by the participating mathematicians.

The project staff designed the final versions of the examination papers and electronic versions were dispatched to schools for them to put in a format as required by the schools. For example, most schools follow the format where the cover page of their examination papers must have the school’s emblem on it.

In order to prevent leakage of the examination papers, the school management teams were approached to timetable the examinations for the same date and time. The five schools agreed to this request. As is normal for school-based end-of-year examinations the responsible mathematics teachers of the schools marked the scripts. Except for two schools, the same teacher taught Grade 10 for the three years. For the one school where this was not the case, the school uses the strategy of a teacher ‘taking the learners through’ from Grade 10 to 12. This school had two teachers involved and they both attended all the project activities. The other school changed the teacher responsible for teaching Mathematics in Grade 10 in 2012 due to the responsible teacher being on maternity leave for the first half of 2013. The other teacher taught Grade 10 for 2013 and 2014 and also attended all project activities.

To ensure consistency of marking across the five schools, a

The marked scripts were collected from the schools once all the administrative procedures that schools are required to do were completed. The score for each item – the sub-sections of a question – for each learner from a school was captured. Therefore, the only data recorded were the scores as reflected on the scripts of the learners.

After the collection of the common examination scripts, the data were checked, cleaned and coded as described in Okitowamba (

The university’s research ethics committee cleared the project of which this particular study is a part with the ethics registration number 11/9/33. The project was also approved by the Western Cape Education Department through a memorandum of understanding between the university and the Western Cape Education Department. In order to maintain anonymity, the names of learners were not used and not recorded in the data files. The scripts were assigned numbers for purposes of checking during the data cleaning phase. The names of schools were also anonymised as indicated in

There is a vast body of literature in which Rasch measurement theory is broadly explained, from its origin to its applications (Andrich,

Similar to the study reported here, Rasch measurement was used to compare different cohorts of students in Australia and to detect improvement in students’ mathematics achievement in lower secondary schools over time (Afrassa & Keeves,

The software WINSTEPS 3.4.1 (Linacre,

Concerning reliability, Rasch measurement provides person reliability index and item reliability indexes. These indices were specifically calculated from the raw data for this article and are presented in

Reliability index by year.

Reliability index | 2012 | 2013 | 2014 |
---|---|---|---|

Item reliability | 0.98 | 0.96 | 0.98 |

Person reliability | 0.83 | 0.87 | 0.86 |

The acceptable range for reliability coefficients of an instrument is that it should be greater than or equal to 0.70 (McMillan & Schumacher,

Regarding validity, content validity was assured through the construction of the test by the set of ‘experts’ – the teachers, mathematics educators, mathematicians and the mathematics subject advisor. The examinations were administered under normal conditions for examinations during the end-of-year examination period and followed the same processes and procedures for such examinations. There were no deviations from the way end-of-year examinations are conducted and administered. This implies that the ecological validity – ’the degree to which an assessment of events, activities, participation, or environments reflects everyday life expectations’ (Crist,

In the Rasch analysis

construct validity focuses on the idea that the recorded performances are reflections of a single underlying construct [

For a test to satisfy unidimensionality only the items within the range 0.5 and 1.5 are deemed productive for measurement (Linacre,

Statistical significance between the scores of 2012 and 2013, 2013 and 2014 and 2012 and 2014 was determined using

‘Effect sizes’ were calculated to assess the effectiveness of the intervention. According to Coe (

‘Effect size’ is simply a way of quantifying the size of the difference between two groups [

Different formulae exist for determining effect size (see, for example, Cohen,

where

_{1} = standard deviation of the mean of group one

_{2} = standard deviation of the mean of group two

Since this study mirrors the one by Afrassa and Keeves (

the size of effect is trivial | |

0.20 ≤ |
the size of effect is small |

0.50 ≤ |
the size of effect is medium |

the effect size is large |

There are other interpretations for acceptability of effects sizes. Hattie (

The results of the analysis are presented in this section. They are given for three periods: 2012 to 2013, 2013 to 2014 and 2012 to 2014.

Mean scores of the five schools for the three cohorts.

Year | Mean | |||
---|---|---|---|---|

2012 | 405 | 26.14 | 13.483 | 0.670 |

2013 | 381 | 22.32 | 13.276 | 0.680 |

2014 | 407 | 35.42 | 16.478 | 0.817 |

Period | Sig. (two-tailed) | 95% confidence interval of the difference |
|||||
---|---|---|---|---|---|---|---|

Lower | Upper | ||||||

2012–2013 | 4.000 | 784 | 0.00 |
3.821 | 0.955 | 1.946 | 5.696 |

2013–2014 | −12.243 | 786 | 0.00 |
−13.105 | 1.070 | −15.206 | −11.004 |

, significant for

The effect sizes for different periods.

Period | Effect size |
---|---|

2012–2013 | −0.29 |

2013–2014 | 0.88 |

2012–2014 | 0.62 |

Trend of learners’ mathematics performance over time.

The calculated effect sizes are given in

In light of the study outcomes reported above, there are positive indicators that the trend of mathematics achievement in school-based end-of-year summative examinations moves in a positive direction over time when an examination-driven teaching strategy is employed. However, this improvement is not necessarily immediate and the nature of examination-driven teaching must be carefully considered and crafted to counter a minimalist view of teaching to the test. Examination-driven teaching is definitely not the only contributor since enhanced achievement in high-stakes examinations is closely related to socio-economic status and ultimately the desirable achievement results that are sought will only materialise if socio-economic inequalities are substantially reduced.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

O.O. conducted the research, did the statistical analysis and wrote the draft of the manuscript. C.J. was the project leader, conceptualised the project, assisted with the data collection and analysis and contributed to the writing of the manuscript. M.M. contributed towards the discussion and conclusions of the research and did final editing.

This research is supported by the National Research Foundation under grant number 77941. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views the National Research Foundation of South Africa.