An investigation into the mathematics performance of grade 6 learners in South Africa

The aim of this article is to investigate mathematics performance of intermediate phase learners. The quality of learners’ scholastic achievement in mathematics are analysed based on the empirical evidence obtained from an international survey. The method of inquiry is based on an analysis of existing statistical data concerning present-oriented problems. The effects of particular variables on mathematics learning are explored, such as male and female learners and those from less advantaged social backgrounds. Overall performance and competence levels are analysed. Although there are no outstanding contributing factors that influence mathematics achievement of grade 6 learners in South Africa conclusions are drawn that may influence school system organisation and also the quality of mathematics teaching.


Introduction
This article is a companion paper to a preceding article, Contextual factors of the mathematics learning environment of grade 6 learners in South Africa (Kotzé & Strauss, (2006), that analysed and related data based on the economic, social and cultural contexts of grade 6 learners and their families to learners' mathematics learning environment. The focus in this study is the mathematics achievement of grade 6 learners in South Africa. Data are based on an international research project undertaken by the Southern Africa Consortium for Monitoring Educational Quality, better known as SACMEQ II (SACMEQ II Research Report, n.d.).
Since mathematical literacy implies the need for a range of competencies at several levels (PISA, 2003) separate skills cannot be singled out in different mathematical contexts. The SACMEQ II assessment approach is based on the premise that different levels distinguish between different competencies. Defined achievement levels reflect explicit targets and expectations of national curricula and international standards. The lowest band of test scores represents low achievement (Andrich, 2002a;Andrich, 2002b). Levels range from simple computations based on definitions, to making connections to solve straightforward problems, to mathematical thinking requiring learners to engage in different reasoning patterns.

Aim and problem statement
The SACMEQ II structure for evaluating mathematics proficiency is based on competency and achievement levels. Scaling procedures are implemented to describe learners' mathematical numeracy levels according to increasing levels of competence and corresponding achievement levels. Levels are identified based on item response theory (IRT), where the proportion of individuals passing a test item varies as a function of the individual's ability. Thus the proportion of individuals getting a valid item correct (that is the p-value) is correlated with ability. An example would be to administer the same test item to two different groups. Suppose the p-value for a group of twelve-year olds is 0.35. Thirty-five percent of the twelve-year-olds answer the item correctly while sixty-five percent do not. Yet the same item may have a p-value of 0.55 for thirteen-year-olds. In other words, the p-value of an item is not a function of the item but a function used with a specific sample. Therefore the test item can be regarded as the independent variable, and the respondents of a specific sample as the dependent variable.
In the case where it is assumed that all items have the same discrimination power, this relationship between the probability of a correct response and ability is called the Rasch model (Andrich, 2002b: 112). This model is regarded as ideal for assessing performance with item structures that do not adhere to normal distribution of classical test theory (CTT) (Salvia & Ysseldyke, 2004: 3). The value of this approach is that test items are ordered according to levels of difficulty, that is, the greater the total score, the greater the ability implied. In addition, the difficulty structure reflects an implied continuum of achievement. Each successive task along the continuum should include elements in earlier tasks, as well as more challenging material.
In this study the level and quality of learners' scholastic achievement in mathematics are analysed based on the empirical evidence obtained from the SACMEQ II survey. Data are presented to answer major questions about the mathematics achievement of grade 6 learners in South Africa.
The following questions are formulated: • What are the mathematics achievement levels of grade 6 learners in South Africa? • What is the relationship in achievement levels between important subgroups?

Research Design
As the inquiry is concerned with the description and explanation of empirical facts, the investigation is grounded in a positivistic paradigm. The research method is based on an analysis of existing statistical data concerning present-oriented problems (Bishop et al., 2003;Romberg & Collins, 2000). The effects of particular variables on mathematics learning are investigated. Mathematics content was based on domains of broad mathematical concepts underlying mathematical thinking ( Figure 1). The main steps used for the construction of the mathematics tests are outlined in Figure 2.
The procedure for the construction of the tests commenced with a comparison of common curriculum elements. Test construction attempted to ensure that the structure was congruent with the content (domains) and behaviour (skills) by means of analysing curricula, syllabi, examinations and textbooks used in the SACMEQ II countries. After restrictions were eliminated, Rasch scaling procedures were implemented to analyse competency levels and establish relationships between competency and probability. A test blueprint was developed followed by writing trial test items. High levels of face validity and construct validity were ensured by achieving congruence between the test blueprint (that is, a framework for test con-struction) and the descriptions of increasing levels of competence. The framework was developed based on a Rasch analysis of item difficulty levels in combination with a skills audit of test items. The initial five levels for the test blue print were extended to eight levels and will be dealt with in more detail below.

Representative test items
After the skills audit new levels were identified and derived. The extra levels that were added tended to reflect deeper levels of understanding of specific mathematics competencies. In order to obtain a better background of different levels, a brief analysis of item levels is provided. Representative items from the eight levels followed by a brief motivation are provided below. Motivation Learners at this level can subtract with carrying and successfully perform arithmetic tasks that require fixed processes.

Number
Operations and number line, square roots, rounding and place value, significant figures, fractions, percentages, and ratios

Measurement
Measurements related to distance, length, area, capacity, money, and time.

Space-Data
Geometric shapes, charts (bar, pie, and line), and tables of data. Figure 1. Domains for the SACMEQ II mathematics test (SACMEQ II Research Report, n.d.)

Level 3: Basic numeracy
Example What shape are these dice?

Motivation
At this level learners know the names of threedimensional regular shaped objects and are able to link them to everyday objects.

Level 4: Beginning numeracy
Example How many 80 g small packets can you fill from a 2 kg packet? A B C D 20 25 40 160 Motivation At this level learners show the ability to combine operations. In this example the task involves a conversion of measurement units followed by division. These steps require knowledge of units and the relationship between the units.

Example
Use the price list to find the total cost of 5 litres of milk, 3 kg of tomatoes and 2 chickens.  Motivation Learners at this level are able to apply specific knowledge of mathematics in a range of areas such as number, measurement and data analysis. The item requires learners to bring to the task some external knowledge of mathematics but it involves conventional mathematics skills that are generally taught in the curriculum. In this example the learner needs to know properties of geometry figures and the notion of conservation of area and combine this with knowledge of areas. If learners possess this kind of knowledge the solution is relatively straightforward.

Example
The table represents a relation between x and y. What is the missing number in the table?
Motivation This level requires learners to be able to extract and convert information (for example, with respect to measurement units) from tables, charts, visual and symbolic presentations in order to identify and then solve multi-step problems. Motivation This level requires learners to identify the nature of an unstated mathematical problem embedded within verbal or graphic information. The learner has to translate the information given into a mathematical approach and then identify the correct mathematical strategies to obtain a solution. At this level the mathematical knowledge and skills required to solve the problem is not immediately obvious.
It is argued that the emergence of extra levels was understandable because the test blueprint had been targeted at learners, whereas the skills audit covered test items given to both teachers and learners. Unfortunately, in South Africa, teachers were not afforded the opportunity to write the test as in other SACMEQ II countries, which is a great impediment in addressing mathematical literacy in South Africa.

Analysis of test
Test scores were analysed with a software interpretation of the Rasch scaling procedures that automatically adjusted the scores to a scale with an arbitrary zero point and a standard deviation of one. The result was that many learners were assigned negative scores. As this is not an ideal situation from an educational perspective, a linear transformation assisted in assigning 500 and 100 to the mean and standard deviation of learners' scores respectively. Therefore the basic feature of the mathematics achievement Rasch model is that the programme performs a linear transformation of the test scores into a scale on which the predetermined mean is 500 and the standard deviation is 100.
By means of the Rasch analysis the ability of the learners is matched to the difficulty of the test items, which allows learners and items to be mapped on to the same scale. Learners could therefore be grouped in the same 'ability' or 'difficulty' range as the items that had similar difficulty. Sixty-three items were used to test the mathematics performance of grade 6 learners.

Presentation of data and discussion
The data analysis firstly involved a comparison of learners' overall scores in 'all' the test items. This is only a relative comparison that can be used to assess the distribution of learners' achievement, for instance among schools within provinces or among provinces. It cannot, however, be implemented to make judgments about the quality of performance in any of these instances. The overall mathematics performance for grade 6 learners in South Africa is presented in Table 1.
For South Africa the overall mean Rasch score obtained by grade 6 learners for mathematics was 486.2. This is below the pre-determined Rasch score of 500. Three provinces exceeded the mean. The standard error of sampling (SE) of the means as reflected in the table above were fairly large which implies that results should be interpreted with due caution. Every result should be interpreted in terms of the standard error of sampling (SE). In the case for South Africa in total, the performances of 95% of the population from which the sample was taken will fall in the performance range 486.2 ± 2(7.18).
For the second dimension of comparison, all the test items were organised into clusters according to levels of difficulty (discussed earlier) and the learner competencies required in answering them correctly. The percentages of grade 6 learners who reached various competence levels in mathematics are presented in Table 2.
For mathematics the modal competence reached by 44.4% of the learners was level 2 (emergent numeracy). In addition, 7.8% of the learners achieved only level 1 (beginning numeracy).
A total of only 24% of the grade 6 learners achieved competence above level 3 in mathematics and a mere 1.3% were competent at level 8. In light of the fact that grade 6 is a key transition stage between the intermediate and the senior phases of the South African school system, these figures and tendencies are extremely worrying. In a normal distribution it would have been expected that a considerably lower percentage of the  8 0.77 44.4 2.32 23.8 1.37 8.8 0.96 6.1 1.47 5.8 1.09 2.1 0.46 1.3 0.48 Table 2. Mathematics competence levels of learners learners should attain level 3 and that more learners would demonstrate higher levels of competence. Hence, comparisons between regions, between sub-groups and between particular characteristics prevalent in different regions are investigated. Three important sub-groups are identified to compare them with the mathematics achievement. Table 3 reflects the data on these sub-groups.
Gender and learner achievement reveal that the modal achievement for mathematics is level 2 (emergent numeracy). The modal achievement for boys was 44.7%, which is slightly higher than the 44.2% for girls. However, the percentages of both boys and girls achieving the highest level for mathematics are remarkably comparable. More boys than girls achieved at the lower end of the scale while more girls tend to perform better on the higher end.
The modal achievement of school location and learner achievement reveal the following: learners from both isolated/rural and small town schools are level 2 (emergent numeracy) achieved by 59.6% and 46.9% of the learners respectively. Learners from the large city schools (21.1%) achieved level 3 (basic numeracy). For isolated/rural and small town schools the percentages of learners who achieved various levels of achievement declined towards the higher end of the scale and ended with no learners from isolated/rural school and only 0.4% from small town schools achieving at level 8. There seem to be factors other than school location that influence achievement because, regardless of the location of school, the majority of learners achieved around the same level (level 2).
Socio-economic differences and learner achievement were determined and expressed in terms of the levels of home possessions, as discussed in Kotzé & Strauss (2006). Two subgroups were identified: learners in the low economic status sub-group (low SES) were those who came from homes whose possessions were half or less than the suggested list of items. Learners in the high economic status sub-group (high SES) were those who came from homes where the possessions were more than half the suggested list of items. The presentation in Table 3 indicates that for both the low and the high socioeconomic status learners the modal achievement in mathematics is level 2 (emergent numeracy), that is 54.9% for the low socio-economic status subgroup and 34.2% for the high socio-economic subgroup. A significant number of learners from low socio-economic backgrounds achieved levels from one to three whereas learners from high socioeconomic levels were fairly evenly spread across higher levels of achievement.

Findings and conclusion
Data from grade 6 learners' scores for mathematics provide the following answers to the problem statement: • The overall learner achievement was noticeably on the lower end of the acceptable limits on the SACMEQ II benchmark.  Table 3. Mastery levels for sub-groups (SACMEQ II Research Report, n.d.)

Percentage of learners reaching the mathematics achievement level
• The relationship between difficulty levels and learner competencies reveals a significant low level of general numeracy, understanding and skills. Test items focusing on higher order meta-cognitive skills were poorly answered. Variations (inequities) in learner achievement were significantly wide both within the provinces and among the provinces.
• Data addressing the achievement levels of important subgroups revealed that although overall achievement in mathematics was generally low, the girls in this study demonstrated higher numeracy competencies than the boys when higher levels are compared. It seems as if the relationship between the learners' socio-economic status and their level of achievement is fairly strong. It also appears as if there are apparently deeper problems regarding the inadequate performances of learners in general. The general findings that emerged are now discussed.
There are no outstanding contributing factors that influence mathematics achievement of grade 6 learners. Poor results are probably due to a range of factors that have financial and logistic implications. It appears that a major emphasis was given in classrooms to the mastery of basic skills. Deeper analysis of the data may also assist in identifying some of the most important factors contributing to the levels of achievement. Achievement comparisons can also provide indications about what can be accomplished, for example, it was possible for learners from some provinces to attain higher levels of achievement. This should serve as an incentive for improvement in other provinces.

The diversity of learners must be recognised.
The effect of age and gender, and the role of parental involvement and family background could be identified as possible influences on achievement. However, learner characteristics such as aptitude, perseverance, prior knowledge, affective barriers as well as attitudes and values towards mathematics could not be determined in this investigation.

Learners' economic and social contexts influence their schooling.
The correlation between the index of home possessions (indicating the home economic status) and achievement showed a considerably high correlation.
The school plays a bigger role than the home in determining mathematics achievement in a developing country like South Africa. Mathematics achievement often lags behind in disadvantaged schools. The question may be posed whether mathematics teaching is a bigger problem in disadvantaged schools than ordinary schools, or whether other contributing reasons exist. On the other hand, there may also be special factors prevalent in such schools because some of the disadvantaged schools show much higher achievement than others, in spite of difficult circumstances.

School location plays a role in achievement.
Among the provinces with lower economic levels lower scores were associated with rural schools and higher scores were associated with urban schools.
One of the limitations to this investigation is the fact that the role of the educator was not taken into consideration with regard to academic content knowledge of mathematics as a discipline. Educator subject knowledge is crucial and has been shown to be a good predictor of learner achievement. The lack of such information may jeopardise academic advancement of mathematics in South Africa.
A further limitation is that the Rasch theory as applied in this study suggests that achievement in mathematics is a one-dimensional trait, even though the items covered topics from a variety of strands in the curriculum, including the three indicated domains. In addition, measurement of the traits is based on 63 achievement items -a small number on which to base an assessment of mathematical proficiency.
Lastly, information was collected through the administration of test items presented in objective multiple-choice format. It is a cost-effective and efficient assessment technique to collecting great volumes of data from many respondents. In this case validity and reliability of data are also guaranteed to a much greater extent than with other data-gathering techniques. Nevertheless, its limitations are obvious. Information about how learners approach the solution of a given problem is more important than whether or not they are able to recognise the correct solution. Valuable conclusions regarding problem-solving abilities, creativity, values and attitudes have been lost.
The data discussed may serve as valuable sources of information against which stakeholders can compare and contrast quality education. The following observations that resulted from this study may serve as recommendations for stakeholders in assisting them in future planning: • Those who are involved in the teaching of mathematics need to find their own solutions to improving quality. By focusing on targeted interventions, specific weaknesses may be addressed. • A sound knowledge base can make a major difference to the quality of mathematics education. Initiatives often require research and/or knowledge that are specific to context and local circumstances. Those who work in the locality or region can generate such initiatives.
• Investment in sources, networks and structures designed to develop and share educational knowledge can yield significant returns by enabling schools to make much better use of limited resources. However, much can be achieved by making better use of existing resources. The investigation provided a more concrete analysis of what learners can actually do. A search for any general theory of successful education reform is difficult to accomplish and unlikely to succeed. In conclusion, it seems that incremental improvements are important: we should not be afraid of growing slowly, we should only be afraid of standing still.