Taxonomy matters : Cognitive levels and types of mathematical activities

We argue that there are intrinsic difficulties in using the Subject Assessment Guidelines for Mathematics (SAGM) for evaluating the standard of a South African matriculation mathematics examination and for determining how it aligns with the curriculum. Our argument has two prongs. First, the SAGM taxonomy conflates cognitive level with type of mathematical activity; we contend that such a conflation leads to problems in assessing the complexity of the examination item. Second, the SAGM taxonomy provides no space for important mathematical activities such as justification and conjecturing despite the promotion of these activities in key curriculum documents (Department of Education, 2003). This absence may obscure weak alignment of the examination and the curriculum. These arguments are illustrated through examples taken from the 2008 Department of Education (DOE) and Independent Examination Board (IEB) Grade 12 Mathematics examination papers. We also examine other well‐known taxonomies in order to discern those aspects of a taxonomy which may be useful for evaluation and alignment. We conclude with the construction of a matrix that could provide an appropriate taxonomy for the South African matriculation Mathematics examination. It has activities on one axis and levels of complexity on the other.

Towards the end of the paper we theorise a desirable matrix that could provide an appropriate taxonomy for assessment in the NCS.Such a taxonomy would have mathematical practices advocated by the NCS on one axis and different levels of complexity on the other axis.

Background
At the end of 2008, South African Grade 12 school learners wrote the DOE National Senior Certificate (NSC) examinations for the first time.These examinations were based on the new NCS.The DOE Mathematics examination was written by about 300 000 learners and provoked considerable public interest in terms of its standard, partly because it was a new form of examination.Up until the end of 2007, students taking Mathematics as a school-leaving subject studied Higher Grade Mathematics or Standard Grade Mathematics.Higher Grade Mathematics was of a higher academic standard than Standard Grade Mathematics and it was generally considered a necessary preparation for further study in engineering or the sciences at tertiary level.Under the new NCS system, Higher and Standard Grade Mathematics have been merged into one subject, Mathematics.The concern of civil society and the tertiary education sector was to what extent this lowered the standard of Higher Grade Mathematics and what mark could be considered as acceptable for entry into various university courses.
Two examination boards, DOE and IEB, exist in South Africa.The majority of schools in South Africa are public institutions and mostly write DOE examinations; private schools write IEB examinations.
The standard of the 2008 Mathematics examination, particularly the DOE examination, has been the subject of a number of comments (Concerned Mathematics Educators, 2009;Dibetle, 2009;Maree, 2009) and reviews (Ministerial Panel, 2009;Umalusi, 2009) and is not the focus of this paper.However these reviews rely heavily on the SAGM taxonomy (DOE, 2008a).The application of this taxonomy to the examination, together with our exploration of alternative taxonomies, leads us to address the issue: How does the use of different taxonomies enable us to assess both the standard of a mathematics examination and the spread of different types of mathematical activities in that examination?

Different taxonomies
Within mathematics, a number of different taxonomies have been developed for various purposes.These purposes may be related to the cognitive levels and/or types of mathematical activities and tasks in an assessment (DOE, 2008a, National Assessment Governing Body, 2008), classroom tasks (Stein et al., 2000) or alignment of curriculum with assessment (Porter, 2002) 1 .Although the purposes of some of these taxonomies are not specifically assessment, we believe that aspects of these different taxonomies suggest useful views of the relationship between taxonomies and assessment.We use them to highlight certain arguments about what features of taxonomy may be useful for setting and evaluating items in the NSC examinations.
The SAGM taxonomy is based on the 1999 TIMSS mathematics survey (DOE, 2008a).Clearly we wish to examine this taxonomy since this is the official taxonomy which informs the setting of the Grade 12 DOE Mathematics paper.The SAGM taxonomy uses categories of knowledge, routine procedures, complex procedures and problem-solving, each with their own descriptors.It suggests the approximate proportion of the examination paper which should be allocated to each category.
The Porter (2002) taxonomy has five categories: memorisation, perform procedures, communicate understanding, solve non-routine problems and conjecture, generalise and prove.Porter's taxonomy is widely used and a respected tool in the USA (Hess, 2006, Porter, 2002) and some of its categories refer to particularly important mathematical practices.For example, categories such as "conjecture, prove, and generalise" are relevant to the NCS's vision which states: "competence in mathematical process skills such as investigating, generalising and proving is more important than the acquisition of content knowledge for its own sake" (DOE, 2003, p. 9).Another important category of Porter's scheme not evident in the SAGM taxonomy but again contained in the NCS's critical outcomes is the need for learners to be able to communicate mathematically (DOE, 2003, p. 2).Porter's category "demonstrate understanding of mathematical ideas" refers to a learner's ability to communicate or explain mathematical ideas and so addresses this critical outcome.Stein et al. (2000) designed a taxonomy consisting of four categories, namely, memorisation, procedures without connections, procedures with connections and doing mathematics.Although this taxonomy's primary purpose is for analysing classroom rather than assessment tasks, we believe it illuminates important distinctions in types of mathematical tasks.In particular we found the distinction they made between the categories "procedures with connections" and "procedures without connections" particularly useful."Procedures without connections" refer broadly to "tasks that require students to perform a memorised procedure in a routine manner" (Stein et al., 2009, p. 1)."Procedures with connections" refer broadly to "tasks that demand engagement with concepts and that stimulate students to make powerful connections to meaning or relevant mathematical ideas" (ibid.).SAGM (DOE, 2008a), Stein et al. (2000) and Porter (2002) all use some form of mathematical practice or activity as their taxonomical categories, for example, routine procedures, memorisation, nonroutine problems, procedures with connections, and conjecture, generalise and prove.Furthermore, they merge these practices with cognitive levels.As we later demonstrate, this conflation of the type of the thinking (as described by the practice, for example, doing procedures or proving and so on) with level of thinking (as described by the complexity of the item) makes it very difficult to use these taxonomies as tools for measuring the standard of an examination.We will show that in order to have an effective taxonomy which can be used to measure both type and level of thinking, these two attributes need to be separated.
In contrast, NAEP (NAGB, 2008) focuses on complexity levels of items only (not mathematical practices).NAEP (NAGB, 2008, p. 18) describe these levels as the "specific level of thinking" which the item is designed to measure.Thus mathematical complexity level refers to an attribute of a task which measures the level of thinking required to complete the task.The NAEP taxonomy distinguishes between three levels of complexity: low, moderate and high.Low complexity items require students to recall or recognise concepts and/or to perform routine procedures.Moderate complexity refers to items for which the method of solution is not directly given; the learner needs to decide on how to approach the problem.More flexibility of thinking is required compared to the low complexity category.Items with high complexity require that students use reasoning, planning, analysis, judgment, and creative thought.In addition they may be expected to generalise or to justify mathematical statements or construct mathematical arguments.
Placement of a task in an appropriate category using any of the above taxonomies can be tricky (Stein et al., 2000).Besides other considerations, the level of thinking required by a task depends on the familiarity of a learner with the mathematical content of the task (NAGB, 2008).For instance, learners must have encountered a mathematical procedure many times in order to achieve the "routine procedure" cognitive demand level.Those learners who are not familiar or lack experience with the tasks cannot successfully achieve this cognitive demand level.Realisation that demands of a task in part depends on the experience of learners necessitates that due consideration be given to learner age and grade level when deciding the cognitive demand level of a task.Furthermore, and as we demonstrate below, conflation of cognitive level with type of thinking required, is problematic.

Applying the taxonomies to the NSC examinations
We explored the four different taxonomies by applying them to each item in 2008 DOE and IEB Mathematics Examinations papers and, when necessary, reaching consensus on a categorisation.From this taxonomical analysis, we chose exemplars (presented below) to illustrate our arguments.

Do the taxonomies provide a useful way of evaluating the standard of the examinations?
The Stein, SAGM and Porter taxonomies purport to reflect the cognitive demand of tasks.Stein et al (2009, p. 1) define cognitive demand as "the kind and level of thinking required of students in order to successfully engage with and solve a task".SAGM and Porter assume a similar conflation of kind and level of activity.In Stein's definition "level" would seem to imply that we can take cognitive demand as an indicator of difficulty, but the fact that it is coupled with "the kind of thinking" gives pause for thought.Indeed our experiences of applying the different taxonomies to analysing the paper suggest that the conflation of the notions of "levels" and "kinds" of thinking makes this chain of deduction less straightforward.
We illustrate the difficulty of applying the taxonomies when level and kind of thinking are conflated, through an example which focuses on the Stein category "procedures with connections".Question 5.3 (DOE, 2008b) states (the learners have been given that 8 and have drawn this graph in question 5.1):

5.3
The graph of f is shifted 2 units to the LEFT.Write down the equation of the new graph.
The fact that a learner has to make connections between different representations led us to classify this problem as a "procedure with connections" task in the Stein taxonomy (a high level cognitive demand category).It is, of course, possible to do this task by recourse to a learnt algorithm (i.e. if a graph shifts 2 units to the left, replace x with x + 2).However as we cannot determine how a learner will do the task we have to rely on the descriptors of the category, which suggest that tasks that involve links between different representations are classified as "procedures with connections".The fact that shifting graphs and its effect on the algebraic representation for a function is a stated outcome in the curriculum and that this kind of problem would thus be very familiar to learners, led us to classify this question as a routine procedure in the SAGM taxonomy (a low level cognitive demand category).What we see here is a focus on two different aspects of the cognitive demand of the task.Noting the connections between different representations that needs to be made seems to prioritise the "kind of thinking" that is needed.Noting that the task is familiar seems to prioritise the "level of thinking" that is needed.
In a related fashion, DOE (2008a) refers to the SAGM taxonomy of categories (memorisation, routine procedures, complex procedures and problem solving) as both taxonomical categories (p.12) and cognitive levels (p.13).Again we suggest that the conflation of kind and level of thinking in SAGM taxonomy makes it difficult to use the taxonomy to evaluate an examination and that it will probably also make it difficult to use the taxonomy to construct an examination.We illustrate our arguments using two sets of examples drawn from IEB and DOE Mathematics Paper 1 (IEB, 2008a, DOE, 2008b).Each example shows how different items placed in the same SAGM taxonomical category may have very different complexity levels.
Question 7(b) (IEB, 2008a) and Question 1.3 (DOE, 2008b) were both classified as "solving problems" in the SAGM taxonomy and "non-routine problems" in the Porter taxonomy.
(2) Hence determine the value of x.We suggest that Question 7(b) is of high complexity because its notation is unfamiliar; its method of solution is not direct and the learner needs to treat 6 6 6 6 ...

   
as an object rather than a process.That is, to solve the problem, the learner needs to start by squaring both sides of the equation, 6 6 6 6 ... x      .This leads to the equation 2 6 6 6 6 6 ...
In contrast we suggest that Question 1.3 is of moderate complexity.The expression 2 4 2 x x   should be familiar to many students.Complexity lies in the need for the student to 'see' that factorisation and cancellation will transform the expression into a simple form, 2, 2 x x   .To solve the problem, the learner just needs to substitute 999 999 999 999 into 2 x  to get 1 000 000 000 001.
These two examples illustrate how the Porter and SAGM taxonomies may obscure subtle differences in complexity between these two questions.In this example, Stein's categories of "doing maths" and "procedures with connections" do distinguish between the different nature of these two questions.Question 7(b) is "doing maths" since it requires non-algorithmic thinking; in addition it requires that the learner understands the nature of certain mathematical concepts and processes (and the interplay between the process and the concept).Question 1.3 is "procedures with connections": its execution uses procedures and in so doing highlights the relationships between factorisation, canceling and substitution.We use one last example to further illustrate how taxonomies which conflate complexity and type of mathematical activity may obscure differences between levels of thinking in a particular item.We consider Question 7(c) and Question 4(c)(2) (IEB, 2008a).According to SAGM taxonomy both questions are "complex procedures".However we suggest that this classification obscures the different levels of thinking (complexity) required.
Question 7 (c) reads: In order to solve this problem, the learner needs to deal with each expression in turn.Computing 1 f x       requires an understanding of how to work with functional notation; this understanding may be purely procedural.That is, to obtain similarly requires procedural knowledge (substitute 3x for f(x)).Finding the inverse function similarly is procedural.Ultimately the learner needs to add these expressions together (a computational task).Given the focus on the use of procedures to simplify several expressions, we regard this task as having moderate complexity.
In contrast, Question 4(c)(2) below has high complexity: (1) Find the equation of the tangent to the curve at the point where x = 1.
(2) Find the x-coordinate of another point on the curve at which the tangent is parallel to the one found in (1).
In order to solve Question 4(c)(2), the learner has to first solve Question 4(c)(1), a routine procedure of moderate complexity.This yields the equation of the tangent as y = 4x + 5. To complete Question 4(c)(2), the learner needs to first realise that the 'new' tangent has a slope of 4 (the slope of the 'old' tangent).This requires a conceptual understanding of what it means for one line to be parallel to another.The learner must then set the derivative of the original function equal to 4, i.e. 3x 2 2x  1 = 4, a complex procedure requiring non standard thinking.The learner then needs to solve this equation to find x = 1 (already given in part 4(c)(1)) or This example illustrates how the SAGM taxonomy is unable to distinguish between a "complex procedure" with high complexity and a "complex procedure" with moderate complexity.Similarly, using the Stein taxonomy both items would be classified as "procedures with connections": Question 7(c) illustrates the subtle relationship between notations and the relevant mathematical objects; Question 4(c)(2) focuses on the conceptual connection between the notion of slope and derivative.With regard to Porter, this example illustrates the difficulty of using this taxonomy.Both Question 4(c)(2) and Question 7(c) require the use of non-routine procedures.So do we classify either of these as "perform procedure" or is it a non-routine problem?Arguments may be made for either classification.
Application of the NAEP taxonomy (NAGB, 2008) on its own is more reliable as a yardstick for measuring the standard of the examination.However it does not distinguish between different types of mathematical practices.For example, 'easy' examination tasks, such as conjecturing a simple pattern, is put in the same category as a familiar and routine procedure, no matter the different nature of the activities.

Do the taxonomies provide a useful way of distinguishing between different types of mathematical practices in an examination?
Porter's taxonomy is attractive in that, unlike Stein or SAGM, it has specific categories for key mathematical practices such as communicating, understanding, conjecturing, generalising and solving non-routine problems.However the conflation of type and level of thinking again confounds the categorisation and leads to questions with very different levels of complexity being placed in the same category.We illustrate through two 'comparable' items, namely Question 8(b)(3) (IEB, 2008b) and Question 1.3 (DOE, 2008c).See overleaf.Using Porter's taxonomy, we classify both Question 1.3 and Question 8(b)(3) in the "conjecture, generalise, prove" category.Question 1.3 can be successfully completed by a straightforward application of the distance formula and the converse of Pythagoras' theorem (low complexity in the NAEP taxonomy).However the route to the proof in Question 8(b)(3) is not straightforward and to construct an entirely coherent and complete proof requires considerable insight (high complexity in the NAEP taxonomy).Porter's category "conjecture, generalise, prove" thus encompasses both low and high complexity items.
Actually, Porter's taxonomy is designed to measure the degree of alignment between curriculum, instruction and assessment, not to reflect a complexity rating.He uses a two dimensional matrix that combines topics and cognitive demand as a tool to describe the content of an intended curriculum, an assessment and of instructional practice separately.He then uses the matrices to calculate the degree of alignment between any two of these (e.g. between the intended curriculum and the assessment).Thus the Although the above broad brushstrokes analysis is not how Porter intended his taxonomy to be used, it illustrates why it is important to distinguish types of activity, such as "generalise, conjecture or prove" from level of complexity.
The notion of using alignment as the criteria for assessing the fairness and spread of an examination is an appealing one.However the mechanisms by which to measure it and achieve it are anything but obvious.We could apply Porter's taxonomy.However his taxonomy has been applied chiefly in contexts where some version of the American NCTM (National Council of Teachers of Mathematics) standards is being implemented and NCTM standards have different priorities compared to the NCS vision.Furthermore, as we demonstrated above, his taxonomy is not useful for answering questions about the standard of an examination.

A new taxonomy
In order to measure both the standard of an examination and the alignment of the examination with the curriculum, we need a tool which distinguishes between key mathematical practices as well as between complexity levels of items.The SAGM taxonomy does not fulfil requirements of such a tool.For example, tasks involving either reasoning or application of mathematics to real-world problems (activities prioritised in the NCS for Mathematics) would both be classified as "solving problems".Similarly, SAGM has no specific space for key mathematical activities such as justification, conjecturing, and communicating mathematical ideas despite these activities being regarded as important components of the NSC curriculum.
Building on the work of Graven (2002) and Parker (2006), Bowie (2009) has argued that four orientations towards mathematics are implicit in the NCS for Mathematics.Bowie makes these orientations explicit as follows: First, Mathematics is important for critical democratic citizenship.It empowers learners to critique mathematical applications in various social, political and economic contexts.Mathematics is part of broader society and is important for all learners.
Second, Mathematics is relevant and practical.It has utilitarian value and can be applied to many aspects of everyday life and in other disciplines.Mathematics modelling is an important tool that can be used in the exploration of contexts.
Third, Mathematics is created and organised using particular processes.These include investigating, conjecturing, generalising, justifying, proving, axiomatising and defining contexts in which concepts are constructed.
Fourth, Mathematics is a body of knowledge and learners at school need to study a subset of it.This knowledge includes specific algorithms, facts, conventions, notations and forms of representation.Bowie (2009) argues that, although the orientation towards mathematics as a body of knowledge predominates, the orientations towards mathematics as relevant and practical and the orientation towards mathematical practices receive particular attention.We suggest that a taxonomy that highlights these aspects is important both because it reflects the implicit NCS and because it is in line with international taxonomies, such as Stein et al. (2009) and Porter (2002).Given the problems associated with using any of SAGM, Stein et al. or Porter, we suggest that, for purposes of setting an examination paper, a taxonomy which distinguishes between low, medium and high complexity on one scale, and categories which derive from the different orientations of mathematics explicated by Bowie, may be more useful.This can be constructed as shown in Table 1.
An elaboration of the above taxonomy is beyond the scope and space constraints of this paper and will be the subject of a future paper.Its presence is to signal the possibility of distinguishing complexity (or difficulty level) from the type of mathematical activity in a taxonomy and of simultaneously using this taxonomy to align an examination with the intended curriculum.

Table 1 :
A new taxonomy