Third International Mathematics and Science Study-Repeat (TIMSS-R): What has changed in South African pupils’ performance in mathematics between 1995-1998?
SARAH J. HOWIE
Tel: +27 12 302 2481
1. WHAT IS TIMSS-R?
The Human Sciences Research Council conducted TIMSS, under the auspices of the International Association for the Evaluation of Educational Achievement, among 15 000 South African students from more than 400 primary and secondary schools. Of the 63 countries that started the study, only 41 completed it. South Africa was the only country in Africa to do so. South Africa’s results from TIMSS 1995 can be found in Howie (1997) and in Howie and Hughes (1998).
In 1998, TIMSS was repeated (and is designated as TIMSS-Repeat or TIMSS-R), with tests and questionnaires administered in 38 countries. TIMSS-R was managed internationally by Boston College in the USA in collaboration with Statistics Canada in Ottawa (Canada), the IEA Data Processing Centre in Hamburg (Germany) and the Educational Testing Service in Princeton (USA).
The following countries participated in TIMSS-R:
TIMSS-R was conducted in South Africa by researchers from the Group: Education at the Human Sciences Research Council. More than 8 000 Grade 8 learners were assessed in 200 schools, and more than 350 teachers and 190 principals of those schools participated.
2. HOW WAS THE INFORMATION COLLECTED?
The TIMSS-R research included tests and questionnaires to collect information about pupils’ achievement and background information about schools, teachers and pupils.
The mathematics and science tests were developed internationally in a collaborative manner. Three different types of questions were included in the pool of TIMSS-R questions: multiple-choice questions, short answer questions and extended answer questions. Some of the questions from 1995 were incorporated into the new tests as trend items, in order to measure the difference in achievement between 1995 and 1999.
Two questionnaires seeking information on national level curriculum plans, reforms, issues and policies with respect to mathematics and science curricula were administered to curriculum specialists. Questionnaires were also administered to the principals, mathematics teachers, science teachers and pupils.
A total of 225 schools were selected at random from all 9 provinces. One intact mathematics class per school was selected. Schools from all former departments of education are represented and both government and private schools are included in this sample. Ultimately, 194 schools and 8147 pupils were included in the international dataset for analysis. South Africa achieved an 85% response rate and the national sample is representative for the country.
3. HOW DID SOUTH AFRICAN PUPILS PERFORM IN MATHEMATICS COMPARED TO OTHER COUNTRIES?
South African pupils performed poorly when compared to other participating countries (see Appendix A - a pdf file. Also see the selected items and results below). The average score of 275 points out of 800 points is well below the international average of 487 points. The result is significantly below the average scores of all other participating countries, including the two other African countries of Morocco and Tunisia as well as that of other developing or newly developed countries such as Malaysia, the Philippines, Indonesia and Chile. As in 1995, pupils from the Asian countries of Singapore (at the top with 604 scale points), Korea, Chinese Taipei, Hong Kong and Japan demonstrated the best achievement in mathematics.
As can be seen in Appendix A, only the most proficient pupils in South Africa (and the same holds for Chile, Morocco and the Philippines) attained the level of the average pupils from Singapore. South African pupils scoring around the country’s average score fell below the least proficient pupils from almost all other countries with the exception of Morocco, the Philippines, Chile and Indonesia.
4. HOW DID PUPILS FROM DIFFERENT PROVINCES PERFORM IN MATHEMATICS?
The province with the highest average scale score for mathematics was Western Cape with 381 scale points, but this was still significantly below the international mean score of 487. Northern Cape and Gauteng achieved the second highest score with 318. Northern Province was below all the other provinces with a score of 226.
Table 1 Inter-provincial results for mathematics
An important finding is that there was no (statistical) difference in the scores of the girls compared to the boys. However, only in Western Cape were the girls’ scores better in mathematics than the boys. The largest differences between boys and girls were found in Free State (34 points) and Mpumalanga (38 points) where the boys perform on average 4-5% higher on the achievement test than the girls.
5. HOW DID SOUTH AFRICAN PUPILS PERFORM ACROSS THE DIFFERENT MATHEMATICS TOPICS?
South African pupils’ performance was relatively low in every mathematics topic (from 37% for algebra to 45% for data representation, analysis and probability). For South Africa, the average score for data representation, analysis and probability is the highest in that the score of 356 points (out of 800 points) makes this the scale with the smallest difference to the international average. Ironically, this is also the area not included in the intended curriculum at Grade 8 level. The lowest achievement was in the area of algebra with a score of 293 points out of 800 points.
6. HOW HAS THE PERFORMANCE IN MATHEMATICS AND SCIENCE CHANGED FROM 1995 TO 1998?
There are 26 countries in TIMSS-R that also participated in the original TIMSS study, including South Africa. Overall, the international average achievement in mathematics did not change, as there is only a small increase by 2 scale points (from 519 in 1995 to 521 in 1999 across the 26 countries). There was no real difference in performance between the pupils in 1999 and those in 1995. Less than 0.5% of pupils in South Africa reached the average score of the top 10% of pupils internationally.
Internationally, there were no significant differences in achievement for mathematics for either girls or boys between 1995 and 1999. In South Africa there was an increase in the score for girls (from 264 in 1995 to 267 in 1999). There was a decrease in the score for boys, who on average scored 10 points less in 1999 (from 293 in 1995 to 283 in 1999). Neither of these results is statistically significant.
In terms of gender differences between girls and boys, there were significant differences in some countries in 1999, as in 1995. An important finding, however, is that fewer countries had these differences (3 countries compared to 6) in 1999. In 1995, the Czech Republic, Iran, Japan, Korea, the Netherlands and Israel all had significant differences in achievement that favoured the boys. In 1999, only three countries still have these significant differences favouring boys; the gap has decreased in others. There were no significant differences found in 1995 or in 1999 for South Africa pupils. Although in 1995, there was a 29-point difference between girls and boys, this gap has narrowed to a 16-point difference in 1999.
The trend data reveal very few changes among the content areas in mathematics, as changes on this level are difficult to effect and take a number of years to show themselves. It is expected that TIMSS 2003 will be able to show more differences in achievement on this level
The only significant difference in achievement (between 1995 and 1999) internationally in specific content areas was found in data representation, analysis and probability. The average percent correct of South African pupils decreased by 2% for fractions and number sense, measurement and data representation, analysis and probability, and 1% for geometry and algebra.
7. WHAT KINDS OF SCHOOLS PARTICIPATED IN TIMSS-R?
The principals reported that:
8. WHO WERE THE GRADE 8 MATHEMATICS TEACHERS THAT PARTICIPATED AND WHAT DO THEY DO AT SCHOOL?
The TIMSS-R results are disappointing. However, the performance of South African pupils’ has remained constant despite an increase in the number of pupils enroling at school. Nonetheless, there is a lack of comparability of South African pupils’ results with those of their peers in other developing countries, a number of which participated in 1998/1999.
Now that access to education and the right to learn has been established for the majority in the country, it is time to set key priorities for the country’s future. If South Africa is to succeed in a rapidly changing and competitive technological world, it will need to develop and protect its capacity to produce well-qualified teachers. In order to improve the lack of change detected by TIMSS-R from developing further, resources have to be put into a variety of well-designed, planned and effective programmes promoting and implementing mathematics and science. Greater collaboration within and between government and the private sector will be required to optimise energies and resources.
SELECTED TEST ITEMS FROM THE TIMSS-R STUDY:
Percentage of correct responses by South African participants by question with the international percentage in brackets: