MCEETYA Paper - Reporting & Comparing School Performances

Click here to download the complete paper (2.7Mb pdf file).

Click on the links to go direct to the relevant section in this website page:

• Communique
• Executive Summary

Communique:

Today, the Australian, State and Territory Education Ministers made further progress in working together to deliver collaborative national reform of schooling in Australia. They confirmed the successful development of the Building the Education Revolution program and arrangements for its timely delivery. They also agreed on the further development of important reforms in Teacher Quality, Literacy and Numeracy and Disadvantaged Schools, based on initial implementation plans for COAG National Partnerships in these areas.

In a major step forward for the shared national transparency agenda, Ministers also agreed on a framework for publication of comparable information about school performance and context: a vital collaborative reform.

For the first time parents, teachers and communities will have access to nationally consistent information that details a school’s results, its workforce, its financial resources and the student population it serves.

From 2009 the new Australian Curriculum, Assessment and Reporting Authority (ACARA) will be responsible for publishing relevant, nationally comparable information on all schools. This will include publication of the 2008 NAPLAN data and associated contextual information.

The information available will enable comparison of each school with other schools serving similar student populations around the nation and with the best-performing school in each cohort of ‘like schools’.

It will also support accountability, school evaluation, collaborative policy development and resource allocation. These same transparency and accountability requirements will apply to both government and non-government schools.

Through better monitoring of performance at the student, school and system level, educational outcomes can be lifted across all schools.

All governments will continue to work together to develop a set of meaningful measures to guide school evaluation, accountability and resource allocation. Improvements in the quality and quantity of information available on school education will allow governments, through the National Partnerships, to target resources to schools. Ministers agreed to release the ‘Reporting and Comparing School Performances’ report prepared by the Australian Council for Education Research (ACER), on which the Ministers drew in developing the school transparency agenda. Ministers agreed that these reforms were not about simplistic league tables which rank schools according to raw test scores.

In important new developments for the shared reform agenda, Ministers agreed to include arts in the second phase of the development of the National Curriculum. In the first phase, the interim National Curriculum Board was asked to develop curriculum for the core subjects of English, mathematics, science and history. Geography and languages have already been confirmed as forming part of the second phase of curriculum development.

Ministers also agreed to reorganisation of Ministerial Council Governance to reflect the shared reform agenda and associated changes in State and Territory and Commonwealth responsibilities across education, early childhood, employment and youth affairs.

Executive Summary

This report provides advice on the collection and reporting of information about the performances of Australian schools. The focus is on the collection of nationally comparable data. Two purposes are envisaged: use by education authorities and governments to monitor school performances and, in particular, to identify schools that are performing unusually well or unusually poorly given their circumstances; and use by parents/caregivers and the public to make informed judgements about, and meaningful comparisons of, schools and their offerings.

Our advice is based on a review of recent Australian and international research and experience in reporting on the performances of schools. This is an area of educational practice in which there have been many recent developments, much debate and a growing body of relevant research.

Our work is framed by recent agreements of the Council of Australian Governments (COAG), in particular, at its meeting on 29 November 2008: COAG agreed that the new Australian Curriculum, Assessment and Reporting Authority will be supplied with the information necessary to enable it to publish relevant, nationally-comparable information on all schools to support accountability, school evaluation, collaborative policy development and resource allocation. The Authority will provide the public with information on each school in Australia that includes data on each school’s performance, including national testing results and school attainment rates, the indicators relevant to the needs of the student population and the school’s capacity including the numbers and qualifications of its teaching staff and its resources. The publication of this information will allow comparison of like schools (that is, schools with similar student populations across the nation) and comparison of a school with other schools in their local community. (COAG Meeting Outcomes) Our work also has been framed by the recently endorsed MCEETYA Principles for Reporting Information on Schooling (see Section 1.4).

Before summarising our specific recommendations, there are some general conclusions that we have reached from our review of international research and experience. The specific recommendations that follow are best understood in the context of these general conclusions:

• Vigilance is required to ensure that nationally comparable data on individual schools does not have the unintended consequence of focusing attention on some aspects of the purposes of schooling at the expense of other outcomes that are as important but not as easily measurable. Parents/caregivers and the public are interested in a broad range of information about schools, and nationally comparable data should be reported in the context of this broader information.

• Although it has become popular in education systems in some other parts of the world to use statistical models to develop ‘measures’ of school performance and to report these measures publicly in league tables, we believe that there are very sound technical and educational reasons why school measures of this kind should not be used for public reporting and school comparisons.

• Related to this point, we are not convinced of the value of reporting ‘adjusted’ measures of student outcomes publicly. Measures of student outcomes should be reported without adjustment.

• To enable the comparison of unadjusted student outcomes across schools, we believe that a ‘like-schools’ methodology should be used. This methodology would allow parents/caregivers, the public, and education systems to compare outcomes for schools in similar circumstances.

• While point-in-time measures of student outcomes often are useful, it is difficult to establish the contributions that teachers and schools make to point-in-time outcomes. In general, measures of student gain/growth across the years of school provide a more useful basis for making judgements about the value that schools are adding.
• Measures of gain/growth are most appropriately based on measurement scales that can be used to monitor student progress across the years of school. The NAPLAN measurement scales are an example and provide educational data superior to that available in most other countries. Consideration should be given to developing national measurement scales for early literacy learning and in some subjects of the national curriculum.

• Initially reporting should build on the understandings that parents and the public have already developed. For example a school’s NAPLAN results should be reported in forms that are consistent with current NAPLAN reports for students. Although much work needs to be done in defining the most appropriate measures, the principle should be to build on the representations of data that are already familiar to people.

Recommendations

Our report makes the following specific recommendations:

student outcome measures

• Nationally comparable data should be collected on the literacy and numeracy skills of students in each school, using NAPLAN (Years 3, 5, 7 and 9).
• Nationally comparable data should be collected on the tertiary entrance results of students in each senior secondary school. These data could be reported as the percentage of students achieving tertiary entrance ranks of 60 or above, 70 or above, 80 or above, and 90 or above (calculated as a percentage of the students achieving tertiary entrance ranks).
• Nationally comparable data should be collected on the percentage of students in each senior secondary school completing Year 12 or equivalent; the percentage of students applying to all forms of post-school education; and the percentage of students completing VET studies.
• Nationally comparable data should be collected on the achievements of students in core national curriculum subjects (English, mathematics, science and history), beginning in 2010. National assessments could be developed initially at Year 10.
• Nationally comparable data should be collected on the early literacy learning of children in each primary school. These assessments will need to be developed and should be administered upon entry to school and used as a baseline for monitoring progress across the first few years of school.

physical and human resources

• Nationally comparable data should be collected about sources and amounts of funding received by each school, including all income to the school from State and Commonwealth governments, as well as details of fees payable by parents, including those that are mandatory and any voluntary levies that parents are expected to pay.
• Nationally comparable data should be collected on the numbers and qualifications of teaching staff in each school. Basic data would include academic qualifications, details of pre-service teacher education, and details of any advanced certification (eg, Advanced Skills Teacher; Level 3 Teacher).

student intake characteristics

• Nationally comparable data should be collected on the socio-economic backgrounds of students in each school. Data should be based on information collected at the individual student level, using at least parental occupation and, possibly, parental education levels, under the agreed MCEETYA definitions.
• Nationally comparable data should be collected on the percentage of students in each school of Aboriginal and/or Torres Strait Islander background under the agreed MCEETYA definition.
• Nationally comparable data should be collected on the percentage of students in each school identified as having a language background other than English (LBOTE) under the agreed MCEETYA definition.
• Nationally comparable data should be collected on the geo-location of each school using a 3-category scale: metropolitan, provincial, and remote.
• Nationally comparable data should be collected on the percentage of students in each school with special educational needs. A nationally agreed definition of this category will need to be developed.

like-school comparisons

• In reporting student outcome data for a school, data for like-schools should be provided as a point of comparison. Like-schools will be schools in similar circumstances and facing similar challenges.
• In determining ‘like-schools’, account should be taken of the percentage of students with Indigenous backgrounds, the socio-economic backgrounds of the students in the school, and the percentage of students from language backgrounds other than English.
• For each school separately, like-schools should be identified as the schools most similar to that school on the above characteristics (rather than pre-defining a limited number of like-school categories).
• Work should commence as soon as possible on the development of an appropriate like-schools methodology.

public reporting

• For the purpose of providing public information about schools, a common national website should be used to provide parents/caregivers and the public with access to rich information about individual schools.
• The national website should provide information about each school’s programs, philosophies, values and purposes, provided by the school itself, as well as nationally comparable data, provided centrally.
• Nationally comparable student outcome data should, wherever possible, provide information about current levels of attainment (ie, status), gain/growth across the years of school, and improvement in a school over time.
• The complete database for each state/territory should be made available to the relevant state/territory departments of education and other employing authorities, enabling them to interrogate data for their schools and to make judgments about school performances using aggregated data and national summary statistics.

We believe that almost all nationally comparable data collected centrally could be reported publicly. The exceptions would arise when the public reporting of data may have negative and unintended consequences for schools. For example, we can envisage negative consequences arising from the reporting of the socio-economic backgrounds of students in a school, or of the financial circumstances of struggling, small schools (both government and non-government). We also believe that data reported publicly should be factual data about a school, and not the results of secondary analyses and interpretations that are open to debate (eg, value-added measures).