There is a tendency for people to evaluate the performance of schools purely based on the raw exam results. How many passed, how many failed, were there any toppers? That sort. This tendency has its genesis in the BCSEA’s annual results analysis that throws out more raw data than the reader can digest. The tendency is further promoted by the media that seems to be equally overwhelmed and opts to simply regurgitate the ‘highlights’. BCSEA’s own reports make little effort to distinguish student performance from a school’s performance, clearly assuming the two to be the same.
An article published by Kuensel (“When passing isn’t good enough“) cleverly added their own indicator of measuring quality, that of the number of students who managed to score enough to earn a seat in a government school. It was clearly not clever enough because, besides being totally arbitrary, it still only used raw data and ultimately confused itself over whether they were measuring student performance or school performance. The ‘research’ set no baseline for comparing results and effectively compared apples with oranges. We were pained to read the article’s final ‘finding’ that “Pelkhil was at the bottom of the quality bar”. A conclusion that was drawn without even defining what was meant by ‘performance’ or ‘quality’. And what on earth is a ‘quality bar’?
While the board exams can easily measure student performance, it is a fallacy to measure a school’s ‘quality’ directly from student results even if one were to define ‘quality’ as being nothing more than academic performance. There are many factors affecting student performance and not all can be attributed to the school. For example students tend to change schools very frequently and the school they ultimately sit for the board exams from, may have had close to zero responsibility for their performance. And it only gets more complex if ‘quality’ is defined as being more than just academic performance.
Anyone who has bought a car and attempted to evaluate the options available in order to make the best choice, will have realized that ultimately what you look at depends on what you want. If you have a budget issue, then the price is the most important factor. If it is engine power or seat capacity or customer care, then the factors you look at will differ. The same thing applies in evaluating schools. If academics is important, then look at the results (and the context in which the results carry meaning). If it is wholesome education, then you must look beyond academics and at the extra activities the school offers. If it is the fees, then clearly government schools are best as they are free.
In Bhutan, the board results seem to be the most popular measure of performance. But what does it really mean and what value can you draw out of the results they publish in the media? Not very much unfortunately, unless the relevant data is properly analyzed.
The hard reality for private schools is that the better performing students (with scores higher than 62% in the BCSE) stay in the government schools after the BCSE, and the bottom half (including the large numbers who did not pass) migrate to the private schools. Therefore comparing the results of a government school and a private school directly is, besides being unfair, meaningless if you want to compare ‘quality’.
A more accurate measure would be the progress or decline the students make when they sit for the BHSEC two years later. The way to measure progress or decline, not perfect by any means, would be to take the mean BCSE score in each subject and compare it with the scores in the BHSEC. A government school which achieved a mean score of 80% in math may look good, but if it had admitted students with a mean score of 85% in the BCSE, then the results actually reflect a decline in performance. On the other hand, a school that achieved a modest mean of 55% but with students whose BCSE mean was just 40%, has achieved an improvement of 15% which is in fact admirable. This is the sort of baseline that is completely missing in the Kuensel analysis. Good research methodology requires proper baseline data and without it, the Kuensel ‘finding’ about school performance is meaningless.
Besides, it is important to note that good academic results can reflect nothing more than good testing skills. When too much emphasis is placed on academic results, teachers and schools tend to oblige by emphasizing on testing skills instead of real learning. Declaring a school with good results as a school of ‘high quality’ is a blatant exaggeration if no other indicator is looked at.
Kuensel need not fret too much over this disastrous piece. Let’s just say that journalism itself still has a long way to go in Bhutan. That would be a fair ‘baseline’ for assessing this article. (Click here for another Kuensel article where the first sentence says it all.)
The article below is reproduced from The Guardian (UK) and talks about the same issues of erroneous assessment. Read on and learn how you can draw better conclusions.
—————————————————————————————————
Parents who pore over exam results should turn their attention to the contextual value added rating, which offers a more accurate assessment by emphasising progress over final results, says Dylan Wiliam
Kenneth Clarke, the former education secretary, famously remarked that he preferred his data raw rather than cooked. It’s a great soundbite. Unfortunately, it’s statistically naive – and a foolish basis for making important decisions such as which hospital to choose for an operation, or the school you choose for your children.
Take hospital mortality rates. If you need heart surgery, and you have a choice of hospitals, then it might seem that choosing the one with the best survival rate might be a good bet. And yet the hospitals with the best surgeons are often those “of last resort” where patients go only if all the other hospitals say there’s nothing that can be done. Comparing hospital survival rates gives you an idea of how good the hospital is only when the patients treated by the hospital have equally good (or bad) prognoses, and this is rarely the case.
The same complexities bedevil school achievement data. The results schools get in national curriculum tests and public examinations are not pure measures of educational quality. Like hospital mortality rates, they conflate two quite separate things; in the case of school results, it’s the quality of education offered by the school, and the level of achievement already possessed by students before they start at the school. Sending your child to a particular school because it gets good results makes no more sense than choosing a cottage hospital for complex open heart surgery just because no one has died there recently.
Very few parents rely solely on academic success rates in choosing a school, but in most cases academic success is an important factor. It is therefore essential that the information that parents do get about school results is meaningful. Ideally, parents want to know how their child will do if they go to a particular school – but of course, it is not possible to tell them that schools can go down as well as up in quality. What we can do, however, is tell them how children similar to their own did at that school over the last few years.
This is what contextual value added (CVA) does. The CVA rating for a school is a measure of how much progress students make at the school, taking into account what they knew before they started, the fact that girls make more progress than boys, and the fact that students from certain minority ethnic communities do better than others. The average CVA rating for a secondary school is 1000.
Because the CVA rating takes into account almost all the factors that are beyond the school’s control, it provides the best way to date of reporting on the quality of the education provided by a secondary school, at least in terms of its GCSE results. This is important, because most government pronouncements treat the raw results – whether in “league tables” or just for each school – as the “gold standard” for information, and CVA results are presented as additional, possibly useful information.
This is the wrong way round. CVA is – by a long, long way – the best measure of the quality of education provided by a school, and it is unlikely that anything better will be available, at least in the foreseeable future. Indeed, there is a strong case for making CVA the only information made available on the academic performance of schools. To see why, consider the case of a typical suburban local authority such as the London Borough of Barnet.
Barnet grammar school (a pseudonym) looks like a very good school. In 2007, 99% of students got five good grades (ie, grades A* to C) at GCSE, including English and maths. But parents sending their child to this school just because they think they will get better GCSEs would be making a big mistake. In fact, in terms of how much students learn at this school, it is only just above, and not significantly different from, the national average (the CVA score is 1004).
At the other end of the scale, Ravenscroft school looks like a “problem” school. In 2007, less than one student in three (29%) achieved five good grades including English and maths. Who would want to send their child there? And yet the CVA score for Ravenscroft is 1028. In other words, the same child would get half a grade better in each subject at GCSE if he or she went to Ravenscroft rather than Barnet grammar. Moreover, this is not a blip – the CVA scores for the previous year showed exactly the same margin: 24 points in favour of Ravenscroft.
These are two extreme cases, but it applies across Barnet. The children of parents who choose schools on the basis of raw results will get lower GCSE grades than those of parents who choose schools for their children on the basis of CVA. The difference is small – about one grade in one subject on average – but significant. In fact, recent research from the University of Bristol has shown that there is very little difference in how much students learn in different schools across the whole country. Across England, 93% of the variation in raw results achieved by different schools is accounted for by differences in the achievement of students before they started at the school, and in Barnet, the figure is 99%.
But the fact remains that in every authority, choosing schools on the basis of raw results will mean that your child is likely to get lower grades at GCSE than if you had chosen the school on the basis of CVA. That’s why CVA should be the most salient (and ideally the only) information made available to parents. Using any other data will lead to worse decisions by parents, worse outcomes for children, and the continued prevalence of schools that are better at attracting able students than teaching them. On the other hand, if parents relied solely on CVA to judge the academic achievement of schools, it would create a relentless pressure on schools to improve student learning. Anyone have a problem with that?
· Dylan Wiliam is deputy director and professor of educational assessment at the Institute of Education, University of London