- Joined
- Jun 21, 2003
That's what is so cool about this statistic. You can say at least something even if all you have is the bare bones as far as data to work with. (Obviously, the more numbers you have the happier the statistician gets.)
And no matter what, there is the question of whether one can ever "prove" anything at all with statistics.
But yes, applying statistical techniques to only one small sample of numbers is not very helpful. But if you do it for many, many different competitions over time, the evidence might start to accumulate.
This in fact is the way the ISU's judges evaluation procedure works. If a few of your marks are high or low, no notice is taken. In order to get hit with an "anomaly"you have to be way off from the average scores of all judges (and I think they include the referee's scores in the average), and in the case of PCSs the anomaly has to be always in the same direction. (That is, if the judge gives a skater way too high a mark on SS and way too low on CH, that's a wash and no attention is called to that judges' scores. This may be related to the question of why the judges tend to give essentially the same scores to each skater in all 5 program components.all 5 PCSs about the same.)
If you get a whole lot of "anomalies", then you get an "assessment." Before getting an official "assessment" you get to present your case to the judge's evaluation committee, together with video evidence. That is, you can claim that you were the only judge who saw that wrong edge take-off, so your scores were right and all the other judges were wrong.If you get a lot of "assessments over the course of a season, you might be hit with a sanction or penalty.
It is really very difficult for any judge to be so far off and so consistently far off that anything comes of this procedure in terms of penalties. I believe that there are only a dozen of so cases per year where the procedure results in penalties, with most of them for judges at the lower levels and most of them in ice dancing.
The details are given in ISU Communication #1631, and whatever revisions have been made more recently. The cool thing is that the ISU never has to try to "prove" that a judge is biased, incompetent or an outright crook. The only thing that is being measured is, "how far off is this judge's scores from what is expected, taking into account all of the judges' opinions." (This is a different procedure from disciplining, for instance, a referee for assigning her sister as judge, as happened last year and is against the ISU ethics rules.)
Anyway, yes of course the more numbers you have more reliable the statistical results are. What was interesting to me, though, was that even in the case where each judge gives only one mark (and that one a ranking, not a real number -- i.e., not a measurement), still even in this weak case statistics provides tools for quantifying "same" and "different."
This particular statistic also has the useful feature that it does not depend on knowing any thing about the underlying population, such as the average score, the range of scores, whether the scores are symmetric above and below the mean, etc. In other words, it doesn't do much but what little it does, it always does without fail.
)
Last edited:
