The more I think about it, though, the more I don’t think the median is really “throwing out judges’ scores.” It is more like, “determining the middle value of a range of scores.” But on the other hand, the median does not discourage cheaters from cheating, as I hoped at first that it might. Here is a simplified example. The first six judges go like this:
Lulu: 8.75 8.75 9.00 9.00 9.25 9.25; Median =9.00
Shel: 8.50 8.50 8.75 8.75 9.00 9.00; Median = 8.75
A close contest, but every judge thinks that Lulu is a little bit better.
Now three conspiring pro-Michelle judges enter the fray. They give
Lulu: 8.50 8.75 8.75
Shel: 9.00 9.00 9.25
The three new numbers knock Lulu’s median down to 8.75 and bump Michelle’s up to 9.00. Michelle wins.
@Math: I don't think cheating can be eliminated tbh. If people wants to cheat they will find the way, but at least I think it's fair that the majority should decide and no anonymity. So if there is a way to combine what's good with the new system with the old one that would be great IMO.
Seven of these judges (all of the pro-Lulu judges, one of the pro-Shel) thought the contest was quite close, giving only 0.25 difference between two skaters.
One of the pro-Shel judges gives 0.5 difference, and one gives 0.75. Those larger differences are enough to tip the balance in Shel's favor, whether the scores trimmed to the medium, or trimmed by high and low, or "Winsorized."
However, we don't know whether those judges honestly thought Shel was significantly better in the areas of skating that they considered most important, which might differ from what the pro-Lulu judges considered most important. We don't know whether they gave larger gaps because they thought Shel was just that much better, or they naturally tend to leave larger gaps in their scoring between all the skaters, or because they were trying to manipulate the outcome toward the skater they wanted to win (whom they might not actually have thought was better at all, if they were cheating). Or, if this was ordinal judging, because they thought Shel was better than Lulu and they also wanted to leave room in between them for a later skater who might be better than Lulu and worse than Shel in their opinion.
If there are two scores per skaters with tiebreakers (second mark in the freeskate), that means that judges who think it's a close contest can give two skaters the exact same total. But if the absolute scores count, not ordinals, it would be stupid to use tiebreakers that way, because it's not really giving the skater that judge prefers a higher score at all and makes the results vulnerable to a single judge who prefers the other skater and doesn't use the tiebreaker give the same total. This was a major flaw in the way pro competitions were often scored, with pro "judges" often thinking in terms of ordinal-style 6.0 tiebreakers.
But Mathman's example shows that even using the smallest increment between scores leaves the possibility that other judges who hold the opposite opinion and use wider increments can override a majority.
Ordinals open up the possibility of flipflops and similar paradoxes.
This example shows that total scores can result in paradoxes where a minority who uses wider gaps between scores (in general or for the specific skaters of interest; as a general principle of scoring or to try to force a particular result) can override a more conservatively scoring majority.
Is the answer to encourage all judges to be bold in their scoring gaps? Or to encourage all to keep the scores close together so that large gaps in minority opinions will stand out for scrutiny?
Accept that judging is subjective. There can never be a perfect system for scoring. Even if the officials are clean as a whistle and never cheat, there will still ALWAYS be results that are "controversial" that some people aren't going to like.
Embrace all of this as part of the fun and drama of figure skating!
It's cool and all when your favourite skaters do well, but if they don't... well pfft big deal. Enjoy the skating. Have a moan about results if necessary, but then move on and let it go - for the sake of your health.
Life's too short...
Agreed, YesWay. At the end of the day, that's all we can do. Enjoy the skating. Because no matter what the scoring system, no matter who the judges are, we're all going to have results we find unfair. Even if we can't ever agree with a certain result (and there are results I'll decry until my dying day), we have to accept that such things happen in this sport.
I agree. The focus shouldn't be on results and who's better than who so much as who is able to deliver wonderful, memorable performances. The joy of skating is about performance not credentials and decimals.
And a whole skating program is complex enough that different individuals will favor different aspects. Even with individual elements there might be differences of opnion as to whose is "better" if they're good (or bad) in different ways.
So most of the time, in a field with numerous skaters including several strong ones at the top, there is not one right answer as to the "correct" results.
The technical panel calls may be black-and-white, yes-or-no decisions that can be clearly right or wrong. Except sometimes when the skater almost/kinda/sorta executes the element or feature as required and it falls in a gray area.
Some kinds of errors on elements require mandatory GOE reductions and may be required to have negative GOE, or specifically -3 GOE, so a judge who gives 0 or better to an element with such an error would be flat-out mistaken on that element.
But otherwise, for all the qualitative decisions that fall under the GOEs and PCS, there aren't right or wrong answers. There are different experts' evaluations within the written guidelines. Some evaluations are better than others, but there may be several that are equally correct depending on the priorities of that particular judging panel at that time.
So the goal of the scoring system can't be to find the single correct answer. But rather to find the best way to combine the disparate opinions of a panel of experts who come to varying decisions, hopefully all tending toward the "better" end of the spectrum.
I have never understood what is so "paradoxical" about flip-flops.The only reason we even attach the negative-sounding name "flip-flop" to this phenomenon is that we have counted our chickens before they hatch. Just tell the skaters, "hold your celebration until everyone skates."Ordinals open up the possibility of flipflops and similar paradoxes.
Last edited by Mathman; 06-10-2014 at 02:46 PM.
The problem is that very, very, very, very few people have faith in the judging system (and even less in the judges).
Until that's fixed FS is screwed in NAmerica.