- Joined
- Jun 21, 2003
The ISU has promised that at the end of the season they will go over all the judges scores (not available to the public) and evaluate the performances of the judges. Those found lacking presumably will be required to undergo more training before they are assigned to any more major events.
One way to measure a judge's performance is to compare that judge's scores against the scores of the majority of the panel. For ordinal judging a standard way to do this is to compute a statistic called the Spearman rank correlation coefficient. Roughly speaking, this gives a percentage mark to each judges ordinal ranking which measures the degree to which that judge agreed with the rest of the panel.
I tried it out for the top six finishers in the ladies' long program in the 2002 Olympics. Here are the coefficients for each judge, identified by nationality, in descending order of apparent competency.
U.S.A. -- 1.00 (perfect match with the final placements -- take a bow, judge Joe!)).
Finland -- 1.00
Germany -- .89
Belarus -- .89
Italy --.83
Slovakia -- .80
Denmark -- .65
Russia -- .34 (the renowned Ms. Danilenko)
You can use the same method to compare the judges against each other, one by one, to see which judges tend consistently to vote together (possible collusion). Here are the highs and lows for each judge.
USA: Most compatible with Finland (1.00) and Canada (.94)
.........Least compatible with Russia (.54) and Denmark (.66)
Finland: Most compatible with USA (1.00)
..............Least compatible with Russia (.54) and Denmark (.66)
Germany: Most compatible with Canada (.94), USA and Finland (.89)
................Least compatible with Russia (.20) and Denmark (.49)
Belarus: Most compatible with Finland, USA and Slovakia (.89)
Least compatible with Italy (.60) and Germany (.66)
Italy: Most compatible with Canada (.94) and Germany (.86)
.........Least compatible with Russia (.37) and Denmark (.49)
Slovakia: Most compatible with Denmark (.94) and Belarus (.89)
...............Least compatible with Italy (.60) and Germany (.71)
Denmark: Most compatible with Slovakia (.94) and Russia (.89)
................Least compatible with Italy and Canada (.49)
Russia: Most compatible with Denmark (.89), Slovakia and Belarus (.77)
............Least compatible with Germany (.20) and Italy (.37)
********************************
The Miss Congeniality award: USA and Finland, with identical scores across the board of 1.00, .94, .89, .89, .83, .83, .66 and .54 (the last two reflect their low compatibility with Russia and Denmark). USA wins the tie-breaker, having a perfect record all the way down through the top ten (this study just includes the top six skaters).
Most cantankerous: Russia, with scores of .89 (v. Denmark), .77 (Slovakia), .77 (Belarus), .54 (USA), .54 (Finland), .42 (Canada), .37 (Italy), .20 (Germany).
*************************************
Conclusion: The Russian judge needs retraining before she is assigned to any more ISU events.
The Russian judge and the Danish judge appear to be in cahoots to skew the vote. Slovakia falls under suspicion, too.
************************************
Under secret judging we will not be able to do analysis like this any more. Thus we are counting on Mr. Cinquanta to do it for us, and to take the necessary and appropriate steps when evidence of incompetence and collusion are found.
MM
One way to measure a judge's performance is to compare that judge's scores against the scores of the majority of the panel. For ordinal judging a standard way to do this is to compute a statistic called the Spearman rank correlation coefficient. Roughly speaking, this gives a percentage mark to each judges ordinal ranking which measures the degree to which that judge agreed with the rest of the panel.
I tried it out for the top six finishers in the ladies' long program in the 2002 Olympics. Here are the coefficients for each judge, identified by nationality, in descending order of apparent competency.
U.S.A. -- 1.00 (perfect match with the final placements -- take a bow, judge Joe!)).
Finland -- 1.00
Germany -- .89
Belarus -- .89
Italy --.83
Slovakia -- .80
Denmark -- .65
Russia -- .34 (the renowned Ms. Danilenko)
You can use the same method to compare the judges against each other, one by one, to see which judges tend consistently to vote together (possible collusion). Here are the highs and lows for each judge.
USA: Most compatible with Finland (1.00) and Canada (.94)
.........Least compatible with Russia (.54) and Denmark (.66)
Finland: Most compatible with USA (1.00)
..............Least compatible with Russia (.54) and Denmark (.66)
Germany: Most compatible with Canada (.94), USA and Finland (.89)
................Least compatible with Russia (.20) and Denmark (.49)
Belarus: Most compatible with Finland, USA and Slovakia (.89)
Least compatible with Italy (.60) and Germany (.66)
Italy: Most compatible with Canada (.94) and Germany (.86)
.........Least compatible with Russia (.37) and Denmark (.49)
Slovakia: Most compatible with Denmark (.94) and Belarus (.89)
...............Least compatible with Italy (.60) and Germany (.71)
Denmark: Most compatible with Slovakia (.94) and Russia (.89)
................Least compatible with Italy and Canada (.49)
Russia: Most compatible with Denmark (.89), Slovakia and Belarus (.77)
............Least compatible with Germany (.20) and Italy (.37)
********************************
The Miss Congeniality award: USA and Finland, with identical scores across the board of 1.00, .94, .89, .89, .83, .83, .66 and .54 (the last two reflect their low compatibility with Russia and Denmark). USA wins the tie-breaker, having a perfect record all the way down through the top ten (this study just includes the top six skaters).
Most cantankerous: Russia, with scores of .89 (v. Denmark), .77 (Slovakia), .77 (Belarus), .54 (USA), .54 (Finland), .42 (Canada), .37 (Italy), .20 (Germany).
*************************************
Conclusion: The Russian judge needs retraining before she is assigned to any more ISU events.
The Russian judge and the Danish judge appear to be in cahoots to skew the vote. Slovakia falls under suspicion, too.
************************************
Under secret judging we will not be able to do analysis like this any more. Thus we are counting on Mr. Cinquanta to do it for us, and to take the necessary and appropriate steps when evidence of incompetence and collusion are found.
MM
Last edited: