Why don't skating protocols gray out outlayer judges' marks | Golden Skate

Why don't skating protocols gray out outlayer judges' marks

lariko

Record Breaker
Joined
Jan 31, 2019
if they were not used in final calculation of the skaters' scores? It would show clearly which judge was bias and that the result should not be considered, making the protocols far easier to read and eliminating guess work. Right now, one has to recalculate to see if each mark contributed or didn’t contribute to the score. This is just a macro that is not hard to implement.
 
I think the lowest mark and the highest mark is thrown out... so it's easy to see.

At the same time, in other sports, like diving, you will see which marks are crossed.. you won't see who is the judge however.
 
In many cases the top and the bottom mark to be excluded are part of a "tie". So the mark to be greyed out in the sheet would be one random out of the tied ones. In those situations, you wouldn't visibly see the bias but rather one score given by others too.
For instance, skating skills for Nakata last night:
J1: 8.25
J2: 8.00
J3: 8.00
J4: 8.00
J5: 7.75
J6: 8.25
J7: 7.75
J8: 7.75
J9: 7.75

So the marks eliminated in the trimmed mean are one 8.25 (J1 or J6) and one 7.75 (J5, J7, J8 or J9).
 
In many cases the top and the bottom mark to be excluded are part of a "tie". So the mark to be greyed out in the sheet would be one random out of the tied ones. In those situations, you wouldn't visibly see the bias but rather one score given by others too.
For instance, skating skills for Nakata last night:
J1: 8.25
J2: 8.00
J3: 8.00
J4: 8.00
J5: 7.75
J6: 8.25
J7: 7.75
J8: 7.75
J9: 7.75

So the marks eliminated in the trimmed mean are one 8.25 (J1 or J6) and one 7.75 (J5, J7, J8 or J9).
They all should be grayed out so there is no guessing. It will be visible and easier to interpret. And if one judge gets the most gray squares, we know who is out to lunch with the fed.
 
The FIS Ski Jumping World Championships are being held in Trondheim (NOR) at the moment. So far, the only competition that has been completed is the Women's Normal Hill Indivdual competition, which was held yesterday.

Here are their equivalent to our Protocols PDF's:

1st Round Results
Final Results

Like in Figure Skating, the highest and lowest Judges' scores get discarded. As you can see, the Results PDF's do not show clearly which scores were discarded.

HOWEVER, the TV graphics do show it. They put a line through the scores that get discarded, even if it is a tied score. So those of us watching can easily spot if a particular Judge is scoring lower or higher than the others.

What I'm getting at is that this isn't just a Figure Skating issue.

Perhaps it is just something that is not possible with the respective software being used.

But, I agree. It would be useful to have it shown on the PDF's. In both sports.

CaroLiza_fan
 
Last edited:
I certainly can't speak to ski jumping, but thank you for providing that example.

The thing is, for every program there will be up to 15 scores (12 GOEs and 3 PCS). For the vast majority of those, there will be more than one judge who gives the highest GOE and more than one who gives the lowest, as @Andrea82 pointed out. For most jumps with falls, all of the GOEs from all the judges will be identical -5s, so they will all be both the highest and the lowest for that element.

For mathematical purposes of arriving at the trimmed mean, it doesn't matter which of many -5s (or which 8.25 or 7.75 component score in Andrea82's example) gets dropped. The mathematical effect is the same.

The purpose of dropping the high and low was never to identify judges who are more biased than the others. There are other calculations used for that purpose.*

(*For rules about how deviations are calculated to determine potential national bias or incompetency, see https://isu-d8g8b4b7ece7aphs.a03.az.../2661_OAC_Communication_2024-251731319942.pdf)

Over the course of an event, most judges will probably have some of their scores dropped, especially if you alternate randomly among those who have the same score.

The purpose is to mitigate the effect of outlying scores, whether as a result of bias, honest differences of opinion, momentary errors, different ways of using numbers**, etc.

***E.g., some judges may consistently be more generous on GOEs or more stingy on PCS across the board, so their scores might end up highest or lowest more often than the rest of the judges', but if they are consistently high for all the skaters (or consistently low for all the skaters), that is not bias.

Similarly if they have been trained to avoid giving 0 GOE whenever possible, so they might give a lot of -1s and +1s for so-so elements that most judges give 0s.

They might take an approach of spreading their component marks to make sure that they clearly distinguish between the skaters who did best and worst in these areas across the field and to have plenty of room to distinguish among the skaters in the middle on each component.

They might also make a point of spreading components for the same skater, so that if most judges gave Skater X PCS of 8.25 7.75 8.0 and this judge gave 9.0 7.0 8.0 (which is the kind of thing that many fans argue would be "more correct" use of PCS), both the 9.0 and the 7.0 would get grayed out, but notice that the total PCS this judge gave this skater are the same as most of the rest of the panel, and their high CO score and low PE score getting grayed out cancel each other out, so they're not giving either an advantage or disadvantage to this skater.

Again, this does not represent bias if it is done consistently across the board.

What is the reason that you want to see high and low scores highlighted on the protocols for each element and each component? If you're just looking to see what the range is at a glance, they could shade all the high scores in one color and all the low ones in another so you can pick out the high and low numbers without singling out individual judges where several of them are in agreement.

If you're looking for individual judges who are frequently out of step with the rest of the panel, that kind of highlighting could allow you to run your eye down each judge's column and see if certain judges had more highlighted scores than the rest of the panel.

But you'd have to examine them more closely to see whether it's because this judge consistently uses a higher range, a lower range, or a wider range, or whether there is a pattern of consistently overscoring particular skaters (especially their compatriots) and underscoring others (their expected closest rivals).
 
Last edited:
if they were not used in final calculation of the skaters' scores? It would show clearly which judge was bias and that the result should not be considered, making the protocols far easier to read and eliminating guess work. Right now, one has to recalculate to see if each mark contributed or didn’t contribute to the score. This is just a macro that is not hard to implement.
Skating Scores is your friend here
 
They don't say which scores where included and which weren't. I want to see them on official protocol not on a fan website.
I'm pretty sure they do, but in any case, the ISU couldn't get headshots right, so implementing a grey box might be harder than you would expect...
 
I don't think that the ISU is interested in stirring this particular pot.

There are many reasons why a judge's mark might be higher or lower than his or her fellows'. Some judges are overall stricter and some more lenient. Some judges are stricter with one aspect of scoring and less strict with another. One judge might spot an under-rotation or a questionable edge that another misses. One judge might be moved by a skater's musical timing, another not so much.

If a fan wants to look at the protocols, it is easy to see which mark(s) in each line are lowest or highest, whether those marks appear in grey or not. As for detecting patterns of systematic bias. that is a much more subtle task. The ISU judges' oversight regulations as posted above by genekelly are not whimsical, although a judge has to be way off quite a few times before any action is taken. I think that this is appropriate. Catch the truly bad guys and otherwise let the judges judge.
 
Last edited:
Back
Top