Scoring bias at the national level

[Andrea82]

Douglas Williams (USA) is found guilty of national bias for his scoring of Levito, Glenn and Tennell at 2023 Worlds. However, he wasn't found guilty of systematically under-marking their competitors.
So he gets away with just a warning, also because he didn't get previous warnings (unlike some of the judges who were recently suspended who had already had previous warnings for their judging).

https://www.isu.org/inside-isu/isu-communications/communications/32839-case-2023-03-isu-v-d-williams-final-decision/file

His level of self-esteem is quite something: The Alleged Offender, in the Statement of Response prepared on his behalf, describes himself as “one of the finest examples of a skating official that the ISU system has ever and could ever produce”.

 

[el henry]

Thanks for the decision @Andrea82 .

Well, for me that decision is multiple pages of gobbledy gook. Are they using standard deviations on a sample size of 24 to mathematically prove discrimination? And they are inferring intent from the SDs? I guess I am perplexed because in trying class action discrimination cases in the US, classes are usually much larger to assure that the SD meant what it meant, and could withstand regression analyses. Certainly if we were trying to argue "but for" causation, which I *think* the decision addresses.

I'm less concerned about Mr. Williams' statement, he is trying to defend himself, than I am trying to understand exactly how the Disciplinary Commission used these numbers. I am perplexed.

 

[Mathman]

Are they using standard deviations on a sample size of 24 to mathematically prove discrimination?

No, an ISU "deviation point" just means the difference between the errant judge's mark (for GOE on a specific element, for instance), and the mean score of the whole judging panel. It is not based on sample size. like a statistical standard deviation.

The big difference difference between assessing "anomalies" in GOEs compared to PCSs is that for GOEs both "too high" and "too low" count against the judge. For PCSs, scores that are too high and scores that are too low for a specific skater azre allowed to cancel, so you are OK if you score a skater too high on choreography but too low on musical interpretation, compared to the other judges.

I didn't read the most recent changes, but this is how they have always done it in the past. There are a couple of easy-to-follow examples given in ISU Connunication no. 1631, pages 5 and 6.

ISU Communication 1631: Judging System Evaluation

ISU Communication 1631 details rules for evaluating figure skating judging and technical decisions, including OAC procedures and mathematical criteria for anomalies.

studylib.net

Interestingly, all the marks given by this judge to the three U.S. ladies in the competition were "within the corridor" -- that is, they were not so far out of line as to trigger an automatic review by the OAC (Officials Assessment Committee). Evidently a separate "ethics compliant" was filed.

As for the defendant's bombast, "How dfare you criticze me -- I am the best figure skating official in the history of figure skating officialdom" (sound familiar? ) -- I am not a lawyer, but I do not think that this line of defense iis likely to impress the court. I think a better tack for the defense would be that the ISU did not follow its own official assessment tool.

 

[el henry]

whole post

Thank you, I was hoping you would post because I knew with my inch wide and inch deep knowledge I was missing something.

I agree, I don't think that "I am the greatest who ever was" will impress the judges, and clearly it did not. However, as far as I can tell from this decision, it was one of many arguments. I didn't blame the respondent for throwing everything up against the wall, including overstated assessments of his own abilities, to see what would stick. Trying it all and seeing what will stick is a time honored legal tradition.

 

[Mathman]

OK, so I went back and looked at the numbers. I think that Mr. Williams has a case. He was not charged with low-balling any of the USA skaters' competitors, and he was not charged with any shenanigans in the short program.

The entire case against him seems to be that he scored Levito an average of 0.66 points above the rest of the panel on the three components composition, musical interpretation and performance in the LP. For Glenn it was an average of 0.73 points "too high," and for Tenell, 0.57. As far as I can see no other "evidence" of wrong-doing was presented.

Legally speaking, pretty thin gruel, if you ask me.

 

[4everchan]

honestly, if it were that thin, the ISU wouldn't not have exposed it. Fine, there are bigger cheaters... and fine, he didn't undermark competitors... but nationalistic boosting is exactly the new, softer kind of cheating we see around from some judges... and I am all for restricting these judges from the big events (and I am aware he just got a warning but that's fine too... warnings can make someone realise that they are out of line and be enough to fix the problem)

 

[Andrea82]

No, an ISU "deviation point" just means the difference between the errant judge's mark (for GOE on a specific element, for instance), and the mean score of the whole judging panel. It is not based on sample size. like a statistical standard deviation.

The big difference difference between assessing "anomalies" in GOEs compared to PCSs is that for GOEs both "too high" and "too low" count against the judge. For PCSs, scores that are too high and scores that are too low for a specific skater azre allowed to cancel, so you are OK if you score a skater too high on choreography but too low on musical interpretation, compared to the other judges.

I didn't read the most recent changes, but this is how they have always done it in the past. There are a couple of easy-to-follow examples given in ISU Connunication no. 1631, pages 5 and 6.

ISU Communication 1631: Judging System Evaluation

ISU Communication 1631 details rules for evaluating figure skating judging and technical decisions, including OAC procedures and mathematical criteria for anomalies.

studylib.net

Interestingly, all the marks given by this judge to the three U.S. ladies in the competition were "within the corridor" -- that is, they were not so far out of line as to trigger an automatic review by the OAC (Officials Assessment Committee). Evidently a separate "ethics compliant" was filed.

As for the defendant's bombast, "How dfare you criticze me -- I am the best figure skating official in the history of figure skating officialdom" (sound familiar? ) -- I am not a lawyer, but I do not think that this line of defense iis likely to impress the court. I think a better tack for the defense would be that the ISU did not follow its own official assessment tool.

The new OAC assessment procedure is here: https://www.isu.org/figure-skating/rules/fsk-communications/31545-isu-communication-2583/file
But it is similar to that described in your link, just with changes in the deviation allowed for PCS because there are only 3 categories now

The "How dare you criticze me -- I am the best figure skating official in the history of figure skating officialdom" defense was unproductive....they turned it against him. At one point, in the decision, they say that as he was such an experienced judge as he said, the marks could have not been due some errors and so it is national bias....

That's because in one previous case (ISU vs Pethes), the judge got away by arguing his marks weren't due to national bias but to mistakes because he wasn't used to judge Pairs and didn't know how to assign correct GOEs.

honestly, if it were that thin, the ISU wouldn't not have exposed it. Fine, there are bigger cheaters... and fine, he didn't undermark competitors... but nationalistic boosting is exactly the new, softer kind of cheating we see around from some judges... and I am all for restricting these judges from the big events (and I am aware he just got a warning but that's fine too... warnings can make someone realise that they are out of line and be enough to fix the problem)

The ISU technical committees actually already lost 2 cases in which they accused a judge of national bias.
1) Michela Cesaro for overscoring Portesi/Chrastecky and underscoring Terreaux/Perron at 2020 Junior Worlds. She based her defense on trying to explain the marks given by referencing to guidelines. She was helped by the fact that some bullet points (the element reflects nuances in the music) were a bit vague....
The Disciplinary Commission concluded that there were suspicion of hidden preferences but it couldn't be proved

https://www.isu.org/inside-isu/legal/disciplinary-decisions/24798-case-2020-02-isu-vs-cesaro-final-decision-for-publication-01-08-2020/file

2) Akos Pethes for overscoring Ioulia Chtchetinina/Mark Magyar and Maria Pavlova/ Balazs Nagy at 2022 European Championships. His scoring sheets was full of bizarre GOEs on twist lifts, lifts and death spirals...so his defense was "The Alleged Offender admits the anomalies and provides a consistent explanation, citing a lack of experience in judging Pair skating." Incompetent yes, biased no! Absolved.

https://www.isu.org/inside-isu/legal/disciplinary-decisions/28500-case-2022-01-isu-vs-pethes/file

 

[4everchan]

The new OAC assessment procedure is here: https://www.isu.org/figure-skating/rules/fsk-communications/31545-isu-communication-2583/file
But it is similar to that described in your link, just with changes in the deviation allowed for PCS because there are only 3 categories now

The "How dare you criticze me -- I am the best figure skating official in the history of figure skating officialdom" defense was unproductive....they turned it against him. At one point, in the decision, they say that as he was such an experienced judge as he said, the marks could have not been due some errors and so it is national bias....

That's because in one previous case (ISU vs Pethes), the judge got away by arguing his marks weren't due to national bias but to mistakes because he wasn't used to judge Pairs and didn't know how to assign correct GOEs.


The ISU technical committees actually already lost 2 cases in which they accused a judge of national bias.
1) Michela Cesaro for overscoring Portesi/Chrastecky and underscoring Terreaux/Perron at 2020 Junior Worlds. She based her defense on trying to explain the marks given by referencing to guidelines. She was helped by the fact that some bullet points (the element reflects nuances in the music) were a bit vague....
The Disciplinary Commission concluded that there were suspicion of hidden preferences but they couldn't be proved

https://www.isu.org/inside-isu/legal/disciplinary-decisions/24798-case-2020-02-isu-vs-cesaro-final-decision-for-publication-01-08-2020/file

2) Akos Pethes for overscoring Ioulia Chtchetinina/Mark Magyar and Maria Pavlova/ Balazs Nagy at 2022 European Championships. His scoring sheets was full of bizarre GOEs on twist lifts, lifts and death spirals...so his defense was "The Alleged Offender admits the anomalies and provides a consistent explanation, citing a lack of experience in judging Pair skating." Incompetent yes, biased no! Absolved.

https://www.isu.org/inside-isu/legal/disciplinary-decisions/28500-case-2022-01-isu-vs-pethes/file

incompetence... let me try this when I do my job badly and see how my boss absolves me

 

[Miller]

Douglas Williams (USA) is found guilty of national bias for his scoring of Levito, Glenn and Tennell at 2023 Worlds. However, he wasn't found guilty of systematically under-marking their competitors.
So he gets away with just a warning, also because he didn't get previous warnings (unlike some of the judges who were recently suspended who had already had previous warnings for their judging).

https://www.isu.org/inside-isu/isu-communications/communications/32839-case-2023-03-isu-v-d-williams-final-decision/file

OK, so I went back and looked at the numbers. I think that Mr. Williams has a case. He was not charged with low-balling any of the USA skaters' competitors, and he was not chargerd with any shenanigans in the short program.

The entire case against him seems to be that he scored Levito an average of 0.66 points above the rest of the panel on the three components composition, musical interpretation and performance in the LP. For Glenn it was an average of 0.73 points "too high," and for Tenell, 0.57. As far as I can see no other "evidence" of wrong-doing was presented.

Legally speaking, pretty thin gruel, if you ask me.

I think the key as to whether he has done anything wrong is contained within paragraph 45 which shows the marks he had effectively given to skaters as if he had judged the whole competition by himself e.g. Levito 143.33 vs 134.62 actual, and Glenn 131.63 vs 122.81 actual.

However to get a fuller picture you really need to go to SkatingScores.com. That way you can see how his marks compare with how other judges scored their own competitors, plus you can also see what happened in the SP where he also scored the American competitors.

This is the SP https://skatingscores.com/2223/wc/sr/women/i/short/tss/ and this is the LP https://skatingscores.com/2223/wc/sr/women/i/long/tss/.

So this is the list of competitors where their own countries' judge was on the panel and the effective score they were given (after converting GOEs, PCS components etc.) compared with the actual, and differential (to nearest whole or half mark).

SP
Levito 75.09 vs 73.03 (+2)
Tennell 69.10 vs 66.45 (+2.5)
Glenn 67.35 vs 65.52 (+2)

H Lee (KOR) 75.67 vs 73.63 (+2)
C Kim 67.57 vs 64.06 (+3.5)
Y Kim 62.45 vs 60.02 (+2.5)

Petrokina (EST) 70.16 vs 68.00 (+2)

Kurakova (POL) 67.07 vs 65.69 (+1.5)

Mikutina (AUT) 60.27 vs 57.05 (+3)

Feigin (BUL) 57.06 vs 54.65 (+2.5)

Van Zundert (NED) 59.07 vs 57.56 (+1.5)

LP
Levito 143.33 vs 134.62 (+9)
Tennell 123.28 vs 117.69 (+5.5)
Glenn 131.63 vs 122.81 (+9)

Petrokina (KOR judge only judged SP, not the LP) 126.98 vs 125.49 (+1.5)

Feigin 107.70 vs 101.09 (+6.5)

Van Zundert 98.78 vs 101.99 (-3)

Repond (SUI) 130.00 vs 131.34 (-1.5)

Sakamoto (JPN) 150.96 vs 145.37 (+5.5)
Mihara 137.15 vs 132.24 (+5)
Watanabe 134.94 vs 131.91 (+3)

Gubanova (GEO) 122.01 vs 119.52 (+2.5)

Conclusion.
You can see that the American judge clearly scored his own skaters higher than the average, especially in the LP. However you can also see that judges from other countries also scored their own skaters higher than the average (of the 16 non-US skater performances, 14 were scored higher than the average, and 2 below). Hence there is a problem across the board, it's just that the US judge was 'too far out', whereas presumably the Japanese and Korean judges were deemed OK (unless there's warnings we don't know about), despite each having 3 skaters scoring higher than the average - I guess there's a 1 in 8 chance of this happening at random, so perhaps nothing definite, whereas the American had 6 scores higher than the average.

Also, was the American judge just scoring higher across the board in the LP and so making his higher scores for the American skaters seem higher than what they were. I looked at his effective scores for the other top 12 skaters in the LP other than Isabeau Levito, and his average score was indeed 2 points higher than the average of the other judges, so he was scoring higher, but not enough to explain the difference. The other judges possibly at question i.e. the Japanese and Korean (and where there's a 'reasonable' sample size) were fine BTW - the Japanese judge scored slightly high in the LP - while the US judge was OK in the SP (but for his own skaters).

N.B. I would have thought this subject (national bias in figure skating/judges being allowed to judge their own countries' skaters) should be worthy of its own thread, rather than buried in one about ISU officials - anyone prepared to split it out?

 

[el henry]

incompetence... let me try this when I do my job badly and see how my boss absolves me

But incompetence would absolutely be a defense against discrimination, at least disparate treatment and in US Law. Of course I do not pretend to know anything about Canadian law.

Disparate impact would require many more examples than we have here.

I do not know if Mr. Williams was discriminating, the evidence seemed thin to me but more importantly to those more learned in mathematics than I am. The ISU hasn't "exposed" anything here except a few scoring differences that can be easily explained. If they have actual evidence of actual bias (disparate treatment), I would love to see it.

 

[Ic3Rabbit]

His level of self-esteem is quite something: The Alleged Offender, in the Statement of Response prepared on his behalf, describes himself as “one of the finest examples of a skating official that the ISU system has ever and could ever produce”.

honestly, if it were that thin, the ISU wouldn't not have exposed it. Fine, there are bigger cheaters... and fine, he didn't undermark competitors... but nationalistic boosting is exactly the new, softer kind of cheating we see around from some judges... and I am all for restricting these judges from the big events (and I am aware he just got a warning but that's fine too... warnings can make someone realise that they are out of line and be enough to fix the problem)

Except, according to Andrea82's quote, the guy is full of himself. So I don't see him trying to change, b/c he doesn't seem to think he did anything wrong.

 

[4everchan]

Except, according to Andrea82's quote, the guy is full of himself. So I don't see him trying to change, b/c he doesn't seem to think he did anything wrong.

Of course, he is probably not going to change, but if it happens enough, they will be able to restrict him.

 

[4everchan]

But incompetence would absolutely be a defense against discrimination, at least disparate treatment and in US Law. Of course I do not pretend to know anything about Canadian law.

never heard about this as a valid defense against discrimination in Canada.

Disparate impact would require many more examples than we have here.

I do not know if Mr. Williams was discriminating, the evidence seemed thin to me but more importantly to those more learned in mathematics than I am. The ISU hasn't "exposed" anything here except a few scoring differences that can be easily explained. If they have actual evidence of actual bias (disparate treatment), I would love to see it.

I think the ISU did what they needed to do. They created evidence they can use later if it keeps going on.

 

[Alex Fedorov]

Generally speaking, for biased judge not nesessary overestimate “his” skaters. It is enough for him to underestimate the most dangerous of rivals. For example, the American judge gives fair score to his skaters, but underestimates Kaori Sakamoto's score.

 

[Magill]

In my simple-minded opinion, whether a judge was misjudging due to national bias or general incompetence is a very interesting question for drawing disciplinary conclusions and lessons learnt from his case for future but the outcome should really be the same in both cases - he should not be allowed to judge any more. I mean, is it better to be judged by a judge who is biased or the one who is incompetent? I guess, depending on the nationality of the judge and the one who is judged....

 

[Mathman]

never heard about this as a valid defense against discrimination in Canada.

I think that in most countries there are certain crimes where the prosecution is tasked with the burden of establishing malicious intent. This is certainly the case in some very, very, very high-profile political cases working their way through the U.S. courts right now.

However, what we havve here is not a criminal matter. The ISU cannot put anyone in jail for being a bad judge. The ISU is a private entity -- sort of. The judges serve at the pleasure of the organization. There are labor laws in various countries that speak to unjust termination, etc., but I can't see anything like that being of legal relevance here, especially since the employee was only given a slap on the wrist. Not to mention questions of jurisdiction.

There is, however, something of a "sauce for the goose, sauce for the gander" aspect. We expect skaters who feel themselves to be the victims of unfair judging to suck it up, grin and bear it, and say "I will work harder next time." Why shouldn't we expect the same of judges?

 

[Mathman]

Generally speaking, for biased judge not nesessary overestimate “his” skaters. It is enough for him to underestimate the most dangerous of rivals. For example, the American judge gives fair score to his skaters, but underestimates Kaori Sakamoto's score.

I think that the lowballing of other skaters is monitored and identified by the same ISU procedures as overscoring is. (Although the perticular example that you give is not the most pressing one, since USA does not have any women at the moment who rival Sakamoto, cheat or no cheat.)

 

[4everchan]

I think that in most countries there are certain crimes where the prosecution is tasked with the burden of establishing malicious intent. This is certainly the case in some very, very, very high-profile political cases working their way through the U.S. courts right now.

However, what we havve here is not a criminal matter. The ISU cannot put anyone in jail for being a bad judge. The ISU is a private entity -- sort of. The judges serve at the pleasure of the organization. There are labor laws in various countries that speak to unjust termination, etc., but I can't see anything like that being of legal relevance here, especially since the employee was only given a slap on the wrist. Not to mention questions of jurisdiction.

There is, however, something of a "sauce for the goose, sauce for the gander" aspect. We expect skaters who feel themselves to be the victims of unfair judging to suck it up, grin and bear it, and say "I will work harder next time." Why shouldn't we expect the same of judges?

As far as I remember, judges are volunteers. There is no termination of employment involved. It's not about labour law, it's not about discrimination... it's just about someone not doing their task properly.

I have had to deal with volunteers. When they do not do the task they are supposed to do, we don't call them again simple as that. They don't have a "right" to be a volunteer

 

[Mathman]

I think the key as to whether he has done anything wrong is contained within paragraph 45 which shows the marks he had effectively given to skaters as if he had judged the whole competition by himself e.g. Levito 143.33 vs 134.62 actual, and Glenn 131.63 vs 122.81 actual.

That is a very cool metric.

Another would be: suppose that we replace all of the errant judge's marks with the average marks of the panel. What effect would that have?

I just did the PCSs for Levito. Her Total (factored) PCS for the LP was 64.84. If the U.S. judge had dropped dead before the competition started and been replaced by a robot that just automatically regurgitated the average marks of the panel, it would have been 68.56 (still edging out Mihara by a few hundredths). The reason that the numbers hardly change is that all of Mr. Williams marks were discarded anyway.

There is also the overall question, if the judging were "perfect," would that equate to "all judges gave exactly the same marks?" Or would it instead be a sign that there are unhealthy forces at play?

A l,ittle off topic, but in discussions of whether judging bias was worse under the 6.0 system than it is now, I do think that there has been two areas of improvement. We don't see much evidence of bloc judging as was constantly alleged in the past, and we don't see as much tit-for-tat -- I'll boost your lady if you recoiprocate by boosting my dancers. Maybe the all-powerful federations are not as all-powerful as the were 30 years ago.

However to get a fuller picture you really need to go to SkatingScores.com. That way you can see how his marks compare with how other judges scored their own competitors, plus you can also see what happened in the SP where he also scored the American competitors.

This is the SP https://skatingscores.com/2223/wc/sr/women/i/short/tss/ and this is the LP https://skatingscores.com/2223/wc/sr/women/i/long/tss/.

So this is the list of competitors where their own countries' judge was on the panel and the effective score they were given (after converting GOEs, PCS components etc.) compared with the actual, and differential (to nearest whole or half mark).

...

Levito 143.33 vs 134.62 (+9)
Tennell 123.28 vs 117.69 (+5.5)
Glenn 131.63 vs 122.81 (+9)

Thank you for these numbers and the link. All the other judges seemed content with a 2 or 3 point boost to skaters from their own country. This seems normal and in fact can easily be accounted for by benign factors, like, these skaters are more familiar to the judge. +9 does stand out as excessive. Moderation in all things, even national bias!

 

[Alex Fedorov]

I think that the lowballing of other skaters is monitored and identified by the same ISU procedures as overscoring is. (Although the perticular example that you give is not the most pressing one, since USA does not have any women at the moment who rival Sakamoto, cheat or no cheat.)

This example was hypothetical, of course. But in any case, I think that deliberate underestimation is more difficult to track than overestimation. In addition, when viewing some protocols, you can see “altruistic judges” who are trying not so much to help their compatriots (sometimes there is simply no one to help), but to reduce the scores of athletes from not favorite country.

For example, judge number 7 at the Olympics clearly deliberately tried to “help” Kaori Sakamoto and at the same time, if possible, reduce the scores of Shcherbakova and Trusova. This is especially noticeable in the PCS section, but the same trend is also visible in the GOE. And this judge is not from Japan.