Scoring bias at the national level

[4everchan]

Except that that is specifically what did not happen in this case. At multiple points during the explanation in the document, you can find the very specific wording of “While his marks were in the corridor, his marks were overall higher" and it even outright states "A Judge’s marks need not be outside the corridor before they can be considered as bias." Doug Williams was found guilty not of judging outside of the corridor, but of a violation of the duties of judges and the ISU Code of Ethics.

For those of you who are interested in how and why his scores got flagged despite that, here's a document explaining the work of the Officials Assessment Commission. Very fascinating read, in my opinion.

interesting read.
How many errors would have Fortin made then. Just curious. (seems that one is obvious with the twist, are the others obvious?)

 

[gkelly]

I think that the whole business of "program components" is fighting with itself. Are we talking about the program taken as a whole (like the old second mark in 6.0) or are we talking about "components" of the proogam (as in, a triple Lutz is one compement on the tech side)?.

What the PCS were intended to measure were various aspects of the program as a whole.

The elements, such as jumps, are considered elements and are reflected in the Total Elements Score.

The ISU has never used the word "component" to refer to discrete pieces of the whole program, such as a single move. Although that word might be used in a similar sense in non-skating contexts, it only muddies the discussion to redefine the word in a way that has never been used in skating.

The program component scores include everything that happens from when the music and movement start to when they stop. The 5, or now 3, different scores each reflect different aspects of the program as a whole.

I think an argument could be made for having only two compnents.. The etchnical aspects of the program as a whole (Skating Skills including the old Transitions) and the artistic/performance aspects of the4 program as a whole.

You could do it that way. But that would give the skaters as well as the public less information about what the judges were thinking when awarding the component scores, and it would give the judges less ability to make fine distinctions between skaters who were close in overall ability but maybe had different strengths and weaknesses.

I think the intention of changing how the components are judging is good, but the problem is how that idea is translated in the common practice... For example i would propose for components to be judged as first related to the ice (with your blades), second related to the public (audience/judges) and the public space (ice rink) and third related to the music and overall storytelling, but the question is how to manage that idea to work the best in everyday practice.

Another possible approach. You could rename and redefine three components to cover those three areas of general qualities to be judged.

From where we are now, it could just be a matter of redistributing some of the current bullet points, or restoring and redistributing some of the criteria from the 5-component descriptions.

 

[Baron Vladimir]

interesting read.
How many errors would have Fortin made then. Just curious. (seems that one is obvious with the twist, are the others obvious?)

The problem with Doug Williams is his explanation that he is 'too good of a judge to make those mistakes' which only implies to me that every mistake he could make is made on purpose and its then some sort of bias If any judge realize that mistakes are inevitable part of the job and try to explain those 'mistakes', i see no problem. For start, just admit you can be wrong and apologize - it can be that simple...

 

[Jumping_Bean]

interesting read.
How many errors would have Fortin made then. Just curious.

Two in the SP (P/F 3LoTh, M/K 3LzTh), three in the Free (S/D 3Tw2 anf 3T+1A+1A, L/N 3STh). No issues in PCS.

(Though I must in general say, 4.5 points in raw PCS is a ridiculously large allowance, that translates to 12 points in actual PCS in the Free Skate for pairs. As the allowance even goes in either direction, judges with 24 (!!!) points of difference in PCS would both not be automatically flagged and be within the judging corridor )

For comparison:
In the SP: CHN and AUS had one error each. CRO, SVK, CAN and USA had two errors each. GER had three.
In the FS: JPN, CRO, SVK and AUS had one error each. CAN and SLO had three.

Just for fun (as this does not seem to be part of the described procedure):
Of the judges who scored both segments, only the Latvian judge (Agita Abele) did not have any errors. She still had significant deviations over 5 points from the average score for three pairs in particular (P/M, P/W, L/E).
Of the three judges with three errors in one segment, two of them only scored one of the segments (GER, SLO).
Of the other judges, across both segments, two had only one error in total (CHN, JPN), two had two (AUS, USA), two three (SVK, CRO) and CAN had 5 errors in total.

 

[snowed]

Well, I absolutely agree that just judging inside the corridor alone says nothing about how correct it is, but there is a difference if judge icewhite from Germany gives Trusova 5s in PCS and Mikutina 9s in PCS while everyone else gives Trusova 9s and Mikutina 7s, or if judge icewhite from Germany gives Hase/Volodin a 2 on their twist and a 0 on their throw while everyone else gives them -1 and -3, gives them higher PCS than everyone else, and is not as generous with the other teams, no?

I agree, this would look like national bias... I was trying to make a point aside from national bias

 

[Jumping_Bean]

The problem with Doug Williams is his explanation that he is 'too good of a judge to make those mistakes' which only implies to me that every mistake he can make is made on purpose and its then some sort of bias If any judge realize that mistakes are inevitable part of the job and try to explain those 'mistakes', i see no problem. For start, just admit you can be wrong and apologize - it can be that simple...

Yeah, he really shot himself in the foot with that "defense". While the OAC could show the scores and explain how they were biased, they wouldn't have been able to prove his intent, in which case he wouldn't have violated the ethics code, just his duties as a judge. Saying that he's too good to make mistakes is basically agreeing that any obvious deviations are not mistakes but on purpose.

 

[4everchan]

Two in the SP (P/F 3LoTh, M/K 3LzTh), three in the Free (S/D 3Tw2 anf 3T+1A+1A, L/N 3STh). No issues in PCS.

So, in the SP, Fortin advantaged an American team and a Japanese team.
In the LP, they had two elements giving an advantage to SD/D and one to an American team.

See what I mean with the "omg.. national bias". That judge made mistakes that advantaged 4 different pairs, two of which were from the USA

(Though I must in general say, 4.5 points in raw PCS is a ridiculously large allowance, that translates to 12 points in actual PCS in the Free Skate for pairs. As the allowance even goes in either direction, judges with 24 (!!!) points of difference in PCS would both not be automatically flagged and be within the judging corridor )

Generally large allowance is your opinion (which I could share... 10 points in the LP for me should be possible) but the ISU sets those guidelines, not us.

For comparison:
In the SP: CHN and AUS had one error each. CRO, SVK, CAN and USA had two errors each. GER had three.
In the FS: JPN, CRO, SVK and AUS had one error each. CAN and SLO had three.

Just for fun (as this does not seem to be part of the described procedure):
Of the judges who scored both segments, only the Latvian judge (Agita Abele) did not have any errors. She still had significant deviations over 5 points from the average score for three pairs in particular (P/M, P/W, L/E).

Did the Latvian judge underscore or overscore these teams ?

Of the three judges with three errors in one segment, two of them only scored one of the segments (GER, SLO).
Of the other judges, across both segments, two had only one error in total (CHN, JPN), two two (AUS, USA), two three (SVK, CRO) and CAN had 5 errors in total.

So the Canadian judge made the most errors. I am not familiar with Fortin. New judge ? What is their pedigree in terms of judging ?

See, I am finding this more interesting than simply making accusations of nationalistic bias. Especially when half the errors were not even favouring Canadian skaters.

 

[Baron Vladimir]

From where we are now, it could just be a matter of redistributing some of the current bullet points, or restoring and redistributing some of the criteria from the 5-component descriptions.

Even if we do that, it would still be a problem (i think) for one judge to judge all the elements and components at one point of time. The best solution i see is to redistribute what judges are judging. For example, 5 judges judging the elements, and 5 judges judging the components - programme as a whole in the short, and they all rotate their job in the free. Maybe to add one more 'artistic' bullet to the elements, and if its needed one more 'technical' aspect for components.

 

[Andrea82]

The tricky part is if you look at a skater who has strong skating skills (worthy of 7s or 8s -- and maybe earning 9s from judges who are especially impressed by successful difficult jumps, speed/power more than complexity, and possibly the skater's nationality) who does not use that power to execute varied or complex skating between the elements.

So if the transitional moves are minimal and simple, but performed at great speed, how should that be scored relative to skaters with average, fair, or poor skating skills performing similarly simple skating?

I would guess this has been a topic of discussion at judging seminars and roundtables, especially when there was a dedicated component for Transitions. Even if everyone agrees that the Transitions score should be lower than the Skating Skills score, judges might disagree about how much lower. 0.25 or 0.5? 1.0? 3.0 or 4.0?

Part of the differences among judges may have to do with how different individuals are neurologically wired to process numbers. Another part, especially when we're talking about new International and not already-ISU-level judges, might have to do with the range of skating ability they have the most experience judging.

They may discuss about it also at the initial judges meeting in major competitions as it is suggested in seminars for Referee (minute 10:00)

2019 Global Seminar - Referees - Program Components - Section 1 - Overview

www.youtube.com

(apart that she is wrong in saying at GPs you don't find low experience judges. Less frequently that in B competitions, but you may find 1-2 recently promoted judges also in GPs)

 

[Jumping_Bean]

So, in the SP, Fortin advantaged an American team and a Japanese team.
In the LP, they had two elements giving an advantage to SD/D and one to an American team.

See what I mean with the "omg.. national bias". That judge made mistakes that advantaged 4 different pairs, two of which were from the USA

So the Canadian judge made the most errors. I am not familiar with Fortin. New judge ? What is their pedigree in terms of judging ?

See, I am finding this more interesting than simply making accusations of nationalistic bias. Especially when half the errors were not even favouring Canadian skaters.

As we already discussed, errors that would be flagged automatically in a review are not necessary for bias. Something the Latvian judge also showed, by not having any errors but still significant deviations in final scores.
The Aussie judge too, by only having one error in the SP (-2 on the 3LzTh of M/K) and still managing to score Hektor and Anastasia over 4 points higher than the average.
Take a look at this chart and you'll see why people were zeroing in on Fortin in the Free. (Still, the funniest judge in the Japanese one, who somehow managed to almost evenly match their negative bias in the Short against M/K with their positive bias in the Free )

The Slovenian judge is interesting too, he really liked the younger Canadian teams (+5.87 for P/M and +10.57 for L/E) and Z/Y (+6.11), and did not like P/F (-5.86). And the German SP judge? Hated everyone (to varying degrees, some more, some less), except for SD/D & G/GM.

Did the Latvian judge underscore or overscore these teams ?

Under. She didn't particularly like many of the other teams here either, her biggest "boost" to a team's score was +2.3 for Z/Y in the Short and +2.15 for W/Z in the Free. Of the top 6 teams, they only gave higher than average scores to M/K (+0.02 in the Short, +1.35 in the Free). Basically, she said f*ck you to most teams, and looked at three teams in particular and said "You know what, f*ck you in particular".

 

[gkelly]

Even if we do that, it would still be a problem for one judge to judge all the elements and components at one point of time. The best solution i see is to redistribute what judges are judging. For example, 5 judges judging the elements, and 5 judges judging the components - programme as a whole in the short, and they all rotate their job in the free. Maybe to add one more 'artistic' bullet to the elements, and if its needed one more 'technical' aspect for components and that's it.

If the panel is to be split, my suggestion would be to have the elements judges also judge the Skating Skills (which may also include technical aspects of what used to be covered under Transitions), and then have the other panel look at what's now Composition and Presentation.

 

[Andrea82]

So, in the SP, Fortin advantaged an American team and a Japanese team.
In the LP, they had two elements giving an advantage to SD/D and one to an American team.

See what I mean with the "omg.. national bias". That judge made mistakes that advantaged 4 different pairs, two of which were from the USA

Generally large allowance is your opinion (which I could share... 10 points in the LP for me should be possible) but the ISU sets those guidelines, not us.

Did the Latvian judge underscore or overscore these teams ?

So the Canadian judge made the most errors. I am not familiar with Fortin. New judge ? What is their pedigree in terms of judging ?

See, I am finding this more interesting than simply making accusations of nationalistic bias. Especially when half the errors were not even favouring Canadian skaters.

Glenn Fortin has been promoted from International to ISU level in October 2023. So 4CCs was his first ISU Championship judging.
His wife is also an ISU judge. They are from Ontario.

 

[Jumping_Bean]

Also, just in case someone isn't yet aware: SkatingScores is a really helpful website, in general and if you want to dig a bit deeper into the scoring system. When you click on GOE total in any of the protocols, you are even shown the mean GOE and errors as described in the OAC document are marked, and by clicking on the judges at the top (from the 2016/17 season on), you can see how they alone scored the event (including the deviation from the mean).

 

[icewhite]

So, in the SP, Fortin advantaged an American team and a Japanese team.
In the LP, they had two elements giving an advantage to SD/D and one to an American team.

See what I mean with the "omg.. national bias". That judge made mistakes that advantaged 4 different pairs, two of which were from the USA

Generally large allowance is your opinion (which I could share... 10 points in the LP for me should be possible) but the ISU sets those guidelines, not us.

Did the Latvian judge underscore or overscore these teams ?

So the Canadian judge made the most errors. I am not familiar with Fortin. New judge ? What is their pedigree in terms of judging ?

See, I am finding this more interesting than simply making accusations of nationalistic bias. Especially when half the errors were not even favouring Canadian skaters.

It's not just "simply making accusations of nationalistic bias", though. There are patterns to be found.
I admit, I love finding patterns in numbers and explanations for them.
Of course all possibilities and probabilities have to be explored and more analysis and comparisons have to be made to make a claim with substance.

Still, what I see in this thread are (to me) weird generalizations that don't look at the details more closely, but less.
Nationalistic "bias", let's use that term for better or worse, happens, deliberate efforts to score athletes higher or lower happen, and there are patterns to be found - the question to me is what these look like, not whether it's worth it to even care.
Sure, there are a lot more important topics in the world, and even in figure skating judging I think there are more important subjects, because yes, the system already targets this kind of judging pretty well.
Still, it should not happen, and if there are cases which can be detected, they should be detected (or better prevented).

 

[snowed]

I got exactly the opposite impression. The judge's marks were actually not outside the corridor and were not flagged as such by the ISU judges' oversight apparatus. The one rule that was broken was a component score of 9.50 to a performace that had a fall. This brought the complaint not to the corridor monitors but to the ISU Technical Committee and from there to the Ehhics Committee that investigates dishonesty and dishonorable activity.

They used this just for Levitto

If most judges on the panel gave skater A an average of 7.0 for all 5 former components,

Judge X who gave that skater scores of 7.5 6.75 7.0 6.75 7.0 would be right inside the corridor.

But Judge Y who gave 8.0 5.0 7.5 7.0 7.5 would also be right inside as well. Same total, different distribution.

And there was a leeway of up to 2.0 total IIRC, so even 7.5 5.0 7.0 6.5 7.0 would not get flagged as anomalous.

All the more reason to encourage judges to separate the different components if they thought there was a significant difference between, in this case, the skating skill and the transitions and program construction. But most judges were not brave enough to go there.

I used "corridor judging" for lack of a different word to express the judges desire to be similar with the rest of the panel so I'll call it "similar judging" not as ISU describe and calculates the "corridor".

from the document explaining the decision
While his marks were in the corridor, his marks were overall higher as reflected in the total
score”. In their opinion he “over-marked in Composition and Skating skills. Conclusion:
National Bias”

And I'll rephrase what I wrote before:
This is exactly where I'm conflicted, the judge that started this thread is was proven in my eyes, to judge not similar with the panel, (he broke the the 9.50 rule I stand corrected). No other rules (scores for specific elements or components) were discussed in the ruling. Explain him, and us the fans, why he's marks are not ok.
So if you were a judge, would you give 4 to Trusova, knowing that the other judges will give 7?

I'll copy from my previous post #42
looking at the marks of the american judge on women FS, he seem to have a wider range than the other judges, here is his overall points compared with what the panel averaged, in order of placement 1)Haein Lee +5, 2)Kaori similar, 3) Chaeyeon Kin +4, 4)Leona Hendrix +1 5) Isabeau +9 Mai 6)Mihara +4, many +/- up to 5 that I won't mention,10)Nina Pinzarone +10 14)Amber Gken+9, 1)Bradie +6 21)Janna -9 23)Alexandra Fegin -11 24)sofia sepchenko -11

His marks were overall higher or lower "as reflected in the total score" for other skaters too. There were recent example of scores varying with more than 15 points, so are those fine because they were not given to their own skaters?

I'm just trying to make the point that I wish for judges to judge correctly (in general and including their own skaters of course), and to not be pressured to judge similarly.

 

[4everchan]

As we already discussed, errors that would be flagged automatically in a review are not necessary for bias. Something the Latvian judge also showed, by not having any errors but still significant deviations.
The Aussie judge too, by only having one error in the SP (-2 on the 3LzTh of M/K) and still managing to score Hektor and Anastasia over 4 points higher than the average.
Take a look at this chart and you'll see why people were zeroing in on Fortin in the Free. (Still, the funniest judge in the Japanese one, who somehow managed to almost evenly match their negative bias in the Short against M/K with their positive bias in the Free )

The Slovenian judge is interesting too, he really liked the younger Canadian teams (+5.87 for P/M and +10.57 for L/E) and Z/Y (+6.11), and did not like P/F (-5.86). And the German SP judge? Hated everyone (to varying degrees, some more, some less), except for SD/D & G/GM.


Under. She didn't particularly like many of the other teams here either, her biggest "boost" to a team's score was +2.3 for Z/Y in the Short and +2.15 for W/Z in the Free. Of the top 6 teams, they only gave higher than average scores to M/K (+0.02 in the Short, +1.35 in the Free). Basically, she said f*ck you to most teams, and looked at three teams in particular and said "You know what, f*ck you in particular".

So it seems that most judges are all over the place... Now if a judge is scoring everyone super low, in theory, a judge who usually scores everyone super high, could look like they're favouring teams and vice versa... Deviation is therefore extremely variable.

 

[4everchan]

Glenn Fortin has been promoted from International to ISU level in October 2023. So 4CCs was his first ISU Championship judging.
His wife is also an ISU judge. They are from Ontario.

Thank you. I was aware that Fortin would have judged inferior level so thanks for letting us know about this promotion.

Let's see how long he lasts. I suspect one bad day at the office is okay.

 

[Baron Vladimir]

If the panel is to be split, my suggestion would be to have the elements judges also judge the Skating Skills (which may also include technical aspects of what used to be covered under Transitions), and then have the other panel look at what's now Composition and Presentation.

I dont think the split on 'artistic' and 'technical' judges would work well, just because it is too much of a split Figure skating is after all a cohesion of all those things, so i would still keep judges to judge more different aspects of it.

 

[icewhite]

They used this just for Levitto

I used "corridor judging" for lack of a different word to express the judges desire to be similar with the rest of the panel so I'll call it "similar judging" not as ISU describe and calculates the "corridor".

from the document explaining the decision
While his marks were in the corridor, his marks were overall higher as reflected in the total
score”. In their opinion he “over-marked in Composition and Skating skills. Conclusion:
National Bias”

And I'll rephrase what I wrote before:
This is exactly where I'm conflicted, the judge that started this thread is was proven in my eyes, to judge not similar with the panel, (he broke the the 9.50 rule I stand corrected). No other rules (scores for specific elements or components) were discussed in the ruling. Explain him, and us the fans, why he's marks are not ok.
So if you were a judge, would you give 4 to Trusova, knowing that the other judges will give 7?

I'll copy from my previous post #42
looking at the marks of the american judge on women FS, he seem to have a wider range than the other judges, here is his overall points compared with what the panel averaged, in order of placement 1)Haein Lee +5, 2)Kaori similar, 3) Chaeyeon Kin +4, 4)Leona Hendrix +1 5) Isabeau +9 Mai 6)Mihara +4, many +/- up to 5 that I won't mention,10)Nina Pinzarone +10 14)Amber Gken+9, 1)Bradie +6 21)Janna -9 23)Alexandra Fegin -11 24)sofia sepchenko -11

His marks were overall higher or lower "as reflected in the total score" for other skaters too. There were recent example of scores varying with more than 15 points, so are those fine because they were not given to their own skaters?

I'm just trying to make the point that I wish for judges to judge correctly (in general and including their own skaters of course), and to not be pressured to judge similarly.

My working thesis at the moment is that giving higher marks to most and having a generally bigger difference in scores can be used as a technique to disguise the pushing of your favoured team a bit. Would have to look at that much closer, though.

 

[snowed]

Except that that is specifically what did not happen in this case. At multiple points during the explanation in the document, you can find the very specific wording of “While his marks were in the corridor, his marks were overall higher" and it even outright states "A Judge’s marks need not be outside the corridor before they can be considered as bias." Doug Williams was found guilty not of judging outside of the corridor, but of a violation of the duties of judges and the ISU Code of Ethics.

For those of you who are interested in how and why his scores got flagged despite that, here's a document explaining the work of the Officials Assessment Commission. Very fascinating read, in my opinion.

If you'll look at my previous post to me it still sounds that he was found guilty of "not judging similarly with the other judges", and I would rather have a commission take element by element and pointing out exactly which element/component was judged incorrectly and why, instead of referring his total score being higher than the average.