Judging bias on the Grand Prix post CoP revision (numbers!)

[Draculus]

Correlation may not necessarily be causation, but that doesn't mean it can't be causation either. Can you come up with a competing explanation for why, for instance, Olga Kozhemyakina (I pick on this judge a lot, but she's a good example because she judges a lot of competitions), consistently overscores Russian skaters and underscores non-Russian skaters other than nationalistic bias?

What do these skaters have in common that could explain her overscoring, other than them skating for the Russian federation?

"Russian style"? Skaters from the same country might have similarities in their styles, skaters and coaches from the same country see each other more often, they have local competitions (and a lot in Russian case), even common training camps/schools.
So Russian judge might just like "Russian style of figure skating". F.e. all Eteri skaters definitely have a lot in common. For me definitely there is a difference between any Russian girl and any Japan girl, do not know about Men/pairs/dance.
Even more, judge might prefer Russians just because they are Russians AND because he/she likes "Russian style". And separate completely legal bias (like that style) from flag-related (OUR figure skater!) bias is not an easy task to do even for the judge herself, if she ever wants to perform such self-reflection.

 

[Miller]

Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

https://www.buzzfeednews.com/article/johntemplon/the-edge

 

[Metis]

Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

https://www.buzzfeednews.com/article/johntemplon/the-edge

Even better: you can read the underlying research from economics professor Eric Zitzewitz, which has been the data-driven elephant in the room since 2010: it’s extremely readable!

 

[Shanshani]

"Russian style"? Skaters from the same country might have similarities in their styles, skaters and coaches from the same country see each other more often, they have local competitions (and a lot in Russian case), even common training camps/schools.
So Russian judge might just like "Russian style of figure skating". F.e. all Eteri skaters definitely have a lot in common. For me definitely there is a difference between any Russian girl and any Japan girl, do not know about Men/pairs/dance.
Even more, judge might prefer Russians just because they are Russians AND because he/she likes "Russian style". And separate completely legal bias (like that style) from flag-related (OUR figure skater!) bias is not an easy task to do even for the judge herself, if she ever wants to perform such self-reflection.

I notice you conveniently left out the list of skaters I referenced. Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin? These skaters encompass a diverse range of program types, skating styles, and skating schools. Personally, even within the same discipline I don't find Voronov and Aliev to be similar skaters at all. If they were all from the same school, maybe you would have more of a point, but skaters from different schools have very distinct styles.

To take an example from ladies since it's the discipline you're most familiar with, sure Eteri skaters may have similar styles, but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well. The idea that she and Alina share the same 'Russian style' seems pretty silly to me--they are more different than they are the same. But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).

And to conclude, I think I will just quote Miller:

Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

https://www.buzzfeednews.com/article/johntemplon/the-edge

 

[Draculus]

Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin?

I have no idea. I do not see why not, if judge doing it for 20 years... Human brain is very good at finding similarities. Even when it is wrong. And with huge experience she might see something I do not.

but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well.
The idea that she and Alina share the same 'Russian style' seems pretty silly to me-

Its fine. But you switching registers. In a world of correlations, statistics and P-value there is no place for "silly". Either we take common sense - and I completely agree, that there are judges who are nationality-biased. Russian nationalism is quite strong. And numbers are not required in this talk, they help but just a little.
Or we take statistics, P-value of the hypothesis, etc. In this world to correctly calculate P-Value it is required to go through all variants. And to decline "russian style" something more that "silly" is required.
For me personally in GPF last years there was a cluster of "russian ladies". May be it is silly, may be in blind experiment my guess russian/not russian would be 50%. But it is impossible to make such experiment so I can just give my opinion, that success rate would be >50%.

 

[Miller]

But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).

She did however do Ondrej Nepela - Rika Kihira 135.39 for her 8 triple, 2 Triple Axel free skate, clean but for a mistake on the final 3S (GOE -0.77 from the judges, so a 2 point swing compared with normal).

Also I notice she is on the list of 16 worst offenders in the Buzzfeed article, as is Yuri Guskov of Kazakhstan (Elizabet Tursynbaeva was in strong contention after the SP) - he gave Rika 137.25, plus the difference between Rika's Free Skate and Elizabet's was only 11.3/11.4 points from both judges despite Elizaveta having 3 URs and a fall in her 7 triple, non-3A Free Skate - http://skatingscores.com/2019/cssvk/ladies/long/tss/

Also this is how the list of 16 worst judges on the Buzzfeed article are performing this year - note that Shanshani's database in the OP doesn't include all competitions, but it is noticeable how one or two seem to be keeping a low profile - I wonder if the article has had some effect on some federations. I have also numbered the number of own skaters and other nationality skaters so you can see how much of a sample size we're talking about, the bigger the sample size the more notable the results obviously.

The points difference is that between how they scored their own countries skaters vs other country skaters i.e. column E in the judges database - https://docs.google.com/spreadsheets/d/1j7A84BTjIkDZdYNBhihzNWhTTR0T8K5vn5mPwJJKQ7k/edit#gid=0

Canada - Nicole Leblanc-Richard - no main competitions judged (2 Challengers, 1 JGP)
Canada - Jeffrey Kukasik - no main competitions judged, 1 Challenger

China - Weiguang Chen - no competitions judged, currently banned
China - Feng Huang - no competitions judged, currently banned
China - Tianyi Zhang - no main competitions judged, 1 Challenger, 1 JGP

Israel - Anna Kantor - +13.49 (2 own skaters/40 other)

Italy - Walter Toigo - +7.82 (but +10.81 in only competition with Italian skater) - 1 own skater/19 other

Kazakhstan - Yuri Guskov - +8.04 (1 own/17 other)

Russia - Maira Abasova - +5.15 (6 own/36 other)
Russia - Elena Fomina - +11.79 (8 own/18 other)
Russia - Olga Kozhemyakina - +8.96 (11 own/40 other)

Spain - Marta Olozagarre - +3.21 (2 own/28 other)

Turkey - Tanay Ozkan Silaoglu - no competitions judged (Agafanova/Ucar retired)

United States - Lorrie Parker - no competitions judged
United States - Sharon Rogers - +5.20 (GP Finland, has also done 2 Challengers not in Shanshani's database)

Uzbekistan - Saodat Numanova - no main competitions (Misha Ge retired) - 1 JGP.

Interesting though how few competitions, relatively speaking, those judges specifically identified by the Buzzfeed News article have done.

 

[Andrea82]

Turkey - Tanay Ozkan Silaoglu - no competitions judged (Agafanova/Ucar retired)

She judged at Bosphorus Cup a couple of weeks ago. She will in 2019 European and World Championships panels as Turkey has been drawn and she is their only ISU qualified judge.

Israel - Anna Kantor - +13.49 (2 own skaters/40 other)

She will be at 2019 Worlds and Junior Worlds. Israel has been drawn for both Men and Pairs and they have 2 judges. So Kantor and Zaydman will one judge Men and other one Pairs. Alexey Beletsky will judge Ice Dance as Israel has been drawn also for ID at Worlds and he is the only one left.

Uzbekistan - Saodat Numanova - no main competitions (Misha Ge retired) - 1 JGP.

She will judge Pairs at 4 Continents. But there won't be skaters from Uzbekistan.

China - Feng Huang - no competitions judged, currently banned

Feng managed to get promoted as ISU Technical Controller this summer. I believe ISU suggested they won't nominate him this season

 

[Eclair]

@shanshani I've already told you on another forum how much I like the statistics you compiled (together with the judges NAMES)! Maybe you can consider adding another column with the deviation and bias of a judge for their countries top competitor. Because usually a judge will overscore their countries no 1 and 2 competitor, but won't do the same with skater no3. So the non-bias shown with no3 will then decrease the end bias in your table. And while some judges do overscore all their competitors, I usually don't care that for example the kazhakstan judge doesn't overscore their 2nd single ladies skater. What matters to me, is how much bias are they showing towards Tursynbava because she is their hope for a podium or top finish.

 

[moriel]

I notice you conveniently left out the list of skaters I referenced. Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin? These skaters encompass a diverse range of program types, skating styles, and skating schools. Personally, even within the same discipline I don't find Voronov and Aliev to be similar skaters at all. If they were all from the same school, maybe you would have more of a point, but skaters from different schools have very distinct styles.

To take an example from ladies since it's the discipline you're most familiar with, sure Eteri skaters may have similar styles, but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well. The idea that she and Alina share the same 'Russian style' seems pretty silly to me--they are more different than they are the same. But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).

And to conclude, I think I will just quote Miller:

While russian skaters are all different, i do feel that there are certain major "streams" such as "russian style" and "north american / western style".
There are some differences in actual skating, the program / choreography / interpretation and so on. Of course, it is a very wide thing, and there is a continuous transition between those two. Of course it is a very generic thing, as any binary classification.

I would say that this is somewhat more visible in pairs and dance rather than in singles, but well.

 

[Shanshani]

I have no idea. I do not see why not, if judge doing it for 20 years... Human brain is very good at finding similarities. Even when it is wrong. And with huge experience she might see something I do not.

Its fine. But you switching registers. In a world of correlations, statistics and P-value there is no place for "silly". Either we take common sense - and I completely agree, that there are judges who are nationality-biased. Russian nationalism is quite strong. And numbers are not required in this talk, they help but just a little.
Or we take statistics, P-value of the hypothesis, etc. In this world to correctly calculate P-Value it is required to go through all variants. And to decline "russian style" something more that "silly" is required.
For me personally in GPF last years there was a cluster of "russian ladies". May be it is silly, may be in blind experiment my guess russian/not russian would be 50%. But it is impossible to make such experiment so I can just give my opinion, that success rate would be >50%.

Unfortunately, it's really not possible to do a blind experiment where you could unambiguously prove causation, so inference to best explanation is the best that we can do here, and I think that the most reasonable explanation is bias. We can argue about the mechanism that causes this bias--for instance, whether it's a conscious attempt to manipulate scores, or more of a semi/un-conscious "boost" that the judge isn't being particularly deliberate about applying--but imo there are no alternative explanations that come close on the ladder of plausibility. Moreover, if it's not national bias, but rather a totally benign preference for certain styles originating from the same culture, why don't we see similar levels of bias across judges from different feds? Japanese judges are less "biased" overall than Russian judges--is that because there's no distinctive Japanese style or because Japanese people have no preference for their own cultural styles over others? That seems pretty implausible to me. There are some smaller feds that don't exhibit any bias at all--is that because they don't like their own skaters' style?

It seems to me that in order to establish the alternative stylistic preference hypothesis, you not need to prove that there is a substantial stylistic difference between Russian and other skaters (which, imo, the burden is on you to show that--since you bring up science, the fact that you *think* you could blindly identify Russian skaters without prior knowledge of their identities doesn't cut it scientifically), but also you need to explain why there isn't a similar effect for judges of other nationalities. So sure, if you want to be scientific about it, maybe I'll retract my statement that this alternative hypothesis is "silly" and instead substitute the statement that this alternative hypothesis requires a lot of evidence and explanation of counter evidence to become viable, and that my own hypothesis that it is caused by some sort of bias mechanism is a lot more parsimonious. I also don't understand your distinction between common sense and statistics--statistics don't interpret themselves, there's always some judgment involved regarding what kind of effects are likely, what kinds of explanations are plausible, and so forth.

(Also, for the record, I am aware of p values and so forth, and have in fact done significance testing on judge biases in the past. The results for many judges were very significant--which matches the results that have been brought up by Miller. I may get around to doing some testing once the data set has built up enough. Of course, this doesn't prove any particular causal mechanism, but if you notice a correlation between houses being left unlocked and houses being burgled, isn't it protesting a bit too much to say that "well it's only correlation, not causation"? Better to just lock your houses instead.)

@shanshani I've already told you on another forum how much I like the statistics you compiled (together with the judges NAMES)! Maybe you can consider adding another column with the deviation and bias of a judge for their countries top competitor. Because usually a judge will overscore their countries no 1 and 2 competitor, but won't do the same with skater no3. So the non-bias shown with no3 will then decrease the end bias in your table. And while some judges do overscore all their competitors, I usually don't care that for example the kazhakstan judge doesn't overscore their 2nd single ladies skater. What matters to me, is how much bias are they showing towards Tursynbava because she is their hope for a podium or top finish.

Hm, sounds interesting. I will consider that. Fortunately, there are official ISU world rankings so I won't have to judge myself who the top competitor is.

 

[Shanshani]

Also, an update: the database for grand prix judges for the 2018-2019 season is complete, and may be found here.

The database linked in the opening post is now in the process of being updated to include Senior B data. Overall, my sense was that Senior B judging was (somewhat predictably) less biased than Grand Prix judging, so I expect this new data to bring down the bias numbers somewhat. But we'll see. Unfortunately, this update process will likely be very slow and the order of competitions added will be somewhat arbitrary. But I'll try to get all of them eventually. In the meanwhile, the "Competitions" page will tell you which competitions have been added in so far. If anyone would like to help put together competition data, you may message me and it will be much appreciated.

 

[WeakAnkles]

I'm no numbers cruncher, so whilst I've been thinking about this all season long, particularly since doing the ID PBPs, I haven't written about it. But the new judging system has ONE OVERRIDING BIAS that I find deplorable: privileging GOE over levels. Essentially the skating PTB are telling its members: you're better off doing the easier elements as long as you do it well. I'm not sure where encouraging people NOT to challenge themselves is a recipe for success in the long term. I haven't seen it work well as a long term strategy ever.

For ID, the new scoring is a disaster. Right now the most important elements appear to be the three choreographic ones in the FD. Since they're all level 1, the tech panel has absolutely no say in judging them. It's all subjective opinions. And we know how well giving judges the chance to manipulate the scores based on "artistic" or "program components" has gone in the past. And it took about 2 competitions for the shenanigans to begin.

I'm not hopeful for this quad at all. I mean AT ALL.

 

[Shanshani]

I'm no numbers cruncher, so whilst I've been thinking about this all season long, particularly since doing the ID PBPs, I haven't written about it. But the new judging system has ONE OVERRIDING BIAS that I find deplorable: privileging GOE over levels. Essentially the skating PTB are telling its members: you're better off doing the easier elements as long as you do it well. I'm not sure where encouraging people NOT to challenge themselves is a recipe for success in the long term. I haven't seen it work well as a long term strategy ever.

For ID, the new scoring is a disaster. Right now the most important elements appear to be the three choreographic ones in the FD. Since they're all level 1, the tech panel has absolutely no say in judging them. It's all subjective opinions. And we know how well giving judges the chance to manipulate the scores based on "artistic" or "program components" has gone in the past. And it took about 2 competitions for the shenanigans to begin.

I'm not hopeful for this quad at all. I mean AT ALL.

It's notable that despite there being fewer points available in ID, it has the worst bias numbers out of all the disciplines including men's, which has a ton of GOE points available from quads as well as extra high PCS factoring.

I think the GOE changes were largely aimed at fixing the issue with there being too little of a penalty for falling on hard jumps in ladies/pairs but especially men's, and also with ugly quads being rewarded too much over well-done triples, but it has the unintended consequence of throwing off the balance of subjectivity versus objectivity in ID in order to keep the scoring consistent between disciplines.

 

[Miller]

Thought I’d do an ‘end of term’ report for this.

Earlier on in the thread I was looking at the effect of what would happen if judges weren’t allowed to judge their own countries skaters, and examples included Savchenko and Massot winning the Olympics Pairs title by 1.63 points compared with 0.43 if the German and Chinese judges weren’t allowed to judge their own countries skaters – the Chinese judge was banned after all (the final margin would have been 2.63 points if neither the Chinese or German judges had judged the contest).

Similarly Mako Yamashita would have won Skate Canada ahead of Elizaveta Tuktamysheva, by 1.74 points compared to 0.26 for Elizaveta, while Kaori Sakamoto would have won silver at GP Helsinki by 0.71 marks, compared to 0.15 for Stanislava Konstantinova, if the Japanese and Russian judges hadn’t been allowed to mark their own countries skater.

Since then I’ve been looking at the various championships to see if any other results could have been affected, and indeed there has been one – the Pairs at 4 Continents. Also it was the Gold medal, though as you’ll see it was ‘one of those things’ as it was really close in the first place (Sui/Han won by 0.06 points), plus as you’ll see there was absolutely nothing to suggest any sort of shenanigans going on. However if judges hadn’t been allowed to judge their own countries skaters Moore-Towers and Marinaro would have won the Gold Medal.

However before then I looked at 2 cases at the European championships, Matteo Rizzo vs Kevin Aymoz for the bronze medal, and Boikova/Kozlovskii vs Della Monica/Guarise for bronze in the pairs. In both cases I got the result to hold up. However Matteo’s margin over Kevin was reduced from 0.74 points to 0.07, while the Boikova/Kozlovskii result comfortably held up.

In the Rizzo case there was an Italian judge in the LP, but there was no French judge in either segment, so an element of luck really played into it, which shouldn’t be the case in figure skating. It’s subjective enough as it is but to have an element of luck as to whose judge is on the panel is taking it too far (IMO). Really judges shouldn’t be allowed to judge their own countries’ skaters, or there should be some sort of substitute judge when a skater whose judge is on the panel is skating.

The exact details – the Italian judge scored it (once GOEs and PCS values converted as if they were the only one judging the contest) Matteo 171.27 vs 165.67 actual for the LP, and the final result would have been Matteo 246.41 vs 246.34 Aymoz, rather than 247.08 vs 246.34. The Italian’s judging of Kevin was just fine by the way, 158.26 vs 158.32 actual.

In the Boikova/Kolovskii case the Russian judge judged both segments and scored it 204.62 B/K vs 205.28 actual, and 200.38 vs 205.14 actual for Della Monica/Guarise, while the Italian judge only judged the LP and scored it 137.62 vs 131.44 actual for DM/G and 127.25 vs 132.70 actual for B/K, so this result would have comfortably held up.

4 Continents. 2 cases. I looked at Elizabet Tursynbaeva vs Mai Mihara and Kaori Sakamoto for the silver/bronze medal, and the aforementioned Moore-Towers/Marinaro vs Sui/Han for the gold.

In the Tursynbaeva case I got that her margins of 0.34 over Mai and 0.67 over Kaori were reduced to 0.13 and 0.14 marks respectively.

The actual scores from the Kazakhstan and Japanese judges were...

Kazakhstan
Tursynbaeva 218.38 vs 207.46 actual
Mihara 206.09 vs 207.12
Sakamoto 203.34 vs 206.79
(Kihira 218.04 vs 221.99 – according to the Kazakhstan judge, Elizabet was the winner).

Japan
Tursynbaeva 209.45 vs 207.46 actual
Mihara 213.87 vs 207.12
Sakamoto 212.58 vs 206.79
(Kihira 227.19 vs 221.99)

However as above the result held up when I removed the Kazakhstan and Japanese judges from the equation – my final scores were

Tursynbaeva – 206.17
Mihara – 206.04
Sakamoto – 206.03

Moore-Towers/Marinaro vs Sui/Han.

This was really close in the first place – Sui/Han 211.11 vs 211.05 Moore-Towers/Marinaro, and after adjusting for judges not being allowed to judge their own countries skaters I got the result to be Moore-Towers/Marinaro 211.02 vs 210.83 Sui/Han.

However the Chinese judge judged it to be 213.08 Sui/Han vs 207.50 Moore-Towers/Marinaro, and the Canadian judge 211.20 Moore-Towers/Marinaro vs 211.98 Sui/Han, so you can see the judging was well within the bounds of acceptability (indeed the Canadian still gave it to Sui/Han), and it’s just one of those things that the result was different e.g. a 0.01 margin for a skater would almost by definition result in a 51% chance of the result staying the same and a 49% chance of it changing even if the judges were to have totally and fairly judged the contest in their own mind - it’s when you get larger swings than this that your antennae start to twitch, though as with M-T/M vs S/H you can always make the case intellectually that judges shouldn’t judge their own countries skaters, plus it would always avoid any element of doubt that there might be.

Finally I looked at Elizabet Tursynbaeva vs Evgenia Medvedeva for the silver medal at the World Championships, and yet again this one held up, but only just. Elizabet’s margin of 0.96 over Evgenia was reduced to 0.04 if judges weren’t allowed to judge their own countries skater. In this example there was no Russian judge in either the SP or LP, but there was a Kazahkstan judge in both, and yet again an element of luck as to who was on the panel almost played its part.

The exact details – the Kazakhstan judge scored it 231.77 Elizabet vs 224.76 actual, and Evgenia 219.29 vs 223.80 actual, and on re-calculating I got it to be 223.84 Elizabet vs 223.80 Evgenia. N.B. If the Kazakhstan judge hadn’t judged the contest at all, in the same way that there was no Russian judge, then the result would almost certainly have been different. However the judges have to come from somewhere, and the Kazakhstan judge’s marks for Evgenia are well within the bounds of acceptability. Not a lot you can do about this, though I do have the theory that if judges aren’t allowed to mark their own countries skaters then any marking down will be greatly reduced as the judges will be sat there doing nothing while their skaters skate and be reminded of the ISU’s concerns.

 

[gkelly]

Thought I’d do an ‘end of term’ report for this.

Thanks for the analysis!

In the Rizzo case there was an Italian judge in the LP, but there was no French judge in either segment, so an element of luck really played into it, which shouldn’t be the case in figure skating. It’s subjective enough as it is but to have an element of luck as to whose judge is on the panel is taking it too far (IMO). Really judges shouldn’t be allowed to judge their own countries’ skaters, or there should be some sort of substitute judge when a skater whose judge is on the panel is skating.

It is possible to control for the luck of the draw under IJS by dropping compatriot judges' scores from the calculations (replacing them with scores from a substitute judge or not).

These would just mean adjusting the accounting algorithms and perhaps adding one more official on the panel who would be present at the venue before the judges' draw in any case.

With either IJS or 6.0 judging, it would be possible to use only judges from 4 Continents federations at Europeans and vice versa, and to use judges only from federations that don't have skaters entered at Grand Prix events. Or, at GP events and especially the final, to make sure that all federations represented on the entry lists are also represented on the judging panel for each respective event.

This would add some complications and probably expenses for the host organizations and for federations sending teams of competitors and officials, but they could be overcome.

At large events open to all federations, such as Worlds, Junior Worlds, and JGP, it would not be possible either to exclude judges from all federations with skaters in the event or to include officials from all federations with skaters in the event on the panel. So one way or another, there will be some luck of the draw as to which skaters in the event also have compatriot judges on the panel. The interventions mentioned above could mitigate that.


However, it still remains true that figure skating is a qualitative sport in which many of the judgments do involve subjective determinations. It may be possible to limit the effects of national bias according to the interventions mentioned above, but it will never be possible to control for all possible biases of the human beings making the judgments -- many of which those humans may themselves be unaware of.

E.g., especially with 6.0 judging where judges choose among the whole "package" presented by each individual skater, some judges might put the highest weight on Skating Skills in general, or on speed or effortlessness or edge quality or multidirectional and one-foot skating in particular. Others might favor artistry in general, or specific kinds of artistry. Some judges might think that the difficulty of the jumps attempted or landed should have a direct effect on the second mark/program components, while others might make a conscious effort to keep those parts of the scores separate. They will each have different pet peeves.

And some judges' opinions of some skaters' skating might be biased by things that they know about those skaters off ice. Again, not necessarily with the judges being consciously aware of these biases.

So there will always be some luck of the draw as to whether the individuals on the judging panel are more inclined to favor Skater A or B or C, assuming their performances are within the same general range of skill level and execution of elements that day.

(We won't even get into issues like the draw for skate order, or unpredictable effects of the quality of the ice surface, etc. Luck of these kinds also has parallels in many other sports including those with much more objective scoring measures.)

I.e., an element of luck will always play into figure skating results. But yes, it would be possible to address the most obvious and identifiable source of bias -- nationalism -- in terms of judges scoring skaters from their own countries.

 

[Andrea82]

Or, at GP events and especially the final, to make sure that all federations represented on the entry lists are also represented on the judging panel for each respective event.

In GP they are already doing that. Feds are asked to nominate a judge when they accept the invitation for their skaters. It is usually unlikely that there are more than 9 countries in the same GP event and therefore everybody who nominate a judge should have one in the panel. There are some cases where some Feds with an invited skater doesn't have a qualified judge at all but as International judges can judge in GP it is a limited number of cases

At large events open to all federations, such as Worlds, Junior Worlds, and JGP, it would not be possible either to exclude judges from all federations with skaters in the event or to include officials from all federations with skaters in the event on the panel.

yes, it would be quite difficult. Looking at this year's entries, in Men they could have done it. On the other hand, in Ladies, I count exactly 13 countries with ISU qualified judges and no skater entered. And some of these countries have only 1 judge, therefore they would rely on them being available for that week. And drawing disproportionally from Feds with only 1 judge, it would also mean having the same individuals every year.

With either IJS or 6.0 judging, it would be possible to use only judges from 4 Continents federations at Europeans and vice versa

IIRC in Ice Dance for Europeans they would not find enough 4CCs judges to fill a whole panel. Because they don't usually have enough judges to fill Ice Dance 4CCs panels without calling Euro judges.