Nationalistic bias in figure skating judging, 2018-present

Shanshani

On the Ice
Joined
Mar 21, 2018
I have put together some quite comprehensive statistics on nationalistic bias in figure skating judging, which includes every single senior level ISU competition. Unfortunately, the write-up is a bit unwieldy for a forum post as it's quite long and contains graphs and so forth, so I will direct you to this website, where you can read the whole thing.

Here are the top line conclusions:

The database currently contains 312 judges, 177 of which I examined for nationalistic bias (the other judges didn’t have an extensive enough judging record), 92 of which showed statistically significant evidence (p<0.05) of nationalistic bias. Of those 74 showed strong (p<0.01) evidence.

Biased judging was widespread across most federations (shout out to Australia the one large-ish federation that was pretty clean), but some federations (notably Russia, France, Germany, and Italy) were particularly bad, though not always in the same way.

Considering that the judges play a quite significant role in the careers of young skaters, I thought I would share this here in order to open up a conversation. Of course, I'm sure you guys have many questions as well, so feel free to ask away.
 

Rashela

Rinkside
Joined
Oct 26, 2011
Thank you for the spreadsheet. Just a little thing - you should check the nationalities, some of the "Japanese " judges are Chinese and vice versa and some other nationalities are probably wrong, too.

AUS judges are exceptional, we could see it especially during ice dance at Worlds, where they didn't get the memo that Polish team is suddenly top ten material.
 

Shanshani

On the Ice
Joined
Mar 21, 2018
Thank you for the spreadsheet. Just a little thing - you should check the nationalities, some of the "Japanese " judges are Chinese and vice versa and some other nationalities are probably wrong, too.

AUS judges are exceptional, we could see it especially during ice dance at Worlds, where they didn't get the memo that Polish team is suddenly top ten material.

Hm, that sounds like the nationality-switched sheet rather than the original sheet. Let me check that they didn't get mixed up somehow. Could you tell me which judge(s) in particular?

Edit: Are you sure the sheets aren't mixed up on your end? There's two versions linked--the one that is normal is called "Judges' sheet", but the other one is called "nationality switches", where every judge got switched to a different nationality for comparison reasons. The far right column tells you which nationality got switched for what.
 

medoroa

On the Ice
Joined
Dec 30, 2017
"(This example from the 2018 ACI men’s free skate is completely random, obviously.)"

I can't tell if this is tongue-in-cheek or if you mean it. Could you elaborate?

Also, I don't seem to see you've mapped this onto how big the staple of judges and elite skaters are in each federation. My country has no judges on that list, but this is probably more because we have few judges at an international level in the first place, and they also have no elite skaters to give biased scores to (Australia is not a "big fed" to me at all which I think is more likely an explanation for their lack of "bias" -- and Russia's size might also explain Russia's large number of "biased judges", although it might also not). Apologies if you mentioned that somewhere and I missed it.

I'm also a little unsure what you're trying to conclude or achieve. National bias exists, this is fact. It has always been considered a fact, especially by the ISU. This is why there are judges from several different countries on a judging panel and why they are drawn randomly; this is an officially-sanctioned method to offset this bias, proving that ISU knows (and has admitted!) national bias exists. I mean, they are not dumb, regardless of how frustrating I sometimes find them. So in light of that, it would be more interesting (to me) to see how the biases fall out when the lowest and highest scores have been omitted. That would be a constructive way to look at whether ISU's method to offset bias and errors is working within reasonable limits.
 

Shanshani

On the Ice
Joined
Mar 21, 2018
"(This example from the 2018 ACI men’s free skate is completely random, obviously.)"

I can't tell if this is tongue-in-cheek or if you mean it. Could you elaborate?

It's tongue in cheek. It is the one instance in which personal bias entered this project--I like Yuzu so I was going to use his scores as an example, especially one where the US judge scored him the lowest by a heftyish margin. :p It doesn't really matter though, the example is just there to demonstrate how the math works--the formula is the same for everyone, which you can confirm by going to the competition sheets. And in fact the US judge used in that example, Doug Williams, did not show statistically significant evidence of pro-US bias in general, so I made no accusations about him. The Japanese judge, Miwako Ando, on the other hand, did make it on to the list (although only by the least strict standard of significance).

Hey, I had to do something to entertain myself after dealing with so much dry statistics :laugh:
 

medoroa

On the Ice
Joined
Dec 30, 2017
It's tongue in cheek. It is the one instance in which personal bias entered this project--I like Yuzu so I was going to use his scores as an example, especially one where the US judge scored him the lowest by a heftyish margin. :p It doesn't really matter though, the example is just there to demonstrate how the math works--the formula is the same for everyone, which you can confirm by going to the competition sheets. And in fact the US judge used in that example, Doug Williams, did not show statistically significant evidence of pro-US bias in general, so I made no accusations about him. The Japanese judge, Miwako Ando, on the other hand, did make it on to the list (although only by the least strict standard of significance).

Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.

eta: And sorry I keep editing my replies, I'm mulling it over in my head as I type.
 

Shanshani

On the Ice
Joined
Mar 21, 2018
Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.

I suppose, but to be honest, I find this kind of criticism a bit lazy. As I mentioned in the comments on my blog post, I have made the entirety of my methodology and mathematics transparent to everyone who bothers to put in the effort. If you have some criticism about the project or worry about bias, you should connect it to some aspect of how the project was done. There's really no way for me to answer the criticism "You like Yuzu, so you're biased" or "You're American, so you're biased" because there is no relationship between those facts and how the project was executed, other than the fact that I decided to use one skater's scores rather than another in my explanation of how the mathematics works, which has no bearing on the results. Nor is there anything I could do (other than stop liking Yuzu or stop being American, I suppose) to answer that criticism, because it has nothing to do with how the results were found.

Again, the same formulas were applied to everyone equally, and determinations of bias were made by comparing a judge to the other judges, and at no point did my judgment about what x skater ought to have scored enter in to it. Someone who likes Nathan, or who loves the Russian girls, or whatever will get the same results if they follow the same steps.
 

medoroa

On the Ice
Joined
Dec 30, 2017
I suppose, but to be honest, I find this kind of criticism a bit lazy. As I mentioned in the comments on my blog post, I have made the entirety of my methodology and mathematics transparent to everyone who bothers to put in the effort. If you have some criticism about the project or worry about bias, you should connect it to some aspect of how the project was done. There's really no way for me to answer the criticism "You like Yuzu, so you're biased" or "You're American, so you're biased" because there is no relationship between those facts and how the project was executed, other than the fact that I decided to use one skater's scores rather than another in my explanation of how the mathematics works, which has no bearing on the results. Nor is there anything I could do (other than stop liking Yuzu or stop being American, I suppose) to answer that criticism, because it has nothing to do with how the results were found.

Again, the same formulas were applied to everyone equally, and determinations of bias were made by comparing a judge to the other judges, and at no point did my judgment about what x skater ought to have scored enter in to it. Someone who likes Nathan, or who loves the Russian girls, or whatever will get the same results if they follow the same steps.

I didn't accuse you of anything. It just stood out like a sore thumb in a post which was otherwise well-presented.

Everyone will get the same numbers with the same steps, but conclusions are always based on how a human being reads the numbers (I've worked at an ad agency, I know). It's relevant information for me if you decide to expand on your conclusions or (even more importantly) you intend to propose changes. That's seldomly, if ever, unmotivated by some form of agenda.
 

jillredhand

Record Breaker
Joined
Oct 20, 2017
I'm also a little unsure what you're trying to conclude or achieve. National bias exists, this is fact. It has always been considered a fact, especially by the ISU. This is why there are judges from several different countries on a judging panel and why they are drawn randomly; this is an officially-sanctioned method to offset this bias, proving that ISU knows (and has admitted!) national bias exists. I mean, they are not dumb, regardless of how frustrating I sometimes find them. So in light of that, it would be more interesting (to me) to see how the biases fall out when the lowest and highest scores have been omitted. That would be a constructive way to look at whether ISU's method to offset bias and errors is working within reasonable limits.

Shanshani does address this idea in the report; the methodology shows that it's not inevitable and intrinsic that all judges are biased in favor of their own skaters. There are many judges who show no statistically significant bias in favor of their own home skaters, or show some bias of a lesser degree. While it's true that some degree of nationalistic bias is natural and would probably be impossible to completely eliminate, analyses like this should make it possible to address the most egregious cases. Just because perfect impartiality may not be humanly possible does not mean that improvements cannot be made.
 

medoroa

On the Ice
Joined
Dec 30, 2017
Shanshani does address this idea in the report; the methodology shows that it's not inevitable and intrinsic that all judges are biased in favor of their own skaters. There are many judges who show no statistically significant bias in favor of their own home skaters, or show some bias of a lesser degree. While it's true that some degree of nationalistic bias is natural and would probably be impossible to completely eliminate, analyses like this should make it possible to address the most egregious cases.

That remains to be seen. The text itself says that as the season progresses, there will be more judges added to the list while (probably) none will be removed. The text itself also says that it does not account for all methods of possible biased judging.

Just because perfect impartiality may not be humanly possible does not mean that improvements cannot be made.

I agree! Which is why I suggested that to me, a more constructive use of this statistic was to measure whether the fail-saves already in place were functioning satisfactorily (for some measure of satisfactory).
 

jillredhand

Record Breaker
Joined
Oct 20, 2017
I didn't accuse you of anything. It just stood out like a sore thumb in a post which was otherwise well-presented.

Everyone will get the same numbers with the same steps, but conclusions are always based on how a human being reads the numbers (I've worked at an ad agency, I know). It's relevant information for me if you decide to expand on your conclusions or (even more importantly) you intend to propose changes. That's seldomly, if ever, unmotivated by some form of agenda.

I do actually agree with you here. Shanshani, I am extremely impressed with your work here, but that tongue in cheek example did leave a questionable impression on me too. I do trust your tranaparity and the objectivity of your methodology, but just from an optics perspective, that example and your tongue in cheek statement will probably result in people drawing unnecessary, even if wrong, adverse inferences.
 

jillredhand

Record Breaker
Joined
Oct 20, 2017
I agree! Which is why I suggested that to me, a more constructive use of this statistic was to measure whether the fail-saves already in place were functioning satisfactorily (for some measure of satisfactory).

I also agree, I'd like to know if discarding the top and bottom scores always (or mostly) gets rid of the biased judge's score. Though even if that failsafe is effective, that still doesn't mean that it wouldn't be preferable to reduce the degree of bias the judges give anyway. Discarding the high and low scores isn't meant just to eliminate national biases, it's also to make up for natural human error and disagreement in the judging.
 

CanadianSkaterGuy

Record Breaker
Joined
Jan 25, 2013
Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.

eta: And sorry I keep editing my replies, I'm mulling it over in my head as I type.

But why would fans who put together a bias of judging ever be biased themselves? :sarcasm:

This is a really important point to make when seeing the source of these criticisms. There's a difference between the person presenting all data and trends and those who pick the trends that criticize the scoring in a manner that props up their faves and condemns their fave's rivals. Of course there's national bias (sometimes blatant - I recall that one Chinese judge), but sometimes skaters are marked lower or higher based on the personal preference of the judges. Important to note that there's a panel, and it's not ordinals anymore, and highs/lows are thrown out, so usually it'll balance out. Although I'm not saying that absolves bad judging but nationalistic bias certainly has less of an effect now IMO than it did under 6.0.
 

drivingmissdaisy

Record Breaker
Joined
Feb 17, 2010
As long as the national federations play any part in training, nominating, and/or selecting judges for international competitions, bias will exist. From the federation's perspective, you want your skaters to place as high as possible and so you'd obviously want judges who will score those athletes favorably (and probably deflate the scores of top competitors).
 

ladyjane

Medalist
Joined
Jun 26, 2012
Country
Netherlands
Sorry, I didn't check the maths. I'm quite good at it, that's not the point. My point is something different: you see I understand a bit of National Bias in the sense that a judge would give his/her own skaters too many points. The judge has seen them throughout the year, maybe he/she has given the skater advice, whatever. That's okay. Maybe not totally fair, but I get that. I probably would do it myself. It's not always intentional bias, it's just you've seen a skater/couple improve throughout the season (after test skates, champs camps, and all these things) and you'll just see them as being better than another might do. As someone stated earlier, the outliers are deleted anyway. However, the problems start when you give your own skaters high points and deliberately lower their competitors's points. Like what happened at the Olympics with the Chinese judge at the Pairs competition. I don't think the judge got penalised for giving the wonderful Chinese pair (too) high points, but for deliberately giving the German pair low scores. Personally (so, this is just my viewpoint, please bear that in mind when reading this) I can totally understand one might score one's own skaters too high, it's when one also gives too low scores to the competitors when it's really problematic. Such things might have been easier to detect under the 6.0 system. It's similar with reputation. Judges shouldn't take that into account, but of course they do. They don't live in a bubble and hear/see nothing.

Don't get me wrong, I admire the effort, and I liked to read the AUS judges are mostly unbiased. But as long as Nationalism and Patriotism remain, so will National bias. It can only be checked as much as possible. But never totally eliminated for the reasons above. Rather human I think.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
Agreed. There is a difference between biased judging and dishonest judging.

We all have biases, which may be based on nationality or familiarity with specific skaters' careers or personal lives or personal preferences about skating style or beliefs in what should count most highly in skating results or any number of other factors. Some of which we are aware of and might readily admit to or consciously try to overcome if we were trying to judge honestly, and others which are so ingrained in our beliefs that we are unaware of them.

That's true for judges and other officials, fans, skaters and coaches, commentators, etc. (And TV commentators and producers may intentionally aim to present home country skaters in the best possible light, downplaying the positive qualities of their closest rivals.)

Only the officials can have a direct effect on the scoring of the event. Are they trying to manipulate the results to get their federations' skaters to the highest possible place in the standings? That's cheating.

Or are they judging as honestly as they know how, without conscious awareness that their personal preferences and personal histories are shaping how they perceive the various performances?

That's bias, and we're all susceptible to it.
 

Rissa

Record Breaker
Joined
Dec 11, 2014
Agreed. There is a difference between biased judging and dishonest judging.

Unfortunately, I find a lot of current judging downright dishonest rather than merely biased, especially on top. Which makes me slowly lose interest in the sport, to be honest.
 

ladyjane

Medalist
Joined
Jun 26, 2012
Country
Netherlands
Agreed. There is a difference between biased judging and dishonest judging.

We all have biases, which may be based on nationality or familiarity with specific skaters' careers or personal lives or personal preferences about skating style or beliefs in what should count most highly in skating results or any number of other factors. Some of which we are aware of and might readily admit to or consciously try to overcome if we were trying to judge honestly, and others which are so ingrained in our beliefs that we are unaware of them.

That's true for judges and other officials, fans, skaters and coaches, commentators, etc. (And TV commentators and producers may intentionally aim to present home country skaters in the best possible light, downplaying the positive qualities of their closest rivals.)

Only the officials can have a direct effect on the scoring of the event. Are they trying to manipulate the results to get their federations' skaters to the highest possible place in the standings? That's cheating.

Or are they judging as honestly as they know how, without conscious awareness that their personal preferences and personal histories are shaping how they perceive the various performances?

That's bias, and we're all susceptible to it.

Exactly! We're all biased. It's how we handle it that counts. I like your differential: biased judging (which everyone does) vs dishonest judging (which nobody has to do).
 

drivingmissdaisy

Record Breaker
Joined
Feb 17, 2010
Unfortunately, I find a lot of current judging downright dishonest rather than merely biased, especially on top. Which makes me slowly lose interest in the sport, to be honest.

Honestly, so much of judging is subjective that I rarely find any scores to be blatantly dishonest. Judges scoring the same element differently by 3 or 4 points in GOE is not terribly uncommon, but those differences aren't necessarily attributable to cheating because the scores for all the judges will usually span that entire range.
 
Top