Scoring bias at the national level

[Mathman]

Well, I'm not a native speaker and I am struggling a bit with the term bias in the first place because it's not what I would use.

I wanted to add that the double meaning of the word bias is a problem in itself. Sometimes it just means that something turned out different from what we expetced -- the data seems to be leaning to one side or another. In common use, however, bias is bad. If you are biased you are an evil person.

The ISU in fact deliberatley confuses these two ideas. In the Assessment of Officials documents they speak of "anaomalies" ( a neutral term) but then if you accumulate too many anomalies, you get tarred with the "bias" brush.

 

[4everchan]

We don't normally see who gets the warnings by the ISU technical committee. Unless they are then brought in front of the disciplinary commission for a suspension.

Ok. so then, let's wait to see if judge Fortin is brought in front of the disciplinary commission.

What do you think ? Is that a strong case ?

 

[4everchan]

In common use, however, bias is bad. If you are biased you are an evil person.

This is interesting to me. Because in my language, if i say I am biased towards this or that, it is not negative or evil.
It simply means I have a "regular" preference..

Silly example :

I am biased towards maple syrup rather than let's say corn syrup

Of course, my bias could be nationalistic.

 

[Mathman]

^ I think that in the United States the word "biased" is heard most often in the context of racial discrimination and other prejudices. I just googled "Is bias a negative word" and the top answer was

"Bias can be either positive or negative. However, all forms of bias (whether favorable or unfavorable) prevent us from judging others fairly. For example, because of explicit bias, a teacher might openly claim that students from a certain ethnic background are exceptionally good in math."

Wikipedia says: "Bias is a disproportionate weight in favor of or against an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair."

 

[4everchan]

^ I think that in the United States the word "biased" is heard most often in the context of racial discrimination and other prejudices. I just googled "Is bias a negative word" and the top answer was

"Bias can be either positive or negative. However, all forms of bias (whether favorable or unfavorable) prevent us from judging others fairly. For example, because of explicit bias, a teacher might openly claim that students from a certain ethnic background are exceptionally good in math."

yes. when applied to people.

But you can be biased towards things too...

 

[Mathman]

^ I think it's tricky. As applied to figure skating. I am biased toward a good Russian split jump (a thing). I like it when a skater includes this element in his program. I like skaters (people) who include this element in their programs and who achieve great height and full split and pointed toes (insofar as you can point your toe in a skating boot). I like Jason Brown. People and things tend to mix together.

I like an Ina Bauer (especially one that looks like it defies the laws of figure skating physics and biology, like Shizuka Arakawa's). I am prejudiced toward toward Jun-huan Cha and would probably give him higher PCSs than the other judges, were I a figure skating judge.

 

[Mathman]

They are referring to the total segment score. The Canadian judge gave S-D/D 140.35 points (11.03 more than their actual FS score ended up being), P/M 127.95 (+5.79) and L/E 121.11 (+5.14). Compared to the untrimmed mean, their score is over 8 points higher on average for the Canadian teams vs. almost 2 points higher than the untrimmed mean for all other pairs.

I can actually see how a judge might end up giving 8 points more in in TES GOEs than the others on the panel. Nathan Chen would make a good example. A judge might be really, really impressed with Nathan's overall jump technique. Every time Nathan launches into the air that judge says to himself, wow! That judge might give one extra GOE point on every jump for that reason et voila.

The other judges might say, yes, it is certainly impressive than Chen can do all those quads, but if you look hard at each jump in isolation, that jump is not always all that.

 

[4everchan]

I can actually see how a judge might end up giving 8 points more in in TES GOEs than the others on the panel. Nathan Chen would make a good example. A judge might be really, really impressed with Nathan's overall jump technique. Every time Nathan launches into the air that judge says to himself, wow! That judge might give one extra GOE point on every jump for that reason et voila.

I am pretty convinced that in pairs, the lifts can create such a difference and Deanna and Max have stellar, not cookie-cutter, lifts... which can be appreciated by some while some others may not care that much for the originality of them but look at other component

The other judges might say, yes, it is certainly impressive than Chen can do all those quads, but if you look hard at each jump in isolation, that jump is not always all that.

 

[Magill]

^ I think that in the United States the word "biased" is heard most often in the context of racial discrimination and other prejudices. I just googled "Is bias a negative word" and the top answer was

"Bias can be either positive or negative. However, all forms of bias (whether favorable or unfavorable) prevent us from judging others fairly. For example, because of explicit bias, a teacher might openly claim that students from a certain ethnic background are exceptionally good in math."

Wikipedia says: "Bias is a disproportionate weight in favor of or against an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair."

Yes, this bit seems crucial to me!
"All forms of bias prevent us from judging fairly".
It's alright to be biased as a fan, we all are, one way or another.
Yet what seems lost in this discussion is that a judge is supposed to be above fans in their ability to judge fairly the quality of what they see, with their personal likes and dislikes, tastes, national and personal affiliations put aside. This is exactly what makes you qualify as a judge just as much as technical expertise, no less. And you should be trained and able to do that.
If you are not able to give a performance without a Russian split the same score you would give to the one with it, or if you always put one school of skating above some other as a rule, just because it happens to be more in tune with your personal or national background, you are not fair and not a good judge. It is just as simple as that. Is it easy? No. But overcoming your personal biases to give every skater and every performance a fair evaluation is what judges should strive to achieve.
What worries me is that it does not even seem to be expected of them.

 

[Jumping_Bean]

I can actually see how a judge might end up giving 8 points more in in TES GOEs than the others on the panel. Nathan Chen would make a good example. A judge might be really, really impressed with Nathan's overall jump technique. Every time Nathan launches into the air that judge says to himself, wow! That judge might give one extra GOE point on every jump for that reason et voila.

The other judges might say, yes, it is certainly impressive than Chen can do all those quads, but if you look hard at each jump in isolation, that jump is not always all that.

I am pretty convinced that in pairs, the lifts can create such a difference and Deanna and Max have stellar, not cookie-cutter, lifts... which can be appreciated by some while some others may not care that much for the originality of them but look at other component

That's not what happened here though. The Canadian judge didn't give significantly higher GOEs on elements SD/D did well, he gave significantly higher GOEs on elements they didn't do well: A +2 on the 3Tw2 (mean -1.11), a 0 on the 3T+1A+1A (mean -2.11) and a +2 on the 3LoTh (mean 0).
When you adjust the GOEs on these elements, his TES drops to 66.14 and the TSS to 136.89, which are still the highest scores respectively, but not so much out of the ordinary (2 and 4 points respectively).

Like I said, here it didn't matter much, because he's just one judge and the highest and lowest scores for each element are thrown out anyway, but you just need two judges to influence a score (in either direction, positively or negatively).

That can be seen in the Men's FS - Take a look at Boyang Jin's protocol, his scores had a spread of 147.63 (USA) to 180.27 (NED), so that both the judging of the Mexican judge (158.99) and the Italian judge (177.56) were counted for most elements & in the PCS, despite still having a big deviation from the actual TSS. In this case, it did not make much difference as there were the same number of outliers in either direction, but that's more down to luck than anything.

 

[el henry]

But who is to say that a Russian split does not deserve the extra GOE?

You? Me? My little cousin?

1. We can't "judge the judgers" by the perception of "Well anyone with eyes to see and ears to hear". My eyes to see and ears to hear may be quite different than yours. And my little cousin's is different from everyone.

2. We can't "judge the judges" by saying well, the Antarctic judge gave the Antarctic skater plus five GOE when everyone else was plus two. To make the enormous unsupported leap to "well it's national bias"????

Nope, without any other evidence, that's just my little cousin talking. The Antarctic judge really could truly believe that was the GOE.

3. Of course we can always "wuzrobbed". That's what skating fans, including me, do. But accusations of actual bias from the judge, without more? Uh uh.

 

[icewhite]

Honestly I don't get most posts from you, @Mathman , and I also don't get the last post from @el henry
I simply don't understand what you want to tell me.
That there cannot be objective judging, that you cannot tell from the scores given whether there is a pattern that indicates manipulation, and that hence you don't really care?

Maybe my position varies so much from yours that I am struggling to even find the right words to express myself and to be able to comprehend your posts.

 

[el henry]

Honestly I don't get most posts from you, @Mathman , and I also don't get the last post from @el henry
I simply don't understand what you want to tell me.
That there cannot be objective judging, that you cannot tell from the scores given whether there is a pattern that indicates manipulation, and that hence you don't really care?

Maybe my position varies so much from yours that I am struggling to even find the right words to express myself and to be able to comprehend your posts.

I am not trying to be obtuse or contrary. NO ONE wants bias or corruption, we all want the same thing.

I think what I am trying to say is that numbers alone on one skater from one comp (unless it fifty points more than any other skater) will not support a finding, for me, that the judge intentionally discriminated.

I confess I am influenced by my background, which is legal. (This is exceedingly simplified of course) In the US legal system, there are two different kinds of discrimination: disparate treatment and disparate impact.

1. Disparate treatment: actual proof of discrimination. Proof that someone says "I won't hire <class of people> because they are lazy and stupid".

In skating, "I scored Skater X higher in return for a promise that Judge ABC would score Skater Y from my country higher."

2. Disparate impact: Company does not hire <class of people> based on numbers of persons hired

Disparate impact requires a large number of people to prove. So that you can present meaningful statistics, you can show "But for" discrimination, of the X number of applications from <class of people>, any company that wasn't discriminating would have hired Y. And they didn't, they hired Y minus 20. Or the "inexorable zero" ( a legal term I love. )

So if we are trying to prove bias based on numbers, for me personally, I need more than one judge at one comp.

I'm sorry to geek out and possibly be confusing, I'm just trying to make a good faith effort to explain my approach.

 

[icewhite]

I am not trying to be obtuse or contrary. NO ONE wants bias or corruption, we all want the same thing.

I think what I am trying to say is that numbers alone on one skater from one comp (unless it fifty points more than any other skater) will not support a finding, for me, that the judge intentionally discriminated.

I confess I am influenced by my background, which is legal. (This is exceedingly simplified of course) In the US legal system, there are two different kinds of discrimination: disparate treatment and disparate impact.

1. Disparate treatment: actual proof of discrimination. Proof that someone says "I won't hire <class of people> because they are lazy and stupid".

In skating, "I scored Skater X higher in return for a promise that Judge ABC would score Skater Y from my country higher."

2. Disparate impact: Company does not hire <class of people> based on numbers of persons hired

Disparate impact requires a large number of people to prove. So that you can present meaningful statistics, you can show "But for" discrimination, of the X number of applications from <class of people>, any company that wasn't discriminating would have hired Y. And they didn't, they hired Y minus 20. Or the "inexorable zero" ( a legal term I love. )

So if we are trying to prove bias based on numbers, for me personally, I need more than one judge at one comp.

I'm sorry to geek out and possibly be confusing, I'm just trying to make a good faith effort to explain my approach.

Sure, there is absolutely no proof in these numbers that deliberate pushing of a certain team happened - I would call it hints or indicators. I would not be able to make a case big or solid enough for it to have actual consequences for the judge.
For me those numbers, how small the sample here might be, do show though that there was inadequate judging favouring the team(s) from the same country. This might come from unconscious prejudice, but the way each team was judged does not look so innocent to me.

Added: It's just a rather random case, we could talk about many more like these.

 

[4everchan]

Sure, there is absolutely no proof in these numbers that deliberate pushing of a certain team happened - I would call it hints or indicators. I would not be able to make a case big or solid enough for it to have actual consequences for the judge.
For me those numbers, how small the sample here might be, do show though that there was inadequate judging favouring the team(s) from the same country. This might come from unconscious prejudice, but the way each team was judged does not look so innocent to me.

Added: It's just a rather random case, we could talk about many more like these.

Here is what I suggest to you
Flag that judge. Look at their other competitions. Are they always doing the same ? Was it just this one time ? Is the alleged bias towards one team, one country, etc. Do they tend to be high scorers compared to other judges ? Are they putting down other teams ? Regularly the same team or country ? Etc

This is what I have done (even at this competition in dance this time)
It happens that in some competition, I got upset with some scores from a judge. I went to look at other events... and that judge was completely within the middle.... it happens quite often. If you find that this one particular judge is constantly playing national favourites, then who knows, maybe the ISU will also conclude to the same thing

I am saying this only because already, it seems like this judge didn't do this at all in the SP, where it matters most as all pairs do the same elements and GOE is huge there. Considering how tight the SP was, if that judge had wanted to help out the Canadian teams, that was a very easy opportunity.

So perhaps, that judge truly believed that these teams all deserved high scores in the LP...
The only element to me that is puzzling is the twist of SD/D. It was clearly not good. Did the judge not see the touching of the shoulder? DId the judge found enough positive bullets to get to counteract the deduction ? I think it's probably that they didn't see it and made a honest mistake. I had not seen it myself until the replay. So there is that. Also, who knows what their angle was at the rink. The twist is performed at the other end of the rink and it's not always easy to spot all the features. That's not an excuse of course, but it can explain a mistake.

Anyways, just my opinion.

 

[Mathman]

If you are not able to give a performance without a Russian split the same score you would give to the one with it...

I think that sometimes we forget that competitive figure skating is a sport. A sport must have an identifiable goal or objective. In the hundred-meter dash the goal is to run as fast as you can for 100 meters and cross the finish line before the other guy. In figure skating the goal is to fill space and time with movement demonstrating athletic control and grace.

I am a big fan of jumps in figure skating. The coolest thing about jumps is their spacing in making full use of the ice surface, their timing in response to the requirements of the music, their overall contribution to the theme and purpose of the choreography.

The ISU rulebook used to invoke the term, “the full skating vocabulary.” As in, skaters are expected to demonstrate mastery of the full skating vocabulary. (Cool use of the word “vocabulary,” incidentally. ). This was, in fact, the heart and soul of the sport.

Well, times change. Something lost, something gained. I can see it from both sides now.

But still, somehow…

 

[Mathman]

Honestly I don't get most posts from you, @Mathman...
I simply don't understand what you want to tell me.

I don't knpw what I think.

Having followed figure skating for decades, my first observation is that the problem of biased and corrupt judging has lesseneed considerably over the years. Whatever gripes we bring to the table now, it was much worse in the past. With Anne Frank, in spite of everything I still believe that most poeple are really good at heart and that the the ISU does take seriously its responmsibility to protect the integrity of the sport.

Second, I believe that the science of statistics is much misused when it comes to trying to "prove" things about human motivation and the like.

Third, yeah, I guess you're right. I do think that it is a plus not a minus that figure skating judges show a range of variation in interpretting the scoring rules.

 

[Magill]

I think that sometimes we forget that competitive figure skating is a sport. A sport must have an identifiable goal or objective. In the hundred-meter dash the goal is to run as fast as you can for 100 meters and cross the finish line before the other guy. In figure skating the goal is to fill space and time with movement demonstrating athletic control and grace.

I am a big fan of jumps in figure skating. The coolest thing about jumps is their spacing in making full use of the ice surface, their timing in response to the requirements of the music, their overall contribution to the theme and purpose of the choreography.

The ISU rulebook used to invoke the term, “the full skating vocabulary.” As in, skaters are expected to demonstrate mastery of the full skating vocabulary. (Cool use of the word “vocabulary,” incidentally. ). This was, in fact, the heart and soul of the sport.

Well, times change. Something lost, something gained. I can see it from both sides now.

But still, somehow…

Well, vocabulary needs a dictionary so that we are all sure we speak the same language and give the same meaning to the same words.
Singling out one element as crucial and more important than all the others while it is not stated expressis verbis and specifically not stated in the rules is arbitrary and unfair. Taking your example, why a Russian split should be more important than e.g. a delayed axel? Unless it is stated expressis verbis in the rules, a fair judge should be able to appreciate both and differentiate on the basis of quality etc and not just because they "like" one element better than the other. I do not think any judge can just make their own rules for themselves and apply them as they like without sharing and discussing and making them known.

 

[Mathman]

Well, vocabulary needs a dictionary so that we are all sure we speak the same language and give the same meaning to the same words.

I think that the dictionary of the ISU is too restrictive and does not reflect the richness of the history and traditions of the discipline.

...why a Russian split should be more important than e.g. a delayed axel?

All is grist that comes to our mill. Give me a Russian split jump and a delayed Axel.

But instead we have triple flip, 4.90. Triple flip, 5.30. Delayed Axel, 1.10. Split jump 0.00. Delayed Axels and Split jumps have been written out of the dictionary. I think that this impoverishes the sport and prevents it from being all that it could be.

 

[snowed]

The only element to me that is puzzling is the twist of SD/D. It was clearly not good. Did the judge not see the touching of the shoulder? DId the judge found enough positive bullets to get to counteract the deduction ? I think it's probably that they didn't see it and made a honest mistake. I had not seen it myself until the replay. So there is that. Also, who knows what their angle was at the rink. The twist is performed at the other end of the rink and it's not always easy to spot all the features. That's not an excuse of course, but it can explain a mistake.

I watched that performance (on TV) and the twist looked perfectly fine, but then the slow motion review was from a different angle and you could see the problem with the catch. So I think is very possible that from a certain angle (like the one we saw on tv), a judge will not see the catch well and the other judges will see it better from different angles