Scoring bias at the national level

[gkelly]

Sometimes it's hard to know whether differing from the panel is because a judge is trying to help a particular skater, or particular skaters at the expense of others, because of which country they're from, or more personal reasons. Or whether they just prefer to reward the kind of skating that that skater exemplifies, or that the coaching in a particular country (or a particular training center in that country) tends to promote.

It would take a lot more complex systems to try to identify biases in favor of certain kinds of skating rather than specific nationalitieis.

 

[Mathman]

It would take a lot more complex systems to try to identify biases in favor of certain kinds of skating rather than specific nationalitieis.

To me, it is not transparent at all why judges judge the way they judge. Sometines I think that this is just the nature, even the definition, of "judging."

Juidge #8 is not responsoble for the marks that judge #7 gives. He is not responsioble for the marks of judge #6. He is not responsoble for judge #5.

If he gives marks that are different from the marks of judge #7 and judge #6 and judge #5, then... ?

Statistics is a funny science. If you have a bunch of numbers and a few of them seem out of line, then statistics can address the question, what is the probability that the numbers might just have turned out that way because of random fluxuations -- that's what numbers do. If that probability is too small, then we accept this as evidence that the discrepancies are not just weird stuff that occurs in every kist of numbers for no particular reason.

So far, so good. It is when we try to use statistics to say, therefore the reason must be specifically X or Y otr Z that we enter shaky ground where statistics alone does not lead to a definitive conclusion.

 

[Rebecca Moose]

we should just burn it all down and go back to 6.0 at this point

 

[alyssamarie]

didn't read all the comments yet but did my masters thesis on judging bias at the olympics so will read into this!

 

[Mathman]

^ Anything you want to share from your research would be much appreciated, especially methodology!

 

[Mathematician]

In my simple-minded opinion

Relax man, you dont have to say that about yourself

 

[Magill]

Relax man, you dont have to say that about yourself

This is called sarcasm, so you can relax, sis...
I will add emoticon this time for you just in case

 

[kolyadafan2002]

I'm glad the ISU took action. That being said, half the ice dance judges would be suspended under these restrictions (some are certainly much much more ergious with it, and get away with it).

 

[Miller]

That is a very cool metric.

Another would be: suppose that we replace all of the errant judge's marks with the average marks of the panel. What effect would that have?

I just did the PCSs for Levito. Her Total (factored) PCS for the LP was 68.84. If the U.S. judge had dropped dead before the competition started and been replaced by a robot that just automatically regurgitated the average marks of the panel, it would have been 68.56 (still edging out Mihara by a few hundredths). The reason that the numbers hardly change is that all of Mr. Williams marks were discarded anyway.

All the other judges seemed content with a 2 or 3 point boost to skaters from their own country. This seems normal and in fact can easily be accounted for by benign factors, like, these skaters are more familiar to the judge. +9 does stand out as excessive. Moderation in all things, even national bias!

I also did the calculation for her GOE. It would have dropped from 8.32 to 7.95 a fall of 0.37 if you'd just taken the middle 6 out of the 8 remaining judges. Overall her score for the LP would have dropped from 134.62 to 133.97 a fall of 0.65 marks. This is in line with previous calculations I've done where I've assumed judges weren't allowed to mark their own countries' skaters - a good rule of thumb is 10 marks in the figures as if the judge had marked the competition by themselves, e.g. Mr Williams had Isabeau as 143.33 vs 134.62 actual, equates to about 1 mark in the final actual result - it doesn't matter if the judge puts in crazy figures as their marks will be discarded, but it does mean figures will otherwise be picked up that wouldn't have been, hence the boost to Isabeau's score.

Obviously the result of the competition wasn't affected in this case (plus other skaters' scores would have also been boosted - those from skaters from countries which didn't have a judge on the panel would have remained exactly the same...) but skating competitions have, and will continue to be won, by smaller amounts than those we're talking about, and it is my opinion that judges shouldn't be allowed to judge their own countries' skaters just to be above board - simply take the middle 6 out of the remaining 8 judges GOEs and PCSs and average/factor accordingly.

 

[Mathman]

As far as I remember, judges are volunteers. There is no termination of employment involved. It's not about labour law, it's not about discrimination... it's just about someone not doing their task properly.

I have had to deal with volunteers. When they do not do the task they are supposed to do, we don't call them again simple as that. They don't have a "right" to be a volunteer

As with everything involving the ISU, I think that the situation is not so simple as it seems. In general, yes, the CEO of an an organization can fire people and can decide not to call back particular volunteers -- but the CEO also answers to the Board of Directors and ultimately to the shareholders.

As I undertand the organizational structure of the ISU, the "shareholders" are the individual national federations that comprise the ISU membership. As for judges, the ISU certifies their qualifications at the international or "ISU" levels and maintains the lists or qualified judges. Prior to a majpr competition there is a random draw (I am not 100% sure that it is truly all that "random," but at least the draw is conductred in public view with many witnesses). The participantrs in the draw are the national federations. The draw determines which countries will be invited to send judges. Much later each federation decides which of its judges to send. The ISU does not say, no, we don't want that guy, send sombody else, please. This would be stepping on the toes of the federations.

It is the same with the skaters. The ISU decides which countries will send skaters to major competitions, but each national federation decides which skaters to send.

On the other hand, the leadership of the ISU hierarchy is tasked with conducting evaluations of judges and has the power to sanction them with varying degrees of severity if they are found not to be performing adequately, including removing tham from the list of qualified judges (the ISU giveth and the ISU taketh away). So... (I don't knolw how to finish this sentence -- so... what?).

 

[alyssamarie]

^ Anything you want to share from your research would be much appreciated, especially methodology!

https://www.beautiful.ai/player/-NmRLBSQZubVqsypSq8- - here is the presentation I made!

 

[SmileHappy34]

I think all judges mark their own country or former country or coaches skaters Higher than other skaters regardless of competition Also , what do you mean volunteers ?. They may not be paid in wages but they are paid. Doesn't the ISU pay for their rooms , flight, and food. Do they pay for them to take classes to understand the scoring system.?

Or they are, volunteer in if they don't get paid wages?

 

[Mathman]

https://www.beautiful.ai/player/-NmRLBSQZubVqsypSq8- - here is the presentation I made!

And very professionally presented, too.

The comment from the 2014 study that national federations pressure and encourage their judges to help out the home team was quite telling. It had been my impression that this was less of a factor than it had been under the ordinal system, when judging was also compromised by voting blocs and deal-making.

I was surprised that China came in for special scrutiny. I think that China has the anecdotal reputation for being extra strict on all skaters.

 

[4everchan]

As with everything involving the ISU, I think that the situation is not so simple as it seems. In general, yes, the CEO of an an organization can fire people and can decide not to call back particular volunteers -- but the CEO also answers to the Board of Directors and ultimately to the shareholders.

As I undertand the organizational structure of the ISU, the "shareholders" are the individual national federations that comprise the ISU membership. As for judges, the ISU certifies their qualifications at the international or "ISU" levels and maintains the lists or qualified judges. Prior to a majpr competition there is a random draw (I am not 100% sure that it is truly all that "random," but at least the draw is conductred in public view with many witnesses). The participantrs in the draw are the national federations. The draw determines which countries will be invited to send judges. Much later each federation decides which of its judges to send. The ISU does not say, no, we don't want that guy, send sombody else, please. This would be stepping on the toes of the federations.

It is the same with the skaters. The ISU decides which countries will send skaters to major competitions, but each national federation decides which skaters to send.

On the other hand, the leadership of the ISU hierarchy is tasked with conducting evaluations of judges and has the power to sanction them with varying degrees of severity if they are found not to be performing adequately, including removing tham from the list of qualified judges (the ISU giveth and the ISU taketh away). So... (I don't knolw how to finish this sentence -- so... what?).

aware of all that already. Also, it is the same in non-for-profit organizations. Guess where many of the volunteers come from? Often related to board of directors. Even the directors themselves are volunteers. When they are not competent or create more trouble than give help, the General Director or CEO if you prefer, still needs to stand firm and let people go. It's not simple but it happens. Been there, done that Of course, it's never simple. But actually, here, it gives bad reputation to the sport when there are judging biases... so the ISU needs to do something and if the federations cannot agree it means that they prefer having rigged judges.

 

[Andrea82]

As I undertand the organizational structure of the ISU, the "shareholders" are the individual national federations that comprise the ISU membership. As for judges, the ISU certifies their qualifications at the international or "ISU" levels and maintains the lists or qualified judges. Prior to a majpr competition there is a random draw (I am not 100% sure that it is truly all that "random," but at least the draw is conductred in public view with many witnesses). The participantrs in the draw are the national federations. The draw determines which countries will be invited to send judges. Much later each federation decides which of its judges to send. The ISU does not say, no, we don't want that guy, send sombody else, please. This would be stepping on the toes of the federations.

It is the same with the skaters. The ISU decides which countries will send skaters to major competitions, but each national federation decides which skaters to send.

For ISU Championships, yes, there is a draw among countries. Only countries with ISU judges can enter the draw. The first draw is done among countries who had a skater in that discipline in previous edition. Then, if necessary, a further draw is done among the rest (for 4 Continents, it is first opened to other 4CC Feds and if there are still unfilled spots on the panel, to European countries too).

The Olympic draw is first down among countries which had a skater qualified in that discipline through World Championships. If it is not enough to fill the whole panel (for Pairs and ID they may be not enough), a second draw is done among those qualifying through the final qualifying competition. If it is still not enough to fill the whole panel, a final draw is done among countries without a skater qualified but who took part in previous year's Worlds in that discipline.

In GPs, the standard practice is to ask countries with a skater invited to that event to nominate a judge.
In all other international competitions, when Feds enter a skater, they also nominate a judge.

So, the situation of a judge judging a skater of their own country is encouraged by how panels are selected (even with the draw, the draw starts with countries which had skaters the year before which are those most likely to have skaters in current year too).

I think all judges mark their own country or former country or coaches skaters Higher than other skaters regardless of competition Also , what do you mean volunteers ?. They may not be paid in wages but they are paid. Doesn't the ISU pay for their rooms , flight, and food. Do they pay for them to take classes to understand the scoring system.?

Or they are, volunteer in if they don't get paid wages?

Judges' hotel and meals are paid by the organizers in all international competitions.
In ISU Championships and Senior GPs, ISU cover the travel expenses too. In the other competitions ISU don't cover the travel expenses and organizers are not obliged. In most cases, the Feds pay the flight for their own judge.
Technical panel members and referees always have the flight paid by the organizing committee.
In ISU Championships and GPs, judges also take a small amount of payment (from 300 to 450 Swiss Francs): https://www.isu.org/short-track/rules/stk-communications/591-isu-communication-2096/file

ISU don't pay judges to attend seminars for re-certification. Their Feds usually cover their expenses.

 

[4everchan]

I think all judges mark their own country or former country or coaches skaters Higher than other skaters regardless of competition Also , what do you mean volunteers ?. They may not be paid in wages but they are paid. Doesn't the ISU pay for their rooms , flight, and food. Do they pay for them to take classes to understand the scoring system.?

Or they are, volunteer in if they don't get paid wages?

Not all. Canadian judges have often marked their skaters lower. They practice "tough love" with many of the skaters. Perhaps not the ultimate favourite like Virtue and Moir.... but certainly up and coming skaters. They do not get any favours.

 

[Andrea82]

I was surprised that China came in for special scrutiny. I think that China has the anecdotal reputation for being extra strict on all skaters.

China had 2 of their judges suspended after 2018 Olympics. One of them (Chen) was also banned to judge at upcoming Olympics. The other one (Huang) managed to pass the exam to be promoted to ISU Technical Controller status while being suspended as judge. So China Fed recommended him to be a TC for 2022 Olympics (as organizers they can propose 2 of the technical panel members/referees) and so he ended being the technical controller for Men competition in Beijing 2022.

https://www.isu.org/inside-isu/legal/disciplinary-decisions/17361-case-2018-06-isu-vs-chen/file

https://www.isu.org/inside-isu/legal/disciplinary-decisions/17359-case-2018-02-isu-vs-huang/file

 

[Mathman]

OT -- Here is (the somewhat irascible) Doug Williams skating with his pairs partner Sharon Carz at the 1990 U.S. Championships.

Sharon Carz and Doug Williams - 1990 U.S. Championships FS

www.youtube.com

 

[Sackie]

For me the ISU needs to look more at the TECH panels and how they show National Bias where their own skaters seem to be the ones getting the higher levels across the board.

 

[yelyoh]

I think all judges mark their own country or former country or coaches skaters Higher than other skaters regardless of competition Also , what do you mean volunteers ?. They may not be paid in wages but they are paid. Doesn't the ISU pay for their rooms , flight, and food. Do they pay for them to take classes to understand the scoring system.?

Or they are, volunteer in if they don't get paid wages?

Not in all cases though yes more often than not. Joe Inman (sp?) a US judge who also found his way to Golden Skate a number of years ago did Michelle Kwan no favors at Salt Lake. And though Shin Amano of Japan is a tech specialist he is tough on everyone. Bless his persnickety heart.