Hypothetical discussion : How would you reduce national bias in judging figure skating

[4everchan]

The only thing is that there is a young Korean skater who the officials suddenly become very confident with giving edge calls every single time, and there's a Georgian skater they pick apart everything other than a completely rotated landing is little more than +1 for GOE. Yet as you say there are recidivists with lutz take off edges (these can be identified in real time, in slow motion with a bad camera angle, etc) but the officials seems to miss it almost every time. I find this very odd.

On the flip side, I watched an American skater recently with 5/7 dirty landings in the free skate and received an enormous score, I saw something similar with a Belgian early in the season. I'm not making this up I even posted screen captures demonstrating the cheated landings but none of it affected scores.

It's all very strange to me these discrepancies.

The good news in what you write is that it seems like it's not nationalistic but some odd cases from various countries. The conversation we are having is about reducing nationalistic bias.

I agree that having one or two more camera angles available would then help reduce discrepancies, but I am not sure those are due to nationalistic biases... perhaps more reputational judging, but that's another animal I would prefer we don't get into, at least in this thread. When people start talking about these, it turns ugly quickly and often, the skaters themselves are the ones receiving dismissive treatment from angry fans. I think we can avoid doing this while discussing ways for the ISU to reduce biases.

 

[Mathman]

At this point the tech is instructed to call what they see...

You know what's a funny sport? American football. OK, so you have to go ten yards to make a first down. The ballcarrier rushes into the line, possibly being pushed from behind by several 350 pound lads (but they did outlaw the play where they picked up the quarterback and threw him over the top of the pile), and he gets close. The referee or linesman eyeballs the play in real time and places the ball on the ground representing his best guess as to how far the runner advanced the ball before his knee hit down. The placement might be off by a foot one way or another, but that's the nature of a judged sport.

Now what? They make an eleborate display of bringing out phyical chains and laying them along the ground to measure if it's ten yards or not. This remarkable technology was introduced in the 1890s. The fans wait with bated breath. When they stretch out the chains, will they reach to where the nose of the football has been placed? The game hangs in the balance.

 

[4everchan]

You know what's a funny sport? American football. OK, so you have to go ten yards to make a first down. The ballcarrier rushes into the line, possibly being pusahed from behind by several 350 pound lads (but they did some time ago outlaw the play where they picked up the quarterback and threw him over the top of the pile , and he gets close. The referee or linesman eyeballs the play in real time and places the ball on the ground representing his best guess as to how far the runner advanced the ball before his knee hit down. The placement might be off by a foot one way or another, but that's the nature of a judged sport.

Now what? They make an eleborate display of bringing out phyical chains and laying them along the ground to measure if it's ten yards or not. This remarkable technology was introduced in the 1890s. The fans wait with bated breath. When they stretch out the chins, will they reach to where the nose of the football has been placed? Thwe game hangs in the balance.

for starters, just the word yard doesn't invite for better tech

 

[Mathman]

for starters, just the word yard doesn't invite for better tech

"A yard was originally the length of a man's belt or girdle, as it was called. In the 12th century, King Henry I of England fixed the yard as the distance from his nose to the thumb of his out-stretched arm."

So, substituting chain for girdle, there you go!

But I will say that for centuries the nose to thumb definition served dressmakers and merchants who sold fabric off the bolt well. If you ordered a three-piece suit, you expected to get the whole nine yards.

 

[el henry]

You know what's a funny sport? American football. OK, so you have to go ten yards to make a first down. The ballcarrier rushes into the line, possibly being pushed from behind by several 350 pound lads (but they did outlaw the play where they picked up the quarterback and threw him over the top of the pile), and he gets close. The referee or linesman eyeballs the play in real time and places the ball on the ground representing his best guess as to how far the runner advanced the ball before his knee hit down. The placement might be off by a foot one way or another, but that's the nature of a judged sport.

Now what? They make an eleborate display of bringing out phyical chains and laying them along the ground to measure if it's ten yards or not. This remarkable technology was introduced in the 1890s. The fans wait with bated breath. When they stretch out the chains, will they reach to where the nose of the football has been placed? The game hangs in the balance.

But they haven't outlawed the "Brotherly Shove" a.ka. the "Tush Push" in football. .. yet. Maybe the hue and cry will stop now that the Eagles have collapsed and aren't a threat to anyone.

(Also perhaps the play will be less subject of more boring and meaningless. screen shots and freeze frames. Dear heavens, my eyes glaze over with those, in any sport, football or skating)

I think the proposal to make referees employees of the ISU, presumably answerable to the ISU, could work in reducing the appearance of impropriety. And appearances are important. Caesar's wife and all that.

As you said earlier, no one likes corruption. No one will be the pro-corruption fan or fed or skater. The problem ... what is corruption?. I agree that it is NOT bias.

Do I think we can eliminate bias? No, nor should we. If I were a judge I would give beautiful spins all the 10s. Just all of them, for everything. If that favored American skaters, so be it. My bias is for beautiful spins, not Americans.

 

[CrazyKittenLady]

The thread drifts happening lately make me think we are in the deepest, darkest off-season. Honestly, it's becoming a bit annoying.

 

[Mathman]

The thread drifts happening lately make me think we are in the deepest, darkest off-season. Honestly, it's becoming a bit annoying.

Sorry to have annoyed you. That was not my intention, not at all.

 

[CrazyKittenLady]

Sorry to have annoyed you. That was not my intention, not at all.

Oh no, don't be sorry, it wasn't directed at you or even specifically this thread, just a general observation lately. Maybe I'm suffering from the Christmas-holiday-figure-skating-withdrawal-blues. I'll take my grumpy self out of this thread now and come back tomorrow. Until then, somebody will hopefully have posted the solution to the national bias conundrum....

 

[Mathman]

Actually, as far as this thread is concerned, we should go back to discussing the main proposal by gkelly in post #5.

 

[gkelly]

No idea how many. We are talking here about a few dozens not a few hundreds. Also, who says we would need nine judges per discipline ? Perhaps 7 would be enough even with still discounting the highest and lowest score, considering that these judges are not representing a nation anymore.

Yes, year-round salary. Their job is to be an ISU official. No day job.

From a quick look at the ISU calendar, it seems that it is not uncommon for there to be 5 or occasionally more international competitions taking place around the world in any given week.

Calendar of Events - International Skating Union

The International Skating Union is the international sport federation administering Ice Skating sports throughout the world.

www.isu.org

The week ending Feb 04, 2024, for example, has 5 smaller internationals taking place in various parts of Europe, plus an ISU championship (Four Continents) taking place in China.

In late summer/early fall, there might be a similar number of JGP+Challenger+lesser internationals taking place the same week.

In general, it would be difficult for officials to travel from one event to another one the following week if they're on opposite sides of the globe, but within Europe (or North America) it might be more possible.

So how many officials would be needed to cover, say two weeks worth of competitions?

Mathman gives some rough estimates of costs:

About the cost, if let's say 100 judges would be required and each paid an anual salary of $US 100,000 (the average salary of an NFL referee is $205,000), that's 10 million dollars. Or would 60 at $90,000 be enough? A substantial chunk of change for an organization with an anual operating budget of 16 million, but still withing the range of what is possible (they could cash in some of their long-term investments in anticipation of increased revenue in the future).

Do we want to encourage judges who reside in smaller skating countries that may host at most one international per year and are far from any other host countries (e.g., Oceania/Southeast Asia)?

So we need to take into account greater travel expenses to be paid for these officials who initially would have less experience, although once hired and traveling around the globe working for the ISU, they would quickly gain experience. And undoubtedly they would like to go home sometimes, especially if they have families.

Some of these internationals may be held in conjunction with domestic open competitions at the same venue in the same week, including for levels lower than junior.

The non-international portions could use entirely local/domestic volunteer officials. These would be good opportunities for non-international judges to meet some ISU-hired elite judges and learn from them, as well as watching elite-level skating. Some of these domestic judges may be aiming to apply for international appointments in the future.

Do I understand that ISU employed officials would be forbidden to judge in any non-international club-hosted events, including events that are not on the ISU calendar but that do allow entries by skaters from outside the host country? That would include if the international judges are already right there for a concurrent smaller international?

All issues to take into account when deciding what this corps of employed officials would and would not be used for, and how many total are needed to cover the calendar of international events.

 

[Miller]

I would have on spare judge to switch out judges when a skater from their country is on the ice. This can't prevent them from underscoring a rival skater, but at least they wouldn't be able to overscore their own skaters. Imagine a football match between Germany and Argentina with a German or Argentinian referee. The very notion is absurd.

This would certainly sort out 'positive national bias'. The 'case of Leanna Caron' wouldn't even apply as she and the French judge wouldn't have
even judged the RD, and Leanna the FD. Also any 'negative national bias' might be reduced as she and any other judge would be sat back and doing nothing while their countries' skater skated. Hence they would be reminded about the ISU's concern about national bias, though from my looking at protocols I would say this occurs far less than positive national bias - I have a feeling that what happens is that once one starts doing it they all start doing it e.g. one judge says my skater is being unfairly judged so gives a 'discreet boost' to them. However other judges spot this (they're bound to look at protocols after the event) and decide to boost their own in return. Hence before you know it lots of skaters are being given a bit of a boost to their score.

Re having a spare judge, they wouldn't be underworked, certainly at major champs. At the 2018 Olympics when I last looked at this there were 137 cases of judges judging their own countries' skater (there's the team event as well), and of these 55 were given the highest 'effective score', i.e. once GOEs/PCS etc. are converted to real scores, of the 9 judges on the panel, 36 the 2nd, 21 the 3rd and 7 the 4th i.e. 119 out of 137 higher than average (the median) - see above comments about 'discreet boosts'. After that 9 were the middle/median score, and only 9 out of the 137 were lower than the median.

However the Doug Williams case in the other thread probably shows that the ISU is clamping down on this far more than before. I have a feeling this wouldn't have been picked up years ago e.g. before the start of SkatingScores, or the ISU's equivalent of it. Also I had a look at the women's LP (chosen at random) from the recent GP series/GPF. There were 73 cases of judges judging their own countries' skater, and of these there were 57 where the effective score was higher than the actual i.e. far more than you would get at random. However most were only 2/3 points higher than the actual, and only a couple were 7/8 points higher than the average. So I would say the ISU is on the case far more. However having another judge who is swapped in when a skater is skating who has a judge is on the panel would eliminate any doubt entirely.

 

[ladyjane]

I like the proposal of the swapping of judges. Probably won't get rid of all bias (I don't think that's possible), but it would of Nationalistic bias for the most part.

 

[Miller]

I do not think the national bias is the biggest problem in judging, because the highest and lower marks are thrown away. Of course I agree the national bias should be monitored, as it seems it is done already to some extent.

However this does mean that GOEs and PCS scores will be picked up that would otherwise be discarded if a judge scores higher than the average. However based on the typical 'corridor' I would say an individual judge could only affect a competition by 2 or 3 points at the most before they are picked up e.g. a 5 point boost in effective score in the SP, and 10 in the LP would equate to about 1.5 points in the final result based on a good rule of thumb I have that 10 points in 'effective score' equates to about 1 point in actual. Repeat this in marking down a main rival and that's your 3 points. However based on the Doug Williams case you probably wouldn't get away with it, 2 points in the final is probably the absolute most.

The biggest problem I see, is the inconsistent "calls" for edges and underrotations, supposedly because of lack of a second video to review.

Definitely. The panel may have a hindered view so don't even review an item. Alternatively they see something, but the video is unclear and they can't call the item because you can't just rely on your initial impression (I assume this is what occurs).

Basically you need another person looking at the ISU feed as the skater skates to see if they spot items that the panel might miss from their view point.

Similarly if they looked at a 2nd video feed at big events they might spot something the first video feed doesn't i.e. the main panel spot something, the first feed is unclear, but the 2nd feed confirms it.

Something like this should improve 'dodgy calls' immensely (plus I would have thought 2 camera angles would be OK for the main panel to look at in the time available, especially if you've got split screen technology to speed up the process). Do this and you might be able to look at 'technical panel bias' more clearly i.e. you can now see the wood for the trees.

Whether anything is going on I don't know - I would imagine 3 person discussions would tend to reduce anything significantly compared with single judges doing their own thing sat by themselves, simply because of the discussions taking place. However it would take someone to explain how it all works just to see if there is anything of concern in the process, plus I guess at the end of the day someone could always report concerns of national bias behind the scenes, something that couldn't occur with individual judges simply because no discussions are taking place.

 

[gkelly]

Re having a spare judge, they wouldn't be underworked, certainly at major champs. At the 2018 Olympics when I last looked at this there were 137 cases of judges judging their own countries' skater (there's the team event as well),

You would probably need at least two and more likely three backup judges to fill this role at any given championship.

Keep in mind that ice dance has different judging appointments than singles/pairs. Some judges have ISU appointments in both categories, but more likely you'd need to get a separate backup for the ice dance event.

The team event is small enough that a single person could probably handle all three of men's, women's, and pairs, even if they all compete on the same day.

At the Olympics, there's only one figure skating event per day, so having the same person on all three events over the course of the two weeks would be feasible, although they would be spending more time on the stands than other judges who are only chosen for one or two events, or sometimes only one segment of some of those events.

At ISU championships, there can be 30-40 skaters in each short program, and 24 in the free skates, with often two non-dance event segments on the same day (e.g., men's short and pairs free), so asking a single person to be the backup for all six of those non-dance segments would be overworking them compared to the non-backup judges.

Additionally, it would make more sense to have the backup judge just judge the whole segment, to keep their own scores consistent for that segment and to also use as a backup just in case any judge happened to be taken ill or otherwise unable to continue during the event.

So they would work just as hard as the rest of the panel, but far fewer of their scores would actually be used. More than the substitute judge's scores under 6.0, where an extra judge would be seated and score the whole event but none of their scores were used.

I'd once heard someone propose swapping out judges for same-country skaters way back in 6.0 days. It made no sense in that context because the judges' scores in 6.0 were only meaningful in relation to the same judge's scores for the other skaters, as placeholders for ordinals.

For IJS, it would be feasible.

 

[icewhite]

When talking about national bias we also need to take the blocks, dependencies and friendships into account. Although we are not in cold war times, friendly judges are not just from the same country.

 

[Miller]

You would probably need at least two and more likely three backup judges to fill this role at any given championship.

Keep in mind that ice dance has different judging appointments than singles/pairs. Some judges have ISU appointments in both categories, but more likely you'd need to get a separate backup for the ice dance event.

The team event is small enough that a single person could probably handle all three of men's, women's, and pairs, even if they all compete on the same day.

At the Olympics, there's only one figure skating event per day, so having the same person on all three events over the course of the two weeks would be feasible, although they would be spending more time on the stands than other judges who are only chosen for one or two events, or sometimes only one segment of some of those events.

At ISU championships, there can be 30-40 skaters in each short program, and 24 in the free skates, with often two non-dance event segments on the same day (e.g., men's short and pairs free), so asking a single person to be the backup for all six of those non-dance segments would be overworking them compared to the non-backup judges.

Additionally, it would make more sense to have the backup judge just judge the whole segment, to keep their own scores consistent for that segment and to also use as a backup just in case any judge happened to be taken ill or otherwise unable to continue during the event.

So they would work just as hard as the rest of the panel, but far fewer of their scores would actually be used. More than the substitute judge's scores under 6.0, where an extra judge would be seated and score the whole event but none of their scores were used.

I'd once heard someone propose swapping out judges for same-country skaters way back in 6.0 days. It made no sense in that context because the judges' scores in 6.0 were only meaningful in relation to the same judge's scores for

For IJS, it would be feasible.

Yes, both very good points, you rarely get an ice dance judge judging pairs/singles and vice-versa. Similarly judging the entire event so that they're 'in the swing of things' when it comes to their scoring rather than just jumping in and out as necessary.

For the length of something like the Olympics you wouldn't need one dedicated judge just for everything. There's lots of judges on site who are qualified, all you need is someone who is independent for each segment (if they have a skater skating their scores won't count anyway because the original 9 are used). Indeed you could use one of the 14 for each discipline i.e. 9 are used in each segment, but one of the remaining becomes the 10th one whose scores are used as necessary.

 

[4everchan]

Yes, both very good points, you rarely get an ice dance judge judging pairs/singles and vice-versa. Similarly judging the entire event so that they're 'in the swing of things' when it comes to their scoring rather than just jumping in and out as necessary.

For the length of something like the Olympics you wouldn't need one dedicated judge just for everything. There's lots of judges on site who are qualified, all you need is someone who is independent for each segment (if they have a skater skating their scores won't count anyway because the original 9 are used). Indeed you could use one of the 14 for each discipline i.e. 9 are used in each segment, but one of the remaining becomes the 10th one whose scores are used as necessary.

With ISU Pro judges, no need for 9 judges however. Feds would not be represented and wouldn't cry about not having a judge in the event and of course, one would expect that the judging would be more homogeneous.

 

[4everchan]

This would certainly sort out 'positive national bias'. The 'case of Leanna Caron' wouldn't even apply as she and the French judge wouldn't have
even judged the RD, and Leanna the FD. Also any 'negative national bias' might be reduced as she and any other judge would be sat back and doing nothing while their countries' skater skated. Hence they would be reminded about the ISU's concern about national bias, though from my looking at protocols I would say this occurs far less than positive national bias - I have a feeling that what happens is that once one starts doing it they all start doing it e.g. one judge says my skater is being unfairly judged so gives a 'discreet boost' to them. However other judges spot this (they're bound to look at protocols after the event) and decide to boost their own in return. Hence before you know it lots of skaters are being given a bit of a boost to their score.

Re having a spare judge, they wouldn't be underworked, certainly at major champs. At the 2018 Olympics when I last looked at this there were 137 cases of judges judging their own countries' skater (there's the team event as well), and of these 55 were given the highest 'effective score', i.e. once GOEs/PCS etc. are converted to real scores, of the 9 judges on the panel, 36 the 2nd, 21 the 3rd and 7 the 4th i.e. 119 out of 137 higher than average (the median) - see above comments about 'discreet boosts'. After that 9 were the middle/median score, and only 9 out of the 137 were lower than the median.

However the Doug Williams case in the other thread probably shows that the ISU is clamping down on this far more than before. I have a feeling this wouldn't have been picked up years ago e.g. before the start of SkatingScores, or the ISU's equivalent of it. Also I had a look at the women's LP (chosen at random) from the recent GP series/GPF. There were 73 cases of judges judging their own countries' skater, and of these there were 57 where the effective score was higher than the actual i.e. far more than you would get at random. However most were only 2/3 points higher than the actual, and only a couple were 7/8 points higher than the average. So I would say the ISU is on the case far more. However having another judge who is swapped in when a skater is skating who has a judge is on the panel would eliminate any doubt entirely.

I personally am OKAY with a bias of 2-3 more as long as these judges didn't mark the direct competitors of their national pet skater way down. I am not okay when it comes to 6-7-8 points more. A natural bias is normal. An "intentional" bias aiming to alter the results of a competition is no longer acceptable.

 

[snowed]

I would have on spare judge to switch out judges when a skater from their country is on the ice. This can't prevent them from underscoring a rival skater, but at least they wouldn't be able to overscore their own skaters. Imagine a football match between Germany and Argentina with a German or Argentinian referee. The very notion is absurd.

I like the proposal of the swapping of judges. Probably won't get rid of all bias (I don't think that's possible), but it would of Nationalistic bias for the most part.

What I don't like about the swapping judges idea is that it would basically change the panel. The new judge will a little milder or more sever than the judge swapped (each judge judges differently), and that would affect the scores and placement for that particular skater. This is similar to why we don't compare the scores from different competitions.

 

[snowed]

I wanted to post here, I posted in the other tread by mistake...

I've got an idea! I think it would make judging more fair/ correct, and would also cover national bias. The idea is inspired by the fact that the tech panel votes to reach a conclusion and it was mentioned that in gymnastics each judge judges just a portion of the program.

Tech panel would do the same exact thing. To avoid national bias there should be no 2 persons on the panel from the same country.

The panel of 9 judges would be split in 3 panels of 3 judges each, and they will vote and reach a single mark (conclusion, see lower) so each single national biased mark/opinion will be outvoted in the panel.

Each panel can judge everything, as now.

OR we can have each panel specialized, one for GOE for jumps, one for GOE for spins and step sequences, one for components.

The panel in charge with jumps double check the calls for edge and underrotations (tech panel would be placed on the opposite side of the rink than the judges, or at least this panel). The judges could not improve the call, just diminish if the tech panel called a jump clean but the judges see it under, it will be under, but not the other way around.

Ideally, each panel would go through all the bullet points and vote on those, and the computer will give the GOE. I think they should have enough time to do that in 2 minutes (because each panel looks at just jumps/ spins&step sequences and components)

This way there would be no extra expense for judges, just an update of the software for the 1st version or a new software for the 2nd version