Hypothetical discussion : How would you reduce national bias in judging figure skating

[4everchan]

Some of us have been discussing one of the main issues with judging : national bias.
Judges are volunteers and almost always quite attached to their national federation : they often have been elite skaters themselves or related to elite skaters. For technical specialists, being a former skater is even mandatory.

The issue, as one can imagine : national bias. How can it not happen when an official has been involved, sometimes even from the grassroots level, within their national federation, often even representing it in international competition.

What would you suggest the ISU could do to minimize this influence ? Is it possible to eliminate all conflict of interest or appearance of conflict of interest ?

For starters, I would suggest that the ISU train and hire permanent judges. These officials would need to "perform" well -meaning be precise and fair in their duties and the ISU would be responsible for making them accountable. Judges could get fired if they are not within respectable criteria. Eliminating the volunteering aspect and the national associations may help.

What would you do ?

 

[Mathman]

For starters, I would suggest that the ISU train and hire permanent judges. These officials would need to "perform" well -meaning be precise and fair in their duties and the ISU would be responsible for making them accountable. Judges could get fired if they are not within respectable criteria. Eliminating the volunteering aspect and the national associations may help.

What would you do ?

When this proposal was put forward years ago, the ISU did seem to be receptive to moving in that direction. This was the whole idea behind the tech specialist position created in 2003. The officials on the technical panel were to be hired, trained and subjected to direct oversight by the ISU. This was intended to diminish somewhat the role of the national federations in the judging process.

So far so good, but the second part of the deal was anonymous judging, This was supposed to free the judges from undue innfluence from their federation because no one knew how the judges scored the competition once they were seated. That is, if the national federation head told the judge, "You better place our skaters ahead of the other guys or else!" then the judge could always give low marks to her country's skaters but then lie to the federation head and say that she didn't. Supposedly this could have prevented the 2002 pairs controversy because it would have allowed the French judge to summon up her courage and double-cross boss Gailhaguet. He would be none the wiser and she wouldn't get in trouble.

Well...let's just say that this aspect of the ISU response didn't work as hoped. Instead it just kept fans and media in the dark and provoked even more suspicion that figure skating judging was crooked to the core.

What should be done now? I have no plan to suggest that does not cost more money than the financial resourses of the ISU can bear. The ISU's current idea seems to be to continue to tweak assessment, oversight and ethics procedures as occasion demands and permits. I have no real quarrel with this approach, although critics will say that this is just sticking another finger in the same leaky dike each year.

 

[Mathman]

By the way, if anyone wants to know how much money the ISU has and what they spend it on, here is their most recent official finantial report (calendar year 2022) as required under Swiis law.

https://www.isu.org/inside-isu/financial-and-other-reports/report-of-the-statutory-auditor/31359-2022-financial-report/file

They had a total of 15,740,506 Swiss Francs in liquid assets, 265,198,067 in long term investments, and 4,625,001 in property. (A Swiss Franc has roughly the same value as a U.S, dollar or a Euro.)

Their expenditures in different categories are listed on page 5. (This report is quite easy to understand -- you don't have to be an accountant to get the gist of it. ) If we ask, why can't the ISU mainatin a core of professional judges like we see in popular team sports -- according to Forbes Magazine the total worth of the NFL is 163 billion dollars, more than 500 times the resources of the ISU.

 

[4everchan]

Please, feel free to suggest ideas no matter how much they would cost.
Sure, the ISU may not be as rich as the NFL, but here the and : if the ISU cleaned itself up, would broadcast companies and sponsors be more willing to generate some revenues for the ISU, considering a broader appeal with fans?

 

[gkelly]

For any new proposal, we could ask how could we get there from here. Or we could just assume that money is no object and that we could start from scratch without worrying about the existing structures or appointments at all.

So, taking the latter approach...

Suppose the ISU has near-unlimited funds.
Would they want all international events to be officiated by judges, tech specialists, etc., who have no current connection to any national federation?

And would these be paid positions?

Even without the money, they could say that no one with an International or ISU-level judging appointment may, e.g.,
*Hold elected office in any national federation
*Hold a paid position in any national federation
*Represent a member federation at ISU congresses
*Hold a national judging appointment for any federation
*Judge any below-national domestic events for any federation
*Compete internationally representing a federation (including adult internationals as well)

OK if you used to do any of those things before becoming an ISU official. All of that is good experience to help qualify you for judging international.

And OK to go back to working with a national federation after you age out of the international position, or resign that position for whatever reason.

And possibly be an immediate family member of anyone who currently does any of those things.
Although that could be overly restrictive. E.g., two siblings are both national judges in their home country. Brother qualifies to become an international judge. Does that mean Sister would also have to resign any domestic positions just because of her sibling's international ambitions?

Would this also apply to referees, technical controllers, and technical specialists?

How about data entry operators, accountants, and technical accountants?

 

[4everchan]

For any new proposal, we could ask how could we get there from here. Or we could just assume that money is no object and that we could start from scratch without worrying about the existing structures or appointments at all.

So, taking the latter approach...

Suppose the ISU has near-unlimited funds.
Would they want all international events to be officiated by judges, tech specialists, etc., who have no current connection to any national federation?

And would these be paid positions?

Even without the money, they could say that no one with an International or ISU-level judging appointment may, e.g.,
*Hold elected office in any national federation
*Hold a paid position in any national federation
*Represent a member federation at ISU congresses
*Hold a national judging appointment for any federation
*Judge any below-national domestic events for any federation
*Compete internationally representing a federation (including adult internationals as well)

OK if you used to do any of those things before becoming an ISU official. All of that is good experience to help qualify you for judging international.

And OK to go back to working with a national federation after you age out of the international position, or resign that position for whatever reason.

And possibly be an immediate family member of anyone who currently does any of those things.
Although that could be overly restrictive. E.g., two siblings are both national judges in their home country. Brother qualifies to become an international judge. Does that mean Sister would also have to resign any domestic positions just because of her sibling's international ambitions?

Would this also apply to referees, technical controllers, and technical specialists?

How about data entry operators, accountants, and technical accountants?

Curious to see how many judges that would eliminate in the current circumstances.. It seems like most.

 

[gkelly]

Yes. They would have to choose between working/volunteering for the ISU or working/volunteering for their federations, at least at the same time. It wouldn't eliminate people who might have worked with/for their federations in the past or prevent them from doing so in the future.

That might make it harder for federations to maintain enough strong officials at a domestic level, if they keep losing their best talent to the ISU.

It would also make it harder for international judges to share their training and the knowledge they gain with the lower level judges at home. Would they be allowed to give domestic seminars to share what they learned internationally?

(I have other questions about how a professional paid judging corps would work, to be discussed later if people think that's important.)

 

[4everchan]

Yeah. That's the issue here : changing the model means that the federation would lose access to judges who are at the higher levels to mentor/train the local judges.

But then, the pro ISU judges could indeed do that.

The way I see it is that the ISU has a bunch of permanent judges, hired and trained by them, going around the circuit. During the high season, there are so many competitions, they would be very busy. During the off-season they could participate in training sessions and also mentor local judges. As ISU judges, they would report to the ISU.

Now of course, if the ISU is crooked we are in trouble but this way, I don't think a federation would have a direct access to judges like Didier did in the past

 

[Mathman]

How many full time professional ISU judges do you think would be needed? Would the proposal be to pay them a sufficient year-round salary that they could quit their day job?

Now of course, if the ISU is crooked we are in trouble but this way, I don't think a federation would have a direct access to judges like Didier did in the past

That's a good p[oint. Who guards the guardians? It is also not clear that this proposal would significantly reduce national bias. The judges might still be homeys even if the ever-so-neutral-and-disinterested ISU is their boss.

Perhaps the biggest advantage would be a cosmetic one. It looks bad when a national federation officer serves on an Olympic panel and then lo and behold, the skater from that person's counrtry ends up getting a huge score. If gkelly's proposal reduces the appearance of possible conflicts of interest, that's a good thing even if national bias remains an area of concern.

 

[4everchan]

How many full time professional ISU judges do you think would be needed? Would the proposal be to pay them a sufficient year-round salary that they could quit their day job?

No idea how many. We are talking here about a few dozens not a few hundreds. Also, who says we would need nine judges per discipline ? Perhaps 7 would be enough even with still discounting the highest and lowest score, considering that these judges are not representing a nation anymore.

Yes, year-round salary. Their job is to be an ISU official. No day job.

That's a good p[oint. Who guards the guardians? It is also not clear that this proposal would significantly reduce national bias. The judges might still be homeys even if the ever-so-neutral-and-disinterested ISU is their boss.

If they show national bias, they are fired. That's their job. So I think the message here is clear. The ISU is responsible to ensure ethical and fair judging.

 

[icewhite]

I would prefer to have a real scoring system with appropriate evaluation and tight, clear rules. No "it's subjective anyway" as a base. Subjectivity has no place in sports competitions. Once it has been established that there is a clear right and wrong, good and bad, it's a lot easier to judge the judges by their performance. To me it does not matter a lot where the mistakes come from. Someone who has shown they are not judging correctly on a systematic level, whether bias or incompetence, they need to be stopped from judging.
I think professional judges are a bit more likely to judge well, but I don't see it as a significant improvement or even real solution.
I also think 7 judges, or less, should be enough, once a better system and more sportive understanding of judging has been established, especially with today's technical possibilities. The 9 judges are only necessary because of the corridor and the "majority is right" system.

 

[Magill]

Yeah. That's the issue here : changing the model means that the federation would lose access to judges who are at the higher levels to mentor/train the local judges.

But then, the pro ISU judges could indeed do that.

The way I see it is that the ISU has a bunch of permanent judges, hired and trained by them, going around the circuit. During the high season, there are so many competitions, they would be very busy. During the off-season they could participate in training sessions and also mentor local judges. As ISU judges, they would report to the ISU.

Now of course, if the ISU is crooked we are in trouble but this way, I don't think a federation would have a direct access to judges like Didier did in the past

I would opt for ISU judges delivering training to local judges but for mixed international training groups or, preferably, only to local judges from different feds than their national one so that local judges from different countries receive the same training and to further reduce the federation - ISU judges link which is deemed unhealthy.
At least for the transitional period they would still be excluded from sitting on panels for competitions where their country placed 1-5 the previous year. With all the judges working for ISU full time, it should not really be a problem to organize it.

 

[Magill]

I would prefer to have a real scoring system with appropriate evaluation and tight, clear rules. No "it's subjective anyway" as a base. Subjectivity has no place in sports competitions. Once it has been established that there is a clear right and wrong, good and bad, it's a lot easier to judge the judges by their performance. To me it does not matter a lot where the mistakes come from. Someone who has shown they are not judging correctly on a systematic level, whether bias or incompetence, they need to be stopped from judging.
I think professional judges are a bit more likely to judge well, but I don't see it as a significant improvement or even real solution.
I also think 7 judges, or less, should be enough, once a better system and more sportive understanding of judging has been established, especially with today's technical possibilities. The 9 judges are only necessary because of the corridor and the "majority is right" system.

I do not see these things as mutually exclusive. Obviously, establishing objective measurement would be the best solution and should be sought and implemented asap wherever possible. Digital measurement will be free of any national bias 100% and probably cheaper and easier to maintain in the long run. This is what should be our ultimate goal, no doubt.
Still, it seems there is a certain, particularly PCS-related aspects, which could be difficult to be scored by computers based on objective measurement or at least it would be difficult to gain wide support for such a solution atm. Here professional judges could make some difference.
I do think after some transitional period and with strong emphasis on objectivity and ethical issues in pro judges training and their ongoing assessment, old loyalties to feds and national / reputation bias could fade to some extent while new attitudes and loyalties could emerge.
Still what I see as crucial for such a solution being any successful is a new emphasis exercised by ISU combined with peer pressure going the right direction and not like today, where, sadly, all kinds of biases seem to be largely accepted or even expected...

 

[gkelly]

I would prefer to have a real scoring system with appropriate evaluation and tight, clear rules. No "it's subjective anyway" as a base. Subjectivity has no place in sports competitions. Once it has been established that there is a clear right and wrong, good and bad, it's a lot easier to judge the judges by their performance.

Here are the current rules about identifying element levels (determined by the technical panel), followed by the judges' guidelines for positive and negative grades of execution, and for program components.

https://www.isu.org/figure-skating/rules/fsk-communications/31153-isu-communication-2558/file

What the technical panel does is already pretty black-and-white but is limited by the perceptions of the human eye from a single view point and a single camera angle for replays. Added technology (that currently exists, e.g., extra camera angles, or that could easily be invented in the near future) could make that side of scoring even more objective.

But look at everything the judges are asked to evaluate, particularly the positive GOE bullets and the program component criteria.

How can those be evaluated objectively?

What can be taken away from the judges and relegated to technological measurement with that can be developed right now?

What rules/guidelines would you change to make clearer for yes-or-no or quantitative assessments?

And then how would you handle all the qualitative assessments that judges are asked to make?

Do you want to remove all the "artistic" aspects of Composition and Presentation from the sport entirely?

Do you have a way to standardize determinations of poor, acceptable, good, very good, etc.? for various GOE and PCS criteria?

Or do you want to get rid of all qualitative assessments entirely and boil everything down to acceptable vs. unacceptable, in which case there would be no benefit for skaters to develop excellence?


Could technology might assist to measure things like height and distance on jumps, rotational speed and centering on spins, etc.,?
Could technology measure speed and ice coverage throughout a program?

Those seem within reach and could become aspects of scoring measured by instruments rather than estimated by eye.

But look at all the other bullet points for evaluating elements. Everywhere the guidelines refer to "good" or "poor" positions, or "effortlessness," or errors that have a range of reduction depending on severity, you're talking about value judgments. I don't see a way to turn these into yes-or-no answers in a way that is fair to skaters.

Same for most of the Skating Skills criteria and pretty much all of the Composition and Presentation criteria. Would you want to get rid of those two components entirely because they're inherently subjective? You could still have a sport of figure skating with elements and in-between skating with no music and no rewards or penalties for how the skater relates to the audience or how pretty the moves look. You'd just be losing most of the qualities that many fans and many skaters love about the sport.

In the Composition area, Pattern and ice coverage could probably be measured with instruments.
For Skating Skills, "speed" could be measured by instruments.
Amount of time spent skating forward vs. backward, clockwise vs. counterclockwise could be counted objectively, and also someone could count all the different types of turns and steps (as the tech panel does for step sequences) and kinds of connecting moves throughout program.
It would just need to be someone dedicated to counting those things in order to get an accurate count, and then there would need to be rules for how that information would get included in the scoring.

But when you get to determining "the ability of the skater to execute the skating repertoire...with blade and body control," "Clarity of edges, steps, turns, movements and body control," "balance and glide," "Flow," and also "Power" in relation to how the skater generates speed, across the program as a whole, it would be much more difficult either to train AI to evaluate these qualities or to train judges to evaluate them "objectively," since they are qualitative assessments.

How can you make evaluations of poor, fair, average, good, very good, outstanding into yes-or-no questions?
Do you have suggestions?

Or do you want to get rid of all qualitative evaluation of skating skill? In that case, we're not talking about skating itself being fundamental to the sport.

 

[skatingguy]

I would prefer to have a real scoring system with appropriate evaluation and tight, clear rules. No "it's subjective anyway" as a base. Subjectivity has no place in sports competitions. Once it has been established that there is a clear right and wrong, good and bad, it's a lot easier to judge the judges by their performance. To me it does not matter a lot where the mistakes come from. Someone who has shown they are not judging correctly on a systematic level, whether bias or incompetence, they need to be stopped from judging.

The only sports that have a completely objective scoring system are those based on time, or distance, and even then are still officials judging certain aspects of the competition. Most sports have a great deal of subjectivity to them, and when you take a sport like figure skating the subjectivity is the point. This skater was more aesthetically appealing than that skater - this skater interpreted the music better - had a better use of body line, etc. One of the things that makes skating an appealing, and engaging sport is that there is going to be disagreement about those aspects of the results. Ensuring the judges have independence from the federations would be a good way to reduce biased judging, but it shouldn't be about removing subjectivity.

 

[icewhite]

Here are the current rules about identifying element levels (determined by the technical panel), followed by the judges' guidelines for positive and negative grades of execution, and for program components.

https://www.isu.org/figure-skating/rules/fsk-communications/31153-isu-communication-2558/file

What the technical panel does is already pretty black-and-white but is limited by the perceptions of the human eye from a single view point and a single camera angle for replays. Added technology (that currently exists, e.g., extra camera angles, or that could easily be invented in the near future) could make that side of scoring even more objective.

But look at everything the judges are asked to evaluate, particularly the positive GOE bullets and the program component criteria.

How can those be evaluated objectively?

What can be taken away from the judges and relegated to technological measurement with that can be developed right now?

What rules/guidelines would you change to make clearer for yes-or-no or quantitative assessments?

And then how would you handle all the qualitative assessments that judges are asked to make?

Do you want to remove all the "artistic" aspects of Composition and Presentation from the sport entirely?

Do you have a way to standardize determinations of poor, acceptable, good, very good, etc.? for various GOE and PCS criteria?

Or do you want to get rid of all qualitative assessments entirely and boil everything down to acceptable vs. unacceptable, in which case there would be no benefit for skaters to develop excellence?


Could technology might assist to measure things like height and distance on jumps, rotational speed and centering on spins, etc.,?
Could technology measure speed and ice coverage throughout a program?

Those seem within reach and could become aspects of scoring measured by instruments rather than estimated by eye.

But look at all the other bullet points for evaluating elements. Everywhere the guidelines refer to "good" or "poor" positions, or "effortlessness," or errors that have a range of reduction depending on severity, you're talking about value judgments. I don't see a way to turn these into yes-or-no answers in a way that is fair to skaters.

Same for most of the Skating Skills criteria and pretty much all of the Composition and Presentation criteria. Would you want to get rid of those two components entirely because they're inherently subjective? You could still have a sport of figure skating with elements and in-between skating with no music and no rewards or penalties for how the skater relates to the audience or how pretty the moves look. You'd just be losing most of the qualities that many fans and many skaters love about the sport.

In the Composition area, Pattern and ice coverage could probably be measured with instruments.
For Skating Skills, "speed" could be measured by instruments.
Amount of time spent skating forward vs. backward, clockwise vs. counterclockwise could be counted objectively, and also someone could count all the different types of turns and steps (as the tech panel does for step sequences) and kinds of connecting moves throughout program.
It would just need to be someone dedicated to counting those things in order to get an accurate count, and then there would need to be rules for how that information would get included in the scoring.

But when you get to determining "the ability of the skater to execute the skating repertoire...with blade and body control," "Clarity of edges, steps, turns, movements and body control," "balance and glide," "Flow," and also "Power" in relation to how the skater generates speed, across the program as a whole, it would be much more difficult either to train AI to evaluate these qualities or to train judges to evaluate them "objectively," since they are qualitative assessments.

How can you make evaluations of poor, fair, average, good, very good, outstanding into yes-or-no questions?
Do you have suggestions?

Or do you want to get rid of all qualitative evaluation of skating skill? In that case, we're not talking about skating itself being fundamental to the sport.

I do not know enough to make good suggestions of how to phrase things exactly.
But first of all: I do not want to get rid of things like PCS and GOE!!! And I think the current system is already pretty good, so I would basically keep it. I would phrase the PCS bullet points more precisely. And I would give them percentages and caps- so that there are no discussions such as "but skating skills are not everything, sure, he lacks speed, flow, glide and security, but the program is complex..." I would not deem it necessary per se, but since these arguments come up it might be necessary after all. How much is complexity worth?

I will have to make a much more detailed post to answer your post adequately.

 

[gkelly]

But first of all: I do not want to get rid of things like PCS and GOE!!! And I think the current system is already pretty good, so I would basically keep it.

Good to know.

You can find the current ice dance GOE guidelines in this document:

https://www.isu.org/figure-skating/rules/fsk-communications/31370-isu-communication-2569/file

They're set up differently from the singles/pairs guidelines. Is this closer to what you would want to see for those disciplines as well?

I would phrase the PCS bullet points more precisely.

Suggestions?

And I would give them percentages and caps- so that there are no discussions such as "but skating skills are not everything, sure, he lacks speed, flow, glide and security, but the program is complex..." I would not deem it necessary per se, but since these arguments come up it might be necessary after all. How much is complexity worth?

Maybe each PCS bullet point should get its own score. Then they could be weighted so that if the ISU decides that "Power and speed" is twice as important as "Variety of edges, steps, turns, movements and directions" (or vice versa), the system would weigh them against each other rather than asking judges to figure out their relative weight in coming up with an overall Skating Skills score.

I will have to make a much more detailed post to answer your post adequately.

Please do. I'm curious about what you're thinking.

 

[Magill]

Maybe each PCS bullet point should get its own score. Then they could be weighted so that if the ISU decides that "Power and speed" is twice as important as "Variety of edges, steps, turns, movements and directions" (or vice versa), the system would weigh them against each other rather than asking judges to figure out their relative weight in coming up with an overall Skating Skills score.

That would be great as it would allow to identify and separate perfectly measurable components, like speed, ice coverage, number of turns, proportion of one foot vs two feet skating etc, measure them with proper tools and have them assessed objectively and without human interference and shown separately on the scoring sheet. This would obviously help to move the whole thing towards more technology-based and objective process. .

 

[gkelly]

That would be great as it would allow to identify and separate perfectly measurable components, like speed, ice coverage, number of turns, proportion of one foot vs two feet skating etc, measure them with proper tools and have them assessed objectively and without human interference and shown separately on the scoring sheet. This would obviously help to move the whole thing towards more technology-based and objective process. .

I think you would probably need to have humans count the numbers of different kinds of turns, steps, transition moves, and evaluate the quality, at least in the near term.

But if you had someone solely tasked with the counting, as is the case for the tech panel in step sequences, that could free judges from keeping track of those numbers while evaluating quality.

Who would determine whether turns are clean or edges are steady? The counter or the judges?

 

[Magill]

I think you would probably need to have humans count the numbers of different kinds of turns, steps, transition moves, and evaluate the quality, at least in the near term.

But if you had someone solely tasked with the counting, as is the case for the tech panel in step sequences, that could free judges from keeping track of those numbers while evaluating quality.

Who would determine whether turns are clean or edges are steady? The counter or the judges?

AFAIK, you can measure the strength and angle with which the blade presses the ice, distribution of the skater's body weight along the blade as well as speed and direction of movements by using sensors. Probably some other features too and they will probably grow in numbers with each year. I guess this could bring answers at least to some of your questions.
As for counting turns, steps etc. I guess a simple app using slow mo good quality video could do it much quicker and with higher credibility than another volunteer. Plus it would not be biased.