Hypothetical discussion : How would you reduce national bias in judging figure skating

[gkelly]

Tech panel would do the same exact thing. To avoid national bias there should be no 2 persons on the panel from the same country.

That is already true. Not counting data entry operators, who don't have a say in the scores -- they just do data input.

The panel of 9 judges would be split in 3 panels of 3 judges each, and they will vote and reach a single mark (conclusion, see lower) so each single national biased mark/opinion will be outvoted in the panel.

By "voting" do you mean discussing, as the tech panel does now?
What the tech panel does is review video replay of the elements that were called for review during the performance.
During the review, the two technical specialists will say what they saw in terms of jump rotations and edges. If those two agree, great. If not, the technical controller will weigh in to break the tie.
The same process might be used if they need to review a spin, for example to determine whether the skater got into a position for 2 full revolutions (or 8 revolutions for the 8-rev feature) or got into the spinning position in less than 2 revolutions after a flying entry, or whether a particular variation meets the definition of "difficult," etc.

These are essentially yes-or-no questions, although in borderline cases different individuals may see things just differently enough to disagree on whether a jump landed at 90 or 95 degrees short, or whether a spin held a position for just under or just over the required revolutions.

So they quickly say what they see, discuss only if there's something especially confusing about what the skater did, and then the decision that at least two of the three agree on becomes the official call.

How would something similar work for judging?

Each panel can judge everything, as now.

OR we can have each panel specialized, one for GOE for jumps, one for GOE for spins and step sequences, one for components.

If "voting" just means entering their scores and taking the score agreed by the majority, that's essentially what we have now. (Except that what is actually used is the trimmed average.)

If you mean that subsets of the judges would discuss each element during the brief review period between the performance and the announcement of the scores, then I think you would definitely need to divide them into smaller groups and have each panel discuss only the subset of scores they are assigned to.

This would be taking place at the same time as the tech panel reviews. So the decisions that the tech panel comes to about the jumps (which are the only tech panel calls that the judges get to see) might not all be ready for the judges until toward the end of this brief period. If the jump judge panel needs to wait until all the tech panel jump calls are complete, that could double the length of the review process (tech panel, then judges), which could make the wait for scores a lot longer than 2 minutes.

If the judges are going to discuss, they would also need headsets, on a different channel than the tech panels or than the other judging subpanels, so they wouldn't interfere with each other's discussions.

The judges could not improve the call, just diminish if the tech panel called a jump clean but the judges see it under, it will be under, but not the other way around.

Judges already have the option individually to lower the GOE of an element for an error that they saw but that was not called by the tech panel.

So a halfway measure to increase the chance that skaters will be penalized for errors even if not called by the tech panel would be to encourage judges to give the maximum relevant GOE reduction if they see a downgrade/underrotation or wrong edge. As long as the judges have access to their own video replay, they can individually go back and check the jumps they saw problems with during the performance.

For e, <<, and < calls, the tech panel calls do also affect the base values. So what you're asking for is for a panel of jump judges to weigh in, beyond their GOE assignments, and duplicate the tech panel reviews for these elements with the possibility of overriding those calls/non-calls?

Ideally, each panel would go through all the bullet points and vote on those, and the computer will give the GOE. I think they should have enough time to do that in 2 minutes (because each panel looks at just jumps/ spins&step sequences and components)

Again, do you want a subset of judges to verbally discuss every possible bullet point for each element they're responsible for

Or only the elements they want to "review"? And should they each mention the positive bullets they awarded and then discuss any that only one or two of the judges mentioned, to come to a consensus of whether the panel of three should award it or not? S
Same for reductions -- each judge mention what reduction(s) they took for the element, and how much they took off for each error since most GOE reductions have a range of, e.g., 1-3 or 2-4?

What about the program components?
If the same judges are awarding program component scores and also discussing multiple elements, they will need a little extra time to input their component scores before or after the elements discussion.

If there will be a separate PCS panel, that would save time. But are the judges on that panel supposed to discuss each of the bullet points for each component and how good they thought that skater was at each of those areas?
Remember, component scores are not yes-or-no decisions.
Judges could say that they thought the skater was Good at one criterion, Very Good at another, only Average at another, etc. And then would they have to convince each other to come up with a single score from the panel for that component?
This would be a valuable process to go through during judge training, or maybe in the roundtable meetings after an event, but would take much too long during an actual event.

It would be quicker not to have discussions but to give judges an interface to score each bullet point separately and then let the computer take averages to come up with that judge's overall Composition or Presentation or Skating Skills score.

Or change to protocols to show the individual bullet points from each judge. Which would probably make each skater's protocol take a couple of pages instead of the current half page, but it would give the skaters (and the public) more detailed information about what the judges thought about their program construction and their skating and performance qualities.

 

[Mathman]

What I don't like about the swapping judges idea is that it would basically change the panel.

I thin k that this point kills the idea of the extra judge swapped in. You can't have a competition where one competitor is rated by one panel and another by a (slightly) different panel.

 

[4everchan]

If we stuck with national fed judges, the best way is to make sure that none of the top 3-5 contenders have a judge on the panel or that all of the top 3-5 contenders have a judge on the panel. The issues are exacerbated when one of the top contender doesn't have their own judge backing them up on the panel while their direct competitors do. You will see this at work for worlds... there are no North American judges in ice dance of all disciplines. Guess who may benefit from that ?

 

[4everchan]

And by saying this ^^, I also don't like the idea of judge swapping.

 

[Mathman]

Ideally, each panel would go through all the bullet points and vote on those, and the computer will give the GOE.

I saw your post on the other thread, but I didn't understand all the details. For individual bullet points for GOES and components -- what would they be voting on? Height of jump, yes or no? Speed across the ice, yes or no? Unison (for pairs) yes or no? How would the computer assign GOEs?

 

[Mathman]

With ISU Pro judges, no need for 9 judges however. Feds would not be represented and wouldn't cry about not having a judge in the event and of course, one would expect that the judging would be more homogeneous.

I am still far from convinced that a professional corps of salaried ISU judges would eleminate the problem (nor, for that matter, resilt in markedly more homogeneous judging -- should that be desired).

The professional judge from France is still from France. He grew up skating for France. He learned the rudiments of judging from French judges and coaches. He is friends with a lot of French skaters, coaches, local organizers, and federation officials. He tends to value value certain aspects of figure skating more than others -- and amazingly enough, these are the very aspects of skating that are most admired in France.

Yes, he attended a lot of seminars, examinations, practice judging, etc. sponsored directly by the ISU, But this is the case of an ISU-level judge now. Yes, as an employe of the ISU the hierarchy can hold his judging performances to rules of ethics and competence -- but they already do.

The main thing that is clearly gained is that the President of the French Federation can't punish him for not toeing the line. Only the ISU can do that.

 

[4everchan]

I am still far from convinced that a professional corps of salaried ISU judges would eleminate the problem (nor, for that matter, resilt in markedly more homogeneous judging -- should that be desired).

I am not

The professional judge from France is still from France. He grew up skating for France. He learned the rudiments of judging from French judges and coaches.

This is where you got this wrong : He learned the rudiments of judging from the ISU because there are no more French judges... but ISU judges. Of course, if you want to look at how it would be next week or next year, it may not be perfect, but down the road, the ISU pro judges would be the ones going around and teaching the rudiments of judging to a whole bunch of people, regardless of nationality.

He is friends with a lot of French skaters, coaches, local organizers, and federation officials. He tends to value value certain aspects of figure skating more than others -- and amazingly enough, these are the very aspects of skating that are most admired in France.

I hope people can have friendships and do their jobs properly. I certainly do

Yes, he attended a lot of seminars, examinations, practice judging, etc. sponsored directly by the ISU, But this is the case of an ISU-level judge now. Yes, as an employe of the ISU the hierarchy can hold his judging performances to rules of ethics and competence -- but they already do.

The main thing that is clearly gained is that the President of the French Federation can't punish him for not toeing the line. Only the ISU can do that.

And only the ISU can keep this judge active and working. High performing judges get high stakes events. It is the way it is in tennis. I know there are various levels of ISU certification for judges, but in each one of these categories, there are better judges than others. The homogeneity would be better greater with pro judges.

 

[gkelly]

This is where you got this wrong : He learned the rudiments of judging from the ISU because there are no more French judges... but ISU judges.

That is not true.

France will still need to have domestic judges to judge their local and national competitions. Those judges will be trained in France and will judge only in France. (Or possibly at club competitions in neighboring countries, etc.)

When the ISU is looking for candidates to train as international judges, would they be better off accepting applicants who already have real-world, often significant judging experience up to the national level?

Even if all their training came from international ISU judges who visit France and share their internationally-trained knowledge, and from ISU sponsored virtual training available identically to all judges around the world, they would still be learning in France and judging primarily French skaters, until and unless they apply to be trained and hired as ISU judges.

And they will discuss their judging experience with other domestic French judges at those domestic events and will probably learn from those discussions with more experienced domestic judges.

Or will everyone who has ever already learned to judge and served as a national or lower judge in any member federation be excluded from consideration, and all international judges must come from the pool of former skaters and non-skaters who have never judged a single competition in their life before, not even a 6.0-scored learn-to-skate event?

Even if the latter, you will still have skaters who grew up skating within a specific skating culture. And everyone, skater, family member, or fan, will have grown up listening to commentators from their country's broadcasters (although less so these days) and in their native language if possible. Even with all skating available for viewing by international streaming, prospective judges will have grown up listening to and being influenced by commentary in languages they understand and not by commentary in other languages that may be played during streaming for other sources but that is just background noise to a viewer who doesn't know the language.

Yes, all ISU judges are required to know English. But that doesn't mean they were fluent in English when they first became enraptured by figure skating as children.

One way or another, all international judges will have already absorbed a lot of their local and national skating culture before beginning the process of becoming international judges.

 

[4everchan]

That is not true.

Depends...

France will still need to have domestic judges to judge their local and national competitions. Those judges will be trained in France and will judge only in France. (Or possibly at club competitions in neighboring countries, etc.)

Not in my version The French judges who will judge locally will still be trained by the ISU. They will be volunteers fair enough but still trained by the ISU and not by the French Fed.

When the ISU is looking for candidates to train as international judges, would they be better off accepting applicants who already have real-world, often significant judging experience up to the national level?

That will also depend. I see it as a brand new crop of judges. Yes, I am perhaps too idealistic but this is why I call this thread hypothetical
It is aiming at not making visions based on the current system but creating a new one that would eliminate biases... This is why I said : please do not care about money... do not care about whatever limitations the current system imposes... What if you could make a brand new system.

Even if all their training came from international ISU judges who visit France and share their internationally-trained knowledge, and from ISU sponsored virtual training available identically to all judges around the world, they would still be learning in France and judging primarily French skaters, until and unless they apply to be trained and hired as ISU judges.

Also depends because the skaters and coaches would have a very good incentive to aim to please the panel of pro judges So the local judges would need to emulate as much as possible what is done in bigger events to give a real feedback to the promising talent.

And they will discuss their judging experience with other domestic French judges at those domestic events and will probably learn from those discussions with more experienced domestic judges.

Or will everyone who has ever already learned to judge and served as a national or lower judge in any member federation be excluded from consideration, and all international judges must come from the pool of former skaters and non-skaters who have never judged a single competition in their life before, not even a 6.0-scored learn-to-skate event?

I would love that to happen

Even if the latter, you will still have skaters who grew up skating within a specific skating culture. And everyone, skater, family member, or fan, will have grown up listening to commentators from their country's broadcasters (although less so these days) and in their native language if possible. Even with all skating available for viewing by international streaming, prospective judges will have grown up listening to and being influenced by by commentary in languages they understand and not by commentary in other languages that may be played during streaming for other sources

Yes. as I said to @Mathman , for a couple generations at most.. but look, we have had the IJS for two decades and it's still not working as well as we would like it to work... so why not give a similar chance to build something that could reduce national bias ?

Yes, all ISU judges are required to know English. But that doesn't mean they were fluent in English when they first became enraptured by figure skating as children.

Most kids nowadays learn English. I don't see this as a problem. The future pro judges would probably be in high school as we speak

One way or another, all international judges will have already absorbed a lot of their local and national skating culture before beginning the process of becoming international judges.

The local culture will be more aware of ISU requirements for good scores and will be less specific in terms of judging.

 

[4everchan]

PS... I think it's clear in my posts that I am not trying to renovate the judging system but completely build a new one... that's why I am not finding much limitations in the concerns expressed. Of course, I am aware that my concept is very far from reality and will not happen... but it doesn't mean it cannot be considered for hypothetical discussion. One step at a time, we may change this sport for the better

 

[Mathman]

I am not

I envy you your convictions. For me, the best I can do is try to keep an open mind.

THe omogeneity would be better with pro judges.

I wish you had said that the homogeneity would be greater Whether this would be better or not, that's another thing that I try to keep an open mind about.

 

[4everchan]

I envy you your convictions. For me, the best I can do is try to keep an open mind.

this is what is required when a conversation is hypothetical

I wish you had said that the homogeneity would be greater Whether this would be better or not, that's anpothier thing that I try to keep an open mind about.

correcting my English again greater is a better word indeed. (but I did spell homogeneity properly in my post though and you misquoted me... making me look like a francophone unaware of the letter 'H" which is most of the time silent for us )

 

[snowed]

all post

I don't have the patience or time to go through each line/ question, but if there is something you really think about pls ask me again but give me just 3 questions at a time...
There are 2 versions on my idea, one more extreme, and a mild one (like a band aid for what it is now). I think it got confusing as I mixed them, but I guess my point is that the idea could be implemented in phases, or not completely. If I would be in charge of this, firstly I would know everything there is in the rules (now I don't), and I would try different ways with help from a mock judges panel on old videos, firstly on paper, then tweaked. Only then a software would me made and tried on old competition videos.
The more extreme idea would be that the 3 panels of judges of 3 judges each, one for jumps, one for spins and steps and one for components. Being that they are so specialized they would have the time to go through each bullet. They don't have to talk about bullets, the computer will award the bullet if 2 of them awarded the bullet. For deductions there are no bullets but ranges of deductions, they can either put individually their deduction or vote as the tech panel does. I understand that the tech panel doesn't really discuss, but vote, one brings something up and the other says yes or no.
The jumps panel would have to vote if they would call edges or underrotations just as the tech panel.

For components I think it is more complicated as the current criteria still overlap and they try to award both quality and quality. Take ice coverage in composition, is not so much how many meters are covered but how it is spread across ice (with the intention to award difficulty in the sense of not going and jumping corner to corner and spinning in the center as in a practice session). But yes, I would like the judges to go through each criteria and give a percentage from 100% that would be an ideal program (or a mark up to 10 as it is now). If they have difficulty, then maybe ISU should work on making them more clear. The middle mark should be the final mark, but maybe if any mark is at a 1 point difference from the middle mark the computer would bring it up so the judge out of range could reevaluate.

This idea of mine would allow for more detailed judging (as I think judges do too much now), so more thoroug and correct. There would be no extra expense (I think this is an important point). I think it would cut the national bias (but only trying it would show for sure). The jumps panel would have time and a different angle to help the tech panel with calls. You said that the judges can do that now, but do they? no, because they are afraid they will be out of corridor" plus, as I said, I think they do to much as it is.

Me, as a fan, I would like to see this level of detailed judging. I guess the skaters would benefit from it too.

I saw your post on the other thread, but I didn't understand all the details. For individual bullet points for GOES and components -- what would they be voting on? Height of jump, yes or no? Speed across the ice, yes or no? Unison (for pairs) yes or no? How would the computer assign GOEs?

Yes, like that. As I understand how judging works now, each judge independently calculates each GOE by first giving a point for each bullet point and then deducting for mistakes using the deduction table.

 

[gkelly]

The more extreme idea would be that the 3 panels of judges of 3 judges each, one for jumps, one for spins and steps and one for components. Being that they are so specialized they would have the time to go through each bullet. They don't have to talk about bullets, the computer will award the bullet if 2 of them awarded the bullet.

OK

For deductions there are no bullets but ranges of deductions, they can either put individually their deduction or vote as the tech panel does. I understand that the tech panel doesn't really discuss, but vote, one brings something up and the other says yes or no.

Sometimes the tech panel has to discuss, when the skater does something that is confusing according to the rules. Most elements are pretty straightforward, but sometimes they have to figure out, e.g., which of two solo jumps in the SP (with or without falls) should get the +COMBO designation. Or whether a certain spin feature can't count for the third spin because it was already awarded for the first spin. Etc.

Currently the judges then have to make sure their GOEs match the tech panel calls. A +COMBO element in a short program must get -5, but the other solo jump maybe could get a less negative GOE.

But it would unnecessarily complicate things to have judges making the decision of which of those jumps to treat as the combo element independently of the tech panel's decision.

Currently, the judges never see the level determinations (let alone the specific features awarded). I'm not sure whether there would be an advantage to letting the spin judges know the levels called by the tech panel, but I can only see negatives to letting them second-guess the tech panel level calls.

The jumps panel would have to vote if they would call edges or underrotations just as the tech panel.

Vote by talking to each other? Or just by individually inputting a code in the computer along with their GOE?

For components I think it is more complicated as the current criteria still overlap ... I would like the judges to go through each criteria and give a percentage from 100% that would be an ideal program (or a mark up to 10 as it is now).

OK

This idea of mine would allow for more detailed judging (as I think judges do too much now), so more thoroug and correct. There would be no extra expense

As long as it really could be done quickly (data input about a few elements, or components only) with no discussions, and no extra equipment needed. If it adds a lot of time to the events, it would add expenses and perhaps reduce income if already-long events would end up lasting an hour longer.

Moving the tech panel or part of the judging panel would require at least small amounts of additional equipment. And might be impossible for some smaller events held in small venues where the ice surface is very close to the wall on some sides.

(I think this is an important point). I think it would cut the national bias (but only trying it would show for sure). The jumps panel would have time and a different angle to help the tech panel with calls. You said that the judges can do that now, but do they? no, because they are afraid they will be out of corridor" plus, as I said, I think they do to much as it is.

It would, however, be a lot easier to encourage judges to mark down errors that they see with assurances that they will not be penalized for disagreeing with the rest of the judging panel, than to set up a whole new system for overriding the tech panel calls, with significant software upgrades and significant retraining.

Me, as a fan, I would like to see this level of detailed judging. I guess the skaters would benefit from it too.

Yes, I think it would be interesting to us and to the skaters to have that kind of information.

If it can be done in a way that doesn't add too much time or clutter to make the competition even less appealing to casual viewers.

Yes, like that. As I understand how judging works now, each judge independently calculates each GOE by first giving a point for each bullet point and then deducting for mistakes using the deduction table.

Yes, that's correct.

 

[Mathman]

this is what is required when a conversation is hypothetical

correcting my English again greater is a better word indeed. (but I did spell homogeneity properly in my post though and you misquoted me... making me look like a francophone unaware of the letter 'H" which is most of the time silent for us )

You beat me on that one. First, I had to look up the spelling of the word homogeneity, while you knew it right off. As for that capital H, thet's a typo. My pet peeve when it comes to keyboard dising is that the "Caps Lock" key is so close to the "A" key that I keep hitting it by mistake.

Sdit: I mean "design."

Edit" I mean :edit."

On the substantive point, though, the quoestion of whether a homogeneous set of marks represents better judging or worse judging than a set of mark with more variation, that is not completely self-evident to me.

 

[4everchan]

You beat me on that one. First, I had to look up the spelling of the word homogeneity, while you knew it right off. As for that capital H, thet's a typo. My pet peeve when it comes to keyboard dising is that the "Caps Lock" key is so close to the "A" key that I keep hitting it by mistake.

Sdit: I mean "design."

Edit" I mean :edit."

On the substantive point, though, the quoestion of whether a homogeneous set of marks represents better judging or worse judging than a set of mark with more variation, that is not completely self-evident to me.

Consensus and stability could lead to better understanding of the sport by casual fans, bringing broader appeal and getting figure skating out of its niche.

 

[snowed]

Sometimes the tech panel has to discuss, when the skater does something that is confusing according to the rules. Most elements are pretty straightforward, but sometimes they have to figure out, e.g., which of two solo jumps in the SP (with or without falls) should get the +COMBO designation. Or whether a certain spin feature can't count for the third spin because it was already awarded for the first spin. Etc.

Currently the judges then have to make sure their GOEs match the tech panel calls. A +COMBO element in a short program must get -5, but the other solo jump maybe could get a less negative GOE.

But it would unnecessarily complicate things to have judges making the decision of which of those jumps to treat as the combo element independently of the tech panel's decision.

Currently, the judges never see the level determinations (let alone the specific features awarded). I'm not sure whether there would be an advantage to letting the spin judges know the levels called by the tech panel, but I can only see negatives to letting them second-guess the tech panel level calls.


Vote by talking to each other? Or just by individually inputting a code in the computer along with their GOE?

Only the judges panel for jumps would "help" the tech panel, and only for edges and underrotations (supposedly missed by the tech panel because of line of view live and video). Any of these 3 "jump" judges could flag a jump for edge or under and they will vote as the tech panel does. Hopefully they would have a video from their angle, (maybe the commercial video), but even without a video they would catch some calls that the tech panel misses

The +Combo +Rep calls for jumps, all other calls and levels for spins and steps would still be tech panel job.

 

[Mathman]

Yes, like that. As I understand how judging works now, each judge independently calculates each GOE by first giving a point for each bullet point and then deducting for mistakes using the deduction table.

Yes, that's correct.

The funny thing is that this count-up-the-bullet-points is already out of synch with what we usually regard as attempting to measure quality rather than quantity (though it is very much in the spirit of the IJS in general). Here are the bullet points for jumps.

1) very good height and very good length (of all jumps in a combo or sequence)
2) good take-off and landing
3) effortless throughout (including rhythm in Jump combination)
4) steps before the jump, unexpected or creative entry
5) very good body position from take-off to landing
6) element matches the music

None of these (except possibly #4) is really a yes or no propostiion. One skater might jump more effortlessly than another. One judge might think that skater's body line is "very good," another that it's only "good."

 

[gkelly]

Only the judges panel for jumps would "help" the tech panel, and only for edges and underrotations (supposedly missed by the tech panel because of line of view live and video). Any of these 3 "jump" judges could flag a jump for edge or under and they will vote as the tech panel does. Hopefully they would have a video from their angle, (maybe the commercial video), but even without a video they would catch some calls that the tech panel misses

Maybe the easiest way to do that would be to have buttons that the jump judges could click to indicate whenever they take a GOE reduction for what they saw as <<, <, or e errors.

Then the "voting" would be through the computer: if two or all three of the judges press the relevant buttons, the computer would either alert the tech panel to review the other video angle, or if you really think the judges should override the tech panel it could just apply the lower base value based on the judges' pressing the buttons.


Oh, and there would be other questions about how this system could be used for lesser events. For domestic events, national bias is not relevant. For small internationals, nine judges per event to be able to divide the panels like this, and an additional camera, could create a financial burden for the host clubs/federations.
So should skating get more expensive at the grass roots level (as it already has with the introduction if IJS vs. 6.0)? Or should lower events use a different system than the elite events?

 

[snowed]

The funny thing is that this count-up-the-bullet-points is already out of synch with what we usually regard as attempting to neasure quality rather than quantity (though it is very much in the spirit of the IJS in general,) Here are the bullet points for jumps.

1) very good height and very good length (of all jumps in a combo or sequence)
2) good take-off and landing
3) effortless throughout (including rhythm in Jump combination)
4) steps before the jump, unexpected or creative entry
5) very good body position from take-off to landing
6) element matches the music

None of these (except possibly #4) is really a yes or no propostiion. One skater might jump more effortlessly than another. One judge might think that skater's body line is "very good," another that it's only "good."

Let's take 1) very good height and lenght, as I understand it doesn't have to be the highest and longest jump, it just have to have these 2 characteristics at the same time to earn that bullet. A correct jump should have some height and lenght, the bullet ask for "very" so in my interpretation the jump should be a little better than a correct average jump, that's it. And it is not that easy especially while skaters develop their technique and their difficulty content. Some skaters jumps are height but don't have lenght (if they don't enter with speed), and others have lenght but not height (because of technique). I guess this specific bullet is easier to see in person...

I think the other bullets can be seen in the videos just fine