Randomization of Judge's Scores (the elephant in the room?)

I'll start this off with a sentence that will likely derail this thread right off of the bat. I do not mind anonymous judging. In fact I believe it is a maybe as important to fair judging as anything. To elaborate and to hopefully ward off an assault of internet tomatoes being thrown my way. I simply don't care what country a judge is from nor do I care how their last name is spelled and if they are from Europe, Asia, or North America. All I care about is being able to see the consistency in their judging patterns during an event. Again I'm feeling a need to clarify myself. I don't care if Judge 1 scores skater A 6.75 in skating skills and Judge 8 scores skater A 5.75 in SS. What I want to see is how Judge 1 and Judge 8 scored the rest of the field relative to how they scored Skater A. Judges not only will disagree when quantifying absolute numbers to award the exact same performance I contend it is to be expected and healthy. What is of absolute importance to me and the best way to see if a judge is scoring fairly is checking a few performances to gauge their marks. Maybe their scale of values is different than mine and yours. Even if so and as long as they apply it equally across the board, then who cares?

I see no need to publish the names and countries of the judges selected in a random draw. Keeping the judges anonymous will however empower some judges to avoid pressure put on them from home town coaches and officials to "see things their way". I'm not naive enough to think that home judges won't score more beneficial to their own federations skaters but in the end I 100% believe this is the most reliable way to curb that to a minimum. I do however think it is of absolute importance to be able to measure what a particular judge's scale of values is and more importantly how it is applied during an event. I don't think it's asking to much and I don't know why this isn't the issue brought up more often as opposed to anonymity. Does anyone agree that maybe there is an elephant in the room?

Anonymity is ridiculous. Never mind absolute ranking of skaters by each judge. Although that's simply another stupid measure by the ISU that continues to minimize accountability. It's no secret that the IsU isn't committed to the fairest judging otherwise judges would immediately have their scores shown. As far as buying off the judges if they are known - that is the most bs rationale the isu has ever given and one of the most faulty logic ever (a la we'll give more guns to avoid mass shootings)

CanadianSkaterGuy said:

Anonymity is ridiculous. Never mind absolute ranking of skaters by each judge. Although that's simply another stupid measure by the ISU that continues to minimize accountability. It's no secret that the IsU isn't committed to the fairest judging otherwise judges would immediately have their scores shown. As far as buying off the judges if they are known - that is the most bs rationale the isu has ever given and one of the most faulty logic ever (a la we'll give more guns to avoid mass shootings)

Click to expand...

I just don't see how knowing where a judge is from or who they are matters in the least. As long as their scores are able to be known than what is the big deal if they are from Spain. Certain posters here are constantly proclaiming events unfair based on the judging panel's nationalities and often before they even have a chance to begin. Of course when it's a home event the crickets come out and all is good. The point is.....who cares who assigned a score if it was done fairly across the board using a reasonably simililar scale of values. To me.....this seems like a much easier battle to win with the ISU and one that the federations and coaches might get behind. We've already seen the results of putting anonymous judging up to a vote. Maybe this is a fair compromise and one that is more than satisfactory for all parties involved.

I feel like randomization and anonymity go hand in hand. It seems possible, if not probable, that the ISU brought in randomization to further protect identities and to make it impossible to tell where each judge is from. It could be possible to extrapolate a judge's nationality from their scores if, for instance, they scored everyone fairly but gave the Spanish skater notably higher marks. People would say, "Aha! Judge 1 is probably the Spanish judge!" The ISU, in its commitment to anonymity, would not want this.

I cannot help but wonder if the ISU is actually quite okay biased or block judging, and simply brought in anonymity and randomization to hide the problem. Alternatively, it is possible that they feel like they cannot stop biased judging, and so they are trying to sweep it under the rug instead. I am convinced that they want to avoid the type of evaluation of a judge that we would all like to do.

I think it is all part and parcel of the ISU mind set on the anonymity issue. Before randomization it was pretty easy to figure out which judge was which, which judges were advancing the fortunes of which countries, and which judges were in cahoots one with another -- or so we amateur sleuths believed, anyway. Randomization of judges' marks threw a monkey wrench into our fun, and also prevented us from figuring out who would have won under 6.0 judging, and other convivial stuff like that. So I am all for eliminating this feature of the IJS.

As for anonymous judging itself, I do not subscribe to the ISU party line that says that anonymity prevents the Godfathers of the National Federations from finding out whether their minions (the judges) betrayed the Family or not. There are ways, there are ways…

Edited to add Moonlight Skater beat me to it. :yes:

Sam-Skwantch, your OP is entirely reasonable. Too sensible to become reality though, of course.

Opinions may differ, but you made a good case for yours...

I agree with MoonlightSkater & Mathman, without randomization there's no true anonymity and therefore incomplete obscuration. The only problem is the tech panel. I think they should use three tech panels for an event and randomly choose the result of one tech panel for perfect disguise. Who cares about the fans and the skaters? Who cares about simplicity and transparency?

Mathman said:

I think it is all part and parcel of the ISU mind set on the anonymity issue. Before randomization it was pretty easy to figure out which judge was which, which judges were advancing the fortunes of which countries, and which judges were in cahoots one with another -- or so we amateur sleuths believed, anyway. Randomization of judges' marks threw a monkey wrench into our fun, and also prevented us from figuring out who would have won under 6.0 judging, and other convivial stuff like that. So I am all for eliminating this feature of the IJS.

As for anonymous judging itself, I do not subscribe to the ISU party line that says that anonymity prevents the Godfathers of the National Federations from finding out whether their minions (the judges) betrayed the Family or not. There are ways, there are ways…

Edited to add Moonlight Skater beat me to it. :yes:

Click to expand...

Well if federations are paying judges to score the way they want, transparent judging would reveal this. And if a federation submits judges who continually receive infractions for suspect judging then those federations should be penalized by how many judges they are allowed to submit for certain events if they can't submit fair judges.

For me, while the reasoning behind anonymity works in theory, it hides those who use it to cheat or judge with a clear bias. What should've/needs to happen is transparent judging where the marks are known and swift, harsh disciplinary action taken when judges–or federations–are caught awarding scores with a clear bias. Already, when looking close enough at the protocols, we can sometimes find what looks like biased scoring (especially in the components). But then on the other hand, I hate that judges are forced to judge within a corridor/aren't free to judge how they feel (but that's another discussion). And while I don't think some GOEs need to be restricted, particularly when a skater is already getting punished with a reduction in base value (e.g., an under-rotated jump that is landed and looks fine/doesn't disrupt or mar the program, like Mao's 3F+3Lo combination in Sochi or at Worlds or Adam's 4Lz at Nationals), judges should be forced to explain why they didn't give a negative GOE to elements with visible errors and maybe even given warning for not following the rules (x amount of warnings until you're suspended for a period of time).

Plus, it's fun/would be fun to see how specific judges score skaters and fun to compare those judges' scores across events.

I too think that randomisation is in aid of anonymous judging, and its sole cause. If there was no anonymity, there would obviously be no reason for it to exist. And without it, anonymity doesn't really exist.

I don't mind anonymous judging anymore, but my reasoning is different because no matter what you do, you cannot make subjective judging fair. Anonymous or not, it is just too easy to manipulate scores in a sport that does not have a definitive measure of winning.

Melon said:

I don't mind anonymous judging anymore, but my reasoning is different because no matter what you do, you cannot make subjective judging fair. Anonymous or not, it is just too easy to manipulate scores in a sport that does not have a definitive measure of winning.

Click to expand...

But there's certainly a lot of room for improvement. Ski jumping f.e. had problems with judging in the past and they have changed rules so that obvious problems don't occur nowadays. It's not that the scoring is always perfect, but it's ok. I couldn't ask for more. Anonymous judging isn't acceptable, it's as if a judge in court would give a sentence without a reason. Does anywone know another sport where there is anonymous judging?

Melon said:

I don't mind anonymous judging anymore, but my reasoning is different because no matter what you do, you cannot make subjective judging fair. Anonymous or not, it is just too easy to manipulate scores in a sport that does not have a definitive measure of winning.

Click to expand...

What do you consider fair? Any judge who agrees with you or is there something more to it than that? I've given my examples of why all I want to know is how consistent a judge is throughout an event. I may totally disagree with them but I'd still be able to call it fair if it was evenly distributed in a fairly consistent pattern.

Not exactly sure why it matters to so many people where the judge is from or what their last name is.

Is it possible to discuss the effect randomization has on the scores without all of the anonymous judging rants?

psusanne said:

But there's certainly a lot of room for improvement. Ski jumping f.e. had problems with judging in the past and they have changed rules so that obvious problems don't occur nowadays. It's not that the scoring is always perfect, but it's ok. I couldn't ask for more. Anonymous judging isn't acceptable, it's as if a judge in court would give a sentence without a reason. Does anywone know another sport where there is anonymous judging?

Click to expand...

I think every sports needs improvement, and FS definitely is one. Heck, even at FIFA world cups the refs sometimes do not call for off-sides, and there's always at least one controversies in Superbowl. Unfortunately, FS is not like speed skating where time is a factor in winning.....it is just too subjective IMO. What I'm trying to say is, anonymous judging is a problem because this sport is subjective; therefore, it really does not matter if FS has transparent judging or not..... it's unfortunate...but true

Sam-Skwantch said:

Is it possible to discuss the effect randomization has on the scores without all of the anonymous judging rants?

Click to expand...

Can you discuss the elephant in the room without mentioning the elephant?

Melon said:

I think every sports needs improvement, and FS definitely is one. Heck, even at FIFA world cups the refs sometimes do not call for off-sides, and there's always at least one controversies in Superbowl. Unfortunately, FS is not like speed skating where time is a factor in winning.....it is just too subjective IMO. What I'm trying to say is, anonymous judging is a problem because this sport is subjective; therefore, it really does not matter if FS has transparent judging or not..... it's unfortunate...but true

Click to expand...

Ski jumping is in part subjective and in part measurable and competitions are often very close. I think it's comparable, altough the figure skating juging system is far more complicated naturally.

Sam-Skwantch said:

Not exactly sure why it matters to so many people where the judge is from or what their last name is.

Click to expand...

People would like to be able to identify incompetence and bias, and to hold judges responsible for their marks. Back in the 1970s it got so bad that the whole Soviet team was suspended from figure skating judging for a year. Fans and skaters have a right to know what to expect when they see the names Balkov or Shekhovtsova on the judging panel. If a judging panel has, let us say, three judges who are naturally allied, and sure enough, judges #2, #5, and #8 all low-ball a certain rival skater -- yes, fans want to know who the culprits are.

Is it possible to discuss the effect randomization has on the scores without all of the anonymous judging rants?

Click to expand...

No, I don't think that is possible.

Anonymous judging was instituted by the ISU to ensure, after the Salt Lake City scandal, that never again would cheating judges get caught, leading to embarrassment and criticism of the ISU. A couple of years later Cinquanta decided that anonymity still had some leaks, and moved to plug them by randomization.

Mathman said:

People would like to be able to identify incompetence and bias, and to hold judges responsible for their marks. Back in the 1970s it got so bad that the whole Soviet team was suspended from figure skating judging for a year. Fans and skaters have a right to know what to expect when they see the names Balkov or Shekhovtsova on the judging panel. If a judging panel has, let us say, three judges who are naturally allied, and sure enough, judges #2, #5, and #8 all low-ball a certain rival skater -- yes, fans want to know who the culprits are.

Click to expand...

What if those "fans" take that "right" to know too far? We all saw what happened when Adelina was guilty only of skating very, very well - the death threats and abuses were sickening.

Mathman said:

Back in the 1970s it got so bad that the whole Soviet team was suspended from figure skating judging for a year.

Click to expand...

Did it or it was just another side of the Cold War? I found a study regarding the national favoritism during the 6.0 era and posted here the link to it. Everyone had national bias as was shown by statistics. And it was also "the block bias". As the number of the eastern block judges was on average lower I could speculate that they were somewhat more "zealous" and only toward the top skaters. In fact that particular case referred to Rodnina-Zaitsev who would win anyway even without Piseev's 6.0. His point of view was that the ISU's tech comittee Bianchetti was tired of the Russian domination in dance and pairs and she hated Piseev personally so that it was just a pretext to get the better of him. In fact the dominance continued for a while and stopped not because of the "better judging"

karne said:

What if those "fans" take that "right" to know too far? We all saw what happened when Adelina was guilty only of skating very, very well - the death threats and abuses were sickening.

Click to expand...

Agree - that was one of the ugliest moments in FS history.

My thoughts:

From an outside observer point of view, fans may want to

*Be able to analyze patterns of judging to try to figure out the thought processes of the judges behind scoring the actual skating according to the rules -- for this purpose, all that's needed is to keep the scores for each judge in the same column; actually naming them is not needed. IJS gives a lot more information of this sort to work with than 6.0, but randomization of the judges' columns skater by skater frustrates those efforts.
This is what the original post asks for, and what would also be of most interest to me personally.

*Be able to analyze patterns of judging according to nationality and individual judges' scoring histories at previous events. This allows fans to try to divine individual judges' thought processes, including possible conscious or unconscious biases as well as apparent pet peeves, etc.

*Feel able to identify dishonest judging, able to act as outside oversight to keep judging honest. This would require being able to identify which judge (by name and/or nationality) gave which scores to which skaters.

Because 6.0 scoring published scores with judges identified by name and nationality and each judge's scores all in the same column for all skaters in the event, but only one or two scores for each skater from each judge in a each competition phase, it was easier to analyze in terms of nationality than in terms of what the judges thought of specific aspects of the skating.

Dishonest judging did happen sometimes.

Because the way scores were presented under 6.0, whenever fans or commentators etc. disagreed with results, the first explanation was usually to look for patterns of national bias.

Also, it's not necessary to have watched the skating or to know anything about skating technique or rules to see nationalist patterns in the scores.

The non-randomized IJS protocols that give detailed scores and also allow scores to be associated with specific judges -- as we see in the JGP and other non-elite international events and domestic events in the US and elsewhere -- would provide the best of both worlds for purposes of outside analysis.

The international events of most interest to fans -- ISU Championships, Olympics, Grand Prix -- are also the places that dishonest nationalist judging is most likely to occur.

From the ISU point of view, according to their explanations, anonymity increases judges' ability to judge honestly at those events by shielding the judges from pressure by their federations or others who might seek to influence results.

Even with the judges' columns presented in different order on the detailed protocols than on the cover sheet listing the judges, it was still possible to figure out which column belonged to which judge, which defeated the stated purpose. Hence the introduction of randomization.

A side effect of anonymity is that fans, the press, or other outside observers can't identify patterns of nationalism or bloc judging. With randomization to improve the effectiveness of anonymity, we can't figure out patterns in the actual skating evaluation either.

That frustrates fans who like to analyze scoring patterns for any of the reasons listed above (to understand and/or to pass judgment against judges).

It may or may not also be an intended purpose of the ISU in introducing randomization -- to keep dishonest judging hidden from the public in order to minimize scandals.