The Judging Controversy Thread

[OS]

Ran the "corridor" numbers. First observation, really really wide corridor for GOE. For example, Adelina scored 22.78 total GOEs (best of night i think). Judge is allowed to be off by +/- 12. In other words, judges can be 50% higher or lower than everyone else and still be in "corridor". For PCS, "corridor" range is 7.5 points, which given total PCS (pre 1.6x multiplier) can be ~45 (assuming 9s), only allows judges to be about ~17% off from average. Perhaps this is why GOE stdev is so much wider than PCS.

No judge tripped "corridor". Closest for GOE was one judge's scores for Caro which totaled 11.2 off average (just below the 12 cutoff). Closest for PCS was one judge's scores for Yuna which totaled (4.4) off average (versus cutoff at 7.5).

Interesting analysis. A couple of things come out of this

A. Did Adelina receive the the highest total GOEs EVER in the history of this sport based on that performance? Surpass anything by Chan, Takahashi, Hanyu, Kim, Mao, Kostner?

B. If so, it seem to indicate GOEs is more vulnerable to manipulation than PCS, because the point variation from PCS at elite level are usually between 8 to 10, roughly 2 points separation and 10 is a rarity. They are Factored differently for short and long which makes FS a far more important event. With GOEs scale of +-3 can be a matter of up to 6 points differences, with 30% reduced GOE since Vancouver reduces the advantage of superior execution. This actually made the Adelina's GOEs scoring even more incredible consider it has been reduced by 30% since Vancouver. At Vancouver, that'd be worth 29.614 (22.78 *1.3) .

C. What also struck me about Adelina's PCS is that it beat Kim's at Vancouver with a inferior program and performance and no precedence in PCS to justify these marks. How is it fair for someone like Kim/Mao/Kostner to take 7/8 years of world championships to earn their PCS, a young skater with no major podium finish (WC, GPF) can gain in a matter of few weeks?

D. The fundamental flaw of all the number crunching analysis is that it assumes people's marking are genuine fair across the board based absolute scale instead of abusing it like ordinal scale of relative scoring. That all judges are marking skaters with the same degree of good conscious and impartiality. That they have no agendas to prop their own skaters while repress their direct rivals? It ultimately shows why anonymity must go, if agenda can not be erased under the COP system.

A useful analysis is not possible without taking account of past judging history, the variance in scoring trends and who they marked.

For example here are the facts.

Despite Adelina and Yulia's young and short careers, why are the judges who gave their most important wins in their young career are on the same Sochi FS panel

Some of the judges that sticks out are are

1. Alla SHEKHOVTSEVA of Russia
- Other than her personal conflict of interest married to the director general of Russia Federation.
- She were at the panel of every single one of Adelina's competiton this season. She is practically her personal judge.
- Marked Adelina and Yulia's at European Championship which saw her PCS rise by 9 points in her FS even though she was 2nd in the FS behind Yulia.
Adelina coming 2nd at that FS received the 7th highest PCS of ALL TIME then. Is this unusual or merely accidental?
- Marking the Russian skaters, at home. She shares no same agenda as her husband I presume.

2. Helene CUCUPHAT of France
- Responsible for 3 of the most important win in Sotnikova's career.
- Marked Adelina at Junior worlds 2012 (Where Yulia was 1st, Adelina 3rd)
- Marked Adelina at European Championships which saw her PCS rose to the 7th highest of all time despite coming 2nd.
- Marking Adelina again at Sochi.

3. Zana KULIK of Estonia
- Responsible for Adelina's most critical win up to date, 2011 Junior world championships.
- Judged here at Sochi which makes a nice Eastern bloc judging if skating history has proved to rear its ugly head again here in Sochi.

4. Can a panel consist of an Italian judge and a Japanese judge and no Korean judge in the FS affect the scoring for Kim? We simply don't know under anonymous judging.

5. Yuri BALKOV of Ukraine.

- Should have been banned for life from his cheating ways at 1998 Winter Games in Nagano, Japan.
Balkov was taped by the Canadian judge Jean Senft explaining what order the competitors would finish in the ice-dancing competition before it took place. He was suspended for one year. He returned to judging and is certified by the international federation next year.
- Yet he judged ice dance panel of judges at Salt lake.
- He is here again at ladies event, does he even have the experience to judge the ladies? Why is he here? Why wasn't he vetoed by ISU's internal Quality control?

Consider all the above, what are the chances they all come together on the same panel on the most important event in Russia ladies figures skating since Salt Lake 2002?

I did wonder what happened to that incredible whistle blower Canadian judge Jean Senft, so I googled and found this old article of her from the TIMES

http://content.time.com/time/arts/article/0,8599,203274,00.html

This bit caught my eye and made me loled since I have been saying the EXACT same thing on Gkelly's judge selection thread. It would be wonderful if this can be adapted as part of the reform process.

TIME: What kind of action would you recommend the ISU take?

Senft: I think they should move to having judges who are part of an independent association. They should no longer be assigned to judge these events according to their national federation. I also think judges should be hired and fired on the basis of competence.

A reform is really necessary to protect the judges as well as the skaters and the sport. For example all the above judges mentioned could have been completely innocent despite what their profile indicate a clear conflict of interest to impartiality to the competition to treat ALL skaters equally, but there's just no way to prove it under anonymity.

They should just reveal all the marking of all the judges and simply let the mark speak for itself. If there are indeed deliberate biases or sabotage then these judges should be penalized, end of. Without removing this shield of secrecy, it encourages all sort of speculation that does the reputation of the sport, the skaters involved no good and it is simply unfair to them all.

Few rotten apples and incompetent governance of this sport should not ruin the great work of every hard working judge who dedicate their life uphold the integrity of the sport with honesty and compassion. Remove anonymity, improve transparency and accountability. Make the ISU judges a professional organization, well run and governed with strong ethical guidance in place. Without it, the sport is sham and will only ever be a tug of war among the old boys' club.

 

[Meoima]

Interesting, but I think investigate these judges alone is not objective enough. What about the rest of judge panel in both free skate and SP program? I think we should do some more research. Because of the anonymous judging, we could never know for sure which judge gives which score.

 

[OS]

Interesting, but I think investigate these judges alone is not objective enough. What about the rest of judge panel in both free skate and SP program? I think we should do some more research. Because of the anonymous judging, we could never know for sure which judge gives which score.

Yeah I totally agree with that. how a 3t3t program beat a superior 3lz3t program, skating order is not good enough of an explanation since the judges should be well aware of who is participating.

Usually if the skater is lowballed in the SP, I do notice the judges will try to make up for it in the long program. There are also usually bonuses for skating clean and particularly skating last (like the 6.0 idea), did it happen at this competition?

 

[leafygreens]

1. Alla SHEKHOVTSEVA of Russia
- Other than her personal conflict of interest married to the director general of Russia Federation.
- She were at the panel of every single one of Adelina's competiton this season. She is practically her personal judge.
- Marked Adelina and Yulia's at European Championship which saw her PCS rise by 9 points in her FS even though she was 2nd in the FS behind Yulia.
Adelina coming 2nd at that FS received the 7th highest PCS of ALL TIME then. Is this unusual or merely accidental?
- Marking the Russian skaters, at home. She shares no same agenda as her husband I presume.

Does the Russian fed even have any other judges? It would appear not. So in one of the largest feds in the world they have to keep using the same judge for ladies events. Not suspicious at all. :sarcasm:

 

[capcomeback]

A technical controller, like Lakernik, has a lot of power toward calling (or ignoring) edge calls and under-rotations, and determining spin and step levels. However, he can be overruled if the both of the other 2 members of the tech panel disagree with him. The correct link is the Technical Manual for how technical calls are made, and
http://www.isu.org/en/single-and-pair-skating-and-ice-dance/special-regulations-and-technical-rules

page 62 of Single and Pair Skating and Ice Dance Special Regulations and Technical Rules, where the job of the Technical controller is described.

The question is would one (let alone two) of the callers have the stones to go against his calls (since he is the head of the ISU Technical Committee).

 

[Sam-Skwantch]

The question is would one (let alone two) of the callers have the stones to go against his calls (since he is the head of the ISU Technical Committee).

That's kind of unfair and purely speculation. Is it not? We need to focus on facts alone if this is going to go anywhere. Character assassination is not going to prove anything except personal bias.

 

[Meoima]

Does the Russian fed even have any other judges? It would appear not. So in one of the largest feds in the world they have to keep using the same judge for ladies events. Not suspicious at all. :sarcasm:

If I remember correctly, she has been judge for years, a familiar face, huh? I don't know. Maybe she is influence (like Russian federation trust her) or it's her husband's influence. Why it's always her?

 

[capcomeback]

That's kind of unfair and purely speculation. Is it not? We need to focus on facts alone if this is going to go anywhere. Character assassination is not going to prove anything except personal bias.

Not meaning to assail anyone's character. My point is that when you have a very powerful individual in the ISU (and don't kid yourself, Lakenik is a powerful ISU official) who also is the Vice President of the Russian Ice Federation who is the defacto supervisor of the Technical Panel (as he has the last word unless overruled by both callers), it creates an untenable position for the callers to have a very powerful someone with a major conflict of interest (and who can determine how many events you will get to work, as he is the Chairman of the ISU Technical Panel) as their boss. Do you think in the case of an edge call or UR that might be relatively close enough (based upon the angle they choose to view it from), that it would be likely that either caller would not be willing to give Lakernik's view "the benefit of a doubt"? The question then becomes would any skater get that same benefit of a doubt?

My point is that there are too many people in positions of power in the ISU who have serious conflicts of interest and it is perfectly reasonable to believe that this crept into the officiating. I think it is fair for all of us to look at the way the ladies' event was officiated to treat it with a jaundiced eye.

 

[capcomeback]

Does the Russian fed even have any other judges? It would appear not. So in one of the largest feds in the world they have to keep using the same judge for ladies events. Not suspicious at all. :sarcasm:

Here is a list of ISU Judges from Russia who can judge singles/pairs (as you can see, 7 others judges could have been selected for the ladies' pool, as two of the other ten would be judging the men's and the pairs events):

Abasova Maira, Ms.
Andreeva Julia, Ms.
Babenko Sviatoslav, Mr.
Dolgushin Igor, Mr.
Fomina Elena, Ms.
Kitashova Sofia, Ms.
Kozhemyakina Olga, Ms.
Labunskaya Lolita, Ms.
Primachenko Natalia, Ms.
Shekhovtseva Alla, Ms.

Source:

INTERNATIONAL SKATING UNION
Communication No. 1812
List of Referees, Judges, Technical Controllers, Technical Specialists, Data & Replay Operators 2013/14 for Single & Pair Skating, Ice Dance and Synchronized Skating (dated 10-08-2013)
http://static.isu.org/media/107397/1...ated-oct-8.pdf

 

[leafygreens]

Abasova Maira, Ms.
Andreeva Julia, Ms.
Babenko Sviatoslav, Mr.
Dolgushin Igor, Mr.
Fomina Elena, Ms.
Kitashova Sofia, Ms.
Kozhemyakina Olga, Ms.
Labunskaya Lolita, Ms.
Primachenko Natalia, Ms.
Shekhovtseva Alla, Ms.

Which of these judges have been at Adelina's other events?

 

[gkelly]

Do we really think there was a plan in advance to support Adelina and only Adelina?

If anything, there may have been a plan to support the Russian skaters. If Yulia had skated better in the individual event, she might well have gotten higher scores than Adelina, and won the event.

 

[OS]

Do we really think there was a plan in advance to support Adelina and only Adelina?

If anything, there may have been a plan to support the Russian skaters. If Yulia had skated better in the individual event, she might well have gotten higher scores than Adelina, and won the event.

I definitely think it is not just Adelina but Russians in general including Yulia. I love Yulia since she was at juniors along with young liza but am sensitive to their struggle at home because Adelina has always been perceived as Russia's favourite and pushed as Russia's no.1 (Putin, Tat are powerful backers). Traditionally Adelina have always received had higher PCS than other young Russians at home even though Yulia is more consistent and I'd argue have a far better packaged program this year.

I thought it was smart for them to use Yulia for teams because whatever PCS she get boosted there, Adelina benefit similar to a spring board to gain even more inflation without having to skate just because she has always earned higher PCS than Yulia in Russia.

I thought Adelina was held back so she can reserve all her energy to the ladies singles event only instead of tiring out and risks ruin her chances with the judges. The Russians probably didn't realise just how much Yulia would have become a media darling and a bigger contender than Adelina internationally.

 

[Mathman]

Thanks for the link Mathman. While I'm not as much of a fan of math as you guys are I'm very interested in the results. It's very fascinating to see how they search out anomalies. Shall I just assume the ISU has already done this excercise?

Yes, although no whistles go off until a particular judge has failed this test three times.

I'll have to read it again to get a better grasp but if a judges scores become an outlier and therefore not used is the judge still responsible to stay within the corridor?

Yes. In fact, the average that determines the middle of the corridor includes not only all the nine judges, but also the marks of the referee, who is scoring along with the judges. (If there are fewer judges on the panel, the referee counts twice).

The corridor is deliberately set so wide that it is practically impossible ever to be outside it. I think the whole idea of the corridor is actually to spot incompetent judges at the lower levels who might need more training, even though the document speaks about catching national bias as well.

The drawback to this corridor test is that it promotes reputation scoring. If "everyone knows" that this skater is great, then as a judge you might feel pressure to give that skater higher marks in an attempt to match the scores that you think the other judges are going to give.

Interesting, but I think investigate these judges alone is not objective enough. What about the rest of judge panel in both free skate and SP program? I think we should do some more research. Because of the anonymous judging, we could never know for sure which judge gives which score.

Anonymous judging has spoiled all the fun of statistics mavens. Back in the day, it was dead easy to test for favoritism and national bias. Just compare the scores that a judge gives to his compatriot with the scores (ordinals) given to that skater by the other judges.

This was also done, in many studies, for all judges from a particular country taken together. Canada always headed the list (greatest degree of national bias). This was because Canadians felt that everyone was ganging up on them to hold Canadian skaters back, so they had to "level the playing field."

Interestingly, Russia was usually at the low end (least amount of national bias). This was because after the break-up of the Soviet Union the judging panel would be sure to have at least two or three judges who were in fact Russians, so no one individual had to be too outrageous.

Sometimes there was a spike for a small federation that had only one skater, and the judge from that country wanted to make sure that he finished 16th instead of last.

The United States was typically in the middle in these studies. The US did, however, get the money bonus. In the 1990s, when Mr. Cinquanta became ISU president, more than half of the entire ISU budget came from American television contracts. It was expected that an American lady would be on the podium in major competitions (the other disciplines, who cares? ) Now, if money is a factor any more, it would be Japan that benefits.

IMHO there was always a gentlemen's agreement that every judge would balance his duty to the National Federation (high scores for homeboys and homegirls) with his duty to the sport (fair judging). If everyone did it to the same extent, and no one went overboard, it would all even out in the end.

 

[capcomeback]

Which of these judges have been at Adelina's other events?

No, but it would be interesting to know. Logically, as Alla seems to have judged all of Adelina's events this season (according to several posters in this thread), you can't necessarily gauge what the other judges might have scored her.

 

[Sam-Skwantch]

Yes. In fact, the average that determines the middle of the corridor includes not only all the nine judges, but also the marks of the referee, who is scoring along with the judges. (If there are fewer judges on the panel, the referee counts twice).

So lets say a judge is deemed to be biased/unfair/and just plain rotten. Would it be possible to replace that judge's score with the referee's or is that something that only applies to a challenge made within a half hour of the event?

The corridor is deliberately set so wide that it is practically impossible ever to be outside it. I think the whole idea of the corridor is actually to spot incompetent judges at the lower levels who might need more training, even though the document speaks about catching national bias as well.

This is as I have suspected all along. The judges all acted within the ISU guidelines likely and there is nothing that can be done except reveal as much truth as possible and hopefully gain reforms. Even little things can lead to bigger things further down the road.

The drawback to this corridor test is that it promotes reputation scoring. If "everyone knows" that this skater is great, then as a judge you might feel pressure to give that skater higher marks in an attempt to match the scores that you think the other judges are going to give.

This is a great point. It does promote preconceived ideals for what scores someone, especially PCS, should receive and kind of forces judges to deliver! If they don't they become outliers and could even face discipline if they aren't in line.

 

[Mathman]

So lets say a judge is deemed to be biased/unfair/and just plain rotten. Would it be possible to replace that judge's score with the referees...?

No. The only thing that happens is that months later, after the season is over, any judge that has acquired three or more anomalies is called on the carpet.

There is, however, a referee's meeting held immediately after each event, where the judges and technical panel sit down with the referee and go over the judging. I have no idea what they talk about -- maybe they just congratulate themselves for a job well done. But the referee can ask a judge or tech caller to explain his or her marks, and presumably this goes into the referee's report for the event. None of this is is made available to the public and I don't know what the ISU does with these referee's reports, if anything.

About the kind of complaint that a federation can file within 30 minutes, I think that can only happen in the case where one of the officials does not follow the rules. For instance, if the referee allows a re-start but the time for a re-start has expired, or something like that. I don't think you can file a protest just for the sake of wuzrobbing.

 

[skatedreamer]

"wuzrobbing" :thumbsup:

Something new for the skating lexicon!

 

[gkelly]

I think the main reason for allowing protests in 30 minutes is to correct any obvious errors on the protocols.

E.g., that wasn't a toe loop, it was a loop (if the TS misspoke and the other tech panel members didn't catch the mistake).

Or that sequence of steps near the beginning of the program wasn't my attempt at a step sequence, it was just transitions -- the full step sequence later in the program is what should have been called.

You're only allowed to protest your own scores/calls, not a rival skater's.

 

[Mathman]

"wuzrobbing" :thumbsup:

Something new for the skating lexicon!

I wish I could take credit for that. I don't know who was the first to coin that verb.

 

[Vanshilar]

You're only allowed to protest your own scores/calls, not a rival skater's.

This brings up an interesting question. A lot has been made of how the Koreans are protesting the composition of the judging panel rather than the scoring directly. But say there was some sort of corruption, i.e. some judge or another giving favorable/unfavorable marks to some skater or another. What is the correct appeals process for this? Is there some avenue to get the scores looked at, to verify whether or not the judges were following the protocols when giving out their scores?

After all, the judging is anonymous already, and the scores are subjective and/or aggregates of different criteria so it's difficult/impossible to go back and check whether or not the scores were given correctly, so if there's not a clear process to double-check if the judging was done correctly, then it seems like the federations set up the system to hide any inconsistencies. As far as I know in response to the ladies' judging the ISU simply released a blanket statement saying all protocols were adhered to, without giving any documentation or explanation.

I'd be curious if the Koreans filed the protest the way they did simply because the system doesn't really allow for directly investigating potential judging corruption.