ISU votes to abolish anonymous judging

[drivingmissdaisy]

But she had sooo many judges with Russian names on her side (one of them was even from Finland ) to ignore two-footed or underrotated jumps:

The technical controller: Lakernik
Assistant Technical Specialist: Baranova
Judge No 2 Balkov
Judge No 4 Kulik
Judge No 6 Shekhovtseva
(Judge No 7 Cucuphat ?)
Judge No 9 Domanska

http://www.isuresults.com/results/owg2014/SEG004OF.HTM

And there were a lot of outraged fans with Korean names. That doesn't mean they are inherently biased. Aditionally, we don't know whose side anyone was on although, presumably, the RUS judge gave her strong marks.

 

[MaxSwagg]

Fun for sure. But judges do not have to answer to coaches or skaters. They just have to judge. :yes:

By the way, I don't think that the results from Sochi provide a very good example of why anonymous judging is good or bad. Sotnikova won by 5 tech points on elements in the long program. Otherwise everything was pretty even with nothing much to remark on in terms of the scores of individual judges.

It would be fun to know whether Ukraine (Balkov), Russia (Shokhovtsova), Slovakia and Estonia ganged up on Germany, Italy, Canada and Japan, and how the French judge voted. Or whether any judge low-balled one skater and pumped up another or whether all the judges just went along with the flow….

Actually, that would be fun.

The components were completely out of line, in my opinion...

 

[jenaj]

But she had sooo many judges with Russian names on her side (one of them was even from Finland ) to ignore two-footed or underrotated jumps:

The technical controller: Lakernik
Assistant Technical Specialist: Baranova
Judge No 2 Balkov
Judge No 4 Kulik
Judge No 6 Shekhovtseva
(Judge No 7 Cucuphat ?)
Judge No 9 Domanska

http://www.isuresults.com/results/owg2014/SEG004OF.HTM

All of the top three skaters got some gifts.

 

[Mathman]

(Judge No 7 Cucuphat ?) [/url]

I think this is a fairy common French name, unrelated to Russian. Helene Cucuphat. (of course you never know who is aligned with whom.)

 

[Mathman]

The components were completely out of line, in my opinion...

But I don't think that this is related to anonymous judging. Out of line or not, the judges probably would have given the same component scores whether anonymously or publicly.

 

[MaxSwagg]

But I don't think that this is related to anonymous judging. Out of line or not, the judges probably would have given the same component scores whether anonymously or publicly.

Oh, I misread. Yes. I'm not one for conspiracy theories. The scores of the label were high across the board so one can't just point te finger at one judge.

 

[hugo]

But she had sooo many judges with Russian names on her side (one of them was even from Finland ) to ignore two-footed or underrotated jumps:

The technical controller: Lakernik
Assistant Technical Specialist: Baranova
Judge No 2 Balkov
Judge No 4 Kulik
Judge No 6 Shekhovtseva
(Judge No 7 Cucuphat ?)
Judge No 9 Domanska

http://www.isuresults.com/results/owg2014/SEG004OF.HTM

Domanska is typical polish name...
I recently read in a polish FS forum Sochi thread and surprisingly (because Polish are very far from being pro-russian...) the majority agreed that Sotnikova should win, not with such big advantage but though...
As about election of Lakernik. I wonder who elected this controversial (and maybe even corrupted :shocked man ? Russian have such power in ISU ?? He got 81 votes vs. 34 of Mari Lundmark (Finland)

 

[MiRé]

Technical panels are transparent yet they fail to make correct judgments so I wonder if being anonymous or not really matters. It's a good move from ISU nonetheless....

 

[chuckm]

The Olympic tech panel:
Referee Diana BARBACCI LEVY SUI
Tech Controller Alexander LAKERNIK RUS
Tech Spec Vanessa GUSMEROLI FRA
Asst Tech Spec Olga BARANOVA FIN (born in Russia)

Free Skate Judges:
Birgit FOELL GER
Yuri BALKOV UKR
Franco BENINI ITA
Zanna KULIK EST
Nobuhiko YOSHIOKA JPN
Alla SHEKHOVTSEVA RUS
Helene CUCUPHAT FRA
Karen HOWARD CAN
Adriana DOMANSKA SVK

The US, GBR, SWE and KOR judges on the SP panel were replaced by UKR RUS EST and FRA on the FS panel.

 

[Mathman]

The US, GBR, SWE and KOR judges on the SP panel were replaced by UKR RUS EST and FRA on the FS panel.

That sounds like it's curtains for Yuna Kim.

And yet … even with US, GBR, SWE and KOR on the short program panel, still they gave Sotnikova equal PCSs to Kim.

So it's hard to draw any conclusions -- 0r at least, it's hard to find any data to offer as evidence to support one's suspicions. Knowing the marks of each judge would be a good thing after all. That way we could be mad at somebody instead of just being mad. :yes:

 

[drivingmissdaisy]

And yet … even with US, GBR, SWE and KOR on the short program panel, still they gave Sotnikova equal PCSs to Kim.

And +GOE that more than made up for the 1 point BV difference.

 

[Blades of Passion]

And a Level 4 call for a Level 3 footwork sequence.

 

[drivingmissdaisy]

And a Level 4 call for a Level 3 footwork sequence.

It's debatable. I can't tell the difference but IIRC gmyers (or gkelly?) said it was L4.

 

[moriel]

Domanska is typical polish name...
I recently read in a polish FS forum Sochi thread and surprisingly (because Polish are very far from being pro-russian...) the majority agreed that Sotnikova should win, not with such big advantage but though...
As about election of Lakernik. I wonder who elected this controversial (and maybe even corrupted :shocked man ? Russian have such power in ISU ?? He got 81 votes vs. 34 of Mari Lundmark (Finland)

As for Lakernik, same question goes for Didier Gailhaguet. He was involved in 2002 skandal and even suspended for 3 years, and well, still made to be french fed president.
ISU things.

 

[moriel]

That sounds like it's curtains for Yuna Kim.

And yet … even with US, GBR, SWE and KOR on the short program panel, still they gave Sotnikova equal PCSs to Kim.

So it's hard to draw any conclusions -- 0r at least, it's hard to find any data to offer as evidence to support one's suspicions. Knowing the marks of each judge would be a good thing after all. That way we could be mad at somebody instead of just being mad. :yes:

+1
The mark between judges were actually reasoanbly consistent, so not sure if actually showing who gave which marks would help much.

 

[Babbette1]

But the East German judges *are* crooks and frauds because they never score Toller above 5.6, even in artistic impression, and instead gift that snooze fest Jan Hoffmann with such high marks...

Oops, wrong decade. But I have a lot to make up for

Cough, sorry, wrong *century*.

 

[eppen]

Out of curiosity started looking for research literature of subjectively judged sports - and there seems to be quite a lot of it, partly fairly recent, partly older. I'm not heavily into statistical methods, so mostly skipped the methodological parts and tried to make sense of the discussions and conclusions (this is somewhat problematic because evaluating methods would also be v. important...). Did not look up the final versions if the text was made available otherwise (paywalls are high and wide in scientific journals). (Some of these have probably been posted elsewhere in the forums, so am very sorry if I'm just repeating what is already commonly known!)

On Cold War figure skating judging: http://users.polisci.wisc.edu/schatzberg/ps616/sala2007.pdf (they found that both Cold War factions were quite active in implementing not-so-unbiased judging )

Nationalistic bias (based on the early date, mostly on 6.0 system): http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.954&rep=rep1&type=pdf (compared ski jumping and figure skating, and found that there is more bias in figure skating particularly in important championships partly because the federations choose the judges unlike in ski jumping where they are selected centrally)

On ISU's attempts to incerase transparency and that they probably have failed: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.668.4950&rep=rep1&type=pdf (written in 2010 and published in 2014; if the skater had a compatriot in the judging panel, his/her/their scores are likely to be higher, ie national bias and vote trading has remained and possibly even increased under the new system - but the last point was not so certain)

On the panels and the randomization of scores: http://kevinfraker.com/school/stats/articles/ch9/skating-yale.pdf (the larger panel of 12 and some kind of randomization of subpanel for each skater is somewhat problematic; a bit difficult to understand what the final outcome was here)

Another one on the panels/subpanels problem in 2006 Worlds pair skating: http://www.stat.yale.edu/~jay/Chance/ChanceOnOffIce.pdf (tight result and different panels or taking all the scores into consideration would have divided the medals differently)

On reputation bias: https://www.researchgate.net/profil...ng_Judging/links/00b7d52956edb9f498000000.pdf (6.0 system, but maybe could also apply these days?)

On serial position bias: http://cbdr.cmu.edu/papers/pdfs/cdr_065.pdf (later get scored better, but again 6.0 system where skaters were compared with each other. The current system evaluates each performance individually, so maybe not so relevant any more? There was a later article also, but not available to read.)

On difficulty bias: http://pirate.shu.edu/~rotthoku/papers/Order Matters.pdf (on gymnastics mainly, but could be applied to skating also? Trying more difficult stuff is valued higher even when the attempt failed.)

On 2010 men's OG judging (abstract only): http://www.tandfonline.com/doi/abs/10.1080/1091367X.2012.639602 (Anomalies in judging transitions for Plushenko and Lysacheck - which could maybe have been repeated in 2014 for Plushenko as he did receive a 4,25 for his transtitions. I talked with a figure skating judge recently who said it was a fair score for what he did... )

So, based on this eclectic selection of analyses, apparently no easy solutions? The nationalistic bias was quite apparent in the 6.0 system and has not really been reduced in the new one. Then there is a host of other biases that influence the outcome... Human nature is hard to erase from the process even with mathematical and statistical methods. There will always be those who are more generous with their marks and those who are harder to please (we probably all have experience from teachers?). Organisational decisions can be made to better the situation like the current decision, but also maybe selecting judges centrally instead of by federations.

I sat behind the technical panel close enough to see the screens during the Europeans this year and it was quite interesting. Shin Amano was the controller for the ladies and almost every skater got red review flags, Baranova for men was more lax. Although this could also be the result of men doing less underrotaed jumps etc? (and they skated mostly really, really badly, so the mistakes were quite clear...) Sometimes the reasons for review were obvious for the viewer, ie an obvious mistake in takeoff-landing etc, but also elements executed at the other end of the rink got sometimes reviewed - maybe because they could not see properly. Sometimes it was not so obvious, eg almost EVERY element Anastasia Galustyan did was reviewed although they seemed fairly ok to me (and she was scored much lower than was expected by the audience, the judges got booed a lot when they exited). They worked very fast - usually the reviews were done before the skater was back at K&C - and yet managed to discuss the more difficult cases.

Maybe because I am working in the academic world and appreciate expertize acquired by hard work and dedication to whatever one is researching, I would like to give the benefit of the doubt to the individual judges - after all, they have been trained to do what they do and many have years of experience in judging. On the other hand, I also acknowledge that there is a lot of bad research and people who are amazingly incompetent in their chosen professions (and yet do well because of a number of reasons). But that is also the human nature and something that is kind of hard to erase completely...

e

 

[Mathman]

Cough, sorry, wrong *century*.

Wrong millennium?

 

[Mathman]

On the panels and the randomization of scores: http://kevinfraker.com/school/stats/articles/ch9/skating-yale.pdf (the larger panel of 12 and some kind of randomization of subpanel for each skater is somewhat problematic; a bit difficult to understand what the final outcome was here)

The conclusion was that the ISU did not chose a different subpanel for each skater, but they did randomize (permute) the order in which the judges scores were listed for each skater.

They could have asked Mr. Cinquanta. The procedure was well-publicized in ISU communications and did not require any statistical analysis. (Unless you thought that the ISU was not following its own rules. )

 

[Mathman]

Nationalistic bias (based on the early date, mostly on 6.0 system): http://citeseerx.ist.psu.edu/viewdoc...=rep1&type=pdf (compared ski jumping and figure skating, and found that there is more bias in figure skating particularly in important championships partly because the federations choose the judges unlike in ski jumping where they are selected centrally)

Thanks for digging up all these studies. This is exactly the kind of thing that is hidden by anonymous judging.

This study form 2002 concludes (I am shocked -- shocked) that USA. Canada, Germany and Italy formed a consistent voting bloc up against France, Russia,Poland and Ukraine resulting in an extra +.17 points (I assume these are 6.0 points, or possibly ordinals?) for one's own skaters and a deficit of -.05 for the other guys'.

Also … the judges that have in the past proven to be the most nationalistically biased are the most likely to be chosen by their federations for high-stakes contests like the Olympics. (Again, duh!)

With the abolition of anonymous judging we will be able, once again, to enjoy reading studies like these. (I doubt, though, that the judging will be any better or worse.)