The abhorrent state of PCS judging

[Mathman]

Plus, tha Alien Bloc might conspire together to win everything against the naive Earthlings who haven't learned yet how to play interplanetary politics.

 

[Khoai]

The goal of the national skating federations is to bring home medals, not ensure that skaters from other countries are judged fairly. It doesn't matter if a judge is new or experienced. No federation is going to promote a judge who consistently under marks his countrymen. The highest score is dropped, so that's really the only practical way to discount the outlier so that the score doesn't impact the result.

The highest and lowest score are dropped so they can ensure the second highest or the second lowest stay. I pretty much agree that almost no federations will send out the judges who are fair. Maybe some of them judge fairly but most don't.

 

[Altie]

perhaps they could ask for a new scoring when the standard deviation between judges is too high, until they converge ?
But that would mean potentially endless deliberations...
A gaussian fitting of the results instead of a simple averaging ? Perhaps not easy to get for the crouds, but probably fairer

 

[Li'Kitsu]

Oh, really? Judges with absolutely no ties to any country?

Where would such judges come from? Mars, Jupiter, Venus, and Saturn?

But first Mars, Jupiter, Venus, and Saturn would have to become ISU members.

And if Mars, Jupiter, Venus, and Saturn become ISU members, then skaters from Mars, Jupiter, Venus, and Saturn would have to be allowed to compete.

But wait ... oh, no ... then there would be the possibility of judges from Mars, Jupiter, Venus, and Saturn being on the judging panels at comps where skaters from Mars, Jupiter, Venus, and Saturn are competing ...

That might surprise you, but there are actually human beings, walking on this very earth, who are not breathing patriotism and sleeping in their national flags at night to keep them warm. I've even seen one once, and it wasn't even in a zoo.

I'm very surprised the idea that people want judges to be more independant from their national federations sparks reactions like that. Even if we accept national bias up until a certain point and try to give the judges the benefit of the doubt as far as possible, and take into account how difficult it would be to get independant judges - I think it's undeniable that the general idea is coming from a place were people want to improve the current state. We might never get 100% independant judges, but I'd think having someone like the head of a national skating federation actively functioning as a judge would raise the eyebrows of most people hearing it (non-skating fans too). We will barely have to go to Mars to find a Canadian for example who isn't the head of the Canadian skating federation.

 

[ice coverage]

... Even if we accept national bias up until a certain point and try to give the judges the benefit of the doubt as far as possible, and take into account how difficult it would be to get independant judges ...

My post was in response to Ares, whose post -- with the word "unaffiliated" -- was absurdly dismissive of the difficulty of attempting to get as-independent-as-possible judges.

Unlike your post, which does acknowledge the difficulty.

I did not say that I am against wishing for judges who are as independent as possible.
I responded to the absurdity of Ares with absurdity.

I would want judges to have knowledge and experience, and they have to start somewhere in acquiring that knowledge and experience. Their home countries are the most natural place for them to get started.

 

[Khoai]

That might surprise you, but there are actually human beings, walking on this very earth, who are not breathing patriotism and sleeping in their national flags at night to keep them warm. I've even seen one once, and it wasn't even in a zoo.

I am pretty much never upset when my National football team lose. So the mob who are crazy about football won't go out and celebrate on the road and let people be at peace.

 

[gkelly]

How would you define "affiliated with the national skating federation"?

Would it differ between large federations, where it might be possible to field internationally qualified judges who have never been directly involved in federation governance, vs. small developing federations where there might be only a handful of families involved in the sport at all?

 

[mrrice]

How would you define "affiliated with the national skating federation"?

Would it differ between large federations, where it might be possible to field internationally qualified judges who have never been directly involved in federation governance, vs. small developing federations where there might be only a handful of families involved in the sport at all?

This is an excellent question!! To say there is a bias in who gets hired for National, or in skating, Grand Prix, National, and World Events, would be an understatement. I'm not sure about skating but, I remember when I first moved from Northridge to Clovis. My name virtually disappeared from the judging "Pool" I got zero assignments my first year. When I asked the head of the judges association for a reason, I was told that they assumed I was leaving their association for the WBA. It is 213 miles from Northridge to Clovis and I was prepared to drive to down to LA to judge. This became a Mini Scandal after I complained and now Judges are allowed to drive up to 300 miles to an event.

Here's where the problem arrives. Money......The SCSBOA pays their judges quite a bit more because they are required to read and understand music. Their days are often quite long because in most cases, there is a Parade, A field Show and an Orchestra Concert. I believe I was paid $500 for the Parade and Field Show and the Music Judges are paid an additional $500 just for the Orchestra concert. So, they made $1000 in a single day.

During the spring, everything changes. When the dancers separate from the band, the judges are hired by the UDA ( The ESPN People) and they come from all over the US. Those assignments are coveted and most of us would do them for free. Now, like Skating Judges, you are not paid for Nationals. They pay for your travel, your hotel, and they give you a per diem for food and ground transportation. This is supposed to prevent a biased or rigged panel. However, it didn't always work and like sports, they were always "Gifts" given to Judges by certain teams. I remember having Brandy and Coffee at the home of the coach from Long Beach State. When I got to Nationals, there was a knock on the door and boom. Flowers, a Card, Starbucks Coffee Beans and a grinder, and a bottle of Brandy. I was so shocked and at first it's flattering that they remembered what I liked. Then, I was like, I hope they're not trying to sway my opinion of their dancers.

Here's Long Beach State from 1992. They were appearing in the Championship for the first time and they ended up winning. I have never received such dirty looks in all my life.

https://www.youtube.com/watch?v=lvi0cAgFu38

 

[andromache]

During the spring, everything changes. When the dancers separate from the band, the judges are hired by the UDA ( The ESPN People) and they come from all over the US. Those assignments are coveted and most of us would do them for free. Now, like Skating Judges, you are not paid for Nationals. They pay for your travel, your hotel, and they give you a per diem for food and ground transportation. This is supposed to prevent a biased or rigged panel. However, it didn't always work and like sports, they were always "Gifts" given to Judges by certain teams. I remember having Brandy and Coffee at the home of the coach from Long Beach State. When I got to Nationals, there was a knock on the door and boom. Flowers, a Card, Starbucks Coffee Beans and a grinder, and a bottle of Brandy. I was so shocked and at first it's flattering that they remembered what I liked. Then, I was like, I hope they're not trying to sway my opinion of their dancers.

I should become a judge of something .

 

[Mathman]

perhaps they could ask for a new scoring when the standard deviation between judges is too high, until they converge ?
But that would mean potentially endless deliberations...

A gaussian fitting of the results instead of a simple averaging ? Perhaps not easy to get for the crouds, but probably fairer

This sort of thing has actually been discussed quite a bit. There are a couple of problems. First, there is no guarantee the the underlying population of scores (this would be the hypothetical scores given by all potential qualified judges) is in fact Gaussian or even symmetric. Not to mention the problem that if they wanted to retain the trimming process of throwing out the highest and lowest, then all parametric methods start to fail in significant ways.

As for using the standard deviation to identify outliers, the current system sort of does that, only they base the calculation on a variation of the range of values rather than the standard deviation. Any way you do it, you cannot insist that the judges (or the ISU) go back and score the competition over again because a mark is out of line with the other judges. (In fact, I believe that there is an official OIC rule about that, for all Olympic sports.)

A little more figure-skaty and less mathy, to me it is not distressing to see a wide range of marks. Different people see things differently. Sometimes we see something like a range of -2 to +2 for a GOE mark, for instance. This might be because a UR or edge, even if uncalled by the technical panel, might be so questionable that some judges though it was OK and others said no. All of this should come out in the wash, averaged over nine judges.

To tell the truth, I think that the rule about throwing out the highest and lowest was not so much to guard against cheating and national bias, but rather to protect against an inadvertent keystroke or touchscreen error. For instance, once in a while we see a judge enter 0.75 when he meant to say 8.75. We would not want such a datum to be included in any averaging process or in any modeling by a Gaussian distribution.

 

[Altie]

This sort of thing has actually been discussed quite a bit. There are a couple of problems. First, there is no guarantee the the underlying population of scores (this would be the hypothetical scores given by all potential qualified judges) is in fact Gaussian or even symmetric. Not to mention the problem that if they wanted to retain the trimming process of throwing out the highest and lowest, then all parametric methods start to fail in significant ways.

As for using the standard deviation to identify outliers, the current system sort of does that, only they base the calculation on a variation of the range of values rather than the standard deviation. Any way you do it, you cannot insist that the judges (or the ISU) go back and score the competition over again because a mark is out of line with the other judges. (In fact, I believe that there is an official OIC rule about that, for all Olympic sports.)

A little more figure-skaty and less mathy, to me it is not distressing to see a wide range of marks. Different people see things differently. Sometimes we see something like a range of -2 to +2 for a GOE mark, for instance. This might be because a UR or edge, even if uncalled by the technical panel, might be so questionable that some judges though it was OK and others said no. All of this should come out in the wash, averaged over nine judges.

To tell the truth, I think that the rule about throwing out the highest and lowest was not so much to guard against cheating and national bias, but rather to protect against an inadvertent keystroke or touchscreen error. For instance, once in a while we see a judge enter 0.75 when he meant to say 8.75. We would not want such a datum to be included in any averaging process or in any modeling by a Gaussian distribution.

Thanks a lot for your very, very insightful answer !
It is very interesting.
And yes I thought after having posting about the case of a judge who gives high (or low) scores to everyone - as it happens often - it would add bias to any parametric model. Sadly (or thankfully ?), FS scores are not independant random variables

 

[gladiolusc]

That might surprise you, but there are actually human beings, walking on this very earth, who are not breathing patriotism and sleeping in their national flags at night to keep them warm. I've even seen one once, and it wasn't even in a zoo.

I'm very surprised the idea that people want judges to be more independant from their national federations sparks reactions like that. Even if we accept national bias up until a certain point and try to give the judges the benefit of the doubt as far as possible, and take into account how difficult it would be to get independant judges - I think it's undeniable that the general idea is coming from a place were people want to improve the current state. We might never get 100% independant judges, but I'd think having someone like the head of a national skating federation actively functioning as a judge would raise the eyebrows of most people hearing it (non-skating fans too). We will barely have to go to Mars to find a Canadian for example who isn't the head of the Canadian skating federation.

^ that's more what I mean when I say "affiliated with national federations". Just like having a former coach of the skater come in and judge that same skater would come off as inappropriate and something ideally avoided, people who are current heads of national skating federations or were team leaders for the national Olympic figure skating team don't strike me as ideal choices as judges. In most fields that I'm aware of, you go out of your way to avoid situations of potential biases (to varying degrees of success). The concept is about as weird to me as... say if a personal friend becomes part of my interview panel. Whether or not she's actually biased towards me is beside the point. She just wouldn't have been on my interview panel at all.

 

[mrrice]

^ that's more what I mean when I say "affiliated with national federations". Just like having a former coach of the skater come in and judge that same skater would come off as inappropriate and something ideally avoided, people who are current heads of national skating federations or were team leaders for the national Olympic figure skating team don't strike me as ideal choices as judges

. In most fields that I'm aware of, you go out of your way to avoid situations of potential biases (to varying degrees of success). The concept is about as weird to me as... say if a personal friend becomes part of my interview panel. Whether or not she's actually biased towards me is beside the point. She just wouldn't have been on my interview panel at all.

True.....In every shape and form and it happens in every judged artistic event. Look at the tech callers, aren't they all former skaters? It's unavoidable to have former skaters on the panel and in my case, I would prefer it. There's nothing like a former skater to understand the intricacies, difficulties, and the amount of training it takes to make these things looks easy. I would love to see Patrick become a tech caller. I know I talk about him a lot but, IMO, he is the textbook addition of the complete skater.

 

[Mathman]

...Just like having a former coach of the skater come in and judge that same skater would come off as inappropriate and something ideally avoided, people who are current heads of national skating federations or were team leaders for the national Olympic figure skating team don't strike me as ideal choices as judges....

There are some ISU rules against such "conflicts of interest." Once in a while the ISU will issue disciplinary actions against a judge who was the sister of the coach of one of the contestants, or something like that. This usually occurs at lower-level competitions, and often it is inadvertent. That is, the sister happens to be available and the event organizer says, "hey, can I get you to sit in as a judge to help us out?"

The test case is whether it is kosher to allow Anna Shekhovtsova, an immensely influential long-time judge who usually specializes in ice dance. She is the wife of Valentin Piseev, the head of the the Russian Federation. This does not stop her from being assigned to judge big-time events, including the 2002, 2006, 2010, and 2014 Olympics, where Russian skaters compete.

 

[karne]

The test case is whether it is kosher to allow Anna Shekhovtsova, an immensely influential long-time judge who usually specializes in ice dance. She is the wife of Valentin Piseev, the head of the the Russian Federation. This does not stop her from being assigned to judge big-time events, including the 2002, 2006, 2010, and 2014 Olympics, where Russian skaters compete.

Actually, I find it much more pertinent to consider the actual Vice-President of the Korean Skating Union who sat as a judge on the panel at Vancouver (but, mysteriously, all those Yunafans shrieking about biases and conflict of interest go very silent about that...)

Alla is NOT the "head of the RuFed" herself, and proclaiming that she would just obediently do the bidding of her husband in such matters is straying into dangerously sexist territory, IMO.

 

[drivingmissdaisy]

Alla is NOT the "head of the RuFed" herself, and proclaiming that she would just obediently do the bidding of her husband in such matters is straying into dangerously sexist territory, IMO.

How many likes can I give this post? I can assure you that when a man is accomplished on his own merits, no one ever cares about who his wife is or what she does. I hate saying that, but it's true.

 

[Mathman]

Alla is NOT the "head of the RuFed" herself, and proclaiming that she would just obediently do the bidding of her husband in such matters is straying into dangerously sexist territory, IMO.

I make no proclamations, sexist or otherwise. I think that it is a conflict of interest just the same.

 

[drivingmissdaisy]

I make no proclamations, sexist or otherwise. I think that it is a conflict of interest just the same.

http://www.usfsa.org/content/Nominating Committee 2014-2015 Bios and SOP.pdf

How about Sam Auxier judging the men's events in Sochi while he was first VP of USFSA, and now president of the organization? Where is the outrage? This is why your post comes off as sexist.

You cannot be a world or Olympic judge without being very well connected within your federation.

 

[magicalwords]

We need Shin Amano and Shin Amano clones to be the tech caller at every event, and somebody who possesses Shin Amano-like qualities to judge at every competiton. /s
But really, this guy's name is the first name that came to my mind when I thought of unbiased officials - he really gives no easy time to Japanese skaters.

Also, How is the above discussion a sexist discussion? People would be making the same argument had the judge been Anton S., the husband of Valentina P. who is the head of RUS Skating Fed. I see it more as nationalism/anti-Russian sentiments than sexism.

 

[Mathman]

How about Sam Auxier judging the men's events in Sochi while he was first VP of USFSA, and now president of the organization? Where is the outrage?

I am not outraged. I said that the situation of Shekhovtsova was the "test case" to determine what the ISU is referring to when they speak of "conflicts of interest." Evidently this situation is OK with the ISU (it "passes the test"), as are the situations mentioned by Karne and by you.

In general, it takes a lot for me to become outraged over figure skating.