Is the standard of Judging deteriorating?

[Mrs. P]

All feelings! I truly believe that the judges are much more objective than us fans. Let's give Akiko a big hug! That's all we could and should do.

By the way, I think that to compare numbers from two different competitions often lead to wrong conclusions.

I do not have skin in the game whether Akiko or Mao wins. I have made a point of saying that I like Mao, her skating and respect her hard work and determination to improve as an athlete. This is not a case of a me slamming one skater to raise the other one up. And heck, I made a point of saying that Akiko shouldn't make so mistakes in the short program, and getting comments from people that I was selling Akiko short (pun not intended) on how good her free skates are! So I don't really have any emotions or feelings to act on. In the end, it's just a competition, life goes on, and there will be other competitions.

However that doesn't mean questioning the judging =/= a lack of willingness to understand the result. I have done my best effort -- as an hobby student of the the sport -- to understand the current criteria of the system. I am the first person to spend hours calculating scores, analyzing them and reading the efforts by other posters (like Doris) to understand parts of the system I don't know about (i.e. Ice Dance levels). And there have been times where a result didn't make sense to me initially, but I went back, looked it over and said, "OK, I get it."

That was not the case here. For this particular competition, I have looked at the protocols several times, watched the YouTube videos several times with the bullet points of each of the program components in mind. I've already established that Mao is superior to Akiko in SS and TR and that my issue is IN/PE/CH. And as for your final point (re comparing competitions), the ISU compares competitions to determine personal and season's bests, so I'm not sure why there is this self-imposed ban to do some comparison between different competitions. That said, I do acknowledge it's not a perfect system because you're dealing with different judging panels. That is why also compared Mao's PCS at this competition between the two segments (using the factoring system). Again I stand firm that it doesn't make sense for Mao to receive nearly same PCS (in fact it was a touch higher) for this FS as her terrific SP.

If I come off frustrated, it's because I don't like being painted as someone that is just acting on emotion/blind fandom when I have made a concerted effort on this board to help myself and others to look deeper at the scores, even the ones I don't agree with.

I think one of the most important simple changes that should be made (under any judging system that combines the results from two or more competition phases, by any mathematical method) is this:

During the final phase of the competition, after each skater skates, before announcing that skater's overall placement, always announce the placement in that phase of the competition only.

Audiences won't always have seen the short program/short dance and those who have seen it might not remember the standings or might not be familiar with the rules for combining results from the separate phases.

What they do know is what they just saw and who they thought was better today. So make sure to let them know what the standings were today.

E.g., instead of "for a total of xxx.xx points for the freeskating. That gives her an overall total of zzz.zz, which puts her currently in first place!"

take a few extra seconds to say "...for a total of xxx.xx points for the freeskating. That puts her her currently second in the freeskate. Combined with her short program score of yy.yy, that gives her an overall total of zzz.zz, which puts her currently in first place!"

That will cut down on a lot of the outrage when it was clear that this skater has already been outskated today, and the judging panel agrees.

There will still be times when the majority of the audience disagree with the majority of the judging panel about today's results specifically, but hey, that's life. Some of the judges may disagree too. It would be interesting to calculate ordinals to find out how many judges (and which ones) gave the highest scores to which skaters, but I don't see that happening as long as the ISU believes that anonymous judging is less subject to pressure from federations.

That is a good idea. I agree it would have reduced some of the outrage at this competition, but it's not like the audience didn't know Mao was first by a margin, they probably had it written down in their programs and the score is flashed on the screen during their skate. I still think many in the audience (and the commentators) believed Akiko did enough to make up the deficit.

[Mathman]

During the final phase of the competition, after each skater skates, before announcing that skater's overall placement, always announce the placement in that phase of the competition only.

...as long as the ISU believes that anonymous judging is less subject to pressure from federations.

I do not believe that the ISU believes that.

[Bluebonnet]

If I come off frustrated, because I don't like being painted as some blind fan that is just acting on emotion/fandom when I have made a concerted effort on this board to help myself and others to look deeper at the scores, even the ones I don't agree with.

No, you are very knowledgable! Better than me who didn't even bother to go through the detailed bullet points in PCS on this particular competition. All I've said was based on my feeling on these two performances. I think you could be sure that I have no particular bias on either of them. I don't care one way or the other. If Mao were to be second on the podium, I wouldn't mind it at all. So let's say that your feelings are different from the judges' feelings this time, but my feelings are close to the judges' feelings at the same time. As equal skating fans, who's feeling should the judges go with? My answer is no one. The results go with the judges' feelings. That's the way it works. Unless we could find any wrong doing, the subjective marks on P/E, CH, and IN are subjective. The results took the judges' subjective views instead of yours and mine. You can't say the judges were wrong based on your subjective views. Because if the judges took your subjective views, it would be against my and many other fans' subjective views. There is actually no clear cut on right or wrong in subjective views.

[Mrs. P]

No, you are very knowledgable! Better than me who didn't even bother to go through the detailed bullet points in PCS on this particular competition. All I've said was based on my feeling on these two performances. I think you could be sure that I have no particular bias on either of them. I don't care one way or the other. If Mao were to be second on the podium, I don't mind it at all. So let's say that your feelings are different from the judges' feelings this time, but my feelings are close to the judges' feelings at the same time. As equal skating fans, who's feeling should the judges go with? My answer is no one. The results go with the judges' feelings. That's the way it works. Unless we could find any wrong doing, the subjective marks on P/E, CH, and IN are subjective. The results took the judges' subjective views instead of yours and mine. You can't say the judges' were wrong based on your subjective views. Because if the judges' took your views, I and many other fans would feel they were wrong.

Well I should clarify and say I'm not accusing the judging of any wrongdoing either, i.e. corrupt judging. When I say I feel the judges are wrong, I mean more that I do not agree with their assessment of the programs based on the current criteria given by the ISU. And I'm just expressing it here. I'm not asking the judges to change the results based on my feeling, but basically saying that this is why I wish they would have scored some things differently. Is that not a reasonable thing to say? Just because I don't agree with the result or the way things were done doesn't mean that 1.) That the people who accept/agree with this win (i.e. judges + fans) didn't make the same good faith effort to come to that conclusion or 2.) Again, that I feel the system is crap and needs to be thrown away.

That's why I take some issue with this subjective views thing. Because if you're acknowledging that the judges reached the result by their own subjective views, I don't think it's out of line for people to say "I don't agree with how this conclusion was reached and here's why."

I've covered many legal hearings and such at my job where each side made valid views based on a good faith effort under various policies, guidelines and laws. I believe that is being done here, for the most part, as well.

[Bluebonnet]

Well I should clarify and say I'm not accusing the judging of any wrongdoing either, i.e. corrupt judging. When I say I feel the judges are wrong, I mean more that I do not agree with their assessment of the programs based on the current criteria given by the ISU. And I'm just expressing it here. I'm not asking the judges to change the results based on my feeling, but basically saying that this is why I wish they would have scored some things differently. Is that not a reasonable thing to say? Just because I don't agree with the result or the way things were done doesn't mean that 1.) That the people who accept/agree with this win (i.e. judges + fans) didn't make the same good faith effort to come to that conclusion or 2.) Again, that I feel the system is crap and needs to be thrown away.

That's why I take some issue with this subjective views thing. Because if you're acknowledging that the judges reached the result by their own subjective views, I don't think it's out of line for people to say "I don't agree with how this conclusion was reached and here's why."

I've covered many legal hearings and such at my job where each side made valid views based on a good faith effort under various policies, guidelines and laws. I believe that is being done here, for the most part, as well.

Well, let's say that I'm just lucky. The more I study on the details of the protocols, the more I'd accept the judges' verdicts, the more I'd feel less need to be outrageous on the results, even to my own favorites which are several, not just one.

No, it's perfectly OK to state your own subjective views anytime. I did not force you or anyone to accept my subjective views. I did not say that my views are the only one which are correct. You could accept them or you could totally against them. So let's shake hands and make peace.

What I don't like so much is constantly being told, if you don't like the results, read more protocols.

No thank you. I'll just continue not to like the results. Like fans of every sport.

Yeah, Math! You've been playing "the devil's advocate" for so long.

[Mrs. P]

Well, let's say that I'm just lucky. The more I study on the details of the protocols, the more I'd accept the judges' verdict, the more I'd feel less need to be outrageous on the results, even to my own favorites which are several, not just one.

Well let's be clear, again, that 95 percent of the time time I do accept the result. I didn't object to Patrick's win in Russia, Mao's win in China or Kiira's win in Russia . I try not to be outraged or the sake of it but only after some consideration and thought. I do not like playing armchair critic, so I try to reserve my criticism to a minimum and only when something sticks out to me.

I do not see the occasional criticism as all that bad or harmful.

[Bluebonnet]

Well let's be clear, again, that 95 percent of the time time I do accept the result. I didn't object to Patrick's win in Russia, Mao's win in China or Kiira's win in Russia . I try not to be outraged or the sake of it but only after some consideration and thought. I do not like playing armchair critic, so I try to reserve my criticism to a minimum and only when something sticks out to me.

I do not see the occasional criticism as all that bad or harmful.

I was outraged on the results several times before. The biggest one was on 2010 Olympics men's result which truly took a few pounds off my weight and I was slim in the first place. I thought I'd quit watching figure skating all together. Damn, Plushenko! I've invested emotion on his win totally and unreservedly. But I didn't understand the detailed scoring as much as I do now. The more I understand the scoring system, the less I feel that I own it logically on that particular argument. Even though I still completely hate the result to reward a quadless champion, I've found now that I was actually more emotional than logical on that one. Sorry to be OT.

[mskater93]

I do not see the occasional criticism as all that bad or harmful.

I see occasional criticism as a means to improve how things are done. ISU should look at fan reactions to controversial results (which is about 5% of the time) and see how the rules and system can be tweaked to improve the results to be more in line with the expectations of more 'refined' fans so people don't need to continually "scream" go look at the protocols....

[Mrs. P]

I was outraged on the results several times before. The biggest one was on 2010 Olympics men's result which truly took a few pounds off my weight and I was slim in the first place. I thought I'd quit watching figure skating all together. Damn, Plushenko! I've invested emotion on his win totally and unreservedly. But I didn't understand the detailed scoring as much as I do now. The more I understand the scoring system, the less I feel that I own it logically on that particular argument. Even though I still completely hate the result to reward a quadless champion, I've found now that I was actually more emotional than logical on that one. Sorry to be OT.

Hm, maybe I need to redefine my definition of outrage. You see my definition of outrage is more of the "What up with that business?" variety. I've only stayed up at night because I'm addicting to watching Korean drama on my iPad.

I see occasional criticism as a means to improve how things are done. ISU should look at fan reactions to controversial results (which is about 5% of the time) and see how the rules and system can be tweaked to improve the results to be more in line with the expectations of more 'refined' fans so people don't need to continually "scream" go look at the protocols....

That would be nice; doubt that happens.

[Mathman]

There will still be times when the majority of the audience disagree with the majority of the judging panel about today's results specifically, but hey, that's life. Some of the judges may disagree too. It would be interesting to calculate ordinals to find out how many judges (and which ones) gave the highest scores to which skaters, but I don't see that happening as long as the ISU believes that anonymous judging is less subject to pressure from federations.

A little bit off topic, but here is a study from 2004 on the subject of "outliers," comparing the 6.0 system with the interim system which featured 14 sitting judges with 7 scores chosen at random to count. (Ah, the good old ISU never let's us down!)

This researcher's conclusion (he is an economist) is that anonymity did succeed in making judges less fearful of giving marks that were different from what that judge thought the rest of the panel might give. (This was before the "corridor.")

http://ftp.iza.org/dp1257.pdf

[Rachmaninoff]

I don't think it's deteriorating; there's always been controversial and downright suspect results. I do fear the scoring system may be going farther away from rewarding the types of programs that wow or mesmerize an audience.

[sarakimm]

Get rid of Anonymous Judging. Now. We need transparency and accountability. We need to see who's giving out what marks. The current system allows judges to hide with pretty much impunity, outside of a narrow group of ISU officials and/or colleagues. No judge who is confident in their training and marking ability, should be afraid to judge in the light rather than the dark, and to be able to justify their scoring if necessary.

ITA!! I am curious: what other Olympic sports have anonymous judging? Anything done in secret is immediately suspect. At least with 6.0 the judges had to hold up their scores and stand behind them. Anonymity needs to go!

[FSGMT]

ITA!! I am curious: what other Olympic sports have anonymous judging? Anything done in secret is immediately suspect. At least with 6.0 the judges had to hold up their scores and stand behind them. Anonymity needs to go!

first of all: welcome! And I agree 100%, in my opinion there should be a separate "Press conference" where the skaters/coaches/fans/journalists should be allowed to ask (politely) the judges why they gave a specific mark or GOE, and they should answer and explain (politely) their opinions about the programs, so the skaters can improve and understand what they need to change! And the biased judges will be immediatly recognized.

[NorthernDancers]

This is great thread! To answer the question.....I'm not sure the judging has deteriorated nearly as much as judges have still not embraced PCS in COP. To me, there seems to be plenty of documentation and guidance about what to look for, but it's not really followed. It's like PCS is really just a new kind of ranking like the old system, where too often reputation, previous rank, personal favourites, etc rule the day. We should be seeing more differences in the numbers across the PCS elements. It's possible a dance team can have great skating skills, but really poor choreography and transitions. That should be reflected in varied scores for these PCS elements. It's possible to have a really crumby program, but still interpret it well. Conversely, it is possible to have great choreography and difficult transitions, but really poor skating skills (ie. line, edge, unison...) The numbers should be more varied, unless we are talking about V/M and D/W where they will be scoring close to the top of the scale due to their uniqueness (and even there, I do think that V/M should edge D/W generally in PCS, and certainly this year, based on some of the criteria related to interpretation, unison, line, edge, musicality, etc.) There are lots of example of this in Canada..... The over scoring of G/P last year at Canadians is probably the most egregious, but there were others within that competition as well. Minto was a pretty clean competition in terms of judging what was there. But not all competitions are the same.

So I think the judges need to all undergo mandatory additional training on PCS and GOE for elements. Further, making judges public will ensure those not following guidelines can be further trained as needed. And if this is deliberate, they can warned and eventually not allowed to judge at certain levels. And if random judges scores are chosen for a competition, and no one knows which scores will count for that competition, then it will be a lot more difficult to influence results outside of the competition. I like the idea of skaters and coaches receiving feedback on each competition. This further encourages quality decision making since there will always need to be an account for the scores. I don't think much is served by including judges and journalists. That could have the potential of getting nasty and ugly with too many partisans. Maybe just general comments added to the score sheet posted online.

[Mrs. P]

first of all: welcome! And I agree 100%, in my opinion there should be a separate "Press conference" where the skaters/coaches/fans/journalists should be allowed to ask (politely) the judges why they gave a specific mark or GOE, and they should answer and explain (politely) their opinions about the programs, so the skaters can improve and understand what they need to change! And the biased judges will be immediatly recognized.

I think the skaters and coaches do get some additional feedback beyond the scores, don't they? The skaters and coaches are always referring to the "feedback" they've received at the competition -- I take that to be some sort of comment beyond the numbers. Or am I wrong?

That said, it wouldn't hurt to have a news conference for the media to explain scores....