Is the standard of Judging deteriorating?

[Bluebonnet]

But I just don't agree that the mistakes detracted from Mao's performance. I thinks she was FAR better at Cup of China even with the jump mistakes there. Heck, let's just go back to Japan Open. I think she did it VERY well there and would have beat a very good Ashley Wagner if not for an invalid spin. Since that initial JO program, I felt she has not performed to to her potential.

The mistakes, to me, put Mao in autopilot mode. The choreography and performance felt forced, rather than that natural lightness I'm used to seeing from her. Any other skater would and have been deducted for doing that. For perspective, her PCS score was higher than Ashley at TEB, who killed it--no mistakes, confidence, strength power. Mao didn't show any of those things.

Again, I'm fine with Akiko's PCS scores. But I have a problem with Mao scoring top PCS marks for a performance that was, quite frankly, below her usual standard, both technically and artistically.

All you've said were subjective, I'm afraid. I think Mao deserved PCS higher than Akiko this time. That was all it mattered. As how much, I can't tell because I'm not trained to do that. And the numbers I give won't count. All I know is that Mao should have PCS above Akiko. Even specifically on P/E which some have pulled out, Mao deserved higher than Akiko. The judges gave her that. The rest, I'll respect the judges' marks.

Mao's surprise look says it all -- even she thought she would lose and didn't look all that happy when she found out she won because she knows she can do much better than that.

This doesn't prove anything. Anyone would be surprised in this situation because Mao has doubled so many jumps. I believe the judges themselves didn't know the outcome either because the difference which has given Mao the edge was so small, only 0.05 when SP and LP were added together. You can call it "fate". Unfortunately Akiko lost by such a tiny margin. Poor Akiko! My heart goes out to her!

I, for one, would totally accept the result if Akiko won this time if the numbers added up and Akiko took the edge in the end.

The results may be "right" according to the way it's scored, but I think the question is now "is they way it's scored what's BEST for skating as a spectator sport?" When most fans scratch their heads over results but really don't have any issues with the way parts were scored when reviewing the protocols, perhaps something about the method of scoring needs to be changed...

I'm sure that even if we let the fans vote for the performances and give the results, some fans will still be surprised for the results by all fans.

I don't think anyone wants to increase the percentage on technical perspect of CoP, right? If we raise the jump values just because this time Akiko did great on jumps but didn't win, next time you will find that Nan Song with a clean skating and two quads and one in combo will win over Daisuke Takahashi.

I don't see anything wrong. That was what the protocols were for. After reviewing the protocols, if you can't find anything totally unacceptable, then this is the result you'll get.

[gkelly]

What the gizmo vote would really be is the audience judging the judges and the judging system.

Not really. Because the judges would still primarily be judging the technical aspects of the skating, as they always have and always should as long as skating competitions are framed as sporting contests. And most audience members are not equipped to assess whether the judges, and tech panels under IJS, have made correct decisions regarding the technical content according to the rules (knowledgeable Golden Skate members excepted, of course).


I can think of three main reasons why general audiences' opinions would tend as a group to believe the judges got it wrong:

1) The judges were not judging according to the rules but rather according to politics or preconceptions or incorrect perceptions, and there are enough audience members whose knowledge of the rules and the technical details, and sightlines, is sufficient to do a better job of judging this competition than the actual panel at this event.

2) The judges judge correctly according to the rules, but the rules are not designed to reward the performance(s) that both experts (including the judges themselves) and lay observers would holistically consider "best."

3) The judges judge correctly according to the rules and values of the sport, and they agree with their own results and so do almost all knowledgeable observers with a good view of what the skaters did on the ice. The skater who performed best technically was not the same skater who gave the best impression to the audience, because the majority of the audience members didn't perceive the technical differences between the skaters and/or because the audience does not put sufficient weight on aspects of the skating that the sport values most highly.

In the case of 1), the audience would be judging the judges, but the likelihood of having an audience that is, on average, knowledgeable enough to know when the judges get it wrong according to the accepted rules and values of the sport is slim.

In 2), the audience would be judging the judging system. If there's a general agreement that the system consistently picks the wrong winner and consistently gets skaters in the "wrong" order according to knowledgeable overall impression, then it would be appropriate for the sport to look at changing the system.

But I expect that 3) would be the most common. And therefore that all the audience's disagreement would tell us is that audiences value different aspects of skating performances than experts do. So the audience disagreement wouldn't be saying that the experts got it wrong according to the official criteria, any more than the official results would be saying that the audience got it wrong according to audience criteria. They're just working with different criteria, or with different weighting.

[Mathman]

After reviewing the protocols, if you can't find anything totally unacceptable,...

*sigh* As a spectator sport, figure skating is doomed.

I don't think anyone wants to increase the percentage on technical perspect of CoP, right?

How about this idea? We could have a mercy rule on tech. If a skater falls three times a big hook comes out and mercifully pulls him off the ice. The jumbotron flashes: "Sorry, it's not your day. Let's give this skater a big round of applause, because that's all he/she will get today." (This rule would have been a great kindness to Alissa Czisny at Worlds. )

Or -- if you pop four jumps, out comes the hook, "Sorry, cowgirl, your ride is over."

[Mathman]

3) The judges judge correctly according to the rules and values of the sport, and they agree with their own results and so do almost all knowledgeable observers with a good view of what the skaters did on the ice. The skater who performed best technically was not the same skater who gave the best impression to the audience, because the majority of the audience members didn't perceive the technical differences between the skaters and/or because the audience does not put sufficient weight on aspects of the skating that the sport values most highly....

...I expect that 3) would be the most common. And therefore that all the audience's disagreement would tell us is that audiences value different aspects of skating performances than experts do. So the audience disagreement wouldn't be saying that the experts got it wrong according to the official criteria, any more than the official results would be saying that the audience got it wrong according to audience criteria. They're just working with different criteria, or with different weighting.

I think this is way too analytical. The whole purpose of the exercise would be so the audience could shout, "Protocols, schmotocols, this kid skated lights out! Give her the medal!"

[gkelly]

Would these rules apply at all senior- and junior-level skating competitions, or only those with television coverage and arenas full of paying spectators? If the best skater at a U.S. regional is trying lots of triple jumps and falling on a few early in the season, should she get dragged off and disqualified so she can't qualify for sectionals and nationals, but weaker skaters who are not trying triples and stay on their feet can move on?

Maybe there should be a worldwide championship for all countries to participate in, with qual rounds and everything, and low-priced tickets to attract diehard fans and locals interested in experiencing the sport live, with technically oriented rules. And then only the top 6-12 go on to a televised, high-priced-ticket event with different rules that are more audience friendly. If so, should skaters be expected to modify their programs between the qualifiers and the high-profile events in order to please those audiences and audience-friendly rules better?

What about the Olympics, in which participating at all is a big deal for many countries and it's already harder to qualify spots? Should that allow 24-30 participants per discipline? Should it drag skaters off the ice if they fall three times and humiliate those skaters even further in their disappointing Olympic experience?

[Mrs. P]

All you've said were subjective, I'm afraid. I think Mao deserved PCS higher than Akiko this time. That was all it mattered. As how much, I can't tell because I'm not trained to do that. And the numbers I give won't count. All I know is that Mao should have PCS above Akiko. Even specifically on P/E which some have pulled out, Mao deserved higher than Akiko. The judges gave her that. The rest, I'll respect the judges' marks.

Eh, PCS scoring is subjective to some extent. As I said, I don't have a problem with Akiko getting lower marks than Mao overall (with Mao getting higher marks in SS/TR), but perhaps not as wide of a margin. I read the bullet points for CH/IN/PE and I still think the judges got it wrong on that one.

This doesn't prove anything. Anyone would be surprised in this situation because Mao has doubled so many jumps. I believe the judges themselves didn't know the outcome either because the number which has given Mao the edge was so small, only 0.05 when SP and LP were added together. You can call it "fate". Unfortunately Akiko lost by such a tiny margin. Poor Akiko! My heart goes out to her!

I, for one, would totally accept the result if Akiko won this time if the numbers added up and Akiko took the edge in the end.

She got only a slightly worse score (117) than what she put forth at COC, which was a much better performance, IMO. Let's face it, she is clearly embarrassed to win under those circumstances. I do agree that Akiko needs to work on the SP so the judges don't have any window for her not to win when she does a great FS. But come on you lose by 0.05 when the performances are close. Yes, Mao had a great SP and bad FS while Akiko had a bad SP and a great FS, but I felt that Akiko was penalized much more heavily. Yes she made that mistake in the short, but how is it that she loses 2 points (factored) in PCS because of it? I see a lot of posters arguing Mao's PCS score should have not gone down because of the technical mistakess. When factored, Mao actually scored nearly the same PCS as her excellent short. Doesn't make much sense.

Heck, I'll use your favorite PC. You will not see me complaining about his win at Cup of Russia despite him doubling a number of jumps as well. Not only did he score well technically for a solid (if not perfect) jump content, but those double jumps, IMO, did not interrupt the program. I still thought he met the bullets for the PCS categories as well.

I'm sure that even if we let the fans vote for the performances and give the results, some fans will still be surprised for the results by all fans.

I don't think anyone wants to increase the percentage on technical perspect of CoP, right? If we raise the jump values just because this time Akiko did great on jumps but didn't win, next time you will find that Nan Song with a clean skating and two quads and one in combo will win over Daisuke Takahashi.

I don't see anything wrong. That was what the protocols were for. After reviewing the protocols, if you can't find anything totally unacceptable, then this is the result you'll get.

Well again it's the matter of the sum being less than the parts, that's the point mskater93 was making. I don't think she's arguing for TES to be a better factor the score, but rather wonder what is missing. I think it is a major problem when you look at the protocol sheets and still scratch your head. My opinion of course, but I think it's shared by others.

[Bluebonnet]

This is an amazing suggestion! Basically the proposal is to let the audience grade the performances by by 6.0.

I disagree. It will totally take away the seriousness of a competition, and make it truly like what Johnny Weir said, "Dog and Cat Shows". That is not what an Olympic sport wants.

This actually all come back to what SkateFiguring has suggested once a few years ago. If we want the audiences involvement, we could do it at the galas after the competitions or at the shows as we've had like Marshalls.

[Mrs. P]

I disagree. It will totally take away the seriousness of a competition, and make it truly like what Johnny Weir said, "Dog and Cat Shows". That is not what an Olympic sport wants.

This actually all come back to what SkateFiguring has suggested once a few years ago. If we want the audiences involvement, we could do it at the galas after the competitions or at the shows as we've had like Marshalls.

I don't think audience-friendly competitions are necessary.

I think in 95 percent of competitions, you'll find that people are fine with the results. It's just a few competitions where people are scratching their heads. And FS is not the only sport where this happens. In football, most people agree with the ref calls, but you'll find a handful of games (the 2006 Super Bowl with the Seattle Seahawks/Pittsburgh Steelers for example) where people STILL to this day wonder why the results is what it is. (Seattle Seahawks fans, sure, but I still hear plenty of of non-fans question the result too). Or those substituent ref games that were really screwed up.

The audience, for the most part, understand the result. So you won't find me saying the system is ABSOLUTELY and TOTALLY broken. But I will say that I do not agree how the judging was done at this one competition. And neither do a lot of other people.

[gkelly]

The audience, for the most part, understand the result. So you won't find me saying the system is ABSOLUTELY and TOTALLY broken. But I will say that I do not agree how the judging was done at this one competition. And neither do a lot of other people.

Fair enough.

And that has always happened under any judging system.

So I think the question is not "Is the standard of judging deteriorating?" -- It would require a detailed analysis of many results over many results over the course of several years to argue that trend.

But rather, "Did the judges get it wrong in this decision at this event?"

Maybe as we discuss the consensus will be yes. Or maybe no more than half of us will completely disagree with the results and the others will either completely agree or see how it could have gone either way.

[AlexRus]

The Judges seem to be all over the place with PCS marks, with some skaters getting very inflated PCS despite multiple falls and popped jumps, whilst others are undermarked in PCS despite skating clean.

That the judges play strange games with PCS - it is familiar for me. Not good but known.

Mao's skating at NHK (both SP and FS) shows new problem - strange games with TES.
What may mean TES = 35.70 for SP without 3+3 combo?
For example, Julia at Junior Worlds had TES 36.43 for SP with 3T+3T and 3Lz, and score for Julia's spins was 0,87 higher than Mao's. This difference is even more than difference in TES (0,73).
I.e. Julia for non-spin elements (with 3+3) has TES lower than Mao (with 3+2). And it is not because "dirty" Julia's elements - we may see on video that they were good enough.
What can it mean? Inflation of TES score.

The same about TES of Mao's FS with three triples - it looks too high.

[Bluebonnet]

How about this idea? We could have a mercy rule on tech. If a skater falls three times a big hook comes out and mercifully pulls him off the ice. The jumbotron flashes: "Sorry, it's not your day. Let's give this skater a big round of applause, because that's all he/she will get today." (This rule would have been a great kindness to Alissa Czisny at Worlds. )

Or -- if you pop four jumps, out comes the hook, "Sorry, cowgirl, your ride is over."

This is cruel to the skaters who prepared for months and months and they aren't even allowed to finish what they've prepared. I think all we could do is to increase the penalty if too many big mistakes were made in one program.

Eh, PCS scoring is subjective to some extent. As I said, I don't have a problem with Akiko getting lower marks than Mao overall (with Mao getting higher marks in SS/TR), but perhaps not as wide of a margin. I read the bullet points for CH/IN/PE and I still think the judges got it wrong on that one.

The only way that we could ensure a desired outcome is to predetermine the winner. For example, in this case, after both performances, the judges had to assume that Akiko should win and they had to make Akiko win no matter what. So they set a number which ensured Akiko would win, then went back to details to change the numbers. Let's make Mao's average on P/E 7.94 instead of 7.96. Akiko got 0.02 more. Then let's make Mao's average on CH 8.12 instead of 8.14. Akiko got another 0.02 more. Let's make Mao's IN 8.12 instead of 8.14. Yay, alright! Akiko got total 0.06 more. Akiko won. Sounds silly! What if Mao did one more triple instead of double? Do the judges have to take off more from here or there to ensure Akiko's win?

I assume that was what 6.0 did - to compare, though I'm not familar with 6.0 system.

She got only a slightly worse score (117) than what she put forth at COC, which was a much better performance, IMO. Let's face it, she is clearly embarrassed to win under those circumstances. I do agree that Akiko needs to work on the SP so the judges don't have any window for her not to win when she does a great FS. But come on you lose by 0.05 when the performances are close. Yes, Mao had a great SP and bad FS while Akiko had a bad SP and a great FS, but I felt that Akiko was penalized much more heavily. Yes she made that mistake in the short, but how is it that she loses 2 points (factored) in PCS because of it? I see a lot of posters arguing Mao's PCS score should have not gone down because of the technical mistakess. When factored, Mao actually scored nearly the same PCS as her excellent short. Doesn't make much sense.

All feelings! I truly believe that the judges are much more objective than us fans. Let's give Akiko a big hug! That's all we could and should do.

By the way, I think that to compare numbers from two different competitions often lead to wrong conclusions.

[Skater Boy]

I am not sure judging is deteoriating at all really. It is what it is - judging and there are certain biases. It takes more than one judge for a Mao or Chan or Carolina to win. The scoring system inherently creates its own biases. That is life. Bickering, complaining, yelling about it may help change the judging system - better or not is questionable but it tends to make non skating fans laugh at skating. So, for example the outrage about Salt Lake City may just have hurt the sport more especially since part of the problem despite the judge bias may lie in the system itself - artistry or the second mark trumps technical back then.

[Mathman]

OK, I take back the hook idea.

It is not at all clear that figure skating aspires to be a big-time, or even a medium-time, spectator sport. If Cinquanta and company have no interests along those lines, then so be it.

But -- if we do want to put on sporting events and charge people $100 to come and see them, then we have to give the people $100 worth of sports experience. For example, a performance like Michelle Kwan's Tosca at 2004 U.S. Nationals was satisfying because when she hit that last triple jump, turned the corner, adrenalin pumping, and came storming down the ice with that victory celebration of a step sequence -- well, that's like the halfback busting through the line, streaking past the last defender, and pounding for the end zone for all he's worth.

What football fans don't need is some expert smugly telling them, "Meet me in the math classroom tomorrow and we will go over the hundredths of a point increments so you will know who won." I already know who won -- I just saw him score the winning touchdown.

As for competitions for children and developmental and recreational level contests, I don't have all the answers. But baseball has no problem running an immensely successful little league program, school and college programs, extensive minor leagues, many, many adult recreational softball leagues (mostly co-ed), as well as the popular and profitable big show. This requires only slight modifications of the rules to accommodate various circumstances.

Anyway, I am not against the CoP. I think most skating contests are well judged. Arguing over the outcome is common to all sports -- it shows we care.

What I don't like so much is constantly being told, if you don't like the results, read more protocols.

No thank you. I'll just continue not to like the results. Like fans of every sport.

[mskater93]

Well again it's the matter of the sum being less than the parts, that's the point mskater93 was making. I don't think she's arguing for TES to be a better factor the score, but rather wonder what is missing. I think it is a major problem when you look at the protocol sheets and still scratch your head. My opinion of course, but I think it's shared by others.

This is exactly my point - in this case the judging SYSTEM got it wrong (and it's happened on multiple occasions in the last several years). I am not saying change the balance, I am saying find a way to reward the whole.

[gkelly]

I think one of the most important simple changes that should be made (under any judging system that combines the results from two or more competition phases, by any mathematical method) is this:

During the final phase of the competition, after each skater skates, before announcing that skater's overall placement, always announce the placement in that phase of the competition only.

Audiences won't always have seen the short program/short dance and those who have seen it might not remember the standings or might not be familiar with the rules for combining results from the separate phases.

What they do know is what they just saw and who they thought was better today. So make sure to let them know what the standings were today.

E.g., instead of "for a total of xxx.xx points for the freeskating. That gives her an overall total of zzz.zz, which puts her currently in first place!"

take a few extra seconds to say "...for a total of xxx.xx points for the freeskating. That puts her her currently second in the freeskate. Combined with her short program score of yy.yy, that gives her an overall total of zzz.zz, which puts her currently in first place!"

That will cut down on a lot of the outrage when it was clear that this skater has already been outskated today, and the judging panel agrees.

There will still be times when the majority of the audience disagree with the majority of the judging panel about today's results specifically, but hey, that's life. Some of the judges may disagree too. It would be interesting to calculate ordinals to find out how many judges (and which ones) gave the highest scores to which skaters, but I don't see that happening as long as the ISU believes that anonymous judging is less subject to pressure from federations.