The Judging Controversy Thread

[cchen24]

If you say you would watch certain competitors' programs carefully, isn't that implying not everyone would get the same attention? Nonetheless, that isn't the main point. You say that any good performances later would have to be compared to Mao. Actually, to be completely fair, all later performances would have to be carefully evaluated against all previously skated performances in the event, not just Mao's. That's where it becomes a bit impractical for these judges to get things exactly as you want to see them in the time they are given to score so many factors of the program.

Of course, every skater should be carefully evaluated.
Let me repeat again how the whole thing played out as a reminder.

After Mao skated, she was in the lead.
All skaters coming up after her would be placed either above her or below her.
Let's just focus on the five categories of PCS.

If you were a judge, would you compare the current skater's skating skills, transitions, etc with Mao's if you thought the performance was comparable?

If not comparable, of course the skater would be marked below Mao and there would be other comparisons to do against other skaters who had skated.

And Mao was still in the lead before the final flight took the ice.
Now, Yulia just finished her skate.
Would you compare Yulia's skate with Mao's to determine if it should be above or below Mao's in terms of PCS?
If you determined it was supposed to be below, then I assume you had to go down the list to see how far below, i.e. you had to compare Yulia with other skaters who had skated.
If you determined it was supposed to be above, then all you needed to do was to decide how far above.

Isn't this the way you would mark the skaters?

I don't think what I outlined above gave Mao an unfair amount of extra attention.
The so-called extra attention she received was simply due to the fact that she remained on top for a long time in ladies LP due to a very early skate order.
In the same token, Yuna got that extra attention in SP because she skated early and she became the one to beat.

 

[yuki]

For me, although Gracie's PCS was below Mao and Julia, it should have been much lower. She looks like a robot out there.

Gracie is a bit stiff and she doesn't pay that much attention to the music (neither does Yulia, to be honest), but she does have good basic skating and nice speed. Then again, both she and Yulia are still very young and have enough time to develop artistically (getting accurate PCS that would encourage them to work on those parts of their skating would help, though ).

Also, Frank Caroll disagrees with you. Apparently, he doesn't see how Mao's interpretation is superior to Gracie's .

 

[gkelly]

The so-called guidelines are always going to be vague because it's not going tell you what, in exact terms, deserves to get 8.5 in skating skills, or 7.25 or 6.75 or 9.75.

Very true.

Judges are supposed to develop their own mental standards for what each of those numbers mean, and there should be at least some vague general agreement across the judging ranks as to what is "average" or "good" or "outstanding."

But it will never be possible to have all judges perceive these qualities on the exact same scale.

And in practice even individual judges may vary slightly in their application. E.g., the excitement of the Olympics, especially the final group, might inflate everybody's scores without the judge even realizing.

At the end of the day, the most important task for the judges is to rank all the skaters.

Actually, a big part of the concept behind the IJS is that the judges are not supposed to rank the skaters. They're just supposed to mark what they see, according to their hopefully-consistent mental standards.

That's one reason why judges are not told the levels of the spins and steps (and various leveled pair moves).

Nor are they supposed to check their notes for previous skaters while marking the current skater, nor allowed to write down previous skaters' totals.

Even if they know all the level rules and are math whizzes and have great memory for numbers, at best they can only estimate what this skater's TES will be and what their total might be combined with the judge's PCS. So they can't award their PCS with any confidence that they will be placing one skater ahead or behind another if the technical content and components are at all in the same general range.


What they can and should do is make every effort to be consistent in their use of PCS throughout the competition. So if they gave good component marks to skater A, and then they determined skater B was not quite as good in those areas, they should give lower marks to B for those components.

They could probably remember the PCS they gave to standout skater A earlier, at least to within 0.25 here or there, and award their PCS for not-quite-as-good skater B accordingly.

But unless there's an obvious difference they can't add up what B will earn for TES and guarantee that they will place B behind A in this program, let alone overall if B was well ahead in the short program.

 

[capcomeback]

It's more the skate order IMO. From the judges perspective, I think this might be the mentality:

Skaters 1 and 2 skate ok. They get marks in the 5's.
Skater 3 skates great. She get's marks in the 8's because she is a lot better than what they have seen so far.
Skaters 4,5,6,7 and 8 skate ok and get marks in the 6's.
Skater 9 skates very well but not as good as skater 3, but better than 4-7. She gets marks in the 7's.
Skater 10 skates great, a lot better than 9. She gets 9's.

It becomes difficult after several groups for the judges to accurately compare skaters who came far earlier in the evening, and the judges do get somewhat boxed into higher scores for later skaters because the middle groups do get some good marks. I don't think asking a judge to grade all the elements, judge the components, and accurately compare each skater to those who skated two or three groups earlier in about a minute is realistic.

Judges should be aware of their placements of skaters when judging performances. If a skater in an earlier group is the best than the 8 you gave them earlier equals 10 and everybody must be scored based upon that performance. If somebody actually skates better, than they get a 9 and that is the new 10. The key thing is that there was no way that Adelina's FS was any better than 4th (behind Yuna, Carolina and Mao in some combination of order). With her score and second place finish in the SP (also questionable) should have netted her a bronze and not a gold.

The officials (equal tech panel and some judges) either rigged it or created a "perfect storm" of bias and ineptitude that allowed a home cooked gold medal. Either explanation does not reflect well on the sport.

 

[Sam-Skwantch]

gkelly

 

[drivingmissdaisy]

And Mao was still in the lead before the final flight took the ice.
Now, Yulia just finished her skate.
Would you compare Yulia's skate with Mao's to determine if it should be above or below Mao's in terms of PCS?
If you determined it was supposed to be below, then I assume you had to go down the list to see how far below, i.e. you had to compare Yulia with other skaters who had skated.
If you determined it was supposed to be above, then all you needed to do was to decide how far above.

Isn't this the way you would mark the skaters?

If time wasn't a factor, sure. But you are asking judges to make this evaluation five times (for each PCS category), not just once for an overall impression. On TR, for example, do you expect a judge to accurately evaluate Julia's transitions against a skater a judge hasn't seen for over an hour? There is a lot of data that the judges are asked to process in a very short amount of time, which means results we don't like are going to happen.

 

[gkelly]

Judges need to be aware of their placements of skaters when judging performances.

That's 6.0 thinking. In IJS they're specifically not supposed to be placing skaters, and there's no way to know exactly how they place skaters in overall totals.

It might come down to a difference in spin or step levels, which the judges never ever know while the competition is going on. Judges have no control over that.

 

[drivingmissdaisy]

The officials (equal tech panel and some judges) either rigged it or created a "perfect storm" of bias and ineptitude that allowed a home cooked gold medal. Either explanation does not reflect well on the sport.

Well I don't think it's ineptitude because Adelina's program was skated in between the other medalists, so the judges had a good chance to evaluate them in close time proximity. Maybe a third explanation was that the judges rewarded the most difficult program, and by difficult I mean the hardest jump content, spins, footwork, transitions, and choreography. I'm not sure why people feel like the intricacy of Adelina's choreography from the ankles-down should not somehow factor in to her score. Yuna caresses her face a lot in her LP, which is very lovely, but it isn't all that difficult.

 

[cchen24]

Actually, a big part of the concept behind the IJS is that the judges are not supposed to rank the skaters. They're just supposed to mark what they see, according to their hopefully-consistent mental standards.

That's too ideal and I fear that it's never going to work that way, i.e. not ranking the skaters.
(I mean ranking skaters in different aspects of their skating, not the final ranking, though.)

What I like about CoP is that it allows the judges to rank the skaters in various aspects of their skating.
Say, skater A may have the best skating skills but is not good in interpretation.
At the end, the one with the highest combined score wins.
It provides skating fans more information than the two marks under 6.0.
It also encourages the skaters to develop themselves into more well rounded skaters.

However, I don't think there is any way to eliminate "ranking" in judges' minds because it is so against human nature.
Also, skaters on the scene change over the years and they all come with different styles and skills,
It's very hard to come up with a set standard for PCS. I believe the judges would always compare all skaters to arrive at their PCS marks. But at least, the judges are allowed to tie many skaters in each of the five program components if they do see fit. That's a nice feature, too.

What they can and should do is make every effort to be consistent in their use of PCS throughout the competition. So if they gave good component marks to skater A, and then they determined skater B was not quite as good in those areas, they should give lower marks to B for those components.

Then, I think we are saying the same thing. The judges do compare or rank the skaters in each program component...

 

[cchen24]

If time wasn't a factor, sure. But you are asking judges to make this evaluation five times (for each PCS category), not just once for an overall impression. On TR, for example, do you expect a judge to accurately evaluate Julia's transitions against a skater a judge hasn't seen for over an hour? There is a lot of data that the judges are asked to process in a very short amount of time, which means results we don't like are going to happen.

I am all for a separate panel for PCS, even one panel for each component if needed.

The skaters work so hard and they deserve to be judged very carefully. I believe you will agree on this one.

 

[drivingmissdaisy]

I am all for a separate panel for PCS, even one panel for each component if needed.

The skaters work so hard and they deserve to be judged very carefully. I believe you will agree on this one.

Honestly, I don't even like PCS because I just feel like the judges should just rank the skaters how they want. The PCS scores give the impression that there is something scientific and more fair when the judges use it to rank the skaters how they would under any subjective scoring system.

 

[gotoschool]

If Lipnitskaya and Sotnikova to a great extent, and Gold to a lesser extent, had been scored more fairly in terms of PCS, tech marks GOEs, edge calls and under-rotation with a figure more in line with what they received during the Grand Prix season for both the SP and LP, even if the scores were somewhat higher, and Mao had been given the score she truly deserved for her stellar LP, remember we're talking a base tech score of 66 with no mistakes and outstanding presence and interpretation, Mao would have received the bronze not Sotnikova.

 

[yuki]

What I like about CoP is that it allows the judges to rank the skaters in various aspects of their skating.
Say, skater A may have the best skating skills but is not good in interpretation.
At the end, the one with the highest combined score wins.

That is one aspect of PCS that works well in theory, but is rarely used properly in practice. Instead of looking at each component separately, judges usually just score in a narrow corridor, slightly varying all the PCS around one mark (usually SS), so your skater with great skating skills but average interpretation would not in fact get IN marks that are that much lower than his SS, even though he should (and it would indicate to him that he needs to work on that aspect of his skating). The wording of the criteria leaves much room for interpretation, which doesn't help either (PE/EX, CH and IN could really be just one component - however, it's easier to factor 5 components rather than 3).
What I'm trying to say is that it's not necessarily the system itself that is at fault, but the way in which it is applied. It doesn't help that the judges need to pay attention to every single element of the program and then also come up with an impression of the program as a whole. Maybe having two separate panels for GOE and PCS would be better in this aspect.

 

[cchen24]

That is one aspect of PCS that works well in theory, but is rarely used properly in practice. Instead of looking at each component separately, judges usually just score in a narrow corridor, slightly varying all the PCS around one mark (usually SS), so your skater with great skating skills but average interpretation would not in fact get IN marks that are that much lower than his SS, even though he should (and it would indicate to him that he needs to work on that aspect of his skating). The wording of the criteria leaves much room for interpretation, which doesn't help either (PE/EX, CH and IN could really be just one component - however, it's easier to factor 5 components rather than 3).
What I'm trying to say is that it's not necessarily the system itself that is at fault, but the way in which it is applied. It doesn't help that the judges need to pay attention to every single element of the program and then also come up with an impression of the program as a whole. Maybe having two separate panels for GOE and PCS would be better in this aspect.

ITA!

CoP has many merits to it by design. But most of the time, the judges simply use PCS as place holders which does not always reflect the actual strength and weakness of a skater. It is a shame.

This is what I have said a few times: the problems aren't the scoring system, but the judges who apply the rules to mark skaters.
But at the very least, now skating fans have a protocol to study instead of two mysterious marks (tech+presentation) under 6.0.

 

[pangtongfan]

If Lipnitskaya and Sotnikova to a great extent, and Gold to a lesser extent, had been scored more fairly in terms of PCS, tech marks GOEs, edge calls and under-rotation with a figure more in line with what they received during the Grand Prix season for both the SP and LP, even if the scores were somewhat higher, and Mao had been given the score she truly deserved for her stellar LP, remember we're talking a base tech score of 66 with no mistakes and outstanding presence and interpretation, Mao would have received the bronze not Sotnikova.

The fact there is even discussion to whether Sotnikova deserved the bronze medal over someone who finished almost 30 points behind her, yet she won the gold, just speaks volumes to the level of curruption of the Sochi ladies event as a whole.

 

[gotoschool]

In terms of the comment that Mao should not even being considered as a possibility for the bronze under an equitable judging system because she performed poorly in the SP, it seems to me that the competition should be based on tallying the point total objectively for both the SP and LP, instead of a judge going out of their way to lowball a competitor performing an outstanding routine with a world record number of triples and a triple axel, because they had one bad skate amid a sea of successful ones during the rest of the season. This is especially true since the freeskate is 2/3 of the competition. This reasoning could be extended to the Grand Prix season or last year's worlds. If a skater has no titles or medal placements, then they should also receive lower scores, at least under PCS, but that did not happen. In fact, the only ones inflated, Lipnitskaya, Gold and Sotnikova, had few if any medals between them.

 

[drivingmissdaisy]

The fact there is even discussion to whether Sotnikova deserved the bronze medal over someone who finished almost 30 points behind her, yet she won the gold, just speaks volumes to the level of curruption of the Sochi ladies event as a whole.

So if I say that I think Kaetlyn Osmond deserved gold is there now a "discussion" that further taints the Sochi results? If you think Mao should have been on the podium after that SP it just shows how grossly biased against Adelina you are. Which we already knew, as you thought people disagreed with the results 30-to-1 but it was actually more like 50-50 (at least in the US). However, finding someone who agrees with you that Mao should have beaten Adelina is easily the best evidence you have presented that there was corruption so keep digging. You're making progress.

 

[drivingmissdaisy]

In terms of the comment that Mao should not even being considered as a possibility for the bronze under an equitable judging system because she performed poorly in the SP, it seems to me that the competition should be based on tallying the point total objectively for both the SP and LP, instead of a judge going out of their way to lowball a competitor performing an outstanding routine with a world record number of triples and a triple axel, because they had one bad skate amid a sea of successful ones during the rest of the season.

Where is this sea of successful programs? She won her competitions on BV and PCS, not great execution.

 

[capcomeback]

Well I don't think it's ineptitude because Adelina's program was skated in between the other medalists, so the judges had a good chance to evaluate them in close time proximity. Maybe a third explanation was that the judges rewarded the most difficult program, and by difficult I mean the hardest jump content, spins, footwork, transitions, and choreography. I'm not sure why people feel like the intricacy of Adelina's choreography from the ankles-down should not somehow factor in to her score. Yuna caresses her face a lot in her LP, which is very lovely, but it isn't all that difficult.

To be honest, I did not see much in terms of choreography from Adelina from the top down. There were times that I'm not sure she even had a sense of what she was skating too (as it seemed that she was focused on the elements, just checking them off as she completed them). This lack of musicality is not totally unexpected from a seventeen year old girl, but should not be rewarded to the extent that it was in terms of her choreography or interpretation. Empty transitions (in other words movements for the sake of movement) should not be rewarded either.

In addition the lack of speed/ice coverage compared to Yuna and Carolina (which is apparent from viewing the overhead view or her program) should be noted as it is ridiculous that Adelina received as high a score on the quality of her skating. As Dick Button pointed out it, "she's not a complete skater". This kind of skater does not deserve to almost set an Olympic record- regardless of scoring system. If this has become a sign of things to come, we might as well drop the PCS score and make it all about the technical aspect of skating After all, to be honest about it, that is what happened in Sochi (assuming you don't subscribe to the conspiracy theory, which I do).

 

[Mathman]

But at the very least, now skating fans have a protocol to study instead of two mysterious marks (tech+presentation) under 6.0.

Yes. We have a protocol to study.

That's what we have. A protocol to study. hwell: