Are judges able to "rank" skaters while they are judging? | Golden Skate

Are judges able to "rank" skaters while they are judging?

Poodlepal

On the Ice
Joined
Jan 14, 2010
This is an offshoot to the conversation about Evan vs. Evgeni. I thought I'd start a new thread to ask my question, to separate it from the quad question.

Under 6.0, they did, correct? They would rank the skaters, saying "OK, I thought Tara was better than Michelle, so I'll give her a 5.9 because I gave Michelle a 5.8" for example. A judge would put the skaters in order as they liked them, and panel of crooked judges could manipulate the whole competition if they wanted to, by placing certain skaters above others en masse.

Under the COP, is this type of manipulation possible? Do they know that in order to make sure Skater A wins, they must not give Skater B more than 7.2 for transitions, for example? Or is it so confusing that they just add or take away decimal points as they see fit, with really no idea what it will all add up to in the end?

People complain about the judging all the time, but maybe there are so many numbers and so many categories, the judges themselves may have no idea what the final score will be for a skater (unless they can add in their heads), and may not know exactly who will be in first or second place. That doesn't let the COP off the hook, if the base values are off, however.
 
Joined
Jun 21, 2003
Great thread question!

My impression is that, no, the judges cannot compute fast enough in their heads to know that they need to give their favorite a 7.2 in transitions to make that skater come out ahead.

But a judge could certainly score the favored skater generously across the board, while low balling that skater's rival. Since all the points of all the judges are added up, this might be even more effective than just to give a high ordinal on the judge's own card. The rule about dropping the highest and lowest mark reduces the impact of this strategy, however.

There is also the potential for the technical panel to call errors more stringently for one skater than for another.
 

janetfan

Match Penalty
Joined
May 15, 2009
Let's not forget that Cinquanta said anonymous judging is necessary to reduce cheating.

This means he believes federations and their judges have the same attitude about scoring under CoP as they had in the 6.0 era.

Personally I think the truest purpose of the CoP anonymous judging is to make it harder to get caught again, particularly at the Olympics which was such a major embarrassment for IOC in 2002.

As to competitions other than the Olympics ISU seemed comfortable enough with 6.0 or CoP judging.

"Judges are human" and changing a scoring system does not change human nature.
 
Last edited:
Joined
Jun 21, 2003
On second reading of Poodlepal's post, I think I did not speak to the right question. It's not about cheating, but about whether judges are still using the scoring system to rank skaters as to who they thought was best, second best, etc. Right?

OK, in that case I think that at first the judges used the PCSs exactly like that. The skater that they thought was best, they gave that skater straight 8.5's. The skater they thought was second best got 8.25's, third best was 8.0's, and so on.

I think that over the years, as the judges got used to the new system, they began to make a more serious effort to score according to the guidelines. So that it would be possible (but still not common) for a skater to get high marks in interpretation but low marks in skating skills if the judge thought such marks were justified.

Still, I do not think that the ordinal approach can be completely eclipsed. At the end of the day, somebody wins, somebody is second and somebody third. The judges will give the higher marks to the skater that the judge thinks skated the best, as always.
 

janetfan

Match Penalty
Joined
May 15, 2009
I think that over the years, as the judges got used to the new system, they began to make a more serious effort to score according to the guidelines. So that it would be possible (but still not common) for a skater to get high marks in interpretation but low marks in skating skills if the judge thought such marks were justified.

.

Do you think Joe Inman might have felt differently about this last season?

We know how Plushy felt about it when he remarked about TR and then again when said Evan's OGM was not about who skated best but which country/federation "needed it more."

When a well respected judge and skater of Plushy's stature make such remarks do you think they are satisfied with the scoring system?

Were they questioning the integrity or the basic nuts and bolts of the system :think:
 

Tonichelle

Idita-Rock-n-Roll
Record Breaker
Joined
Jun 27, 2003
When a well respected judge and skater of Plushy's stature make such remarks do you think they are satisfied with the scoring system?

chances are in the 6.0 system or any system had Plushenko gotten anything but first place his nose would have been severely bent out of joint and he would have demanded his platinum medal just the same. :sheesh:
 

janetfan

Match Penalty
Joined
May 15, 2009
chances are in the 6.0 system or any system had Plushenko gotten anything but first place his nose would have been severely bent out of joint and he would have demanded his platinum medal just the same. :sheesh:

and what about Judge Joe?

Was he implying that too many judges were incapable of scoring the PCS properly?

If that is true then how are the same judges going to arrive at a good and fair point total?
 
Joined
Jun 21, 2003
Do you think Joe Inman might have felt differently about this last season?

I think that Joe Inman (and almost everybody else who thinks about the CoP) believes that there is a gap between what the CoP says on paper and what the judges actually do.

Were they questioning the integrity or the basic nuts and bolts of the system :think:

I do not think they were questioning the integrity of the individual judges. In Inman's case, I think it was more an accusation of laziness on the part of the judges, that they were not working hard enough to make distinctions between all the compartments and bullets that the program components are divided up into.

As for the basic nuts and bolts, I do think there is a big problem with the CoP. Underlying the CoP is the tacit assumption that it is possible to list enough criteria that a judge can discriminate, in a consistent and objective way, between a program that deserves 7.5 in Performance/Execution and one that deserves (objectively and consistently) 7.75.

Personally, I think this claim is absurd. But it is possible to say, of these two performances, Skater A's was superior to Skater B's in terms of performance values.
 

janetfan

Match Penalty
Joined
May 15, 2009
I think that Joe Inman (and almost everybody else who thinks about the CoP) believes that there is a gap between what the CoP says on paper and what the judges actually do.



I do not think they were questioning the integrity of the individual judges. In Inman's case, I think it was more an accusation of laziness on the part of the judges, that they were not working hard enough to make distinctions between all the compartments and bullets that the program components are divided up into.

As for the basic nuts and bolts, I do think there is a big problem with the CoP. Underlying the CoP is the tacit assumption that it is possible to list enough criteria that a judge can discriminate, in a consistent and objective way, between a program that deserves 7.5 in Performance/Execution and one that deserves (objectively and consistently) 7.75.

Personally, I think this claim is absurd. But it is possible to say, of these two performances, Skater A's was superior to Skater B's in terms of performance values.

Let's say we are both judging and you think Michelle deserved a 5.9 for presentation.
I only gave her a 5.8 because I liked Sasha better and gave her a 5.9.

It's still subjective, just as GOE is subjective along with the PC points.
We even read complaints that the levels are called wrong on steps at times so I will also call that subjective.

Is 7.75 anymore subjective than 5.9?

I think it is because the human mind can only accurately process so much information.

I am sure I know who I liked better but to get it right (to the best of my ability) I would need to watch it 3-4 times and have a good 20-30 minutes to break it down the way the CoP does.

I see CoP as little more than a smokescreen at times and what certain fans like most about it is what I find most troubling.

Joe Inman was asking for what is extremely difficult to do. Watch a program once and then decide in a couple of minutes how to score it to the hundredth of a point in so many different categories.

Of course they get it wrong many times as they are human.
 
Last edited:
Joined
Jun 21, 2003
Let's say we are both judging and you think Michelle deserved a 5.9 for presentation.

I only gave her a 5.8 because I liked Sasha better and gave her a 5.9.

Yes. That is the nature of judging. That is why judging is different from measuring. That is why quality is different from quantity. That is why figure skating is not downhill skiing.
 

janetfan

Match Penalty
Joined
May 15, 2009
Yes. That is the nature of judging. That is why judging is different from measuring. That is why quality is different from quantity. That is why figure skating is not downhill skiing.

Not sure if I get your point here. Are you suggesting that

6.0 = judging

CoP = measuring

:)
 
Joined
Jun 21, 2003
Not sure if I get your point here. Are you suggesting that

6.0 = judging

CoP = measuring

Yes. In basic intent. (I do not think that the CoP succeeds in its intent, however.)

"Ordinals" means putting things in order: first, second, third. That's what 6.0 judging did, and was not ashamed to admit it.

The CoP says, you get so many points for this and so many points for that. The winner is determined by arithmetic. I think there is a fundamental mismatch between this scoring philosophy and nature of figure skating.

I am not really against the CoP, however. I just think it is a little bit naive to believe that it can do what it claims.
 

prettykeys

Medalist
Joined
Oct 19, 2009
Yes. That is the nature of judging. That is why judging is different from measuring. That is why quality is different from quantity. That is why figure skating is not downhill skiing.

Not sure if I get your point here. Are you suggesting that

6.0 = judging

CoP = measuring

:)
No, Mathman is stating that the nature of a judged sport such as figure skating can include some degree of differences in judgment, unlike measurement where people can agree on e.g. who was the fastest.

Edit: Oh, I see that that is indeed what Mathman meant.

As for Poodlepal's question, I am sure that judges are able to rank skaters. The question is, should they? Some fans seem to say yes, because only by getting a general idea of rank could they make the outcome of the scoring/judging more or less "correct". Some fans say no, possibly because it brings too much bias and not enough analysis into the scoring/judging.

I really think the most damaging aspect of the IJS right now is the anonymity of the judges. I would like to see if I can reasonably see consistency and fairness in a judge's overall method of scoring. If the CoP's strengths include objective criteria to score with, then a judges' scoresheet should be justifiable.

Anonymous judging in either 6.0 or CoP, I would imagine, would make it easier to manipulate scores and get away with it.
 

prettykeys

Medalist
Joined
Oct 19, 2009
Yes. In basic intent. (I do not think that the CoP succeeds in its intent, however.)

"Ordinals" means putting things in order: first, second, third. That's what 6.0 judging did, and was not ashamed to admit it.

The CoP says, you get so many points for this and so many points for that. The winner is determined by arithmetic. I think there is a fundamental mismatch between this scoring philosophy and nature of figure skating.

I am not really against the CoP, however. I just think it is a little bit naive to believe that it can do what it claims.
I don't know if I agree with that. It is possible to rank skaters according to e.g. performance or e.g. TES for a double axel or e.g. transitions. Although the differences in a PC score for one aspect may be just 0.25 of a point, making the overall impact of the difference on the scoresheet smaller than the impact on the overall performance (or vice versa.)
 
Joined
Jul 11, 2003
Judges are like Fans. The judges are human and apparently love the skating (it is a voluntary job? No?). Judges also know the very elite skaters they are about to judge, as do the Fans. They may even have a preference for some skaters for whatever reason and will up their individual scores for them. Fans would do that too, if they had the power of scoring.

The Elements have been given, arguably, base values, but even those are modified by Grades of Execution which are supposed to clarified by their descriptive notes. Are they? This portion of the competition is as close as we can get for quantifiable purposes.

By and large, the entire PC scores are a matter of opinion. The question should be: Are the judges qualified to give such 'truths' in that score? Do they know the difference between a Carravachio and a Botticelli for scoring purposes?

So my conclusion is the CoP continues the opinion method of the 6.0 albeit differently and in more detail. On the Plus side is that collusion is more difficult. Maybe they will find a way to collude again.

Buttle/Joubert and Plushenko/Lyacek remain questionable? or decisive? Personally, I'm mute.
 
Joined
Jun 21, 2003
I don't know if I agree with that. It is possible to rank skaters according to e.g. performance...

That is the point that I am bumbling around trying to state.

It is possible to rank performances. First, second, third is a ranking. This is what ordinal judging called upon the judges to do.

I do not believe that it is possible to score a performance. In terms of choreography, say, this is a 6.25 performance. Period. It is not a 6.00 performance and it is not a 6.50 performance.

That is the what (nominally) the CoP asks judges to do.
 

Robeye

Final Flight
Joined
Feb 16, 2010
That is the point that I am bumbling around trying to state.

It is possible to rank performances. First, second, third is a ranking. This is what ordinal judging called upon the judges to do.

I do not believe that it is possible to score a performance. In terms of choreography, say, this is a 6.25 performance. Period. It is not a 6.00 performance and it is not a 6.50 performance.

That is the what (nominally) the CoP asks judges to do.
Completely agree with this.

-As far as the "artistic" components of PCS are concerned, which are by definition analogue in nature, they are potentially amenable to accuracy (eg comparative ranking), but are resistant to fine precision (that is to say, better by how much?). This will always be true.

-In practice, the "technical" components face similar issues (due to the real-time, high speed nature of many elements, and the very subtle and fine-grained distinctions that need to be made by human beings in the heat of the moment). In principle, however, and in contrast to the artistic aspect, the technical components should be susceptible to higher degrees of quantification and measurement than is currently the case.

Measurement: the bottleneck is the limitations of human perception. A strictly quantitative measure of rotation (3, or 2.75) is used for a triple-jump, for example, but the measuring device is the human eye and the human brain, which works qualitatively. Even with the aid of slow-motion, it's not always clear-cut. In the longer term, it may be feasible to alleviate this bottleneck through a form of machine scoring, using digital motion capture technology and software.

Quantification: As a consequence of such developments, it's possible to envision more strictly quantified definitions of many, if not all, technical elements, including jumps, spins, spirals, speed, deep edging, etc., as well as GOE bullets (speed, height, ice coverage, flexibility, et al). The current definitions, which are often neither completely fish nor fowl, are necessarily the way they are because they need to be applied by human beings using imperfect tools of measurement.

-The issues with going digital: Institutional conservatism, inertia and politics, for one. And the not inconsiderable cost and effort involved, for another. But I'm of the view that all sports will be going down this road eventually, although some of them may not yet know it. At some point, though (IMHO), such technologies will be sufficiently established in the wider sporting world, and the costs will have come down so significantly, that figure skating will find it difficult to hold out, given the benefits that it offers to such a technically opaque sport.

And, too, I wonder if committed skating fans shouldn't be careful what they wish for; sure, these developments might make skaters and general audiences happier, but I sometimes muse that we skating fans are a masochistic lot, and would feel positively deflated without controversial calls to parse endlessly. :laugh:
 

Poodlepal

On the Ice
Joined
Jan 14, 2010
Thanks for answering my questions. I think that judges do play with the numbers in order to get "their" skater to win, just as much as they did under 6.0, but now it looks like the results are "scientific" and "statistically significant."

As a teacher, I don't (or at least try not to) grade the papers to make certain that a certain final average is reached, or so that student A gets a higher average than student B (to get the subject award, for example.) I grade each test or paper separately, and may not know whose average is higher until I put them in the spreadsheet. That's what the judges are supposed to be doing--just judging each skater's performance, checking those edges and transitions and so on, without regard to anybody else, and then being sort of surprised by the results.

In reality, I don't think they are doing that. I think they can kind of decide who they want to win and then prop scores up or keep them down, and I think with the anonymous judging, there is a lot of room for backroom deals and bloc judging. Maybe even more than before.

Interestingly, very few people outside of us hardcore fans even know about the code of points. If there's an offhand reference to figure skating in a commercial or during banter on a radio show, for example, they always mention getting a 6.0. In video games with figure skating, the little avatar gets a 6.0. The COP hasn't caught on at all in pop culture that I can see. I liked it a lot at first, when a lot of surprise winners came from behind, but now I'm not so sure.
 
Joined
Jul 11, 2003
^^^^^
That is the way it is in Modern Figure Skating, but there is really no fundamental difference between it and the 6.0 system.
 
Top