Does anyone miss the "old" judging system?

[andromache]

I think both examples that you give should be punished. A fall is obviously the worst mistake on the scale of mistakes. Actually, I think falls should be worth a drop in BV too. So, the Lutz on the correct edge but fall should score less - but both would lose points, and both should be discouraged.

I do not like the suggestion that the ISU floats every now and again of "call it on the edge" (meaning if your flutz is bad enough it gets called a flip etc). I think - I'll correct that to "I know" because I, myself, actually, shamefully, have the beginnings of a Lutz that is ! on a good day and e on my bad days - there are differences in the mechanics of the jump that are overall significant enough that a flutz is NOT the same as a flip, and should not be counted as such. So, a flutz should be counted as a poorly-executed Lutz, and -GOE is necessary, as it would be for a fall.

I think I just talked myself in circles but I would like to see tech panels be stricter about the correct edges at the lower levels, so that the problem can be corrected earlier. For me, I do a lot of Lutz-entry exercises with my coach in an attempt to correct the edge problem before I even rotate the jump (which I can't do - yet). I have seen others at the rink working on corrections. It is something that CAN be fixed, and it is better, IMO, to fix it early, at the beginning, then try and fix it later.

Thank you for explaining! So I think we largely agree. Because, IMO, it's a shame to see a nicely-landed flutz score less than a lutz fall. My only problem with the previous mandate for -GOE on edge calls came from how wrong-edge jumps would ultimately be worth less points than a correct-edge fall when I feel that a fall is a worse, more disruptive error.

By the way, I've seen you talk about your adult skating before...I'm so jealous! I'm hoping once I finish my MA and (fingers crossed) get a real job with a grown-up salary I can do it too!

 

[gkelly]

Well, it's currently legal to give +1 to a jump with an e call and +2 if there's a ! (assuming there are enough other positive qualities to the jump). If judges could be reminded of that rule change such that they'd feel free to use those scores where appropriate, the current rules would work for me.

 

[Mathman]

Well, it's currently legal to give +1 to a jump with an e call and +2 if there's a ! (assuming there are enough other positive qualities to the jump). If judges could be reminded of that rule change such that they'd feel free to use those scores where appropriate, the current rules would work for me.

I would hope the the judges would use that option sparingly, though. For many skaters, going off the wrong edge on a Lutz makes it easier to get full rotation, height and distance, flow on the landing, etc. We would not want a skater to be rewarded for making a deliberate choice to cheat the take-off edge.

By the way, for those of us who are curious as to whether it really makes any difference, 6.0 or IJS, we can at least look at some events and see if the result would have been different with factored placements, assuming other aspects of the scoring to be the same. In the three Grand Prix events so far this tseason two winners would have been different (out of 12 competitions). By factored placements, at Skate America Gold would have beaten Medvedeva and at Cup of China Hongo would have won over Asada.

 

[Sam L]

What I hate the most about the new judging system is that it's a numbers game. I loved how in the 6.0 ordinal system, it was a ranking. That's how the sport started out, skaters got ranked. It's about places. Now, it's not about places, it's about numbers based on your performance. When it was about places, it created a lot of drama and a lot of speculation and possibilities about who beats whom and the likely outcomes. Like the 2002 Olympics. It all depended on whether Slutskaya (the last skater) beat Kwan or not. Nowadays, it's just about whether you can get the numbers to put yourself in first place or not. That drama and possibilities was what made Figure Skating, Figure Skating imo. But unfortunately, in this sport, things have been given way just because the public "doesn't understand" it.

The new system is not hard to understand. It's the GOEs and the inconsistencies around it and the anonymity of the judging that is the problem.

 

[Leonardo]

So we would have:

Lutz
Flatz (!)
Flutz (e)

I just want to strip the tech panel of as much power as possible. The judges should be able to counter a bull headed tech panel or have the freedom to agree with them.

agreed!

 

[aa456]

The sum of the parts doesn't always equal the whole and that is what IJS is not able to capture. Additionally, all the crazy requirements to get the highest level on step sequences have led to some very mediocre and labored steps. Frankly unless it's someone like takahashi or asada, it's really boring to watch never ending step sequences for me.
The free skate isn't "free" anymore which is sad because most programs look the same now or at the very least have the same layout.

 

[Sam L]

The sum of the parts doesn't always equal the whole and that is what IJS is not able to capture. Additionally, all the crazy requirements to get the highest level on step sequences have led to some very mediocre and labored steps. Frankly unless it's someone like takahashi or asada, it's really boring to watch never ending step sequences for me.
The free skate isn't "free" anymore which is sad because most programs look the same now or at the very least have the same layout.

Hmm... I agree actually. It's really a Short Program and Long Program these days. Like literally.

 

[NanaPat]

I don't miss the old judging system at all. Yes, it has the advantage of being simple, and knowing immediately that a 5.8 is a pretty good mark, 5.9 is excellent, 5.1 stinks (at a senior international; at a local competition it might be outstanding).

But I have two major problems with it. First of all, the numbers are too close together. So the three top skaters might all have 5.7s and 5.8's. Suppose there are 7 judges. Skater A might have 6 5.8 and 1 5.7, skater B 4 5.8 and 3 5.7, skater C 3 5.8 and 4 5.7. That means that some judges had ties, i.e. two (or even 3) skaters the same mark. Skater B "beats" skater C based on one little mark by a single judge.

When you add ordinals into it, it gets even messier. Suppose we have the situation above, and an additional skater D who got 5.5 from all 7 judges. Once it's converted into ordinals, the differences disappear. The skaters are ranked A, B, C, and D. But the fact that B and C were almost the same, while clearly D was clearly worse, is lost. Skater C gets the same ordinal as he/she would get if he had all 5.6s, or even 1 5.6 and the rest 5.5s. That seems really unfair.

Under 6.0, there is also the problem of ordinals flip-flopping. Skater A can beat skater B initially, but the positions can be reversed after skater C comes along. That clearly is unfair and counter-intuitive. Let's hear the commentators explain that!

Ordinals also mean that it is almost impossible to move up a lot in the free skate under the 6.0 system. It's common to have have maybe 3 skaters with really good scores, 10 skaters "in the middle" with scores bunched together within a small range, and a few skaters with very bad scores. In fact, the bell curve, which describes a lot of natural distributions (test scores, height, weight, IQ, etc) says that's what will usually happen; a few scores very low or high, most of them bunched up in the middle. Under the 6.0/ordinal system, the person in the middle or at the bottom of that middle group can't improve much in the free skate, because there were so many people "above" them, when in fact they were very close to all those people. Whether skater A beats skater B shouldn't depend on whether there are only two skaters in the competition or 30, and it very much does in the 6.0/ordinals system.

 

[drivingmissdaisy]

I had the opposite experience from you. Watched the skate without knowing the score. I knew some jumps would be scrutinized, but I still expected around 74 (considering the inflation this season + she's done 78 in the past with no 3-3 and no lutz). I do understand where some people are coming from, especially regarding whether the flutz, but if Mao skates like this and winds up below Ashley/Evgenia... something just doesn't add up.

The thing is, if Mao skates well in the SP and ends up behind Ashley and Evgenia, it's because she made poor choices with jump selection. At Worlds, she absolutely should not go for the 3/3 in the SP. She has not had one credited in (I think) almost 6 years, and the SP is all about doing jumps you can do well. The lutz is also a jump that she should exclude in the SP; it isn't worth the risk of the likely edge call.

 

[Sandpiper]

I don't miss the old judging system at all. Yes, it has the advantage of being simple, and knowing immediately that a 5.8 is a pretty good mark, 5.9 is excellent, 5.1 stinks (at a senior international; at a local competition it might be outstanding).

But I have two major problems with it. First of all, the numbers are too close together. So the three top skaters might all have 5.7s and 5.8's. Suppose there are 7 judges. Skater A might have 6 5.8 and 1 5.7, skater B 4 5.8 and 3 5.7, skater C 3 5.8 and 4 5.7. That means that some judges had ties, i.e. two (or even 3) skaters the same mark. Skater B "beats" skater C based on one little mark by a single judge.

When you add ordinals into it, it gets even messier. Suppose we have the situation above, and an additional skater D who got 5.5 from all 7 judges. Once it's converted into ordinals, the differences disappear. The skaters are ranked A, B, C, and D. But the fact that B and C were almost the same, while clearly D was clearly worse, is lost. Skater C gets the same ordinal as he/she would get if he had all 5.6s, or even 1 5.6 and the rest 5.5s. That seems really unfair.

Under 6.0, there is also the problem of ordinals flip-flopping. Skater A can beat skater B initially, but the positions can be reversed after skater C comes along. That clearly is unfair and counter-intuitive. Let's hear the commentators explain that!

Ordinals also mean that it is almost impossible to move up a lot in the free skate under the 6.0 system. It's common to have have maybe 3 skaters with really good scores, 10 skaters "in the middle" with scores bunched together within a small range, and a few skaters with very bad scores. In fact, the bell curve, which describes a lot of natural distributions (test scores, height, weight, IQ, etc) says that's what will usually happen; a few scores very low or high, most of them bunched up in the middle. Under the 6.0/ordinal system, the person in the middle or at the bottom of that middle group can't improve much in the free skate, because there were so many people "above" them, when in fact they were very close to all those people. Whether skater A beats skater B shouldn't depend on whether there are only two skaters in the competition or 30, and it very much does in the 6.0/ordinals system.

I don't think judges were allowed to give the exact same marks to two different skaters. A judge could give 5.6/5.7 to Skater A, and 5.7/5.6 to Skater B... but that is not, functionally, a tie. The first mark is the tiebreaker in the SP, the second mark in the LP. Judges know this, and they know the scores are simply placeholders for ranking. Let's say this was an LP--that means the judge chooses Skater A over Skater B, and their mark isn't "meaningless" just because it appears to be a tie.

And anyway, close results happen under COP too (Cup of China: K/S's win over S/H, Elena's bronze over Anna). If anything, COP opens up the possibility of a dedicated minority being able to overrule a majority, which is impossible under 6.0 (as Mathman has explained many times).

I don't see the need to reflect "B and C were almost the same, while clearly D was clearly worse". As long as placements are correct, why do we need to go, "Haha, Skater D, you scored a full 40 points below everyone else"?

IIRC, the rules were changed in the '90s to prevent ordinal flops. I think it was called "one beats one" system? So if the majority of judges chose Skater A over Skater B, that placement would stay that way, no matter what scores other skaters received. For the life of me I can't remember what year or event caused this change (though I have some vague memory of it involving Urmanov )

The thing is, if Mao skates well in the SP and ends up behind Ashley and Evgenia, it's because she made poor choices with jump selection. At Worlds, she absolutely should not go for the 3/3 in the SP. She has not had one credited in (I think) almost 6 years, and the SP is all about doing jumps you can do well. The lutz is also a jump that she should exclude in the SP; it isn't worth the risk of the likely edge call.

You're right. Mao indeed will get punished for her stubbornness and ambition. My point is, I personally applaud her for going for the most difficult layout and wish the system rewarded her more for a 3F-3Lo (even a UR one, as long as she doesn't step-out/fall) versus doing a 3Lo-2Lo and a solo flip.

 

[gkelly]

I don't think judges were allowed to give the exact same marks to two different skaters.

They weren't supposed to. Occasionally it happened by accident, e.g., in the middle of a large field.

A judge could give 5.6/5.7 to Skater A, and 5.7/5.6 to Skater B... but that is not, functionally, a tie. The first mark is the tiebreaker in the SP, the second mark in the LP. Judges know this, and they know the scores are simply placeholders for ranking. Let's say this was an LP--that means the judge chooses Skater A over Skater B, and their mark isn't "meaningless" just because it appears to be a tie.

Yes.

I don't see the need to reflect "B and C were almost the same, while clearly D was clearly worse". As long as placements are correct, why do we need to go, "Haha, Skater D, you scored a full 40 points below everyone else"?

There's an argument to be made that margin of victory should matter in combining the results of two or more phases of competition.
Others will argue that it shouldn't.

Under 6.0, the margin was defined as the number of other skaters who placed between any two skaters being compared. Under IJS, it's defined by total points reflecting the specific scoring of the skating. If (and, I grant you, only if) we believe that the size of the point differences faithfully represents legitimate differences in the quality of the performances, I think it's fairer to preserve that information.

Also it gives the skater information about how the judges and tech panel rated their skating, directly, whether or not there were any other skaters in the event close to their level, or for that matter at all.

IIRC, the rules were changed in the '90s to prevent ordinal flops. I think it was called "one beats one" system? So if the majority of judges chose Skater A over Skater B, that placement would stay that way, no matter what scores other skaters received. For the life of me I can't remember what year or event caused this change (though I have some vague memory of it involving Urmanov )

Yes, OBO (one by one), initiated after the multiple flipflops at 1997 Europeans, which Urmanov ended up winning when all the results shook out.
Details can be found here: http://www.frogsonice.com/skateweb/obo/

It didn't completely eliminated flipflops but made them less likely than with majority calculations. (Which are still used in the US for certain types of events, e.g., lower level freestyle)

 

[NanaPat]

There's an argument to be made that margin of victory should matter in combining the results of two or more phases of competition.
Others will argue that it shouldn't.

Under 6.0, the margin was defined as the number of other skaters who placed between any two skaters being compared. Under IJS, it's defined by total points reflecting the specific scoring of the skating. If (and, I grant you, only if) we believe that the size of the point differences faithfully represents legitimate differences in the quality of the performances, I think it's fairer to preserve that information.

That's exactly what I was getting at. Number of skaters between is much more arbitrary than number of points. The number of points may not be absolutely correct and exact, depending on how you feel about the rules applied in compiling the points, but it is fairer than number of skaters.

Under the ordinal system, a skater who did badly in the short program was toast, even though the long program is supposed to be worth more than the short. The long program is much more meaningful under the new system.

 

[Sam L]

That's exactly what I was getting at. Number of skaters between is much more arbitrary than number of points. The number of points may not be absolutely correct and exact, depending on how you feel about the rules applied in compiling the points, but it is fairer than number of skaters.

Under the ordinal system, a skater who did badly in the short program was toast, even though the long program is supposed to be worth more than the short. The long program is much more meaningful under the new system.

But on the flip side of the coin, you could have 3 or even up to 6 or virtually every skater in the competition, theoretically, within a few points and thereby rending the short program completely irrelevant because it is all about the free skate and whoever wins it on the night. The ordinal was important because you are being judged against your competition. Figure Skating is ultimately about your competition. It's not Speed Skating where you're essentially competing against time.

 

[NanaPat]

Thank you for explaining that there aren't actually ties. However, I think the ordinals are sufficiently complicated that even though the raw numbers are easy to comprehend, what they mean isn't. You have to go "well, judge 3 gave A 5.7/5.8 and B 5.8/5.7, so judge 3 ranked skater A higher (or is it lower, depends on the program), then repeat for all the other judges, then take a majority of the judges. That isn't really all that easy. So even if you THINK you know the meaning of 5.8 more than you know the meaning of 82.78, it's not so simple to say who is ahead, given a typical collection of 5.7's and 5.8s. At least I know 82.78 is bigger than 80.23, even if they don't have any intrinsic meaning. And I know that they are very close to each other, and if it's a short program score, I know that the difference is very slim (less than one jump or spin), and that the long program will be exciting. And if the scores are 90 and 80, I know that the leader has a big lead and has a good chance of winning. But there may be other contests that are close! And I can spot them on the leader board, by comparing the numbers.

 

[NanaPat]

But on the flip side of the coin, you could have 3 or even up to 6 or virtually every skater in the competition, theoretically, within a few points and thereby rending the short program completely irrelevant because it is all about the free skate and whoever wins it on the night. The ordinal was important because you are being judged against your competition. Figure Skating is ultimately about your competition. It's not Speed Skating where you're essentially competing against time.

It's absolutely true that all the short program results could be very close and it will all be decided by the free skate. And why shouldn't it be, if everyone skated about the same in the short? Of course, they didn't all skate exactly the same, some did better spins but missed or popped a jump or two, some may have skated well overall but missed an element entirely because of a fall, etc. But that's true in either system; a single number (or two) are used to represent a skate that had lots of parts and aspects.

 

[Sam L]

It's absolutely true that all the short program results could be very close and it will all be decided by the free skate. And why shouldn't it be, if everyone skated about the same in the short? Of course, they didn't all skate exactly the same, some did better spins but missed or popped a jump or two, some may have skated well overall but missed an element entirely because of a fall, etc. But that's true in either system; a single number (or two) are used to represent a skate that had lots of parts and aspects.

I see the merit in the new system, believe me. But I just feel that Figure Skating lost a little something special and unique about itself when it got rid of the 6.0 system. But then, I started following the sport under that system.

I feel like the IJS is too much like a race. It's a race to get the most points you can and I just don't think that's what Figure Skating is about.

 

[karne]

But on the flip side of the coin, you could have 3 or even up to 6 or virtually every skater in the competition, theoretically, within a few points and thereby rending the short program completely irrelevant because it is all about the free skate and whoever wins it on the night. The ordinal was important because you are being judged against your competition. Figure Skating is ultimately about your competition. It's not Speed Skating where you're essentially competing against time.

You mean like at Skate America, which allowed Brendan Kerry to slay those who didn't skate as well as he did in the long program?

But under 6.0, would a little no-name on Grand Prix from a little no-name country even be allowed to come anywhere near the Olympic bronze medallist, a skater who was fourth at Euros,, a top Japanese skater, and a French National Champion? Somehow, I doubt it. But the numbers didn't lie at Skate America.

 

[Sam L]

I wasn't thinking about anyone in particular. Just thinking about how the systems worked. That's a judging flaw of the sport itself. Not the 6.0 system. Nothing in the 6.0 system said that you have to inherently favor reputation judging.

 

[Mathman]

I don't miss the old judging system at all. Yes, it has the advantage of being simple, and knowing immediately that a 5.8 is a pretty good mark, 5.9 is excellent, 5.1 stinks (at a senior international; at a local competition it might be outstanding).

But I have two major problems with it. First of all, the numbers are too close together. So the three top skaters might all have 5.7s and 5.8's. Suppose there are 7 judges. Skater A might have 6 5.8 and 1 5.7, skater B 4 5.8 and 3 5.7, skater C 3 5.8 and 4 5.7. That means that some judges had ties, i.e. two (or even 3) skaters the same mark. Skater B "beats" skater C based on one little mark by a single judge.

When you add ordinals into it, it gets even messier. Suppose we have the situation above, and an additional skater D who got 5.5 from all 7 judges. Once it's converted into ordinals, the differences disappear. The skaters are ranked A, B, C, and D. But the fact that B and C were almost the same, while clearly D was clearly worse, is lost. Skater C gets the same ordinal as he/she would get if he had all 5.6s, or even 1 5.6 and the rest 5.5s. That seems really unfair.

Under 6.0, there is also the problem of ordinals flip-flopping. Skater A can beat skater B initially, but the positions can be reversed after skater C comes along. That clearly is unfair and counter-intuitive. Let's hear the commentators explain that!

Ordinals also mean that it is almost impossible to move up a lot in the free skate under the 6.0 system. It's common to have have maybe 3 skaters with really good scores, 10 skaters "in the middle" with scores bunched together within a small range, and a few skaters with very bad scores. In fact, the bell curve, which describes a lot of natural distributions (test scores, height, weight, IQ, etc) says that's what will usually happen; a few scores very low or high, most of them bunched up in the middle. Under the 6.0/ordinal system, the person in the middle or at the bottom of that middle group can't improve much in the free skate, because there were so many people "above" them, when in fact they were very close to all those people. Whether skater A beats skater B shouldn't depend on whether there are only two skaters in the competition or 30, and it very much does in the 6.0/ordinals system.

This is a wonderful post. It is very interesting to me, because almost everything that you mentioned as a drawaback to the 6.0 system, to me is a plus.

First of all, the numbers are too close together. So the three top skaters might all have 5.7s and 5.8's. Suppose there are 7 judges. Skater A might have 6 5.8 and 1 5.7, skater B 4 5.8 and 3 5.7, skater C 3 5.8 and 4 5.7. That means that some judges had ties, i.e. two (or even 3) skaters the same mark. Skater B "beats" skater C based on one little mark by a single judge.

In any judging system, there will always be really, really close contests. Under IJS a skater might win by a score of 182.39 to 182.32. One mark by one judge might have made the difference. This, in fact, is not so rare as one might expect.

When you add ordinals into it, it gets even messier.

That is, it gets more mathematically interesting.

Suppose we have the situation above, and an additional skater D who got 5.5 from all 7 judges. Once it's converted into ordinals, the differences disappear. The skaters are ranked A, B, C, and D. But the fact that B and C were almost the same, while clearly D was clearly worse, is lost. Skater C gets the same ordinal as he/she would get if he had all 5.6s, or even 1 5.6 and the rest 5.5s. That seems really unfair.

I don't see what's unfair. The skaters mount the podium. Skater A gets the gold medal, B gets the silver, C gets the bronze, and D gets the pewter. Who has been treated unfairly?

Ordinals also mean that it is almost impossible to move up a lot in the free skate under the 6.0 system.

I could go either way on that. Yes, under 6.0 you had to skate two good programs to win. You couldn't mess up one and hope to make it up on the other.

The other side of the coin is that under IJS a skater can get such a humongous lead that there is no point in holding the LP at all. It's not much fun to see the second place skater give the free skate of her life and totally outskate the frontrunner, but have no chance to win because she started too far behind, pointwise.

Under 6.0, there is also the problem of ordinals flip-flopping. Skater A can beat skater B initially, but the positions can be reversed after skater C comes along. That clearly is unfair and counter-intuitive. Let's hear the commentators explain that!

I am not a commentator , but "flip-flops" have been unfairly picked on. :yes: All a "flip-flop" means is that Skater A counted her chickens before they were hatched. The illusion of unfairness arises only because of the premature announcement that "Skater A is leading Skater B so far." There is no "so far" in ordinal judging because later skaters can steal ordinals from the frontrunners.

It is like an election where there is a third party candidate. Candidate A beats Candidate B head to head, but then the third party candidate comes along and steels some votes from A, giving B the victory.

 

[Mathman]

Yes, OBO (one by one), initiated after the multiple flipflops at 1997 Europeans, which Urmanov ended up winning when all the results shook out.

Details can be found here: http://www.frogsonice.com/skateweb/obo/

The strange thing to me, in reading about this, is that the "flip-flops" that people were in such an uproar about in that competition were almost entirely due to factored placements, rather than actual violations of the technical condition of "independence of irrelevant alternatives" (as game theorists call it).

That everything turned upside down and Urmanov came out on top was simply the result of the fact that all the skaters who did well in the short program did badly in the long program, and vice versa. So obviously the leaders after the short were no longer leading after the long.

The "flip-floppiness" of reporting in real time came about because they started out with a preliminary ranking by short program placements, then every time another competitor skated he shook up the list (either by being low ranked from the SP but beating a lot of skaters ahead of him in the LP, or else vice versa).

IMHO all the befuddled commentators needed to say was, "Well folks, hold on to your hats. We are in for a big ranking roller coaster this evening, if the trend continues of the leaders in the SP falling apart in the long."