Evaluating judging trends: A study on GPF

[Mrs. P]

Is there an easy source to look up protocols from events that took place from 2000-2010? I'd be curios to look at some videos of programs and compare them to today's standards. Looking at things like difficulty and complexity and weigh them out against the scores given back that and what we see have become today's standards. It's always been my opinion that scores were just too low back then as opposed to being too high now.

10's in components should be something achievable just like 6.0's were. It really shouldn't be something people imagine to be an impossible feat.

2004-2015 ISU Championships can be found using
isuresults.com/results/YYXXXX

** YY being the comp = WC Worlds EC Euros FC 4cc
*XXXX being the year.

GPF can be found with isuresults.com/results/gpfSSSS
SSSS== curent season year (so 2004 GPF is gpf0405)


web adress format for GP events prior to 2010 ==
http://www.isuresults.com/results/gpusa03/

other events chang country code (can/rus/fra/jpn/chn) and year.

 

[solani]

I think the ISU wanted a kind of balance between the TES and the PCS, so that the TES cannot outweigh the PCS and vice versa. So it seems natural that if the TES rises overall (as it happens now), the PCS has to rise as well.
This whole world record thing that the ISU has installed is a joke, no one should take that seriously anyway, considering all the rule changes that happened. Records that were achieved under different rules are not comparable. (The weightlifting union changed the weight categories a couple of times, and therefore reset the world records). The ISU should reset the world records if they change the rules. This would have the advantage that the audience and the coaches (and maybe also Hanyu ) all love records and if a world record isn't broken after, lets say, 5 years, the ISU can make a rule change and we get a bunch of new world record holders in a very short time and everyone is excited and happy.

 

[qwertyskates]

The first 3-3 ever in pairs? And girls are super strong these days. There are plenty super talented skaters out there

I'm ok-ish with the clean-ish gold medal winners, but even the very subpar singles and pairs got gifted. They received higher scores than when they skated better at other GP events.

I might sound like Scrooge, but this is rather unfair to the skaters who couldn't make it to GPF, especially Tut, Max, and even Sui/Han, etc., especially if scoring falls back down to earth.

I'm not even comparing to 5-10 years ago, heck just look at Adelina's 74+ PCS in 2014 when she skated lights out and received death threats for them.

It seems if the skater is popular anything goes. High TES wasn't ok if Jin scored them honestly, without e or ur, with gorgeous difficult jumps and inching close to 100, but not a beep when Hanyu broke all TES records doing the same. It boggles the mind...

This contradicts the spirit of sports, where fairness and a level playing field is essential.

 

[Sam-Skwantch]

2004-2015 ISU Championships can be found using
isuresults.com/results/YYXXXX

** YY being the comp = WC Worlds EC Euros FC 4cc
*XXXX being the year.

GPF can be found with same doman using gpfSSSS SSSS== curent season year (so 2004 GPF is gpf0405)

Wait....have you told me this before. I'm having déjà vu

 

[HanDomi]

It seems if the skater is popular anything goes. High TES wasn't ok if Jin scored them honestly, without e or ur, with gorgeous difficult jumps and inching close to 100, but not a beep when Hanyu broke all TES records doing the same. It boggles the mind...

.

Who had problems with Jin TES ? Jin has great 4lz, but on all other jumps Hanyu quality beats him tremendously + Hanyu is doing his quads and other jumps in and out with transitions

 

[mrrice]

Am I alone to feel that scores are actually fine here?
Evgenia Medvedeva and Hanyu are properly awarded, especially for Evgenia! In GPF, both programs were very well executed. Her jumps are pristine, her footsteps are crystal clear, her spins are L4 and there are transitions in between. I am quite sure, she will break Yuna's LP world record soon.

You are definitely not wrong in thinking the scores were fine. Now, I can't speak for the ISU but, in dance we are told to "Select a winner" and to make it clear who you thought was the best team at the event. The further into the season we'd get, the more clear it would become who the top teams would be by Nationals. I'm sure the Judges were told during their pre-event meeting. This is the final event of the GP and now is the time to send a message to the skaters and their coaches where you think they'll end up at Worlds.

IMO, that's what happened with the Men's and Ladies event's.

 

[padme21]

Hanyu's program is brilliant - choreographed by Shae-Lynn Bourne - who else? It's perfect for him.

I personally do not think Yuzuru, Javier, or even Shoma got overscored this season. Why? Because they have quality. In varying degrees of course but to take Yuzu as an example- his jumps were flawless and effortless, and it never once looked like he was setting up a jump and landing it. They were seamlessly incorporporated in the program. On top of that, his components are just a cut above the rest. You see that even without skater eyes when you watch all the men side by side.

Who I did think was waaay overscored was Medvedeva. I am not sorry to say that her free program is NOWHERE near the calibre of Mao's Sochi FS, Kim's record breaking FS from Vancouver- hell, it wasn't even half as good as Carolina's Bolero. WHERE the hell did those judges get 72 on components? THAT is outrageous. Satoko had the best components that evening. I just had to keep rolling my eyes whenever the Eurosport guys essentially lose their minds over how good Evgenia was. SHe's good. But not that good.

I agree with the first part! I didn't watch the women so I can say if I thought they were over scored or not.

 

[gkelly]

A sport can never be fair when it's judged subjectively

That being the case, and accepting that many aspects of figure skating can only be judged subjectively, it would follow logically that figure skating can never be fair.

But can it be "as fair as possible," given the subjective nature of some components and the qualitative nature of the other components and GOEs? And how can "as fair as possible" be determined?

one thing I'm really curious about is whether there has been a shift in how the judges approach PCS scoring. I think in the early days of COP, I think judges were really reluctant to give high PCS scores for whatever reason. A 70+ PCS score was a really big deal. Heck even a high 60s PCS was huge too. I can't help but wonder if, as some people pointed out in the competition threads, that the judging of PCS is shifting from hey --reserve this just for a "special performance/skate" and more like "if this skater meets the given criteria, it is reasonable to score this way."

I do think there is a trend and probably there have been communications from the technical committees to the referees to the judges in exactly that direction.

Also both the skaters/coaches and the judges are getting the hang of how to maximize component skills and scores more than was the case 10-12 years ago, so even if the PCS scale were 100% stable and invariable across individuals, judging panels, and years, more skaters actually deserve scores in the 9s than was the case earlier in the history of IJS.

But it is a problem if skaters are earning 9.75s and 10s in all components while there is still room for even further improvement by that skater or by the sport in general.

I'm sure the Judges were told during their pre-event meeting. This is the final event of the GP and now is the time to send a message to the skaters and their coaches where you think they'll end up at Worlds.

I would hope that is not what the judges were told. They're supposed to judge each event on its own merits, NOT to establish a ranking ahead of time.

To the extent that judges tended to default toward protocol judging under the 6.0 system, much of what IJS was designed to do was to try to prevent that approach.

Strictly speaking, with IJS scoring judges no longer supposed to be ranking skaters at all. They're supposed to be evaluating each element and each component on its own merits and not worrying about how the total scores will add up.

Nor do they have all the information (e.g., levels assigned to non-jump elements) they would need to figure out the likely totals for themselves. And I doubt many if any judges have memorized the Scale of Values including positive and negative GOE values for each element, and the PCS factors especially for events other than junior/senior men where they're nice and easy, or have the time or the mathematical skill to do all those calculations in their heads while scoring an event.

At best, they can decide that skater B was slightly or significantly better than skater A in all areas and make sure to give slightly or significantly higher scores, accordingly.

 

[CanadianSkaterGuy]

For those who say Medvedeva is overscored (and yes, she slightly was) on PCS... I think people are holding a skater like her to a higher standard simply because she's just out of the junior ranks. Yes, she is out of the junior ranks, but she has good speed, quality elements, huge difficulty (two triple-triples), many transitions in her program, and she performed it well.

Compared to Uno, who got 89.58% PCS for his clean skate, Medvedeva scored 90.46% PCS for her clean FS -- and nobody is really complaining that Uno's PCS score was so high after going clean.

It's honestly a lame argument at this point to say "They're just out of juniors... how can they be scoring higher than [insert senior skater]?" To me, if they have all the requirements and are executing things well, it's unfair to slap them with a PCS threshold, just because they're not established yet. That being said, the judging still needs to be fair and reflect the level and quality of performance.

BTW, I would say Medvedeva's performance was definitely on the calibre of Mao's Sochi FS (which was quite underscored on PCS), and Kim's OWG2010 FS from a technical standpoint -- 7 triples, two 3-3 combos (Kim did one triple-triple, a 2A+3T and only 6 triples; Mao's 3F+3L and 2A+3T were UR but she did have a 3A)... and 5 jumping passes in the 2nd half, compared to Kim (3) and Asada (4).

No matter how you cut it, Medvedeva delivered one of the best senior SP+FS of all time -- oh but, she just came out of juniors so let's trivialize her performances for it. If you had swapped her face with Lipnitskaia's (who's established), people would have been saying her PCS was underscored.

 

[YesWay]

I hope somebody in the skating media or academia explores this topic because I think it could shed light on why the scoring has trended upward in recent years.

I'm sure there are many factors involved... but I think one part of it, is that skaters and coaches have become "smarter" at playing the IJS/CoP system...

They learned from the skaters who scored the highest over the years etc... and incorporated what they learned into the training and the competition programs for each newer generation. They know better, how to maximise the points for a given skaters skills and abilities.

I'd say the top competition programs these days... have become visibly more difficult, intricate and crammed with elements, than the top programs of years gone by. ie. it's not just arbitrary "inflation" even if that is indeed a factor... because the top skaters have been getting better over the years.

 

[mrrice]

gkelly, I wish I could tell you that you're correct but, that's not how judging associations work. They are trained by the same people and I can promise that if the scores were to completely flip from event to event. You'd see some judges out of a job.

Can you imagine what some of those judges would have to hear if Gracie won an event and then had the same skate and came in third at her next event? Scores normally increase as the season goes on and it takes a bad skate for a skater to get a lower than their previous event.

Lastly, Frank has a lot of power in the world of figure skating. He's like Mishin, you don't give his skaters bad scores unless you can completely justify doing so, or you're fired. Sadly, I have fired Judges for not following the rules in our guidelines.

 

[bara1968]

CanadianSkaterGuy, well, you are talking about TES to justify Med's high PCS. 7-triple argument is weak to build your case, as Yuna Kim did perform 7-triple program and Mao Asada did so with two 3-3s, and 3A back in 2007-2008, yet their PCS were barely 60 when they were clean (or clean-ish). The recent and more comparable example may be Zijun Li at 2013 Worlds, who was of similar age as Med at the time; she had a clean, lovely long program with 7 triples (except the edge call on 3Lz but Med is also a flutzer) yet her PCS was barely 58+. Of course, PCS has been widely inflated over time but even so, I personally do not think the level of skating Med showed was not there yet, to earn 72+ PCS. (IMHO, I personally do think that Med's LP here was not that much better than that of Zijun Li back in 2013 Wolrds.) For instance, her ice coverage, speed, edge control are nowhere near Caro K or Yuna Kim at Sochi yet she was getting that much SS at GPF. I think it is too much of inflation to bear, at least for me, and many others in this forum.

 

[Ophelia]

Sanity check: Evegnia may likely break the WR for the LP. Is her performance WR, 150 point worthy? Um...no (and this is coming from an Evegnia fan).

I remember when scoring in the 140s meant something even 2 years ago. Now upper 140s are getting handed out like candy.

 

[CanadianSkaterGuy]

CanadianSkaterGuy, well, you are talking about TES to justify Med's high PCS. 7-triple argument is weak to build your case, as Yuna Kim did perform 7-triple program and Mao Asada did so with two 3-3s, and 3A back in 2007-2008, yet their PCS were barely 60 when they were clean (or clean-ish). Of course, PCS has been widely inflated over time but even so, I personally do not think the level of skating Med showed was not there yet, to earn 72+ PCS. For instance, her ice coverage, speed, edge control are nowhere near Caro K or Yuna Kim at Sochi yet she was getting that much SS at GPF. I think it is too much of inflation.

I was referring to the comparison of Medvedeva's skate to specifically Yu Na's 2010 Olympics FS (which had 6 triples and one clean 3-3) and Mao's 2014 Olympics FS (which had 6 ratified triples, although one was a 3A, and no clean 3-3 (although the tech spec was a bit harsh on that one). And I did say that Medvedeva did get overscored on her PCS (would have given her about a 70). I totally think Caro/Yuna/Mao in Sochi thoroughly beat Medvedeva at the GPF out on SS/CH/IN/PE... although Medvedeva does have a program packed with transitions, including some very difficult ones into her jumps and is almost on par with them, IMO.

I would not necessarily use the marks that Caro and Kim got in Sochi as examples, because that was blatant PCS reduction for those two (and inflation for Sotnikova) -- which is annoying because that's the best Kostner ever skated, and one of the best ever performances by Kim, on the biggest stage of them all (but let's not get into THAT).

And we can't compare PCS scoring in 2007-2008 to today, because judges were reserved in how high they allowed PCS marks to get. Kim's PCS would have been around 75/76 (and Mao/Joannie in the mid-high 60's) for her Vancouver skate if PCS were scored back then the way they are now. Only 3 skaters scored above 63+ PCS in the Vancouver FS... in Sochi, 8 skaters got 63+.

 

[gkelly]

gkelly, I wish I could tell you that you're correct but, that's not how judging associations work. They are trained by the same people and I can promise that if the scores were to completely flip from event to event. You'd see some judges out of a job.

International judges are nominated by their national federations but must receive training by the ISU to get international appointments.

If a judge is consistently out of line with the other judges and showing incompetence or national bias, they can be sanctioned and suspended/demoted by the ISU. It doesn't happen very often -- not often enough, in the opinion of many observers who believe the ISU lets judges get away with corruption.

It's the national federations who decide which of their judges to send to an event where they can send judges. Federations may aim to send their best, most fair judges, or they may aim to send the judges who will be best at acting on national bias to help their home skaters without getting caught. We've heard judges complain that they had to go along with pressure from their federations to help their skaters because otherwise they wouldn't get selected for international assignments by their federations.
They are all volunteers and cannot be easily fired. Were your dance judges paid employees? That's a big difference right there.

The ISU has less of a vested interest in specific skaters winning than the individual federations do. So the ISU would "fire" judges (i.e., not assign them to events, under the current volunteer setup) if the judges were doing a bad job of reflecting that day's performances, whereas the corrupt federations would stop assigning judges who failed to hold up their countries' skaters, or those of other countries they'd made deals with, regardless of how the skaters performed on that day.

People have often argued that judges should be hired or at least appointed by the ISU, not by their federations, to reduce the effects of national bias.

Can you imagine what some of those judges would have to hear if Gracie won an event and then had the same skate and came in third at her next event?

But how did the other skaters perform? Were they the same other skaters? It's entirely normal for a skater to skate the same at two events and have different results because the rest of the field was different.

Skaters are much less consistent in how they perform than dancers (and freestyle skaters are less consistent than ice dancers), because of all the high-risk moves especially jumps. So it's very rare for all skaters to perform exactly the same from one event to the next. It is to be expected that there will be some movement in the standings from one event to another. The bigger problem people have perceived with skating judging is that skaters skate differently from one event to another and still receive very similar PCS -- that just because skater P is better than skater Q in general, if P makes more visible errors on a given occasion, her PCS should suffer compared to cleaner performances or compared to a clean performance by Q.

We wouldn't expect huge differences in PCS for the same program performed with pretty much the same jump success, but some variations are expected. E.g., see this training video on scoring the Interpretation component that gives a specific example of the same skater skating the same program "on" and "off" the music on different occasions and advising that the Interpretation scores should be different for those two performances.

 

[bara1968]

To CanadianSkaterGuy : (I think you repliled while I was fixing my original post to include Zijun's PCS but) Okay, then it seems that we are pretty much on the same page. Thanks for more thorough explanation.

 

[koatcue]

CanadianSkaterGuy, well, you are talking about TES to justify Med's high PCS. 7-triple argument is weak to build your case, as Yuna Kim did perform 7-triple program and Mao Asada did so with two 3-3s, and 3A back in 2007-2008, yet their PCS were barely 60 when they were clean (or clean-ish). The recent and more comparable example may be Zijun Li at 2013 Worlds, who was of similar age as Med at the time; she had a clean, lovely long program with 7 triples (except the edge call on 3Lz but Med is also a flutzer) yet her PCS was barely 58+. Of course, PCS has been widely inflated over time but even so, I personally do not think the level of skating Med showed was not there yet, to earn 72+ PCS. (IMHO, I personally do not think that Med's LP here was not that much better than that of Zijun Li back in 2013 Wolrds.) For instance, her ice coverage, speed, edge control are nowhere near Caro K or Yuna Kim at Sochi yet she was getting that much SS at GPF. I think it is too much of inflation to bear, at least for me, and many others in this forum.

You really say that Zhenia's blade control isn't on par with the best of the best, don't you? Hm, her edges are great. The problem is that she IS young AND super talented, as many other young girls nowadays. It might turn out that Yuna isn't the only one great skater in post 6.0 system. And to compare Zijun and Medvedeva. Just, really? They are on different ends of everything! Tes and Pcs wise.

 

[bara1968]

^ ^ Well, I am comparing Med to Zijun at her best, am I not? And that was a very nice performance too - Her jumps were light and effortless, choreography was lovely, very nice interpretation "at one particular competition". Yet she scored 58+ PCS. I understand that Zijun did not have a program packed with transitions as Med has, but I would not score her PCS 14 points less than that of Med either for example. Well, if you persist on edge control- then how about ice coverage and speed across the ice for example? Of course Med is of HUGE talent. What I am saying is I do not think she deserved +9 SS YET. (along with other PCS category but I would not go there as it seems that we will simply disagree there. )

 

[CanadianSkaterGuy]

To CanadianSkaterGuy : (I think you repliled while I was fixing my original post to include Zijun's PCS but) Okay, then it seems that we are pretty much on the same page. Thanks for more thorough explanation.

I think the "problem" with Li though is that she was basically unheard of, and wasn't so it was unexpected what to do with her. She was also 12th after the SP, and that placement and being not in the final flight meant lower PCS. Medvedeva on the other hand, was leading after the SP of the Grand Prix Final, and was clean with incredible difficulty (moreso than Li).

Had she been a World Junior champion, Russian bronze medalist, and GP winner and silver medalist, like Medvedeva, and had it been 2 seasons later (when scoring was higher), she probably would have deserved mid-60s. (Although, I think Medvedeva's performance at 2015 GPF was far superior to Li's, even with how great Li skated.)

 

[koatcue]

^ ^ Well, I am comparing Med to Zijun at her best, am I not? And that was a very nice performance too - Her jumps were light and effortless, choreography was lovely, very nice interpretation "at one particular competition". Yet she scored 58+ PCS. I understand that Zijun did not have a program packed with transitions as Med has, but I would not score her PCS 14 points less than that of Med either for example. Well, if you persist on edge control- then how about ice coverage and speed across the ice for example? Of course Med is of HUGE talent. What I am saying is I do not think she deserved +9 SS YET. (along with other PCS category but I would not go there as it seems that we will simply disagree there. )

The thing is that she was judged according that particular field. Would it be ok if she received the same PCS or TES as Satoko?...Everybody got inflation there. I understand what you are saying, but even if Med was overscored, she really isn't that far away from our well-known leaders of everything If we talk about PCS just, because her TES should be bigger than theirs. Aaand to reply to your latest Li sentiment - why don't you compare Zijun with Mao or Yuna? If her presentation was good that night and so on and so forth. It's easy to manipulate words.