PCS per judge - Data Analysis AWG Mens FS

[ice coverage]

Wasn't no more anonymous judging introduced this season? http://www.insidethegames.biz/artic...em-in-figure-skating-to-increase-transparency Judges names were listed at 2016 Europeans, but in detailed protocols says 'in random order' http://www.isuresults.com/results/season1516/ec2016/ec2016_Ladies_FS_Scores.pdf. However this year not http://www.isuresults.com/results/season1617/ec2017/ec2017_Ladies_FS_Scores.pdf. Could just be a 'cock up' of course, not updating how it should be, but of course they should have got it right, or at least said ISU and their country.

Judging is no longer anonymous this year.
The 2017 Euros protocols give results by specific judge -- whose identities are published for anyone to see.
And the 2016-17 ISU document at my link above (from the ISU website) lists each federation's "ISU judges" (as well as "international judges").

Everything is out in the open.
I don't know what you are complaining about.

 

[cathlen]

Thanks for the analysis, Geek on Ice! It's good to see you back You are missed in Takahito FanFest :hap85:

I'm kind of sad for skaters from smaller Feds. In ISU Championships there are usually judges in all disciplines from big Feds, so when one judge is underscoring, the other will overscore and it's more or less averaged. It actually kind of worked out quite well in non ISU comp, like Four Nationals Championships. In all 3 disciplines, there was judge for all 4 countries (Czech Republic, Slovakia, Poland and Hungary), there wasn't even inflation much, because it was nicely levelled, by the fact that no country had majority. But in any comp skater which Federation is not represented by judge, will probably be underscored by some but there is no judge to overscore, so ends up being lowballed... Unless there is already reputation score. Then country bias has less significance.

 

[Rissa]

Thanks for the analysis, Geek on Ice! It's good to see you back You are missed in Takahito FanFest :hap85:

I'm kind of sad for skaters from smaller Feds. In ISU Championships there are usually judges in all disciplines from big Feds, so when one judge is underscoring, the other will overscore and it's more or less averaged. It actually kind of worked out quite well in non ISU comp, like Four Nationals Championships. In all 3 disciplines, there was judge for all 4 countries (Czech Republic, Slovakia, Poland and Hungary), there wasn't even inflation much, because it was nicely levelled, by the fact that no country had majority. But in any comp skater which Federation is not represented by judge, will probably be underscored by some but there is no judge to overscore, so ends up being lowballed... Unless there is already reputation score. Then country bias has less significance.

I've got a solution to that: judging panel shouldn't contain a judge from a country who has a skater/team with, let's say, top 10 season's best score. It'd be unlikely judges would have their own candidate for the top placements and thrperefore wouldn't drastically over or underscore others. Big federations get depowered, small ones empowered. Of course, it also means no judges from USA, Canada, Russia at all, Japan only judges Pairs and Dance and China only judges Ladies and Dance.

Can you imagine the upheaval?

 

[Miller]

Judging is no longer anonymous this year.
The 2017 Euros protocols give results by specific judge -- whose identities are published for anyone to see.
And the 2016-17 ISU document at my link above (from the ISU website) lists each federation's "ISU judges" (as well as "international judges").

Everything is out in the open.
I don't know what you are complaining about.

I'm saying that wouldn't it be far better for the judge's country to be on the protocol, rather than having to know about ISU communication 2027 on the 6th October 2016 to be able to translate their name to a particular country. It'd be a lot easier and far more transparent, especially in a world of no more anonymous judging.

 

[ice coverage]

I'm saying that wouldn't it be far better for the judge's country to be on the protocol, rather than having to know about ISU communication 2027 on the 6th October 2016 to be able to translate their name to a particular country. It'd be a lot easier and far more transparent, especially in a world of no more anonymous judging.

I think the point of the view of the ISU (which I think is understandable) prob. is that it should not be automatically assumed that any judge's federation would matter.

I don't blame the ISU for not enabling those who are inclined to assume that (for example) a judge from France will be biased in favor of skaters from France, or biased against skaters from the U.S. or anywhere else.

The need to click on one link doesn't make things any less transparent.
The need to click on one link makes things a tiny bit less convenient for those who are inclined to make such assumptions. That's their "problem" (micro-"problem"), not the ISU's. IMO.

 

[Sugiady]

I've got a solution to that: judging panel shouldn't contain a judge from a country who has a skater/team with, let's say, top 10 season's best score. It'd be unlikely judges would have their own candidate for the top placements and thrperefore wouldn't drastically over or underscore others. Big federations get depowered, small ones empowered. Of course, it also means no judges from USA, Canada, Russia at all, Japan only judges Pairs and Dance and China only judges Ladies and Dance.

Can you imagine the upheaval?

It will be another mess battle.
Think about it, if Australia judge gives lower scores to Japanese men's single skater (and there's no Japanese judges in this panel), what will happen to Australia skaters when they meet Japanese judges？

 

[gkelly]

For example, why can't someone have SS 9.50 and have 6.00 in TR? That would be the true assessment, IMO.

The true assessment of what? Did you have a particular program in mind?

I'm trying to imagine what a program that deserves SS 9.50 (outstanding) and TR 6.00 ("above average" for skating as a whole, below average among elite competitors) would look like.

If the skater truly deserves such a high SS score, he must have a lot of difficult, high-quality skating throughout the program, including multidirectional and one-foot skating -- probably a number of choctaws, counters, and rockers, and also brackets and maybe outside mohawks. They may almost all be in the step sequences where they wouldn't help the transitions score, but I wouldn't expect a skater to earn 9.5 for Skating Skills without demonstrating some of those skills elsewhere in the program. In which case they would also count toward the Transitions score.

So it wouldn't be on quality or difficulty of the skating movements that the transitions are lacking.

Variety could be lacking, if there were practically no other categories of transitions besides skating movements. (No arm or body movements. No unlisted jumps. No highlighted glides in positions.)

Quality could be lacking elsewhere on the component if, for example, the skater did some spirals, spread eagle, etc., and the edges were strong but the positions sucked.

Continuity of movements could be lacking if most of the elements were telegraphed: good difficult skating, square off and glide on a nice edge simple edge for a long time, enter a jump or spin, hold the exit on a nice simple edge for a long time, rinse and repeat.

Does that sound like a program that deserves 9.5 for skating skills, when someone else might have almost as good speed and flow and steps/turns content but also more agility to connect the turns and steps to the elements?

 

[Geek On Ice]

This is actually the first time I've ever had an avatar. I was so jealous of MaiKatze's cute Shoma avatar that I wanted one too! Next time I will probably use a photo. But for now, I like the cartoon one. I like your Mura one too! Hoping that Mura does well at Printemps! Thx again for your efforts on our behalf!

Thanks, Arriba627! It is definitely a cuuuuute avatar!! I really like that costume on Shoma.
Yeah, next stop for both Mura & Shoma will be Coupe de Printemps, eh? I'm sure both of them will be on fire again!!!!!

 

[Geek On Ice]

It must be hard work to generate these stats, thanks for doing it!

I'd be wary of drawing any conclusions using stats from just a single event, but if you can keep this going across many competitions, I think together they could become something very important. I'm very interested in any patterns that may be revealed by such stats. Because proper, systematically accumulated, "real" evidence regarding judging bias... is rather thin on the ground. --- I guess even if all the important/relevant factors were identified, accounting for them might make it all too much hard work? Obviously it would be convenient if we do indeed only need to consider top 6 in order to draw some "meaningful" conclusions - but is this the case? I'd just like some reassurance?

Thank you for your insightful comment, YesWay!
TBH, I actually don't know how I really feel about the country bias and I was not planning to draw any 'conclusions', just wanted to hear what you guys thought about 'country bias', because as you said (& I wrote in the original post) this is just a quick sample I did.
I agree that this partial sample data alone should NOT be used in any way to come up any solid conclusions. You're totally right, we need much more data. Also, even if we gather up millions of data, it's rather 'correlation', may not be 'solid' evidence.

Meanwhile though... I guess we're free to discuss about country bias and so many other interesting things you pointed out, as long as we don't rush to make any incomplete/inappropriate conclusions.

And it was my bad, that I made this thread a bit confusing (maybe 2 things in one thread...), but I guess we can still talk about the data & AWG itself too? Hmm.

 

[Geek On Ice]

Thanks for the analysis, Geek on Ice! It's good to see you back You are missed in Takahito FanFest :hap85:

I'm kind of sad for skaters from smaller Feds. In ISU Championships there are usually judges in all disciplines from big Feds, so when one judge is underscoring, the other will overscore and it's more or less averaged. ---

Awww! I missed you so much, cathlen, the President of Takahito Mura's Sexy Hair Club !!
I'll make sure to join you guys at FanFest soon! :agree2:

I totally agree with your point about the skaters from smaller Feds. I hope it does not become a vicious cycle... unless they have one huge star, like for example Javi, who will help smaller Feds to 'grow', it will be hard for those to avoid country bias issues... no?

 

[Geek On Ice]

The true assessment of what? Did you have a particular program in mind?

Hi, gkelly! I really missed your knowledgeable posts!
First of all, it's my bad, about "SS 9.50/TR 6.00", I exaggerated the numbers a bit, to make my point more obvious! :dumb:
Second, I just randomly picked SS and TR, but it's not only limited to them.
Third, I shouldn't have used the word 'true', I meant more like 'organic' or 'realistic' or 'natural'...?? Sorry my English vocab is not that great... :

I really enjoyed reading your post and it does make sense totally in those extream cases.
So it's not like, at a judge's meeting etc, they decide on a certain "max/min" for judges to stay within, rather, the similarities are the results of 5 components so closely related, right?

Still, do you think it's possible to see a gap like 1.0 though (unlike huge gap I said before)? Or is it really rare? Usually the gap is within 1.0 or so, no?
(Okay, I do remember seeing an individual judge's score, not final PCS, having a gap of 1.75 - SS 8.75/TR 7.00 from my man MURA's scores... But see, while I may not agree w/ that mark, I actually don't mind seeing this gap. It seems more 'natural' to me. )

For example, I always wondered about this program - Takahiko Kozuka at Worlds 2015 FS
https://www.youtube.com/watch?v=s2-x-tExC80

Here is the judge's scores. Kozuka is 9th.
http://www.isuresults.com/results/wc2015/wc2015_Men_FS_Scores.pdf
SS 8.00 | TR 7.54 | PE 7.54 | CH 7.64 | IN 7.64 (TSS 152.54 TES 76.82 PCS 76.72 -1.00 Starting #1)

I felt that Kozuka could have received higher SS (but I guess StSq2 may not help, eh?) and slightly lower TR because of telegraphic jump prep (I love it though). There is already 0.46 gap b/w SS/TR, but could it be more?

Also, if what you are saying is true (which probably is, as I almost always trust your knowledge! :yes, why are we having 5 different areas if they are SO closely related? Could we have one big chunk of PCS mark instead?

Maybe the wording label of 'SS - SKATING SKILLS' bothers/confuses me. Because as much as I LOVE Boyang (and I actually don't think his skating skill is that bad as others say), to see his "Skating Skills" higher than Mura/Ten etc, even if you explain to me why it is so (& I may accept it), it still psychologically doesn't sit well with me. But that's just me...
:scratch3:
Many of my questions about judging is NOT coming from me suspecting judges are not doing their jobs fairly (well, I do a little ). They probably are following rules and guidelines like you explained to me. Rather, they are questions about current 'system' iteself, I think.

 

[eppen]

Sometimes - but very, very rarely - judges have given PCS scores with large differences. In Sochi, there were judges who did not think very highly of Plyushcheko's SS or TR - in the SP he received a "realistic" 4.75 for TR from one judge (and there were a couple of scores below and around 7.00s). The same judge gave him also 6.75 for SS. The range in SS was 6.75-9.75 and in TR an incredible 4.75-9.25. In FS the SS his score ranged from 7.75 to 9.50, TR from 6.25 to 9.25. The rest of his PCS were around 8-9. But Plyushcheko was the only guy in the top group who got these kinds of scores...

IMO Plyushcheko's programs in Sochi were distinctly "old school" when it came to transitions (or rather lack of them) and I guess it is not surprising perhaps that the judges were very divided.

What Dave spoke about in the TSL 4CC scoring session of a basic score for everyone in TR kind of makes sense to me. The quantification of the transitions made last year was interesting as it also showed the differences in what the top 3 guys did compared to the others, but that these differences were fairly subtle in the end. I hope someone has the patience, skill and time to do a similar analysis for this season with the increased number of quads and how that might have changed the transitions in particular...

E

 

[Mathman]

why are we having 5 different areas if they are SO closely related? Could we have one big chunk of PCS mark instead?

IMHO I think it would be best to have two marks, SS and TR together and the other three together.. (We could call them the first mark and the second mark. )

It does seem sort of artificially nit-picky to say, well, this program had great choreography which was well executed by the skater, but he did not interpret the music worth beans.

Maybe the wording label of 'SS - SKATING SKILLS' bothers/confuses me. Because as much as I LOVE Boyang (and I actually don't think his skating skill is that bad as others say), to see his "Skating Skills" higher than Mura/Ten etc, even if you explain to me why it is so (& I may accept it), it still psychologically doesn't sit well with me. But that's just me...
:scratch3:

I, too, think that the ISU should come up with a better name for this category. To me, the whole program is about demonstrating that your skills at skating are better than the other guy's. This is, after all, a skating contest.

 

[Yatagarasu]

I hope someone has the patience, skill and time to do a similar analysis for this season with the increased number of quads and how that might have changed the transitions in particular...

Disregard the potential screaming but these are actually excellent resources
SP GPF 2016
http://chibura.tumblr.com/post/153879348150/original-post-in-russian-by-yulena
FS GPF 2016
http://chibura.tumblr.com/post/154115680890/transitions-in-mens-free-programs-at-gpf-2016

And she did this for last season too, so
SP at Boston
http://chibura.tumblr.com/post/143787559605/full-translation-of-the-whole-analysis-of-top-men
FS at Boston
http://chibura.tumblr.com/post/144143945205/full-translation-of-the-russian-article-analyses

We'll see the final one only post Helsinki but still, it's a good way to look at both.

 

[gkelly]

IMHO I think it would be best to have two marks, SS and TR together and the other three together.. (We could call them the first mark and the second mark. )

I think that would be workable -- and that judges would have to get 0.1 instead of 0.25 increments to work with again -- but I'm not sure how it would be better than the current 5 marks.

Why is it preferable to see that judge A gave Bill and Sam both 8.5 for "the first mark" and judge B gave Bill 8.4 and Sam 8.3,
as opposed to knowing that both judges thought Bill was better than Sam on Skating Skills and that Sam was equal to or better than Bill at Transitions?

It does seem sort of artificially nit-picky to say, well, this program had great choreography which was well executed by the skater, but he did not interpret the music worth beans.

Sometimes. But other times there's a big difference between the quality of the choreography and the quality of the skater's interpretation of the music. Baiul 1994 freeskate comes to mind as an example of a notable mismatch (in the other direction) between those two areas. But hey, while we're in Lillehammer, maybe Kerrigan would be a good example of choreography outshining the interpretation.

I, too, think that the ISU should come up with a better name for this category. To me, the whole program is about demonstrating that your skills at skating are better than the other guy's. This is, after all, a skating contest.

Right. But sometimes skaters rely on difficult elements and charisma to earn points even when their actual blade-to-ice skills are not so impressive. Whereas a skater with easier elements and/or less personality might have the best technical skating skills and deserve to be rewarded for them. So what else would you call that part of the score?

 

[Mathman]

Why is it preferable to see that judge A gave Bill and Sam both 8.5 for "the first mark" and judge B gave Bill 8.4 and Sam 8.3,
as opposed to knowing that both judges thought Bill was better than Sam on Skating Skills and that Sam was equal to or better than Bill at Transitions?

I think that it might forestall some of the complaints about how outrageous and despicable it is that we have all these lovely bullet points but the evil and incompetent judges are not following them. Conversely, the judges may have thought that Bill was better at "rhythmic knee action" and Sam was better at "effortless glide," but I don't think we gain anything by replacing the SS category by RK and EG.

We can always propose more components than we have now, or fewer than we have now. The reason that I favor fewer is that I have become a convert to "individual elements" versus "program as a whole" as the defining feature of the IJS versus other scoring systems. Within the program-as-a-whole it makes sense to me to have the further main division, "technical proficiency sustained throughout the program as a whole as opposed to individual highlight elements like jumps and spins" and "artistic merit of the program as a whole." Further sub-divisions do not seem to me to be very useful, although there is nothing wrong with listing all the bullet points that judges are supposed to take into account.

Whereas a skater with easier elements and/or less personality might have the best technical skating skills and deserve to be rewarded for them. So what else would you call that part of the score?

I have tried to think of a better name for "Skating Skills," but without success. At one time I thought that "Basic Skating Skills" might be more appropriate, understanding by this term things like secure edges, one-foot skating, turning in both directions, varied steps and turns, speed and acceleration, etc. But this world not be such I good name either, because a person could say, "Well, every senior competitive skater has extraordinary basic skills, so what are we talking about here?"

Blade-to-ice Skills would be another idea. But then you could say, all skating is blade to ice except jumps, so again the name doesn't really capture the spirit of SS.

This is not so important, though. I am sure that skaters could come up with an appropriate term. I think Skating Skills is as good as any for a combined SS & TR component. For the other, Performance would be OK. I would even go along with "Artistic Merit" except that all the artists would scream bloody murder at speaking the name of ART in vain.

 

[gkelly]

I think that it might forestall some of the complaints about how outrageous and despicable it is that we have all these lovely bullet points but the evil and incompetent judges are not following them.

Maybe. But that would be changing the system in response to the discussions by a subset of diehard fans who are devoted enough to study the scores in detail and also are inclined to distrust and malign the authority figures as a matter of principle.

Casual fans probably don't care one way or another. Fewer numbers might feel less overwhelming.

But what about the skaters themselves? Do they benefit from less information (and therefore less potential for conflicting information) in the scoring?

What about diehard fans who love studying the protocols and maybe mock-judging on their own and consider the official judges no more evil or incompetent than themselves?

Conversely, the judges may have thought that Bill was better at "rhythmic knee action" and Sam was better at "effortless glide," but I don't think we gain anything by replacing the SS category by RK and EG.

We can always propose more components than we have now, or fewer than we have now.

True. There's only so far you can break things down. I have seen some fans -- one I'm thinking of is a competitive skating parent -- suggest judges check off each bullet point to show their work in even finer detail. That would probably be useful for the skaters and for the fans who love to analyze (whether or not they're predispose to consider judges evil until proven to agree completely with themselves), but not very practical unless they have a way to enter their scores without looking away from the ice, and more judges with divided labor so they would have time to record more detail about the fewer elements or components they're responsible for.

The reason that I favor fewer is that I have become a convert to "individual elements" versus "program as a whole" as the defining feature of the IJS versus other scoring systems. Within the program-as-a-whole it makes sense to me to have the further main division, "technical proficiency sustained throughout the program as a whole as opposed to individual highlight elements like jumps and spins" and "artistic merit of the program as a whole." Further sub-divisions do not seem to me to be very useful, although there is nothing wrong with listing all the bullet points that judges are supposed to take into account.

Makes sense.

I have tried to think of a better name for "Skating Skills," but without success. At one time I thought that "Basic Skating Skills" might be more appropriate, understanding by this term things like secure edges, one-foot skating, turning in both directions, varied steps and turns, speed and acceleration, etc. But this world not be such I good name either, because a person could say, "Well, every senior competitive skater has extraordinary basic skills, so what are we talking about here?"

Degrees of competence?

5.00 is defined as "average," and that seems to be approximately the level that non-elite senior competitors tend to achieve. They might stand out as exceptional at their local rinks, but by international standards, or national standards in strong federations, they would not be considered extraordinary.

Still, for an average senior skater, or someone following that skater's career, to see his/her scores increase from 4s to 5s to 6s as his/her skating improves should be meaningful.

Blade-to-ice Skills would be another idea.

Sure. But a little unwieldy. Maybe just "blade skills." Or "edge skills." Toe steps etc. may use the blades in ways that make for fun choreography and are certainly athletically challenging but they don't really count as skating in the sense that skating = gliding.

But then you could say, all skating is blade to ice except jumps,

And lifts, in relevant disciplines.

 

[Raomina]

Maybe. But that would be changing the system in response to the discussions by a subset of diehard fans who are devoted enough to study the scores in detail and also are inclined to distrust and malign the authority figures as a matter of principle.

Casual fans probably don't care one way or another. Fewer numbers might feel less overwhelming.

But what about the skaters themselves? Do they benefit from less information (and therefore less potential for conflicting information) in the scoring?

What about diehard fans who love studying the protocols and maybe mock-judging on their own and consider the official judges no more evil or incompetent than themselves?

I think, at the end of the day, the categories have to be such that, when the skaters/fans look at the protocol to find ways to improve, it should be obvious where they can focus on to gain points for that particular category. For example, the answer to 'how can I improve my SS score' should be in the ballpark of 'do more stroking exercises and improve edges' etc. and not 'do more difficult jumps or improve my spins'. And later, when the edges have improved, the scores should reflect in the SS score. Perhaps an easier way to judge an elite skater would be 'how else can this skater improve' and score the categories accordingly by deducting from the maximum points. 10 does not necessarily equal absolute perfection, but the fact that it's the maximum score in this system means that there is inherently a 'perfect' level of what that category could be in the judges mind So if the impression that the judges get from a skate is that 'he/she could have better transitions and should work on that', then they should deduct from the transitions score instead of trying to quantify it by starting from 0.

 

[gkelly]

Perhaps an easier way to judge an elite skater would be 'how else can this skater improve' and score the categories accordingly by deducting from the maximum points. 10 does not necessarily equal absolute perfection, but the fact that it's the maximum score in this system means that there is inherently a 'perfect' level of what that category could be in the judges mind So if the impression that the judges get from a skate is that 'he/she could have better transitions and should work on that', then they should deduct from the transitions score instead of trying to quantify it by starting from 0.

So elite skaters should have a completely different scoring system (subtractive) than regular skaters (additive system)?

At what point does a skater become "elite" enough to switch over to the subtractive system? Which competitions would use which?

I can't imagine more than one system being used in the same competition.

 

[Raomina]

So elite skaters should have a completely different scoring system (subtractive) than regular skaters (additive system)?

At what point does a skater become "elite" enough to switch over to the subtractive system? Which competitions would use which?

I can't imagine more than one system being used in the same competition.

I'm not thinking about a different scoring system, but rather a different scoring mindset.