Fair/Over/Underscoring - Tricky Subject, Tread Softly

[prettykeys]

When I signed up to Golden Skate last week, this is actually what I wanted to discuss: in a performance sport where judges determine scores, how does one evaluate whether the judgments were fair, high, or low?

This is an age-old question that is not exclusive to figure skating, but as tensions run higher in an Olympic season, this could be a useful discussion. If you wish to discuss this, try to be as unbiased as possible. If you think this is a pointless discussion, then leave this thread uncluttered by "get over it" types of comments for those who think this is important or at least interesting, please.

In one of the Rostelecom threads (the Men's Free Program?), this subject and its relationship to statistics was raised by Mathman and discussed by Buttercup and myself, especially as applied to Program Components (PCS).

The PCS have specific guidelines and criteria as to what constitutes certain scores (thanks to Common on the Yu-Na Kim Forum.)

http://www.isu.org/vsite/vfile/page/fileurl/0,11040,4844-168551-185769-65184-0-file,00.pdf

PCS is very subjective and we can't deny that it's changing as time goes by. When NJS was first introduced 5 years ago PCS was really high marking 68.88(8.6+ per each component), Sasha Cohen's PCS in SA 2003 which is even higher than current Yuna's PCS 66.46(8.3+ per each component).

Until last season's Worlds, highest PCS was considered as 63+(7.88+ per each component, which means the skater got 8+ in 1 or 2 out of 5 components). And that of the average well performed program was around 60+(7.5 per each component). Judges tended to give 60+ when the skater executed all the planned elements nicely, though not very extraordinary.
That said, we can easily make an educated guess on the judging trend. There existed certain barrier between 7 (good) and 8 (very good) points. And she broke that barrier in LA.

The rest of that discussion is here: http://yunaforum.com/index.php?showtopic=927

My take on this is that there may be guidelines that are supposed to be more or less objective, but it requires experience and time for judges to determine examples of what is "good" vs. "very good", "poor" and "exceptional". Once these sorts of examples and standards are established, one would hope they can apply them consistently towards an individual skater; between skaters, and between years. I do not believe this is happening, although I am unsure about what the consequences may mean in the long run.

 

[prettykeys]

Now I'm going to elaborate on this in regard to my favourite skater whose fans I ticked off on her fan forum: YuNa

Being a fan means you support someone whether they are on a roll or in a slump (check.) But being fair is also important. For example, last night's UFC bout between Lyoto Machida (who I'm a fan of) and Mauricio "Shogun" Rua clearly showed (to me and MANY others) that Shogun won, but somehow all 3 of UFC's judges unanimously gave Machida the decision win and left a bitter taste in my mouth. But that's another topic.

My foray into this subject began seriously when I said that I believed YuNa's marks at TEB 2009 had been a little inflated by 2-3 points; it didn't change the medalling outcome, but it allowed me to give some sympathy to fans of other skaters. At the same time, some anti-YuNa's got downright catty and insulting.

My initial explanation was that because the CoP system is still fairly new, and no one has yet thrived under it like YuNa is starting to do, figure skating fans and judges are unaccustomed to the amplified effect that small differences can make in the scoring; that is, differences are multiplied in CoP, whereas small 5.8-6.0 type of differences in the old system were only additive. But thinking back over the years, I cannot see how YuNa's record scores indicate so much difference in skating ability - between herself over several competitions, and also versus other quality skaters over the years. So ultimately I do believe there was inflation related to both an increasing trend of CoP scores as well the fact that in the past two competitions, YuNa did not have a close competitor.

Skate America 2008
Here are YuNa's Danse Macabre PCS
7.70 Skating Skills
7.20 Transitions
7.70 Performance
7.75 Choreography
7.70 Interpretation
Judges Total Program Component Score (factored 0.8) 30.44

Miki was 2nd in total SP score; here are her PCS:
7.30
6.85
6.95
7.30
7.25
Judges Total Program Component Score (factored) 28.52

YuNa's Scheherazade (1st)
7.60
7.10
7.70
7.60
7.50
Judges Total Program Component Score (factored 1.6) 60.00

Yukari got 2nd with 56.64

Skate Canada 2008
Joannie Rochette got highest FS with PCS of 60.16

Cup of China 2008
YuNa Danse Macabre SP 1st
7.60
6.90
7.40
7.25
7.30
Judges Total Program Component Score (factored) 29.16

YuNa Scheherazade LP 1st
7.75
7.30
7.90
7.80
7.80
Judges Total Program Component Score (factored) 61.68

Miki 2nd LP with 56.40

2008 Grand Prix Final
YuNa Danse Macabre
7.80
7.30
7.65
7.65
7.65
Judges Total Program Component Score (factored) 30.44

Mao 2nd SP
7.65
7.10
7.45
7.55
7.35
Judges Total Program Component Score (factored) 29.68

Mao 1st LP
7.80
6.95
7.60
7.50
7.40
Judges Total Program Component Score (factored) 59.60

YuNa 2nd LP Scheherazade
7.85
7.40
7.60
7.65
7.45
Judges Total Program Component Score (factored) 60.72

Worlds 2009

Note that things were fairly consistent until the spike at the last World Championships -- and yes, everyone's LP PCS was accordingly inflated.

YuNa Danse Macabre 1st
8.45
7.75
8.50
8.05
8.15
Judges Total Program Component Score (factored) 32.72

Joannie 2nd 30.40, Mao 3rd 30.16

YuNa Scheherazade 1st
8.50
8.25
8.70
8.60
8.70
Judges Total Program Component Score (factored) 68.40

Miki 2nd 63.92, Joannie 3rd 62.80, Mao 4th 62.88

Now compare with the Olympics 2006 performances:
Sasha SP 1st
7.75
7.54
8.00
7.89
8.07
Judges Total Program Component Score (factored) 31.40

Sasha was ON FIRE, her Performance grade of 8.00 could be taken as a good measuring stick.

YuNa's WC2009 Danse Macabre hit a Performance value of 8.50, even though two judges scored it at 7.50 and 7.75 (I believe anywhere from 7.85 to 8.25 would have been more reasonable.)

Joannie's overall Performance was 7.70, Mao's was at 7.60 (both reasonable.)

Shizuka won in the LP
8.04
7.61
7.93
7.86
7.93
Judges Total Program Component Score (factored) 63.00

In my opinion, the cool, cautious skate of Shizuka's here was reminiscent of YuNa's Gerschwin Concerto at TEB 2009. Substituting in YuNa's Skating and Transitions (8.30, 8.00), her overall score of (40.02 x 1.6 = ) 64.03 seems to me more appropriate than the 66.40 that she actually got at TEB 2009 (and which surprised even YuNa.) But except for the blip at 2009 Worlds, it would also be hard to argue for the leap from what was once YuNa's own high 7.xx Skating score to low-mid 8.xx. Bring her Skating value down to 8.00 as well, and you get 63.55 which is very reasonable.

YuNa TEB 2009 Gershwin LP:
8.30 (8.00)
8.00
8.45 (7.93)
8.35 (7.87)
8.40 (7.93)
Judges Total Program Component Score (factored) 66.40 vs. 63.55

Mao's TEB 2009 LP was 2nd:
7.75
7.30
7.75
7.70
7.60
Judges Total Program Component Score (factored) 60.96

Nothing too unusual about Mao's PCS. But from my outline above, it seems YuNa was a little overscored in the range I said; by about 2-3 points. I could feel it, but now I've shown it for those who can't see it and/or think I'm a fake YuNa fan and being a broken record for its own sake while others are just being sore. There's a real trend of inflation going on, guys. This is not YuNa's fault; I am not blaming or hating on YuNa. This just shows how judging is still questionable, inconsistent, and fluctuating.

 

[Spun Silver]

(I'm just happy Mathman has someone to commune with.)

 

[PftJump]

Again~ Yuna Overscore thing...Yay~~~

But Yuna, at least have never got PCS score that was higher than her TES score.

IMO, the suspicious one is someone relieved by fixed PCS score -that is way over TES score- regardless of her/his performance.

 

[Blades of Passion]

GREAT post, prettykeys.

Sasha's SP in Torino was, IMO, the best performance we have ever seen from a lady in the SP under CoP. She should have received Plushenko-like PCS and, like you say, that performance was indeed a great measuring stick.

 

[prettykeys]

(I'm just happy Mathman has someone to commune with.)

Sasha's SP in Torino was, IMO, the best performance we have ever seen from a lady in the SP under CoP. She should have received Plushenko-like PCS and, like you say, that performance was indeed a great measuring stick.

I think it was, too.

Performance: is the involvement of the skater/couple/teams physically, emotionally, and intellectually as they translate the intent of the music and choreography.

One could even argue that she wasn't really given enough credit--I can't imagine an audience getting too much more involved in a performance like they did with hers...or a performer being more jubilant.

 

[Mathman]

Again~ Yuna Overscore thing...Yay~~~

But Yuna, at least have never got PCS score that was higher than her TES score.

IMO, the suspicious one is someone relieved by fixed PCS score -that is way over TES score- regardless of her/his performance.

I think there are many performances where the PCSs are higher than the TESs, and quite rightly. Remember that the "T" in TES does not stand for "technical." The distinctions is that the TES is the total score for individual elements while the PCS is the mark given to the program as a whole.

Alissa Czisny is a good example of a skater who makes many mistakes on individual jumps, but still skates a beautiful program overall.

There are actually two ways in which a skater's PCSs can turn out to be much higher than her PCSs, without controversy. Second tier U.S. skater Katrina Hacker could not achieve the highest levels in TES because she did not have a flip or Lutz. But her programs were breathtaking.

The other thing that can happen is that a skater might lose many points on the TES side for downgrades due to underrotation, but this does not disrupt the program, and indeed, is usually not even visible to the audience.

 

[janetfan]

Such interesting stuff - but it also leads me to wonder how it is possible for judges to watch a program, digest all of this in a couple of minutes and then present an accurate score. No wonder so many feel so many judges are getting it wrong.

A first thought would be if the judges today are still primarily doing this because they come from very well to do backgrounds and happen to be skating fans.

I am sure there are test requirements - but one's social standing might be as important as any real skating experience. Is it still true today that many judges have little or no practical background in skating?
Perhaps in 2009 that statement is false but going back only to 2002 there were judges who had very limited or no skating experience at all.
Perhaps that is OK at the Westmisnter Dog Show but shouldn't skating judges have a really thorough knowledge of figure skating, one that goes beyond being a fan and simply coming from a wealthy family?

That is not really my point but I think it explains why reputation is still so important. If a judge really doesn't know the difference it is simply logical and safer for them to award higher marks to the top skaters. Much of what we suspect could be cheating could be nothing more than incompetence.

Wondering about the tech callers - that seems to be a good idea. Using replay seems good. Why not try and get it right since it is supposed to be a sporting competition.

Why not rethink the whole judging concept and have specialists for many of the elements. Ex: if all a judge had to do was be resposnible for the step sequences I would tend to believe they could be more accurate in their appraisal. Why not a judge who would only rate artistic/musical interpretation?. If that is all they were looking at without having to worry about so many other aspects of a performance wouldn't they be able to offer a more accurate analysis and score?

Is the old excuse, "well his/her program is really harder" valid? I say a program that is designed to be harder should not get too much consideration if the skater doesn't execute it very well. Presentation should be based on it's meaning: "presentation." Splatting all ove the ice should not be rewarded or overlooked just because it may be a difficult program. CoP trends towards this and it is why we see skaters consistetly unable to deliver a clean program.

Just a few thoughts on this. Reading some of this stuff makes my head spin but thanks for discussing marks and scoring which still remains a troublesome topic in figure skating.

 

[i love to skate]

A first thought would be if the judges today are still primarily doing this because they come from very well to do backgrounds and happen to be skating fans.

I am sure there are test requirements - but one's social standing might be as important as any real skating experience. Is it still true today that many judges have little or no practical background in skating?
Perhaps in 2009 that statement is false but going back only to 2002 there were judges who had very limited or no skating experience at all.
Perhaps that is OK at the Westmisnter Dog Show but shouldn't skating judges have a really thorough knowledge of figure skating, one that goes beyond being a fan and simply coming from a wealthy family?

Not sure about other countries but to become a judge for Skate Canada you do need to have a skating background and need to have passed tests up to a certain level. Why do you say that skating judges come from wealthy families? The large majority of judges hold hown full time jobs and take time off from work (without pay) to go and sit in cold arena for hours on end. It takes quite a commitment to become a judge.

 

[janetfan]

Not sure about other countries but to become a judge for Skate Canada you do need to have a skating background and need to have passed tests up to a certain level. Why do you say that skating judges come from wealthy families? The large majority of judges hold hown full time jobs and take time off from work (without pay) to go and sit in cold arena for hours on end. It takes quite a commitment to become a judge.

Thanks for the info regarding Canadian judges.
For some judges, it is indeed quite a committment. For others possibly not as much if they are from wealthy families, or are very well taken care of by their federations.

It is possible my views are slanted by my age and things I have read about judges over a few decades of following skating.
I hope to hear a few other views about this and about other federations.
I would be happy to be very wrong (or have an outdated view) about this.

 

[i love to skate]

Thanks for the info regarding Canadian judges.
For some judges, it is indeed quite a committment. For others possibly not as much if they are from wealthy families, or are very well taken care of by their federations.

It is possible my views are slanted by my age and things I have read about judges over a few decades of following skating.
I hope to hear a few other views about this and about other federations.
I would be happy to be very wrong (or have an outdated view) about this.

Not sure if you know this and if you do I apologize - a judge is not a paid position. It is strictly volunteer - your airfare, meals, and accommodation are paid for but you receive no payment for the hours you are at the rink.

 

[janetfan]

Not sure if you know this and if you do I apologize - a judge is not a paid position. It is strictly volunteer - your airfare, meals, and accommodation are paid for but you receive no payment for the hours you are at the rink.

I am sort of aware of what you said but am not sure if it points a finger at most judges be regular working or middle class folks. To the contrary it seems to point to upwardly mobile people or wealthier people being the one's who would have the time and flexibilty in their lifestyle.

No apology necessary, I always enjoy your POV.
You have also made me a Joannie fan :yes:

 

[i love to skate]

I am sort of aware of what you said but am not sure if it points a finger at most judges be regular working or middle class folks. To the contrary it seems to point to upwardly mobile people or wealthier people being the one's who would have the time and flexibilty in their lifestyle.

No apology necessary, I always enjoy your POV.
You have also made me a Joannie fan :yes:

I have had the opportunity to meet many judges -some even at International and ISU levels - and they have not had high incomes. Many that I have met work in social services or education and those are not high paying fields. I'm sure there are some judges that come from wealthy families and I know there are some "bad apples" but the majority are good, honest people who love skating.

I am glad you are now a Joannie fan!!

 

[janetfan]

Well I am a judge and let me tell you I do not have a high income - I am a student after all I have had the opportunity to meet many judges -some even at International and ISU levels - and they have not had high incomes. Many that I have met work in social services or education and those are not high paying fields. We do it because we love it! I'm sure there are some judges that come from wealthy families and I know there are some "bad apples" but the majority are good, honest people who love skating.

I am glad you are now a Joannie fan!!

Thanks for your insights. I appreciate it and feel better about judges, atleast in Canada.

 

[HCOSurfer]

But Yuna, at least have never got PCS score that was higher than her TES score.

This is where you're wrong m'lady:
http://www.isuresults.com/results/wc2009/SEG004.HTM

 

[prettykeys]

But Yuna, at least have never got PCS score that was higher than her TES score.

IMO, the suspicious one is someone relieved by fixed PCS score -that is way over TES score- regardless of her/his performance.

This is where you're wrong m'lady:
http://www.isuresults.com/results/wc2009/SEG004.HTM

OOooOooOoOooh, nice catch!

Re:Joannie, she is probably the skater right now who has the least publicity considering how good she has become in the past few years. She's improved by leaps and bounds in a very short time.

That's all I want to say for now. I'm really busy, I have at least 3 assignments every week and now it's midterm time so I got almost no sleep last night and another killer midterm is coming up this Friday. I'm saying this especially to you, janetfan, because you wrote a lot that deserves a good response but I must apologize that I can't read and reply just yet. Maybe someone else will; if not I'll come back to it!

I'm happy that no one's pounced me and gotten too upset or used my post as fodder for hate. I know it's a touchy subject.

 

[janetfan]

OOooOooOoOooh, nice catch!

That's all I want to say for now. I'm really busy, I have at least 3 assignments every week and now it's midterm time so I got almost no sleep last night and another killer midterm is coming up this Friday. I'm saying this especially to you, janetfan, because you wrote a lot that deserves a good response but I must apologize that I can't read and reply just yet. Maybe someone else will; if not I'll come back to it!

I'm happy that no one's pounced me and gotten too upset or used my post as fodder for hate. I know it's a touchy subject.

Good luck on your mid-terms. And don't worry, I get shredded enough as it is

 

[prettykeys]

Oh, I wasn't going to shred you, I just felt that someone who puts a lot of thought into a big post deserves a thoughtful response.

 

[Mathman]

In my opinion, the Big Fiction of the program component scores is this. The ISU claims that they can write down point by point guidelines with such clarity and accuracy that any judge with proper training can distinguish between a program that properly deserves 4.25 for performance/execution and a program that objectively deserves 4.50.

This boast is so absurd that it is not worth discussing. What judges can do, though, is judge whether one skater did better than another in this program category. But any way you slice it, when one of the criteria is "does the skater establish an invisible connection with the audience" there are bound to be judgment calls along the way.

Well, what's wrong with that? That's why we call them judges instead of tape measures or stop watches. They make judgments.

I think in practice, what the judges do is something like this. Let's say you are judging a juvenile or intermediate competition. You know from experience that skaters at this level are expected to get somewhere between 2.75 and 4.00 in interpretation. So the skater that you like best, you give him a 4.00, and the skater whose program seemed not so magical gets a 3.00..

I don't see how it could be otherwise, considering the subjectivity of the criteria. Skaters at the Yu-na Kim / Mao Asada level get high eights. So if Kim really nails it she might get 8.75, while if Mao is sloppy this time out she gets a 7.75.

 

[prettykeys]

Why not rethink the whole judging concept and have specialists for many of the elements. Ex: if all a judge had to do was be resposnible for the step sequences I would tend to believe they could be more accurate in their appraisal. Why not a judge who would only rate artistic/musical interpretation?. If that is all they were looking at without having to worry about so many other aspects of a performance wouldn't they be able to offer a more accurate analysis and score?

Is the old excuse, "well his/her program is really harder" valid? I say a program that is designed to be harder should not get too much consideration if the skater doesn't execute it very well. Presentation should be based on it's meaning: "presentation." Splatting all ove the ice should not be rewarded or overlooked just because it may be a difficult program. CoP trends towards this and it is why we see skaters consistetly unable to deliver a clean program.

I was just revisiting this thread, and I must say that I agree that having judges judge fewer things in a more focused way should result in better analyses and more accurate scoring. Not only is it harder to be an expert across multiple issues, it is likely more difficult to pay attention to everything in a balanced and comparative way. :think:

And yes, judgment should ideally be targetted for the presentation at hand, not based on expectations or past precedent.