Globe and Mail: Figure skating judging system still has flaws

[Bluebonnet]

^ Huh? His 2006 Olympic programs were of the few memorable ones

It's interesting how one program perceived by different people. I agree with Jaana that Weir's 2006 Olympic LP was empty and uninspiring. He used some recycled movements from other programs of his. It's really blown out of proportion to emphasize "how successful he was" there.

His "Child of Nazareth" was a great one but he felt foreign with that program and never skated it clean. His "King of Chess" SP was great in the debute in the beginning of the season. But it was watered down and threw out a few gem of the movements at the end of the season.

 

[dorispulaski]

I don't think anyone can say that Johnny's 2006 SP Swan Camille wasn't memorable

His LP is open to discussion. He didn't skate it well.

 

[hurrah]

If in fact underrotations that were clearly visible to you and other viewers weren't called, that suggests limitations either in the video equipment available to the technical panel or in the competence of the members of the technical panel. It doesn't suggest predetermination.

That's an incredible suggestion. I kind of assumed judges knew what they were doing. But your suggestion is as likely as the conspiracy theory. If your suggestion has any truth to it, ISU really ought to get that sorted out.

 

[seniorita]

It's interesting how one program perceived by different people. I agree with Jaana that Weir's 2006 Olympic LP was empty and uninspiring. He used some recycled movements from other programs of his. It's really blown out of proportion to emphasize "how successful he was" there.

His "Child of Nazareth" was a great one but he felt foreign with that program and never skated it clean. His "King of Chess" SP was great in the debute in the beginning of the season. But it was watered down and threw out a few gem of the movements at the end of the season.

Truth is I remember programs that i feel the skater brought the program to his/her measure and put something of his soul in there instead of making it a perfect copy of the choreo he / she was given so i dont mind watered down programs. Whenever I see a fake drama or smile I cringe.
I think Weir was all personality in his programs, so most I remembered them, yes and the Vancouver ones

 

[dorispulaski]

Yes, I will remember Johnny's Olympic 2010 programs as well.

Memorable does not always mean winning.

I have strong memories of Wing & Lowe's 12th place FD to a John Lennon Medley in 2003. To tell you the truth, I can't remember what DelSchoes skated to that year. Or Dubrieul & Lauzon. Or Faiella & Scali. Or Grushina & Goncharov or Chait & Sakhonovsky, all of whom finished ahead of Wing & Lowe. But I have a clear, bright memory of Aaron & Megan.

But I remember Lobacheva & Averbukh (and not liking their program), Bourne & Kraatz (being happy they finally won), Navka & Kostamarov (like the program), Denkova & Staviyski's Baroque OD , and Tanith & Ben's Elvis Medley, and Lang & Tchernychev's Scorpions' Still Loveing You.

Memory is very idiosyncratic, like love. Different people love and remember different things.

 

[Blades of Passion]

He was at his best with a 6.0 choreography, where he could glide on the ice and do beautiful hand gestures and poses.

Gliding on the ice while forming shapes with your hands and body are part of figure skating. There's no two ways about it. CoP currently undervalues those elements while overvaluing busy work with the blades.

Here's what most people who support the "OMG transitions are most important" camp seem to not understand: It requires both talent and effort to perform with emotion, move your body into poses, and interpret the music. Thinking about emoting and exactly where to place your hands at the exact beat of music, while under the stress of competition and thinking about your technical elements and moving across the ice, is NOT easy. It's especially not easy to do that throughout an entire performance and to come up with a variety of brilliant movements specifically tailored to the music.

Measuring such things is a somewhat more subjective task (note that scoring the skating skills and transitions of a performance is ALSO subjective) but without them figure skating is far less interesting. The current CoP is trying to hide the fact that figure skating competitions can never be absolutely measured when it should instead be embracing those qualities.

 

[Bluebonnet]

^ Huh? His 2006 Olympic programs were of the few memorable ones and his Child of Nazareth and Love is War of 2007 and 2008 were also a fantastic program. I believe the Notre Damme program didnt have the exposure it deserved. All under CoP but not all the same.
In 6.0 programs you didnt glide slowly anyway.

I don't think anyone can say that Johnny's 2006 SP Swan Camille wasn't memorable

His LP is open to discussion. He didn't skate it well.

I'm sorry that I meant to say his 2010 Olympics LP. The whole time I was thinking about his 2010 Olympics LP while I was writing.: I liked his 2006 Olympics LP but he didn't skate it well. His "The Swan" is the trandmark of Johnny Weir. It IS memorable though it was too over-the-top for my taste.

 

[gmyers]

Although Weir jumped well at the Olympics, his choreography was real empty. It was originally created by David Wilson (in my opinion the best choreographer), but watered down. I have always thought that happened because Weir is not powerful enough to skate to a good CoP-choreography. He was at his best with a 6.0 choreography, where he could glide slowly on the ice and do beautiful hand gestures and poises.

His 2010 programs were not empty. They were full but not of the right things. Not a lot of nonsense moves for no reason. They were COP wrong but not empty. Maybe they did say they will just be a slave to COP but changed and moving away from COP was certainly right.

 

[Mathman]

Is the CoP hard to understand?

You get points for all the tricks you do, add them up. What’s hard about that?

Let’s take a look at ONE of those numbers that comprise the total. Here is a line from Caroline Zhang’s Four Continents protocol.

CCoSp4 3.50 0.64 1 2 0 1 3 1 1 1 2 4.14

What does a fan need to know to understand this?

First, what does CCoSp mean? So you look that up. It means Change (of foot I think) Combination (several positions I believe) Spin. Level 4 means that the skater did four of the following 8 things:

1) A difficult variation in a basic or (for spin combinations only) in an intermediate position
2) Another difficult variation in a basic position which must be significantly different from the first one and:

● spin in one position with change of foot – on different foot than the first one

● spin combination without change of foot – in different position than the first one

● spin combination with change of foot – on different foot and in different position than the first one

3) Change of foot executed by jump

4) Backward entrance/Difficult variation of flying entrance/Landing on the same foot as take-off or changing foot on landing in a Flying Sit Spin

5) Clear change of edge in sit (only from backward inside to forward outside) or camel
6) All 3 basic positions on both feet

7) Both directions immediately following each other in sit or camel spin

8) At least 8 rev. without changes in pos./variation, foot or edge (camel, sit, layback, difficult upright), counts twice if repeated on another foot.

Now we look up the base value for this designation: 3.50.

Next come the GOEs. In this example these range from 0 to 3. The criteria for positive GOE go like this: You need 2 of the following 8 bullets for +1 GOE, 4 for +2, and 6 for +3.

1) good speed or acceleration during spin

2) ability to center a spin quickly

3) balanced rotations in all positions

4) clearly more than required number of revolutions

5) good position(s) (including height and air position in flying spins)
6) creativity and originality
7) good control throughout all phases

8) element matched to the musical structure

In this example one judge thought she did zero or one of these, one judge thought she did 6 or more, ant the others are in the middle. Throw out the highest and lowest.

Now convert the remaining +1s and +2s according to the following rule (the rule is different for each element) +1 = 0.5 and +2 = 1.0. If there had been any negative GOEs the translation is different: -1 GOE = -0.3, -2 GOE = -0.6, etc.

Finally average the seven remaining converted scores. This comes to 0.64, rounded to the nearest hundredth.

Add 0.64 to the base value of 3.50. That’s your score for this element, 4.14.

Here’s another, from a different skater.

3F< 4.07x -2.10 -3 -3 -3 -3 -3 -3 -3 -3 -3 1.97

This one’s easy. 3F = triple flip . Look up the base value for a triple flip: 5.3.

< means under-rotated by more than 90 degrees but less than 180 degrees. So we multiply 5.3 by .70. (This time we round only to the nearest tenth, not the nearest hundredth.) 5.3x.70 = 3.7.

But there is also an x. This means the element was done after the half-way mark in the program. Multiply 3.7 by 1.10 (leave the answer in hundredths; do not round). 3.7x1.10 = 4.07.

This time the GOEs are easy. All -3’s. Naturally that means we should subtract 2.1 from our adjusted base value of 4.07. Score for this element is 4.07 – 2.10 = 1.97.

I am not making this up!

Do this 286 more times (twelve elements for each of 24 skaters) and you will understand the TESs for the ladies long program at Four Continents. (I will omit the PCSs – they are a little tricky. )

 

[emma]

You had me reading with bated breath until we cot to the -3's followed by the word 'naturally'.....then I realized, once again, you got me! And of course, that means my computer screen got 'gotted' too

Impressive, though, how well you understand AND can explain this....you might find yourself converting people (something like careful what you wish for turned into careful how/what you instruct...)

 

[skatinginbc]

Well, I don't think analysis of presidential debates is any more objective than analysis of figure skating programs. And I also believe that the public can be taught to adopt different opinions about what matters in a presidential debate and who "won" any given debate depending on which media analysts they pay attention to or what the media tell them beforehand will be important. Different questions (about what constitutes a winning debate) will generate different answers. mmcdermott suggests that there may be cultural differences across figure skating countries/regions regarding the kinds of questions they encourage judges to ask about what makes good skating. I think there are also cultural differences between skating experts as a group vs. fans. And on a finer scale between coaches vs. judges vs. TV commentators.

Those are issues of survey design and sampling. Did you notice I said "experts and representatives" and "proper sampling methods"? The word "representatives" implies the existence of various strata or communities (coaches vs. judges vs. commentators vs. federations, etc.). Don't worry. An expert hired by the ISU should be able to handle those issues.

That's because of the required element rules, not the judging system.

The term "judging system" is confusing. In your definition, it means the system of cumulative evaluation. It hasn't changed since the ISU implemented the NJS. To me, the "judging system" (measurement) has changed many times. Every change of the rules is a change to the measurement.

 

[gkelly]

The term "judging system" is confusing. In your definition, it means the system of cumulative evaluation. It hasn't changed since the ISU implemented the NJS. To me, the "judging system" (measurement) has changed many times. Every change of the rules is a change to the measurement.

Yes, we may be using the term in different ways.

Usually in these discussions we're talking about ordinals based on two scores for whole programs vs. absolute scores added up for individual elements and (for lack of a better word) program components -- 6.0 vs. IJS -- as general principles. Another completely different approach could also be possible (and we have seen some examples in the past in professional competitions).

In looking at past and present program content and competition results, all too often people conflate different kinds of changes and their effects, blaming an effect of one kind of rule change on a completely different change. I think it's valuable to try to maintain a distinction among the different kinds of rules.

The rules of the sport tell us what kind of skating content is being judged: how many phases of competition, what kinds of skills are included in each phase.

Eliminating school figures was a big change in the total content of what was being measured, but there was not an accompanying change at the same time in how the remaining content was measured.

In ice dance, the change from Original Set Pattern dance to Original Dance was a change in What, but there was no change in How that took place at the same time.
More recently, there was a big change from Compulsory Dance(s) plus Original Dance to Short Dance. The details of how specific pieces of this phase of dance competition have changed along with the change in exactly what the pieces are, but the general principle of breaking the dance into elements, a technical panel assigning levels, and a judging panel assigning GOEs and components has not changed.

I would also distinguish between what I call the judging system, by which I mean the system by which judges evaluate the content that the program rules require, and the scoring system, by which I mean the way that the judges turn their evaluations into numbers and what the accountants or computers do with those numbers.

E.g., there were several significant changes between the 1998 and 1999 seasons. The rules for senior men's short program jump content changed significantly. The rules for Original Dance and Free Dance content changed significantly. (Both Rules of Sport changes) And the scoring algorithms for international competitions changed significantly, from Majority to One-by-One calculations. (Scoring System change) But the judges kept doing exactly the same thing they had been doing -- there were no changes to the Judging System that year.

If you want a global term to encompass all the rules about what is measured and how, I would recommend using the term Evaluation System rather than "Judging System." The judging system is only a subset of the evaluation system as a whole.

In our hypothetical situation of hiring evaluation consultants to design a whole new judging system or judging+scoring system from scratch, are we also going to ask them to decide on program content and competition structure? Or are we going to present them with the current rules about short program and long program (or short dance and free dance) content and say "This is what we want to evaluate. Please help us figure out the best way how to evaluate it"?

 

[skatinginbc]

In our hypothetical situation of hiring evaluation consultants to design a whole new judging system or judging+scoring system from scratch, are we also going to ask them to decide on program content and competition structure? Or are we going to present them with the current rules about short program and long program (or short dance and free dance) content and say "This is what we want to evaluate. Please help us figure out the best way how to evaluate it"?

Say, a national council would like to develop a screening test to ensure the English proficiency for non-native speakers studying in American universities. Two groups of experts are involved: Language experts and testing experts. Scholars of Second Language Acquisition may differ significantly in opinion from those of English Literature regarding what abilities should be included in the test. Even experts of Teaching English as a Second Language may disagree with each other. Whose views are correct? How do we reconcile differences among those experts? It is the job of the testing experts to facilitate the decision process.

What native-like English proficiency is should be a stable concept, and so should be what constitutes a good piano player, a good baseball player, a good figure skater, and so forth. Although a trend may change across a larger time frame, it should not change every other year. The ISU may tell evaluation consultants "This is what we want to evaluate. Please help us figure out the best way how to evaluate it", and the consequence: Those who have the most political powers define what good skating is about, and whoever is good at lobbying dictates the rules. If the ISU has hired qualified evaluation consultants, there should be generalizability coefficients (similar to "reliability coefficients" in Classical Test Theory) reported somewhere. They should be able to answer such questions as "How the consistency of scores may change if 7 judges are used instead of 9" and "How the consistency of outcomes may change if the judges rate a skater against others (i.e., norm-referenced) rather than against given guidelines (criterion-referenced)". They may answer a lot of questions. Strangely, since the implementation of the NJS, I haven't seen coefficients published. All I found was unanswered questions.

If the ISU cannot afford hiring consultants, they may offer scholarship to some doctoral students for conducting studies on the NJS. I know for certain that Educational Testing Service gave out scholarship in exchange for research and development on TOEFL (Test of English as a Foreign Language). Two decades ago they were particularly interested in performance-based assessments (for English speaking skills) that involve a number of raters (judges) with scoring guidelines. Maybe the ISU does not want the judging system scientifically validated by outsiders. Maybe they would like to keep it full of smoke and mirrors, as complicated as possible so you can’t see what goes on behind closed doors.

 

[gkelly]

Yes, it would definitely be preferable if the ISU had done a more thorough study of how the stakeholders (officials, coaches, competitors) wanted to define good skating, hired experts in evaluation to help design a new system, and tested it more thoroughly with shadow panels or test events before adopting it as the official system.

I don't know how much more of that they would have done if the scoring system change had been a longer-term project rather than rushed in after the SLC scandal. Certainly the 2003 Grand Prix (and Nebelhorn) should be considered test events.

 

[Mathman]

*Skating is a highly technical sport based on the different ways the human body can direct blade edges to glide across the ice on curves. All skating skills should be rewarded in terms of the wide range of ways to vary and combine the basic edges and to transition from one edge to another (including multiple rotations in the air). Spectacular and athletic variations are worth rewarding. So are skill shown in executing and combining these moves with visual beauty and in time with music or even better expressing nuances of the music. But the control of edges on ice is always fundamental.

For this view, I wonder if our consultants would come up with something like this. A two stage competition. In the first stage the skaters would demonstrate their blade control by tracing set figures. They would be judged according to how well their figures matched geometric perfection.

Then there would be a free skating part, with two marks. One for the spectacular and athletic aspect, and one for the visual beauty and music aspect.

 

[ivy]

Gliding on the ice while forming shapes with your hands and body are part of figure skating. There's no two ways about it. CoP currently undervalues those elements while overvaluing busy work with the blades.

Here's what most people who support the "OMG transitions are most important" camp seem to not understand: It requires both talent and effort to perform with emotion, move your body into poses, and interpret the music. Thinking about emoting and exactly where to place your hands at the exact beat of music, while under the stress of competition and thinking about your technical elements and moving across the ice, is NOT easy. It's especially not easy to do that throughout an entire performance and to come up with a variety of brilliant movements specifically tailored to the music.

Measuring such things is a somewhat more subjective task (note that scoring the skating skills and transitions of a performance is ALSO subjective) but without them figure skating is far less interesting. The current CoP is trying to hide the fact that figure skating competitions can never be absolutely measured when it should instead be embracing those qualities.

I agree with this whole post.

Elements that are beautiful but may appear simple are undervalued. Gliding on an edge and spinning in place for an extended period are unique to skating. Acrobats can jump and spin, dancers can execute amazing foot work. Only with skates can you perform an spread eagle that flies through space and time or a layback spin that goes on and on. And when they're performed to music with grace or power or finesse, or what ever special thing a skater can bring they can creates a moment with emotional depth that is not fully quantifiable. Though of course we do need to try, since it is a competition. CoP, which I generally support, is taking some of the grace and emotional content out of skating and losing part of what makes skating special.

One other brief note re MM's example. I so wish the protocol sheets would just list the actual name of the element, rather then a list of letters. It's all computers, no one has to type it out. Way to many 'S's - spins, spiral, steps, sequence, salchow and when is 'Ch' change or when is it 'choreograph', maybe it never is? I can look these things up, but why not just use words!

 

[gkelly]

One other brief note re MM's example. I so wish the protocol sheets would just list the actual name of the element, rather then a list of letters. It's all computers, no one has to type it out.

Space on the page.

The US results on IceNetwork do give the spelled-out names of the elements on the summary for each skater with the averaged scores (if you click the + sign next to the total score in the results list), but the protocols that list all the judges GOEs and components still use the space-saving abbreviations.

Way to many 'S's - spins, spiral, steps, sequence, salchow and when is 'Ch' change or when is it 'choreograph', maybe it never is? I can look these things up, but why not just use words!

Good point about the S's.

Ch is only used for Choreographed sequences.
For spins with change of foot, the abbreviation is C. But then that gets confusing with the C for camel and Co for combination spin.

Since CCoSp (combination spin with change of foot) is required in most levels of short programs and has been required in some free programs, it's probably the single most common spin abbreviation and people, especially technical panel people, use the letters in conversation.

If your coach is also a technical specialist, he might well ask you to show him your cee-cee-oh-ess-pee.

 

[emma]

^ thanks for pointing this out, gkelly re: icenetwork and spelled out elements, I hadn't realized this before. I've been using ISU communication 1611 on the scale of values to help me, but it is a lot to scroll through. This makes it easier, so again, thanks.

 

[ivy]

Space on the page.

This is where you hire a good graphic designer to create a clear and readable protocol sheet. All the abbreviations do is to create a document that is not easily accessible to the average fan. I read the protocol sheets for every competition I watch and sometimes even for competitions I don't watch. Intentional or not, the abbreviations are just another hurdle the bureaucracy at the ISU places between skaters and their fans.

 

[mskater93]

These are common abbreviations for coaches/choreographers/skaters when writing down a program layout and a lot of judges used to use something like those for their 6.0 judging sheets (each judge had own nomenclature but it was similar - S-Salchow, T-toe loop...)