The Good Judging/Judging Theory Thread

[gkelly]

There seems to be a lot of fan interest in calling out bad judging.

But we can’t know what judging looks like done wrong unless we first have a good idea of what it looks like done right.

Personally, I’m most interested in analyzing the skating, within the established rules and guidelines (which often change slightly or significantly from year to year). What fascinates me about this sport is how complex and qualitative it is, requiring analysis and judgment calls and balancing of varying degrees of different qualities to reach any determination of what was good or better.

So I’d love to have a thread where we can talk about what we believe good judging looks like in theory. I don’t want to focus on specific judges or specific skaters, but on general principles – with some real-world examples where relevant.

I’m probably more fascinated by program components than GOEs, but we can talk about both here.
And maybe some discussion of how different rules or different structure of the scoring system would affect the evaluation tasks.

If you think you could do a good job as a judge, how would you describe what you’d be good at?

 

[Izabela]

To be honest, fair judging in figure skating doesn't really have to be that difficult to achieve (if we apply the CoP guidelines appropriately with enough expertise and exposure to programs). There's no such thing as an absolutely objectively valid judgement, and all of us judge in relation, in differentiation. As I've mentioned elsewhere, the GOE guidelines seem to operate like the 4.0 Grading System (or any other grading system). You can get a 4.0 in a subject but it doesn't necessarily mean that the professor thinks you are best in your class or the best student they had; simply that you performed all the requirement excellently to meet the 4.0 grade. It's almost impossible for a judge to recall who did the best execution of an element or so and scale all their scores based on that, so the GOE guidelines function to give judges more freedom in scoring a jump based on itself: whether it has very good height and length, whether it looks effortless and so on. It's a neat system in that it is quite easy to follow (if you follow it fairly and with good intention). In short, I really don't have much problem about the current judging system we have in comparison to how the system is used/applied.

The problem arises when judges don't have the time to differentiate and review their own scores during competitions. For example, in the first program you have to score, you gave Skater A +4 for their 3A based on the GOE guidelines: they meet your definition of those three first bullets and you think the element matches the music. And then you watched Skater B, and also gave them +4 for the same, but now you worry, because you still think Skater B performed the jump better than Skater A, while also thinking it didn't satisfy +5? How can you account for that difference? You may end up compensating by increasing some points in the PCS for example. And then you watched Skater C, and gave them +5 for their jump, but now you worry, since you think the "1" difference between the Skaters' A&B and Skater C isn't enough to reflect the difference in quality among those 3As. So, again, you may be compelled to compensate by increasing some points in the PCS. Now think in larger scale where judges have to make these differences for 24 individual programs; it's really no wonder that judges tend to rely on reputation (and tiers of skaters) and are conservative in their scoring at the earlier part of the competition.

Would it be a better system then, for judges to be given more time to review and analyze the programs and the scores they did? Rather than have their scores be announced right after the skaters performed their respective program? I mean, even in other performance competitions, they don't give the scores right after you perform, but they wait until all the performances are done, and then the judges also were given more time to sit together to deliberate and find consensus in their scoring and placement. Sometimes they post the scores the next day. By giving that room, they may be able to readjust their score; lower that +4 to +3 because the other judge gave a valid reason why it should just be +3. They don't have to do this for all 24 individual skaters; I think it's really not that difficult to pick which scores and programs they should be evaluating. I understand that we want judges to judge independently, but I think it would be a better way of judging (particularly in this kind of sport) if we give judges that ability to talk with each other, deliberate and reevaluate their own scores. I personally wouldn't mind it if they will post all scores the next day after the competition.(And hold the award ceremony together for all 4 disciplines, mandating that they all wear their costumes of course ). But then again, it doesn't necessarily have to be the next day, but that the scoring shouldn't be announced as quick as they do it today. I think it's a quite a disservice for this sport to make its scoring as real time as possible (like basketball or tennis) since figure skating is more like a music competition or any performance competition where the norm is everyone wait for all performers to do their thing before they reveal the scores and rankings. It's really the ideal way for me, although this may not be good to attract audiences because they do want to know who wins as soon as possible.

(I mean, this doesn't really touch the much bigger program of national biases and politicking and all, but if we hold that judges as fair, I think this system will result to a better judging.)



Also, again, as has been pointed out several times in this forum, the first thing we need to do is to divorce the judges from the feds. Create a pool of competent judges that are trained by ISU and are subjected to annual evaluation. Probably where it's open for everyone and the acceptance is based on that person's resume, and they can be assigned from lower tier events until they build their resume for major international competitions. The tricky part is where will their compensation be coming from? Should they be treated as ISU employees (can they do that?) and have their salaries come from ISU? If they found them incompetent or found them colluding with certain fed, ISU can just fire them out. Or something like that.

 

[Eclair]

Shin Amano does pretty fair tech calls most of the time and more often than the average technical judge. For me, good judging is being harsh to everyone. It's also one of the less controverse results of the season.

In general, I disagree with the notion that the job of a judge is to score skates from his own country the highest as possible without getting caught cheating, while trying to be somewhat objective to all other skaters.

The job of the judge is to score what they see at that day at competition, based on their understanding of what good skating is no matter nationality or politicking. These skaters all train hard, they deserve the to be judged fairly.

 

[Yuzuruu]

If you think you could do a good job as a judge, how would you describe what you’d be good at?

I can call a flutz in real time while some tech specialist cannot even do it in slo-mo

 

[yume]

Good judging is when a badly badly badly badly badly badly badly, seriously seriously seriously, severely severely severely bombed program doesn't get just 5 points below the perfect PCS score.

 

[Henni147]

If you think you could do a good job as a judge, how would you describe what you’d be good at?

I think this is a very legit question. It's easy to criticize the judges or the system as such, but try to find a solution, everyone is happy with... :giveup:
No matter how hard you try to make the judging as objective and fair as possible, there will always be some space for misuse, especially when it comes to artistry.

Just two examples:

Advantages of subsequent judging:
1. You have enough time to review every single element and component ---> enough time to check every GOE bullet and PCS criteria
2. You don't have to hold back with points before the final skate (It would be very interesting to see, how many points judges would add to a textbook performance in an earlier group or at the beginning of the final group in a global review...)

Advantage of realtime-judging: Figure skating is not only a sport but also a spectators' event with entertainment factor. Try to imagine figure skating without kiss & cry moments! :shocked: Without scores the competitions wouldn't differ from the EX gala...

Advantage of conference judging: there wouldn't be extreme anomalies between the judges like 5x GOE+3 and 1x GOE0 for the same element (4CC 2017 *cough*)
Advantage of isolated judging: biased judges can't join and influence neutrals


It's really hard to decide... :whack:

 

[OS]

Would be great if this is a standard practice and not just a theory!

I fear judges are put into an impossible situation from the start. When their interests cannot be aligned with the Olympic Motto, less to do with 'fairness' but what they can do to serve their federation. All athletes performances on the day are secondary. You can see it in all the grey areas in PCS and GOEs.

Judges probably work very hard study and follow all the rules, to get to the ISU level, but it is absolutely human nature to have a tendency to corrupt once you reach the top. It is not a matter of capability or competency, but 'alternative facts' can become a weapon for self-serving, integrity is easily compromised. Also because they know the rules so well, it is very easy to get away with it within 'acceptable' tolerance. If they can't, they will just change the rules. Not unlike the recent immigration law that got passed through the US supreme court once the lawyers went through it, masking its real intentions. Elite judging at the top governing the sport runs a bit like that, rather feudal in setup. While the rest of the lower rank world suffers from their fickle whims. Elites of anything have a tendency to hoard all the glory, creating new barriers and new rules if challenged - to avoid sharing the riches or erode their power. There are also the rare elites at the other end of the spectrum trying to set by example, put up the good fight, but these are rare and far in between. Without strong ethics review and governance, all systems will eventually breaks down and get overturn through a revolution.

You can not change human nature in a 100% human judged sport. Any goodwill naivety blue sky thinking will eventually get dirty at the top when judges had to swim in the muddled water with the sharks, fighting surival or make them your foe/clansmanship. The more they have to lose, the more knee-jerk reactions they are going to get, which explains all these rule changes at every whim from the elites that affect the rest of the lower level. Skaters interests are always placed last.

The only way to really re-enforce good judging is to ensure judges are not affiliated with any federations, and their loyalty should be to the sports and athletes, more than the federations and ISU. Also, they need to be provided with better judging aids, that can measure and account for factual sporting performances (height, distance, speed, trajectory, coverage etc..) instead of relying on impressions or crowd reactions...

COP need to implement stronger automatic deduction via software not human 'options' for badly performed programs, even just through showing indicators or relative score comparison from skater's previous programs/performances/outing and the score they received during marking. These are useful aids for judges to work to a consistent 'quality assured' standard.

 

[gkelly]

If you were an official judge, what do you think you would be best at and what would you prefer? Would they be the same?

*Technical panel calls (start a separate thread for tech panels?)

*Grades of execution

*All program components

*GOEs and PCS at the same time

*GOEs and PCS equally, but more accurate if only judging one or the other

*Performance/Composition/Interpretation only

*6.0-style big picture scoring and ranking skaters, with knowledge that the referee might ask questions that put you on the spot in the post-event meeting about how specific technical aspects affected your decision

*Elite skaters only

*All senior and junior level skaters

*Lower level skaters only

*All levels

*Large events

*Six or fewer skaters at a time


How much have you tried filling any of these roles in a simulation of what it would be like to judge/score in real time without knowing results?

 

[Mathman]

I’m probably more fascinated by program components than GOEs, but we can talk about both here.

If you were an official judge, what do you think you would be best at and what would you prefer?

*Performance/Composition/Interpretation only

I do not have the expertise to contribute to the thread, but I would like to mention the part that is most interesting to me. I would like to have a clearer understanding of what judges are, or ought to be, looking for in making judgments about

"Physical, emotional and intellectual involvement in the translation of the intent of the music."

"Every step, movement,all its parts, big or small, seem necessary to the whole, and there is an underlying vision or symbolic meaning that threads together the entire composition."

For instance, is it better choreography if the skater portrays a character like Sinbad fighting off the pirates in Scheherazade, or should he rather concentrate on matching figure skating elements to musical structure and phrase?

I confess that I am at the stage of, I don't know anything about choreography, but I know what I like. For instance, I like it just fine when a skater's "choreography" consists of a blazing display of technique: Here's my triple Lutz -- Kapow! Here's my inside spread eagle -- Ta-da!"

 

[gkelly]

I would like to have a clearer understanding of what judges are, or ought to be, looking for in making judgments about

"Physical, emotional and intellectual involvement in the translation of the intent of the music."

That is one of the less technical criteria!

I just understand it to mean full commitment to the performance, how much the skater is going for it with the whole body vs. holding back; how much do they look like they care about what they're doing (even better if they're showing specific emotions connected to the music and those emotions evolve throughout the program); how much do they look like they understand what they're doing and why

Facial expression would probably fit in there more than for most other criteria

Back in 6.0 days people used to talk about skating with "attack" -- I don't hear that so much lately, but it seems this is where that would best fit in, although that implies a more aggressive approach than may be appropriate for softer music

"Every step, movement,all its parts, big or small, seem necessary to the whole, and there is an underlying vision or symbolic meaning that threads together the entire composition."

For instance, is it better choreography if the skater portrays a character like Sinbad fighting off the pirates in Scheherazade, or should he rather concentrate on matching figure skating elements to musical structure and phrase?

You'd probably get some different answers from different experts depending on what's most important to them. The guidelines don't rank the different criteria within each component, but there might be a general consensus about what's most valued.

My take is that matching the movement to the musical structure and phrase is expected/fundamental to the Composition and Interpretation components. Largely because precise timing and nuanced expression require greater control of the technique to pull off. And that telling a story (perhaps an abstract one, e.g., a generalized theme of "growing up" or "grieving" or "falling in love" or "conquering obstacles" -- which could be represented by slaying the technical challenges of the program) or portraying a character is a plus but not required. Both would be great, but if a skater can only have one or the other, following the music closely would be more valued than telling a story that might reflect the idea behind the music or its source material but that doesn't actually involve moving to or with the music.

 

[Izabela]

Would this be a good start to understand what judges are (or should be) looking for in each category under PCS? It's a document USFSA published way back in 2004 which I always use even to this day. I think it's still relevant despite different changes of rules.

To answer Mathman's question though, it doesn't have to be either-or. You could portray a character (and make lines/movements to that affect) while keeping the music phrasing in mind. The first one had to do with your performance and interpretation skills, the other your mastery of technique (blade control, how you can control your upper body by not rushing your movements and showing commitment to the movements you make, how well trained you are in terms of musicality) to be able to pull it off. So yes, I agree with gkelly here that the ability to listen to the music and have your movements in line with the phrasing and structure of the music should come first and is more fundamental than just simply portraying a character on ice. It's easy to throw in movements on ice to portray the character of "Sinbad fighting off pirates in Scheherazade" but more difficult to do this in line with the timing and structure of the music.

On the other hand, it's also not a good choreography IMO when skaters only hit the notes without purpose (as in, they raise their hands to follow the beat of the music, and drop it to follow the next beat). I believe choreography/composition is more concerned about how you can make your every movement/every element you put in there as necessary part to make a whole (although may not be achievable under CoP, so we can say as much as possible). Everyone is given the same materials to build a house, but the difference is how convincing you are that the one you're making is actually a house, and then how beautiful that house is. To quote the document, IMO this should be the most important one when judging choreography:

Unity – purposeful threading of all movements
A program achieves unity when: every step, movement, and element is motivated by the
music. As well, all its parts, big or small, seem necessary to the whole, and there is an
underlying vision or symbolic meaning that threads together the entire composition.

You can set a mood or communicate a purpose in the beginning part of a program (or in some parts of it); but it's a different achievement all together to maintain that mood/communicate that purpose throughout the entire program. I believe that should mark the difference between great choreography and the good (and the really, really bad) and I believe as judges, once you get exposed to different programs as years go by, can actually detect this.

 

[gkelly]

Oh, another thing that's not explicitly stated in the rules but that I think is important is for the musical interpretation or thematic or storytelling or character-developing movements to happen as much as possible with the whole body and with the blade work and during elements, not primarily during posing sections and arm gestures while stroking.

Although at least posing and arm gestures are something, so that would be a step up choreographically from a "program" that just consists of one element and then another one and another with nothing to tie them to each other or to the music.

 

[CanadianSkaterGuy]

Good judging = scoring the same for a skater's elements as if that skater were skating with a paper bag over their face, and scoring a skater's PCS based on the performance they put out that day and no competition prior to that.

Good judging is strict and takes into account errors, and penalizes skaters equally and rewards them accordingly without paying heed to what other skaters have done and what skaters have yet to skate.

Good judging rewards strong skating and elements regardless of how little your experience and popularity is, and punishes errors and sloppy performances regardless of how extensive your experience and popularity is.

Good judging will also give perfect scores only to those that are truly flawless and exceptional in their performance.

 

[Mathman]

Here is one of the best-received programs ever, Liz Manley at the 1988 Calgary (“Stampede”) Olympics. Is this good choreography? Does it have a unifying theme or mood, other than “Here we go, catch me if you can!”

To me, the timing of the jumps to the music is outstanding, mainly because the jumps themselves are wonderfully executed. The blend of “slow part” music into the razz-ma-tazz is well done, although Manley was not a lyrical skater. The program had a satisfying esthetic balance (although I am not 100% what this means).

How would this program be evaluated against the modern PCS criteria?

https://www.youtube.com/watch?v=hV179v0zjkM

 

[gkelly]

Keep in mind that 3 or 4 different musical selections showing different musical styles/rhythms/moods without necessarily any unifying connection between them was pretty much par for the course in 1980s long program construction. Katarina Witt skating to music from Carmen and also trying to portray the character/tell the story was cutting edge at the time.

By the 1990s, programs with unified themes were more common and still are. But not required.

Only since IJS, as far as I know, has there been any official language explicitly referencing "purpose" or "unity." As of 2001, the second-mark criteria were
A. Harmonious composition of the program as a whole and its confirmity with the music chosen
B. Variation of speed
C. Utilization of the ice surface and space
D. Easy movement and sureness in time to the music
E. Carriage and style
F. Originality
G. Expression of the character of the music
(plus Unison for pairs)

So what constitutes harmonious composition of the program as a whole and conformity with the music chosen? If you've chosen several different pieces of music, do they make some kind of sense together, even if just to say "This program portrays the many moods of Skater" or "This program showcases Skater's versatility"?

Similar considerations for a medley approach might be in play as when a producer determines the most effective order of acts in a variety show (including an ice show consisting of multiple independent acts).

Ultimately it's the actual movement on the ice and not the choice of music that's being evaluated. The musical choices and arrangement are just the first step -- what's more important is what the skater (and the choreographer) does with them.

So each judge would have to decide for themselves how harmoniously composed this performance of this program was and how well it conformed to the music chosen. Or in IJS terms, how much the movement to this music demonstrates an intentional design and underlying vision -- whatever that intention and vision might be.

If as a judge you find the blend of slow music into razz-ma-tazz to produce a satisfying aesthetic balance, then you can reward that as an effective structure that was effectively executed by the skater. If someone else finds it less effective, they could give a lower score.


The document that Izabela linked referes to "the principles of proportion, unity, space, pattern, structure, and phrasing." Judges might be introduced to these principles in judging seminars. Judges who have a background in visual or performing arts (either as participants or from an "appreciation" or theoretical perspective) might have studied these principles in more depth in off-ice contexts. Judges who haven't studied arts and humanities in their off-ice lives and rarely consume high culture or think about pop culture theoretically might have no more training in these principles than an occasional seminar, the ISU components videos, and conversations with other judges in the judges' room -- less than artistically minded fans.

Now they are given explicit guidelines to consider these principles, so they should be making more effort now than 30 years ago, but they can each only do so at their own level of understanding.
A judge who is also a professional painter would probably focus on different aspects of proportion, unity, space, pattern, structure, and phrasing than a judge who is also a professional violinist (e.g., "space" might be more salient to the former and "phrasing" to the latter). Both would probably bring more attention to these principles than a judge who is an engineer or a lawyer. But an engineer or lawyer might bring different expertise to other aspects of judging that are not the artists' strengths.

 

[CanadianSkaterGuy]

Good judging is when a badly badly badly badly badly badly badly, seriously seriously seriously, severely severely severely bombed program doesn't get just 5 points below the perfect PCS score.

Agreed. But define severely bombed... how many falls are we talking? In the past, some skaters have fallen 4-5 times and gotten PCS scores even above 9.00 in some categories from some judges.

That's not just bad judging. That's like the worst of the worst judging.

 

[labgoat]

Here is one of the best-received programs ever, Liz Manley at the 1988 Calgary (“Stampede”) Olympics. Is this good choreography? Does it have a unifying theme or mood, other than “Here we go, catch me if you can!”

To me, the timing of the jumps to the music is outstanding, mainly because the jumps themselves are wonderfully executed. The blend of “slow part” music into the razz-ma-tazz is well done, although Manley was not a lyrical skater. The program had a satisfying esthetic balance (although I am not 100% what this means).

How would this program be evaluated against the modern PCS criteria?

https://www.youtube.com/watch?v=hV179v0zjkM

The best thing about Liz was the solid jumps with speed, height and amplitude. Her spins were speedy and controlled. Her footwork was quick, precise and musical. Her enthusiasm and love of skating was palpable. All of these qualities should be marked forward as Mr. Button would say.

 

[yume]

If you were an official judge, what do you think you would be best at and what would you prefer? Would they be the same?

*Technical panel calls (start a separate thread for tech panels?)

*Grades of execution

*All program components

*GOEs and PCS at the same time

*GOEs and PCS equally, but more accurate if only judging one or the other

*Performance/Composition/Interpretation only

*6.0-style big picture scoring and ranking skaters, with knowledge that the referee might ask questions that put you on the spot in the post-event meeting about how specific technical aspects affected your decision

*Elite skaters only

*All senior and junior level skaters

*Lower level skaters only

*All levels

*Large events

*Six or fewer skaters at a time

Grades of execution. Because they are the easiest marks to give. There are just a few, clear and precise bullets. Just by just following the criteria you know what a jump or a spin worth. Simple, when you're honest. And i'm honest

 

[Ladskater]

Well I'm certainly not a FS judge, but I do recall the incomparable Toller Cranston, once declaring "the judges made deals in the back room before the competition took place" or something to that effect. I do think judging has improved in this area. It does seem to be more above board, and more accountability with the judging than in the 6.0 days and days of caddy girls holding up the marks after the skater finished his/her program. To answer your question this past Olympics was an example of good judging.

 

[Mathman]

Thanks to labgoat and gkelly for the renarks about Liz Manley's program.

As of 2001, the second-mark criteria were
A. Harmonious composition of the program as a whole and its confirmity with the music chosen
B. Variation of speed
C. Utilization of the ice surface and space
D. Easy movement and sureness in time to the music
E. Carriage and style
F. Originality
G. Expression of the character of the music

Of these, I think the performance was excellent ion criteria B, C, D, E and G, and good on criterion A.

Here is another one from the same era. This is Paul Wylie at the 1992 Olympics. American fans and commentators loved it, in part because of Wylie’s backstory as a perennial underdog “nice guy.” (He is still a nice guy.) Looking at the judges’ scores, though, there does not seem to be any national bias pro or against, compared to Viktor Petrenko who got 8 out of 9 first place ordinals, Canada dissenting.

https://www.youtube.com/watch?v=8nLhA7CVIok

Looking back, I think Wylie was overscored on the technical mark. In the modern judging system he should have got negative GOE or at most 0 on almost every jumping pass (+1 for the triple toe out of steps late in the program). Lovely moves in the field. The choreography and musical interpretation held together but the total effect was diminished by little lapses in technical precision. (Should this be reflected in the second mark/PCS?)