Pre-comp judging: inevitable or unfair?

[Eclair]

I often wish that the judges would score the skaters according to what they actually did in competition and not according to their potential.
Some commentators once said, that judging the levels in the step-sequences is so complex und challenging, that judges look at them in practice beforehand and then just adjust their scores in the real competition according to whether the execution was better or worse than in practice. While I understand that this might be inevitable with step-sequences, I would like to question if this is also necessary for jumps, spins and PCs. There're two issues I would want to hear your take on:

1. My understanding is that judges attend practices and afterwards have an idea of what to expect in competition, thus giving out e.g. PCs according to whether the performance was better or worse than in practice. This way, it's harder for a skater to break out of a pre-competition formed range of PCS or even GOE's.

2. We often see judges talking to each other during practice sessions. While I understand that if one is not sure about something or just want to hear another opinion, discussion can be productive and result in a better understanding of e.g. the program. But there's a thin line between constructive discussion and manipulating the other one. Is it really necessary for judges at practice sessions to discuss with other judges? Can't they sozialize and connect with each other during dinner/ gala/ buffet = not during they are giving out scores, even if they may not be the final scores?


I know this may be hard to realize, but do judges really have to attend practices? They can be given a scoresheet of what the planned element are and see the execution in competition for the first time. This way they're more prone to judging what they actually see and not what they expect. Also, this way, judges can arrive later, therefore cutting hotel costs for the host and saving time for the judges. What do you guys think?

 

[jimini]

Some commentators once said, that judging the levels in the step-sequences is so complex und challenging, that judges look at them in practice beforehand and then just adjust their scores in the real competition according to whether the execution was better or worse than in practice. While I understand that this might be inevitable with step-sequences, I would like to question if this is also necessary for jumps, spins and PCs.

The levels are determined by the technical panel, not the judges. The judges probably know what the difficulty is, but it's not their responsibility to do the "bean counting" during competition. For the technical panel, it's to their advantage to know what the skater is planning to do in order to reduce the amount of replays they need to do to count levels.

I would say there are a couple key benefits of the judges watching routines pre-competition:

1. Ensures they know where all the key highlight elements are and where all jumps are positioned, so that they're not looking down at the wrong moment while writing/typing a mark/note.
2. Flipside to #1 - if they do happen to miss a big artistic move or something because they're having to write/type something about the previous move, they also don't forget the skater does it so it will still get reflected within PCS.
3. Allows the judges to see skaters on the ice together for comparison. It's sometimes hard to notice a skater's relative skating speed, relative flow, posture, effortlessness/lack of setup (or vice versa), spin speed, jump height, jump distance, etc. in isolation. But if you see multiple skaters on the ice at once, you start to notice the minute differences between them. This can have a big impact on PCS.

(Side note: Point #3 was even more important pre-IJS/COP, since the artistic mark was always a "ranking" of skaters.)

 

[gkelly]

I often wish that the judges would score the skaters according to what they actually did in competition and not according to their potential.
Some commentators once said, that judging the levels in the step-sequences is so complex und challenging, that judges look at them in practice beforehand and then just adjust their scores in the real competition according to whether the execution was better or worse than in practice.

Can you point to what the commentators actually said about the step sequence levels and who looked at them in practice?

The technical specialists might watch step sequences in practice to get a sense of what level the skater is aiming for -- what features they are trying to include, especially the variety of steps and turns, since it's most challenging to count those all in real time. It's technical specialists and controllers who look at the levels, not judges.
Did the commentator actually say "determining the levels in the step sequences is so complex and challenging, that technical specialists look a them in practice beforehand"?

Judges don't have to know the levels of the step sequences. They're not shown levels during or after the program, and it's not their job to figure it out themselves.

Judges' job is to determine the Grades of Execution. For step sequences, they do not have to count twizzles and rockers etc. the way the tech panel does. All they have to do is determine how well the skater fills the following positive criteria add to the GOE:

1) good energy and execution
2) good speed or acceleration during sequence
3) use of various steps during the sequence
4) deep clean edges (including entry and exit of all turns)
5) good control and commitment of the whole body maintaining accuracy of the steps
6) creativity and originality
7) effortless throughout
8) element enhances the musical structure

and look for the following negative criteria to reduce the GOE:

SP: listed jumps with more than half rev. included
Fall
Less than half the pattern doing steps/turns
Poor quality of steps, turns, posiitons
Stumble
Does not correspond to music

The criterion that I boldfaced is closest to what the tech panel also looks at and most likely to benefit from watching what the skater plans.

There's no need for judges to count exactly how many of each kind of turn the skater does -- the tech panel will take care of that -- but getting a general sense that the skater is using some difficult turns and both backward and forward, clockwise and counterclockwise, inside and outside turns might be useful to know in advance even if the judge does not have to keep track if the skater has included the right kind of variety to achieve a higher level.

E.g., a skater could do all their difficult turns counterclockwise and forward-to-backward, and only turn clockwise or back to front with steps or easy turns and still meet the level 3 guidelines for the tech panel, but a judge could consider that lacking in variety.

Conversely, a skater could plan the required content needed for level 4 and make one little unobtrusive mistake that invalidates one turn or cluster that was needed, only get called at level 3, but still impress the judges with the variety of steps and turns as well as enough other performance-related qualities to earn a +3 GOE.

The rest of the criteria are all about how the skater actually performs during the program, so there's no point in keeping track of what they do in practice.

Back in 6.0 days, there was no tech panel, so then judges were responsible for keeping track of the difficulty and variety as well as the quality of performance, and therefore it would have been even more useful for judges to watch in advance. But there weren't specific rules about exactly how to reward the difficulty that they saw.

I would like to question if this is also necessary for jumps, spins and PCs. There're two issues I would want to hear your take on:

1. My understanding is that judges attend practices and afterwards have an idea of what to expect in competition, thus giving out e.g. PCs according to whether the performance was better or worse than in practice. This way, it's harder for a skater to break out of a pre-competition formed range of PCS or even GOE's.

I think it's possible for judges to get a sense of the level of skating skills (speed, effortlessness, edge quality, multidirectional skating, etc.), so that might give judges who watch practices a sense that this skater seems to be capable of 5 or 7 or 9 level of Skating Skills. Of course, if they have an especially good or an especially bad performance, the decimal places will reflect that, and if s/he's much better or much worse in competition than in practice, that could bump the score up or down a whole point or more from what the judge had anticipated.

Watching practices could be very useful for taking note if a skater is planning a difficult transition, an unexpected entry into an element, an interesting exit or quick transition from one element to the next. Then the judge can be on the lookout for those transitions during the performance and be sure not to look down just at the wrong time and miss something important that should factor into the Transitions score.

Some aspects of the Composition component (e.g., travel patterns and element placement) could be hard to keep track of while also assessing everything else, so it might be useful for a judge to watch a runthrough with only that in mind so they go into the competition already knowing what to expect from this program in terms of spatial layout.

But that assumes that the skater actually performs a full runthrough in the practice that the judge watches.

Often during practice sessions skaters only skate part of their programs when their music is played. They might start later or end earlier in the music, or they might take a break in the middle, for various reasons.

Sometimes they intentionally leave out or water down some elements so they can focus on performance quality and end up doing better on component areas during the practice than they can in competition with all the difficult content.

Other times they might spend a practice runthrough

2. We often see judges talking to each other during practice sessions. While I understand that if one is not sure about something or just want to hear another opinion, discussion can be productive and result in a better understanding of e.g. the program. But there's a thin line between constructive discussion and manipulating the other one. Is it really necessary for judges at practice sessions to discuss with other judges? Can't they sozialize and connect with each other during dinner/ gala/ buffet = not during they are giving out scores, even if they may not be the final scores?

If they saw something relevant during a practice session, they could mention it to other judges afterward in the judges' room, out of sight of skaters, fans, and media, so we'd never know what they said, just that it wasn't at a time when they could directly point it out and say "watch now, he's about to try it again."

As you say, there's a distinction between constructive discussion and manipulation and sometimes the exact same words could be interpreted as one or the other depending on tone of voice or nationality of the skaters or judges involved.

I know this may be hard to realize, but do judges really have to attend practices? They can be given a scoresheet of what the planned element are and see the execution in competition for the first time.

What the elements are is the least of what they need to judge during the competition.

Even if there were a place on the scoresheet to indicate that a skater plans a difficult entry or exit to an element or quick connection between elements and give some warning when judges should not glance away to write notes or input scores, it would be practically impossible to notate exactly how difficult or surprising or close together the various moves are. If a judge sees it for the first time in competition and isn't sure exactly what they saw because it was so unusual, they can't properly evaluate that aspect of the GOE or Transitions score.

At major events where judges have access to video replay, they may be able to go back and confirm entrances and exits -- I'm not sure how much control judges have over what they can rewatch after the performance. If there's a cool creative move or variation during an element, they should be able to rewatch that to figure out what it was -- maybe not if there was a cool creative highlight move between elements.

But more to the point, there's no way to give judges a scoresheet that notates in advance that Skater A's program is mostly circling around counterclockwise to set up jumps, maybe a few straight lines, and puts two of the spins in exactly the same spot on the ice and the other one at the mirror image point on the other end, or that Skater B's program is full of complicated patterning in both directions and unexpected twists and turns. They have to see that for themselves.

So if a skater is doing a lot of creative stuff not only with the elements but also with the choreography, it only helps the skater to let judges know what to expect.

If the choreography is simple and predictable, well, maybe those skaters would prefer not to let judges figure that out in advance and hope that they'll be too distracted by the elements and the music during performance to notice.

 

[Barb]

If determining levels is so complex and challenging, why are there only 3 persons in the technical panel vs 8-9 judges?

 

[Violet Bliss]

If determining levels is so complex and challenging, why are there only 3 persons in the technical panel vs 8-9 judges?

I think determining levels requires total focus as elements happen fast. Watching practice helps in seeing what to expect. But it's not really that complex or flexible because there are strict quantitative criteria to go by and 3 persons, with one taking part only when there is a disagreement, is quite sufficient, especially with the aid of video replay when necessary. Judging on the quality of execution on each element, OTOH, may be subjective even with bullet points, thus requiring a large panel to get a general consensus.

 

[Eclair]

Can you point to what the commentators actually said about the step sequence levels and who looked at them in practice?

I doubt there's any footage, since Eurosport only storages it for 1 week, but it was in context of ice dance step sequences, when the commentators said something along the line: The step sequences in ice dance are so complex that of course, the judges don't judge the levels during competition. They look at it at practices and take notes and then at competition, they look at whether it was executed better or worse than in practice. That's all I can remember. I think they probably meant the Technical specialist look into the levels at practice and the judges look into GOE's at practice.
But my point is, that since judges and not only technical specialists attend practice, they too, probably think about what GOE's and PCS they'd give skater A or skater B and then in actual competition awards them a little bit more or less of the score they thought about.

I think it's possible for judges to get a sense of the level of skating skills (speed, effortlessness, edge quality, multidirectional skating, etc.), so that might give judges who watch practices a sense that this skater seems to be capable of 5 or 7 or 9 level of Skating Skills. Of course, if they have an especially good or an especially bad performance, the decimal places will reflect that, and if s/he's much better or much worse in competition than in practice, that could bump the score up or down a whole point or more from what the judge had anticipated.

Here's where I see the problem. They see skaters at practice and see how skater A has better SS and speed than skater B. So, in their mind, they have a pre-formed ranking. Then at competition, skater A is slower than usual and makes a few mistakes, while skater B skates the lights out and is now as fast as skater A, plus projects more. Often, I feel the PCS for skater A and skater B then turns out to be: skaters A gets a little bit higher PCs than skater B, because the judges gives skater A a little less PCs than they'd have given in practice and skater B a little bit higher PCs than they would have in practice. But it would be more fair to give skater B the same or slightly higher PCs than skater A, since even though usually skater B potential is lower than skater A, in actual competition he/she was better.

Watching practices could be very useful for taking note if a skater is planning a difficult transition, an unexpected entry into an element, an interesting exit or quick transition from one element to the next. Then the judge can be on the lookout for those transitions during the performance and be sure not to look down just at the wrong time and miss something important that should factor into the Transitions score.

That's definitely true, but since skaters often don't do full run-throughs anyways, the impression won at practice whether a program has lots of transitions or lack of it can also be a false one and is not really that reliable. Replaying it at the actual competition might be a more accurate way to look for those, especially since judges are not going to pay equal attention to trickiness of transitions of every skater equally during practice. And I think most of the time, it's not really how many or how tricky the transitions, the higher the PCs. It's the better the SS, speed and performance, the higher the PCs.

Some aspects of the Composition component (e.g., travel patterns and element placement) could be hard to keep track of while also assessing everything else, so it might be useful for a judge to watch a runthrough with only that in mind so they go into the competition already knowing what to expect from this program in terms of spatial layout.

But do judges really have to be present and watching at practice to get a feel for travel patterns and element placements? Don't they get that when they see the program in competition anyways? Especially since I feel right now, judges don't take time to assess element placement and program construction anyways, but just go with their impression. And then, I'd rather have them score according to their impression in actual competition than in practice.

If they saw something relevant during a practice session, they could mention it to other judges afterward in the judges' room, out of sight of skaters, fans, and media, so we'd never know what they said, just that it wasn't at a time when they could directly point it out and say "watch now, he's about to try it again."

Sure they could, but in practice they have hours to talk and exchange opinions on skaters, while afterward in judges room the discussion surely will be less. Which is, what I think is more favorable, though I understand that there're legitimate arguments for judges to exchange their views.

As you say, there's a distinction between constructive discussion and manipulation and sometimes the exact same words could be interpreted as one or the other depending on tone of voice or nationality of the skaters or judges involved.

And that's why I think that discussion about specific skaters should be minimized. When judges talk about what's favored, it's best to keep it general with examples of earlier competitions underlining and clarifiying what's meant. I'm in no way against judges discussing what good skating is about, I'm just against it while they score/ rank skaters - whether in actual competition or in their minds pre-comp.

What the elements are is the least of what they need to judge during the competition.

Even if there were a place on the scoresheet to indicate that a skater plans a difficult entry or exit to an element or quick connection between elements and give some warning when judges should not glance away to write notes or input scores, it would be practically impossible to notate exactly how difficult or surprising or close together the various moves are. If a judge sees it for the first time in competition and isn't sure exactly what they saw because it was so unusual, they can't properly evaluate that aspect of the GOE or Transitions score.

At major events where judges have access to video replay, they may be able to go back and confirm entrances and exits -- I'm not sure how much control judges have over what they can rewatch after the performance. If there's a cool creative move or variation during an element, they should be able to rewatch that to figure out what it was -- maybe not if there was a cool creative highlight move between elements.

I don't know how much control judges have over replay and alike during competition either, but I don't think it's technically that challenging to enable judges watching whatever element or transition or move they want.

But more to the point, there's no way to give judges a scoresheet that notates in advance that Skater A's program is mostly circling around counterclockwise to set up jumps, maybe a few straight lines, and puts two of the spins in exactly the same spot on the ice and the other one at the mirror image point on the other end, or that Skater B's program is full of complicated patterning in both directions and unexpected twists and turns. They have to see that for themselves.

If judges would give out choreography scores more according to choreography and less according to the other aspects of PCs, then I would agree that this would be a great argument for judges to watch practices. But since I feel choreography rarely gets scored accurately, I wonder if it's sufficient for the judges to just see those things in competition. I feel, they score program composition and choreography according to their 'feeling' and impression anyways.

So if a skater is doing a lot of creative stuff not only with the elements but also with the choreography, it only helps the skater to let judges know what to expect.

I'm absolutely for skaters to get more credit for thoughtful choreography and creativity. But right now, I just don't see how creativity and choreography gets more credit than oh, that was nice and intricate or that wasn't that nice. A lot of skaters who don't have much transitions or thoughtful, original choreography gets high PCs because of their SS, speed, performance or reputation. While lots of B ranked skaters who're not as good in SS have very thoughtful choreography that doesn't get credit.

If the choreography is simple and predictable, well, maybe those skaters would prefer not to let judges figure that out in advance and hope that they'll be too distracted by the elements and the music during performance to notice.

I think that's already happening because choreography doesn't gets judged as it's own category. I don't think judges scores for choreography would be much different with or without them attending practice.

 

[gkelly]

If determining levels is so complex and challenging, why are there only 3 persons in the technical panel vs 8-9 judges?

The tech panel's job is to determine exactly what the skater did in each element. Some of the elements are so complicated (even moreso when two partners are involved) and/or happen so fast that it can take more than one set of eyes to determine exactly what the skater did.

Replay is available, but it's not practical to have one person rewatch every single element at least once and complicated ones 4 or 5 times in order to see every detail. Thus Tech Specialist and Asst Tech Specialist and Tech Controller each watch a different aspect of the element, and if they're each sure of what they saw in real time they say whether the skater achieved the feature or not, the element is called, and they move on.

Replay is only for elements where there was some question about what they saw in real time. And if there are still questions when watching the replay, the tech panel members discuss and come to consensus. If there were more than 3 people discussing, the review process would take a lot longer without adding more accuracy.


Judges are not deciding simple What or Yes-or-No questions -- they're evaluating the quality of the various elements and program components on a scale of -3 to +3 or 0 to 10. Ultimately these evaluations are opinions, based on thorough understanding of the rules and standards. And then their numerical scores are averaged (after dropping high and low).

Different good judges might have different emphases in what they look for and how they translate those evaluations into numbers. So the more different expert opinions get averaged into the final result, the more fair for all the skaters.

And judges are not allowed to consult with each other during the competition. So there's no question of more judges slowing things down. Each judge has to make all his/her own decisions individually by the time the tech panel finishes their reviews.

 

[mrrice]

From gkelly,The tech panel's job is to determine exactly what the skater did in each element. Some of the elements are so complicated (even moreso when two partners are involved) and/or happen so fast that it can take more than one set of eyes to determine exactly what the skater did.

I have always wondered about this area. When I was judging dance, The Tech/ Execution Judge only judged how well the moves were being done. It was my job to determine what the moves was, determine whether or not it fit the music, and it's worth on a basic scale. In dance, the execution judge is the person who gives the GOE number. A triple pirouette will get a great basic score. However, If the execution judge see's 5 dancers fall off of "Relevette" their score will drop after it's tabulated. I hope that made sense.

 

[Barb]

I know this may be hard to realize, but do judges really have to attend practices?

At this point I think we all understood that some judges or technical especialists need pre-comp analysis. So, if they do not attend practices they will watch videos of previous competitions or practices anyway. And they can discuss it at other places too. When we see a program many times usually we note mistakes or details, so maybe those judges attending practices are the ones who really try to be more accurate.

 

[gkelly]

I doubt there's any footage, since Eurosport only storages it for 1 week,

I was hoping it might have been something that someone put up online. Oh well.

but it was in context of ice dance step sequences, when the commentators said something along the line: The step sequences in ice dance are so complex that of course, the judges don't judge the levels during competition. They look at it at practices and take notes and then at competition, they look at whether it was executed better or worse than in practice. That's all I can remember. I think they probably meant the Technical specialist look into the levels at practice and the judges look into GOE's at practice.

Yes, probably the latter. The judges never judge the levels, the tech panel does. And they have more to look at with dance teams or pairs than with single skaters.

The judges may look at step sequences to see all the GOE bullet points in practice (where relevant -- they won't see how it relates to the music if the skater's music isn't playing at the time) -- they shouldn't be using that to decide on their scores in advance, just to know what to look for in competition.

But my point is, that since judges and not only technical specialists attend practice, they too, probably think about what GOE's and PCS they'd give skater A or skater B and then in actual competition awards them a little bit more or less of the score they thought about.

Part of that thought process might involve automatic thoughts about what GOE they would have given to the element as performed right then in practice -- it probably becomes a mental reflex to note the various bullet points and turn that into a number. And the next time the skater/team practices the same element a few minutes later, the judge might have different thoughts if the skater/team performs it differently.

They shouldn't be trying to remember their practice thoughts and refer to them during the competition. They have a lot of skaters to watch, not just the medal contenders. There's no way they could remember every element they saw in practice and what their thoughts about scoring were during the practice unless they took notes. Which they're not supposed to do and are not allowed to bring with them to the competition.

A few especially good or especially problematic practice elements might stick in their heads though.

Here's where I see the problem. They see skaters at practice and see how skater A has better SS and speed than skater B. So, in their mind, they have a pre-formed ranking. Then at competition, skater A is slower than usual and makes a few mistakes, while skater B skates the lights out and is now as fast as skater A, plus projects more.

Are these skaters who are close in general ability, but A is usually a little better than B? Or are they skaters at completely different skill levels?

Maybe in practice a judge thinks A is a well-rounded skater who usually deserves scores around 8.0 and B is similarly well rounded at more of a 7.0 level. Then A has an off day in competition and the judge gives them 7.5 for Skating Skills, and B skates lights out and the judge also gives 7.5. And then because A had several small breaks in the performance and B projected more than A, maybe some judges will go up to 7.75 or higher for B’s PE component and down to 7.25 for A’s.

But what if A has a lot of good qualities that B just hasn't mastered yet and A is able to show many of those qualities during the performance despite slowing down and making a few visible errors, then A's good points might still add up to more than B's good points even on B's best day. Maybe A has quieter turns and smoother flow and more multidirectional and one-foot skating in his program, and those things are still true even when he slows down on a bad day. Maybe B’s program relies a lot on crossovers to regain speed, even one or two here and there in the middle of the step sequence between elements, and there are almost no clockwise turns or one-foot back-to-forward turns outside the step sequence, and those things are still true even B’s best day. (And similar pervasive strengths for other components, or unison and spatial awareness throughout the program for pairs and dance teams.)
So the mere fact that B skated at the top of his potential that day and A skated below potential doesn't necessarily mean B deserved higher scores than A. That's a judgment each judge has to make for herself while watching the competition, regardless of whether or not she attended practices the day before. Other judges, or fans, might reach different conclusions. That’s why a larger judging panel is advantageous, as far as costs allow.

I don't have strong feelings about whether judges should or shouldn't watch practices. In my earlier post I pointed out some areas where watching practices might be helpful for evaluating the competition-day performances more accurately. You’ve offered some concerns about how it might interfere with judges evaluating only what they see in competition. If both are true, at least sometimes, then it’s a tradeoff as to which is more common and has more effect. I don’t know the answer to that.

That's definitely true, but since skaters often don't do full run-throughs anyways, the impression won at practice whether a program has lots of transitions or lack of it can also be a false one and is not really that reliable.

It’s definitely true that they often don’t do full runthroughs.

If a skater is working on a difficult transition into an element or between elements, especially if it’s a difficult element, they will probably practice it in isolation during the practice session. And they may well practice it in the 6-minute warmup during the competition, when judges are sitting right there. If so, some judges might see it and others might be looking the other way. Do you think it’s a good or bad thing if a judge who sees it during the warmup thinks “Wow, that looked like a cool entry into the axel. I hope he can pull it off in the program. I’ll make sure to look for it and see if I can figure out exactly what he was doing.”? If it’s bad, do you want to find a way to prevent judges from looking at the ice during the warmups?

Replaying it at the actual competition might be a more accurate way to look for those,

That would work for individual elements, including entries and exits if judges have control over when to start and end the replay segments. (I don’t know how it works. I do know that tech panels see each element separately and the video operator may not mark the clips early enough to include the whole entry.)

It wouldn’t help for larger structural issues about program construction. Which they should be able to see during the performance, but it’s one thing that is more likely to be consistent from one runthrough to another, as opposed to the success of elements for example.

especially since judges are not going to pay equal attention to trickiness of transitions of every skater equally during practice.

However, if a lesser known skater has a tricky transition or variation, showing it off in practice is one way to start getting attention from judges to put them on a more equal footing with the favorites.

I'm in no way against judges discussing what good skating is about, I'm just against it while they score/ rank skaters - whether in actual competition or in their minds pre-comp.

So then the question is: What exactly do judges do when they attend practices? Do they think in terms of specific scores and comparisons between skaters and do they compare observations with each other? That’s what you’re worried about, and if that is in fact the bulk of what judges do at practices, then there is good reason for concern.

If the bulk of what they’re doing is getting a sense of the overall quality of the field (which was more important with 6.0 than with IJS judging), and looking for tricky moments or subtle details that would be easy to miss if they never saw them before the program in competition, and familiarizing themselves with the skaters they don’t already know so that the stars and veterans don’t have a built-in advantage over the newcomers, and pre-scoring represents little or none of their thought processes, then there would be reason to welcome it.

I just don’t know what’s really going on in judges’ heads.

 

[gkelly]

I have always wondered about this area. When I was judging dance, The Tech/ Execution Judge only judged how well the moves were being done. It was my job to determine what the moves was, determine whether or not it fit the music, and it's worth on a basic scale. In dance, the execution judge is the person who gives the GOE number. A triple pirouette will get a great basic score. However, If the execution judge see's 5 dancers fall off of "Relevette" their score will drop after it's tabulated. I hope that made sense.

It sounds as though the division of labor in the kind of floor/stage dance judging you did was completely different from the way IJS divides the tasks in skating. So trying to relate what the "technical" judge did in a completely different kind of competition is more likely to confuse than shed light on what skating judges and tech specialists are doing.

The same would be true of gymnastics scoring (which is also divided differently than skating). As I understand, there are some similarities, but also some significant differences.

And of course IJS judging in skating is very different in many ways from 6.0 judging, even though the content of what's being judged may be practically identical. (E.g., in 2004 season, when many of the same skaters competed with the same programs under both systems.)

 

[gkelly]

Apologies for 3 posts in a row.

From a fairness point of view, which of the following do you think would benefit most from judges attending practices and which would do better if judges see them for the first time (or first time this week) on the SP warmup?

Skater C tends to underrotate many jumps and touch down quickly with his free foot. He’s very musical, very expressive to the audience, and skates with great speed and commitment. But there isn’t a lot of content in the programs -- few level 4 or even level 3 elements, not much clockwise skating or difficult transitions. Fans love him and are often bewildered why he rarely scores well.

Skater H has been around for years but is a headcase, and judges never know what kind of day to expect. Watching a practice could give some sense of whether this week will be good or bad, but it won’t be terribly surprising if he has a good practice and them bombs in competition, or struggles in practice and then puts it together in competition. He often practices amazingly creative skills including rare quads, unusual combinations, unusual spin variations or unique single jump variations as transitions or in choreo sequence, surprising connections into and out of jumps and spins -- and just as often leaves them out in competition hoping to go clean, so the only place to see them is in practice.

Skater I has medaled in the past but has been struggling with injury or illness recently, hasn’t competed for about a year, so no one knows what to expect. He’s recovered enough to do all his elements well in isolation, but stamina is a problem so this year’s programs are simpler than in the past. Knowing he’s hasn’t skated a clean freeskate runthrough at home and even the SP is iffy, he chooses only to practice pieces of the program when his music is played during practice and otherwise just works on individual jumps and spins.

Skater J dominated at the junior level last year and is making his senior debut this year. Judges all know who he is, some of them have judged him before and most if not all have seen video of the winning junior performances. They’re eager to see how he compares live to the more experienced seniors.

Skater L is a late bloomer. He finally started landing clean consistent triple axel and quad toe in his 20s and is now making his international debut. Fans and judges in his home country have long appreciated his artistry, but most international judges have never seen him before -- maybe they recognize his name but have low expectations of what he can do.

Skater N is also making his senior debut. He never got much attention as a junior nor competed at Junior Worlds or any JGPs far from home. Some judges from his part of the world know who he is, but most never heard of him before. As of this year he now has several quads in his repertoire -- when clean, they can look so effortless it might be easy to mistake them for triples. He’s been working on his program component skills, which are stronger than last year but not yet up to senior medal standards.

Skater S burst on the scene a few years ago and has been winning medals and acclaim ever since. If he’s in a competition it’s a guarantee his programs will be televised. Casual viewers remember him from the last Olympics. Fans frequently debate his strengths and weaknesses. Judges and referees use them as examples. Maybe there has been some discussion about how his charisma/star quality has led them to overscore some of his components.

Skater V has been competing for years and slowly moving up in the ranks as he matured. He’s pretty consistent, with solid edges, difficult turns etc., and medium speed, but generally shy and introverted. He’s never medaled at a senior championship before, but at his best he’d be well positioned to take advantage of others’ mistakes. And he does have especially well choreographed programs this year, even if he doesn’t have the personality sell them in a flashy way.

 

[karne]

That's definitely true, but since skaters often don't do full run-throughs anyways, the impression won at practice whether a program has lots of transitions or lack of it can also be a false one and is not really that reliable. Replaying it at the actual competition might be a more accurate way to look for those, especially since judges are not going to pay equal attention to trickiness of transitions of every skater equally during practice.

[snip]

I don't know how much control judges have over replay and alike during competition either, but I don't think it's technically that challenging to enable judges watching whatever element or transition or move they want.

That would work for individual elements, including entries and exits if judges have control over when to start and end the replay segments. (I don’t know how it works. I do know that tech panels see each element separately and the video operator may not mark the clips early enough to include the whole entry.)

*poofs into thread like magic*

Hello! Let me explain how the video replay works. The video operator marks the "in" and "out" points and the pieces of video are attached to elements. At the request of the technical panel, the video operator can move the "in" or "out" points earlier or later. Since the video operator is a part of the technical panel, it is their needs that they consider when cutting in and out, not the judges', so an interesting transition into a jump might get cut off or partially cut off. (Personally, I like to cut elements tight; it's the best way to look at them, when you only have the jump itself and no fuss). The judges see the same video cuts that the TP does (though they have less fancy buttons - TP has slo-mo, super slo-mo, frame-by-frame, reverse slo mo, reverse super slo mo, etc, the judges just have "play" "pause" and "back").

Because the pieces of video attach to the elements in the system, video replay of transitions during the program is not possible and not practical.

 

[Eclair]

They shouldn't be trying to remember their practice thoughts and refer to them during the competition. They have a lot of skaters to watch, not just the medal contenders. There's no way they could remember every element they saw in practice and what their thoughts about scoring were during the practice unless they took notes. Which they're not supposed to do and are not allowed to bring with them to the competition.

Okay, that's an important thing to know. I had assumed, that since the commentators spoke so nonchalantly about it as if it was something quite obvious, that the practice of pre-scoring was well established and widely practiced. It's good to read, that it's something judges aren't supposed to do. That changes things.

Are these skaters who are close in general ability, but A is usually a little better than B? Or are they skaters at completely different skill levels?

Well, I pictured two skaters with different strengths, skater A a little bit better in SS and speed, while skater B. is really good in performance, choreography. Both equally good in transitions and edges. Usually, the one with better SS and speed will receive higher PCS, but when he/she underperforms and is slower than usual, I think it would have been fair if skater B received an equally high score or even a little bit higher one than skater A, since their speed is now the same and skater B should get more points for performance/ choreography than skater A for that little bit of better SS. But that rarely happens and I attributed that to the preformed ranking in PCS in the judges minds, because if skater A still gets higher PCs than skater B then skater A was judges based on potential and not what skater A laid on the ice at that day.

Do you think it’s a good or bad thing if a judge who sees it during the warmup thinks “Wow, that looked like a cool entry into the axel. I hope he can pull it off in the program. I’ll make sure to look for it and see if I can figure out exactly what he was doing.”? If it’s bad, do you want to find a way to prevent judges from looking at the ice during the warmups?

I think it's definitely a good thing if the judge notices the cool entry, but not if he thinks during practice: wow, that skater can do an axel with a cool entry and then at competition the skater stumbles, but the judge gives him higher GOE's than a skater who didn't show off that cool entry at practice and had a similar stumble at competition. I believe, both skaters should receive similar GOE's for that jump with similar execution. The skater who can do a cool entry but doesn't do it in competition shouldn't score higher, just because he impressed the judge at practice.

It wouldn’t help for larger structural issues about program construction. Which they should be able to see during the performance, but it’s one thing that is more likely to be consistent from one runthrough to another, as opposed to the success of elements for example.

However, if a lesser known skater has a tricky transition or variation, showing it off in practice is one way to start getting attention from judges to put them on a more equal footing with the favorites.

I'd love for judges to look out more for program construction, because I don't feel they do that very often.
Same story, I think it doesn't matter whether the lesser known skater can do a tricky transition in practice, what matters is whether he does it in competition or not. If judges use practices to look out at competition for that tricky transition, I think that would be very cool. But my concern is, that the information about what the lesser known skater can do in practice, influences the scores in competition, which should not be the case.

So then the question is: What exactly do judges do when they attend practices? Do they think in terms of specific scores and comparisons between skaters and do they compare observations with each other? That’s what you’re worried about, and if that is in fact the bulk of what judges do at practices, then there is good reason for concern.

If the bulk of what they’re doing is getting a sense of the overall quality of the field (which was more important with 6.0 than with IJS judging), and looking for tricky moments or subtle details that would be easy to miss if they never saw them before the program in competition, and familiarizing themselves with the skaters they don’t already know so that the stars and veterans don’t have a built-in advantage over the newcomers, and pre-scoring represents little or none of their thought processes, then there would be reason to welcome it.

I fully agree with the paragraph above. Now that you have pointed out, that judges are not supposed to take notes with them and use practices to pre-score/ rank the skaters mentally, my concerns are not a strong as before. I had assumed, that pre-scoring was an approved and often used tool to help judges score. I still think, what a skater does in practice still affects whether a judge will be harsh or lenient on GOE's on them in competition though.

 

[gkelly]

I had assumed, that since the commentators spoke so nonchalantly about it as if it was something quite obvious, that the practice of pre-scoring was well established and widely practiced. It's good to read, that it's something judges aren't supposed to do. That changes things.

Good.

Well, I pictured two skaters with different strengths, skater A a little bit better in SS and speed, while skater B. is really good in performance, choreography. Both equally good in transitions and edges. Usually, the one with better SS and speed will receive higher PCS, but when he/she underperforms and is slower than usual, I think it would have been fair if skater B received an equally high score or even a little bit higher one than skater A, since their speed is now the same and skater B should get more points for performance/ choreography than skater A for that little bit of better SS.

Are you talking about Skating Skills score or total PCS?

If A is objectively better at Skating Skills than B, even with a less clean and slower performance than usual, then A could still deserve a higher SS score, but not as much higher as usual.

If B is really better at Performance and Choreography (and Interpretation), then B should get higher scores in those components.

Because faster and more confident performance affects the overall perception, it's possible that A will score higher in those areas when A skates faster and more confidently than B, even if Performance and Choreography are not her strengths, but B can score higher in those areas when B skates her best and A has an off day.

Under IJS, judges aren't supposed to consciously balance out the different component scores with each other or with the technical scores (which judges can only partly guess at because they don't know the levels called). So if they give higher Performance scores to A, it's because their perception was that A performed better.

If you disagree, it may be because the judges were unconsciously influenced by their expectations or by skate order. That is always a concern, which judges should be aware of and guard against, but watching practices at the current event is not likely to have more influence than already knowing [some] skaters' history and world standings and short program results.

Or you may disagree because you give more weight to different Performance criteria than most of the judges on that panel, or because you had a different view of the skating than the judges did (Were you watching on video with more closeups and changes of camera angle? Were you sitting on the other side or much higher up in the arena?), or because you were influenced by unconscious effects yourself.

I think it's definitely a good thing if the judge notices the cool entry, but not if he thinks during practice: wow, that skater can do an axel with a cool entry and then at competition the skater stumbles, but the judge gives him higher GOE's than a skater who didn't show off that cool entry at practice and had a similar stumble at competition.

If Skater A does a cool entry into a jump and then stumbles on the landing, the judge might think "+1 for unexpected/creative/difficult entry and clear recognizable steps/skating moves preceding the jump; -2 for stumble; final GOE -1."

If Skater B telegraphs the same jump and stumbles in the same way on the landing, the judge might think "Long preparation -1 to -2; stumble -2; final GOE -3."

That thought process would be the same regardless of whether the judge had watched the practices or not. If the skater does the cool entry in the competition and then stumbles on the landing, he should be rewarded for the cool entry done in competition and penalized for the error on the landing.

A better example might be if the skater had planned and practiced a cool exit from the jump landing leading directly into a spin following. In practice the judge might think that the difficult connection between the jump landing and the spin, if performed as well as it was in practice, would be worth one + on the jump GOE, one + on the spin GOE, and positive consideration on the Transitions score.

But then the skater stumbles on the landing of the jump and doesn't do the cool exit and transition to the spin. So those positive bullet points for jump and spin GOEs would be gone, as would PCS credit for that planned but not executed transition, and the jump would get negative GOE for the stumble. Judges are not supposed to reward something that didn't actually happen in the program during competition.

But a difficult entry to a jump that stumbled on the landing did happen in the program and should be rewarded.

If judges use practices to look out at competition for that tricky transition, I think that would be very cool. But my concern is, that the information about what the lesser known skater can do in practice, influences the scores in competition, which should not be the case.

There might be some unconscious influence, but that would be comparatively minor. And anything that puts unknown skaters on a more even playing field with known skaters is a plus in my eyes.

What could be more useful about watching practices is making judges more aware of details that they might miss in competition if they don't know to look out for them. But if they're on the lookout for Skater Q's cool transition from the jump landing into the spin and remember not to look down right after that jump -- and then Skater Q doesn't do the transition after all, it shouldn't get credit.

I still think, what a skater does in practice still affects whether a judge will be harsh or lenient on GOE's on them in competition though.

It could work against a skater, though, if she does good stuff in practice and leaves it out in competition. Judges could be disappointed and penalize the skater more harshly than if they weren't expecting the good stuff in the first place.

 

[Eclair]

Skater C tends to underrotate many jumps and touch down quickly with his free foot. He’s very musical, very expressive to the audience, and skates with great speed and commitment. But there isn’t a lot of content in the programs -- few level 4 or even level 3 elements, not much clockwise skating or difficult transitions. Fans love him and are often bewildered why he rarely scores well.

I think Skater C would benefit from judges not watching him practice. Of course his UR and two-foots have to be scored accordingly, as well as his levels in spins and steps. But judges haven't spend two days watching him two-footing and under rotating his telegraphed jumps, as well as skating only clockwise and I guess are less hesitant to award him if he finally lays out a clean skate.

Skater H[/B] has been around for years but is a headcase, and judges never know what kind of day to expect. Watching a practice could give some sense of whether this week will be good or bad, but it won’t be terribly surprising if he has a good practice and them bombs in competition, or struggles in practice and then puts it together in competition. He often practices amazingly creative skills including rare quads, unusual combinations, unusual spin variations or unique single jump variations as transitions or in choreo sequence, surprising connections into and out of jumps and spins -- and just as often leaves them out in competition hoping to go clean, so the only place to see them is in practice.

While I enjoy watching unusual variations and personally like to watch practices, I think it would make the scoring fairer if in this case the judges don't consider that he can do those creative combinations. Ideally, the scoring range of that skater would be very big, depending on his form he could be in the top 3 or the top 20. But if his creativity doesn't show in his actual program, but only at practice and he fails to deliver, then I think he should be scored just as low as someone who can't do those variations at all. But if he delivers, then he should of course score higher. What sometimes bugs me, is when skater H doesn't deliver, but still gets less harshly scored than someone who skated equally bad.

Skater I has medaled in the past but has been struggling with injury or illness recently, hasn’t competed for about a year, so no one knows what to expect. He’s recovered enough to do all his elements well in isolation, but stamina is a problem so this year’s programs are simpler than in the past. Knowing he’s hasn’t skated a clean freeskate runthrough at home and even the SP is iffy, he chooses only to practice pieces of the program when his music is played during practice and otherwise just works on individual jumps and spins.

If the judges don't see him at practice, this one is similar to skater H - nobody knows what to expect. the probability is high that with skater I stamina will be an issue, resulting in a few mistakes. Without his lack of stamina being clear beforehand, I think judges are less prone to be overly harsh and those moments of high energy will be noticed as what they are (something positive), instead of the judges constantly thinking: doesn't he look a little tired?

Skater J dominated at the junior level last year and is making his senior debut this year. Judges all know who he is, some of them have judged him before and most if not all have seen video of the winning junior performances. They’re eager to see how he compares live to the more experienced seniors.

I think this one is what I wish would happen with all skaters. Judges are looking at him with an open mind. Not really expecting him to dominate the field, but not excluding the possibility. I often think that first season seniors are scored the most fair. I think in this case, it doesn't matter whether judges see him in practice or not. His shortcomings or well-doings relative to the more experienced seniors will get noticed.

Skater L is a late bloomer. He finally started landing clean consistent triple axel and quad toe in his 20s and is now making his international debut. Fans and judges in his home country have long appreciated his artistry, but most international judges have never seen him before -- maybe they recognize his name but have low expectations of what he can do.

If judges have never seen him before, then they'll probably score him according to what he does on ice. But if they already 'know' him as a B rate skater, then skater L, as someone who has improved on jumping consistency theoretically would benefit from judges watching him practice. In reality, I think, the progress skater L has made will go unnoticed by the judges, even if they watch him practice.

Skater N is also making his senior debut. He never got much attention as a junior nor competed at Junior Worlds or any JGPs far from home. Some judges from his part of the world know who he is, but most never heard of him before. As of this year he now has several quads in his repertoire -- when clean, they can look so effortless it might be easy to mistake them for triples. He’s been working on his program component skills, which are stronger than last year but not yet up to senior medal standards.

I think he would benefit from judges not watching practice. A landed, clean, easy quad is hard to overlook, so that the judges will notice him in competition anyways. They'll also notice his subpar SS, but I guess won't be as harsh as they would be if they had watched him skate on flats for the whole week. As Sandra Bezic once said, the 'magical moment' should be worth something, too.

Skater S burst on the scene a few years ago and has been winning medals and acclaim ever since. If he’s in a competition it’s a guarantee his programs will be televised. Casual viewers remember him from the last Olympics. Fans frequently debate his strengths and weaknesses. Judges and referees use them as examples. Maybe there has been some discussion about how his charisma/star quality has led them to overscore some of his components.

Skater S gets lots of attention from the judges. If he underperforms and still gets high PCS I think that's a case of reputation judging, which is another matter than whether the judges expect him to skate clean or not depending on his practices.

Skater V has been competing for years and slowly moving up in the ranks as he matured. He’s pretty consistent, with solid edges, difficult turns etc., and medium speed, but generally shy and introverted. He’s never medaled at a senior championship before, but at his best he’d be well positioned to take advantage of others’ mistakes. And he does have especially well choreographed programs this year, even if he doesn’t have the personality sell them in a flashy way.

If others' mistakes are punished accordingly and scores not held up by reputation or expectation judging, I think skater V would be in a good position to take advantage of others' mistakes, especially if judges have see him at practice and noticed his intricate programs. The problem is, that I doubt, they would notice it. They would probably focus on those with flashy technical repertoire and overlook the slow, but steady improvements. When the skaters with flashy technical repertoire fails to deliver, they would still be less harsh to him than they would be with other skaters, resulting in skater V not getting the medal and moving up the ranks as he would deserve. But that's a difficult one.

All in all, I think if judges use practice sessions to actually look at the program composition and details of not only the top skaters, but all skaters, practice attendance makes sense. But if they use it as to rank the skaters according to their potential, then it's getting problematic.