Rescoring of 2010 Olympics

Blades of Passion said:

You're supposed to set as accurate of a mark as possible within the objective scale and then reference that as needed. Nobody is getting judged relative to skaters to come, rather they should all be getting judged within the framework of an overall objective scale, but WHILE being mindful of scores that were already given, so as to properly rate each performance in relation to each other. Perhaps there is added difficulty to keeping many performances in mind while assessing each mark, but a great judge should be sufficiently capable of handling this task, and referencing the numbers would only serve to improve judging consistency and mindfulness.

Click to expand...

Out of simple curiosity... What would you do if, say, a person gave a "10" level performance in group 4 according to the scale you have in mind, and then in group 6, someone gives an even better performance, which is now the "10" in your objective scale?

For me, it is simple. I don't judge in real time, and can adjust my scores. This is an entirely genuine question.

Well that's the inherent flaw of the system, right? However, I would say a 10 should be almost impossible to obtain, because there's always something to improve if you want to be picky. So you cap your mark off at 9.75 just in case there is even better yet to come. Realistically, it's just not going to happen that someone gives "the all time greatest performance ever" and then another competitor tops it, and then another competitor tops them, and then yet another competitor is even better than that, etc.

Blades of Passion said:

Well that's the inherent flaw of the system, right? However, I would say a 10 should be almost impossible to obtain, because there's always something to improve if you want to be picky. So you cap your mark off at 9.75 just in case there is even better yet to come. Realistically, it's just not going to happen that someone gives "the all time greatest performance ever" and then another competitor tops it, and then another competitor tops them, and then yet another competitor is even better than that, etc.

Click to expand...

I agree it's an inherent flaw of the system but it didn't seem to me that you, gkelly, baron Vladimir, and CanadianSkaterGuy were even arguing on the same plane of thought and reality. Kind of like postmodernists.

As I said, it's time that is lacking in the judging with the way scoring is supposed to happen. With all the things that are scored, we need to divide things up amongst judges AND confer for a bit AND give them some more time - and maybe still fail. Unlike an art competition, where technique exists simply as a tool for artistry, here technique is also evaluated separately, for an overall result. If we could get a ranking system like 6.0 and combine it with the flexibility of CoP where someone 10th in the short can end up winning, that would be better.

Yes, 10 should be nearly impossible to obtain. But they do exist, and may be updated by the time the next competition comes along. BUT it's a theoretical flaw I picked out when taking about the same competition -a result of the way the system is structured - that can't be fixed unless something like I said happens.

Also, I'll say that it being a spectator sport changes a lot of things. "Ideally", we could just do a bulk review after everyone skates, and take an entire day to get through the results. Not really a thing anyone's going to watch, however.

CanadianSkaterGuy said:

Yes, because a level 1 footwork is easier than a level 4 footwork.

Click to expand...

In 2006, it was possible to do a Level 4 step sequence that would only get Level B today. What counts as difficult has been frequently redefined.

Blades of Passion said:

Well that's the inherent flaw of the system, right? However, I would say a 10 should be almost impossible to obtain, because there's always something to improve if you want to be picky. So you cap your mark off at 9.75 just in case there is even better yet to come. Realistically, it's just not going to happen that someone gives "the all time greatest performance ever" and then another competitor tops it, and then another competitor tops them, and then yet another competitor is even better than that, etc.

Click to expand...

Let's see another one, and it could be that I don't understand you. Say group 4 guy is given 9.75. And then someone in group 5 gives a slightly lesser but still great performance, and they are given 9.5. And then someone comes in group 6 and falls in between those two. What must they be given?

9.75? 9.5?

So is the next argument "well, 9.5 guy should be given 9.25 as a ranking"?

I also try to use all-time scale but only as a reviewer, so it's silly to me to say it's also what goes through the judges' heads in real time. It is maybe not possible they'll come to the same conclusions as you when they judge that this is how scores should be dealt out - even if they agree results should be on an all time scale, because they simply might want to leave room for the skaters ranked above the ones in group 4. (I get that this flies in the face of how they keep inflating scores as they move to higher groups in reality, but we're talking flaws with this system in a single competition). And then group 4 guy giving the best performance and getting a 7 will be criticised by everyone, and everyone else will be left confused as to why they got 6.5s. (again could be drawing a poor conclusion here).

If scores are ranks anyway, then just use a 6.0 system, but patch it so that we can have CoP's flexibilty? We need to tell skaters there's room for improvement, so I get "all-time" scores are to be used, but I doubt what we have on hand passes any meaningful criterion set by some true science. I even don't think "all time" scores are used in any other competition, sport or otherwise, that uses a rubric (citations definitely needed). I am not one to believe the judges have any understanding of artistry, or technique, and of course that would fix some, but if we are to get to "accurate" judges, probably we'll always fail with this system. But maybe one's aim is to get to the greatest efficiency possible, which still is doubtful given time constraints to me, but I guess more achieveable with the things you've suggested.

I'm also curious if I'm actually correct in my understanding that this system would work in a bulk review, but not in a real time judging environment. Anyone have an opinion there?

Skatesocs said:

Let's see another one, and it could be that I don't understand you. Say group 4 guy is given 9.75. And then someone in group 5 gives a slightly lesser but still great performance, and they are given 9.5. And then someone comes in group 6 and falls in between those two. What must they be given?

9.75? 9.5?

So is the next argument "well, 9.5 guy should be given 9.25 as a ranking"?

I also try to use all-time scale but only as a reviewer, so it's silly to me to say it's also what goes through the judges' heads in real time. It is maybe not possible they'll come to the same conclusions as you when they judge that this is how scores should be dealt out - even if they agree results should be on an all time scale, because they simply might want to leave room for the skaters ranked above the ones in group 4. (I get that this flies in the face of how they keep inflating scores as they move to higher groups in reality, but we're talking flaws with this system in a single competition). And then group 4 guy giving the best performance and getting a 7 will be criticised by everyone, and everyone else will be left confused as to why they got 6.5s. (again could be drawing a poor conclusion here).

If scores are ranks anyway, then just use a 6.0 system, but patch it so that we can have CoP's flexibilty? We need to tell skaters there's room for improvement, so I get "all-time" scores are to be used, but I doubt what we have on hand passes any meaningful criterion set by some true science. I even don't think "all time" scores are used in any other competition, sport or otherwise, that uses a rubric (citations definitely needed). I am not one to believe the judges have any understanding of artistry, or technique, and of course that would fix some, but if we are to get to "accurate" judges, probably we'll always fail with this system. But maybe one's aim is to get to the greatest efficiency possible, which still is doubtful given time constraints to me, but I guess more achieveable with the things you've suggested.

I'm also curious if I'm actually correct in my understanding that this system would work in a bulk review, but not in a real time judging environment. Anyone have an opinion there?

Click to expand...

You can't define every individual judges work as a skaters rankings. Judges are giving an evaluation of the skaters performance and then the ISU mathematical system is the one who is doing the ranking. This is how one judge explained how judging of components work. While they are watching the program, they are taking a quick 'notes' about it. For example if they see that skater is exceptionally good at one foot skating they are checking that criteria with plus, or if it is not good at all they are checking it with minus. Then for the rest of criteria, if the skater use flow and deep knees most of the time during the programme, lets say 90% of the skating they can put number 9 at that criteria, if they use multidirectional skating lets say 70% of the skating time they will put number 7 at that criteria. Also, many of the things related to the SS, they can see at the very begining of the programme, and they check those criteria on the paper. Then they are looking if the skater is able to sustain those positives during the whole programme, or how the programme going on if the skater is doing other things written in the criteria, or not doing anything special at all. And at the end of the programme they do the approximation of all the criteria they are evaluated on paper and put the final score for it. Now, if the next skater can demonstrate even better flow and glide than the previous skater and the rest of the criteria is equally done, they can just up their score for one mark higher, but it is rarely the case during the competition that you can compare skaters easily like that. Because different skaters will take an advantage of different criteria they are good at, and you need to evaluate every criteria in every category individually (or at least you need to try to). How that is almost impossible to do, there are multiple judges of the panel and the approximation of all of theirs opinion should give a better evaluation of one component (as many of them can see and evaluate more of the things than what only one can see ).

Baron Vladimir said:

You can't define every individual judges work as a skaters rankings.

Click to expand...

I don't think you understood what I was saying. Please read again

ITA, @Baron Vladimir.

Skatesocs said:

Let's see another one, and it could be that I don't understand you. Say group 4 guy is given 9.75. And then someone in group 5 gives a slightly lesser but still great performance, and they are given 9.5. And then someone comes in group 6 and falls in between those two. What must they be given?

9.75? 9.5?

So is the next argument "well, 9.5 guy should be given 9.25 as a ranking"?

Click to expand...

This is less of an issue with IJS than with 6.0 (or pro-style 10.0 rankings, where the scores were added up rather than used as place holders for ordinals).

With IJS there are 5 different components, so even if a judge does not have room to give three different skaters with similar overall skating skills three different scores for the SS component, there will probably be differences on the other components that will end up with that judge giving these skaters different total PCS.

And if by chance three skaters do end up with the same total PCS from any individual judge, the panel as a whole will probably have enough disagreement that the averaged scores will differ -- although sometimes by less than 0.25 because of the averaging. In that case, the TES will end up determining the winner.

Or vice versa in the even rarer situations where two skaters end up with the exact same TES.

Since the job that judges are assigned under IJS is to score each element and component on its own merits, rather than to rank the skaters, giving the same score for any specific component does not amount to tying the skaters.

I'm also curious if I'm actually correct in my understanding that this system would work in a bulk review, but not in a real time judging environment. Anyone have an opinion there?

Click to expand...

That is probably true. But for the sake of both the skaters and the spectators, and the organizing committee that has to pay for the ice time, that's never going to happen.

(Maybe for a virtual competition not conducted in real time. With 6 or 12 skaters and not 30.)

Skatesocs said:

I don't think you understood what I was saying. Please read again

Click to expand...

Well, the conclusion of what i was saying is that your individual mark is not the only one important for the skaters score. You just need to choose between 9.5 and 9.75 as the rest of the judges. And statisticaly most likeable scenario is that skater will get 9.62 or 9.63 as the final score for that component, or something which will be between 9.5 and 9.75 for sure. Individual judge is not thinking about to put someone's mark between someome's else marks. But the average of all judges numbers will most likely give you that between mark.

Vlad, the reason I asked you to read my post again, isn't because I believe the scores are ranks, or all the judges believe scores are ranks. It's because the way Blades of Passion phrased his answer to my initial question to him - that 9.75 should be a cap, and not an objective reality score, to someone who skates "perfectly" in group 4. Doing so, to my perception, would indeed change this particular score into a simple ranking - BUT this conclusion of mine might be wrong. The rest of the post was asking him to explain if I understood him properly by presenting to him another example.

Based off this, if you have any insight, please let me know.

gkelly, yes, I understand that it is not possible to do bulk review in a competition setting. My rambling is thus based on my obsession with systems in theory vs practice.

Blades of Passion said:

The current rules for footwork do not encompass the full range of difficulty that is a possible. It's very possible to do a footwork sequence that is objectively around the same level of difficulty as what a current Level 4 might be, but that footwork will not get the credit it deserves. The same can also be said about spins to a degree..

Click to expand...

You mean SUBJECTIVELY. Because a Level 4 is based off of specific requirements for which the technical panel assigns a level. It's fine if a skater does something that is "viewed" as harder - like a footwork sequence all on one foot. But it's not a listed ELEMENT. They will be given credit in other areas such as program composition and performance. But elements should have specific requirements so as to be able to assess skaters within a consistent framework. Yes, some transitions or spin positions will be harder than others, but that's the skater's choice to go above and beyond. If a skater wants to do something that doesn't happen to garner technical points, that's fine.

Blades of Passion said:

Yes it is. The judges are told how to score certain skaters by their "bosses" and other such "lobbyists" before the competition starts. Skaters are currently held under the thumb of these politics, with big federations having the most sway, making the judging less objective. As for tech panels, they are able to talk and give their opinion about aspects of the tech elements throughout the entire competition, that's a fact. So any claim of "judging would be hurt by judges talking to each other" is something that's already affecting competitive skating and always will as long as there's a tech panel.

Click to expand...

Have you ever been on a tech panel? Or listened to one? You have no evidence that tech panels invariably assess elements of skaters relative to each other - their job, in fact, is supposed to be the most objective -- not "Oh, we should give skater B a ! because I recall skater A having more of a flutz and we gave them an e call.". And even if you were correct and this is the status quo (judges conferring with each other, and lobbying, etc.), wouldn't it be more ideal if judges were independently able to assess skaters (free from interference of their federation/makeup of the competition/conferring with other judges during the competition)? Judges not talking to each other is something that can be enforced to increase the level of independent assessment and accountability.

Blades of Passion said:

You're supposed to set as accurate of a mark as possible within the objective scale and then reference that as needed. Nobody is getting judged relative to skaters to come, rather they should all be getting judged within the framework of an overall objective scale, but WHILE being mindful of scores that were already given, so as to properly rate each performance in relation to each other. Perhaps there is added difficulty to keeping many performances in mind while assessing each mark, but a great judge should be sufficiently capable of handling this task, and referencing the numbers would only serve to improve judging consistency and mindfulness.

Click to expand...

Judges "leave room" all the time. Plushenko in the SP went after several poor skaters - and if he were marked relative only to them and not skaters yet to come, his scores would have been even higher.


In an ideal world, yes, nobody would be judged relative to skaters who have yet to come or who previously skated - a good judge will mark a skater absolute as possible, including their elements. Certainly when it comes to GOE they should not be "leaving room" and should judge elements absolutely. And when it comes to PCS, the skater should be marked according to the scale and what that judge is looking for. In IJS, skating order theoretically should not matter when it comes to the marks that a skater receives because then that skater's scores are contingent not only on how well/poorly their preceding competitors performed, but also how their upcoming competitors MIGHT perform. Judges should score every skater as if they were all the very first skater to skate.

If Skater A competes after 5 excellent skaters, wouldn't their marks likely be lower than if they got to compete after 5 mediocre skaters? Why should there be different scores for the same performance, simply because of how the Skating Order shaped up?

Skating is unlike other judged sports in that there is a considerable amount of subjectivity based upon skating order, and judges will often not score a skater as an absolute. In diving, judges score the dive in that moment. Yes, there are other competitors who dove or have yet to dive, but that athlete is being assessed based on what they do in that moment.

CanadianSkaterGuy said:

You mean SUBJECTIVELY.

Click to expand...

He means objectively - since his objective reality is predicated on his experience as a skater. Unlike yours, which is based on authority. At some point, one of you has to recognize that the other is defining words in entirely different realms of thought.

Diving is less subjective because of the nature of the sport.

Scoring dives (or gymnastics vaults) would be more similar to scoring a skating single jump or spin or lift in isolation -- base value determined by the tech panel and quality/GOE determined by the judges.

As you say, there's no reason why scoring elements in isolation like that should be comparative between competitors and be affected by skate order.

In theory there's no reason why judges' scoring of PCS should be affected by skate order either. But almost all parts of PCS are qualitative rather than quantitative, and many parts are subjective in the sense of evaluating skaters' ability to demonstrate and evoke emotional response. To the extent that judges are human beings subject to their own emotional responses, as influenced by the mood in the arena at the time, and also subject to fatigue during long events, there may some unintended effects of skate order on these scores, unrecognized by the judges at the time. Judges should be aware of the possibility of skate order effects and guard against them -- they definitely should not be leaving room or making conscious comparisons as they did under 6.0 -- but that's more difficult whenever if they're not consciously of how their perceptions are being affected.

Slowing down the event for detailed discussions between judges would only exacerbate that tendency, aside from the other problems with such a procedure.

gkelly said:

In theory there's no reason why judges' scoring of PCS should be affected by skate order either. .

Click to expand...

There are evidences from cognitive psychology that the order of presented things can influence the cognition. A lot of things figure skating judges are subjected to in the process of judging is inherent to humans cognitive system, and it is even happening to the 'trained cognition'. So the judges (and other human professions who are working with evaluation of things, as teachers for example) can just learn to be aware how 'the outside' things, like that one, may affect the process of judging, but they can't be totally removed. Human cognitive system just has its own restrictions and affections. More 'mistakes' in the judging are happening because of the restriction of the cognitive system than because of some subjective/emotional or socialy related bias (even one is always affecting the other). The best possible way to deal with it, beside the training of course, is to put more individuals to judge the same thing and not to overload those individuals with too many tasks (of too much 'different nature').

gkelly said:

Diving is less subjective because of the nature of the sport.

Scoring dives (or gymnastics vaults) would be more similar to scoring a skating single jump or spin or lift in isolation -- base value determined by the tech panel and quality/GOE determined by the judges.

As you say, there's no reason why scoring elements in isolation like that should be comparative between competitors and be affected by skate order.

In theory there's no reason why judges' scoring of PCS should be affected by skate order either. But almost all parts of PCS are qualitative rather than quantitative, and many parts are subjective in the sense of evaluating skaters' ability to demonstrate and evoke emotional response. To the extent that judges are human beings subject to their own emotional responses, as influenced by the mood in the arena at the time, and also subject to fatigue during long events, there may some unintended effects of skate order on these scores, unrecognized by the judges at the time. Judges should be aware of the possibility of skate order effects and guard against them -- they definitely should not be leaving room or making conscious comparisons as they did under 6.0 -- but that's more difficult whenever if they're not consciously of how their perceptions are being affected.

Slowing down the event for detailed discussions between judges would only exacerbate that tendency, aside from the other problems with such a procedure.

Click to expand...

Agreed, and good point about the emotional responses/audience reaction/etc. swaying their scores. I think the point we were trying to make is to minimize this as much as possible by eliminating as many variables as possible that could influence a judge's assessment of a skater, bearing in mind some things are unavoidable.

Judges will invariably "make room" whether conscious or subconscious. Some people complained that Kim/Plushenko were in earlier flights due to not having competed much prior to Olympics and were thus "held back" on their components scores, as the tendency is for judges to allot lower PCS for earlier flights - so even a really artistic skater won't get the PCS they "deserve" whereas a more mechanical skater will be on par with them for PCS simply for starting later on in the competition, and that of course isn't right.

And yeah, slowing the event for detailed discussions between judges would be unreasonable from a logistics standpoint and unfair from an assessment standpoint. There's the officials' post mortem after competitions for that. But anything before that is like kids leaning over during their tests and copying each other to make sure they all got the right answer. They should be prepared to assess and do their job.

Tangent -- Ever played the game Codenames? Sometimes players react to a clue being given which causes their teammates to get information or sway their responses for future turns, instead of simply using the clues given. Whereas they are supposed to be stoic, so their teammate can use the clues objectively and the game is as fair as possible without additional reactions perturbing the results. That's how judging should theoretically be, as much as possible. It would be interesting to see how a judging panel would work if they were only given the tools to input GOE/PCS about an individual skater's performance and no paper/pen for "tracking" the PCS/GOE they'd given to previous skaters, and if they remained stoic or even had dividers between them to not communicate with each other, and if they had zero heads up of the skating order or even which competitors were in the field so they wouldn't "leave room". Obviously, that's impossible, but it would probably lead to much more objective judging than what we see.

A judging panel isn't a collaborative network of assessment, but rather 9 individuals who are trained and tasked with competing up with their own individual assessments and then this gets averaged out to yield placements/points/results.

Skatesocs said:

He means objectively - since his objective reality is predicated on his experience as a skater. Unlike yours, which is based on authority. At some point, one of you has to recognize that the other is defining words in entirely different realms of thought.

Click to expand...

LOL, do I really have authority? I'm just a poster on a skating forum. For the record, I'm an experienced(ish) skater, but I also don't think experience or lackthereof should necessarily preclude one from being an expert, or having an opinion. I grew up competing under 6.0 so a lot of IJS doesn't apply to my previous competitive skating experience (and not trying to age BoP, but I'm guessing the same applies to him to).

If anything, when I skate with IJS in mind as an adult skater, I'm a lot more mindful of choreo/transitions/spin variations+number of rotations/etc. than I was under 6.0. You weren't thinking about what points you got, but you were making a concerted effort to include every jump "type" in your program, have 3 spins, etc. but there weren't really specific rules that pushed skaters to do difficult variations or transitions (I mean, I don't even think I heard the word "transitions" for most of the 6.0 performances I watched growing up). It's interesting to see a skater like Plushenko have to "upgrade" his spin variations/rotations as well as add greater complexity to his footwork sequences because those weren't really a thing under 6.0 (see his 2002 programs vs. 2006/2010). Sometimes in 6.0 you'd see a flying camel with 3 rotations and then the skater moves on. Which is fine for back then - but sports evolve. Skating under IJS is much harder than 6.0, but I'm personally enjoying it more because it's giving an actual structure of difficulty you can "aim" for, and causes you to be more mindful of incorporating skating transitions and basics into your choreography. Back then I would have never even thought about doing arm variations or a counter/bracket entry into my axel but IJS even in its rules/parameters has introduced difficulty and creativity into my own skating that I didn't have before.

And yes, BoP and I are certainly from different realms of thought. I'm definitely very okay with that though.

CanadianSkaterGuy said:

It would be interesting to see how a judging panel would work if they were only given the tools to input GOE/PCS about an individual skater's performance and no paper/pen for "tracking" the PCS/GOE they'd given to previous skaters,

Click to expand...

As far as I'm aware, mostly from witnessing local competitions, the judges are using paper and pencil to take notes on each program as it happens in order to aid their scoring of that program.
E.g., to make notes relevant to PCS that they will be inputting on the PCS input screen after the program is over.
To make notes of which elements they want to watch video review of afterward.
To make notes of ambiguously executed elements, including possible edge and rotation calls, that they need to wait till they see the tech panel's final call before finalizing their GOEs.
To make notes of weird errors they want to look up the rules for at the end of the program.

And also using the pencil erasers to touch the touch screen.

To prevent judges from keeping track of previous skaters' scores, in case any were so inclined, they could be required to turn in their paper for the most recent skater before the next skater is announced.

and if they remained stoic

Click to expand...

Most of them try, or are supposed to. Or so it seems anyway. If you watch the judges while they're judging, they tend to look pretty stonefaced.

and if they had zero heads up of the skating order or even which competitors were in the field so they wouldn't "leave room". Obviously, that's impossible, but it would probably lead to much more objective judging than what we see.

Click to expand...

It would be impossible not to know who's competing, especially if they're watching practices.

It would theoretically be possible to keep them ignorant of the skate order until the last minute. I'm not sure that would make much difference if they are familiar with past work/reputation of some of the skaters they know will be competing later.
And of course they would see the warmups.
[/QUOTE]

CanadianSkaterGuy said:

Judges will invariably "make room" whether conscious or subconscious. Some people complained that Kim/Plushenko were in earlier flights due to not having competed much prior to Olympics and were thus "held back" on their components scores, as the tendency is for judges to allot lower PCS for earlier flights - so even a really artistic skater won't get the PCS they "deserve" whereas a more mechanical skater will be on par with them for PCS simply for starting later on in the competition, and that of course isn't right.

Click to expand...

While you can say that is 'not right' for 'objectivity' of one exact competition, when i took all the factors involved, i kind of find it as a OKish thing. You can look at a skating in the last group as a 'small reward' for skaters who pushed themselves more to compete across the different ISU competitions and to be constantly better during the course of one whole season (or even two). Skating in the last groups implies a higher involvement to initially get there for any skater, instead to compete in only one ISU competition per year. It is probably not fair for some skaters who didn't have the same opportunity or luck to begin with, but that's life.

Blades of Passion said:

Well that's the inherent flaw of the system, right? However, I would say a 10 should be almost impossible to obtain, because there's always something to improve if you want to be picky. So you cap your mark off at 9.75 just in case there is even better yet to come. Realistically, it's just not going to happen that someone gives "the all time greatest performance ever" and then another competitor tops it, and then another competitor tops them, and then yet another competitor is even better than that, etc.

Click to expand...

Okay, but what if you give a 9.75, and then a better skater comes along, followed by an even better skater?

Someone brought up the various combinations for which a skater can get the same GOE scores so a jump with greater amplitude could get the same GOE as one that's small but has transitions/extension/better air position, or a spin with fewer rotations that's well centered could be on par with one with many rotations/positions but poorly centred, etc. The same applies for PCS - there are several criteria that go into each PCS score. For example a skater might have greater variety of transitions but okay quality of transitions, whereas another has fewer transitions but the security and depth of edge is better, so they end up deserving the same transition mark.

Point being, it's okay for skaters to tie in certain PCS aspects. In fact, it's an inevitability. Worlds 2019 had 35 skaters in men's, and assuming most will score 5.00 or higher (which is 21 possible scores from 5.00 to 10.00).