Today Show Investigation on Scoring

[GGFan]

Forgive me in advance but this is something I've noticed several times on GS: a) someone makes a statement, b) someone else provides some evidence that the person is or has reasons to be biased, and therefore it means c) we can dismiss everything in the original statement.

Huh??? That's not how the world works. People have biases all the time. It's a fact of life. I think it's wonderful to consider biases and to account for them but a biased person can often tell the truth. If someone biased is a witness, what you do is to get another witness or find another means to corroborate the information, you don't say "well we will not listen to anything you say because you're potentially biased." If all biased or potentially biased statements were just thrown out per se, we would have very little left in the world. topic:

 

[CanadianSkaterGuy]

Forgive me in advance but this is something I've noticed several times on GS: a) someone makes a statement, b) someone else provides some evidence that the person is or has reasons to be biased, and therefore it means c) we can dismiss everything in the original statement.

Huh??? That's not how the world works. People have biases all the time. It's a fact of life. I think it's wonderful to consider biases and to account for them but a biased person can often tell the truth. If someone biased is a witness, what you do is to get another witness or find another means to corroborate the information, you don't say "well we will not listen to anything you say because you're potentially biased." If all biased or potentially biased statements were just thrown out per se, we would have very little left in the world. topic:

I fully agree with this. But then again, maybe that makes me biased towards supporting bias?

 

[Step Sequence4]

Ugh -_- this is the first of many im sure, as its that time of year. Personally I agree with the Sochi results but thats not what were discussing here. I do think the system of judges should be changed and small reforms to the scoring system should be made but nothings ever gonna perfect, for example i think national federations should send in a list of possible judges they want, then the Isu should decide. Side note here, i was quite surprised to see how low on the graph of juding bias Canada was, and thats from a canadian.

 

[Sam-Skwantch]

I tried something similar (although far less comprehensive, obviously) a while back. Here's my blog post: http://bunniko.blogspot.com/2017/11/figure-skating-scores-judging-and.html
Can't post photos on this site but there are some graphs in there.

Wow..that was interesting. Very nice work. :agree:

Im not sure all these stats really do much for me though. They need to be tied directly to what happened on the ice to mean anything to me and I can’t find an effective way to do so. Like maybe for a hypothetical example a judge from (Fed1) favors a home skater/team by 5 pts more than the other judges. That only really matters to me if the same judge were to also show a noticeable bias against that teams rival competition (fed2). So how do we correlate this into a reasonable stat and would it even be reliable? What would an acceptable corridor even be? Bias +x(fed1) in relation to -x(fed2)?

Still...sometimes a skater deserves the points awarded by the judge from (fed1) even if they favor them by 5pts but also scores the rest of the feild +2 or +3 higher than the rest of the other judges. . I don’t think this is being addressed in the article because it seems that they are saying (fed1) judges favor (fed1) skater by (insert amount) and leaving the rest of the equation and relevant details discussion up to us. It’s vital to assess how the judge’s variance correlates with the entire field. I can’t imagine this is possible to do with a pool of judges.

The data is interesting but hardly unexpected for me based on the performances I’ve seen and I’m afraid people can use stats like this to cause unfocused doubt.


I honestly think our judging system is better than we give it credit for but I think we need to not be afraid to call out individual incompetence. Still...I like the judging system and think only a few tweaks are needed. I don’t think the host federations should ever invite or make accommodations for ISU judges. They can flip the bill but leave the selection to the ISU or lottery.

 

[Eclair]

Good proposition but I object the one regarding overlooked UR call. It would toxify the whole competition and taint relationships between skaters, coaches etc. I think the current restrained approach at calling out your opponents' mistakes is the only rational option. It's very rarely used though ... the only example that springs to my mind is last year in Ice Dance at the European Championships when French filed a protest (not sure whether it's the right term here ...) to give -1 point deduction to Capellini / Lanotte already after SD as tech panel overlooked their mini-lift that exceeded allowed number of revolution or something of this sort. Later on C/L SD score have been changed.

But imagine everyone doing it. It wouldn't taint relationships at all if everyone does it.

If Eteri calls out Osmond's uncalled flutz, will that taint the relationship between Eteri and Ravi? Not really.
If Mie Hamada calls out Evgenia's flutz, will that taint the relationship between Mie and Eteri? not really.
If Frank Carol calls out Satoko's UR, will that taint any relationships? no ...

If Rafael calls out Shoma's UR, will that taint relationships? No ..
If Brian calls out Nathan's UR, will that taint any relationships? Not really either.

The result of this little, minimal and cost-efficient change would be, that if a skater UR, falls, two-foots, etc. it will definitely and consistently for every skater be called. Still the tech panel holds the deciding power and can adjust the level of harshness or lenientness across the board.
At the same time, if a skater doesn't flutz or UR but gets called, then it will definitely be objected on and reviewed again before the scores come out.

Of course, coaches who just object every jump of the opponent skaters without them actually be UR in any way, would be looked down on. But I don't even think that they need to implement a punishment for this as I don't think coaches would do that just for fun. All median calculating and voting won't help, if all of the votes are biased in one or another way...

And it wouldn't take any effort to implement this. just publish the calls and levels given before the scores comes out and give the coaches like 15 seconds after the slo-mo-replay to object.

 

[Sam-Skwantch]

The coaches and competitors I know personally hardly ever watch other skaters unless they are in different groups and even still then maybe only if they know them personally or are already finished. You’d almost have to bring someone along to oversee things and make decisions on challenging the calls for skaters competing in your own group. And what about coaches with multiple skaters?

The judges are already there and watch all of the skaters presumably with their full attention on the performances. Having the judges uphold or deny the TP’s calls by a majority vote would eliminate taking a coach away from preparing their skater to compete in order to monitor and make challenges.

I do see your point though. If Eteri were sitting on the boards staring at the judges when Kaetlyn skates they might be more inclined to call her flutz and vice versa. I don’t know though...we’d probably still need to have the judges on the panel review the “challenge” and vote. If we just go straight to a nine person vote to uphold or deny the TP calls on UR’s and Edge calls I think we’d see the same results anyway.

My biggest grumble with the judging these days is that the judging seems to often reflect the judges’s personal preferences more than the effort skaters are putting forth on the ice. I’m not sure how to solve this honestly. People often let their biases influence them when evaluating performances. The judges aren’t much better than the fans who present their own sets of challenges in this regard.

 

[russianfan]

I’m not sure how to solve this honestly. People often let their biases influence them when evaluating performances. The judges aren’t much better than the fans who present their own sets of challenges in this regard.

slow-mo replays?
some kind of PCS limit for "performances" like Carolina's at Euro?

 

[Ares]

But imagine everyone doing it. It wouldn't taint relationships at all if everyone does it.

If Eteri calls out Osmond's uncalled flutz, will that taint the relationship between Eteri and Ravi? Not really.
If Mie Hamada calls out Evgenia's flutz, will that taint the relationship between Mie and Eteri? not really.
If Frank Carol calls out Satoko's UR, will that taint any relationships? no ...

If Rafael calls out Shoma's UR, will that taint relationships? No ..
If Brian calls out Nathan's UR, will that taint any relationships? Not really either.

The result of this little, minimal and cost-efficient change would be, that if a skater UR, falls, two-foots, etc. it will definitely and consistently for every skater be called. Still the tech panel holds the deciding power and can adjust the level of harshness or lenientness across the board.
At the same time, if a skater doesn't flutz or UR but gets called, then it will definitely be objected on and reviewed again before the scores come out.

Of course, coaches who just object every jump of the opponent skaters without them actually be UR in any way, would be looked down on. But I don't even think that they need to implement a punishment for this as I don't think coaches would do that just for fun. All median calculating and voting won't help, if all of the votes are biased in one or another way...

And it wouldn't take any effort to implement this. just publish the calls and levels given before the scores comes out and give the coaches like 15 seconds after the slo-mo-replay to object.

It would make complicated figure skating even more complicated so I oppose that. Just not a good idea but each to its own.

 

[karne]

Getting rid of biased judging is actually pretty easy. For the tech calls:
- Tech panels calls the jumps and levels just as it is now.
- They respeat each jump in slo-mo after the skate in the arena - for the most time, they do that already anyways
- They have to publish the calls and levels etc. they made before the final scores
- Coaches of competitive skaters or the skater's coach themselves can file a formal protest or objection if they think that the tech panel has overlooked an UR or wrong edge or has wrongly accused the skater of an wrong call. This all happens before the scores are announced.
- the tech panel must review the element, that the coach think was overlooked/ wrongly called. They then can decide if they want to change their call or not.
- If they don't change the call and the coach still thinks it's blatantly unfair, then he can offer a formal protest after the competition and this element will be reviewed by a broader panel.
- this system would benefit fair judging, as the skater's coaches can appeal to wrongly and unfairly made calls etc. in time, while it can at the same time prevent the tech panel from overlooking UR, two-foot and edges (or levels).

So every competition now goes on for three weeks, rather than a few days.

Do have any idea how long this would take? If the delay is greater than 10 minutes you have to run another six minute warmup. There's no way you'd be getting this done in under 10 minutes every time.

 

[Eclair]

So every competition now goes on for three weeks, rather than a few days.

Do have any idea how long this would take? If the delay is greater than 10 minutes you have to run another six minute warmup. There's no way you'd be getting this done in under 10 minutes every time.

How long would it take for the tech panel to review one or two elements per skater again?
10 seconds? that would make competitions 30min longer at most.

And that is when you have 60 skaters with all of them required to review at least 3 elements.

it's not like I want 20 coaches all object the same overlooked underotated jump discussing endlessly with the judges. Use computers. If one coach object to one jump in UR, then other coaches can't object to it anymore/ it will be marked as being reviewed for UR.

 

[Eclair]

The coaches and competitors I know personally hardly ever watch other skaters unless they are in different groups and even still then maybe only if they know them personally or are already finished. You’d almost have to bring someone along to oversee things and make decisions on challenging the calls for skaters competing in your own group. And what about coaches with multiple skaters?

The judges are already there and watch all of the skaters presumably with their full attention on the performances. Having the judges uphold or deny the TP’s calls by a majority vote would eliminate taking a coach away from preparing their skater to compete in order to monitor and make challenges.

I do see your point though. If Eteri were sitting on the boards staring at the judges when Kaetlyn skates they might be more inclined to call her flutz and vice versa. I don’t know though...we’d probably still need to have the judges on the panel review the “challenge” and vote. If we just go straight to a nine person vote to uphold or deny the TP calls on UR’s and Edge calls I think we’d see the same results anyway.

My biggest grumble with the judging these days is that the judging seems to often reflect the judges’s personal preferences more than the effort skaters are putting forth on the ice. I’m not sure how to solve this honestly. People often let their biases influence them when evaluating performances. The judges aren’t much better than the fans who present their own sets of challenges in this regard.

voting doesn't solve the problem though. If you have 5 out of 9 judges doing block voting, then those voting processes are not worth anything .. also, voting takes even longer than reviewing the element by the tech panel.

 

[whatif]

This article and the discussion is just a gibberish. It misses a lot of other factors, such as "vote trading", which I think is even a bigger issue than the so called "national bias". For example this Olympics, Russians do not really have Gold medal contenders in at least 2 or perhaps 3 disciplines. Often what really affects the judging is selling one's votes where one does not have a contender for favourable voting by a competing country judge for their vote in another discipline where they do. This is I think why it is not a problem you can solve in the type of sort that is FS.

 

[Blades of Passion]

I'm disappointed they didn't go into more detail, the interview process I did was quite extensive and they seemed to do plenty of other research too, but I suppose at the end of the day all they were trying to do with the article and video was quickly bring attention to the subject for the general public. It shows how little figure skating matters to people anymore that the Olympics are around the corner and this is all the air time that can be given to it, because of perceived market worth.

 

[GF2445]

As said before, the report to us is a little bit of a laugh, because it was a news report on something that is very old news. But for the casual fan, it gave a very basic overview and it presented a problem (one of many) that is present in judging. Something that I was thankful that the report brought up is the ISU's unwillingness to harshly sanction and ban cheating judges. The ISU's, judges, and official's attitude towards judging in figure skating has never moved on from 6.0. Whether using ordinals or decimals, if it's a high stakes/major competition, reputation is king.

I would rather a tech panel with officials from neutral nations, and in the judging panel, ensure that the nations of the top contenders in the event are present on the panel. We know if a certain nation's judge is not on the panel, other nations may gang up on that nation. It's politics 101.

 

[GF2445]

Also, give the tech panel another camera. There is no excuse for 'overlooking.'

 

[rocketry]

I was going to do something very similar for an econometric project, but my advisor encouraged me to do something more “socially relevant” for my project instead. :sad4:

Yes, as the poster suggested upthread, it’s a “simple linear regression,” but sometimes that’s all you need to understand that there are systematic deviations. These numbers are basically validating what we already suspected. A larger sample size would probably yield the same results.

But the numbers themselves don’t give us contextual information or answer questions about whether or not the skaters deserved the scores or not. As a result, i think the study is useful in telling us that nationalistic bias does exist, but the extent which it can be applied to actual outcomes has to be looked at on a case-by-case basis.