Today Show Investigation on Scoring

[GGFan]

I was scared to start this thread but the story was about more than just throwing accusations. They started with Sochi but more interestingly they presented research:

1. NBC tallied the 164 judges "available" for the Olympics--20% hold leadership positions in their countries federations

2. A Dartmouth economist analyzed figure skating scores and showed that there's significant nationalistic bias.

3. Who got called out by name on the show; Ukraine, Russia, South Korea, US

4. The ISU said "they have a robust system for monitoring judges."

None of this is new to us hardcore figure skating fans, but the more media coverage and academic studies, the more likely that the ISU will actually do something. Outside monitoring is one of the only ways things will change.

ETA: Here's the video story

ETA: Here's the Zitzewitz paper.

 

[Sam-Skwantch]

The first three paragraphs were laughable

The only thing different this year is the US isn’’t really in the mix for a lot of medals? The judging hasn’t suddenly become more bias...lol.

ETA:: Fortunately the article isn’t biased as it is speculative/inconclusive.

 

[tjb]

and what's the story?
some random american guy did a research on his own and published the results in the american media?
can i do my own research?
i'm not an economic professor, but i can ask for opinion of my friend, who is a sales consultant. his word is also might be weight something

 

[GGFan]

and what's the story?
some random american guy did a research on his own and published the results in the american media?
can i do my own research?
i'm not an economic professor, but i can ask for opinion of my friend, who is a sales consultant. his word is also might be weight something

Given the amount of complaints about outrageous judging, I thought folks would be happy to see that someone is crunching the numbers to see what's going on. It's something that Americans are guilty of so it's not a geopolitical issue. The Dartmouth economist is not trying to help the US win medals.

I think what will be interesting is whether the ISU feels any pressure to step up its oversight of judging in the new quad.

 

[YesWay]

Zitzewitz found that in 2016 and 2017 judges added an average of more than three points to the cumulative scores of their compatriots

Only 3 points? I expected more like 30, based on the... how can I put it... "stuff"... that gets posted on these boards...

Zitzewitz cautions that the rankings are based on a limited amount of data and should be treated as approximate.

Not really interested in the lightweight narrative that reads too much like propaganda... would rather hear more about the details of Zitzewitz' study - what his methods were, what data did he use, why "limited amount of data"?, etc.

Are these the kind of statistics we can trust? Or the other kind?

 

[lyverbird1]

We can laugh all we want about the obvious bias in this article but at the end of the day, there are still obvious problems and if this is the kind of stuff that's being read by neutral non-fanatics of skating, then we're not going to get many more fans into the fold. Is there any article more thorough that addresses the problems in judging that we can hold up as a more reasonable discussion? Any recommendations gratefully received.

PS On a lighter note, I'm enjoying saying the name Zitzewitz out loud. It has a nice, lyrical ring to it.

 

[GGFan]

We can laugh all we want about the obvious bias in this article but at the end of the day, there are still obvious problems and if this is the kind of stuff that's being read by neutral non-fanatics of skating, then we're not going to get many more fans into the fold. Is there any article more thorough that addresses the problems in judging that we can hold up as a more reasonable discussion? Any recommendations gratefully received.

PS On a lighter note, I'm enjoying saying the name Zitzewitz out loud. It has a nice, lyrical ring to it.

I haven't read his paper. But I'm assuming that it's a lot more straightforward and less biased than the Today article which is meant to get attention from public.

 

[Sam-Skwantch]

Alright ....my coffee pot is taking forever so I just went ahead and finished the article. Meh..,it’s good to have a healthy cynicism of judging. I agree we should hold the judges accountable but what this story lacks is actual evaluations of the skating. Just saying judges favor their own skaters is kind of misleading unless you show individual cases and spell out exactly what about that performance the judge scored incorrectly. Just looking at stats is very inaccurate because take the JGP for example. In about 99% of the events a a Russian or Japanese judge should favor their athlete because they are better then the rest. So yeah...the stat increases but in most cases it should. What percentage of judges should be favoring their own skaters because they deserve it?

Sonia Bianchetti...isn’t she the former ISU judge that came out right before Sochi with an article at IN that interjected her personal bias toward Caro right before the Olympics. Going after both Yulia and Adelina.

Simple solution....your scores are automatically tossed out when a skater from your federation performs in addition to the two outliers. IMO...you just can’t throw too many of the judges marks out. I’d even support center ice graphics on the Jumbotron like the old ‘Press Your Luck’ Whammy’s that shame the judge and sweep away the outlier scores publicly for all to see and laugh at!! Let’s keep score of the judge’s whammy stats! I’m not even joking!!

 

[GGFan]

Ok, I've added the paper to the OP. His last paper on the topic is from 2014 and it's not controversial or biased. It was looking at "compatriot" judging before and after the IJS reforms. He found that the nationalistic bias actually increased a bit after the reforms.

That's a solid and useful finding to consider and something that tracks GS discussions. Many of us have said that the additional numbers make the system seem more objective when it truly isn't in many ways.

I’d even support center ice graphics on the Jumbotron like the old ‘Press Your Luck’ Whammy’s that shame the judge and sweep away their scores publicly for all to see and laugh at!! Let’s keep score and judge’s whammy stats! I’m not even joking!!

Love this! Old game shows were the best.

 

[pearly]

Simple solution....your scores are automatically tossed out when a skater from your federation performs in addition to the two outliers. IMO...you just can’t throw too many of the judges marks out. I’d even support center ice graphics on the Jumbotron like the old ‘Press Your Luck’ Whammy’s that shame the judge and sweep away their scores publicly for all to see and laugh at!! Let’s keep score and judge’s whammy stats! I’m not even joking!!

Like in Salt Lake with the French judge and Russian pair

I wonder if IJS needs a "police officer" judge - or algorithm - which would check the scores for situations where rules require a -2 reduction in GOE and a judge awards 2 or more. Which should be impossible with a max possible GOE of +3 (+3-2=1). If your GOE is more than 1, it lights up in red and you have to change it.

 

[CanadianSkaterGuy]

Well the issue isn't if a judge from the home country of a skater's marks should be thrown out since home country judges could still theoretically mark properly. A better way would be to drop the 2 highest and 2 lowest judge scores every time so the marks used in the eventual score are more consistent.

It's nice to see some work being done on this. Obviously some people will just dismiss it as bias if it's not saying what they want it to say. But I think some of this holds truth

It seems this bar graph was based on GOE? But it would be neat to see one based on PCS. While an average of just over 3 points more doesn't seem like a lot, we have seen many events where that could make a difference.

I think National bias is unavoidable but at least this gets the conversation going with an actual study.

 

[Sam-Skwantch]

Given the amount of complaints about outrageous judging, I thought folks would be happy to see that someone is crunching the numbers to see what's going on.

In theory yes...I welcome it. They just left out the part where you connect the stats to actual figure skating and prove the bias on specific circumstances beyond saying the stats suggest it. Like..show me instances where the “bias” was actually justified and then some where it wasn’t and let’s discuss why it happened and who did it. It’s important to point out that “bias” from the “home” judge isn’t always an unusual thing we should not expect,. However...what the Chinese judge did to Shoma at 4CC was beyond reasonable.

https://www.goldenskate.com/forum/s...ng-Anomalies&p=1954609&viewfull=1#post1954609

I think the only way to change anything is to stop lumping people into stats and by nationality and expose them as individuals. It needs to be a personal responsibility of the judge and they need to take ownership of their marks.

 

[CanadianSkaterGuy]

Like in Salt Lake with the French judge and Russian pair

I wonder if IJS needs a "police officer" judge - or algorithm - which would check the scores for situations where rules require a -2 reduction in GOE and a judge awards 2 or more. Which should be impossible with a max possible GOE of +3 (+3-2=1). If your GOE is more than 1, it lights up in red and you have to change it.

Yes! Along with a loud siren going off above those judges who give +3 for elements with a touchdown or 10.00's for programs with a fall.

It would be great if a tech specialist can recognize an error and then put a GOE cap in the system based on mandatory deductions/reductions. So a judge is physically incapable of giving +3 for an error.

 

[GGFan]

In theory yes...I welcome it. They just left out the part where you connect the stats to actual figure skating and prove the bias beyond saying the stats suggest it. Like..show me instances where the “bias” was warranted and then some where it wasn’t and let’s discussed why it happened and who did it. It’s important to point out that “bias” from the home judge isn’t always and most likely not even usually an unnatural or unwarranted thing. However...what the Chinese judge did to Shoma at 4CC was beyond reasonable.

https://www.goldenskate.com/forum/s...ng-Anomalies&p=1954609&viewfull=1#post1954609

I think the only way to change anything is to stop lumping people into stats and nations and expose them as individuals. It needs to be a personal responsibility of the judge and they need to take ownership of their marks.

I don't know how much time the researcher has to follow up, but this is very much the nature of the beast when it comes to research. You have to keep running more studies and experiments. For example, this is much more complicated because the coding is subjective, but they need to do a study that zooms in on the problem areas. If there's a visible error, how many judges are taking mandatory deductions? etc.

That's just one crude example, but the point is that there's a lot of analysis that could be done to tighten up this system. The ISU says it has robust oversight but the numbers tell a different story.

 

[Mrs. P]

Just a bit of a nitpick.

This is actually a story of NBC News' investigation department. The Today Show often broadcasts stories from NBC News, but these are bonafide investigative journalists.

In theory yes...I welcome it. They just left out the part where you connect the stats to actual figure skating and prove the bias on specific circumstances beyond saying the stats suggest it. Like..show me instances where the “bias” was actually justified and then some where it wasn’t and let’s discuss why it happened and who did it. It’s important to point out that “bias” from the “home” judge isn’t always an unusual thing we should not expect,. However...what the Chinese judge did to Shoma at 4CC was beyond reasonable.

https://www.goldenskate.com/forum/s...ng-Anomalies&p=1954609&viewfull=1#post1954609

I think the only way to change anything is to stop lumping people into stats and by nationality and expose them as individuals. It needs to be a personal responsibility of the judge and they need to take ownership of their marks.

The thing is the judges are chosen by the federation and as noted many times hold positions in the federation. I don't think you can uncouple the two.

 

[[email protected]]

Quite a revealing article. If we believe the data it says that among the skating powers Japan inflate their skaters by 1.5 points and Russia by 3 points. The rest are in between. Even if it were true Russian skaters would receive about 0.3 point score advantage over Japanese due to the national bias. Hence, only very very close wins could be suspicious. Means that if Medvedeva and Zagitova bring 5 to 10 point margin it could be anything but corrupted Russian judge.

 

[Sam-Skwantch]

It would be great if a tech specialist can recognize an error and then put a GOE cap in the system based on mandatory deductions/reductions. So a judge is physically incapable of giving +3 for an error.

I’m 100% against this because I want to remove all of the TP’s power I can. In fact..any call they make like UR’s and Edge Calls I wish were voted on by the panel. So of the TP calls UR then right after the performance the judges get prompted to uphold or decline the calls via a quick “public” vote on their computer screen with up to two quick video reviews....just like Snapchat it then disappears. You can’t involve too many checks and balances with this stuff IMO.

It may take longer but they just need to improve the in arena entertainment while more accurate scores are posted.

 

[Neenah16]

Wow, real research and not the "analysis" we usually get from various sources on FS

I should have some free time soon to review the paper. I am very interested as what their data sources where and the analysis methods used :hap85:

 

[eppen]

The actual paper came out in Journal of Sports Economics 15 (2014), pp. 3-30. I have not checked out whether much has changed from the draft to the final version.

This is perhaps not a hugely surprising piece of news - or revolutionary research - as there are so many analyses of how the block voting went on in the Cold War era etc.

National bias is perhaps impossible to eradicate in the end - and when you look at the score sheets now with the immediate connection between country, judge and skater, well, unfortunately just about everybody does it, it seems. But what I would like to know is which is more effective - skater's own judge scoring high or the main opponent's judge scoring low?

E

 

[Bluediamonds09]

The real problem with the scoring system is the judges not following the rules and not being ethical or fair in their scoring. Cheaters cheaters cheaters, all of them. They know what they’re doing. They’re sick.