Today Show Investigation on Scoring

[GGFan]

i can't seriosly debate an article that starts with some hilarious sotnikova rant.
and goes on like this

"Zitzewitz started studying figure skating after a scandal at the 2002 Salt Lake City Olympics rocked the sport. The Russian mafia was accused of fixing a gold medal victory for Russian pairs skaters, beating out a Canadian team despite the Canadians' superior performance."

this guy must be really credible and non biased "professor" of figure skating sciences

Because this is an NBC story it might seem like his loyalties are to them if you do not live in this country. However, let me assure you that they're not and we should not conflate his academic work with the NBC story.

Zitzewitz wrote this non-funded paper for academic purposes. He knew his data would be checked by academics and if he was biased he was putting his credibility on the line. Now there might be issues with his methodology or corrections that should be made to his numbers, but those are different critiques than writing off his work for bias.

 

[tjb]

Well, now I have confirmation you didn't read any of his work. He didn't write the NBC article, try reading the byline.

Edit: I'm gonna drop this, if you can't tell the differences between a journal article containing statistical analysis and a puff piece from NBC this is pointless.

why should i read an article from some random economics guy? nbc piece are not awakening my desire to do so, that's for sure.
but i'm happy for you if you actually read those works. what's your last name btw, zitzewitz?

 

[[email protected]]

It's not about the credentials of the guy or economist vs. mathematician. Advanced econometrics deals with issues where we have optimization problems with multiple variables with some covariation among them. One usually uses Monte-Carlo method, then applies Cholesky decomposition matrix, etc. I remember we tried to do something along the lines while developing the future yield curve of Treasury bonds when I was a student at Stanford. What the guy has done is a primitive regression analysis on, as he said himself, "a limited sample of data". Any undegraduate student can do the same if he or she has time and motivation.

But what is even more important is that it is much ado of nothing. A lot of text is written to state the obvious: there is national bias. So what? The paper lacks rigorous analysis of whether this bias results in continuous abberation of results or had occassional bursts of big time unfair competition results. Instead, the guy widey uses terms like "Russian mafia", "scandal" while the math outcome of his tables is not material 0.3 bias for Russia vs. 0.2 for USA and 0.15 for Japan is nothing. This makes me think that it is yet another political effort - there have been so many of them recently.

 

[Mathman]

why should i read an article from some random economics guy?

No, you should not. You will not learn anything from it.

 

[schizoanalyst]

I don't think that, in practice, this would help much. If you define "independent" as evidence that the judge gives marks within a close range of the mean, you could still have voting blocs drive up scores for certain skaters, forcing an "independent" judge to mark those skaters higher just because the judge wants to be closer to the mean score. Furthermore, it discourages judges from awarding outlier scores that the judge genuinely believes are accurate. For example, we sometimes see a judge award much lower TR scores for big name skaters and, usually, these lower scores are justified. It's all very complicated and I think the ISU has taken reasonable steps towards fairness, including identifying judge scores and staffing the tech panels in big events with representatives from countries who aren't expected to compete for the top spots in that event.

Nobody would ever measure "fairness" as "closeness to the expected mean." If you did that solely, you'd induce a beauty contest game. It can be a component. You'd look at difference in scores for compatriots vs other athletes. You'd look at compensating bias (which is a valid measure of fairness in strategic games), which would go against "closeness to the mean". Low scores for a "big name" skater would be evidence of compensating bias and would be awarded as more fair, for example.

 

[GGFan]

It's not about the credentials of the guy or economist vs. mathematician. Advanced econometrics deals with issues where we have optimization problems with multiple variables with some covariation among them. One usually uses Monte-Carlo method, then applies Cholesky decomposition matrix, etc. I remember we tried to do something along the lines while developing the future yield curve of Treasury bonds when I was a student at Stanford. What the guy has done is a primitive regression analysis on, as he said himself, "a limited sample of data". Any undegraduate student can do the same if he or she has time and motivation.

But what is even more important is that it is much ado of nothing. A lot of text is written to state the obvious: there is national bias. So what? The paper lacks rigorous analysis of whether this bias results in continuous abberation of results or had occassional bursts of big time unfair competition results. Instead, the guy widey uses terms like "Russian mafia", "scandal" while the math outcome of his tables is not material 0.3 bias for Russia vs. 0.2 for USA and 0.15 for Japan is nothing. This makes me think that it is yet another political effort - there have been so many of them recently.

A good 95% of academic papers are of the so what variety. The paper is just about moving the ball a tiny bit. His research should lead to other research to see if it holds up. It could be that when you go deeper you find out that there's not much there or even if there is there's not much that can be done.

Regardless, the point isn't that any one paper will answer all of life's questions. It's just supposed to move things forward and provide us with more data. Replication is still necessary. I would love to see more work done along these lines. If the ISU is doing it internally we don't really hear about it much or see any changes being made.

ETA: the whole point of the IJS reforms was to improve the system and reduce some of the nationalistic bias. I think it's valid to compare what occurred before an after the reforms were instituted.

 

[TunaKeem]

I'm not privvy on judge composition for ISU competitions but I thought judges were mixed by country in competitions.
Like in Sochi there were judges from multiple countries that made up the panel but it seems that the entire judging panel's scores were inflated for certain skaters. Could 1-2 judges from a country that inflate their scores to show favoritism to a skater from their country really skew the scores high enough to matter?

Maybe automatically excluding the highest and lowest scores from a scoresheet could work? It seems like the way to go is to not even be comparing scores across events for athletes because the judging is so skewed. I've seen this mentioned in commentary while watching videos where they say going earlier can be disadvantageous for skaters since the judges will save their PCS in case a better skate is done but that's so arbitrary. The scoring system isn't built for fans to be able to objectively measure if they have improved or not. I'm realizing now it can't even be argued to not compare scores across events when ISU shows seasons best and personal best for each skater.

 

[Mathman]

Nobody would ever measure "fairness" as "closeness to the expected mean."

Actually, I think the ISU does.

IMHO (I am a mere mathermatician, not a statistician ), the simpler the study the more useful and interesting it is. For some reason the link to the Zitzewicz article doesn't work for me. But anyway, just do this. How much higher do the judges from USA score American skaters, compared to the the scores given by judges from other countries to the same performance? This is the information that it would be interesting to know.

By the way, this guy is a serious scholar with a substantial publication record in refereed economics journals and well as popular news outlets. He is not just some bozo who is unaware of the science of statistics. Here is his resume from the Dartmouth web site.

https://www.dartmouth.edu/~ericz/

 

[tjb]

so it's an academic paper already? i'm just curious in what science?
the guy is obviosuly doing great if a 15 years long research leads to nothing

 

[GGFan]

so it's an academic paper already? i'm just curious in what science?
the guy is obviosuly doing great if a 15 years long research leads to nothing

I'm all for a good joke, but right now I don't think you're succeeding in making him look bad. His cv is available online. He's an economist. You do not have to agree with his work but I don't see the sense of trying to ridicule his credentials.

 

[tjb]

I'm all for a good joke, but right now I don't think you're succeeding in making him look bad. His cv is available online. He's an economist. You do not have to agree with his work but I don't see the sense of trying to ridicule his credentials.

i'm just a member of russian mafia in figure skating. that's what we do. we always are trying to discredit serious academic works without any success

 

[GGFan]

i'm just a member of russian mafia in figure skating. that's what we do. we always are trying to discredit serious academic works without any success

On that note . . .

 

[narcissa]

Actually, I think the ISU does.

IMHO (I am a mere mathermatician, not a statistician ), the simpler the study the more useful and interesting it is. For some reason the link to the Zitzewicz article doesn't work for me. But anyway, just do this. How much higher do the judges from USA score American skaters, compared to the the scores given by judges from other countries to the same performance? This is the information that it would be interesting to know.

By the way, this guy is a serious scholar with a substantial publication record in refereed economics journals and well as popular news outlets. He is not just some bozo who is unaware of the science of statistics. Here is his resume from the Dartmouth web site.

https://www.dartmouth.edu/~ericz/

I tried something similar (although far less comprehensive, obviously) a while back. Here's my blog post: http://bunniko.blogspot.com/2017/11/figure-skating-scores-judging-and.html
Can't post photos on this site but there are some graphs in there.

 

[Mathman]

so it's an academic paper already? i'm just curious in what science?

Well, his publication record lists an earlier paper that was published in something called the Journal of Sports Economics, so I guess "sports economics"is a science. (Sports is certainly big business.) In an earlier paper (2012) Zitzewicz investigates whether anonymous judging increases or decreases national bias. (I can't access the paper now without paying for it, but if it the paper that I am thinking of, he puts quite a bit of effort into the interesting question of, how can we tell which judge is from which country, if this information is not supplied up front by the ISU.

Are we really criticizing Zitzewicz' actual paper here, or is it the Today Show's take on it that is so objectionable?

 

[Mathman]

I tried something similar (although far less comprehensive, obviously) a while back. Here's my blog post: http://bunniko.blogspot.com/2017/11/figure-skating-scores-judging-and.html

Great read. Thank you.

 

[[email protected]]

I'm all for a good joke, but right now I don't think you're succeeding in making him look bad. His cv is available online. He's an economist. You do not have to agree with his work but I don't see the sense of trying to ridicule his credentials.

Once again, no one is trying to ridicule his credentials per se. I believe that the guy is serious. One has to look at relevant things, though. And they are not in the essence of his paper. The fundamental question is: "why this and why now?" And it is more important because the scientific value of the analysis is minimal but the paper is presented by NBC to a broad public right before the Games. And it contains the words and phrases like "Russian mafia", "scandal", etc. which you would not find in your regular journal articles.

And what perplexes me is why this serious guy spent so many years for such a weak work while there has been much more thorough analysis on block voting in 6.0 era. I gave a link to it once - feel too tired to look it up one more time. But the idea is clear - let's see whether some countries vote in favor not only their compatriots but also their "friends". There are abundant theories on GS that Europeans in general and judges with "slavic" names tend to favor the Russians. I can put forward my theory that judges from Australia tend to underscore the Russians. The outcome of this analysis should be twofold. 1) whether the block theory can be proved on a large sample of data 2) if yes, whether it had material effect on the previous competitions' outcomes. And the second is more important because if the bias is there but even if it was not the results would be the same then who cares and why bother?...unless it is another chord in a big political game we are watching now.

 

[karne]

I was scared to start this thread but the story was about more than just throwing accusations. They started with Sochi but more interestingly they presented research:

Oh yay, another Adelina bash-fest. Not even thinly veiled, this one.

1. NBC tallied the 164 judges "available" for the Olympics--20% hold leadership positions in their countries federations

20%, but apparently the only one that's important is that Alla is married - married, I tell you! - to someone in a leadership position.

2. A Dartmouth economist analyzed figure skating scores and showed that there's significant nationalistic bias.

*gasp* I'm stunned. Stunned, shocked. Whoever could have imagined such a thing?!

3. Who got called out by name on the show; Ukraine, Russia, South Korea, US

Okay, this made me laugh. "But Yuna was robbed!" *South Korea gets called out* Yeaaaaaaaaaaahhhhhhhhhhh............

 

[GGFan]

Once again, no one is trying to ridicule his credentials per se. I believe that the guy is serious. One has to look at relevant things, though. And they are not in the essence of his paper. The fundamental question is: "why this and why now?" And it is more important because the scientific value of the analysis is minimal but the paper is presented by NBC to a broad public right before the Games. And it contains the words and phrases like "Russian mafia", "scandal", etc. which you would not find in your regular journal articles.

And what perplexes me is why this serious guy spent so many years for such a weak work while there has been much more thorough analysis on block voting in 6.0 era. I gave a link to it once - feel too tired to look it up one more time. But the idea is clear - let's see whether some countries vote in favor not only their compatriots but also their "friends". There are abundant theories on GS that Europeans in general and judges with "slavic" names tend to favor the Russians. I can put forward my theory that judges from Australia tend to underscore the Russians. The outcome of this analysis should be twofold. 1) whether the block theory can be proved on a large sample of data 2) if yes, whether it had material effect on the previous competitions' outcomes. And the second is more important because if the bias is there but even if it was not the results would be the same then who cares and why bother?...unless it is another chord in a big political game we are watching now.

Oh, ok I think there is some misunderstanding here. We should separate out two things:

1. NBC. They are a TV network and they're carrying the Olympics. They want to generate interest in the Olympics and want stories to sound more salacious than they are. In terms of having people listen this is the ideal time for them to run this story. It's not about right or wrong; it's about getting viewers. In general the media is very bad about presenting research. Research papers are usually measured and only look to answer a narrowly defined question. None of that sounds sexy on TV. Without even thinking about it I just completely ignored the NBC wrapper. Who cares?

2. The paper. Zitzewitz did his research years before this story on NBC and the two are not related. He was following up on his previous research on bloc voting, and as researchers do he had a question he wanted to answer: has bias decreased since the IJS reforms? That's a valid question to ask. His study provides one tentative answer. Other studies should be run to keep pushing this analysis forward. The fact that he's American has very little to do with the analysis. Did he use the right numbers? Are his assumptions defendable? etc. This is not even the focus of his academic work--he does lots of other things, so let's not assume that this is his life's work and he should feel bad that he hasn't accomplished more.

 

[tjb]

Oh, ok I think there is some misunderstanding here. We should separate out two things:

so, now it's separated?
do you think that the guy who did "the research" is against the nbc throwing russian mafia accusations? why he agreed to use his name then?
and why you was so scared to start a thread about this amazing nbc "investigation" (which you completely ignored)?
do you think that someone might kill you?

 

[GGFan]

so, now it's separated?
do you think that the guy who did "the research" is against the nbc throwing russian mafia accusations? why he agreed to use his name then?
and why you was so scared to start a thread about these amazing nbc "investigation" (which you completely ignored)?
do you think that someone might kill you?

Feel free to send me a PM if you want to know more about me personally. I have no interest in pushing any kind of agenda or relitigating 2014. As I've mentioned on GS before my days of true disappointment ended in 1998 when Michelle lost (and she lost to another American). All of this stuff since is interesting to talk about but does not devastate me or move me very much. In terms of your questions:

1. Someone at NBC probably wanted to do a story on the Olympics. His research paper was out there so they asked him to comment. Academics don't get a lot of attention so I'm sure he was thrilled to put his research out there. His paper and NBC's language have nothing to do with each other. He is not an editor at NBC or employed by them. I'm not so sure why that is difficult to understand. I can provide a quote for a newspaper and not at all agree with the conclusions of the writer.

2. I said I was afraid because I didn't want this to turn into a Sochi thread, which I am very tired of. That's why I tried to focus it on the research. I'm interested (like many of us are) in improving the system. My questions have to do with whether the ISU is looking and conducting this kind of research and whether they can implement reforms to ameliorate the issue.