How do federations influence the results?

[petitebrie]

I see this person is affiliated with The Skating Lesson. Some interesting biases. Even using ordinals for PCS evaluation. Can our technical experts give us an evaluation on the credibility of the analyses posted in this link?

I'm guessing the ordinals are hypothetical based on the protocols? If you calculate the PCS (and GOE if you're into that, though it's more labour intensive unless you have a program) from each individual judge you can see how they score skaters relative to each other on components, and you can use those scores to infer a ranking. Stories would obviously vary depending on whether you add the technical scores, but it's interesting to see which judges are generous or stingy, how some judges seem to prefer a given style over others, how outlier scores affect the final result, the existence of national bias can be assessed by statistical analysis, etc.

Sorry if I'm misunderstanding you here!

 

[TGee]

I'm guessing the ordinals are hypothetical based on the protocols? If you calculate the PCS (and GOE if you're into that, though it's more labour intensive unless you have a program) from each individual judge you can see how they score skaters relative to each other on components, and you can use those scores to infer a ranking. Stories would obviously vary depending on whether you add the technical scores, but it's interesting to see which judges are generous or stingy, how some judges seem to prefer a given style over others, how outlier scores affect the final result, the existence of national bias can be assessed by statistical analysis, etc.

Sorry if I'm misunderstanding you here!

As I mentioned on another thread, as a Canadian fan, from the country where 6.0 was loathed, the interest in manufacturing ordinals as a means to assess bias just strikes me as darn odd.

And putting my math hat on, I can anticipate that there are biases that will be introduced when you try to shift an absolute score into an ordinal for comparison.

Issues include:
- ignoring TES which for many is the larger portion of the total score
- if not using SOV but including TES, assuming the impact of judging error is the same on all elements, which it isn't
- assuming that the degree of error is the same across all elements or components, while it may actually be severe but limited to specific elements......
- a small amount or trivial amount of absolute difference that may have no material impact on the total score will look as large as an extreme one

Using a biased measure to assess whether bias exists, may risk slamming certain judge's reputations....

So, in my view if folks want to go with, back of the envelope or "easy to relate to" calculations, I would really hope that they be honest with themselves and others about it, and be cautious about what they think they might be seeing in the data....

 

[andromache]

I see this person is affiliated with The Skating Lesson. Some interesting biases. Even using ordinals for PCS evaluation. Can our technical experts give us an evaluation on the credibility of the analyses posted in this link?

What biases are you noticing? I couldn't find anything. How is using ordinals biased? I actually found the conversion of PCS to ordinals to be fascinating. Showed the American bias for C/B (only judge other than Japan to rank C/B 1st) and the Canadian judge's bias by placing them 5th.

 

[Mango]

I'm guessing the ordinals are hypothetical based on the protocols? If you calculate the PCS (and GOE if you're into that, though it's more labour intensive unless you have a program) from each individual judge you can see how they score skaters relative to each other on components, and you can use those scores to infer a ranking. Stories would obviously vary depending on whether you add the technical scores, but it's interesting to see which judges are generous or stingy, how some judges seem to prefer a given style over others, how outlier scores affect the final result, the existence of national bias can be assessed by statistical analysis, etc.

Sorry if I'm misunderstanding you here!

You have explained the process, thank you! I got that bit, but I didn't get why they are using this system. Because the ISU does not use it. Nor does an ordinal ranking matter when you have a high and low scores for an element dropped. This does not look to be an objective statistical analysis, but I would like to hear from the experts whether or not this system of evaluation has merit. Clearly the ISU does not care for it and has not since 2002.

As I mentioned on another thread, as a Canadian fan, from the country where 6.0 was loathed, the interest in manufacturing ordinals as a means to assess bias just strikes me as darn odd.

And putting my math hat on, I can anticipate that there are biases that will be introduced when you try to shift an absolute score into an ordinal for comparison.

Issues include:
- ignoring TES which for many is the larger portion of the total score
- if not using SOV but including TES, assuming the impact of judging error is the same on all elements, which it isn't
- assuming that the degree of error is the same across all elements or components, while it may actually be severe but limited to specific elements......
- a small amount or trivial amount of absolute difference that may have no material impact on the total score will look as large as an extreme one

Using a biased measure to assess whether bias exists, may risk slamming certain judge's reputations....

So, in my view if folks want to go with, back of the envelope or "easy to relate to" calculations, I would really hope that they be honest with themselves and others about it, and be cautious about what they think they might be seeing in the data....

This is what I was I thinking TGee.

What biases are you noticing? I couldn't find anything. How is using ordinals biased? I actually found the conversion of PCS to ordinals to be fascinating. Showed the American bias for C/B (only judge other than Japan to rank C/B 1st) and the Canadian judge's bias by placing them 5th.

You see, the ISU cares about what judges score an element or a PCS category relative to other judges' scores. Not relative to how they scored another skater or team. So for example, the Italian judge can give a 9.00 to the Italian dancers, and that could be the highest PCS score they gave that night to any team. But if the rest of the panel gave them 9.25 and up, then the 9.00 the Italian judge gave would be the lowest mark and (from my understanding) will be tossed. Obviously judges have biases themselves, they are human with citizenship and loyalty to a nation. This may be part of why the ISU does not use ordinals any longer, after dealing with 6.0 for decades. It's a bit less subjective when you get rid of high and low scores. Then you also have judges who are just harsh markers. I can't recall which Challenger event it was this year, but there was a judge #6 who just scored everyone low. How do you explain that solely with ordinals? The ISU also has reviews on judges marks when they go out of a range, so you should see extreme judging getting corrected.

Though seeing as this individual is affiliated with The Skating Lesson I can already notice that there are somethings they are trying to say. Not entirely objective analysis going on here. That's what is making me cautious and seek further opinions.

 

[qwerty]

You have explained the process, thank you! I got that bit, but I didn't get why they are using this system. Because the ISU does not use it. Nor does an ordinal ranking matter when you have a high and low scores for an element dropped. This does not look to be an objective statistical analysis, but I would like to hear from the experts whether or not this system of evaluation has merit. Clearly the ISU does not care for it and has not since 2002.



This is what I was I thinking TGee.



You see, the ISU cares about what judges score an element or a PCS category relative to other judges' scores. Not relative to how they scored another skater or team. So for example, the Italian judge can give a 9.00 to the Italian dancers, and that could be the highest PCS score they gave that night to any team. But if the rest of the panel gave them 9.25 and up, then the 9.00 the Italian judge gave would be the lowest mark and (from my understanding) will be tossed. Obviously judges have biases themselves, they are human with citizenship and loyalty to a nation. This may be part of why the ISU does not use ordinals any longer, after dealing with 6.0 for decades. It's a bit less subjective when you get rid of high and low scores. Then you also have judges who are just harsh markers. I can't recall which Challenger event it was this year, but there was a judge #6 who just scored everyone low. How do you explain that solely with ordinals? The ISU also has reviews on judges marks when they go out of a range, so you should see extreme judging getting corrected.

Though seeing as this individual is affiliated with The Skating Lesson I can already notice that there are somethings they are trying to say. Not entirely objective analysis going on here. That's what is making me cautious and seek further opinions.

The fact that out-of-range marks will (probably) be thrown out doesn't discount their interest to skaters and fans alike.

What interested me about the Canadian judge wasn't the fact that he was a consistently low marker: he gave Virtue/Moir very high PCS. He gave Chock/Bates very low PCS period, and also low in comparison to how he marked everyone else. The fact that in his tally Chock/Bates were 5th is not insignificant, IMO.

I didn't find it to be biased analysis.

 

[Mango]

The fact that out-of-range marks will (probably) be thrown out doesn't discount their interest to skaters and fans alike.

What interested me about the Canadian judge wasn't the fact that he was a consistently low marker: he gave Virtue/Moir very high PCS. He gave Chock/Bates very low PCS period, and also low in comparison to how he marked everyone else. The fact that in his tally Chock/Bates were 5th is not insignificant, IMO.

I didn't find it to be biased analysis.

Alright then, can we repeat this analysis after Rostelecom Cup too? I think everyone would love to see how things change or remain the same.

I am interested in a lot of statistics, even at the advanced novice level. That doesn't make those stats meaningful. I tried to explain what's most important to the ISU, but I guess it's not that important to fans. Of course, you are free to look at the protocols and analyze as you wish.

 

[qwerty]

Alright then, can we repeat this analysis after Rostelecom Cup too? I think everyone would love to see how things change or remain the same.

I am interested in a lot of statistics, even at the advanced novice level. That doesn't make those stats meaningful. I tried to explain what's most important to the ISU, but I guess it's not that important to fans. Of course, you are free to look at the protocols and analyze as you wish.

Yes, I'm also looking forward to seeing more protocols!

Another reason that I believe low-balling is not insignificant is because if a judge is fairly certain their marks will be thrown out, then the next-lowest mark will have to be counted. I really doubt that this line of thinking is practiced much by the judges, but just a thought. For instance, since the Canadian judge gave a 7.75 in Skating Skills to C/B, they had to then swallow two 8.5s from Judges No. 6 and 8.

 

[Mango]

Yes, I'm also looking forward to seeing more protocols!

Another reason that I believe low-balling is not insignificant is because if a judge is fairly certain their marks will be thrown out, then the next-lowest mark will have to be counted. I really doubt that this line of thinking is practiced much by the judges, but just a thought. For instance, since the Canadian judge gave a 7.75 in Skating Skills to C/B, they had to then swallow two 8.5s from Judges No. 6 and 8.

It was discussed earlier in the thread, but judges really don't know unless there is prior collusion what the others will give. It could just be personal preference. Like people who hate Sam Smith's vocals may just give low marks to a program with such vocals. It makes more sense to look over scores from both the short and free. I saw that with Mao's score at Finandia - one judge gave her low marks in the SP but higher marks in the FS. They just liked the FS more.

 

[qwerty]

It was discussed earlier in the thread, but judges really don't know unless there is prior collusion what the others will give. It could just be personal preference. Like people who hate Sam Smith's vocals may just give low marks to a program with such vocals. It makes more sense to look over scores from both the short and free. I saw that with Mao's score at Finandia - one judge gave her low marks in the SP but higher marks in the FS. They just liked the FS more.

I hadn't looked much at the SD scores until you mentioned this, but when I did, the Canadian judge (sorry to single you out, Sylvain) gave C/B quite low scores again (just .25 above Stepanova/Bukin who were fifth in PCS). Perhaps personal preference, but I can't help but feel this was a concerted effort to bolster Virtue/Moir's chances of success. Hope this doesn't sound too conspiratorial.

 

[Nathan13]

It was discussed earlier in the thread, but judges really don't know unless there is prior collusion what the others will give. It could just be personal preference. Like people who hate Sam Smith's vocals may just give low marks to a program with such vocals. It makes more sense to look over scores from both the short and free. I saw that with Mao's score at Finandia - one judge gave her low marks in the SP but higher marks in the FS. They just liked the FS more.

Just a thought...Why should one's personal preference to music affect the skating skills score? Performance score, interpretation score, choreography, maybe... But I don't see how Skating Skills should be affected by personal preference.

 

[Mango]

Just a thought...Why should one's personal preference to music affect the skating skills score? Performance score, interpretation score, choreography, maybe... But I don't see how Skating Skills should be affected by personal preference.

Because judges are human too. If we acknowledge national bias, then we have to acknowledge other biases too. I'm sure there are judges out there who hate "Feeling Good" and other warhorses and may mark strictly (due a lack of originality in music selectoin). It'd be reflected in other PCS than SS, of course.

 

[Nathan13]

Because judges are human too. If we acknowledge national bias, then we have to acknowledge other biases too. I'm sure there are judges out there who hate "Feeling Good" and other warhorses and may mark strictly (due a lack of originality in music selectoin). It'd be reflected in other PCS than SS, of course.

That may be true, but in the case of Chock and Bates, they received a 7.75 in Skating Skills, but an 8.50 in Composition and Performance (which are the most closely linked to the music style). Even if personal preference plays a role in scoring Performance, Composition, and Interpretation, it should in no way be attached to Skating Skills.

*I am in no way a conspirator--just a Chock and Bates fan who likes to look at protocols*

 

[Mango]

That may be true, but in the case of Chock and Bates, they received a 7.75 in Skating Skills, but an 8.50 in Composition and Performance (which are the most closely linked to the music style). Even if personal preference plays a role in scoring Performance, Composition, and Interpretation, it should in no way be attached to Skating Skills.

*I am in no way a conspirator--just a Chock and Bates fan who likes to look at protocols*

I am and have been discussing results in general.

Chock/Bates could likely tell you what feed back they got from the judges if someone asked them.

 

[Ice Dance]

OK. With regard to the, "Why not invite judges who do not have their own athletes in the event?" question:

1. Because we want educated judging panels. Often the most educated judges come from countries in which the sport & discipline have existed for many years.

2. Because there are FAR more figure skating countries in Europe. If you eliminate U.S., Canadian, Japanese, Chinese, and Korean judges from all the judging panels in which those athletes compete--in some cases this might be every GP & major international event in every discipline--then the panels would be even more Euro-centric than they already are. And this is a legitimate problem because different styles of skating are respected and valued in different parts of the world.

3. Just because a judge does not represent a major skating power does not mean he or she is not strongly linked to one. Most athletes from minor skating countries ultimately train with big-name coaches or work with big-name choreographers elsewhere. Galit Chait is a tech specialist (one I respect because she has a reputation for being tough). She is A. an Israeli, B. moved to live in the U.S. when she was about the age of two, C. skated with a former Russian athlete, D. is married to someone from . . . Italy, was it?, and I'm sure has many other ties within her life. A mighty diverse international resume. Korea has three ice dance teams. One trains in Russia. One in the U.S. One in Canada. Does this make them unbiased? Who knows? The Belarusian team has Russian coaches. They have been invited to Rostelecom twice in the past two years. Is a Belarusian judge, therefore, unbiased when judging Russian athletes? Dubreil & Lauzon train French, Canadian, American, and Danish dance teams. Marina trains the Finns, the Japanese, and Americans (also has a Canadian singles skater & choreographs for athletes worldwide). Igor trains Americans, French, Russian, Israeli, and Ukrainian teams.

You can make yourself crazy trying to find an unconnected ice dance judge/tech controller/coach. And when you finally do, that judge is going to wind up being the one who can't tell the elements apart and might well be ready to give anything in order to find some support for his or her team.

So, while I ask myself exactly the same question every time I see a judge give the highest mark for an athlete from his/her country, I then remind myself that if that judge gives the highest mark and also gives the major competition the lowest mark, those scores are getting thrown out anyway.

 

[TGee]

That may be true, but in the case of Chock and Bates, they received a 7.75 in Skating Skills, but an 8.50 in Composition and Performance (which are the most closely linked to the music style). Even if personal preference plays a role in scoring Performance, Composition, and Interpretation, it should in no way be attached to Skating Skills.

*I am in no way a conspirator--just a Chock and Bates fan who likes to look at protocols*

I'd like to drop in another thought....beyond the math methodology concerns that are quite serious for me, especially when folks do rough and ready calculations with a single data point and then try to make inferences, ......

So, here goes, Chock and Bates have made a moderately courageous choice in their music and choreography for the Free Dance. And that is a risk when taking a new program to a diverse international panel, especially in Ice Dance.

I happen to really like it, but I also really liked the original version of Gilles and Poirier's FD last year and was disappointed that they felt they needed to tone it down. It was good dance, and while I can appreciate and enjoy what folks in the dance world call "storybook ballet", I really prefer dance that is out there, edgy and modern.

While Chock and Bates FD isn't as far out as that, and may seem less out there to an American audience as it is rooted in American dance, one can't expect it to be as easily understood or appreciated outside the US.

Franky, I believe that it's a testament to the increasing openness of ice dance judges to diverse and new dance styles that Chock and Bates won the Free Dance.

This is their first GP and we did not hear the usual about "giving the judges time to understand /appreciate what they were doing" or "needing to make adjustments."

For a courageous ice dance program in October, that's as close to a slam dunk as it gets.:agree:

Perhaps then, their fans could declare victory rather than looking for inconclusive evidence that they were not given something they deserved.....

 

[Nathan13]

I'd like to drop in another thought....beyond the math methodology concerns that are quite serious for me, especially when folks do rough and ready calculations with a single data point and then try to make inferences, ......

So, here goes, Chock and Bates have made a moderately courageous choice in their music and choreography for the Free Dance. And that is a risk when taking a new program to a diverse international panel, especially in Ice Dance.

I happen to really like it, but I also really liked the original version of Gilles and Poirier's FD last year and was disappointed that they felt they needed to tone it down. It was good dance, and while I can appreciate and enjoy what folks in the dance world call "storybook ballet", I really prefer dance that is out there, edgy and modern.

While Chock and Bates FD isn't as far out as that, and may seem less out there to an American audience as it is rooted in American dance, one can't expect it to be as easily understood or appreciated outside the US.

Franky, I believe that it's a testament to the increasing openness of ice dance judges to diverse and new dance styles that Chock and Bates won the Free Dancce.

This is their first GP and we did not here the usual about "giving the judges time to understand /appreciate what they were doing" or "needing to make adjustments."

For a courageous ice dance program in October, that's as close to a slam dunk as it gets.:agree:

Perhaps then, their fans could declare victory rather than looking for inclusive evidence that they were not given something they deserved.....

This is exactly my sentiments--and yes, a 112 in October speaks highly of the scoring potential for the rest of the season. Personally, I don't really think that whether or not a judge likes their music or chose style of dancing should really influence PCS anyway.

 

[andromache]

I'd like to drop in another thought....beyond the math methodology concerns that are quite serious for me, especially when folks do rough and ready calculations with a single data point and then try to make inferences, ......

So, here goes, Chock and Bates have made a moderately courageous choice in their music and choreography for the Free Dance. And that is a risk when taking a new program to a diverse international panel, especially in Ice Dance.

I happen to really like it, but I also really liked the original version of Gilles and Poirier's FD last year and was disappointed that they felt they needed to tone it down. It was good dance, and while I can appreciate and enjoy what folks in the dance world call "storybook ballet", I really prefer dance that is out there, edgy and modern.

While Chock and Bates FD isn't as far out as that, and may seem less out there to an American audience as it is rooted in American dance, one can't expect it to be as easily understood or appreciated outside the US.

Franky, I believe that it's a testament to the increasing openness of ice dance judges to diverse and new dance styles that Chock and Bates won the Free Dance.

This is their first GP and we did not hear the usual about "giving the judges time to understand /appreciate what they were doing" or "needing to make adjustments."

For a courageous ice dance program in October, that's as close to a slam dunk as it gets.:agree:

Perhaps then, their fans could declare victory rather than looking for inconclusive evidence that they were not given something they deserved.....

For me, the "analysis" (which I understand your concerns with but I just see it as an interesting exercise with interesting results) is not an attempt to "prove" or provide evidence for any wrongdoing. It is often discussed here how PCS aren't that far off from a glorified ordinal system, where scores are used to "rank" the skaters, either according to how the judges objectively view them or who the judges want to win. Seeing the high skating skills scores for Ashley Wagner (my girllllllll), for example, makes me inclined to say that judges thought she gave the best performance and wanted to reward her all around by putting her in first and didn't pay that much attention to each individual PCS mark.

Anyway, I definitely thought it was neat (not nefarious) that the Canadian judge gave the highest PCS to V/M and G/P, the US judge to C/B, and the Italian judge to C/L.

 

[TGee]

For me, the "analysis" (which I understand your concerns with but I just see it as an interesting exercise with interesting results) is not an attempt to "prove" or provide evidence for any wrongdoing. It is often discussed here how PCS aren't that far off from a glorified ordinal system, where scores are used to "rank" the skaters, either according to how the judges objectively view them or who the judges want to win. Seeing the high skating skills scores for Ashley Wagner (my girllllllll), for example, makes me inclined to say that judges thought she gave the best performance and wanted to reward her all around by putting her in first and didn't pay that much attention to each individual PCS mark.

Anyway, I definitely thought it was neat (not nefarious) that the Canadian judge gave the highest PCS to V/M and G/P, the US judge to C/B, and the Italian judge to C/L.

I can definitely agree that the current PCS approach in practice seems to be a bit of a muddle between COP and some kind of 6.0 like approach.

The upper bound and factoring really make it less bottom up than it should be. . . For example, why should singles skaters be getting higher scores on SS than ice dancers, who everyone acknowledges are stronger in his area?

And I agree with Nathan13 that SS and TR should be the parts of PCS that are most independent of the perception of overall performance. And it appears that many judges are looking at those against the criteria in a bottom up way, while others seem to be giving an "impression score". So even if there is bias in PCS scoring some judges may do it across the board, while others only do it on components other than SS and TR....

Interestingly, we hear enormous fan criticism both ways. When Carolina Kostner or Patrick Chan have a flawed technical performance but they still skate programs that hit all he SS and TR criteria, folks talk about PCS inflation when they get high SS and TR scores. These folks seem to want PCS to be assessed like the old "artistic impression" ordinal.

But there are also folks like Nathan13 and me who are looking for these components to be assessed independently and wonder if a skater has been underscored when we see strength in these areas despite a performance that is otherwise not as strong as it might have been.

So, I'd be less worried about looking at ordinals on the PCS alone

1. If it weren't for the fact that folks are making some strong statements about bias and gaming the system while not being clear that they are only looking at one part of the scoring (one poster actually said "the numbers don't lie" on another thread)

2. If folks looked at SS and TR separately from the other program components to see if the pattern varied.

 

[aromaticchicken]

Hi there, I just wanted to write in because I am the editor of @skatingprotocol. I've been a long time poster on Golden skate, though I've posted less frequently recently.

It's not my goal to reveal any major biases in my analyses. You'll see that I've called out judges from all countries, including American, which is my own nationality. My investment in this is to enjoy the statistics (I created a program that calculates all the scores), and also because, well... No one else was doing it. The whole point of transparency in figure skating and IJS is to call out unfair judging and make the sport more fair. Judges who score way far off the range (higher or lower) are defeating that purpose.

I am indeed friends with Dave Lease, but I can assure you that I am the one writing all of the posts -- he encouraged me to start it up since I mentioned it, and is resharing a lot of my posts because hopefully it is good and interesting content. I also intend on writing full length posts on the Skating Lesson page. And for what it's worth, he and I disagree plenty on a number of skaters and teams. But hopefully that doesn't come across anyway, since my goal is to report the statistics fairly.

Happy to field any further questions. - Mark

Sent from my SM-N920T using Tapatalk

 

[aromaticchicken]

What biases are you noticing? I couldn't find anything. How is using ordinals biased? I actually found the conversion of PCS to ordinals to be fascinating. Showed the American bias for C/B (only judge other than Japan to rank C/B 1st) and the Canadian judge's bias by placing them 5th.

I see this person is affiliated with The Skating Lesson. Some interesting biases. Even using ordinals for PCS evaluation. Can our technical experts give us an evaluation on the credibility of the analyses posted in this link?

Oh - one final note (please see my earlier reply in the thread)-- I've been using ordinals only when it was easier to see among multiple scores (e.g.,with 3-5 teams across 9 judges) which way the judges went. When possible, I've been erring on the side of posting the full scores. They are time intensive to calculate, so I usually try to include them since I have to calculate them anyway.

Sent from my SM-N920T using Tapatalk