- Joined
- Mar 28, 2006
I saw this recently, but did not have time to write about it before now. https://medium.com/@renatamachado_7...he-figure-skating-judging-system-54ef2e033096
The “data science” caught my eye in the first place and got me to click it – something I am interested in for my work life and something that often gets me infuriated and/or amused. A lot of the big data or similar data analyses with “amazing” results esp. when they concern human activities and culture particularly. That is because often it seems that the researchers have not really thought what they are studying and what their results really might mean… This blog left me with similar feelings. Amused because the intention was so clear – Nathan Chen is overscored! – and infuriated because the analysis was really not doing what it was claiming to do.
The author took the results of 10 male figure skaters who have done well in the past two years, isolated the GOEs from their results for a number of years, calculated means and did cool diagrams and scatterplots to visualize the results. She used the final GOEs calculated as a percentage of the BV, not the raw values of -5 to +5.
The premise for the conclusions is that GOEs are a bit like PCS, they get better with time and experience. To a certain extent that is correct, particularly when it comes to steps, spins and chore sequences (IMO). There are rarely such major errors as falls in steps and spins which means that you can follow the development in them. They don’t vary that much year by year (I would imagine without looking at any data actually). Jump GOEs can also be seen as developing with time and practice. A skater introduces a new jump element, it can be not very brilliant at first, but gets better with time and experience, adding more difficult entries is possible etc.
However, when it comes to jumps the GOEs do not follow such a linear development as perhaps imagined by the author. The risk is higher than in the other elements and major mistakes are easier to make. You can have a perfect 3Lz usually, but if you fall you still get -5s and -4s whenever you do that mistake. Skaters can struggle with their jumps one season and be realizing their jump potential to the fullest in another. And it is not a given they will ever do well with the difficult jumps they are desperately trying to do for many seasons (just look at Kevin Reynolds’s track record with quads!).
So, basically calculating the GOE means for jumps per season and comparing them shows how well the skater has been doing jumps that season. If they do really well, the mean is high; if they do hot and cold, it will be average; it they do bad, it can be even in the negative. In addition, using the final GOE means that if a skater does many quads and does them well, the final GOE is very high as the BV is high. For example, Hanyu’s 4Lz got 3,94 as GOE and his 3Lz 1,60 (slightly lower raw scores though) – more than two times higher for the more difficult jump. If a skater with high BV jumps is consistent in many competitions through the season, the GOE mean is going to be also quite high. If a skater does only triples, the GOE will not be that high even though they would get +4s and +5s for every one of them.
The fishy case of Nathan Chen’s GOEs in the last two seasons can be explained by looking at the protocols and realizing that in 2018-9 he did no jump very well, esp. his big jumps were often in the negative side – he was not rewarded unjustly for the poor execution. His only really good competition when it comes to jumps was the Worlds. He won every competition, but mostly because his rivals were doing worse than he was. This fall, on the other hand, Nathan has been pretty consistent in the SP compared to last season. In the FS, not always brilliant with his jumps, but slightly better than last season. The GOEs are generally on the plus side. His jump repertoire consists mostly only of the most difficult jumps, so when he does well, the GOE reward is big.
Also, according to the Skating Scores stats, Nathan’s jump GOE mean in ISU events for 2018-9 was 1.45 and for this season 2.19, which is not even close to the 300% increase as claimed by the author? Which one is the correct one? There seem to be other differences also when one looks at the years in the blog’s per year/diagram, so I wonder if her 2018 actually means calendar year 2018, not season 2018 which would seriously mess up the +3 and +5 systems. If so, forget about this analysis.
Then there is Hanyu who has been getting the top GOE raw scores for quite a few seasons whenever he pulls off any jump. His seasons have been fairly similar for a number of years now – good and bad performances making the GOE totals and means high but not as high as Chen’s this season. If he did a season of mostly good and consistent competitions, he would get a higher GOE mean also. With the 4Lo and 4Lz and a perfect program, probably higher than Chen’s current record. But the difference between his seasons might not be as big as that of Chen’s 2018-9 and 2019-20, because his average is quite high. (And btw, looking at the Skating Scores means for seasons, Hanyu’s GOEs for jumps in 2014-15 0.41 and for 2015-16 is 1.26. And that IS an increase of more than 3 times between two seasons.)
So basically, when you do your any kind of analyses, quantitative or qualitative:
1) know what you are analyzing: what GOE measures and how it works in different contexts 8Spins vs jumps), how its evaluation has changed in the rules over the period you are looking at (and here it is important to maintain the seasonality even with the old +3 system, rule changes apply to seasons, not calendar years)
2) make sure your algorithms work right and don’t mess up different seasons
She did a lot of work of her own when she could have just checked the stats from Skating Scores (which she used to collect the data) and compared them... The sad thing is that this will probably float around the cyberspace for some time and might be used to argue for god knows what...
E
The “data science” caught my eye in the first place and got me to click it – something I am interested in for my work life and something that often gets me infuriated and/or amused. A lot of the big data or similar data analyses with “amazing” results esp. when they concern human activities and culture particularly. That is because often it seems that the researchers have not really thought what they are studying and what their results really might mean… This blog left me with similar feelings. Amused because the intention was so clear – Nathan Chen is overscored! – and infuriated because the analysis was really not doing what it was claiming to do.
The author took the results of 10 male figure skaters who have done well in the past two years, isolated the GOEs from their results for a number of years, calculated means and did cool diagrams and scatterplots to visualize the results. She used the final GOEs calculated as a percentage of the BV, not the raw values of -5 to +5.
The premise for the conclusions is that GOEs are a bit like PCS, they get better with time and experience. To a certain extent that is correct, particularly when it comes to steps, spins and chore sequences (IMO). There are rarely such major errors as falls in steps and spins which means that you can follow the development in them. They don’t vary that much year by year (I would imagine without looking at any data actually). Jump GOEs can also be seen as developing with time and practice. A skater introduces a new jump element, it can be not very brilliant at first, but gets better with time and experience, adding more difficult entries is possible etc.
However, when it comes to jumps the GOEs do not follow such a linear development as perhaps imagined by the author. The risk is higher than in the other elements and major mistakes are easier to make. You can have a perfect 3Lz usually, but if you fall you still get -5s and -4s whenever you do that mistake. Skaters can struggle with their jumps one season and be realizing their jump potential to the fullest in another. And it is not a given they will ever do well with the difficult jumps they are desperately trying to do for many seasons (just look at Kevin Reynolds’s track record with quads!).
So, basically calculating the GOE means for jumps per season and comparing them shows how well the skater has been doing jumps that season. If they do really well, the mean is high; if they do hot and cold, it will be average; it they do bad, it can be even in the negative. In addition, using the final GOE means that if a skater does many quads and does them well, the final GOE is very high as the BV is high. For example, Hanyu’s 4Lz got 3,94 as GOE and his 3Lz 1,60 (slightly lower raw scores though) – more than two times higher for the more difficult jump. If a skater with high BV jumps is consistent in many competitions through the season, the GOE mean is going to be also quite high. If a skater does only triples, the GOE will not be that high even though they would get +4s and +5s for every one of them.
The fishy case of Nathan Chen’s GOEs in the last two seasons can be explained by looking at the protocols and realizing that in 2018-9 he did no jump very well, esp. his big jumps were often in the negative side – he was not rewarded unjustly for the poor execution. His only really good competition when it comes to jumps was the Worlds. He won every competition, but mostly because his rivals were doing worse than he was. This fall, on the other hand, Nathan has been pretty consistent in the SP compared to last season. In the FS, not always brilliant with his jumps, but slightly better than last season. The GOEs are generally on the plus side. His jump repertoire consists mostly only of the most difficult jumps, so when he does well, the GOE reward is big.
Also, according to the Skating Scores stats, Nathan’s jump GOE mean in ISU events for 2018-9 was 1.45 and for this season 2.19, which is not even close to the 300% increase as claimed by the author? Which one is the correct one? There seem to be other differences also when one looks at the years in the blog’s per year/diagram, so I wonder if her 2018 actually means calendar year 2018, not season 2018 which would seriously mess up the +3 and +5 systems. If so, forget about this analysis.
Then there is Hanyu who has been getting the top GOE raw scores for quite a few seasons whenever he pulls off any jump. His seasons have been fairly similar for a number of years now – good and bad performances making the GOE totals and means high but not as high as Chen’s this season. If he did a season of mostly good and consistent competitions, he would get a higher GOE mean also. With the 4Lo and 4Lz and a perfect program, probably higher than Chen’s current record. But the difference between his seasons might not be as big as that of Chen’s 2018-9 and 2019-20, because his average is quite high. (And btw, looking at the Skating Scores means for seasons, Hanyu’s GOEs for jumps in 2014-15 0.41 and for 2015-16 is 1.26. And that IS an increase of more than 3 times between two seasons.)
So basically, when you do your any kind of analyses, quantitative or qualitative:
1) know what you are analyzing: what GOE measures and how it works in different contexts 8Spins vs jumps), how its evaluation has changed in the rules over the period you are looking at (and here it is important to maintain the seasonality even with the old +3 system, rule changes apply to seasons, not calendar years)
2) make sure your algorithms work right and don’t mess up different seasons
She did a lot of work of her own when she could have just checked the stats from Skating Scores (which she used to collect the data) and compared them... The sad thing is that this will probably float around the cyberspace for some time and might be used to argue for god knows what...
E

). I've always felt that this has an effect on the scoring and it would be interesting to check if the numbers support that observation.

I feel like I should just start writing biased articles analyzing tech calls that are made specifically to pander to those who love the most popular/buzz worthy skaters, and make money off all the site traffic. It would be so easy and one could make a killing because, instead of considering balanced analysis, people are chomping at the bit to read articles/stats that reinforce their views especially ones that are presented/formatted in such a way to make it seem like they have actual scientific basis (see above).

