In Praise of the Judges | Golden Skate

In Praise of the Judges

GGFan

Record Breaker
Joined
Nov 9, 2013
This is crazy but I'm going to do something I never thought I would do: I'm going to defend the judges! :shocked: I think this is necessary because every season we have heated discussions about how terrible the judging is. Well for once I'm gonna take the opposite side and argue that blaming the judges is akin to people blaming the media for everything.

For the sake of catharsis it is quite easy to pick a scapegoat to blame for our issues. I have no problem with that. We all do it and it's a common human experience. I think we should complain and keep the judges honest. But sometimes we get on soapboxes and get really righteous unnecessarily. We act as if there is some conspiracy on the part of the judges to keep some skaters down

I think the NHK result between Liza is a perfect example. Folks are roasting the judges for placing Liza ahead of Satoko. Some have gone as far as to suggest home cooking. But these arguments are built on several faulty premises.

1. We assume that the judges are one person that decides who should end up in what spot. The truth is that the judges are 9 individuals and 2 specialists, and even if they sought to collude there are different interests and incentives that would keep them from agreeing. I'm sure someone with game theory experience could elaborate here.

2. We assume that the judges have the amount of time to digest the performances that we have. The judges are making quick decisions. Yes, there is some prejudging but they also have to make decisions that I'm sure they would slightly revise with the benefit of time and hindsight.

3. We assume that the judges decide the placings following everyone's performance. They have to judge the event as it goes. That gives us a huge advantage over them.

4. We tend to ignore the law of averages. *I'm not a math person so Im really sorry if I get some details wrong here. Because there are so many elements to judge and so little time no one judge can be as strategic as we assume. In order to make distinctions between competitors they have to keep track of the scores they gave the competitors on each element. That's a lot for the human brain to keep up with. Thus to really distinguish and be safe they would have to make bigger distinctions (it's easier to remember that you gave one person an 8 vs. 9 as opposed to an 8.25 or an 8.50). But if you make your distinctions too big you risk having your score thrown out for being too high or low.

5. We assume that the judges can control tight decisions. When it gets really close like it did between Satoko and Liza the judges are in a crapshoot situation. If we for example have them skate tomorrow the outcome would likely be different. Because Liza and Satoko excel at opposite things it is really difficult for the judges to control the outcome. #4 would be really hard for any one individual judge to manage because they would have to be even more extreme in their distinctions since it's hard to figure out how the TES+PCS score would turn out so you have to be more extreme to be safe. But being extreme would make it more likely that their score would get thrown out.

I think there are things we could do to improve judging but we need to step back once in a while and put ourselves in the shoes of the judges. There's is no monolithic evil person manipulating the scoring and trying to hurt your favorite. The system is flawed but the math does keep the judges somewhat honest. :devil:
 
Last edited:

HanDomi

Record Breaker
Joined
Feb 27, 2014
The judging this season is definitely big improvement, but still some way to go. At least they started to call UR's
 

lesnar001

Record Breaker
Joined
Jan 19, 2005
Whenever the score is so close, there will be peope complaining one way of another.
To me, the placements were absolutely fine.
Satoko's TES was even marked down by UR's.
If you want to look at 1 deciding factor - Liza had a "hitch" or something after she landed the triple axel and got -GOE.
If she had landed that jump cleanly, like she did in the SP, she would have been 2nd here and not 3rd.
 

Tolstoj

Record Breaker
Joined
Nov 21, 2015
The problem is not about a mistake, but multiple, and that's the same argument all the time.

It got really close between Satoko and Liza because judges wanted to get it that close.

I don't honestly get the argument "Satoko = bad technique; Liza = bad artistry so that's why scores are close" because i don't think the two issues are comparable at all.

Elizaveta Tuktamysheva's only real weakness in the components is lack of transitions; the performance is there, the interpretation is there, she has decent skating skills (obviously not on par with Miyahara),... Transitions is 1 element out of 5.

On the other hand Satoko has many mistakes on her jumping technique (edge issues for both flip and lutz, bad take-off, prerotation, underrotation, and the jumps are the smallest you can see in senior ladies), which quite frankly shouldn't let her being a top contender at all, i'm always shocked when i see her TES over 40 and over 70, because that means some judge gave her +3 and +4 for her jumps which is an actual mistake according to the new rules.

It bugs me soo much because we've seen this soo many times, ISU even changed the rules and here we are again.
 

GGFan

Record Breaker
Joined
Nov 9, 2013
I don't honestly get the argument "Satoko = bad technique; Liza = bad artistry so that's why scores are close" because i don't think the two issues are comparable at all.

I think this may be a major part of the difference in opinion among some. There are those for whom the jumps matter a lot. It's a sport and the differences between their jumping prowess is great.

But there are also those who focus in on the details and see a wide gap between Satoko and Liza. The PCS is not only performance. Once you get into the details Satoko is beating Liza everywhere.

The TES also takes into account spins and footwork and Satoko beats Liza there as well. As Mishin well knows Liza's lack of content outside of the jumps helps her to perform better. Would she land everything as well if she put in more content and focused on moving her entire body like Satoko does--we don't know because she has not done that. That's a choice they've made. It's completely legitimate for the judges to dock them for that choice.

And in the end it was really close.
 

moriel

Record Breaker
Joined
Mar 18, 2015
I think we should know our heros.
Praise to judges from TPE, GER, AUS.
Judge from CHN... Well, we will let him live.
Russian and japanese judges supported their own skaters.
Judges from CAN, KOR and USA did not deserve any chocolate today.
Judge from USA, specifically, deserves a spank for being a bad girl and scoring Leonova like that.


My thoughts here - if we take the PCs as given by each judge, and subtract the actual PCs, we should have some positive and negative values.
If the judge is not biased, roughly half of the scores he gives to japanese will be positive, and half negative, same for russians.
If judge is equally biased towards both groups he will give lower or higher scores to everybody, regardless nationality (see the grumpy german judge).

Of course, no statistical evidence due to small sample size, and all the comments here are just my thoughts.

Scores above average given by judge:
to japanese girls / to russian girls
TPE 2/4 - this guy deserves a chocolate
RUS 3/6 - supported their own
GER 1/1 - this guy hates everybody, deserves chocolate
CAN 3/0 - underscored russian ladies
JPN 4/1 - supported their own by screwing other people
AUS 5/4 - this guy loves everybody, deserves chocolate
KOR 1/4 - underscored japanese ladies
USA 4/0 - underscored russian ladies, deserves trophy for worst judging for the -9,26 he gave to Leonova

CHN 4/6 - this guy deserves a chocolate


PCs by each judge, with actual score subtracted from it
Japanse
SP------- Kihira - Miyahara - Mihara
TPE ____ -0,59 ____ -0,35 ____ 0,32
RUS ____ -1,99 ____ -1,55 ____ 2,52
GER ____ -0,19 ____ -1,15 ____ -0,48
CAN ____ 1,81 ____ 0,85 ____ -2,48
JPN ____ -0,79 ____ 1,25 ____ 0,72
AUS ____ 1,21 ____ 1,65 ____ 4,12
KOR ____ -0,99 ____ -0,75 ____ -0,08
USA ____ 0,41 ____ 0,05 ____ -1,08
CHN ____ 1,21 ____ 0,25 ____ -2,08

FS------- Kihira - Miyahara - Mihara
TPE ____ -1,15 ____ 1,7 ____ -0,51
RUS ____ -2,75 ____ 0,1 ____ 1,09
GER ____ -1,95 ____ 1,7 ____ -3,31
CAN ____ 1,25 ____ -1,9 ____ -0,91
JPN ____ 1,65 ____ 0,9 ____ -1,71
AUS ____ 1,25 ____ -2,7 ____ 1,89
KOR ____ -1,95 ____ -0,7 ____ 1,49
USA ____ 2,05 ____ 0,5 ____ -0,11
CHN ____ 1,25 ____ -0,3 ____ 1,09

Russians
SP------- Tuktamysheva - Leonova - Sotskova
TPE ____ 0,01 ____ 0,74 ____ 2,4
RUS ____ 1,61 ____ 1,34 ____ 0,4
GER ____ 0,21 ____ -0,06 ____ -1,2
CAN ____ -0,79 ____ -2,06 ____ -1
JPN ____ -1,99 ____ 0,54 ____ -1,8
AUS ____ 3,81 ____ 4,34 ____ 0,4
KOR ____ -1,19 ____ -1,06 ____ 0,8
USA ____ -1,39 ____ -3,26 ____ -1
CHN ____ 1,41 ____ 0,54 ____ 1,4

FS------- Tuktamysheva - Leonova - Sotskova
TPE ____ -0,64 ____ -0,06 ____ 1,77
RUS ____ 3,36 ____ 2,74 ____ 5,77
GER ____ -3,04 ____ -4,86 ____ -4,63
CAN ____ -0,24 ____ -0,46 ____ -2,63
JPN ____ -1,84 ____ -1,66 ____ -1,43
AUS ____ -0,24 ____ 3,54 ____ -1,83
KOR ____ 1,76 ____ 1,54 ____ 3,77
USA ____ -1,44 ____ -9,26 ____ -6,23
CHN ____ 1,76 ____ 3,14 ____ 4,57
 

GGFan

Record Breaker
Joined
Nov 9, 2013
Thank you so much for that detailed analysis moriel! I love when I learn things on GS. :clap::thank:
 

Shayuki

Record Breaker
Joined
Nov 2, 2013
The main issue with judging still is how skaters aren't differentiated enough for different things. Tuktamysheva-Miyahara is an extremely easy example so let's use that. The PCS should be further apart, as should StSq / spin GOE. However, the jump GOE should be much further in Tuktamysheva's favor. Perhaps the end result would be acceptable even if it was the same in the end. But it honestly is a problem to me that every jump gets the same GOE as long as it's essentially clean. It's like they completely failed to achieve what -5/+5 was supposed to achieve.
 

Jeanie19

Record Breaker
Joined
Oct 20, 2017
Country
United-States
I think we should know our heros.
Praise to judges from TPE, GER, AUS.
Judge from CHN... Well, we will let him live.
Russian and japanese judges supported their own skaters.
Judges from CAN, KOR and USA did not deserve any chocolate today.
Judge from USA, specifically, deserves a spank for being a bad girl and scoring Leonova like that.


My thoughts here - if we take the PCs as given by each judge, and subtract the actual PCs, we should have some positive and negative values.
If the judge is not biased, roughly half of the scores he gives to japanese will be positive, and half negative, same for russians.
If judge is equally biased towards both groups he will give lower or higher scores to everybody, regardless nationality (see the grumpy german judge).

Of course, no statistical evidence due to small sample size, and all the comments here are just my thoughts.

Scores above average given by judge:
to japanese girls / to russian girls
TPE 2/4 - this guy deserves a chocolate
RUS 3/6 - supported their own
GER 1/1 - this guy hates everybody, deserves chocolate
CAN 3/0 - underscored russian ladies
JPN 4/1 - supported their own by screwing other people
AUS 5/4 - this guy loves everybody, deserves chocolate
KOR 1/4 - underscored japanese ladies
USA 4/0 - underscored russian ladies, deserves trophy for worst judging for the -9,26 he gave to Leonova

CHN 4/6 - this guy deserves a chocolate


PCs by each judge, with actual score subtracted from it
Japanse
SP------- Kihira - Miyahara - Mihara
TPE ____ -0,59 ____ -0,35 ____ 0,32
RUS ____ -1,99 ____ -1,55 ____ 2,52
GER ____ -0,19 ____ -1,15 ____ -0,48
CAN ____ 1,81 ____ 0,85 ____ -2,48
JPN ____ -0,79 ____ 1,25 ____ 0,72
AUS ____ 1,21 ____ 1,65 ____ 4,12
KOR ____ -0,99 ____ -0,75 ____ -0,08
USA ____ 0,41 ____ 0,05 ____ -1,08
CHN ____ 1,21 ____ 0,25 ____ -2,08

FS------- Kihira - Miyahara - Mihara
TPE ____ -1,15 ____ 1,7 ____ -0,51
RUS ____ -2,75 ____ 0,1 ____ 1,09
GER ____ -1,95 ____ 1,7 ____ -3,31
CAN ____ 1,25 ____ -1,9 ____ -0,91
JPN ____ 1,65 ____ 0,9 ____ -1,71
AUS ____ 1,25 ____ -2,7 ____ 1,89
KOR ____ -1,95 ____ -0,7 ____ 1,49
USA ____ 2,05 ____ 0,5 ____ -0,11
CHN ____ 1,25 ____ -0,3 ____ 1,09

Russians
SP------- Tuktamysheva - Leonova - Sotskova
TPE ____ 0,01 ____ 0,74 ____ 2,4
RUS ____ 1,61 ____ 1,34 ____ 0,4
GER ____ 0,21 ____ -0,06 ____ -1,2
CAN ____ -0,79 ____ -2,06 ____ -1
JPN ____ -1,99 ____ 0,54 ____ -1,8
AUS ____ 3,81 ____ 4,34 ____ 0,4
KOR ____ -1,19 ____ -1,06 ____ 0,8
USA ____ -1,39 ____ -3,26 ____ -1
CHN ____ 1,41 ____ 0,54 ____ 1,4

FS------- Tuktamysheva - Leonova - Sotskova
TPE ____ -0,64 ____ -0,06 ____ 1,77
RUS ____ 3,36 ____ 2,74 ____ 5,77
GER ____ -3,04 ____ -4,86 ____ -4,63
CAN ____ -0,24 ____ -0,46 ____ -2,63
JPN ____ -1,84 ____ -1,66 ____ -1,43
AUS ____ -0,24 ____ 3,54 ____ -1,83
KOR ____ 1,76 ____ 1,54 ____ 3,77
USA ____ -1,44 ____ -9,26 ____ -6,23
CHN ____ 1,76 ____ 3,14 ____ 4,57

Moriel,

Will you do the same for ladies of Skate Canada.
 

GGFan

Record Breaker
Joined
Nov 9, 2013
The main issue with judging still is how skaters aren't differentiated enough for different things. Tuktamysheva-Miyahara is an extremely easy example so let's use that. The PCS should be further apart, as should StSq / spin GOE. However, the jump GOE should be much further in Tuktamysheva's favor. Perhaps the end result would be acceptable even if it was the same in the end. But it honestly is a problem to me that every jump gets the same GOE as long as it's essentially clean. It's like they completely failed to achieve what -5/+5 was supposed to achieve.

Agreed. But then we also have to admit that even when individual judges distinguish averages tends to lessen the gap. Also, as experts like gkelly have pointed out the PCS has to cover all levels of skating. With the top skaters you're really just distinguishing between 7-9s. Once you throw out highs and lows, etc. things get close.

And that's really my point. Even with moriel's analysis you can see that there are 4 judges that keep the outliers honest. It's pretty unusual (and then it's usually obvious) for all 9 judges and the callers to work together to favor any one skater. There are too many individuals and inputs involved.
 

Nathan13

Medalist
Joined
Dec 1, 2015
My thoughts here - if we take the PCs as given by each judge, and subtract the actual PCs, we should have some positive and negative values.
If the judge is not biased, roughly half of the scores he gives to japanese will be positive, and half negative, same for russians.
If judge is equally biased towards both groups he will give lower or higher scores to everybody, regardless nationality (see the grumpy german judge).

Of course, no statistical evidence due to small sample size, and all the comments here are just my thoughts.
Agree with the sentiment, however I do think it is important to not take the actual scores that the skaters received as "correct." It's just the opinion of 9 highly qualified people, while another 9 highly qualified people could come up with a totally different result.

Having said that, I agree that Leonova deserved far better PCS than some judges awarded her, so this is an interesting way of looking at it.
 

moriel

Record Breaker
Joined
Mar 18, 2015
Agree with the sentiment, however I do think it is important to not take the actual scores that the skaters received as "correct." It's just the opinion of 9 highly qualified people, while another 9 highly qualified people could come up with a totally different result.

Having said that, I agree that Leonova deserved far better PCS than some judges awarded her, so this is an interesting way of looking at it.

Oh, I don't take the scores as "correct". I just look at the relative behaviour.
Subtracting this "average" score out of individual scores makes it easier to see who consistently scores high and who consistently scores low.


Will do skate Canada laters tonight
 

Leonardo

Final Flight
Joined
Feb 17, 2010
i'm always shocked when i see her TES over 40 and over 70, because that means some judge gave her +3 and +4 for her jumps which is an actual mistake according to the new rules.

If you look at the NHK trophy protocols, Satoko got only one +4 in the sp for her jumps (eliminated) and only two +4s (eliminated) for her jumps in the free. The other 90 goes for her jumps were all +3 or below. According to the new rules she can get up to +3 in goe for her jumps (including musicality, body position, flow, transitions etc). So, no, she isn't scoring +4s for her jumps.

Imo she is indeed overscored for her jumps, but seriously underscored in pcs (especially in the past seasons). Zagitova was scoring much higher than Satoko in pcs in her first senior season, that's absurd, but of course, nobody cares. When the judges agree with you (not necessarily you, it's just an example), the judges are right. When they don't, they are horrible.

Liza is an amazing skater, but the difference in skating skills, transitions, the sophistication and level of performance, choreography, interpretation, spins, steps and chsq between her and Satoko is absolutely visible (except for someone like Kovtun, lol). So yes, the issues are comparable. It takes years and years to develop both incredible pcs, spins and steps or incredible jumps.
We know Liza doesn't have many transitions or good spins or sophisticated choreo/interpretation (more eurotrash like), but we like her, we respect her and we don't insisting on calling her over and over about that even when she scores +4s for spins, steps and even 9s for pcs. It's tiring and unecessary.

And yes, Satoko's a top contender because she is consistent, she's fierce, she's very well trained, an extremely hard worker, a fighter and an absolutely gorgeous skater with unique artistry and many other skills, except for technique in some of her jumps. I'm sorry but nothing people say will change that or the fact the she touches so many hearts and fans. The worst part for some is: she keeps improving and she keeps getting medals. :dance2:
 

yume

🍉
Record Breaker
Joined
Mar 11, 2016
Zagitova is getting slashed since last season because of her PCS. So i don't think that nobody cares.
Liza's spins and overall artistry is not great. That's why she doesn't receive sky high GOEs and components for that, despite delivering difficult tech contents. Some can argue that she can receive slightly less, like 2 points, but that's not big overscoring IMO. And i see many people point what she lacks everytime she competes and complain about her PCS, including at NHK.
 

Tolstoj

Record Breaker
Joined
Nov 21, 2015


The problem for me personally is that we keep minimizing the issue to "hey her jumping technique is not as impressive, but the rest is perfect", but it's not just that her jumping technique is not impressive, Medvedeva's jumping technique is not very impressive.

Satoko's jumping technique is absolutely atrocious, it's not just "some of her jumps" cause she has zero height and no distance plus prerotation for all her triples + edge issues for both the Flip and the Lutz + underrotations + the infamous blade assist on the lutz. So how can you give a pass to all of that for every single jump + giving her positive GOE?

No wonder she is that consistent, less height also mean less risks to lose balance, and Satoko in particular barely leaves the ice on her jumps.

When you call a jump underrotated, the GOE for that jump is usually a 0 or negative, rarely a +1, that's the reality for most skaters, but not for Satoko where GOE are still positives somehow even in the rare occasions where judges decide to call one of her underrotated jump. (despite all of them are always suspicious)

It strikes me cause between Juniors and Seniors i've always seen skaters getting screwed for smaller mistakes, judges had so many chances to give her the carrot cake she honestly deserves, instead they suddenly decided it was okay for her to keep all these mistakes.

The most annoying part is that in Figure Skating every fan has its own favourites, so all these discussions are getting dimished as "it's just a comparison between x and y, fan x vs fan y" while for me it's not just that, i've always thought her scores are a bit made up by the judges, and there have been many other occasions in the past like this one where she beats skaters who were in my opinion deserved more. (like few weeks ago Kaori Sakamoto at Skate America, or two years ago Anna Pogorilaya at GPF, i bet there will be a new one at GPF this season)

"TES is also spins and footwork", it's time to stop that as well, because while it's true, they don't count that much.

If you combine Satoko's 3 spins + stsq +chsq from her free skate = 22.59 points out of 71.89 points (her TES), the rest (49.3 points) is all jumps.

49.3 : 71.89 = X : 100

68.5% of her TES in the Free skate is jumps, so when you decide to inflate that part, you're essentially overscoring most of her TES.

I feel sorry for the athlete cause she seems a lovely person, she works hard like every other skater at that level, but you shouldn't pretend her mistakes don't exist because of that.

There is soo much more i'd like to add about the GOE for jumps and spins but it's almost 4 am here so i'll end here.
 

skatenewbie

Medalist
Joined
Mar 16, 2017
Satoko is overscored in TES. Clearly she wad pushed to be in front of Tuktamysheva in SP, almost 41 TES is a crime tbh. Some of her triples GOE is higher than the likes of Leonova, Mihara, Tuktamysheva etc. She deserve the PCS though but i still think Liza should have been 2nd
 

GGFan

Record Breaker
Joined
Nov 9, 2013
the same arguments about Satoko are repeated without paying attention to the main argument of the thread which is that judges do not vote together as a block and that they do not get to decide placements after everyone has skated. There's not universal skating conspiracy. I just think people want to be able to say that the results mirrored what they wanted.

Unless the judges make a rule specifically against Satoko's jumps you will not get your wish. She is not per se disqualified from winning competitions because of her jumps. That is not how it works.
 

Roast Toast

Medalist
Joined
Apr 13, 2017
It's true, i was in the arena and sadly Satoka was only jumping singles which were erroneously marked as triples by the judges and given +5 GOes :cry:(((( The truth is that she was actually OVERSCORED because she only got two edge calls and two URs and her 3Lze!<<<+3T<<< got a fraudulent 0.67 points bonus score.

I also believe Lisa was UNDERSCORED in PCs but sadly because i sat in corner of the rink I never got to see her up close. She has wonderful ice coverage especially near center ice
 

Shanshani

On the Ice
Joined
Mar 21, 2018
I wasn't gonna post this until I was finished, but I've also been tracking judge's scores, and tbh there isn't that much to praise.


GP Events

GP USA
GP Finland
GP Japan (incomplete for men's, pairs and dance, but ladies is done. Edit: men's and pairs done too)

(someone is helping me with Skate Canada, but it's not done yet)

Random Senior Bs
Autumn Classic

Ondrej Nepala


On each spreadsheet you can see how much a judge over/undermarked each skater relative to the average of the other judges. You can also see their average score deviation for same-nation skaters ("MEANSAME"), the average score deviation for different nation skaters ("MEANDIFF"), and the difference between the two ("DELTA"). Underneath you can also see the same stats but for GOE and PCS raw scores.

If you look through, you'll see that there are lots of cases where the difference between how a judge marks their own nation's skaters and different nations' skaters is as much as 10 points or higher, and there are very few cases where judges under-mark their own skaters (which should happen around half of the time if judging were completely fair and score deviations completely random).
 
Top