Fair/Over/Underscoring - Tricky Subject, Tread Softly

seniorita · Nov 1, 2009

prettykeys said:
Now I'm going to elaborate on this in regard to my favourite skater whose fans I ticked off on her fan forum: YuNa

Being a fan means you support someone whether they are on a roll or in a slump (check.) But being fair is also important.

wow, whole post :clap:

and mathman finally found someone to play with! :laugh:

Tonichelle · Nov 1, 2009

i love to skate said:
I have had the opportunity to meet many judges -some even at International and ISU levels - and they have not had high incomes. Many that I have met work in social services or education and those are not high paying fields. I'm sure there are some judges that come from wealthy families and I know there are some "bad apples" but the majority are good, honest people who love skating.

sounds like a lot of the officials I've met over the years at nationals... just in love with their sport (I've sat with judges who are learning the new system, learned a lot by listening in on their discussions... especially the Mens LP in 2006)

gkelly · Nov 1, 2009

Most of the judges I've met in the US fall into one of the following categories:

Former child skaters, many of whom had tested up to junior or senior level

Adult skaters

Parents of competitive kid skaters

Offhand I think I know one judge who started out as just a fan

Of those in the latter two categories, all the ones I know took some skating lessons themselves after starting out on the trial judging path, so essentially they became adult skaters themselves, even if they didn't test or compete.

Regarding the question of whether judges are scoring fairly or over- or underscoring, I think that's not a very useful question to ask.

Often we disagree with the judges' average scores and the results. But how can we assess whether high scores were too high or low scores were too low or whether they were fairly arrived at?

Whenever we disagree with the panel, there are three possibilities:

The panel's opinion was right and ours is wrong
The panel's opinion was wrong and ours is right
Both opinions are fair, honest, and arrived at based on the details of the skating and the rules, but because of the subjective aspects of many of the judgments different individuals or the consensus of different groups of individuals can arrive at different results, both valid

It's probably more useful to ask "Do I agree with these scores or these results?" If the answer is No, then we can explain why we would have scored it differently.

There are differences of opinion on the judging panels. There are differences of opinion among fans.

Disagreeing does not necessarily mean that the other side is wrong. I think it's a great opportunity for discussion about the complexity of what is being evaluated (the skating) and the tools for evaluating it (the rules).

prettykeys · Nov 1, 2009

gkelly said:
It's probably more useful to ask "Do I agree with these scores or these results?" If the answer is No, then we can explain why we would have scored it differently.

There are differences of opinion on the judging panels. There are differences of opinion among fans.

Disagreeing does not necessarily mean that the other side is wrong. I think it's a great opportunity for discussion about the complexity of what is being evaluated (the skating) and the tools for evaluating it (the rules).

janetfan · Nov 2, 2009

gkelly said:
Most of the judges I've met in the US fall into one of the following categories:

Thanks for your insights gkelly.

Perhaps there is another category for judges:

Judges who will watch the competition and do their best with NO preconceived notions about the skaters and their reputations.

Judges who will always give skaters like Mao, Plushenko, Miki, etc., the highest pcs no matter how they skate their program.

Is that a false assumption?

I just read in three different reviews the following statement about COC Ladies Gold Medal winner Akiko:

"once the judges know her better they will start to give her higher pcs."

My, my, isn't that nice of the judges.

ETA: CoP was created and implemented after the SLC Pairs scandal. It's primary objective was to create a more fair system and to reduce the chances for cheating,
But isn't favoritism about the same thing as cheating?

Here is an interesting article about the random elimination of three judges' scores (which has become even more relevant with the recent reduction in the size of the judging panel).

http://www.stat.yale.edu/~jay/EC2006/

gkelly · Nov 2, 2009

janetfan said:
Perhaps there is another category for judges:

Judges who will watch the competition and do their best with NO preconceived notions about the skaters and their reputations.

Judges who will always give skaters like Mao, Plushenko, Miki, etc., the highest pcs no matter how they skate their program.

Is that a false assumption?

I think it's still asking the wrong questions.

Let's assume for the sake of argument that all judges will watch the competition and do their best with no preconceived notions about the skaters and their reputations.

Most of the program component criteria have nothing to do with the success or failure of the elements. Performance/Execution will probably be most affected by mistakes, or in the other direction by a skater being especially "on" that day.

Skaters who have scored well in the past will often deservedly earn high PCS again even when they don't skate their best. Their marks may be a bit lower than they receive at their best, but they would still be deservedly higher than marks for skaters with lower overall skill levels.

That is especially true for Skating Skills, which is hard to evaluate as accurately as the judges when watching from distant seats in the arena and especially when watching on video at home.

So to a large degree the high scores are in fact deserved based on the actual skating that day and not on preconceived notions. The best skaters consistently get the best scores.

Now, there can be other factors that can exaggerate that effect unduly.

Among other things, Skating Skills addresses the effortless and security with which the blades move across the ice. A skater who does this extremely well will make a stronger overall impression that will legitimately affect the other components as well.

However, Skating Skills is the first component listed. Some judges may have developed a tendency to assign the SS score first and to think of the other components in terms of going up or down from that number. That's not the ideal way for PCS to be used -- they're supposed to be independent. So judges who fall into that habit, perhaps without consciously realizing it, may not be doing the best job of assigning PCS they're theoretically capable of, but they are still honestly trying to judge what they see.

Judges are human. Human perception is influenced by expectation. So to some extent, like the rest of us, judges often see what they expect to see. When watching a skater who has often skated very well in the past and skates slightly less well today, they may perceive a better overall impression than from a skater from whom they expect less.

This effect would account for the phenomenon that "once the judges know her better they will start to give her higher pcs." That does often tend to happen with newcomers to the elite scene. But much of it is an unconscious effect of expectations.

For all of the above reasons, honest judges who fall into your first category will still tend to give high scores to skaters who already have a reputation of being among the best. Most of it will be deserved. Some of it will be unconscious psychological effects that result in honest judges not doing the best job of assigning scores that they possibly could.

Then you might also get some other effects that are less honest and would push the judges into your second category:

Judges who assign scores more with an eye toward staying inside the "corridor" of marks on the panel. That means predicting how the rest of the panel would mark, using past scores for that skater as a clue in addition to what they see on the ice today. They may also be afraid to go out on a limb with drastically different marks for one component that stands out as extremely better or worse than the rest of the skater's performance.

That's more an effect of timidity, incompetence, or fear of being out of line.

Judges who out of feelings of national loyalty consciously overmark the skaters from their own country.

Judges who are pressured by their federations to hold up their own country's skaters and undermark their expected rivals, and perhaps to comply with deals to hold up skaters from other federations in exchange for similar consideration from that country's judges.

Those two are dishonest motivations.

From the outside, though, just looking at the numbers, it's pretty hard to figure out which effects are in fact leading to the numbers we disagree with. It might be mostly the honest reasons or mostly the incompetent or dishonest reasons or a mix of all of them. There's no need to jump to the conclusion that every time a score seems wrong to us there must be a nefarious reason.

Another thing to consider is that judges on the panel disagree with each other in various ways. One may be a generous marker who gives high scores to everyone, and another may tend to mark low in general or to be a real stickler for a particular pet peeve that other judges and fans don't care so much about. Some may make an active effort to spread their PCS for each skater, or for the field as a whole, as widely as possible, and others may tend to stick to a much narrower range. Some judges may emphasize some particular criteria for each component, and other judges or fans may emphasize different criteria. Some may reflect certain areas of overlap between the PCS in one of the relevant components that another judge reflects in a different component.

Most of those differences will get flattened out in the averaging. So there might be one or more judges who scored similarly to the way you would have scored, but they were balanced out by other judges who perceive things differently.

The same thing would happen if we got a bunch of fans to score the programs and then averaged our scores.

No one individual has the one correct score, with everything else automatically being wrong.

Mathman · Nov 2, 2009

Janetfan said:
Here is an interesting article about the random elimination of three judges' scores (which has become even more relevant with the recent reduction in the size of the judging panel).

http://www.stat.yale.edu/~jay/EC2006/

(B)ut the system has introduced the unsettling possibility of dumb luck influencing the medal standings. In a close competition with skaters separated by only a few points, the outcome will likely be determined by the random choices of panels of nine judges.

This was a common, but, as it turned out, fallacious criticism that was popular when the interim judging system first came out in the 2002-2003 season. It is true that if you select nine judges at random from a panel of twelve, then you mught get different results dependng on which nine you chose.

However, you have already made a random selection of the twelve from among the total population of all judges. Statistically, it works out exactly the same whether you choose 9 judges at random from the total population of 1000, or whether you first chose 12 from the 1000, then 9 from the 12.

In other words, the "dumb luck" factor is already built into the system, no matter how you go about selecting your nine voting judges.

Some time later (2008?), the ISU refined their procedure for selecting the judging panel. The statistical waters are muddied a bit by the fact that of the total panel of 16, some are selected to score the short program only, some the long program only, and some both -- and then there is a random draw on top of that.

In my humble opinion, the reason why the ISU should discontinue the procedure of dropping two scores at random is not related to statistics, but to public relations. The procedure is stupid on its face, plus it gives the impression that the ISU is trying to trick us somehow.

gkelly · Nov 2, 2009

Mathman said:
In my humble opinion, the reason why the ISU should discontinue the procedure of dropping two scores at random is not related to statistics, but to public relations. The procedure is stupid on its face, plus it gives the impression that the ISU is trying to trick us somehow.

And for the statistical reason, using more judges is going to give a better result than fewer judges. So as long as they're paying for more judges to be there, why not use them all?

Mathman · Nov 2, 2009

gkelly said:
And for the statistical reason, using more judges is going to give a better result than fewer judges. So as long as they're paying for more judges to be there, why not use them all?

That, however, is a different point from the one that Prof. Emerson was addressing.

As for the number of judges, reducing the number of scores from 9 to 7 increases the margin of error by about 13%. Every little bit helps, but IMHO there are so many uncontroled variables in figure skating judging that this is not really a very important consideration.

I think a stronger argument against randomly throwing out the scores of two judges is just that it is inane, ridiculous, and utterly devoid of merit or sane purpose. In fact, I think the whole idea was just something tossed into the ring by the ISU after the 2002 judging controversy, to try to trick the IOC into thinking that Cinquanta was out there taking control and seizing the day.

gkelly said:
Skaters who have scored well in the past will often deservedly earn high PCS again even when they don't skate their best. Their marks may be a bit lower than they receive at their best, but they would still be deservedly higher than marks for skaters with lower overall skill levels.

Mao Asada is a great example. Her blade and body control are so outstanding that she earns and deserves high marks in the skating skills component even when she falls on her jumps.

Plus, I think the opposite can happen, too. If the judges are expecting something amazing and it doesn't happen, they may unconsciously give lower scores even in the areas where the skater performed well.

gkelly said:
However, Skating Skills is the first component listed. Some judges may have developed a tendency to assign the SS score first and to think of the other components in terms of going up or down from that number.

If I remember correctly, the ISU has done soem experimenting with this. When the judges were asked to do P/E, Chor and Int first, the results overall were different.

Audrey19 · Nov 3, 2009

Mathman said:
This was a common, but, as it turned out, fallacious criticism that was popular when the interim judging system first came out in the 2002-2003 season. It is true that if you select nine judges at random from a panel of twelve, then you mught get different results dependng on which nine you chose.

However, you have already made a random selection of the twelve from among the total population of all judges. Statistically, it works out exactly the same whether you choose 9 judges at random from the total population of 1000, or whether you first chose 12 from the 1000, then 9 from the 12.

In other words, the "dumb luck" factor is already built into the system, no matter how you go about selecting your nine voting judges.

But the problem is actually that by drawing the judges randomly, not every skater in the same competition is scored by the same judges. Some might be lucky and have the judges that had scored them low not be drawn, others won't. I really hope they will get rid of that random draw...

gkelly · Nov 3, 2009

Audrey19 said:
But the problem is actually that by drawing the judges randomly, not every skater in the same competition is scored by the same judges. Some might be lucky and have the judges that had scored them low not be drawn, others won't. I really hope they will get rid of that random draw...

No, that doesn't happen. Every skater is scored by the same judges. The judges are that are randomly selected not to count do not count for the whole competition, all the skaters.

For each individual GOE and PCS, the high and low marks from the judges who do count are dropped. That isn't random. And it will change for each element and each component.

Audrey19 · Nov 3, 2009

Oh really? I didn't know that, I always thought they were selected for each skater... Thanks for clearing that up...that makes it indeed sound less apalling!

Mathman · Nov 3, 2009

If I were a figure skating judge I would be really mad if they made me sit at the scoring table pantomiming the judging process, then said at the end, "Sorry, the computer just threw out all your scores -- ha ha, what a fool."

janetfan · Nov 3, 2009

Mathman said:
If I were a figure skating judge I would be really mad if they made me sit at the scoring table pantomiming the judging process, then said at the end, "Sorry, the computer just threw out all your scores -- ha ha, what a fool."

and would it really be so difficult for someone to rig that software and throw out the judges less randomly?

prettykeys · Nov 3, 2009

gkelly said:
No, that doesn't happen. Every skater is scored by the same judges. The judges are that are randomly selected not to count do not count for the whole competition, all the skaters.

For each individual GOE and PCS, the high and low marks from the judges who do count are dropped. That isn't random. And it will change for each element and each component.

Do you know if the same judge appears in the same column in a scoresheet, for all the skaters?

For example, is Jerk Judge #9 on http://www.isufs.org/results/wjc2006/WJC06_Ladies_SP_Scores.pdf who clearly has a thing against YuNa ( :frown2: ) the same Judge #9 for the other skaters?

MasterB · Nov 5, 2009

As this is a sport I would like to get rid of the PCS score and let the best jumper or the one who lands the most jumps win. Jumping is something a non-fan can see and understand immediately.

Now that we micro-manage everything even I have a hard time understanding some of these scores.

janetfan · Nov 5, 2009

MasterB said:
As this is a sport I would like to get rid of the PCS score and let the best jumper or the one who lands the most jumps win. Jumping is something a non-fan can see and understand immediately.

Now that we micro-manage everything even I have a hard time understanding some of these scores.

Don't feel bad, atleast you are honest

Many former skaters and coaches don't get the logic in some of the rules either.

It seems only a select few "enlightend ones" - who also happen to be posters at GS - have acquired the necessary knowledge to love and cherish in sickness and in health and for as long as they live the the scoring system known as CoP

prettykeys · Nov 5, 2009

MasterB said:
As this is a sport I would like to get rid of the PCS score and let the best jumper or the one who lands the most jumps win. Jumping is something a non-fan can see and understand immediately.

Now that we micro-manage everything even I have a hard time understanding some of these scores.

In the past I and others here toyed with the idea of giving out skating medals for separate events like SP, LP, and possibly a jumps-only competition (e.g. see which skater has the best quality for all standard jumps for females/males), and then having medals for the all-around "bests". Maybe a compulsory figures event can be brought back, too.

I definitely wouldn't want to get rid of artistically programmed skating, though.

MasterB · Nov 6, 2009

Perhaps PCS should only be used when two skaters in any competition land an equal amount of jumps.

The thing with choreography is that it is not easily measurable. A jazzy number could be highly entertaining with simple choreography while a much more intricate number can be lost on the audience due to some abstract piece of music(boring).

As an example: I think that Ashley Wagner's music is highly annoying and never seems to build to an exciting (Olympic)moment but, because I've been skating an eternity I know that the program is well choreograph and therefore have an appreciation than a layman who happens to see skating only during the Olympics might not have. Flatt on the other hand skates to Sing, Sing, Sing a much more peppy music that is choreograph well but not necessarily more difficult that Wagner's. I would say the same for Kim's Bond program.

A perfect way to judge PCS would be to have a choreographer create a routine and teach the same routine separately to three skaters and see which one delivers it better. Yes, that would be boring(kind of like watching compulsory dance).

Mathman · Nov 6, 2009

prettykeys said:
Do you know if the same judge appears in the same column in a scoresheet, for all the skaters?

Yes.

Fair/Over/Underscoring - Tricky Subject, Tread Softly

seniorita

Tonichelle

Idita-Rock-n-Roll

gkelly

prettykeys

janetfan

Match Penalty

gkelly

Mathman

gkelly

Mathman

Audrey19

gkelly

Audrey19

Mathman

janetfan

Match Penalty

prettykeys

MasterB

janetfan

Match Penalty

prettykeys

MasterB

Mathman

Similar threads

Connect with us