Bring back the 6.0 mark

[Norlite]

Joe Inman has posted his thoughts and impressions on judging for the first time using CoP. He judged this past weekend at Nebelhorn

http://www.fsuniverse.net/forum/showthread.php?s=&threadid=6803

 

[Panther2000]

As far as I can see the judges will still be able to do this under the CoP by manipulating the five "Program Component" marks to make the final point totals work out the way they want them to. This is not necessarity a bad thing, however. This is, after all, a judged sport.

Mathman

You again, have hit the BULLEYES. It is a judged sport. This new juding system is not 100% & it will never be.

Michelle said it best. Unless you use a stopwatch. It is all in the judges hands. Just because, they are going by each element. Doesn't mean that the cheating will stop.

This is still not the answer. But, we will see how it works

 

[hockeyfan228]

As far as I can see the judges will still be able to do this under the CoP by manipulating the five "Program Component" marks to make the final point totals work out the way they want them to.

While this may still be true, under the 6.0/ordinals system, any judge could "justify" why s/he had given a particular score or placement, by emphasizing whatever s/he wished. Under the new system, there is specific wording for each point range. So if the judge gives a 7 for choreography, arguably the "poofiest" of the five program elements, s/he needs to justify that the "program composition demonstrates a unity of skater, music, and content to present a 'story' via the program -- may have a clear theme or motivation which skater portrays for more than 75% of the program", "good distribution of highlights," and "good placement of elements through entire program," each of which has specific criteria. Not that the skater(s) had two or three poses or attempts to graft on style, but that the skater(s) maintained the style throughout the program.

If the judge gives a skater with two falls a 7 in performance/execution, s/he will have to justify why the skater qualified for an 8, less the two -.5 deductions.

 

[Rgirl]

Norlite,
I feel as if you are speaking exactly what I feel, only in a much more concise and effective manner Great stuff. And thanks for the link to Joe Inman's FSU post. In fact, I would like to add what Inman said at the end of that post:
"But what I want you to all know, if interested, is yes, there needs to be tweeking in many areas; but when I left the podium I truly felt I had judged and given the MOST honest accessments of all parts for the first time in my judging life. THanks---joe inman."

As for understanding the COP, I think those who don't should give yourselves some time. After all, Rome wasn't built.... Just kidding. Anyway, you can download the whole COP kit and kaboodle from the ISU site and/or get started with Paula's condensed versions. The ISU COP is no different generally and conceptually from the one used in gymnastics, though of course it is different in its specifics. IMO, a COP that breaks down the elements is going to be the best answer for judging in terms of statistical accuracy as is humanly possible in a judged sport with a large dose of subjectivity. The rest of it has to come in terms of really cracking down on cheating or deal-making judges. The latter is a completely human endeavor, meaning that if the people at the top want it, it will happen. If they don't, it won't. Of course the COP will need revisions, that goes without saying, but I really urge everyone to read all of Joe Inman's post on using the COP at Nebelhorn. It's from the horse's mouth.

As far as I'm concerned, 6.0 R.I.P.
Rgirl

 

[shine]

Add me to the list of those who have changed from an opposer of CoP in fear that it might destroy the "art" aspect of the sport, into one of the biggest supporters of the new system. The technicality of the system, though difficult for both the fans and the judges to learn, will force the judges to base every single point of their marks on each individual element that the skaters do. "Holding up" of certain skaters will no longer be possible and I really can't imagine fans or skaters screaming "wuz robbed" under this system of marking, since a specific mark is assigned to each element which then adds up to a total score for everybody to see. "Impression" will no longer hold skaters back either. There are many skaters in the lower ranks (and vice versa) who I feel should be ranked much higher, yet never have been because of the certain impression they've made which keeps them stuck in the back of the judges' mind as "second tier". Really, God knows how much the judges even bother following the 8 criteria of presentation when they mark (even thought they are clearly written in the rule book). I really, really hope CoP will change everything. I expect there to be major movement in all desciplines- nothing could be more exciting than this!
And even though CoP is not yet to be used in the major championships, I expect it to change forever the way judges think and mark, which, needless to say, will affect them greatly whenever they mark.

 

[Joesitz (RIP)]

It can't be a secret for you Joe, since you just recommended basically the exact same thing in your post above.
So there you go NorthernLite (nice name BTW) Joe approves.

Not quite, says Ole Joe. Dropping the highs and lows assists in illiminating some of the 'cheats', but we still don't know who marked whom because the scores are given in ascending order and differ with each skater. Tough to put them to individual judges. You did notice that, didn't you, Norlite?

Northernlite - Dropping 5 judges scores leaves 9 scores to be used; and in other contest dropping 3 judges scores leaves 7 scores to be used. Imo, it's the best way to go with the 6.0 system bearing in mind we still want to see which judge gave whom that silly score.

Joe

 

[Norlite]

I've been arguing about the big and small aspects of this radical change elsewhere. For now, I just have one question of everyone - how do you feel about the marks of five judges out of a panel of 14 being the ones which count?

I rather like the idea. I like it a whole lot better than the system they used last year. I particularly like the idea of *after* the random selection of 5 being eliminated, the two highest and two lowest are still thrown out. I think it adds in the process of a truly unbiased (and I mean truly unbiased, as in cultural differences as well as any other dealings) set of scores.

I know many have picked on this point over and over, even asking Mr. Inman how he felt about it. And I really can't see how it would matter to a judge one way or another.

Could you give me some insight as to why this is bothering so many people.

Rgirl- Thanks. I also appreciate your POV, especially your posts on athletic injuries. I take it this is your field?

 

[hockeyfan228]

Really, God knows how much the judges even bother following the 8 criteria of presentation when they mark (even thought they are clearly written in the rule book).

Sure they did, they all did. It's just that for one judge, there was Variation of speed (usually interpreted as "speed", Carriage and style , Expression of the character of the music, Use of ice surface , Unison (pair skating) , Harmonious composition, Ease of movement/sureness , and Originality.

For another judge there was Variation of speed (usually interpreted as "speed", Carriage and style, Expression of the character of the music, Use of ice surface, Unison (pair skating), Harmonious composition, Ease of movement/sureness, and Originality, etc.

And they didn't even have to apply their relative biases equally across all skaters in any competition.

 

[Norlite]

Oh....duh....


Took me awhile hockeyfan but I finally got it!

 

[Joesitz (RIP)]

Hockeyfan - Thanks for clearing up Shine's question. I was kinda worried about that too. It does seem from your example that some judges look at the criteria differently - more emphasis on one criterium than another. Hmm. Get those seminars moving!!:\

With regard to Originality... How is that defined? Haven't we seen all these skating moves in Dance, Gymnastics and, of course, Vaudeville. I guess it's everything old is new again.


Joe

 

[Mathman]

Could you give me some insight as to why this is bothering so many people.

Are you asking about eliminating five judges at random at the outset, or about throwing out the two high and the two low marks of the nine that remain?

If the former, the reason that it bothers so many people is that the five who were randomly eliminated might have swung the contest the other way. Suppose the 9 "real" judges split 5-4 in favor of skater A, while the 5 "dummy" judges (the ones who were eliminated at the outset) all favored skater B. Then there is a sense in some people's mind that skater B was the real winner, 9 to 5, and was deprived of her rightful reward by an unlucky role of the dice. It is widely believed that this happened to Sasha Cohen against Viktoria Volchkova in the Cup of Russia last year, for instance.

This objection is not quite mathematically sound, but has strong emotional appeal, especially to fans of skater B.

The business of dropping the highest and lowest, I don't think aroused much opposition.

I am still composing my thoughts about the mathematical aspects of these procedures.

Mathman

 

[Norlite]

The business of dropping the highest and lowest, I don't think aroused much opposition.

Yes, it does seem to be. Actually, all of it, including dropping the high and low marks.

Most of what I can gather from opposition, is not *just* dropping highs and lows, or *just* randomly selecting judges, but the simple fact that only 5 of 14 judges are actually being used. Like it is a waste, or insultive to the judges, plus narrows the level of expertise on the panel. That is the impression the opposition is giving. Like I said, even asking Mr. Inman how he feels about the possibility of his marks not even counting. Maybe they are just bringing up other points in order to not have to say " I wuz robbed" I don't know? I would like to here all opinions from those who oppose it, to get a clearer understanding of their reasoning.

I think one of the opposition groups issued a release this week which includes this issue. I'll try to find it and post it.

 

[Norlite]

If the former, the reason that it bothers so many people is that the five who were randomly eliminated might have swung the contest the other way. Suppose the 9 "real" judges split 5-4 in favor of skater A, while the 5 "dummy" judges (the ones who were eliminated at the outset) all favored skater B. Then there is a sense in some people's mind that skater B was the real winner, 9 to 5, and was deprived of her rightful reward by an unlucky role of the dice. It is widely believed that this happened to Sasha Cohen against Viktoria Volchkova in the Cup of Russia last year, for instance.

(Hmm....please bear with me here, I know this has been done to death, and I'm more or less thinking out loud anyway)

I have a hard time believing that is the reasoning for the "organized" opposition groups. I can, however, believe it to be believed by most fans.

I think that those who are involved enough to the point of being organized realize that last year's throwing out of marks were "shadow judging" meaning, those marks were never really there in the first place (yes, yes, I know, we saw them on screen). Having the "real judges" selected before the comp even began.

I think they realize that those shadow marks where there two years ago, and three years ago, and four etc. etc. The only difference being location, and publication. The judge sat at the panel instead of directly behind it, and for the first time the public got to see them.

That's why I think they have other concerns.

Maybe I'm wrong and they don't...maybe it's still about "I wuz robbed"

 

[Ptichka]

As I understand it, this is more complex than just throwing out top and bottom. They throw away top and bottom FOR EVERY ELEMENT. This is significantly different than throwing away top and bottom OVERALL scores.

Example: suppose we have a judge who just loves Morozov style footwork, and gives unreasonably high marks for all skaters who have it. Suppose the rest of his judging is OK. Under CoP, only his marks for the footwork would be thrown out; the rest would still be counted.

 

[Mathman]

To Norlite, Part 1.

I haven't really read much from the people who oppose this system. But I think that the argument against discarding so much data (dropping 4 judges from the sitting panel of 14, then dropping 4 more for high and low marks) goes something like this.

There is, in the grand cosmic scheme of things, a Right Score for each performance. This is the score that God would give, were She a figure skating judge instead of just an enthusiastic fan. (I wonder if God would be ticked off if the computer randomly threw out Her marks.)

Anyway, since we don't know what this Right Score is, it can be best approximated in principle by, let us say, taking the average of all the marks that might ever be given for that performance by all the properly qualified figure skating judges that are, ever were, or ever could be. In the language of sampling theory, this is the "population." The Right Score is the mean score of this population.

To attempt to get the best possible estimate of the Right Score, we take a "sample" from this population -- the panel of judges. The average mark of the panel of judges is the best unbiased estimate that is available to us of the Right Score (i.e., the population mean).

The "standard error" for this elementary problem in statistical estimation is S.E. = Sigma/(square root of n), where sigma is the standard deviation of the population (we can estimate this by the standard deviation of the sample), and where n is the sample size. That is, as you include more and more judges, the sampling error goes down by a factor inversely proportional to the square root of the number of judges. If you quadruple the size of the judging panel, you cut the error in half.

Conversely, if you cut a panel of 14 down to a panel of 5 (pretending for a moment that this is done randomly, without the "high and low" thing), this increases the sampling error by 67% (sqrt 14 / sqrt 5 = 1.67).

So, bottom line: You should have as large a judging panel as possible and use all the data available in making your estimate of the Right Score.

MM

 

[Mathman]

Part 2.

What about dropping the two high and low scores?

In any set of numerical data the two most elementary and most useful measures of central tendency are the mean and the median. The median is just the middle number. In a panel of nine judges, if you threw out the four highest and the four lowest, the score of the remaining judge would be the median. This does not give that judge any particular power or special privilege, its just the he or she happened to end up in the middle of the ranked scores.

The (arithmetic) mean is what most people think of when they hear the word "average." Add up all the scores and divide by the number of judges.

Both the mean and the median have their uses. And they both have drawbacks, in the sense that it is possible to contrive sets of data where either the mean or the median or both gives a false, or at least incomplete, summary of the central tendencies of the data. For instance, the mean is very sensitive to extreme values. Applied to figure skating, this implies that the mean (a very useful statistic in other ways) would be disproportionately skewed by a single judge giving a mark that is way out of line with his colleagues.

The median, on the other hand, suffers from a loss of information. Plus, being what is called a "non-parametric" statistic, mathematicians don't like it very much because it does not fit into any nice formulas and if you want to treat it mathematically, you usually have to make some (possibly invalid) assumptions about the underlying population.

The "cropped" or "trimmed" mean is a compromise between these two statistics. It is like the median in that excessively high or low values are "trimmed," and it is like the mean because -- well, because it is the mean of the trimmed data set.

Personally, I do not see any mathematical basis for objecting to the use of the trimmed mean in this setting.

MM

 

[Mathman]

Part 3.

About chosing 9 "real judges" and demoting the other 5 to the status of "shadow judges."

As you say, Norlite, there have always been shadow judges and practice judges. This obviously has nothing to do with the outcome in any way. Mathematically, it doesn't matter if the final (real) judging panel is chosen months before by pulling names out of a hat, or just before the event by a computer. So I can't really say that there is anything mathematically invalid in this procedure.

However, since it is basically a foolish waste of time, the burden is not on the objectors to prove that there is something wrong with the procedure. Rather, the onus is on the ISU to give some justification for doing it. The justification is that it protects the privacy of the judges and makes it virtually impossible to figure out which judge gave which scores. If you think that privacy in this regard is a good thing, then choosing 9 out of the original 14 is a good thing.

On the other hand, if you think that all this secrecy is a bunch of baloney, then you will conclude that it is pretty silly having 5 dummies sitting up there along with the real panel, making fools of themselves even as the ISU tries to fool us.

The ISU's position is not unreasonable, however. The Salt Lake City pairs scandal was blamed not on a judge being dishonest, but on the fact that she was under pressure from her federation. So the ISU's basic position is that the judges are basically honest people, but they need to be protected from outside pressure (from national federations pushing their own agendas, from parents of skaters who might bribe them, from Mafiosos who might threaten them, from the audience who might boo them, from public opinion which might revile them, etc.)

It is the same principle as the secret ballot. When we vote in an election, we are protected from public scrutiny so that politicians can't hire goons to beat us up if we vote the wrong way.

(Still, phooey on secret judging.)

Mathman

 

[Norlite]

Wow Mathman, thanks.

I'll have to take time and digest all this. I work in organization and details. Numbers / Statistics / Mathmatics ( whatever this is) is not my bag.

Thanks again!

 

[mpal2]

I still have never received a good answer on why anonymity will protect a judge from pressure. I've been told that it helps the honest judges from feeling pressured into voting against what they feel is correct. But I'm not sure how valid this is.

If the federation is the one applying the pressure to judge a certain way, won't the judges in that federation be groomed to go along with the group think anyway? Wouldn't the judges from the "bad" federation still be tainted somehow if the federation was really that determined to cheat?

Even though there is a chance that their marks won't be counted, there still is a chance that they will. It does make it harder to determine if pressure has been applied to a judge or if results from "deals" between federations has been successfully counted in the scores. But wouldn't you take that chance that the deal might help if you were a "bad" judge/federation anyway?

Just from personal life experience, most people are honest. Some people are honest as long as they are afraid they will be caught cheating. Some people just don't give a damn and do whatever they please. Give them an inch and they take a mile. How does anonymity protect skaters from the last 2 groups?

 

[hockeyfan228]

Wow Mathman, thanks.

ITA.