Suggested Revisions to 6.0 System Submitted by Austrailia

Suggested Revisions to the 6.0 System Submitted by Australia

Found this at MKF (MKsmiral's post). Sounds like a definite improvement over the existing systems to me... what are your thoughts?

http://www.iceskatingintnl.com/current/content/revised6.htm

Hey Mathman - Check this out. I'm busy to read it thoroughly so I just scanned it. I'm all for appointing judges from various regional groups rather than by country.

The rest of it seems to be based on total scores and reaching the mean. If I remember my elementary statistics course, the mean gave a better result than the average.

Joe

I think this is an excellent proposal.

Finding a statistical mean and throwing out extreme deviations would eliminate phony high and low scores such as we have seen under 6.0 Random Sekret Komputer Selektion.

And it seems only fair to have a judging panel representative of all regions, and not so heavily European.

Of course, such a proposal has the chance of a snowball in Hades of being passed.

ITA, eurterpe - This system would be hard to figure out how to cheat or worse make deals.

Joe

Thanks for the post IndieBoi, and welcome to Golden Skate.

Mr. Cinquanta's nemesis, the evil Dr. Rossano, strikes again, LOL:

"In the CoP approach, which does not use relative comparison of the skaters, the individual scores are statistically nearly meaningless. Rather than being a distraction, considering placements in a relative comparison is essential to obtaining the best possible scoring accuracy, and is unequivocally the fairest way of scoring an event using human judges."

Click to expand...

Where's Moyesii when we need him, LOL?

First, the trimmed mean business is kind of a red herring because that's what the CoP does anyway. The only difference is that under the CoP the trimming takes place after the random draw (which would be eliminated under this proposal. Actually, I expect Cinquanta to drop the random draw anyway.)

Another red herring is the part about selecting judges by geographical (or geopolitical) region. This is an idea that has considerable merit, and I think that it should be formally proposed to the ISU as a separate issue. But this could be done equally under the CoP, under any version of the old ordinal system (OBO, etc.), or under the scoring system proposed by the Australian federation. So this should be a separate debate, rather than mixed in with discussions about the CoP versus ordinal systems.

The most significant part of this proposal is that it retains the "comparison" feature of the old ordinal system. Rather than earn point totals based on an absolute scale, skaters are ranked by the judges: best, second best, etc. Dr. Rossano claims that retaining this feature (in a somewhat diluted form) is "10 times more statistically meaningful than just 'marking in the moment' (judging on an absolute scale) alone."

Well, I'm all for 10 times better judging. The judges themselves, however, do seem to like the CoP because they don't have to remember whether the performance of the 4th skater to perform was better than that of the 29th entrant. The Australian proposal speaks to this issue, although not with the same vigor as does the CoP.

BTW, there is something interesting about the politics of this proposal. It is offered as an alternative to the "existing ordinal based system," rather than as an alternative to the CoP. In other words, the proposers are adopting the point of view that the CoP is still in the proposal stage and the ordinal system is the official judging system of the ISU. This wording seems to be a deliberate counter to Cinquata's attitude of regarding the CoP as a fait accompli.

Mathman

Mathman...

...your knowledge of statistics truly has me in awe...and I could never hope to keep up with you on that front! Thank goodness you are here to lend your interpretation. I can only speak for myself, but I certainly could never figure all that out on my own.

One of the things you mentioned in your post really leaped out at me...the point about the CoP measuring elements v. generally ranking the skaters. For us armchair quarterbacks (speaking for myself here) we only get to see MAYBE the "top" six skaters in an event, and it's easy to forget that these judges have to rank many, many more skaters than we ever see.

So...I find myself still liking many of the facets of CoP. (I'm sure it can use some improvement) If I had to watch 30 skaters and rank them all 1 - 30 on very general terms, I can't imagine how I would do that and *know* I did it fairly. But by scoring each element, and then breaking down the performance section into separate areas, I would sure feel better that the result was what it should be.

Of course none of this addresses the cheating judge issue. I still think CoP at minimum makes it more complicated to cheat (more scoring areas to manipulate with limited time to figure it out) which is sad, but probably a good thing.

Foremost, these skaters, coaches and families work very hard at what they do. I think they deserve detailed feedback which addresses all technical elements specifically, and the presentation elements more specifically as well.

So... I personally would like to see the CoP evolve. Supposedly after the GP series a very detailed evaluation was to be made of the judging, the results, the system, etc. Other than general comments from the ISU saying "it worked" I haven't seen anything detailed - like "here's what we think can be improved"....etc. I somehow doubt we will ever see that kind of detail. But just in case...has anyone seen anything beyond the "it worked" general commentary from the ISU?

DG

To start off with, any time somebody uses a phrase such as "[is] unequivocally the fairest way of scoring an event using human judges" my screwball detectors go up. To use the words "unequivocally," "fairest," and "human" in a phrase to make the same point is to me like saying "Capitalism is unequivocally the fairest economic system for humans."

I know this next thing is nitpicky, but I couldn't help but chuckle at the term "human judges." I know Rossano meant judged events vs. timed events or those events where the result is determined (or "judged") by something like how many times you get a ball in a basket, but when he writes "human judges" I want to say, "As opposed to Vulcan judges? Monkey judges? Computer judges?" Sorry, just being human.

As for the proposed system, while I'm glad that at least the Australian federation is proposing an alternative to the pre-'01 ordinal based system as well as the COP, like Mathman I think some of it addresses political issues rather than scoring or statistical issues. Examples: Anonymity of judges and assigning judges from different geographical or geopolitical regions. As Mathman said, these changes can be implemented with any system and I agree they should be a separate discussion. However, anytime the problem of anonymous judging is brought up is another time Speedy sees that people don't like it, so on that account, I guess I don't care when or how people talk about it as long as they do.

What I find problematic about this system is that it attempts to do many of the same things as the COP, but IMO without the tools. For example, it breaks up the technical mark into three sub-marks: "In singles, jumps would account for one-half of the Technical Merit mark (0 to 3.0), spins would account for one-quarter (0 to 1.5), and sequences and connecting moves for one quarter (0 to 1.5)." Rossano goes into how this would work for pairs and dance as well.

"In this approach," says Rossano, "the contribution of each type of element to the skater's score is rigorously established. For singles, jumps would make up 25% of a skater's total score, compared to CoP where jumps make up about 42% of the total score. Spins under Australia's proposal would make up 12.5% of the total score compared to 8% under CoP. With the publication of these sub-marks all would know exactly the relative contribution of each type of element to the total score, and were each skater stands for each type of element."

Although the advantage to this system relative to the COP is that it's simpler, I think it's also the disadvantage. There are only three technical components to the the technical merit score--jumps, spins, and everything else. Since this is a short paper and I'm sure not intended to go into all the detail of the ISU's 25-page COP communique, I'm not sure about some things. For example with jumps, Rossano says they would be scored from 0 to 3.0, which, BTW, doesn't square with his assertion that the scoring will go to two decimal points, ie, 3.00. Anyway, does the 0.00 to 3.00 score mean each judge's impression of how the skater did all his/her jumps? Or is each jump scored the way it is in the COP except without a base value, eg, .25 for a clean 3Lutz, .20 for a clean 3flip, etc.?

What does the skater see on his numerical scores? For example, let's say that for jumps a skater scored 2.25, for spins 1.40, and for everything else 1.45, for a total score of 5.10. What specific information does that give the skater other than his jumps were scored at 75% of the maximum, that his spins were scored at about 93% of the maximum, and that everything else was scored at about 97%? Okay, so jumps in the technical score is what needs work. If the skater fell, that could account for it, but what if the skater didn't fall? Did the skater two-foot, take off or land on the wrong edge, pitch forward, touch down with a hand or the free leg, whatever? Rather than go into all the "maybes" I'll just say that one of the things I like about the COP is that everything a skater does is covered specifically. True, there are bugs in the system of having the caller "see" if a jump was significantly underrotated, flutzed or whatever, but I think this problem can be addressed. Anyway, with the COP, a skater can see that by adding footwork going into a clean jump, he gets anywhere from +1 to +3 points, or that because of two-foot landings, he has points deducted.

OTOH, A skater may find that by adding footwork before a jump, his speed is compromised, resulting in two-foot landings on that jump. So the skater and the coach can decide: Right now, is it worth keeping in the extra footwork given the risk of having a two-foot landing? I would hope an intelligent skater and coach would first of all say, "We've got to work on your/my jump technique and certain strength issues to do this jump clean with the added footwork." Secondarily, they would have to decide for the next competition what to do, but they would have the specific information to make the choice. I think it gets even fuzzier with spins, spirals, footwork, and other areas. We saw how Sasha, Shizuka, and Fumie all upgraded their spiral sequences by seeing they were only getting a Level 2 in the COP and finding out what they could do to upgrade it to a Level 3. The skaters get more points and we get better skating, providing the skaters do the changes well. I don't see provissions in Rossano's system for skaters and coaches being able to make such changes with either the technical or presentation score.

Also, what about the differences in number of jumps between the SP and the LP? There are at least twice as many triple jumps in the LP as the SP, so to assign a jump sub-score of 0.00-3.00 for both seems problematic.

Rossano says his proposal his superior because under the COP about 42% of the total score is based on jumps whereas in his system it's 25% of the total score. I'm all for doing things to make the "other stuff" in figure skating important, but IMO jumping is what separates meat from the potatoes. I didn't know that jumping accounted for about 42% of the total score in the COP but I was quite pleased to read it (assuming it's correct). I think about 40% for jumping is just fine. IMO, 25% is far too low. For as difficult as jumping is and for as few people in the world who can do it well and consistently, I want skaters who jump well to get credit for it. I don't want "jumping beans" but neither do I want 75% of a skater's score to be based on everything else. Figure skating is a lot more than jumping, but jumping (triples, quads, combos) requires the greatest amount of skill and the greatest risk in the shortest amount of time. That's why I think 25% is much too low and that somewhere around 40% is about right. OT: Under the COP, I wonder if the percentage of the total score for jumping is greater for men than ladies.

Rossano spends a lot of his time talking about relative vs. absolute scoring, that is placement (how one skater did relative to another) vs. judging "in the moment." But if the results of the competition are to be based on the double trimmed mean of the scores of 11 judges, how does relative placement get in there? (BTW, "average" and "mean" are the same.) In other words, under Rossano's system, as I understand it, each skater's technical and presention scores would be added (I don't know if they would be added together or separately in order to get the mean); the two high and two low scores would be thrown out (I guess he means for the technical and presentation scores each, which would be the two high and two low technical scores and the two high and two low presentation scores, which would actually be eight scores, though I'm not sure on that since it would depend on whether he combines the scores first and I don't think he said); the remaining scores would be divided by the number of remaining judges (11 - 4 = 9); and that would give us our mean (again, don't know if he's keeping the technical and presentation scores together or separate). The skater with the highest mean score would win the gold, second highest would win the silver, and so on.

So where does relative placement happen? Did I miss it or did Rossano not specify?

There are advantages to Rossano's proposal over the COP--easier to implement, can compare past results, fans are used to the 6.0 system, and other things he mentioned in his paper. But I think these are relatively minor advantages compared to the weaknesses of the system and the advantages of the COP.

OT: Doggygirl asked if we had heard anything about how the COP worked from the ISU. I don't think we'll hear anything about it until well after the entire season is over, the ISU has it's big COP meeting, and possibly not even until early next fall.

So, interesting points in Rossano's proposal but my vote still goes to the COP with changes.
Rgirl

Re: Mathman...

Doggygirl said:

Of course none of this addresses the cheating judge issue. I still think CoP at minimum makes it more complicated to cheat (more scoring areas to manipulate with limited time to figure it out) which is sad, but probably a good thing.

Foremost, these skaters, coaches and families work very hard at what they do. I think they deserve detailed feedback which addresses all technical elements specifically, and the presentation elements more specifically as well. -- DG

Click to expand...

I agree 100%. These are the most important issues. In fact, the reason that I don't like to be drawn too far into statistical debates about judging systems is because non-statistical issues like these are so much more critical than anything the statisticians have to tell us.

Rgirl -- Everything in your post is grist for my mill, LOL. But "in the moment" I will just remark on one thing:

Rossano says his proposal his superior because under the COP about 42% of the total score is based on jumps whereas in his system it's 25% of the total score.

Click to expand...

I think that these are not the right numbers to be looking at. It's not the per cent weight given to different kinds of elements that counts, it's the difference between the scores given to the skaters in these categories that matters.

Example: Suppose jumps count 20% and everything else counts 80%. Skater A gets a score of 2.5 for jumps and 2.4 for everything else, for a weighted mean of 2.42.

Skater B outskates skater A in the most imposrtant elements (the 80%) and gets a 2.5, but messes up on a couple of jumps and only gets a 2.0. Skater B has a weighted average of 2.40. Skater A wins.

These numbers are not just hypothetical. According to an analysis of the Skate America and Skate Canada CoP statisitics, in the men's competition jumps are almost everything, with quads being the most important indicator of placement. The reason is that even though points for spins, footwork, etc., are given due prominence by the CoP, in practice the judges are not really using them to distinguish between skaters, giving out very similar marks to everybody.

(One statistical analysis that I read (by CoP critic Dr. Dirk Schaeffer) concludes his analysis with the statement that the only reason that the CoP has not produced a string of disastrous embarrassments is that the judges are ignoring it as best they can, LOL.)

"Supposedly after the GP series a very detailed evaluation was to be made of the judging, the results, the system, etc. Other than general comments from the ISU saying "it worked" I haven't seen anything detailed..." -- DG

Click to expand...

I think we can expect more than just "it worked." I fully expect to see ISU issues to the effect: "The CoP worked exquisitely.It's perfection was beyond our expectations, yes, beyond the furthest reach of the human imagination, making all critics of Mr. Cinquata look like the fools they are!"

Mathman, monkey judge emeritus.

IMO, everyone likes the CoP in principle, but are the judges really following it?

It would seem to me that a judge, particularly, an inept one, will just mark things up because the skater had certain tricks and another skater had a wonderful (subjective) presentation, and that one skater had the best of both.

I'm hoping for a good review of the system by the brass now that the test has been completed.

Joe

The proposal does not impress me. I think CoP has had a good beginning; it certainly needs fine tuning, but nothing in the article convinces me that that "other" system is any better. One thing I like about CoP is its flexibility. For example, if ISU decides that LESS attention needs to be paid for the jumps, it can simply lower the base mark for jumps, and increase base mark for in-betweens.

Mathman...

...you crack me up! The ISU should hire you to write the review. I'm sure you could do the analysis too, but I'm sure that's not necessary.

DG