How can CoP be improved?

It has now been used in a major event for the first time, how can it be improved? I'm not really a numbers person, so I will lay out a few items for thought.

1. Should falling be more heavily penalized in CoP?

2. Judges look at both the difficulty level of a move (level 1,2,3) as well as how well it is done (GOE). Should difficulty or quality be emphasized more or should both be equal? Do you feel the current system does a good job of balancing difficulty and quality?

3. Are the criteria for the PCS scores (skating skills, choreography, transitions, performance/execution, interpretation) clear enough?

4. Are there certain moves which you feel should be worth more or less? (e.g. Joubert talking about how the quad isn't given enough weight in CoP.)

5. Are there ways CoP could be simplified to make things easier for everyone while still being a workable system?

I think there's one big thing:

make it serve its purpose by actually being OBJECTIVE.

(that's not technically possible since you have judges, but at least the judges can start by judging on what happened THAT NIGHT instead of letting politics or reputation influence them. Wishful thinking, yes. But it really has to be that way, or people won't watch it anymore. And for me, it certainly gets old. What's the use watching if the right person doesn't win?)

I also think that come Olympics there should be a BIG explanation of the Cop system. I'm almost sure the average viewer won't even know the 6.0 system is not being used anymore.

For me the one big thing is teaching the judges how to implement the system properly before messing around with changing weights, scores etc.

At the moment skaters have no idea if they are coming or going with regards the CoP. Case in point one of Arakawa's step sequences' was marked as a level three at one competition and exactly the same step sequence marked as a level one at another, now how on earth the skaters are supposed to compete under a system like that is beyond me. It is my feeling that the judges really don't know what they are doing with this system and as a result there are massive inconsistancies across the board and we are still seeing skaters who are skating in the final group benefitting over earlier group skaters and place holding in certain areas of the competition similar to when the 6.0 system was used.

IMO you can have the greatest and theoretically most fair marking system but if those using is don't what they are doing then it is pointless.

As Hockeyfan has suggested: Have a separate set of judges do the PCS scores. This will prevent a rush to judgement and assist in better scoring for the lower tier skaters.

Joe

Joesitz said:

As Hockeyfan has suggested: Have a separate set of judges do the PCS scores. This will prevent a rush to judgement and assist in better scoring for the lower tier skaters.

Joe

Click to expand...

ITA Joe and Hockeyfan. To me this is the one big change I hope they consider for next year, along with a lot of training for the judges on the PCS side.

If they are going to tinker with base values for elements, they better do it soon in fairness to the athletes who will shortly be starting to work on next years programs. I hope they don't do too much tinkering just in the interest of minimizing confusion. If there is one change I'd like to see here, it would be rewarding combo jumps a little more than the sum of the parts.

DG

Anonomyous

Remove anonomyous judging and require the judges to defend the marks they gave.
Linny

Linny said:

Remove anonomyous judging and require the judges to defend the marks they gave.
Linny

Click to expand...

Which would also put away the reason for the random choice of judges, so they would e.g. only need 9 judges for TES instead of 12 - the other 3 could do the PCS

1. Remove anonimity.

2. If they insist on throwing out random scores, put in a rule that if a judge from a skater's country is on the panel, then that judge's scores automatically get thrown out. (The "other side" will always consider them unfair)

3. I like the idea of having separate jusges do PCS. If not, break the components down even further. The more it's broken down, the more objective it will be. This (the streamlining of PCS) is what I consider CoP's main problem.

Ptichka said:

2. If they insist on throwing out random scores, put in a rule that if a judge from a skater's country is on the panel, then that judge's scores automatically get thrown out. (The "other side" will always consider them unfair)

Click to expand...

Except they're not randomly choosing which scores to throw out skater by skater, but rather for the whole competition segment.

I once had a discussion with someone about doing this with the ordinal system -- in that situation it would make no sense at all, because one judge might be marking high throughout and another low, so the ordinals only make sense if one judge's marks are used for the whole event. (E.g., 5.7/5.7 might be first-place marks from one judge and fifth place from another.)

With adding up total scores, it does make a bit more sense. Although all it would achieve would be that, if judges consistently score their own skaters higher than the rest of the panel, their high scores will be thrown out (or kept, but still be high, if another judge happens to be using a higher range throughout).

If they were really trying to cheat against "the other side," they would also be consistently undermarking their skaters' perceived closest rivals, and your suggestion wouldn't protect against those artificially low scores. Nor is there any way to figure out in advance which skaters a cheating judge would consider a threat who needs to be undermarked, because it partly depends on pre-event buzz and partly on how the skaters actually perform during the competition.

So throwing out the home country judge scores would really be more a cosmetic solution than an actual one.

3. I like the idea of having separate jusges do PCS. If not, break the components down even further. The more it's broken down, the more objective it will be. This (the streamlining of PCS) is what I consider CoP's main problem.

Click to expand...

My initial reaction when we were first introduced to the system last season was that one possibility to protect against individual judges deliberately or accidentally controlling the results by using larger ranges between their scores (for the medal contenders or whichever skaters we're worried about) would be to add up the GOEs and component scores by column, attach the same base mark as determined by the callers to each column, and convert each judge's total scores into ordinals and crunch those numbers by OBO or majority. I.e., the most complicated aspects of both system, but gaining the benefits of both. *Really* tedious to do without computers, though.

My other thought was, if they're just going to add up the scores anyway, and identifying levels of elements plus assigning grades of execution plus giving component scores is too much for any single individual to manage in real time, which is why they needed to separate the caller and judge functions, perhaps they need to break it up differently. Because grading the execution of all the elements *and* rating the components does appear to be too much to do at the same time.

So maybe, for singles, you would have:

-one set of officials identify and grade the jumps and spins (these are relatively easy for anyone who's motivated to learn to identify even if they never skated themselves, so it would be a good first assignment for judges from smaller, newer skating countries who have trouble fielding experienced judges)
-one set to identify the difficulty of the step (and spiral) sequences and also to rate the skating skills and transitions (this definitely takes more skating knowledge and experience)
-one set to rate the performance, choreography, and interpretation (these officials should have a significant degree of training in the arts, probably from other sources in addition to ISU seminars)

Obviously for pairs and dance the breakdowns would need to be different.

If you combine the identification (calling) and grading functions in the same individuals, then maybe instead of some aspects being rewarded or penalized in the level of the element and others in the GOE, there would just be a certain number of bonus points and deductions available for each element that the judges could award for both added difficulty, outright errors, and positive or negative quality.

But now that the existing system is in place, I doubt they'll actually reconsider the basics that drastically. Although putting in more checks and balances to the calling function and letting a separate panel assign component scores would be less drastic, so maybe we can hope for it.

gkelly said:

So maybe, for singles, you would have:

-one set of officials identify and grade the jumps and spins (these are relatively easy for anyone who's motivated to learn to identify even if they never skated themselves, so it would be a good first assignment for judges from smaller, newer skating countries who have trouble fielding experienced judges)
-one set to identify the difficulty of the step (and spiral) sequences and also to rate the skating skills and transitions (this definitely takes more skating knowledge and experience)
-one set to rate the performance, choreography, and interpretation (these officials should have a significant degree of training in the arts, probably from other sources in addition to ISU seminars)

.

Click to expand...

Yes. I like this idea.

Anonymous Judging

I agree to some extent with those who think the judges' identity should be made known, but I think there is some validity to the theory behind anonymous judging. It was meant to free judges from the pressures exerted by the federations to hold skaters up or down. I can see how anonymous judging would help judges give the marks they truly feel are appropriate without fear of retribution.

I think the skaters who benefit the most are those from small federations with a medal contender in only one discipline, because it will prevent inter-disciplinary deal making like the dance/pairs deal from 02. In the past skaters from under-represented countries were ordinal pawns that were placed up or down to acheive the desired result for the power country skaters.

brad640 said:

I agree to some extent with those who think the judges' identity should be made known, but I think there is some validity to the theory behind anonymous judging. It was meant to free judges from the pressures exerted by the federations to hold skaters up or down. I can see how anonymous judging would help judges give the marks they truly feel are appropriate without fear of retribution.

.

Click to expand...

With CoP there are suppose more detail and specific rules, as long as they strictly judge according to the rules why should they afraid of?

And how about the personal bias and imcompeted judges how are they going to be monitored?

Don't know any judged sports with anonymouse judging. Not in gymnastics, not in diving. These two both have very similar judging methods.

mzheng said:

And how about the personal bias and imcompeted judges how are they going to be monitored?

Click to expand...

According to the ISU documentation,

http://www.isu.org/vsite/vfile/page/fileurl/0,11040,4844-152090-169306-63933-0-file,00.pdf

after each competition,

For Technical scores:
Each judge's scores are compared, element by element, to the trimmed mean (7 or fewer judges) or double trimmed mean (more than 7 judges) awarded to the skater. The deviations are added up. The judge's performance is considered acceptable if

the total deviations/# elements <= 1.

Example:

Total deviation in SP (8 elements) is 7.92 points. 7.92/8 <=1, and is in line

Total deviation in LP (14 elements) is 15.2 points. 15.2/14 > 1 and is considered out of line -- potentially.

For components
Total deviation is compared to 15% of the total maximum points or the difference between the average and the referee's total, whichever is greater.

Example: Total points for LP is 100. Trimmed mean is 50. First acceptable range calculation is +/- 7.5 points.

Example A: referee's score is 60. Acceptable range is now +/- 10 points.
Example b: referee's score is 55. Acceptable range is +/- 7.5 points.

There are similar algorithms for Compulsory Dances.

The results are reviewed by the Official Assessment Committe, comprised of two members from each discipline. The OAC members get the reports for all judges, but without identification.

For levels and referee decisions
The members are responsible for identifying bias and/or "serious" errors made by the Technical Specialists and Controller and referee.

The OAC is responsible for sending reports to the Technical Committee and ISU Council. If the ISU Council disagrees with the conclusions, they are supposed to resolve this with the OAC members, but the Council has the final word, as it seems to have in everything.

If the ISU Council decides that sanctions are in order, the ISU Secretariat is responsible for telling the official.

---------------

In my opinion, the main issue with CoP is that the judges are evaluated by how close they conform to the other judges' scores, not to the written code. If judges were evaluated by how they conform to the written code, then either the existing judges would have to award the PCS correctly or prove themselves incompetent. The current evaluation system rewards conformity with the herd, not with the rules.