Does skate order matter in CoP scoring?

[Mathman]

The seeding system for the short program for Skate America, in which the competitors skate in the inverse order of their ISU rankings, raises again the question of "scoring drift." In the new judging system, do the scores (especially the program component scores) tend steadily to rise as the event goes on? Do the skaters in the last group(s) to skate have a built-in advantage over skaters who perfromed earlier?

G. S. Rossano has recently completed a statistical analysis of a variety of recent contests at various levels. Here is the link to the article on his web site.

http://www.iceskatingintnl.com/current/content/scoring drift.htm

Rossano concludes that even taking seeding procedures into account, typically there is a 4 to 8 point advantage, at all levels, to skating later in the competition.

After careful study of numerous event segments where there is a random draw, including domestic and international competitions from the Juvenile to Senior level, we find there is an unmistakable systematic upwards drift in the judges' marks that is present in nearly all event segments. This drift amounts to typically 4-8 points during events, even for events as small as 12 competitors. This drift has a clear impact on which skaters advance from initial rounds to final rounds, and which skaters receive medals.

Interestly enough, Rossano's earlier studies of 6.0 judging did not show this trend, contrary to almost everyone's expectations.

It was widely said under 6.0 that placements drifted downwards (results improved) during an event, and thus there was a disadvantage in skating earlier in an event. In a study of 6.0 marks ten years ago, and again in 2004, we found that was not in fact the case.

Under 6.0 one finds events where the placements trend up, others where they trend down, and still others which show little trend at all. Averaged over several event segments, placements in 6.0 do not show a statistically significant trend for earlier skaters to be placed better or worse than later skaters in unseeded event segments with a random draw. Thus, under 6.0, the judges had the demonstrated ability to mark events with a consistent standard that did not put skaters at an advantage or disadvantage due to start order.

Rossano does conclude, however, that the base values of technical scores do not suffer from the same upward drift as the PCSs and GOEs, and credits the technical specialists with being able to maintain a consistent standard throughout a competition.

For this black cloud [the upward drift of PCSs and GOEs], there is one silver lining, that being the Base Value scores from the Technical Panels. The Technical Panels appear to show consistent judgment throughout an event, and even from one event to another for a given day/competition.

(The article is reinforced with a number of graphs that are easy to read. A line sloping upward shows that the scores increased as the contest went on; a downward sloping graph indicates the opposite.)

 

[PolymerBob]

When I checked the starting order at Skate America, it seems that the Americans skate last, regardless of standings.

http://www.isufs.org/results/gpusa07

 

[Tinymavy15]

yeah, I think it matters a somwhat. not as much as it used to though. Skting last will help. Look at Irina in 2005, Miki 2007, Shizuka 2006 Olympics. All skated last, or next to last.

 

[hockeyfan228]

Couldn't an upward drift under CoP in a random draw mean that random skaters would receive the benefit of upward drift, giving each skater an equal chance at reaping the benefit, as opposed to a pre-determined ranking under 6.0?

 

[Vash01]

I prefer the random order in the SP, so that those with less experience or lower past scores don't have to start with a disadvantage.. For the LP it may be OK to group them based on the SP performance. I have always thought that skating order mattered even in the cop.

 

[slutskayafan21]

I prefer the random order in the SP, so that those with less experience or lower past scores don't have to start with a disadvantage.. For the LP it may be OK to group them based on the SP performance. I have always thought that skating order mattered even in the cop.

I agree. Random draw for SP, reverse order of SP finish for LP. Perfect idea.

 

[Mathman]

When I checked the starting order at Skate America, it seems that the Americans skate last, regardless of standings.

http://www.isufs.org/results/gpusa07

I think you are looking at the list of entries, not the skating order. The entries are listed in alphabetical order by country, so USA is always listed last (unless there is someone from Uzbekistan).

 

[Mathman]

I believe that some of the changes about seeding were insisted upon by NBC. For instance the rule about separate random draws for the top three and then the next three for the final group in the free skate.

But the short programs at State America are not being shown on TV, so that shouldn't make any difference.

Couldn't an upward drift under CoP in a random draw mean that random skaters would receive the benefit of upward drift, giving each skater an equal chance at reaping the benefit, as opposed to a pre-determined ranking under 6.0?

I suppose. But one would rather have the contest decided by who skates the best, not by who got lucky in the draw.

 

[hockeyfan228]

But one would rather have the contest decided by who skates the best, not by who got lucky in the draw.

That would be ideal. But do you believe that under 6.0 the rankings were accurate and not reputation-based?

 

[Mathman]

That would be ideal. But do you believe that under 6.0 the rankings were accurate and not reputation-based?

I was very surprised that Rossano's numbers show that the problem is worse and not better under the new judging system. I (and I think most people) would have expected it to be the other way around.

I think that the biggest part of Rossano's concerns come not from World Championships but sectional and regional contests, some for juniors, novices, etc. In these contests I wouldn't have thought that "reputation" would play as big a role. Yet Rossano argues that even so, skating later in the order translates into higher scores on the average, with or without seeding.

 

[enlight78]

I was very surprised that Rossano's numbers show that the problem is worse and not better under the new judging system. I (and I think most people) would have expected it to be the other way around.

I think that the biggest part of Rossano's concerns come not from World Championships but sectional and regional contests, some for juniors, novices, etc. In these contests I wouldn't have thought that "reputation" would play as big a role. Yet Rossano argues that even so, skating later in the order translates into higher scores on the average, with or without seeding.

I think it makes perfect since. In 6.0 skaters are judged against each other. In COP skaters are supposed to be judge against a universal standard. I believe judges start off with a high standard at the beginning of an event and then gradually lower their standard as the event goes on. I believe judging PCS can be so exhausting that the judges are unconsciouslly giving out higher scores. Or could be because some of the early skaters make so many mistakes that they just drop their standards. Maybe skating should be from highest rank to lowest so skaters without repitations can get a break.

 

[Ptichka]

This is an interesting issue. I remember when my cousin was a TA for a graduate economics course at NYU, he frequently had to grade exam papers. When he evaluated that marks he gave out, he found that he tended to grade higher as the night progressed; some trivial conclusions that would prompt him to penalize a student at 5 pm would solicit no more than a sigh by midnight, while any shimmer of original thinking would get a bigger reward for a student at 1 am than it did around dinnertime. So, the question is - how is it possible to get rid of, or at least minimize this bias for an ultimately subjective sport like figure skating.

 

[mzheng]

If that's the case, IMO, they should didio at first. Then using two group of judges judge from video, once forwd order, once reverse oder, then avg.

 

[feraina]

In Psychology, there is a well known "recency" effect. That is, the last item in a list tends to be the most memorable. In psychology experiments, different trial types are always randomized to minimize ordering effects. It's completely inherent to the human psychology NOT to be perfectly objective and oblivious to ordering effects. As long as there are going to be humans judging skating, this will continue to be true. The only way around this is to completely randomize the order of skaters (easy), or to use a completely computerized way of judging (hard). No other way to be fair.

My 2 cents.

 

[Joesitz (RIP)]

If we accept the fact that judges are human, then they too, will have problems with the order of skate. If you think about 30 competitors divided into 5 groups interspersed with Zamboni sweeps, may be difficult to keep a relaxed mind.

NBC wants to bring in ratings, so the order of skate is upper most in their acceptance of viewing the competition. Random drawing which would be the fair way will not go over with the Networks if top skaters skate early on.

Challenging skaters should be in the penultimate group and Top skaters in the final six to skate. I think there should be no change affecting that system. Whatever works to keep it that way.

Die hard fans will sit though anything.

Joe

 

[feraina]

Challenging skaters should be in the penultimate group and Top skaters in the final six to skate. I think there should be no change affecting that system. Whatever works to keep it that way.

I agree that's a good compromise between fairness and capitalistic gains (oh, and also viewers' access to great skating).

I HATE the new rule in GP that _strictly_ orders the skaters by ranking or SP results. That's really unnecessary even from the viewpoint of the TV networks, and totally exacerbates the unfairness problem.

 

[Mathman]

I HATE the new rule in GP that _strictly_ orders the skaters by ranking or SP results. That's really unnecessary even from the viewpoint of the TV networks, and totally exacerbates the unfairness problem.

Totally unnecessary in the case of Skate America in the U.S., since NBC TV did not show the short programs at all.