- Joined
- Jul 26, 2003
But it's such a good one.moyesii said:Oh I thought this thread was finished...
As opposed to ordinals, where a judge, with a typical human brain, see a relatively small number of "events" in a row, and condenses that into a judgement, sees a few more, aggregates that into the initial judgement, sees a few more, etc. until the very end, when he or she must then compare that final judgement against what she has seen over the last 20 or so programs? It seems to me that there is a greater chance of that logical chain breaking down and many more errors seeping in.But in fact, despite the apparent transparency of the system, each of the links in the scoring chain are very weak, such that the system has a good chance of breaking down at any point in the chain.
You've cited nothing to convince me that ordinals are no less error prone, or how under ordinals, a skater cannot be screwed by the single hand of one judge -- i.e. 5/4 split. As far as random chance is concerned, because one set of judges is chosen at the beginning -- not for each scoring element -- starting with 10 and choosing 7 is no different than having 7 in the first place, and using trimmed mean, like is done in diving. What the data shows is the variance among 10 judges instead of the variance among 7, and that if a different combination of judges were chosen -- outright or randomly -- there would be a different result. I see no difference between CoP and ordinals here.On the other hand, the system itself is crude and error-prone, so that the skaters can be majorly screwed by the single hand of one judge or random chance.
If it were for the fans, the fans would vote like on American Idol.In effect, what most fans seem to be saying is, the system sucks for the skaters, but at least we get to see what the judges are doing to them. Yay for the fans!
Yes, and in contrast to ordinals, mistakes are transparent to everyone looking at them. Or another was of looking at is, that the standards aren't clear, and judges interpret the rules as having more discretion. For example, one judge may think that -3 is mandatory for a fall, while another might believe that if the entrance and rotation were good, that s/he can give the skater a -2. That can be addressed through clarification and training.My 2nd response is, if the CoP "provides data to determine whether judges are adhering to the code," then what disciplinary actions against cheating judges are available to us in a secretive, pooled scoring system? We have already seen numerous times in the GP series that judges have not been taking the proper deductions.
I disagree that the bias for elements are just as difficult to decipher or rationalize. Computers and statisticians are quite good at finding patterns, and there is the videotape to confirm the basis for the pattern.Yes, and under CoP, bias will be a factor in ALL of the different element and component scores, and those biases are just as difficult to decipher or rationalize.
I agree that the program elements have been, for the most part, a mere substitution for ordinals, and a bust to date. Part of the underlying cause will not go away, and that is the aggregate nature of the judgement. There are some things that I think can be addressed in several ways, such as dividing the judges into technical judges and presentation judges, who would be specialist for a maximum of two of the PE, like is done in ski-jumping, and quantifying transitions in a similar way that elements are quantified -- i.e. counting each one and having a predetermined difficulty.Unfortunately, it turns out that each of the 5 program components are just as obscurely marked as the single presentation mark. That's why we have concluded that ranks are less error-prone than an absolute point scale, and a single ordinal introduces less error into the results than do 5 separate marks for program components.
What does it mean when one judge gives a skater 5.8 for pre and another judge gives that skater 5.6 for pre, and the results are decided by one ordinal? We think the same thing that is happening under CoP: that it was extremely close, but that might not be the case, because the actual score in the ordinal is meaningless in its own context; it is a way to "leave room," even when the skaters are extremely close. Under CoP, we might think that the decision is close and statistically insignificant, and either skater could have won. We might look to see if the first judge was judging relatively higher across the board than the second judge, or if the second judge really thought that the skater was that much worse or better.What does it mean when one judge gives a skater 5.50 points for Skating Skills and another judge gives the exact same skater 7.25 points? Then, what are we supposed to think when the final results are decided by less than a tenth of a point?
Humans are biased. A judge is not a critic; a critic can have as many biases as s/he wants. A judge is someone who is trained against that natural bias and is audited against whether s/he judges to the defined standard without bias. One of the main points of any judging system is to be able to detect and address that bias.Just because there is more information provided by the CoP, doesn't imply anything about the integrity of the enormous amounts of information being given. Judges in BOTH systems are going to be biased.
The important thing is that in addition to the error from bias, the CoP also introduces random error into all the scores and calculations. The random error is far more serious, because bias cannot be completely gotten rid of in a judged sport, but random error should be minimized. A large amount of error means that the system is not producing reliable results.
A large amount of error may mean that the system is not producing reliable results, but there is no evidence that 6.0 produces equal or more reliable results. A large amount of error could point to the difficulty in judging figure skating, or that the judges need to be trained more.
I mean that ordinals place the favored skater ahead, with no explanation or auditability of the value judgement.I'm not sure I understand. Do you mean that ordinals simply place the better skater ahead, regardless of the "quantifiable" difference between skater A and skater B?
I disagree. Under 6.0 there are written standards for the judges to follow that have explicit value judgements built in. That is why the judges go to training seminars. Under 6.0 they've taken the leeway and have been allowed to decide their own balance based on preference for each skater in any competition.First of all, it's not always possible to quantify instead of qualify the differences among skaters. It would only be possible if skating was a jumping contest. But every skater has their own strengths and weaknesses that cannot be added and subtracted into totals that can be quantifiably compared meaningfully. Really, skaters can only be compared relatively.[/b]
Notmyjob. But it is the ISU's, according to their bylaws.Well I didn't know this was your job j/k
Yes, it is how I think judges judge, however simplified. By aggregating a series of judgements, it means that each judge can decide to factor or ignore any element, any level of difficulty, on any scale. It's not just a matter of bloc judging. It is a matter of giving leeway that can be justified in almost all cases "I thought "X" outweighed "Y' so I gave them ordinal "Z."First of all, I really hope you don't think that that's how judges judge. Obviously, bloc judging totally confounds the results process.
According to your logic, a wide range of scores should be acceptable -- it is, after all, what we have under 6.0 -- as long as the relative scoring by each judge is consistent. Under CoP it is possible to determine if this is the case.I gave these numbers based on how I recalled their skates. And basically, that is what the judges do in the CoP: At the end of the skate, they assign 5 numbers to the program components.
I think the opposite. It think that the absolute scale is far from subjective and inaccurate; I believe that in many cases the judging has been subjective and inaccurate, and if anything, some scores are not justifiable, legitimate, and incontestable, based on a comparison of the written code to the performances.The absolute scale is subjective and inaccurate enough that any WIDE range of scores by the judges will seem justifiable, legitimate, and uncontestable.
Again, the random factor is the same as having chosen 7 judges outright, while 3 were armchair judges. The variability of the judges marks is only relevant to the final placement if relative standards aren't consistent. I think it is a bad thing that the presentation scores are so out of line with the standards, but the data makes this very clear.I made the scores above so that one judge preferred Baiul, and one preferred Kerrigan. Basically, we see that one judge likes Baiul, because of her balletic style and choreography. The other judge preferred Kerrigan, because of her strong, clean lines and perfect execution. Now, how the competition turns out is up to anyone and no one under the CoP system, because of the variability in the judges' marks and the random count of the scores.
The accountability takes place when judge A is asked, and exactly how did skater A's performance meet the criteria of score A?Would you feel better now, knowing that one judge scored Kerrigan higher in Perf/Execution, even though another judge scored Baiul higher in the same mark? Or are we back where we started with the ordinals, except now we have 5 marks instead of one and a multitude of confounding variables, and no accountability.
That the ISU has not had the data with which to prove bias, because a single score, or single sets of scores can be justified by the relative judgement, weighting, and standards of the judge.I don't know what your point is. The ISU or the judging system?
No, I mean a more transparent scoresheet for technical andprogram elements, because the data I've seen so far makes it clear to me that the standards aren't being used. Overall secrecy -- hardly. It is secret to me and thee, but it is not secret to the ISU after the event.Let's be specific here. You mean a more transparent judging chain, a more transparent scoresheet for technical elements, but still equally untransparent marks for program components, and overall secrecy in the judges' marks.
Three, actually, the specialist, the assistant specialist, and the controller. All of whom consult with each other and are mandated to use the videotape, if there is an issue. All three are hired, trained, and fired by the ISU. They are hardly infallible, but they are better trained, technically, than the average judge, based on the qualifications needed to be hired for the position, and the ISU is not subject to the choices of the Federation, which is the case with judges.Yes, that is most definitely a positive. Now here is one for OBO:Each judge marks the elements, including non-jump or spin elements, as THEY see them and not as some technical specialist calls them. In other words, in a 9-judge panel, there are nine judges making independent decisions, instead of a single person making calls for the entire group.
I disagree that the relative difficulty of elements should be judged by nine judges making independent decisions. That is like saying that a baseball referee should have discretion on whether a runner is called safe or out or whether a basketball player travelled or whether a play is offside. Yes, officials will make mistakes -- and, on occasion, receive death threats because of this -- but the rules are codified, and unlike the ISU, the leagues can fire officials that aren't following the code.
And these more subversive ways would be?Do you mean Baiul at the Olympics? That was more likely bloc judging. If there's a group of collaborating judges working the CoP system, they can easily manipulate the system in more subversive ways.
I think the program element scores are not being judged properly. However, I haven't seen a consistent pattern of exaggerating the component marks, partly because so few skaters' programs have been televised on US networks. (Which doesn't mean that I haven't missed them.)Also, under CoP we have seen many times already in the GP that jumping mistakes are often "forgotten," by exaggerating the component marks.
Actually, they don't. They can take into consideration any combination of things they want, and weigh any combinations of things they want in a different ratio for each skater within each competition.When did it ever do such a thing in the ordinal system? In both systems, the judges must take into account both the strengths and weaknesses of a skater's performance.
Based on the standards, only two of the five program elements should be subjective, and then not highly so -- choreography and interpretation. There are clear technical standards for skating skills and performance and execution, and, as I mentioned earlier, I think that transitions could be made more quantifiable.The CoP seems to be more literal and systematic in its calculations, because of the TES mark, but people are forgetting that the TPC is highly subjective, and has been the deciding factor in numerous events in the GP.
It sounds like you think that this is happening already, because the judges can look at the scores that they gave in the moment, and decide to use PE scores to "adjust." The only reason to "adjust" a technical score is to review the video to see if the original score was unduly harsh or too lenient.In addition, the problem of the systematic adding of the total elements and program components is that the marks are riddled with random and human error. Some consider the leeway with which judges have to award marks in the ordinal system to be a bad thing, but I consider it a good thing because it allows judges to check for error that would otherwise be present in the calculated results.
I agree that some things could use some tweaking, and I'm basing this on what skaters and coaches have said about relative difficulty that is nowhere differentiated by CoP, ex: a 2T/3T is more difficult than a 3T/2T. Since computers are extremely good at making these calculations automatically, there's no reason to stint on the number of differentiations and weights.Well some people consider this a bad thing. Some people think that the values that have been assigned to elements in the CoP don't make sense.
Whether I think spins should be more highly valued is irrelevant; this is a decision that the ISU must make, and I can cheer or hiss or beat my breast and say that the sport's apocalypse is here. But at least the skaters and coaches can look at the relative weights and adjust accordingly to maximize results.
And those tyrants would be, the ISU technical committee? A group that has been responsible for setting the technical standards for technical scoring since it was formulated? Do you really think that officials of a sport should ignore the rules, regulations, and standards of the sport -- because having standards is what makes them eligible for the Olympics?I like the idea of a democratic judging panel (more reflective of the population) rather than a tyranny of what is considered valuable in skating.
The only way that ordinals are adaptive is the anarchic changes that are made by individual judges.Also, a technically and artistically rapidly evolving sport like skating NEEDS an adaptive system like ordinals, not a system rigidly coded with values for certain elements whose emphasis may or may not change with the times (even year to year or event to event!!)
The ultimate goal of scoring is to assign placements. It's the same goal as in diving, gymnastics, and ski-jumping.We have also seen that skaters can build an insurmountable lead after the SP... But, that is not the main problem. As I discussed earlier in this thread, the ultimate goal of a judging system in skating is to assign placements, not scores.
What is your justification for that assertion?And there can never be such a thing as a "statistically insignificant difference" between any two skaters. This is a fallacy or myth generated by the CoP.
I don't agree that AMPLIFICATION is the most important element of a judging system. 6.0 gives a false impression that the judgement was definitive, when it may have been anything but definitive.The strength of the ordinal system is that it is able to AMPLIFY the marginal differences between any two closely matched competitors into meaningful differences in placements.
Just as under 6.0 the placements could have been very different under a different set of judges.The CoP is unable to do this, because any skaters that finish within about 5 points of each others' totals will not have a clear and definite justification for their placements (due to error and variability), and statistically there is no reason why those placements wouldn't change with another random draw.
That's not the only interpretation. The scores under 6.0 could tell me that the under the system, the judges made arbitrary decisions in scoring skaters whose performances were very close, because the requirement was to have to choose.The only thing the scores of these closely ranked skaters tells you is that the system was unable to differentiate the skaters. The placements of these undifferentiated skaters are entirely due to chance.
There is no evidence that there was anything conscious, deliberate, or thoughtful about the decisions made, just that a decision was made.Not so with ordinals. The judges must make a conscious, deliberate, and thoughtful decision to put one skater ahead of another so that there is no misunderstanding that skater placed #4 was better than skater placed #5.
There are two ways to look at this: 1. Is the judge internally consistent in his/her scoring, even if the score doesn't match code 2. Does the score match the code?And how do you prove that the judges "aren't following the code" when it is undetectable? I will quote myself here:
"The absolute scale is subjective and inaccurate enough {due to norms in human acceptance of variability} that any WIDE range of scores by the judges will seem justifiable, legitimate, and uncontestable."
The new system is the cost of remaining in the Olympics. I'm sure that the Olympic TV revenue will offset the cost of the system, that a number of skaters have said has already helped them improve their programs from competition to competition.Yeah, who knows, maybe the ISU is just buying itself some time, and even if the CoP is passed at the ISU congress, maybe after years and years of CoP failures and continuous modifications, like maybe after a couple of years of Olympic scandals, the ISU will introduce a "new" system of ordinals to fix the problems of the "old" CoP system. So maybe the ISU is buying itself some time with this experiment... It's not so bad since they're only wasting tons of money and affecting the results of competitons in the meantime.