The CoP is intended to inject a certain amount of uniformity and objectivity into the judging process. Our main statistical tool for deciding whether this goal is being met is called "Analysis pf Variance."
Here is a simplified example. There are two skaters and three judges. The scores were
Skater A: 2 3 4 (average 3)
Skater B: 4 5 6 (average 5)
The grand mean of all of these six numbers is 4. But some of the scores are higher than 4 and some are lower. Why?
There are two factors. First, the scores might be different because the performances of the skaters are different: Skater B was better in this competition.
Secondly, the scores might be different because the criteria of the judges are different: Judge #1 appears to be stingier with his/her marks across the board, and Judge #3 is more generous.
The technique of Analysis of Variance assigns numbers to gauge the relative importance of these two sources of variation. First we compute the total variation. For each of the six scores, how far is it from the average value of 4? Square the differences and add them up:
Sum of squares (total) = 4+1+0+0+1+4 = 10 units of variation in all.
To measure how much of this variation is due to the actual skating performances, we eliminate the differences in the judges by replacing each judge's score with the average for that skater.
Skater A: 3 3 3
Skater B: 5 5 5
Now the variation among the scores (the sum of the differences between the score and the grand average of 4) is
Sum of squares (Skaters) = 1+1+1+1+1+1 = 6
Thus 6 units of variation, or 60% of the total, reflect the fact that skater B really did outperform skater A.
But how much variation is there among the three judges? This time we are judging the judges, not the skaters, so we replace the scores by the average for each judge.
Skater A: 3 4 5
Skater B: 3 4 5
Sum of squares (judges) = 1+0+1+1+0+1 = 4 units of variation.
So in this example 60% of the total variation is correlated with the actual skating performances and 40% with the personal peccadilloes of the judges.
In a perfectly objective scoring system we would have 100% skaters, 0% judges.