You can measure anything,
I think this claim is worth revisiting.
Here are the criteria for judging program components.
https://www.isu.org/figure-skating/rules/sandp-handbooks-faq/31474-program-components-chart/file
Under “Presentation” we see, for example, “expressiveness and projection.” Earlier versions of this document used the language “the skater establishes an invisible connection with the audience.” Can this be measured?
Well, we could install applause meters and the skaters are awarded points in proportion to the recorded decibels. Or we could go high tech and fit the audience with heart monitors. The skater wins who elevates the aggregate heart rate the most.
Of course we could just eliminate this bullet point altogether as being vague and not readily measurable. But we
want to include this sort of thing (entertainment value, perceptions of “artistry”.) in figure skating.
Moving on to the question of judging the judges by applying statistical means to measure how well their marks match those of the majority of experts, here is the ISU document about the workings of the “Officials Assessment Commission” regarding “Evaluation of Judging – Assessments for the Figure Skating Branch.”
https://www.isu.org/figure-skating/rules/fsk-communications/31545-isu-communication-2583/file
Every year a few judges are so egregiously out of line as to run afoul of the ISU, but in general a judge would have to be way, way off before any action is taken. For the most part, a cheating judge must be caught by actual (non-statistical) evidence – Mr. Balkov is caught on video tapping his toe to indicate what scores the other judges should award to ice dance performances, Mme La Gougne tearfully and publically confesses, Yes I did it – Didier made me.
The suggestion that “majority of opinions” ought to constitute evidence of whether a judge is doing his duty, this actually is a point in favor of the old ordinal system as opposed to “add up the points.’ It is not so easy to decide that a skating performance contains 6.25 points worth of “Expressiveness and Projection.” But it is usually pretty straightforward to opine, “this performance was better than that one, with the other coming in third." In the case of performances that are close in quality, majority (of the judging panel) rules.
In general, when we use statistical methods, we must be careful to state exactly what it is we are studying. If we use statistics to measure the extent to which h a judge’s opinions match those of the the majority of experts, well, that is what we are measuring. To add on the emotionally charged value judgment “and if not, then the judge is not exercising due diligence,” this is a technical quibble using the word “diligence” in a specialized way rather than in its ordinary meaning in discursive English.
Of course, dishonest people will find a way to weave their nefarious plots no matter what the system.