Skaters/Judging experts on GS - Question

[Mathman]

Speaking of "flip-flops" under ordinal judging, I really cannot see what the fuss is about or why this is considered an undesirable feature of a judging system -- so much so that we had to change the judging system, twice, because of it.

This is how flip-flops occur.

After five skaters, with one to go, here is the provisional ranking.

A
B
C
D
E

Now skater F performs. Although he did not skate well enough to escape last place, he did skate well enough to steal a couple of ordinals from D.

A
B
C
E
D
F

What is so terrible about that?

 

[gkelly]

Well, the big confusion came when the flip was more like
A
B
C
D
E

in the long program changing to

A
C
B
D
E
F

So it changed places that weren't adjacent to F's placement and that were medal positions, and also along with the factored placement changes it changed the overall results from something like

B
C
A
D
E
F

to

A
D
B
C
E
F

B went from gold medal position before F skated to bronze in the final results, without F himself getting anywhere near medal position (from most judges -- but it was the one or two outliers who made the difference).

That kind of thing didn't happen very often, but when it did, it got people upset.

The more benign kind of place switching that you mention happened all the time but no one really cared except the skaters who may have left the arena after they skated and came back later to find themselves in an unexpected final position.

 

[gkelly]

Another example that affected medal results:

1994 Europeans dance competition

After the Compulsory and Original Dances, both Usova/Zhulin (won both CDs) and Torvill/Dean (won the OD) were in control of their own destiny; Grishuk/Platov were behind on factored placements so they would have needed help to win.

After Usova/Zhulin and Torvill/Dean skated their Free Dances, U/Zh were ahead in the free dance as well as overall. Since the last team G/P couldn't catch them just by beating them in the FD, they seemed assured of gold.

Well, the ordinals got mixed up so that U/Zh and T/D flipped places in the free dance, and that mixed up the overall standings as well.

Before G/P skated, the FD standings were
U
T
and so were the overall standings.

After G/P skated, the FD standings were
G
T
U

and the overall standings were
T
G
U

You can see the source of Zhulin's consternation.

The Britons, who skated to "Face the Music and Dance" by Irving Berlin, entered the evening with only the slimmest advantage over the defending world champions, Maya Usova and Aleksandr Zhulin of Russia. Before the evening of free dancing ended, both couples would be upstaged by the second Russian pair of Oksana Gritschuk and Yevgeny Platov, whose exuberant performance earned them the highest scores of the competition, including three 6.0's for artistic impression.
Ads by Google

"We thought we were the winners when we saw those 6's," said Platov, who skated after the other two couples had finished. Mathematics Competition

Instead, those 6's ended up benefiting Torvill and Dean by knocking Usova and Zhulin out of first place in the free dance. By the time the mathematical smoke had cleared, Torvill and Dean were declared the overall winners because six of the nine judges placed them in second while five put Usova and Zhulin in first or second place.

"It was too algebraic," said Zhulin, who ended up settling for the bronze medal with Usova, behind Gritschuk and Platov.

"I'm still in shock," said Torvill.

 

[Dragonlady]

I’ve mentioned before that when my daughter was skating (under 6.0), there was a competition where all of the judges placed her 3rd, but her final placement was 4th. It’s very difficult to explain to a 9 year old who thinks she’s won a medal, that no, in fact you did not win a medal. And while I understand how the ordinal grid works, it still seems wrong to me that she didn’t get a medal. If I feel that way, imagine the reaction of parents who DON'T understand how the grid works.

These kinds of flip flops happened far more frequently that Mathman would have us believe and it made the results appear to be off every time it happened. While people on discussion boards may only be interested in elite skating, there is much more to the figure skating world that the handful of skaters we see on TV and nobody liked flip flops.

If parents think their kids are getting screwed over by the scoring, they won’t waste further money on the sport.

 

[Mathman]

You can see the source of Zhulin's consternation.

To me, the only reason Zhulin is upset is because of how the sequence unfolded in time, not because of how the voting system works.

Suppose that all three teams skated the same, that all of the judges voted the same way, but G&P had skated first instead of last. Then all is well

G
T

G
T
U

Factor in the OD:

T
G
U

The "flip-flop" has magically disappeared. That's because it wasn't there is the first place, partial rankings in the middle of the event being irrelevant if not misleading.

Here is 2002 Olympics.

After Sarah and Michelle had skated, the overall rankings were

M
S

But Michelle knew she was out of luck. Sure enough, Slutskaya inserted herself and it ended up.

S
I
M

Michelle has no reason to be puzzled or outraged. This is exactly the same as if the skate order had been I, M, S.

I
M

S
I
M

Or if the order had been S, I, M.

S
I

S
I
M

Moral of the story -- don't count your chickens until they hatch.

I’ve mentioned before that when my daughter was skating (under 6.0), there was a competition where all of the judges placed her 3rd, but her final placement was 4th. It’s very difficult to explain to a 9 year old who thinks she’s won a medal, that no, in fact you did not win a medal.

I can see where it would be difficult to explain to a nine-year-old that you have to look at all the ordinals for all the skaters, not just your own.

I think most adults can understand how a skater can get a couple of firsts, a couple of seconds and a couple of fourths, and end up ahead of a skater with all thirds.

 

[gkelly]

To me, the only reason Zhulin is upset is because of how the sequence unfolded in time, not because of how the voting system works....
The "flip-flop" has magically disappeared. That's because it wasn't there is the first place, partial rankings in the middle of the event being irrelevant if not misleading.

Yes, but the partial rankings in the middle of the event are announced, which means that the way the event is presented to the public and to the participants is misleading by design (not by intention, but by the system itself).

Here is 2002 Olympics.

That's purely a factored placements flipflop, which were more common and more predictable. Kwan would know once she lost the LP to Hughes that she would lose the gold if someone else beat her in the LP.

The situation would have been comparable if Kwan had been ahead in the freeskate as well until Slutskaya skated.


But yes, since factored placement flipflops were very common, the skaters surely learned to expect them even if the less experienced fans did not.

 

[antmanb]

Totals:

Urmanov 3+1 = 4
Candeloroo 2+2 = 4
Zagorodniuk 1+3 = 4
Kulik .5+5 = 5.5
Yagudin 2.5+4 = 6.5
Vlacenko 1.5+6 = 7.5

For the three skaters tied with 4 points, the LP breaks the tie.

Ant, my man! Is this your example of something that is hard to understand? Let's go over it again. 3+1 = 4, ...

No it was more the working out of the majority ordinals etc detailed in the Sandra Loosemore link that gkelly provided. That and the icecalc matrices that were generated for orindal judging that i never knew how to read or understand. Nowadays, the protocls and totals make it easier to understand! Really the only point I was making was when people say that COP is soooooooo difficult to understand, 6.0 was waaaaaaaaaaaaaay easier. I don't think it was.

Re the copyright query, i'm not sure what happens for shows but for competitions I believe the skaters have to submit the names and cuts of the music along with their application form, and I always assumed it was to clear it from a public playing point of view for copyright. Way back when i did the adult nationals I even had to write down the code from the CD....i wonder howyou do that nowadays with downloads?

 

[Mathman]

No it was more the working out of the majority ordinals etc detailed in the Sandra Loosemore link that gkelly provided. That and the icecalc matrices that were generated for orindal judging that i never knew how to read or understand...

But it is very interesting!

One salient feature of majority of ordinals is that it allows a compromise candidate to win who is no one's first choice but at least none of the voters despise him.

Let's say you are trying to hold your fragile democracy together and it is being pulled apart by warring factions A and C. Here are the preferences of four voters (the first column is the preferential ordering of the first voter, etc.)

A.....A.....C......C
B.....B.....B......B
C.....C.....A......A

Neither A nor C has a majority of first-place votes. But B has a majority of first and second place votes. So the compromise candidate, B, wins by the majority of ordinals method, thus saving the nation from civil war.

The problem with OBO is that it invites rock, paper, scissors problems. I put out rock, you put out paper, gkelly puts out scissors. Who wins? (This is less likely to cause difficulties if there are more judges than skaters, however.)

Speaking of nine-year-old kids, I wish I had been a figure skater when I was nine. I would have had a field day with all this. I could have explained in great detail to all my friends exactly why they lost. Imagine how popular I would have been!

 

[Dragonlady]

In my daughter's case, there were 3 judges, all of whom placed Tink 3 giving her a total of 9 points. Another girl had two 4th place ordinals and one 1st placed ordinal. Statistically they were tied at 9 but the other girl had a higher ordinal than Tink, giving her the medal. I understood this even if my daughter didn't. She looked at the row of 3's beside her name and wanted to go get her medal.

There is a saying "Not only should the result be fair, but it must be SEEN as fair" and the problem with flip flops and they don't seem to be fair. In my daughter's case, a majority of the judges said my daughter beat this girl. One judge saw it differently, so it wasn't even a majority opinion to give it to the other girl. That's the part that I have difficulty with.

 

[Mathman]

^ Oh, that is very interesting, Dragonlady. They added the ordinals?!

That is neither majority of ordinals nor OBO. I don't think that system was ever used (for good reason) in elite skating. No, that is not good at all.

The reason why ordinals can't be added is, well, think about it. Numbers can be added: 4+4+1 = 9 (yes). But placements can't be added: 4th place plus 4th place + 1st place = 9th place (no).

I guess for an event for children with only three judges, they figured this was an easy way to do it. (This is a called a "Borda count" method, by the way.) But under either of the two IJS 6.0 scoring methods, majority of ordinals or OBO, your daughter would have won (depending on other skaters, of course). She had the "majority of 1st, 2nd, and 3rd place ordinals" over the other girl (majority of ordinals) and also head to head (OBO).

But you can't blame the 6.0 systems for that. In fact, adding up the ordinals is more like CoP scoring (add up the points) than it is like true ranking systems. Under CoP two judges might have given your daughter (in PCSs, let's say), 30 points and one judge who didn't like her 10 points. All judges gave the other skater 25 points. Same thing. Two of the three judges thought your daughter was better, but in an add-up-the-points system like the CoP, the other girl wins. :disagree:

Edited to add Let me put this more emphatically. There is no purely ordinal voting system in which your daughter could have lost to the other girl, holding two ordinals out of three.

In an add up the points (or -- yikes! -- add up the ordinals) system, yes, weird things like that are not rare at all.

(BTW, your daughter's experience had nothing to do with flip-flops.)

 

[skatinginbc]

Speaking of "flip-flops" under ordinal judging, I really cannot see what the fuss is about or why this is considered an undesirable feature of a judging system -- so much so that we had to change the judging system, twice, because of it.

The majority criterion, which ordinal voting systems aim to satisfy, is NOT essential in sport. Medal winners shall be determined, not by the consensus of a "society", but by their actual performances. Sport judges are expected to give absolute measures in terms of, for instance, the height of a bar cleared and the number of goals scored, rather than the relative rank distance between competitors. If Jumper A clears 2.02m and Jumper B fails it, then their relative placement in that high jump competition should remain constant regardless of the performances of others. If Dai was truly better than Hanyu in the Short Program at the last Japan Nationals, it should hold true even if the judges re-watch the footage later after they have seen an indefinite number of performances during the time interval. In other words, it should be the absolute values (point) not the relative values (ordinal) that are being compared in the decision of the rank order of performances, and so indispensable are such criteria in sport ranking as independence of irrelevant alternatives, independence of clones, and the consistency criterion that any violation will be deemed suspicious and scandalous by the mass of the sport fans. As a consequence, the 6.0 One-by-One system (a Condorcet method) or any voting system that fails to escape Arrow's impossibility theorem is a dead-end approach. Cardinal voting systems (e.g., range voting) and interval-scale measurements are the only options that seem fair, logical and relevant in sport.

Can you name any sport competition (other than the old 6.0 in figure skating) that uses an ordinal voting system in judging?

I think we should abandon the application of traditional social choice theory to sport judging. Sport is not an election. It is more about performance evaluation than about preferential voting.

 

[antmanb]

See now my head just hurts, thus proving, yet again that COP is not more complicated and dificult to understand than 6.0

 

[skatinginbc]

now my head just hurts...

Sorry. Let me paraphrase my post in plain language: Flip-flopping is bad in sport because most fans will not understand how it can happen and thus will suspect something wrong with the judging system. If to earn credibility from the fans is one of the goals of a sport judging system, any ordinal system that permits flip-flopping will not work. Even if someone is familiar with the mathematical aspect of it and understands how it can happen, it does not mean that it should happen or it is a logical way of judging. If A performs better than B, then A performs better than B. It should have nothing to do with how well other competitors perform. When this principle is violated, the spirit or the perceived fairness of the sport is also violated. The old 6.0 system (an ordinal voting system) was a rarity in sport for it went against the conventional logic of sport. Its demise was for a good reason.

 

[Mathman]

If Dai was truly better than Hanyu in the Short Program...

It is not possible to determine whether this is true or not. (Note the undefined term "better.") Different observers have different opinions, and there is no external standard, like a tape measure or stop-watch, that can settle the matter independent of human observation and interpretation.

If A performs better than B, then A performs better than B.

It is not possible to determine this with certainty or objectivity. The best we can hope for is some kind of consensus among expert observers. Lacking unanimity, there are various schemes, none perfect, to resolve the conflict.

Figure skating is not like other sports, nor should it wish to be. A well-crafted figure skating performance makes my heart sing. To say that it sings 168.31 decibels loud reduces figure skating to -- well, to a mere sport. (YMMD )

 

[skatinginbc]

It is not possible to determine whether this is true or not. (Note the undefined term "better.") Different observers have different opinions, and there is no external standard, like a tape measure or stop-watch, that can settle the matter independent of human observation and interpretation.

It is possible to determine which skater is "better" as long as a standard is established by a well-sampled interest groups.
Though subjectivity is involved, the system (FINA rules) used for judging an Olympic diving contest is "transparent and easy to understand" according to Michel Balinski and Rida Laraki's Majority Judgment: Measuring, Ranking and Electing. "Clear-cut, absolute meanings are ascribed to each of the numbers...Scores determine rankings, so there can be no flip-flops...The simplicity, in contrast with the rules for skaters and gymnasts, is striking." (http://books.google.ca/books?id=sSXxXBxuorIC&pg p. 147-148).
Indeed, differences between judge ratings are inevitable. It however can be estimated, accounted for, and hence controlled in a measurement model.

Figure skating is not like other sports, nor should it wish to be.

Let me put it this way: Do you think figure skating is more of a sport or more of a piano contest or more of a dance competition or more of a beauty pageant? Sport ==> Measurement Theory. Piano Contest ==> Measurement Theory. Dance competition ==> Measurement Theory. All three types of international competitions currently employ criterion-referenced measurements rather than preferential voting. Beauty pageants sometimes involve preferential voting. For instance, the top five finishers in Miss America are ranked by the judges and the winner is determined by the Borda count method. Electronic voting by the TV viewers is incorporated from time to time in Miss Universe competition.

 

[gkelly]

It is possible to determine which skater is "better" as long as a standard is established by a well-sampled interest groups. Though subjectivity is involved, the system (FINA rules) used for judging an Olympic diving contest is "transparent and easy to understand" according to Michel Balinski and Rida Laraki's Majority Judgment: Measuring, Ranking and Electing.

The difference between diving and skating, however, is that dives are judged as individual elements and a skating program is judged as a whole performance, a collection of elements and the transitional skating between elements, which is arguably more important than the elements themselves. (Without even getting into the question of artistry.)

I think that absolute scores assigned according to clearly defined (albeit complex) rules for the individual elements are similar to scoring for dives and can be considered transparent and easy to understand once everyone agrees on what the rules are and the values in the Scale of Values.

Yes, there will be some differences of opinions on GOEs and some gray areas on the level or underrotation calls, but the vast of the decisions will be straightforward and the vast majority will result in clear rankings of who was best, second best, etc., overall on elements.

Personal opinion about what should be worth more is another story. Ultimately the rules and Scale of Values will enforce a consensus even though individuals will disagree with individual rules.

But then we get to the program components, which are purely qualitative and in some cases based on criteria that go well beyond the actual athletic and technical skills demonstrated.

I don't think it's merely a matter of voting for preferences, even under 6.0. The preferences are based on criteria on which there is general consensus, but with room for disagreement in close cases and in cases where one skater's areas of strength are weaknesses for another and vice versa.

Under IJS, the qualitative evaluations have to be quantified and turned into absolute numbers.

So even if everyone agrees that Skater C was better at skating skills, transitions, and choreography and Skater D was better at performance/execution and interpretation, if they can't agree on exactly how much better each of the skaters was at her respective strengths, then different judges may end up with different rankings.

When judges disagree with each other about who was best in one phase of competition, then the question is whether combining the disparate expert evaluations should use cardinal differences in absolute scores or ordinal differences in comparative rankings to produce the consensus result. There are potential problems and advantages with both approaches.

Same if there is unanimous agreement about each phase but the skaters reverse standings from one phase to the next -- should the margins of victory carried from the first phase to the second be based on the absolute score differences between skaters or between the number of other skaters who ranked between them?

Let me put it this way: Do you think figure skating is more of a sport or more of a beauty pageant or more of a piano contest or more of a dance competition? Sport ==> Measurement Theory. Beauty Pageant ==> Measurement Theory. Piano Contest ==> Measurement Theory. Dance competition ==> Measurement Theory. All four types of international competitions currently employ criterion-referenced measurements. None of them is based on preferential voting.

I can't speak for Mathman. I'd say skating is a sport with aspects of a dance competition layered on top.

I'm not familiar with the types of criterion-referenced measurements used in beauty contests or piano contests or dance competitions. Can you enlighten us at all?

 

[Mathman]

^ (responding to skatininbc's post). I think the confusion between quality and quantity and between judging and measuring is rampant not only among lay people but also in the ranks of experts who study these things. Even the vocabularies are schmushed together. We hear people say, of these three candidates for public office, Carlos is qualified, Diego is more qualified, and Maria is most qualified.

No, no, no! Carlos is well qualified, Diego is better qualified, Maria is best qualified.

Takahashi's performance was better than Kozuka's. Takahashi's performance was not "more" than Kozuka's. And then to go on to say, not only was Takahashi's performance more than Kozuka's, it was 7.62 more -- not, mind you, 7.61 more -- I think we have tricked ourselves, by abuse of language, into believing falsely that we have measured something.

Beauty Pageant ==> Measurement Theory.

Really? You mean Miss Texas got a score of 4.38 worth of filling-out-a-bathing-suit points, while Miss California only looked good enough in a bathing suit to get 3.79 points?

But Miss California made it up in the talent portion of the contest. By sawing away on a cello she got 0.29 points, whereas Miss Texas' ventriloquist act was worth only 0.19.

Never fear, Miss Texas came roaring came back by answering the question, "What man do you most admire" with "My father" (5.92), completely outpointing Miss California's lame, "The President" (2.18).

The CoP is far cooler than I thought!

I liked the old-fashion way of judging beauty contests better. When the traveling circus came to town the local sheriff's daughter won the beauty contest in exchange for the sheriff looking the other way while the circus' pickpockets worked the crowds.

Sort of like figure skating judging after all.

 

[skatinginbc]

I'm not familiar with the types of criterion-referenced measurements used in beauty contests or piano contests or dance competitions. Can you enlighten us at all?

In the elimination stage, the competitors in a beauty contest, for instance, Miss America, is scored on various categories (e.g., Preliminary ==> Lifestyle and Fitness in Swimsuit - 15%, Evening Wear - 20%, Talent - 35%, Private Interview - 25%, On-Stage Question – 5%. Finals ==> Composite Score - 30% (Top 16), Lifestyle and Fitness in Swimsuit - 20% (Top 16), Evening Wear - 20% (Top 10), Talent - 30% (Top 8), On-Stage Question (Top 8)). The top five is then ranked according to a preferential voting method (http://www.missamerica.org/news/press-kit/national-judging-process.aspx). A typical scoresheet in a beauty contest is as follows (http://www.angelfire.com/biz/tncharm/judging.html):
1-9 NEEDS WORK
10-14 AVERAGE
15-19 GOOD
20-25 VERY GOOD

This is how International Chopin Piano Competition is judged:
"In principle, the Jury will admit to stage II no more than 40 competitors, and to stage III no more than 20. To the final, the Jury will admit no more than 10 competitors. At stages I, II and III, the competitors' playing will be assessed according to two systems: points and 'yes/no'. The assessment in the final will be based on points alone...The points system involves the assessment of a performance on a scale from 1 to 100 (no fractions allowed), with 1 as the lowest mark and 100 as the highest. The arithmetic mean of the points awarded to a pianist, as referred to below, will be obtained by dividing the sum of the points awarded to him/her by the number of assessing Jurors, with an accuracy of two decimal points. The points system is of secondary importance in stages I, II and III. However, in the final the points obtained over the successive stages will provide the basis on which the decisions will be taken concerning the awarding of prizes. If the number of points awarded by one of the Jurors deviates from the arithmetic mean of the total points obtained by a pianist in a given stage by more than the following:
10 points in stage I,
8 points in stage II,
6 points in stage III,
5 points in the final,
then a second, auxiliary arithmetic mean of the total points obtained by that pianist will be calculated, excluding the deviant marks. All the marks falling outside the above-specified norm of deviation from the auxiliary mean obtained in this way will then be corrected to the nearest whole number within this norm. The final, determinant mean will be that obtained from the marks corrected in this way." ( http://konkurs.chopin.pl/en/edition/xvi/rules/965_rules_of_the_competition_jury).

Here are some examples of criterion-referenced score sheets used in dance competitions: http://www.kentstatecheerleadingcamps.com/Dance_Score_Sheet.pdf, http://www.midwestdancefest.com/forms/score_sheets_09.pdf, http://www.backtobatoche.org/resources/Judges Score Card- Square Dancing edited[1].pdf

After more research, I have to admit that I made a mistake: A 6.0-like scoring system is often used in the final stage of dance competitions. Such ranking system is called "Skating System" (http://www.mso-dance.org/assets/docs/MSOSkatingSystemDefined.pdf, http://en.wikipedia.org/wiki/Skating_system). The International Dancesport Federation (IDSF) specified in its rules that "In finals adjudicators shall place couples. 1 is the best place marking. The same placing must not be given to more than one couple....no more than 6 couples should participate in the final...The Skating System shall be employed." (http://www.fedansa.org/documents/rules/IDSF COMPETITION RULES.pdf). However, it has undergone a change mimicking the one that ISU has gone through. Since 2009, a CoP-like system has been implemented (http://www.dancefloor-magazin.ch/wp-content/uploads/2009/10/idsf_new_judging_system-2009.ppt). It is funny that in dance competitions there are "Program Component Scores". And this is what they had to say about the change: "Again, the new system produced results that are perfectly in line with those obtained through the traditional skating system. One only needs to look at the classifications of the recent World Standard and this Grand Slam: identical! The closest contest in Wetzlar, GER, was certainly the one between Segatori – Sudol, GER, and Konovaltsev – Konovaltseva, RUS, for third place. If anything, the fight between the same couples got even closer in Shanghai: 220.14 versus 219.79. But this time it is perfectly quantified in all the different Programme Components!" (http://www.worlddancesport.org/Blog/Live_From/Shanghai,_CHN/New_IDSF_Judging_System-504)

 

[spikydurian]

This is an interesting thread which is highly technical but interesting to me. The debates constantly point to whether fs should be classified as 'sport' living within the realms of sporting definition or 'art' where emotions subjectivity and connection rule. May I, a layman and non-skater ask a question from the list of experts on this thread? (I vaguely remember reading a thread related to fs as sport in another forum but nevertheless will pose my question differently).
Why don't the figure skating federations take fs OUT OF THE OLYMPICS ie. sports realm so that fs do not have to conform to the 'sporting rules' ie. turn fs into an artistic (like ballet and dance)? Certainly to the ordinary folks, watching a dance competition is more fun than a sports competition which tends to be very technical and in this case, fs is a compeition where many skills are displayed in that few minutes unlike gymnastics whereby it is an element at a time.
Over to you... experts.

 

[skatinginbc]

I think the confusion between quality and quantity and between judging and measuring is rampant not only among lay people but also in the ranks of experts who study these things....No, no, no! Carlos is well qualified, Diego is better qualified, Maria is best qualified.

I was trained to quantify qualities. For instance, approval voting is seen as a variant of range voting with only two levels: approved (1) and disapproved (0), or yes (1) and no (0) as in the preliminary stages of the Chopin International Competition. Charlos is well qualified (+1), Diego is better qualified (+3), Candidate X is better qualified than Charlos but worse qualified than Diego (+2), Maria is best qualified (+4), and I am not qualified at all (0). The numbers in the above examples are NOT treated as interval scales since equal interval is not warranted.

Really? (regarding Beauty Pageant ==> Measurement Theory)

The top five finishers are ranked by a Borda count. So, yes, the prize winners of Miss America are the products of a preferential voting.