You Be the Judge

[skatinginbc]

If what you're measuring is primarily based on personal preference or qualitative perception of the same skills, then pure ranking makes sense.

It is not always true. As a matter of fact, it is usually not true--more often untrue than true. Most of the performance-based classroom assignments (e.g., research paper) and standardized performance tests such as GRE Analytical Writing and RCM Piano Exams are scored against established criterion, rather than against other participants of the test. International Chopin Piano Competition, for instance, employs criterion-based scoring by the jurors despite what is being measured is mainly "personal preference or qualitative perception of the same skills".

[gkelly]

It is not always true. As a matter of fact, it is usually not true--more often untrue than true. Most of the performance-based classroom assignments (e.g., research paper) and standardized performance tests such as GRE Analytical Writing and RCM Piano Exams are scored against established criterion, rather than against other participants of the test. International Chopin Piano Competition, for instance, employs criterion-based scoring by the jurors despite what is being measured is mainly "personal preference or qualitative perception of the same skills".

So is this an argument in favor of the "levels" now given to segments of the set pattern portions of the current short dance?

[skatinginbc]

If you asked a judge "look at these two performances; how would you score them in relations to each other?", then she would have to ask "under what scoring system?" before she could answer the question.

I chose the Japanese Nationals for a reason. The SP performances are to screen out any invalid scoring methods. Whatever the scoring method a judge (poster) uses results in a ranking order different from the official result is an invalid one. Say, if several tests are designed to separate the native-English speakers from the non-native. Theoretically a native-speaker should pass all those tests. If one test fails the native-speaker, the validity of that test is severely in doubt.

So is this an argument in favor of the "levels" now given to segments of the set pattern portions of the current short dance?

I don't follow ice dance. What I said is an argument in favor of scoring against a set of well-defined criterion. It not only complies with the spirit of sport (Most, if not all sports are scored against criterion) but also makes most sense in terms of measurement practices (Performance-based assessments are usually criterion-based).

[Mathman]

The SPs performances are to screen out any invalid scoring methods. Whatever the scoring method a judge (poster) uses results in a ranking order different from the official result is an invalid one.

Now I am confused about the goal of this exercise.

Are you saying that if my ranking in the short program is different from the official outcome, then I am a non-CoP speaker?

Or are you saying that if I can establish my CoP-speaking credentials independently, but i still get it wrong, then the validity of the CoP to order the skaters correctly is compromised?

Edited to add: Wait, OK, I got it. We are studying [i]only[//] the various ways to put together the results from the short and long programs, not the ranking of each program separately. Is that it?

[skatinginbc]

Are you saying that if my ranking in the short program is different from the official outcome, then I am a non-CoP speaker?

No. I meant: If that is the case, I will secretly discard your data. The differences among their SP performances are so obvious that any valid scoring system should produce the same result.

[Mathman]

What I said is an argument in favor of scoring against a set of well-defined criterion. It not only complies with the spirit of sport (Most, if not all sports are scored against criterion) but also makes most sense in terms of measurement practices (Performance-based assessments are usually criterion-based).

Desirable as this might be, I think the counter-argument is that it is not possible to achieve this in figure skating.

Indeed, we push too hard in this direction only at mortal peril. What would it profit the sport of figure skating to gain the most comprehensive and definitive list of criteria, but lose its soul.

The first words of the Tao Te Ching are (as I am sure you know , "The Tao that can be written is not the true Tao."

In other words, any attempt to confine something within the bounds of syntax and grammar automatically leaves something out -- perhaps the most important part. (In mathematics, this statement can actually be given a formal proof -- it is Goedel's Incompleteness Theorem. [A big private to Seniorita.])

The translation of the Tao te Ching into skating parlance would go something like this: Any attempt to reduce to measurement the true Kwanliness of a skating performance is doomed before we begin.

[skatinginbc]

The first words of the Tao Te Ching are (as I am sure you know , "The Tao that can be written is not the true Tao."

"道可道，非常道，名可名，非常名。無名，天地之始，有名，萬物之母". The literal translation of its first sentence: The Tao (= The way of all things) that can be described is not the universal, eternal Tao. A name that can be assigned is not the universal, eternal name. Nameless is what it was when the sky and the earth first formed. Naming is the mother of all things (= By naming a particular thing, we thus recognize its existence. For instance, English contains so many words for different colors. Without those color labels in Chinese, I didn't grow up with the concept of a fine gradation of color like English speakers do).

Without the 5000 words Laozi used to describe Tao, most of us would probably not even know the existence of such thing called "Tao". Some of us may sense that something is out there. What is it? Nobody can tell. The importance of "有名，萬物之母" (Naming is the mother of all things) shall not be underestimated. The category of "Presentation" in CoP is so broad that different judges may focus on different aspects and overlook others. By naming, the existence of "Transition" becomes obvious so much so that no one can easily ignore.

It is the "Golden Middle" or fine balance between naming (describing) and not naming (not describing) we are searching for. I'm against a strict count of successful features in level rating because it defines skills in such great detail that the skills (Tao) named are no longer the universal skills (Tao). Yet, at the same time, I'm against a category so broad that is basically useless in defining the skills (Tao).

[dorispulaski]

Would it help if COP referenced this book:

http://www.amazon.com/Tao-Pooh-publi...0378814&sr=1-3

[skatinginbc]

Would it help if COP referenced this book:

http://www.amazon.com/Tao-Pooh-publi...0378814&sr=1-3

"The Tao of Pooh!... in which it is revealed that one of the world's great Taoist masters isn't Chinese..."

A bit off topic: The great Taoist master Laozi was likely not a Han Chinese. Laozi was an honorific title for the founder of Taoism Li Er 李耳, which means "tiger" in several minority languages (揚雄方言: 虎…江, 淮, 南楚之間謂之李耳). Due to possible language infusion, we cannot tell for certain if Li Er's native language was Austro-Asiatic or a branch of Sino-Tibetan such as the Tujia language. Regardless, he was very likely an ethnic minority.

[skatinginbc]

My conclusion from my exploratory "Be the Judge" study:

CoP has been criticized for its disproportional emphasis on technical elements. The statistics based on 2011 Worlds data show that TESs contributed most to a skater's total score and to the variances among them (e.g., men's SP: TES Mean = 37.18, Standard Deviation (SD) = 4.96, while PCS mean = 33.83, SD = 3.91; men's LP: TES mean = 73.06, SD = 11.42, while PCS mean = 69.73, SD = 9.46). This scoring practice contradicts the findings of my exploratory study, in which expert opinions (albeit limited due to insufficient participants) suggested a heavier weight for PCS. It is therefore a legitimate concern about CoP's validity in terms of its relative weight on each category.

[gkelly]

Have you looked at other disciplines at 2011 Worlds, or at other events? Maybe next week you could do 2012 Junior Worlds.

One discipline from one competition probably isn't enough of a sample to draw any conclusion one way or another. I wouldn't be surprised if the effects of TES vs. PCS have different trends for men vs. women or juniors vs. seniors.

[Mathman]

...the findings of my exploratory study, in which expert opinions (albeit limited due to insufficient participants)...

That is, due to an insignificant number of participants.

[skatinginbc]

That is, due to an insignificant number of participants.

Bad English!!! (English is bad, not me ).

Have you looked at other disciplines at 2011 Worlds, or at other events? Maybe next week you could do 2012 Junior Worlds. One discipline from one competition probably isn't enough of a sample to draw any conclusion one way or another. I wouldn't be surprised if the effects of TES vs. PCS have different trends for men vs. women or juniors vs. seniors.

I would not be surprised either. I imagine that in Juniors' or ladies' competitions TES and PCS would be more balanced in weights. Please keep in mind that the performance videos I chose were of elite male skaters. Say, if a panel of experts review the footage and mostly agree that PCS should carry more weight, it is the population of elite male skaters that they are talking about, not the ladies, nor the juniors. Is CoP a valid method in ranking the said population? That's a question needed to be answered by further study (Note: The exploratory study was to identify areas for further research. Due to "insignificant number of participants", I can only identify one question or concern worth further examination.)

[drivingmissdaisy]

The statistics based on 2011 Worlds data show that TESs contributed most to a skater's total score and to the variances among them (e.g., men's SP: TES Mean = 37.18, Standard Deviation (SD) = 4.96, while PCS mean = 33.83, SD = 3.91; men's LP: TES mean = 73.06, SD = 11.42, while PCS mean = 69.73, SD = 9.46).

This is the way it should be. It means that what the skater's actually put out there on the ice matters more than reputation (which PCS tends to be).

However, I don't agree with your conclusions fully. Just because TES Mean > PCS Mean doesn't contradict a heavier weight for PCS. If I give an exam where you get 60 points for writing your name and then you do 40 questions worth 40 points total, the name writing mean would be 60, and the answer questions mean could be, say, 32. The questions are more outcome determinative but the mean score for writing your name was higher. So mean should not factor in here at all.

Standard deviation matters though But still, I'm not sure exactly how much. First, there is a ceiling on PCS while there isn't on TES. More importantly, the SD for an individual skater's scores between competitions will be greater for TES than PCS because a bombed performances gets you hit harder in TES. So a few great skates and a few bombs are going to throw the TES SD off, where they might not as much in PCS.

[skatinginbc]

Have you looked at other events?

2011 GPF (men)
Short TES: Mean = 40.65, SD = 3.35; PCS: Mean: 39.96, SD = 2.90
Long TES: Mean = 82.15, SD = 7.18; PCS: Mean: 81.68, SD = 4.94

the SD for an individual skater's scores between competitions will be greater for TES than PCS because a bombed performances gets you hit harder in TES.

Is PCS a stabler trait that should result in a smaller variance?
After excluding ""bombed" performances or those with fall(s) from analysis, the 2011 Worlds LP results would be: TES mean 76.38 > PCS mean 71.6 (significant), and TES SD 10.49 > PCS SD 9.07 (insignificant). Or if we count the mandatory deduction as part of PCS, the results would be: TES mean 73.06 > PCS mean 69.15 (significant), and TES SD 11.42 > PCS SD 9.72 (insignificant).

If we look at a small sample (e.g, 2011 GPF), the effect is clear:
After excluding ""bombed" performances, the LP results would be: TES mean 86.38 > PCS mean 82.08, and TES SD 1.30 < PCS SD 3.21. Or if we count the mandatory deduction as part of PCS, the results would be: TES mean 82.15 > PCS mean 81.01, and TES SD 7.18 > PCS SD 4.99.

Indeed, the PCS seems quite stable, but should it be so?

BTW: Javier FERNANDEZ improved his skating skills by hook or by crook less than a year from 6.68 at 2011 Worlds (PCS = 65.78) to 8.21 at 2011 GPF (PCS = 81.40). One can add transitions, hire a better choreographer, and with a fitting program suddenly have a better presentation and interpretation skills. Skating skills, however, are in my opinion a rather stable trait, which requires years of hard work, or am I wrong?