Page 3 of 4 FirstFirst 1 2 3 4 LastLast
Results 31 to 45 of 56

Thread: ISU experiment: dividing tasks among judges??

  1. #31
    Custom Title Mathman's Avatar
    Join Date
    Jun 2003
    Location
    Detroit, Michigan
    Posts
    28,331
    No, this is terrible. Who would want to be a judge under such a system?

    Let's say you are judge #2. All we want from you is to count Transitions and then to give an opinion on Performance and on Choreography. If you have something to say about the skater's blade to ice skills (SS) or about the way they interpreted the music, or about the quality of the skater's triple Lutz -- shut up, we're not interested. Who do you think you are, judge #8?

    The ISU is fiddling while Rome is burning. This monkey business will not produce better programs. It will not produce better skating. It will not win back disaffected fans. It will not improve their product at the elite level and it cannot be implemented in lower level contests with fewer judges. It will not eliminate cronyism and disruptive politicking within the organization.

  2. #32
    Custom Title Mathman's Avatar
    Join Date
    Jun 2003
    Location
    Detroit, Michigan
    Posts
    28,331
    Quote Originally Posted by Alba View Post
    Mathman, how much time it will take you to come up with an idea to beat this system?
    That is the one good thing about it. It is harder to beat!

  3. #33
    Custom Title
    Join Date
    Apr 2014
    Posts
    1,770
    Mathman, when you break the experiment down like that, it looks less confusing... but infinitely more crazy.

    If they're going to split the judges, just have some judges specialize in GOE and some specialize in PCS.

    Granted, I don't think the ISU is stupid enough to actually implement this judging division, but...

  4. #34
    a cat watching figure skating alebi's Avatar
    Join Date
    Jan 2014
    Location
    Italy
    Posts
    234
    I think we are on the right track, I'm looking forward to see if it works well or the judges'll find it too complicated even if I still believe that 1 tech specialist, 6 GOEs' judges and 6 PCS' judges is the easiest and best way.

    Actually, one thing I would like to see different is the PCS range. While I can clearly identify a -3 from a -2 or a +1 from a +2 on a technical element, I can't understand the difference between a 6.25 and a 6.50 or 7.75 from a 8.25 skater. I've always found this range too big, with too many options inside that honestly don't give you the real idea of a skater. A 1 to 5 range would be more immediate to understand (for example 1 is poor on that element, 2 is sufficient, 3 is decent, 4 is good, 5 is majestic). Or if they want to diversify more the skaters they can still use 1 to 10 but without decimals.

  5. #35
    Custom Title Mathman's Avatar
    Join Date
    Jun 2003
    Location
    Detroit, Michigan
    Posts
    28,331
    Quote Originally Posted by alebi View Post
    I think we are on the right track, I'm looking forward to see if it works well or the judges'll find it too complicated…
    I have the opposite concern, that the judges will find it too simple. If you are an individul judge all you have to do is worry about, say, SS, P&E, and Interpretation, and otherwise you can take a nap for four minutes.

    Actually, one thing I would like to see different is the PCS range. While I can clearly identify a -3 from a -2 or a +1 from a +2 on a technical element, I can't understand the difference between a 6.25 and a 6.50 or 7.75 from a 8.25 skater.
    It's a dilemma. The problem that any scoring system must deal with is that a single scale must be able to accommodate skaters at all levels from beginners to world champions. 6.0 had the same challenge, hence the need for decimals. If you didn't have enough divisions then every elite skater would automatically deserve the highest mark, in comparison to all the skaters in the world.

  6. #36
    Custom Title FSGMT's Avatar
    Join Date
    Sep 2012
    Location
    Italy
    Posts
    2,871
    1 technical specialist, 6 GOE judges and 6 PCS judges would be the best option for me, since there is another problem: training the judges, since "normal" judges (I think) are not trained in all the small details required to assign level/e or UR calls, so creating "technical judges" who evaluate both levels and GOEs would cost a lot...

  7. #37
    a cat watching figure skating alebi's Avatar
    Join Date
    Jan 2014
    Location
    Italy
    Posts
    234
    Quote Originally Posted by Mathman View Post
    I have the opposite concern, that the judges will find it too simple. If you are an individul judge all you have to do is worry about, say, SS, P&E, and Interpretation, and otherwise you can take a nap for four minutes.
    eeeeh?! How can you evaluate a performance or interpretation without watching the whole exhibition?
    What you're saying is a lack of work ethic that, honestly, we can find under any kind of system. BTW watching a judge falling asleep during an exhibition would be epic


    Quote Originally Posted by Mathman View Post
    It's a dilemma. The problem that any scoring system must deal with is that a single scale must be able to accommodate skaters at all levels from beginners to world champions. 6.0 had the same challenge, hence the need for decimals. If you didn't have enough divisions then every elite skater would automatically deserve the highest mark, in comparison to all the skaters in the world.
    I think the only "automatic" mark is SS, which is obvious higher for elite skater than beginners and you don't need to watch the skater closely to evaluate it. But sometimes it happens that, for example, a junior can give a better interpretation than a senior skater. So I don't see any problem if the first get a 7 and the second one get a 5, I don't think it has to do with their category or level. Maybe a 1 to 10 scale is appropriate and perfectly understandable when 6 is the sufficiency, I can also accept a 0.5 mark... but having so many decimals for PCS... it's not like in school when you get a mark based on the number of errors, how can you say a performance was 0.25 better than another one? And it has nothing to do with your level but with your exhibition of that day. So the same skater could get a 5 on a bad day and 8 on a good day. I want to see this kind of diversification, not marks based on categories so that you inevitably need all those decimals

  8. #38
    Custom Title
    Join Date
    Jul 2003
    Posts
    3,894
    Quote Originally Posted by Sandpiper View Post
    Mathman, when you break the experiment down like that, it looks less confusing... but infinitely more crazy.

    If they're going to split the judges, just have some judges specialize in GOE and some specialize in PCS.
    They tried that as an experiment at Nebelhorn a number of years ago but decided not to adopt that split. What I heard was that the judges who were judging GOEs-only were bored.

    It looks like this breakdown is an attempt to give at least some of them something else to do as well. And to give the PCS-only judges less to do so they can be more analytical about the components that they are judging, plus they won't all be tied to the Skating Skills mark since some judges aren't judging Skating Skills at all.

    There's no guarantee this experiment will be deemed successful and its approach officially adopted. And if it is, the breakdown of who does what would have to be flexible since not all competitions would be able to bring in 12 judges plus a tech panel.

    Quote Originally Posted by alebi View Post
    I think we are on the right track, I'm looking forward to see if it works well or the judges'll find it too complicated even if I still believe that 1 tech specialist, 6 GOEs' judges and 6 PCS' judges is the easiest and best way.
    This could be something like what Mathman suggested above.

    The tech specialist could be responsible for identifying how many elements were executed and getting the codes for each one input into the computer. If they make a calling mistake, the data entry operator (if there is one) or the referee or any of the judges could alert them (did you really mean toe loop? it looked like a flip to us!) to fix it at the end of the program.

    Element judges could identify jump errors that reduce the base value (but might not agree, so with this year's rules and five judges, the base value for a flip or lutz with questionable rotation and questionable takeoff edge could be different for every judge).

    These judges would also subtract quality/GOE reductions for other errors and add points for good quality and for difficulty features. How those pluses and minuses would be displayed on the protocol would need to be determined.

    Or keep the levels and 3-person tech panel system already in existence, but split the judges so that some ("technical judges") evaluate elements and Skating Skills and Transitions, and others ("performance judges") judge Performance Execution, Choreography, and Interpretation, with separate appointments and training for those two different roles, although any individual is welcome to pursue both.

    Actually, one thing I would like to see different is the PCS range. While I can clearly identify a -3 from a -2 or a +1 from a +2 on a technical element, I can't understand the difference between a 6.25 and a 6.50 or 7.75 from a 8.25 skater. I've always found this range too big, with too many options inside that honestly don't give you the real idea of a skater. A 1 to 5 range would be more immediate to understand (for example 1 is poor on that element, 2 is sufficient, 3 is decent, 4 is good, 5 is majestic). Or if they want to diversify more the skaters they can still use 1 to 10 but without decimals.
    Quote Originally Posted by Mathman View Post
    It's a dilemma. The problem that any scoring system must deal with is that a single scale must be able to accommodate skaters at all levels from beginners to world champions.
    Yup.

    When starting out learning IJS, it's best to focus on the whole numbers. See the Program Components Overview (linked at the bottom of the page).

    The numbers are defined as
    Outstanding 9-10
    Very Good 8
    Good 7
    Above Average 6
    Average 5
    Fair 4
    Weak 3
    Poor 2
    Very Poor 1
    Extremely Poor <1

    For Skating Skills especially, these correlate with technical skill levels. I think of 10 as being a great performance by an all-time great skater, 0-1 as being a beginner with very little one-foot skating or identifiable edges. So what is Average/5? Seeing how judges have been using the numbers, I think of it as acceptable senior quality, nothing special in senior competition -- but a strong score at lower levels, more exceptional the lower you go.

    If 5 is basic senior quality, then scores lower than that will be more common at lower levels, and scores higher than that will be more common at the international competitions. Judges who are experienced at judging all levels will quickly be able to peg skaters to a general range (e.g., 5s or 7s) on the full scale.

    Fans who only watch seniors, or only elite seniors, will see a narrower range of skills and might think of 5 as a very low score. But watching a lot and analyzing the criteria for the various components could allow even fans who don't know much about skating technique to predict whether a skater they've never seen before (i.e., no reputation judging) will likely earn 6s or 7s or 8s.

    The decimal places allow for finer distinctions among skaters who are basically in the same skill range. That's where judges might start thinking comparatively between skaters in the same event, even though strictly speaking they're not supposed to. They also allow judges to balance out the various criteria on the same component, in case a skater is notably stronger at some criteria and weaker at others.

    At that level of distinction, there aren't really right or wrong answers.

    The other components don't need to be directly tied to the Skating Skills skill level, although some of the criteria (e.g., Difficulty and Quality of Transitions, Carriage under Performance/Execution, Pattern and ice coverage under Choreography, Effortless movement under Interpretation) do rely at least to some degree on control of the technique.

    So each judge needs to develop a mental standard of what is "Weak" or "Average" or "Very Good" performance or interpretation, across the full range of skaters from beginners to world champions. Here's where I think more detailed guidelines and training would be useful. But judges do develop a consensus of what they consider average, etc., by judging with each other, reading protocols to compare their marks to the whole panel, discussing in the judges' room afterward what they liked and didn't like.

    Fans can develop a sense of above-average performance or very good interpretation too, at least at the whole number level. Since these don't rely so much on technical skating knowledge, fans' evaluations in these could be just as valid as judges', especially for fans with performing arts backgrounds. But fitting their evaluations to the 10-point scale means understanding the range of the scale, having a sense of what to expect from non-elite skaters as well as the elites.

    6.0 had the same challenge, hence the need for decimals. If you didn't have enough divisions then every elite skater would automatically deserve the highest mark, in comparison to all the skaters in the world.
    Yes.

    The other reason 6.0 needed decimals, along with tiebreakers, was to have enough room to rank the skaters in large fields. If you only have 10 numbers available for each mark, you would have to use the full range of numbers for every competition regardless of the skill level of the skaters. There would not even be a rough correspondence between scores and skill level -- they would be nothing but place holders. And judges still might run out of numbers/get "boxed in" pretty quickly. Deductions (as in short programs) could not be taken from the actual scores but just considered mentally by each judge when deciding which placeholder scores to put up to rank the skater appropriately.

    (With compulsories, which only received one mark for each figure or for each dance until the early 1990s, it would be impossible to distinguish 30 skaters with only 10 possible numbers.)

    Quote Originally Posted by FSGMT View Post
    1 technical specialist, 6 GOE judges and 6 PCS judges would be the best option for me, since there is another problem: training the judges, since "normal" judges (I think) are not trained in all the small details required to assign level/e or UR calls, so creating "technical judges" who evaluate both levels and GOEs would cost a lot...
    Yes, any major reassignment of duties would require more training.

    If the tech panel as it now exists were to be abolished and individual judges were assigned to independently score difficulty along with quality for each element, I think it might make sense to get rid of the current definition of levels and just allow element judges to give extra points for extra difficulty as well as for extra quality. But they would still need to be retrained.

    Quote Originally Posted by alebi View Post
    I think the only "automatic" mark is SS, which is obvious higher for elite skater than beginners and you don't need to watch the skater closely to evaluate it.
    I don't think you need to watch the skater closely to get a sense of what general range they belong in. But two skaters with similar power and edge depth, for example, may show different levels of mastery of multidirectional skating and one-foot skating. So paying close attention allows judges to reward skaters who actually show more skills even if the first impression is similar.

    But sometimes it happens that, for example, a junior can give a better interpretation than a senior skater. So I don't see any problem if the first get a 7 and the second one get a 5, I don't think it has to do with their category or level.
    Yup.

    Maybe a 1 to 10 scale is appropriate and perfectly understandable when 6 is the sufficiency, I can also accept a 0.5 mark... but having so many decimals for PCS... it's not like in school when you get a mark based on the number of errors, how can you say a performance was 0.25 better than another one?
    As I mentioned, strictly speaking judges aren't supposed to be thinking "I already gave Skater P 6.5 on this component, and Skater Q was a little better, so I'll give her 6.75." Although I wouldn't be surprised if some do think that way.

    What it's really for is for a judge to say to himself something like "Skater Q was better on this component than just Above Average, but not quite Good. Halfway in between? No, I'd say Q was closer to Good in my mind, almost there, but those couple little problems/weaknesses won't let me go all the way to 7 on this score."

    Imagine that you're judging numerous skaters who are all very close in overall ability on this component. How much room should be available to reflect slight differences? If they're all pretty much average and nothing special, should they all earn 5.0? Or can the ones who have average skill and are having a good day earn 5.5 or even 6.0, and the ones having a bad day earn 4.5 or even 4.0?

  9. #39
    Custom Title
    Join Date
    Apr 2014
    Posts
    1,770
    It looks like this breakdown is an attempt to give at least some of them something else to do as well. And to give the PCS-only judges less to do so they can be more analytical about the components that they are judging, plus they won't all be tied to the Skating Skills mark since some judges aren't judging Skating Skills at all.
    Hmm, when you put it that way, it makes sense. I think the current PCS scoring is too tied with skating skills. Someone with good SS immediately gets their P/E and INT scores raised, which is wrong to me. On the other hand, I actually do think PCS should be tied to TES to some degree, since some skaters would never break through if other skaters can make boatloads of mistakes and still win on PCS.

    But: The judges were bored? Well, I guess, if they're bored they won't do a very good job, but... it's their job. You do what you're required to do.

  10. #40
    Custom Title
    Join Date
    Feb 2010
    Posts
    3,634
    Quote Originally Posted by Sandpiper View Post
    Hmm, when you put it that way, it makes sense. I think the current PCS scoring is too tied with skating skills. Someone with good SS immediately gets their P/E and INT scores raised, which is wrong to me. On the other hand, I actually do think PCS should be tied to TES to some degree, since some skaters would never break through if other skaters can make boatloads of mistakes and still win on PCS.
    I kind of agree with you, but another solution could be to increase penalties for errors so that PCS is less likely to save a skater with multiple falls.

    As far as SS being a more important factor in PCS, I definitely agree. On TR, Yuna at her best would score ahead of Mao at her best, even though Mao typically has more intricate programs and performs transitions admirably. However, to me SS makes a skater stand out from the sea of pretty princesses with similar programs so I don't necessarily mind it being rewarded.

  11. #41
    Yulia and Ruslena team forever! Alba's Avatar
    Join Date
    Feb 2014
    Location
    Milan
    Posts
    3,392
    Quote Originally Posted by Mathman View Post
    The ISU is fiddling while Rome is burning. This monkey business will not produce better programs. It will not produce better skating. It will not win back disaffected fans. It will not improve their product at the elite level and it cannot be implemented in lower level contests with fewer judges. It will not eliminate cronyism and disruptive politicking within the organization.

    I absolutely agree.

    We say in Italy: Parole sante.

  12. #42
    Yulia and Ruslena team forever! Alba's Avatar
    Join Date
    Feb 2014
    Location
    Milan
    Posts
    3,392
    Quote Originally Posted by Mathman View Post
    That is the one good thing about it. It is harder to beat!
    But not impossible right? As with all things, it just needs more time to figure out.

  13. #43
    Custom Title
    Join Date
    Jul 2003
    Posts
    3,894
    Quote Originally Posted by drivingmissdaisy View Post
    I kind of agree with you, but another solution could be to increase penalties for errors so that PCS is less likely to save a skater with multiple falls.
    There are other kinds of errors besides falls. Some aren't even obvious at a casual glance, don't disrupt the overall impression of the performance, but do cost a lot of points in TES.

    As far as SS being a more important factor in PCS, I definitely agree. . . . However, to me SS makes a skater stand out from the sea of pretty princesses with similar programs so I don't necessarily mind it being rewarded.
    If any single component should have the most weight, I would want it to be Skating Skills. This is, after all, a skating competition first and foremost.

    But should that be happening because it's the first one listed and therefore the first one most judges score, because some of the criteria for other components presume skating competence to fulfill?

    Or should it be possible to evaluate some or all of the other components completely independently from Skating Skills, but then give them lower factors because they're less important in determining how well each competitor skated?

  14. #44
    Custom Title
    Join Date
    Apr 2014
    Posts
    1,770
    Quote Originally Posted by drivingmissdaisy View Post
    I kind of agree with you, but another solution could be to increase penalties for errors so that PCS is less likely to save a skater with multiple falls.

    As far as SS being a more important factor in PCS, I definitely agree. On TR, Yuna at her best would score ahead of Mao at her best, even though Mao typically has more intricate programs and performs transitions admirably. However, to me SS makes a skater stand out from the sea of pretty princesses with similar programs so I don't necessarily mind it being rewarded.
    I totally agree about increasing penalties for mistakes. But also: What if one skater skates a super easy program, but performs it beautifully? While another skater skates a far tougher program but comes across as "lacking" in PCS areas? I think PCS should take into the account the difficulty of doing transitions, showing off skating skills, emoting, ect. while doing far harder elements. The second skater isn't necessarily worse in PCS areas--it's just harder to do these things as your elements get harder. So I think the second skater should be compared in PCS to those with similar technical content. Within reason, of course. I wouldn't want everyone to turn into Timothy Goebel and still get sky-high PCS.

    I don't mind skating skills being rewarded, but not at the expense of other components. Even those with good skating skills should be encouraged to show off their edges/turns in ways that relate to the music and project to the audience. Right now, it seems a little "Well, if I do this hop or that turn, my SS and TR go up and the rest of the components just follow suit automatically."

  15. #45
    Custom Title Mathman's Avatar
    Join Date
    Jun 2003
    Location
    Detroit, Michigan
    Posts
    28,331
    Quote Originally Posted by gkelly View Post
    Essentially there are three primary tasks in scoring a program:

    *Identifying each of the elements according to the rules in place

    *Evaluating how well each element was performed

    *Evaluating the overall performance as a whole in terms of several broad areas
    In general I would prefer a scoring system that does not artificially separate or compartmentalize these three features. (In this regard I agree with Sandpiper's post above.) A well executed triple flip should avoid errors such as wrong edge take-off and short rotation, should exhibit athleticism as measured by height and ice coverage, should have pleasing embellishments such as a proper air position and well-controled flow on the landing edge, should match the musical phrasing, and should serve as a well-integrated choreographic highlight.

    I would not like to see a system where one judge is assigned to watching the replay for edge calls, another times how long the skater is suspended in the air, a third records the loudness of the accompanying musical crescendo, a fourth evaluates the placement of the jump within the vision of the choreography, and so on.

    To me, a judging system like that deliberately misses the heart of figure skating. It does not honor those marvelous performances in which the whole is greater than the mini-sums of all of its parts of parts.
    Last edited by Mathman; 07-11-2014 at 10:53 AM.

Page 3 of 4 FirstFirst 1 2 3 4 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •