Thread: ISU experiment: dividing tasks among judges??

1. 0
Originally Posted by Alba
Sorry everyone. I really have to use italian here:
So, 5 giudici su 7 danno il goe, 4 di questi 5 anche un solo aspetto dei PCS. Gli altri 7 giudicano solo tre PCS. Il tutto fatto in modo che comunque l'atleta abbia 5 valutazioni su ogni PCS.
Did I got that right?

we have a total of 12 judges so

5 giudici valutano i GOES
7 giudici valutano i components

4 dei 5 giudici dei GOE valutano anche una sola voce dei components (differente per ogni giudice)
i 7 giudici dei components valutano 3 voci, non tutte quelle presenti

this is how I got it

2. 0
Originally Posted by Blades of Passion
The ISU continues to take good ideas and then implement them incorrectly.

There should just be 6 technical judges and 6 program component judges. It's so easy to figure out. You only need 1 technical specialist, rather than a panel of 3, because all of the technical judges need to be tech specialists in the first place. They would all vote to determine rotation calls for jumps and they would each look for a different criteria in the footwork sequences.
easy things are always the most difficult to be done

3. 0
Originally Posted by Blades of Passion
It still would be majority vote. 6 tech judges + 1 tech specialist. They would each review the jumps and vote. 4 out of 7 would determine how the call goes.
By the way, not completely right though (I'm in total confusion with this topic ).
The tech specialist will determine the level right? While the tech judges will give the GoE's only. So in the end we still have a problem here.

4. 0
The idea of dividing tasks is good (one person can't give GOEs and at the same time evaluate 5 different PCS categories accurately), but why do we need all these complications, with some judges doing more things than the others? Just make one GOE panel and one PCS panel! (But giving only GOE scores would be extremely boring, yes )

5. 0
If they had six "technical judges" they wouldn't need technical specialists at all. Each of the six judges could separately judge the level, under-rotations, wrong edge, etc., and come up with a total score for each element.

6. 0
Originally Posted by Mathman
If they had six "technical judges" they wouldn't need technical specialists at all. Each of the six judges could separately judge the level, under-rotations, wrong edge, etc., and come up with a total score for each element.

So at least 3 different opinions, if not more, if the level is 3 or 4.
With the tech. specialist at least we have one (3 at max) to blame only, and know who they are.

7. 0
Originally Posted by Alba
So at least 3 different opinions, if not more, if the level is 3 or 4.
To me, that is the nature of a judged sport. If three expert observers think that the skater did enough to earn a level four and three other equally expert observers disagree, what's wrong with averaging those scores?

With the tech. specialist at least we have one (3 at max) to blame only, and know who they are.
Do away with anonymous judging.

8. 0
Originally Posted by FSGMT
The idea of dividing tasks is good (one person can't give GOEs and at the same time evaluate 5 different PCS categories accurately), but why do we need all these complications, with some judges doing more things than the others? Just make one GOE panel and one PCS panel! (But giving only GOE scores would be extremely boring, yes )
I have thought for a long time that a split was the best answer particularly for fairly judging PCS. PCS stays so static through the season regardless of a skater's actual performance and I have always assumed it is because the judges are not able to adequately evaluate it. How does one get a sense of choreography or connection to the music when so busy marking GOE on each element? They have to kind of make do on some of those marks and I think they end up just making assumptions based on knowledge of the skater and his/her previous performances--so the system builds in a reputation bias.

9. 0
Originally Posted by Mathman
To me, that is the nature of a judged sport. If three expert observers think that the skater did enough to earn a level four and three other equally expert observers disagree, what's wrong with averaging those scores?
Nothing wrong, that's the judged sport but it's also now with the GoE's and PCS's and we still complain and blame them.

10. 0
Originally Posted by Mathman
To me, that is the nature of a judged sport. If three expert observers think that the skater did enough to earn a level four and three other equally expert observers disagree, what's wrong with averaging those scores?
Some questions/possible issues:

Would these technical judges both call a level (or underrotation/downgrade) and also give a separate GOE?

It's easy to average GOEs, and it's possible to average levels.

But since underrotations and downgrades -- and as of this year edge calls -- change the base value of the element and possibly the value of the GOEs, it wouldn't work well to average the base values and average the GOEs separately and add/subract the two, as is done now. It would make more sense to take the base value +/- GOE for the element to get total element score from each judge and then average those totals.

So what would be shown on the protocol? E.g., just 3Lz or CCoSP in the element column, and then << e -3 or level 4 +1 along with or without the final element score in each judge's column? That would require wider columns.

Or just the final element score for each judge without publishing how they each called the element or rewarded/penalized in GOE?

What if a judge makes a mistake on identifying the basics of the element, e.g., calling 2A instead of 3A or 3T instead of 3F or CSp instead of FCSp, etc.? Would that judge's score for that element be based on that judge's mistake identification or would there be some mechanism to alert the judge that they probably made a mistake?

What if the skater makes a mistake such that it's unclear exactly what they did/what they should get credit for. E.g., a hop on the landing of a jump that turns into a single loop (no longer scored for juniors and seniors, but could affect whether that jump is considered a jump combination). Or off-balance steps between two jumps: Should it be called the first jump +SEQ and no value for the second jump? Or as two separate jumping passes, which would give credit for the second jump here but could lead to too many jump elements and no credit for the last jump pass in the program? Same with a wide step/recenter between two feet of what was supposed to be a change-foot spin.

In the current system with a technical panel working as a team, for those cases the tech panel can review the element after the program and determine the most appropriate call, and then alert the judges, who will award accordingly.

If each judge makes an independent determination of what the skater actually did, different judges might have significantly different lists of elements, with different numbers of elements. How would the computer or an as yet unidentified official determine that elements 7 and 8 should be considered 2A+SEQ (with the 2T after a step out ignored) and CSp, not 2A and 2T?

If there's no separate technical panel, who would keep track of the number of falls? The element judges? The PCS judges? Both? (What if they disagree?) Or would that responsibility go to the referee? Or get rid of the fall deduction and charge the PCS judges with reflecting falls in their PCS?

Would each judge be responsible for keeping written notes on level features, edge/rotation calls, and errors/+GOE bullet points as well as inputting both element codes and GOEs into the computer in real time? That's going to take their eyes off the skaters for several seconds between elements, which could be a big problem if the next element follows immediately. At least now the tech panel gets a data entry operator to do the data entry for them.

I think each judge would end up missing some of the feature and GOE details they're supposed to be responsible for. Hope that the rest of the panel will catch what each one misses and it'll all average out in the end?

This will be even more true in pairs and dance where each judge with a single set of eyes needs to keep track of both "what?" and "how well?" details of what two skaters are doing. How often will they they be accurate about catching different errors from each partner -- which may affect the element code/base value -- on side-by-side elements?

These technical judges would have little attention to spare for the "Element matches the musical structure" bullet point, so that would rarely be awarded in GOE -- but PCS judges could pay attention and reward it under Choreography and/or Interpretation.

11. 0
I think it would work fine if each of the six technical judges decided for himself what element the skater did, what level to assign, what errors should be deducted for and what positive GOE features should be rewarded. Judges could have access to replay if they were in doubt about whether the skater did a double of a triple Axel, whether the flutz was bad enough to deserve a deduction, etc. Put all these scores together for each judge, then average the totals.

I agree that there is a problem with displaying the results on the protocols, for instance if one judge scored a combination and another scored two separate elements. Maybe the judges should be allowed to confer on the name of the element, in the same manner that the three-person tech panel does now. Fall deductions, costume deductions, music deductions I suppose could be handled by the referee -- but I am not sure we need a separate fall deduction anyway. A fall would negatively impact GOE (and possibly even the name of the element), in addition to detracting from presentation, choreography, and interpretation.

One judge could make a bad mistake, but the tech panel can, too. Mistakes by the tech panel affect the marks of all judges, whereas a mistake by a single judge would tend to get averaged out. In the case of a borderline downgrade, for instance, the either-or choice of the technical panel can wipe out the whole element, whereas if three judges thought the jump was an under-rotated triple and three thought it was an over-rotated double, both opinions would be represented in the average. There could also be a trimming of the mean to mitigate the effect of bad errors in judgement.

I guess my instinct is that if the proposal is to dedicate six of the twelve judges to the examination of each element in isolation, well, tha;'s the job of the technical panel, too. Do we need to put nine people on this task?

12. 0
Essentially there are three primary tasks in scoring a program:

*Identifying each of the elements according to the rules in place

*Evaluating how well each element was performed

*Evaluating the overall performance as a whole in terms of several broad areas

Under 6.0 judging each judge did all of the above and were free to consider as many or as few details as they liked, with no report of what they actually did and did not consider.

IJS has added more officials and broken up the tasks in one way.
This Nebelhorn experiment proposes to keep the same tasks and the same assignments (but several new rules) for the tech panel but vary how many of the judging tasks are assigned to each of the judges.

Some posters in this thread have proposed other ways of dividing the tasks among officials. Still other methods are also possible. We could discuss how various alternatives might work. -- How much to keep certain assumptions that have been part of IJS since the beginning and how much to start from scratch, which the ISU has shown no evidence of doing?

13. 0
Interesting concept, though I'm more concerned about how it would trickle down to club comps.

Though I imagine people will only be happy with it until one day there's a Russian judge on the GOE panel and a Russian skater beats their favourite to win gold - then it'll be all CORRUPTION! EVILLE! THIS IDEA WAS OBVIOUSLY CREATED BY RUSSIA TO CONTINUE THEIR EVILLE!

14. 0
My first thought: What an unnecessarily complicated mess. I don't mind dividing duties, but why do it in such a confusing, haphazard fashion?

15. 0
Originally Posted by Sandpiper
My first thought: What an unnecessarily complicated mess. I don't mind dividing duties, but why do it in such a confusing, haphazard fashion?
I don't know about haphazard -- there is actually a method in their madness.

But madness it is none the less. There has to come a point at which someone says to the ISU, "hold, enough."

Here is the proposal for the tasks of the twelve judges, which will be tested at Nebelhorn.

Judge #1 SS TR PE
Judge#2 TR PE CH
Judge #3 PE CH INT
Judge #4 SS TR CH
Judge #5 SS TR INT
Judge #6 SS PE INT
Judge#7 TR CH INT

Judge #8 GOEs and also SS
Judge #9 GOEs and PE
Judge #10 GOEs and CH
Judge #11 GOEs and INT
Judge #12 Just GOEs

This way each of the five program components is evaluated by five judges, and GOEs are judged by 5 judges also. (Too bad the math worked out so that there was one GOE judge left over with no component to judge -- TR already had five judges. Oh well, nobody's perfect.)

The judges must arrive a day early to attend a seminar, led by the technical specialist, at which the duties of each judge will be assigned and explained.

So this isn't crazy after all -- except that it is.

Page 2 of 4 First 1 2 3 4 Last

Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•