I agree with this. I was thinking mostly of events like Worlds. Mao Asada's goal in Los Angeles in March is to win the World Championship. I do not think she is so much interested in season's bests, whether she got her step sequence up to a level four instead of a level three, or whether the Bulgarian judge gave her 7.50 instead of 7.75 in choreography. (As Michelle Kwan famously said, I don't mind a 5/4 split, as long as the 5 are for me! )
Joannie Rochette's goal is to make the podium (third place ordinal or higher.)
At their nationals, their goal is to win the event, but that's not much challenge if they skate reasonably well. Straight 1s would tell them nothing. A low Transitions score, an edge call or downgrade on a jump, a lower level than expected on a spin or step sequence -- that gives a lot more useful information about what they need to fix before they go up against each other.
The two American ladies' goal is to get 6th and 7th, or 5th and 8th, or 4th and 9th. The opportuntity to get feedback from the judges about what they need to work on next year I think is farther down on their list of priorities.
Suppose a skater lands 5 triples and finishes 9th behind another skater who only completes 4. Wouldn't she want to know why? Even if she still thinks she should have placed higher, at least she wouldn't have speculate whether "the judges" didn't like her risky music choice, for example. She could see where she lost points on elements and she could also see how many judges preferred her choreography and interpretation and how many preferred those aspects of the skater ahead of her.
At Worlds, that 9th-place skater's ordinals might very well range from 15th to 4th. She'd feel good about the higher ones, dismiss the lower ones, and still all she'd know is that the judges disagreed on how she stacked up against a bunch of other skaters, with no idea why.
She might even have won a head-to-head comparison against the skater who ended up 8th and have to delve into the intricacies of the majority or OBO calculations to figure out why the official results didn't come out that way. And the reasons would be more about math than about skating. Some skaters and fans are interested in those details, some aren't. Casual viewers generally aren't.
No matter what the scoring system, the skater is going to get a placement. If the most important thing is the placement and the skater isn't much interested in how it was determined, regardless of scoring system she won't pursue the details and regardless of scoring system she'll know where the mysterious system placed her.
But very reluctantly, I have to agree with you that the judging system ought to be primarily for "Category 5" (skaters, parents and coaches.) I say reluctantly because then we do not have the right to complain, or to be surprised, when Categories one though four gradually lose interest and drift away.
Category 4 was lower-level skaters. Adult skaters or kids who never intended to be more than recreational skaters may drift away from watching elite skating if the scoring system bores or confuses them and is irrelevant to their own skating, but most would only drift away from going to the rink if they're not having fun on the ice any more. I've known plenty of skaters who put in their lesson and practice time every week but couldn't name the current world medalists -- under both scoring systems. If they watch skating on TV, they may be more interested in appreciating the skill level and interesting moves of the elite skaters rather than in following specific skaters or competitive results.
The kids with some talent who debate whether to continue to the higher levels may be influenced by the kinds of skating they see at those higher levels or how they perceive the scoring. Usually money or desire for a normal lifestyle and schedule would be the biggest reasons not to train toward those levels. Injuries or frustration that the technical requirements are getting too challenging at the higher levels would be other reasons to back off. In various eras, the forbidding technical demands might have been school figures, triple jumps, or complicated spins and steps.