The main issue that I can see is that nothing ever happens to the judge who marks in that way. Like a judge who gives -2 for a fall on a jump when the rest have uniformly given -3.
There is actually quite an elaborate procedure for "judging the judges," but, as you say, it is not clear whether or not anything ever happens as a result.
First, as I understand it, the referee holds a meeting of all the judges immediately after the event to go over the scores. I can't find the ISU doicument that talks about the "anomaly" procedure, but it is something like this. For each score on the protocol, if a judge's mark is outside the "corridore" defined by the judges' marks in total, then that score is marked as an "anomaly." I think the corridore is plus or minus three standard deviations.
This is quite a generous spread. On a panel of nine judges, if eight of them give -2 and one judge gives +2, the standard deviation is 1.33 and so the out-of-step judge is only just on the borderline of being outside the corridore. (I assume this "generousity" is deliberate so that judges will not feel intimidated into trying to guess what their colleagues will give out and score accordingly.)
Anyway, IIRC, three anomalies and your name comes up before some sort of ISU judges' oversight committee. The committee reviews the judge's performance for evidence of bias or incompetence. The "accused" is allowed to defend his/her marks and to present evidence (videotapes, etc.) in support of his/her case. If the committee finds just cause, it can "punish" the recalcitrant in various was, like demoting him/her to lower level competitions for a time.
As far as I know, none of this is ever made public, so it is impossible to know whather anything ever comes of this procedure or not.