In an ideal world there would be an unlimited supply of qualified knowledgeable judges from all over the world, including countries with no elite skaters whatsoever, who can get to any location at any time for the same modest costs. If that were the case, avoiding judges' familiarity with or allegiance to specific top competitors could always be a top consideration.
In real life, judges need to gain experience judging elite skaters before they can be deemed qualified enough to judge the big events. They will usually have judged all their own country's top skaters in domestic competitions. Judges from newer federations need experience judging junior and senior B events before qualifying for championship events. Judges in Europe can often drive or take trains to competitions in other European countries, making it easier for them to be invited to judge international events at the beginning of their international judging careers, so they will be more familiar with other European skaters before they reach the elite levels. American and Canadian judges may be familiar not only with their own country's skaters but some of each other's who cross the border for non qualifying competitions, as well as any foreign skaters who train in North America and enter nonqual competitions for practice.
Judges often have relationships with other officials or with coaches of similar age and background dating back to their own skating years.
Some sources of familiarity and affiliation with the skaters they judge can be considered too much of a conflict of interest and reason to disqualify that judge from serving on a panel with that skater. Others are too numerous to be anything but inevitable, and usually too tenuous to make much difference.
Where is it practical to draw the line? E.g., say we all agree that no parent should judge his/her own child, or no tech specialist who is also a coach should serve on a panel of an event in which a current student is competing. But just having judged the same skater a couple times before during the same season? Keeping track of all such connections would be a full-time job in itself.
If the main concern is the high-stakes medal contenders at the big events, maybe the rule could only apply to the Grand Prix and to ISU championships.