AFAIK Kozuka and Chan are both always listed as either 5'6" or 5'7" so they are both small, but to me Kozuka actually gives the appearance of being taller than Patick because he is more slight in build and has longer limbs. In regular terms you could consider them dwarf like, but in the skating world they are pretty average. Rippon, Brezina, Lambiel are all around 5'7" (Plushy too....I don't believe he's 5'10") as were Orser and Buttle. Takahashi is only 5'5" and skaters like Oda and Sawyer are even shorter. Abbott, Verner, Joubert may look tall but they are only around 5'10", so average by normal standards. Evan at 6'2" is the only top skater I can think of that would be considered tall in the real world, and he towers above skaters like Joubert and Abbott who would otherwise look pretty tall out there. So essentially, in the skating world, Chan and Kozuka are not dwarf like by any means, and are actually probably the ideal height/size for a man in skating.
Lambiel is taller than Brezina he´s almost 5´9 (1,75 cm), Plushenko and Joubert have around 5´10 (1,78 and 1,79), Verner is taller than Joubert (1,80 cm I would say), Abbott too, Lysacek is definitelty the tallest guy there.