That's an interesting question. I think it's fluctuated at different times. I can't speak for the really early years, but I suspect the guys could get very much older then because they didn't have to train full-time or as strenuously--there were only double jumps. (I'm talking the 1920s and 1930s here). Then during the fifties and sixties, a lot of the men were college guys who didn't stay around much past their early twenties. (I'm going by the medalists I know a bit about from my reading.) John Curry and Toller Cranston were a bit older--I think both were 26 when they medaled (Curry gold, Cranston bronze) in 1976, while Dorothy Hamill was 19, pretty typical for girls. Guys who stayed in to win multiple world championships or medals through the eighties and nineties tended to be early to mid twenties because nobody really hit his stride before 20 or 21. By that time, with all the triples, men needed a lot more muscle behind their skating, I guess. So Boitano, Hamilton, and Browning, who stayed through most of an Olympic cycle, were in their mid twenties when they peaked and then left eligible skating. (Browning had two Olympics because the cycle changed in 94.) Come to think of it, though, both Petrenko and Urmanov were early twenties when they won. Paul Wylie was in his mid-twenties because it took him so long to hit his stride. Yagudin, Kulik, and Plushenko all hit the top a bit earlier. I think they each won the OGM in the early twenties, but I seem to remember that both Yagudin and Plushy won their first world medals as teenagers. I'm not looking stuff up as I speak, so please correct me if I'm wrong.
Maybe now they're getting older again? I can't tell. Chan peaked unusually early, and so did Joubert. I don't think Oda's very old, but it may be that he just looks so young and is so small.
If we factor in guys in pairs and especially ice dancing, the age goes up, I imagine, though Charlie Davis and Scott Moir are very young. I don't know whether Tessa Virtue is the youngest female ice dance gold medalist (Klimova was considerably younger than her partner, for instance), but there's a chance that Scott Moir could be the youngest male one!
Mathman, where are you when we need your expertise?