I actually interpreted it as him saying his opinion doesn't and shouldn't matter to the *skaters*--as in, they're young and need to do what they need to do without regard to what him and others are saying. Actually a rather sweet sentiment, when taken that way.