It's not just the women, either. I read recently that Takahito Mura has lost 1.5kg or something, because he'd "bulked up too much to jump". I remember a video with Patrick Chan talking about how he lost weight and was still "bigger" than pencil-thin Yuzuru and Boyang.
Mm, I remember a couple of seasons ago Max talking about changing up his diet because he wanted a "leaner" look, and some discussion that the PTB at USFS didn't want their men to have such large engines (this was during the 2014-2015 phase where Max was at his bulkiest). It gave me the shudders. He might have been bulky but there is not an ounce of fat on that boy.