It would be very financially irresponsible to put your child into skating if you have to go into debt to do so. There are sacrifices that you can make to make it work, but those sacrifices just aren't possible or realistic for every family. And that's okay!
Maybe they don't come from "affluent" backgrounds, but I think the vast majority come from families with above-average/above-median incomes. The average household income for a US family is $59,039 (in 2016). How far that money goes all depends on cost of living, etc., - but let's also remember that the best places for pursuing skating are also pretty high cost-of-living places. Maybe US ice dance has been able to do so well in part because Detroit, which has been the center of US ice dance for over a decade, is low cost of living? Just an interesting observation I only now realized.
I think the men have been more successful than ladies because they are able to develop more slowly to get to the top. They don't have to make skating a full-time pursuit at so early of an age as the ladies do to be successful. I think American parents are more likely to "invest" in their promising preteen than their promising 8-year old.
And I really, really don't think we should emulate gymnastics. Sending kids away from their parents probably makes them more likely to be targets of abuse.