In the U.S. it's money. All money.
Rinks cost money to build and maintain, and are either publicly-funded through taxes or privately-funded by investors. Once built, the rinks have to attract a clientele willing and able to pay for their ice sport of choice (hockey, figure skating, broomball, curling, and possibly intro to speed skating (very limited).
Of these sports, figure skating is one of the most expensive, if not THE most expensive. 30 years ago, my husband and I were paying around $24,000 (twenty four thousand) dollars a year to keep our daughters involved with figure skating , and they were only doing 1 freestyle a day, five days a week, along with an elite (at that time, but not now--team has disbanded) synchro team involvement. And of course, "elite synchro" is hardly the money-pit that a singles skater falls into every day, week, and year.
And although both of our daughters were able to find decent-paying coaching jobs in college, it was hardly enough to cover their college expenses--the coaching income mainly paid for their good times, clothing, cars, gas, etc. in college, and since they were at the rink much of their spare time, their "good times" was not often the traditional college kid idea of good times (which, to put it bluntly for non-Americans reading this, means parties and alcohol!).
Many parents simply don't have the income to pay for this sport! One-half of all marriages in the U.S. end in divorce (I don't know what the stats are for same-sex married couples, but I suspect it's probably not as high, but still high). This means single parents, which often (but not always) means a decrease in income and even more important, a loss of spare time to spend getting kids to and from extracurricular activities.
If a child does appear to have some athletic ability, parents will often encourage the child to get involved with a sport that can lead to a college scholarship. In the U.S., many colleges offer full-rides (all expenses paid) to athletes, and there are several divisions of NCAA. Do you know what sport in the U.S. garners the MOST college scholarships at this time? ---hint, it's NOT figure skating! It's BOWLING!!! Yes, that humble sport of Fred Flintstone and Barney Rubble! You can still bowl a game in the U.S. for under $5, and regular bowlers pay even less! And a bowling ball, shoes, and jersey costs a whole lot less than figure skates and ice time! And bowling has soooooo many sponsors! What figure skating needs is BUDWEISER or MILLER to sponsor the sport--but how likely is this?!!! ha ha ha
Also do not forget that for many years, the U.S. has had a negative population growth, and our birth rate is dropping every year. This year, out of a population of 350 million, only 3.5 million babies were allowed to be born. That means that there simply isn't a big enough field of children to provide a large influx of new skaters. And one of the main reasons for the declining birth rate is finances--parents can't afford to have more than 2 children, and once they have those children, they seek out the cheapest activities for their children that are likely to result in school and college success, job success, physical health/fitness success, and mental/spiritual/life satisfaction and joy. Figure skating is often seen as a sport in which only a FEW children achieve success--I personally think that U.S. Figure Skating should attempt to change this perception, because many skaters never come near winning ANYTHING at any level of the sport, but they can still find great enjoyment and friendship in the sport throughout their lives.
But mainly, it's money. We just don't have enough parents making incomes that can support the rise of elite singles skaters, let alone pairs and dance skaters. Heck, even our synchro teams have been off the World medal podium for a few years now (at the Senior level), and who know what will happen after this pandemic year of very limited opportunities for synchronized skating teams to practice.
I hope this post is helpful as people think about the excellent question posed by the OP.