I am curious, on a conceptual level, what the SP is supposed to serve as its purpose, and why there are certain requirements and restrictions in it (e.g. the required solo double Axel.)
Good question. In a general sense, it's supposed to demonstrate skaters' ability to perform specific skills that are deemed important for their event (singles vs. pairs, men vs. women, junior vs. senior) and to do them on demand, with no do-overs allowed. The specifics, including how specific the restrictions are, have changed over time.
To hit a couple of high points in the conceptual development of the short program:
Originally singles skaters did compulsory figures and freestyle programs. Pairs skaters only did free programs.
Around 1960, a short program was introduced for pairs so they would have to demonstrate some required moves that all pairs would have to do, and not just whatever they were good at.
Meanwhile, there was a movement afoot to increase the importance of freestyle skating and decrease the importance of figures for singles skaters.
In 1972-73, a short program was introduced for singles. It was only worth 20% of the total score at first, but short plus long program together added up to more than the figures.
I believe that double axel was always a required element for both senior men and ladies in the SP, until 1998-99 when men were allowed to do triple axel instead.
The other jumps either rotated specified takeoffs every year or allowed the skaters to choose (at least one jump in the combo has always been the skater's choice). When other specific takeoffs were required in the 1970s and 80s, they were required as double jumps, on the theory that every senior skater should be able to execute them. In the later 1990s and 2000s the jump out of steps rotates for juniors among loop, flip, and lutz, and skaters have the choice to do double or triple.
When the junior SP was introduced, I think at first both junior men and junior ladies had to do single axel, then the men were allowed or required to do double axel, then junior ladies were allowed and then required to do double, and very recently junior men have been allowed to do triple axel as the solo axel.
The spin requirements have also sometimes been very specific and other times offered skaters more options.
The shape of the step sequence was originally specified. That ended in 1988-89, which is also when the ladies' spiral sequence and the second step sequence for men were added.
So the requirements have always been either something that everyone was expected to be able to do, a choice between easier elements everyone could do or harder elements that would separate the top contenders, or free choice.
But there has always been some kind of required axel.
Why is the axel considered so important? I can only guess it's because the forward takeoff is believed to demonstrate skating (edge control) skill more than the other jumps, and not just athleticism.
Anyway, although the restrictiveness of the SP requirements has changed over time, the SP has always been about doing specified elements on demand, with no second chances to try a missed element again and penalties for executing moves not according to the SP requirements. The required elements are chosen to reflect elements that everyone in the field should be able to execute according to the definition, so that skaters could be compared on how well they execute the same moves, or in some cases the same kind of moves with opportunities for the more skilled skaters to show more difficulty within the requirements. Originally it was supposed to be comparing apples to apples -- more recently especially in seniors it might be more like comparing different varieties of apples as skaters choose different triple jumps and different spin variations.
The free program was historically an opportunity for skaters to showcase their own best skills in whatever combination of elements they could fit into the time limit. The only real restrictions were on repeating triple jumps, starting in 1982-83, and later limits on jump combos and sequences. In the mid 1990s there started to be a move toward standardization with recommendations for what would constitute a "well-balanced program," and in the early 2000s there were minimum requirements for each kind of elements (jumps, spins, steps, field moves) with deductions for leaving them out.
Then came the new judging system, and with it much more restrictive limits on how many of each kind of element a skater could include. Some of this is necessary for fairness, so that a good jumper won't rack up points by filling the program with dozens of triple and double jumps and not spend much time on spins or steps, for example. But the result is that the long program content is now almost as regimented as the SP content; it's not really a "free" program any more.
One additional requirement that was added to the long program along with the new system was "an axel-type jump." In the LP, that requirement can be filled by a double axel, a triple axel, a single axel with or without variations, including a one-foot axel (landed on the other foot). The skater can choose whether to fill that requirement with a difficult, high-base-value element, or to use a lower value element and hope to earn +GOE points for quality. Or both. Unlike the short program, there's no penalty for doing a single instead of a double except in the lower base mark.
So again, it seems to be important to the powers-that-be for skaters to demonstrate some mastery of the forward outside jump takeoff. But there's less sense of comparing apples to apples.
Under the current rules, there's a lot less difference in the level of restrictions or requirements between long program and short than there was >20 years ago when SPs were more uniform in their requirements and LPs were more free. The main differences are just in the time limits and number of jump elements. So it's harder to define what the purpose of the short program is. I think it still exists mainly because of tradition and to reward consistency across more than one performance.