First, to be a pairs skater (even more than any other discipline) you need a very special body type. Many people I thinka re encouraged to make the switch (especially short, tiny ladies) simply because they have this body type and almost totally beyond any other considerations.
Once you've crossed that threshold, you have to be comfortable executing lifts and throws, both of which are much harder than they look. You have to learn to skate in unison with your partner (much much harder than it looks) and you have to have or work to develop some 'connection' with your partner that others can see. You need to be committed not just to skating but also to the partnership.
So I think it is way more nuanced than just 'those who can't jump, pair' and it is really not true that the jumps are harder than other elements that the pairs are expected to execute.