Very true. Some very high profile skaters have missed at least one season in the past couple of seasons, and Virtue/Moir are the only ones who come to mind as not having struggled at least somewhat, for whatever reason. Chan, Mao, Kostner (not saying she wouldn't consider this year's results a success, but it was clear at Worlds she's not currently where she was), V/T, etc. And dance hasn't leapt forward technically the way men's has. Josh isn't as high profile as them, but it just goes to show that what you said is true. Comebacks are hard.
It's easier to come back and start back up where you left off in Ice Dance since there are no big elements like jumps required and Ice Dance hasn't changed at all really since V/M last competed. Getting jumps back is harder to do then any element in Ice Dance especially for a top team like V/M.