Well, you have never met such people so you may doubt whether they exist at all. I have met them many times, in different walks of life, and in different periods of my - not so short - life so I have no doubts they do in fact exist and I need no more proof of that. This in spite of the fact that skating (
not figure skating) for fun as a form of winter entertainment and a winter pastime is rather popular here...
Now, how numerous they are and how representative of a bigger population their views are, none of us knows without research, right. But it stands equally true for the claim they form an important fraction of non-watching non-fans as for the opposite statement that they are just a tiny insignificant minority of them all. Who knows, who can tell? One thing that can be said for sure based on our
combined experience, is that they happen to exist, even though you have never met them

As for the NFL argument, I am sorry but my intuition is that team sports fandoms and individual sports fandoms have very different dynamics. I never watched NFL in my life, and I assume not many people in my country did, as it is not popular in Europe at all. I suppose, though, the fandom dynamics is more comparable to that of soccer, a highly popular sport here, with huge money involved, and based on teams. Now, these teams or better say, clubs, have been a part of their local environment in many localities for ages, and rooting for them to some extent is a part of the local identity. They are often huge contributors to various events and undertakings in the neighborhood. Individual players and coaches come and go but the team/club stays there and generation after generation roots for them, no matter what. Each match is a huge event drawing in crowds, regardless of the results. Participation alone offers a huge reward in the sense of belonging to a community. Similar for international soccer competitions where people mostly just root for their country at the times of the games, with all the city streets dropping empty and then going wild with joy or tears, no matter what.
That's a very different dynamic from a sport where you mostly root for individual athletes whose careers are relatively short, especially when compared to teams/clubs. Now, if you come from a country which is not a big fed, with scarce or hardly any medal chances, your patriotic motivation is not that strong so you are more picky about whether you follow a sport or not, and what you expect from it. Judging perceived as rigged can be a huge no-no factor in such circumstances, believe me