I was lucky enough to live in Japan when 1998 Nagano Winter Olympics happened. I didn't have any tickets, very expensive and also very hard for foreigners to get them, but I decided to go anyway, post myself in front of the arena and just hope that someone has extra tickets for figure skating. To this day I cannot believe my luck to find not one ticket, but 4, for all my family members from someone whose family couldn't come!
This remains one of my life's highlights, so yes, no matter how much the cost of the tickets, if you have the opportunity to go to Olympics just do it, you would not regret it!