I know there will be a large loss with no spectators. The loss without any games would be much more significant. No TV revenue, loss of exposure on special areas of interest videos always shown as fillers during the games. If you have watched the games before I don't need to go further, if you haven't watched before there is no need to have even started this discussion. We need these games, we have given up more than enough already. The talking heads have said the virus will be with us from now on, the question is will you continue to hide or will you come out and reclaim what life you have left? The athletes are coming out to play, they know the risks, let em live.
I agree that they should go on, separated largely from the Japanese population. And, with athletes generally vaccinated, having (reportedly) access to vaccinations, testing protocols, etc. and with the Athlete's Village essentially a sort of bubble, I don't see this as an event likely to be a super-spreader throughout the world and certainly not throughout the US.
That said, you are being (in my opinion) too "one size fits all" in your remarks. There are areas (and Japan is one, apparently, and lots of other nations) where vaccines are not well distributed, rates of infection are still (relatively) high and coming out and reclaiming the life they have left could mean making that life very, very short. There are areas in the US where folks are resuming life as normal at very high risk - it is within their rights to do so, of course, but I wouldn't rush if I was unvaccinated in an area of remaining significant spread. If my wife did not remain at personal high risk (immunosuppressed) I would probably be pretty much back to normal, as it is not an area presenting undo risk to someone vaccinated. Since she can't, I can't.