Actually, there is almost no possibility that there will be spectators, or that there will be any foreign visitors not "part" of the Olympics.
Essentially, they are talking a "bubble" (huge) of Olympic competitors and staff. However, there is a valid concern that the virus could spread via Japanese "support" staff - the cooks, cleaners, folks running the mechanicals, etc. etc.
Its all about the money. That said, if it could be run in the way most generally outlined, I would not think it was horrible to hold the Olympics. If they think spectators are a good idea, don't have vaccine requirements for athletes, allow folks from all over the world to fly in and stay in hotels - well, then I do not support holding the Olympics.