That's not a Philly problem, that's a drunk sports fan problem.
It's also the difference between the NFL and college.
If you're wearing an away team shirt at a college football game, you're going to be invited to every tailgate you pass. (With the exception of NC State....they seem cranky. At West Virginia, they all but take up a collection to buy you a house near the stadium.)
In the NFL, just leave the jersey at home. Corporate bro who got the tickets from his brother-in-law's insurance broker is there to defend his turf.