No it's been ok for you since 2004 because that's ALL YOU KNOW. And fans like you make intolerable douches! Thanks for the obvious Waq - you're the best!!
One thing about your situation mau, you grew up in the middle of CT, which isn't necessarily Patriots country, especially when a team is a laughing stock (not just bad--a complete and total joke of a franchise). So you wouldn't expect many Pats fans.
And then you moved to Mass, which is the heart of Patriots country, so you'd expect a whole lot more. And you see it through this lens, so naturally it appears there are way more bandwagon fans than for other teams. Your particular geographical circumstances enhances the effect.
Every team that gets good has a LOT of bandwagon jumpers. Look at how Dallas became 'America's Team' back in the day. Realistically, nobody outside of Texas and neighboring areas should give a lick about them, but they were a national brand.
Look at the Yankees of the 1990s. If you traveled around the world, you'd see Yankee hats everywhere. It was only towards the mid 2000s that you started to see some Sox caps showing up.
Heck look at UCONN. The vast majority didn't support prior to when JC stared winning. I'm willing to bet the vast majority didn't even know they had a basketball team.
Do pink hats annoy me? Sometimes. Especially those who no nothing about the game. But I take it as inevitable and a small price of success and enjoyment. Just the nature of the beast.