The answer really is simple: Refi when it is worth it to you. What are your goals: Do you want to reduce the time you owe on a mortgage? This is a great way to do it. Say you got a 30-year at 4.5% a couple of years ago, and two years later find you could get a 20 year at 3.25% (I'm making up the numbers, but they're not far off) for close to the same monthly payment. Assuming the mortgage company is not gouging you with the up-front costs when you refinance (and look those costs over very carefully), this is a no-brainer. You just dropped eight years of largely-interest payments off your mortgage for about the same monthly payment. Watch how much more quickly you accumulate equity in your home this way, too. We did this on our last house and it really helped.
Or say your circumstances have changed and it'd really be helpful to have lower payments. You can refi your 30-year mortgage into a new 30-year with a much better rate and lower payments.
Just keep shopping around. One thing that has happened a lot is that many of the most aggressive advertisers -- Lending Tree and Rocket, for example, will NOT show you what your rate will be until you give them your phone number. This will subject you to at least a few calls each from really aggressive if not obnoxious mortgage salespeople who will keep you on the phone with low-rate come ons that they cannot possibly fulfill.
It used to be you could go to Bankrate.com and put your info in and you'd get a bunch of rates without giving any personal information, but not any more.
The only place I know of that will do this is the company where we got our mortgage, Aimloans.com. You can go to their page, fill out a few lines of information and other than your Zip code, you will have to give them no personal information at all and they will tell you what their rate is for at least a 30 and 15-year mortgage. As I say, I know of no one else that does this, although there probably are others. It will give you an idea of what you can get right now, without the really annoying phone calls.
Also, I believe that right now is NOT the best time to refinance, for a variety of complex reasons having to do with the way the mortgage and bond markets work. Wait a few weeks and the rate will get better, I think.