Worse if it comes Friday after lunch.
Never. The only time to worry is when you are called into an office with your boss, and your bosses boss. I've seen cops standing outside to escort the employee out.
If after lunch is the first opportunity to speak with someone, then wait until then to speak with them. Saying, "we need to talk" without saying why, is never a sign of leadership.
I banked personal days for days like this. Monday rolled into Tuesday. Often Wednesday. There was never once where the "urgent" matter couldn't have been handled in an email, or a two minute conversation.