The first episode the White Walker lets one of the guys go. A walker also allowed Sam Tarly to live; I think it was the one Jon killed. I fully believe they let Jon go. Sometimes they choose to kill, sometimes they choose to spread fear. They only attacked because Jon was trying to usher out the wildlings they had been tracking. I suspect they had been keeping tabs on this group, hoping that others would amass there since it certainly did not seem as if 100,000 wildlings were there. Oh, and I think the Darth Frost guy fully appreciates and knows what Jon Snow is. Also don't think they would waste the time to gather what was at most a couple thousand people by attacking the ships, when they can now break through the wall with relative ease in the absence of Snow. I don't read the books but I just think that the walkers will hit the wall/castle black before Jon gets back.