You might be interested in reading this question: In ASoIaF what is the difference between a wight and an Other? The creature that was killed in episode 1, season 3 was a wight, not an Other. The wights are slow and clumsy and easily killed. Well... more easily killed than Others. They are quite resilient compared to humans, but susceptible to fire.
However, on a personal note, that's the first time I've seen a wight using a weapon. That does not occur anywhere in the books that I can remember. They only use their hands. So I can understand the confusion.
Also, that whole scene is a cluster fudge. It's a snow storm, and people go bare-headed. (Yeah, you can tell people from sunny California wrote this script.) Ghost is there, and not with Jon. The wight used an axe. Sam didn't send off the birds (in the books he sent off all the birds). So, take it with a grain of salt, perhaps.
Why are they afraid of white walkers? I don't know that they are. Where did you get that impression? I think actually that most people of Westeros consider the Others mythical creatures that don't really exist. The white walkers have not been seen for thousands of years, and people have forgotten.
That is not to say, of course, that people do not fear them, because they appear in mythology and "ghost stories", such as those told by Old Nan from Winterfell. But of course, grown-ups don't believe in ghosts, so there you have it.
Also... (mild spoiler season 3, episode 2?)
The wildlings are not thriving. They are fleeing from the army of undead.