The defenders had a bunch of things going for them. First, they had nowhere to run. This is the situation Sun Zu called Desperate or Death Ground. "死地則戰" or "on desperate ground, fight". A shaky army, such as made of civilian militia, may fight ferociously if they have no other hope. This is part of Theoden's gambit of taking his people to Helm's Deep. Prior to going to Helm's Deep, they are on "dispersive ground"; they may run to defend their individual homes, or think they can scatter and hide. Once at Helm's Deep (desperate ground) they cannot run, and if they surrender they will be slaughtered. They must fight.
The defenders were behind extremely good fortifications and they were very old. Old things, in Tolkien's world, are much better than new things.
The defenders knew the terrain very well. While Saruman might know the terrain generally, his armies would not.
The defenders had time to set up defenses and traps.
There was just one way in, and it was narrow, creating a bottleneck which greatly reduces the advantage of the attacker's numbers and greatly concentrates the defenders. And it was open, creating a killing ground. This is Sun Tzu's "hemmed-in ground" where "a small number of [defenders] would suffice to crush a large body of [attackers]".
The attackers had disadvantages. The obvious disadvantage was attacking an extremely well built fixed fortification down a narrow bottleneck.
They had to march a very long way with siege equipment and immediately enter battle. Despite Tolkien's depictions of huge feats of endurance, these are not the heroes. Many a battle has been lost by an army showing up exhausted.
The attackers were divided and contentious, held together by a hatred of the Rohirrim, a love of plunder and Saruman's will. Orcs of various tribes and varieties, trolls, humans and who knows what else. "Give me allies to fight against", attributed to Napoleon, meaning allied armies will prove quarrelsome and poorly coordinated and apt to splinter when the going got tough.
This leads to further problems for Saruman. The bottleneck could have been used to their advantage, they could have blockaded the fortress and starved the defenders out. Instead, they attack in haste. Tactically, it is a mistake to make a frontal assault on a strong defensive position. Strategically, Saruman had little choice; it is unlikely he could have held his army together for a boring siege of many months with little fighting or plunder.
Morale-wise, the attackers were in the opposite situation of the defenders. They were in strange territory. They were fighting for conquest and plunder, not defending their homes. They had a long, vulnerable supply line behind them. They had the option to simply walk away from the fight (until the Huorns show up).
Finally, Saruman may be The Wise, but that does not make him a great commander. He is an amateur and makes amateur mistakes. He has no trained command staff to advise him and point out flaws in his plans, instead he has sycophants like Wormtongue and obedient servants who would not dare question him. He is not a military leader, and would not have the experience to successfully lead troops into pitched battles. He has disdain for his soldier's abilities, confidence in his superior numbers, and will happily throw them away rather than use good tactics. Saruman continuously demonstrates contempt for his enemies and arrogance in his superiority.
I may be mistaken on this as it's been a while since I've read the books, Saruman commits a classic blunder and is absent from the battle. He is not there to exert his will on his squabbling armies and shore up their morale. He is not there to personally observe the battle moment to moment and make the necessary small adjustments. Many a battle has been lost by absentee commanders interfering using out of date or incorrect information.
As it turns out, the reckless, frontal assault does its job. Rather than a weeks long slog, the walls are breached quickly (I don't recall how many days) and the defenders are forced to their last bastion and must use their last reserve. Theoden's ride is a desperate last attempt to push the attackers back. Even if successful, it would have left them exposed on all sides and slaughtered. With their leader dead, their best troops dead, their defenses breached, the battle would have been lost.
Two things save the day. Saruman's army is surrounded by the Huorns. No army likes to be surrounded in the open, never knowing which way the attack may come. They like it even less if they're surrounded by an army of creepy, vengeful trees. This shakes their morale. They've already taken great casualties. Some will question why they are here.
The second is Gandalf's attack on their exposed flank at just the right moment. The attackers are entirely oriented toward Helm's Deep, and are totally exposed. This is a massive failure of scouting on Saruman's part, a classic mistake of inexperienced or overconfident generals who think the enemy is incapable of a counter attack.
In warfare, to win a battle you do not need to kill the enemy, you only need to break their morale. The attackers are surrounded deep in enemy territory. They are divided in their purpose. They have taken huge casualties. The defenders they thought were on their knees have now counter attacked. And now a fresh army appears on their flank. Once a few start to run, the rest will go quickly. Their morale cracks and they flee.
Many parallels can be drawn to The Battle Of Stalingrad. This featured a superior army attacking deep into enemy territory lead by an interfering and overconfident amateur (Hitler) to attempt a rapid assault of a heavily defended position (Stalingrad) for political rather than military reasons. The defenders cannot retreat, they have their backs to the River Volga and there are no further natural barriers. There is nothing else to stop the Germans. The day is saved by Operation Uranus, the massive Soviet attack on the weak German flanks; an attack Hitler (though not his generals) thought the Soviets incapable of. The attackers become surrounded and destroyed.
Another is the defense of the Mannerheim Line during the Finnish/Soviet Winter War. The tiny Finnish army, made up of mostly reservists but with a very professional core, excellent leaders, and good defensive positions, holds off an enormous and lavishly equipped Soviet army of ill-trained soldiers and incompetent and unimaginative commanders who are forced to fight in bad terrain and bottlenecks. Unfortunately for the Finns, Gandalf does not appear, and there is no flank attack to save the day.
