We know that she couldn't see them due to the Cloak, but snakes have other senses other than sight. Snakes can sense the vibrations on the ground made by their prey's movements, have infrared sensors that detect body heat and an acute sense of smell. Nagini couldn't sense this. Why?
She was floating above the ground.
The room beyond was dimly lit, but he could see Nagini, swirling and coiling like a serpent underwater, safe in her enchanted, starry sphere, which floated unsupported in mid-air.
Hence she doesn't feel the vibrations made by their movements - because she wasn't on the ground.
The sphere is the reason Nagini didn't sense them using her infrared detectors and sense of smell.
This is the tricky and speculative part. One can only assume that Nagini's enchanted cage also "dulled" her senses. The cage served to "protect" Nagini from outside attacks, it may have served to "prevent" Nagini from smelling or sensing others while inside the cage. Here's a description of the sphere:
"It is the only way, Nagini," he whispered, and he looked round, and there was the great, thick snake, now suspended in mid-air, twisting gracefully within the enchanted, protected space he had made for her, a starry, transparent sphere somewhere between glittering cage and tank.
Also, with Voldemort and Snape being in the same room, she might not have been able to differentiate their smell and the Trio's.
Nagini's emotions were clouded by Voldemort's.
Answered excellently in this question which states that Nagini could sense Voldemort's emotions. As such, Voldemort's control state of fury and anger at Snape affected Nagini as well, and thus she too was focusing only on Snape, and didn't discern the Trio who were there.
Voldemort's tone was musing, calm, but Harry's scar had begun to throb and pulse: pain was building in his forehead and he could feel that controlled sense of fury building inside Voldemort.
The stab of rage felt like a spike driven through Harry's head: he forced his own fist into his mouth to stop himself from crying out in pain.
Nagini felt these emotions too, and hence was too preoccupied in being "angry" with Snape to notice that the Trio were present.
So what's the big diff. between the events of Godric's Hollow and the Shrieking Shack?
In the Shrieking Shack, Nagini was just too preoccupied to notice the Trio, whereas in Godric's Hollow, she could focus all her attention on studying ("smelling") Harry and Hermione.
