Why not wear foam earplugs? The right earplug, properly used, have a NRR of 33dB, which is really close to the 35 dB rating of your earmuffs. 3dB is considered the minimum human detectable threashold for sound, so it's unlikely that you'd notice any difference even in ideal conditions.
It's the ideal conditions part that typically causes problems with earmuffs. To work properly, ear protection, plugs or muffs, has to block the path of the sound from reaching your eardrum. They achieve this by creating a sealed barrier between the sound and the eardrum.
While slightly more effective, the muffs have three disadvantages in achieving this seal. First, they're sealing against a much larger and flatter surface area and using a fairly weak spring to do so. This means the seal can be disturbed fairly easily and noise can intrude. Second, there are lots of common things that can get between the seal on the muff and the wearer's skin, typically eyeglass temples and hair are the big offenders here. Third, they're much more likely to become dislodged.
I suspect that in your case, it's some combination of those factors coupled with airflow over your head pushing on and disrupting the seal between the muff and your head.
Earplugs on the other hand achieve a good seal much easier and unless you have really hairy ear canals like Yoda, they also don't have problems with foreign object disrupting the seal. The biggest problem with earplugs is in teaching people how to wear them correctly, but it's easy to do. The other big plus is that earplugs are much more discreet than those big yellow Peltor muffs.