The difficulty with defrosting food outside of a refrigerator is depending on the density and makeup of the item, the outside will always be closer to temperature danger zone than the inside, and in large items, this may exceed a safe time.
Take for instance a large frozen turkey. For argument sake, this could take 24 hours to defrost in a warm kitchen, exposing the outer meat to 22 hours in the temperature danger zone (8 - 63°C). The amount of time food should be in this zone should not exceed 2 hours as by this point, the amount of bacterial reproduction will start to reach poisonous levels. The inside of the turkey may have just defrosted at this point, but unlike the outside, would be theoretically safe to cook.
Compare this to say, a shelled frozen prawn. This will have defrosted entirely within 30 minutes, but if left out on the counter for 24 hours, the entire prawn will be unsafe to eat as both the inside and outside will have been in the danger zone for over 23.5 hours.
So yes, in theory it would be possible to defrost an item by monitoring the internal and external temperatures so they don't exceed those limits, but it would be more luck than judgement as this would require the precise density and thickness of food to work inside the safe temperature/time window. This would be almost impossible to reproduce consistently, due to bones, fat, type of produce etc.
By defrosting in a refrigerator you guarantee that the food will not rise above 5°C at any time. Defrosting on the counter is a bit like Russian Roulette, you might get away with it, you might not.