Newer dimmer switches work by a computerized circuit: pushing the switch causes a circuit board to relay to a motor to adjust the resistance on the wire, where more resistance translates to a dimmer light, and vice versa. Other models involve applying electric currents through a coil, which moves the resistor. These would clearly be a problem whether you left the light on or not, as you’re activating this circuit regardless of how bright the light currently is.
Older switches work by directly adjusting the resistance, with the switch being on a slider; sliding it one way increases the resistance, and sliding it the other way decreases it. While there might not be an issue of lighting a fire, there are several other potential prohibitions involved with electricity on Yom Tov, and some of them, like Boneh and Minhag, couldn’t be circumvented in the manner you describe, as they’re not permitted on Yom Tov in the first place.