Look at the II-VI options from a chemical perspective.
For silicon, Group III boron p-doping would be replaced by Group II beryllium doping, and Group V phosphorus n-doping would be replaced by Group VI sulfur doping.
The beryllium would tend to act like a Lewis base that donates a pair of electrons to some nearby silicon atom, forming a stronger and more localized bond than is possible with the odd electron count created by boron doping. Similarly, the sulfur would tend to act like a Lewis acid and form a localized sulfide bond with some nearby silicon atom, versus the odd electron count created by phosphorus.
So, the main reason for picking elements immediately "next door" (off by one electron) is to avoid introducing paired electrons, which on average will bond more strongly and more locally via a Lewis base-acid process.
That said, there's enough complexity going on there that it's still possible that you could get II-VI or maybe even I-VII doping to work in the right situations.