When loaded above the yield point ($Y$ in your sketch), ductile material will start to deform plastically. When you stop at some point before the material fails and reduce the load again, the material will relax.
Citing the wikipedia article on plasticity: "Plasticity in a crystal of pure metal is primarily caused by two modes of deformation in the crystal lattice, slip and twinning. Slip is a shear deformation which moves the atoms through many interatomic distances relative to their initial positions. Twinning is the plastic deformation which takes place along two planes due to a set of forces applied to a given metal piece."
Therefore, the relaxation will be elastic. However, since the internal structure has been changed somewhat during plastic deformation, residual stresses will remain inside your specimen. There also is some lasting deformation. When loaded into the opposite direction, the yield limit is reduced. The length of the elastic line is the same for the pre-deformed specimen as for a new specimen (which is $2\tau_Y$).
The underlying principle is the same for torsion and for pure tension. The major difference is that when twisted, only an outer layer of the material will undergo plastic deformation when loaded above the yield point, while the tension probe will undergo plastic deformation in the whole cross section. This is because strain and stress increase linearly with radius in a torsion probe (see here for an example), but are independant on the location in a tension probe.