We consider the Kelvin-Voigt model for viscoelasticity and prove propagation of $H^1$-regularity for the deformation gradient of weak solutions in two and three dimensions assuming that the stored energy satisfies the Andrews-Ball condition, in particular allowing for a non-monotone stress. By contrast, a counterexample indicates that for non-monotone stress-strain relations (even in 1-d) initial oscillations of the strain lead to solutions with sustained oscllations. In addition, in two space dimensions, we prove that the weak solutions with deformation gradient in $H^1$ are in fact unique, providing a striking analogy to the 2D Euler equations with bounded vorticity.