Sounds like it's time for a new mirror

i have had good success in fixing the prob.  the little plastic
circular holder that the swivel 'ball' goes into is most likely
cracked.  remove the complete mirror assy, pull the mirror out of the
holder socket. now put a slim piece of paper 'whatever'  over  the
socket, and push the mirror back into the socket.  reinstall .  has
worked for me..  -gary