QS: Remove listeners only if added

handleDestroy() calls setListening(false) unconditionally. This
causes an error if no listener was never added. Check if we added
a listener and then try to remove it.

Change-Id: Iefe8aacbe87ba1c2c19c9e77cf132fc7c84798f9