Don't call listeners after remove on the same thread.

It was possible for the listener to be called after removing it on the
same thread; which does not seem desirable and could lead to UI bugs
(e.g calling UI listener after UI is destroyed).

Bug: 226122027
Bug: 226099985
Test: atest CtsSafetyCenterTestCases
Change-Id: I9ce8cb3d433b13397604076a30a413517bc47fc1