Ensure proper lock is held for all removal ops

We need to hold the local lock when invoking any operation which can
cause local callbacks, etc.

Bug: 162715416
Test: manual + presubmit
Change-Id: I4f95b58310c7d1eff00656c02aae6dfb69255cdf