Do not enforce unlocked state when there is no caller

Do not enforce unlocked state when running on the background thread. User can
be in the stopping state or removed by the time the message is being processed.

Bug: 28471878
Change-Id: I1862849661d93b424a07ea94e80563bea7a94ce5