Fix race condition when destroying view

Calling ensureView before triggering mRemoveViewRunnable
would halt view re-inflation. Now we make sure that the
view will be removed.

Test: Increased handler delay to simulate race condition.
Change-Id: Ic987f307c268b973058a80ab546da87f43fefb1b
Fixes: 63890571