Ensure view is attached to window before removing it

I'm not able to reproduce the bug, but we can avoid calling removeView
if the view isn't attached.

Bug: 180114844
Test: Manual verification of existing behavior
Change-Id: I1173f7d3656aaa10815f07717c09d3a3869d9878