Ensure activity view is always cleaned up

- remove mInflated flag and just use whether the views are null or not
  as a signal
- unify where views are nulled out / expanded view cleaned up, always
  ensure to clean up expanded view first

Test: manual
Bug: 152535659
Change-Id: I9ebaafe7d678fc7c3af790e007f0b6d51dfeb00b