Don't invalidate list data during layout.

When a FastScroller is enabled, we were invalidating the
list data during layout, which left the UI in a bad state.
Long-presses didn't work properly, and TalkBack touches
were impossible to perform.

Bug: 31160338

Test: Manually tested long pressing, and verified
that TalkBack now works with the TestActivity in ag/1567328.
Change-Id: Iecd2d597259748a3ecbf5573950e565a0e9faa42