Fix layout oscillation at the root of a few bugs.

Note: The CL which introduced this bug was Ic9cd89e899a03010a523bee4f1ef72fbfbd8b43f

Test: visual, manual
Test: Use TalkBack to validate that the size of the touch target is still >48dp tall.
Test: Use an app (e.g. LINE) which only posts the most recent message in an update to send 2+ messages to the device (to make the unread number visible), and validate that 1) the text is visible, and 2) there are not thousands of View errors spamming adb.
Fixes: 182474419
Change-Id: Ia23113f5903fe2c508fc2dbad99160471dcec5fd