Fatal abort on invalid state

Use mCaches.size() to track item count instead of trying
to do it manually on the side.

Bug: 31856965
Test: manual, minimal. Repro steps for bug are unknown, fix
is speculative as a result
Change-Id: Ic32f27262548a99be95440881420d224ede4db74