Replace List<Integer> with faster IntArray.

These were uncovered by the recently added EfficientCollections
Error Prone detector, and several of them are in hot codepaths.

Bug: 155703208
Test: make
Change-Id: I4cff2d6b846edc9a7482d21a803117132cbcdadf