Fixes a concurrent modification crash issue

Made the logic more clear. Instead of removing keys while iterating the map, identified keys to remove prior.

Fixes : b/319369215

Test: Manually tested. Existing test passes.
Change-Id: I5e4531260623e479ff895530d3b613b027c61c66