Do not throw exception on task vanished

Becasue ag/22162082 make children task info might remove on task
info changed, it might cause we throw exception on task vanished
due to the children task remove from list early. We should remove
it.

Fix: 275022724
Test: manual
Test: pass existing tests
Change-Id: I92f034564d89fef4cf9af4172d64db7630adee42