Wait before computing contact affinity

Instead of assuming we'll have an answer as soon as the handler
gets around to handling the rankingreconsideration.

Change-Id: Id8f2147f59abf6435d869b9ffcc5cf2c2b52f0f7
Fixes: 73010730
Test: cts-verifier