Put NaN checks where it is otherwise not expected

We are getting NaN failures during touch events on the PanelView.
I was able to narrow this down to two possible sources, but it is
unclear which is actually causing the problem.

No repro steps are available at this time, so this is the best
short term fix.

Fixes: 152817428
Test: manual
Change-Id: I04ceb2128c54f88289f560f0e5636f3093f405d0