Revert "Simplify our build file now we don't support non-neon."

This reverts commit d8fae023.

Reason for revert: DroidMonitor-triggered revert due to breakage https://android-build.googleplex.com/builds/quarterdeck?branch=aosp-master&target=ndk&lkgb=9213008&lkbb=9213057&fkbb=9213057, bug 255439661.

Bug: 255439661
Change-Id: I335a8f671292c1e1f3f6e3ed251a81b695d4ce94