Simplify our build file now we don't support non-neon.

I also tested that adding all the source files to the default set of
sources doesn't break riscv64.

Test: treehugger
Change-Id: Ib3a7faf2d5509eed722e8841293ada650c7dff95