Add -lm to the default libs for Linux & Darwin

libm is a default library for device builds, so default it for host
builds as well.

Also removes duplicate additions of -ldl, -lpthread, -lm and -lrt.

Test: m host
Change-Id: I8f7e799d48a1f427e48dcfb1d0ccba93c5f9780b