spi/pl022: remove function cannot exit

The remove function in the PL022 driver cannot abort the remove
function any way, so restructure the code so as not to make that
assumption. Remove will now proceed no matter whether it can
stop the transfer queue or not.

Reported-by: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>