target: check hex2bin result

Now that hex2bin does error checking, on error add debugging error msg.

Changelog v1 (update):
- fixed definition of 'ret'
- hex2bin now returns an int

Signed-off-by: Mimi Zohar <zohar@linux.vnet.ibm.com>
Acked-by: Andy Shevchenko <andy.shevchenko@gmail.com>