Fix UB in ResourceTable::stringToInt.

Was here because UBsan found integer overflow in the parsing for hex
numbers, since hex numbers here are actually unsigned but assigned to
a signed integer.

Also fixes a number of missing error conditions.

Change-Id: Iaea576daedfc6c75521cde02de3fe9dd0198a3b7