tty: serial: altera_uart: Handle pdev->id == -1 in altera_uart_remove

Commit 6b5756f1 introduced the
possibility for pdev->id being -1 but the change was not done equally in
altera_uart_remove. This patch fixes this.

Acked-by: Anton Vorontsov <cbouatmailru@gmail.com>
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>