Only attempt terminating ARC for audio devices

Fixes an issue introduced in ag/12629370 which inverted the condition
checking the logical address of the removed device.

Bug: 181095118
Test: atest + manual
Change-Id: I33ee58f45d6e366ce1793e76b2083f0aef2773a2