Properly destroy TrustAgentWrapper when not bound

Also fixes a potential issue where refreshing agents
for a user that no longer exists would result in a crash.

Bug: 18318629
Change-Id: I3589ea7e0f2e63fca02daeecf3ca964a8a8e4b3b