Remove unused member in VulkanManager.

A large part of the previous change was specifically around removing
this ref counted data member, and then I just forgot to delete it...

Test: manual testing

Change-Id: Ie9a7f7495789d45a65900c08a27a9d83ad64617c