Destroy prefetched layers when the context is destroyed

Also fix a few places preventing the tests from running
in SkiaVulkan, although that is still not enabled in general yet

Fixes: 283315634
Test: CanvasContext_buildLayerDoesntLeak unit test
Change-Id: I4d8a50db447e47afd8e8a267df332a173c88e888