drm/vmwgfx: Prune modes based on available VRAM size

This needs to be reviewed once we support screen objects and don't rely
on VRAM for the frame-buffer.

Also fix some integer overflow issues pointed out by Michel Daenzer.

Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>