drm: check for minor master before allowing drop master.

When fast user switching a lot eventually we get to the point,
where we were checking for the wrong thing in this function.

Signed-off-by: Dave Airlie <airlied@redhat.com>