netfilter: xt_recent: avoid high order page allocations

xt_recent can try high order page allocations and this can fail.

iptables: page allocation failure: order:9, mode:0xc0d0

It also wastes about half the allocated space because of kmalloc()
power-of-two roundups and struct recent_table layout.

Use vmalloc() instead to save space and be less prone to allocation
errors when memory is fragmented.

Reported-by: Miroslav Kratochvil <exa.exa@gmail.com>
Reported-by: Dave Jones <davej@redhat.com>
Reported-by: Harald Reindl <h.reindl@thelounge.net>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>