[0/7] iptables: introduce cache evaluation phase

Hi Phil,

Happy new year.

This patchset introduces a new step to parse rules into a list
of commands:

 input -> parser -> list of commands -> list of jobs -> netlink -> kernel

This allows us to estimate the cache requirements from the list of
commands and to avoid in-transit cache cancelations while handling
a batch.

This batch is likely disabling your optimizations where selective
chain/set netlink dump speeds up things. From reading the code,
I'm not sure what paths are specifically benefiting from this
optimization since any of the existing nftnl_chain_list_get() and
nftnl_set_list_get() calls might exercise this selective netlink dump.
I think it should not be too hard to restore those, I remember you have
a few tests to evaluate the speed up. Note that this patch also disables
the pre-parsing from xtables-restore.

Downside is that there's an extra memory allocation, that could be
consolidated in the future, by having one single list of commands/jobs.
But would probably make this batch even larger, but I tried to reduce
complexity as much as possible. Debugging cache issue is hard in my
opinion, if we can avoid fetch/cancel/fetch cache scenario, the better.

The approach that 'among' follows to use the set infrastructure is
interesting, probably there's a way to simplify this, but I understand
there might be more urgent stuff to have a look.

Let me know,
Thanks.

P.S: I started this patchset by Dec 25 while traveling, tests look
     OK here, I'm just releasing this because I'm not sure I can come
     back to this anytime soon.

Pablo Neira Ayuso (7):
  nft: do not check for existing chain from parser
  nft: split parsing from netlink commands
  nft: calculate cache requirements from list of commands
  nft: restore among support
  nft: remove cache build calls
  nft: skip table list release if uninitialized
  nft: missing nft_fini() call in bridge family

 iptables/Makefile.am                               |   2 +-
 iptables/nft-arp.c                                 |   5 +-
 iptables/nft-bridge.c                              |  18 +-
 iptables/nft-cache.c                               |  72 ++--
 iptables/nft-cache.h                               |   1 +
 iptables/nft-cmd.c                                 | 389 +++++++++++++++++++++
 iptables/nft-cmd.h                                 |  79 +++++
 iptables/nft-shared.c                              |   6 +-
 iptables/nft-shared.h                              |   4 +-
 iptables/nft.c                                     | 310 +++++++++++-----
 iptables/nft.h                                     |  49 ++-
 .../shell/testcases/ip6tables/0004-return-codes_0  |   2 +-
 .../shell/testcases/iptables/0004-return-codes_0   |   2 +-
 iptables/xtables-arp.c                             |  26 +-
 iptables/xtables-eb-standalone.c                   |   2 +
 iptables/xtables-eb.c                              |  26 +-
 iptables/xtables-restore.c                         |  35 +-
 iptables/xtables-save.c                            |   3 +
 iptables/xtables.c                                 |  57 ++-
 19 files changed, 863 insertions(+), 225 deletions(-)
 create mode 100644 iptables/nft-cmd.c
 create mode 100644 iptables/nft-cmd.h