[nft,v3,00/14] libnftables: JSON support

This series adds JSON input and output support to libnftables via
libjansson.

The first five patches prepare the existing code for the actual
implementation which follows in patches 6 and 7. Patches 8 and 9 extend
the simple Nftables Python class in py/nftables.py. The remaining ones
deal with Python testsuite in tests/py: After a bit of cleanup, the last
patch finally extends the testsuite to cover JSON, at least for positive
rule tests.

JSON output support is relatively trivial: If enabled, do_command_list()
redirects to a new function do_command_list_json() which resides in
src/json.c and mimicks the former's behaviour. For translating the
elements of an nftables ruleset into JSON format, the various '_ops'
structs are extended by a new callback which returns a libjansson json_t
object. After the translation is finished, the resulting JSON object is
dumped as a string to output_fp.

To enable JSON output, libnftables API is extended by two new functions:

* nft_ctx_output_get_json() and
* nft_ctx_output_set_json()

these toggle JSON output in the same manner as other libnftables
configuration getter/setters do.

JSON input support is implemented as an alternative parser to the
existing lex/yacc-based one and resides in src/parser_json.c. It's two
entry functions are:

* nft_parse_json_buffer() and
* nft_parse_json_filename(),

which are called from the generic nft_run_cmd_from_*() functions
libnftables API exports iff JSON output has been enabled. Note that this
is rather a performance optimization: If JSON parsing fails early, i.e.
the given buffer or file doesn't look like JSON input at all, regular
syntax parsing is performed as fallback. So out of all combinations of
JSON or standard input or output, only JSON input and standard output
is not possible, but practical significance of that is probably
negligible.

Please note that this series of patches is not fully complete yet. The
(known) missing pieces are:

* JSON format documentation,
* echo and monitor support,
* a dedicated (but simple) test suite to increase test coverage.

The latter two of those are already WiP, for the first one I'm a bit
undecided in which form this should be done - dedicated man page,
README, new wiki article?

Heads-up for reviewers: In order to enforce constraints on which
expression types are allowed in a certain place, parser_json makes use
of context flags which are set by some routines (mainly
json_parse_*_expr()) and then checked in json_parse_expr() (and
elsewhere). This whole thing is so ugly it might burn your eyeballs and
possibly everything behind them. I'm more than happy for any suggestion
for improving this. You have been warned.

Changes since v1:
- Fix patches 6, 7 and patch 14 (former patch 13).
- New patch 13.

Changes since v2:
- Rebased series onto current master branch.
- Adjusted code to upstream changes in struct handle.
- Minor improvement in JSON output debug code.
- Aligned expected JSON output of one test in inet/ip_tcp.t with
  expected standard output.

Phil Sutter (14):
  include/linux: Add required NFT_CT_MAX macro
  libnftables: Put bison parsing into dedicated functions
  libnftables: Make some arrays globally accessible
  libnftables: Make some functions globally accessible
  libnftables: Introduce a few helper functions
  libnftables: Implement JSON output support
  libnftables: Implement JSON parser
  py: Add getter/setter for echo output option
  py: Add JSON support to nftables Class
  tests/py: Reduce indenting level in nft-test.py
  tests/py: Simplify parsing of 'set' lines
  tests/py: Don't read expected payload for each table
  tests/py: Highlight offending parts in differences warnings
  tests/py: Support testing JSON input and output as well

 configure.ac                          |   14 +-
 include/ct.h                          |    4 +
 include/datatype.h                    |    5 +
 include/expression.h                  |    5 +
 include/fib.h                         |    3 +
 include/gmputil.h                     |    1 +
 include/json.h                        |  181 ++
 include/linux/netfilter/nf_tables.h   |    2 +
 include/meta.h                        |    4 +
 include/nftables.h                    |    3 +
 include/nftables/libnftables.h        |    2 +
 include/rt.h                          |    2 +
 include/rule.h                        |    5 +
 include/statement.h                   |   10 +
 include/tcpopt.h                      |    4 +
 py/nftables.py                        |   70 +-
 src/Makefile.am                       |    5 +
 src/ct.c                              |   50 +-
 src/datatype.c                        |   11 +-
 src/expression.c                      |   17 +-
 src/exthdr.c                          |    2 +
 src/fib.c                             |    3 +-
 src/hash.c                            |    1 +
 src/json.c                            | 1564 ++++++++++++
 src/libnftables.c                     |  124 +-
 src/main.c                            |   11 +-
 src/meta.c                            |   11 +-
 src/numgen.c                          |    1 +
 src/parser_json.c                     | 3141 +++++++++++++++++++++++++
 src/payload.c                         |    3 +
 src/rt.c                              |    4 +-
 src/rule.c                            |   10 +-
 src/statement.c                       |   42 +-
 src/tcpopt.c                          |    2 +-
 tests/py/any/ct.t.json                | 1352 +++++++++++
 tests/py/any/ct.t.json.output         |  575 +++++
 tests/py/any/dup.t.json               |   30 +
 tests/py/any/fwd.t.json               |   32 +
 tests/py/any/fwd.t.json.output        |   25 +
 tests/py/any/limit.t.json             |  374 +++
 tests/py/any/log.t.json               |  169 ++
 tests/py/any/log.t.json.output        |   16 +
 tests/py/any/meta.t.json              | 2418 +++++++++++++++++++
 tests/py/any/meta.t.json.output       |  577 +++++
 tests/py/any/queue.t.json             |   86 +
 tests/py/any/queue.t.json.output      |    9 +
 tests/py/any/quota.t.json             |  136 ++
 tests/py/any/rawpayload.t.json        |  157 ++
 tests/py/any/rawpayload.t.json.output |  104 +
 tests/py/any/rt.t.json                |   14 +
 tests/py/arp/arp.t.json               |  877 +++++++
 tests/py/arp/arp.t.json.output        |  148 ++
 tests/py/bridge/ether.t.json          |  180 ++
 tests/py/bridge/ether.t.json.output   |   77 +
 tests/py/bridge/icmpX.t.json          |   82 +
 tests/py/bridge/icmpX.t.json.output   |   52 +
 tests/py/bridge/meta.t.json           |   23 +
 tests/py/bridge/reject.t.json         |  221 ++
 tests/py/bridge/reject.t.json.output  |  260 ++
 tests/py/bridge/vlan.t.json           |  468 ++++
 tests/py/inet/ah.t.json               |  557 +++++
 tests/py/inet/comp.t.json             |  316 +++
 tests/py/inet/comp.t.json.output      |  166 ++
 tests/py/inet/ct.t.json               |   39 +
 tests/py/inet/ct.t.json.output        |   16 +
 tests/py/inet/dccp.t.json             |  340 +++
 tests/py/inet/dccp.t.json.output      |   17 +
 tests/py/inet/esp.t.json              |  256 ++
 tests/py/inet/ether-ip.t.json         |   85 +
 tests/py/inet/ether-ip.t.json.output  |   40 +
 tests/py/inet/ether.t.json            |   84 +
 tests/py/inet/ether.t.json.output     |   29 +
 tests/py/inet/fib.t.json              |  128 +
 tests/py/inet/fib.t.json.output       |   39 +
 tests/py/inet/icmpX.t.json            |  116 +
 tests/py/inet/icmpX.t.json.output     |   86 +
 tests/py/inet/ip.t.json               |   41 +
 tests/py/inet/ip_tcp.t.json           |  158 ++
 tests/py/inet/ip_tcp.t.json.output    |  121 +
 tests/py/inet/map.t.json              |   66 +
 tests/py/inet/map.t.json.output       |   66 +
 tests/py/inet/meta.t.json             |  198 ++
 tests/py/inet/meta.t.json.output      |   35 +
 tests/py/inet/reject.t.json           |  240 ++
 tests/py/inet/reject.t.json.output    |  224 ++
 tests/py/inet/rt.t.json               |   56 +
 tests/py/inet/rt.t.json.output        |   23 +
 tests/py/inet/sctp.t.json             |  594 +++++
 tests/py/inet/tcp.t.json              | 1552 ++++++++++++
 tests/py/inet/tcp.t.json.output       |  134 ++
 tests/py/inet/tcpopt.t.json           |  418 ++++
 tests/py/inet/tcpopt.t.json.output    |   30 +
 tests/py/inet/udp.t.json              |  730 ++++++
 tests/py/inet/udplite.t.json          |  526 +++++
 tests/py/ip/ct.t.json                 |  213 ++
 tests/py/ip/ct.t.json.output          |   25 +
 tests/py/ip/dnat.t.json               |  255 ++
 tests/py/ip/dnat.t.json.output        |   65 +
 tests/py/ip/dup.t.json                |   46 +
 tests/py/ip/ether.t.json              |  151 ++
 tests/py/ip/ether.t.json.output       |   40 +
 tests/py/ip/flowtable.t.json          |   23 +
 tests/py/ip/hash.t.json               |  230 ++
 tests/py/ip/icmp.t.json               | 1382 +++++++++++
 tests/py/ip/icmp.t.json.output        |   85 +
 tests/py/ip/ip.t.json                 | 1781 ++++++++++++++
 tests/py/ip/ip.t.json.output          |  226 ++
 tests/py/ip/ip_tcp.t.json             |   60 +
 tests/py/ip/ip_tcp.t.json.output      |   49 +
 tests/py/ip/masquerade.t.json         |  411 ++++
 tests/py/ip/masquerade.t.json.output  |  118 +
 tests/py/ip/meta.t.json               |   99 +
 tests/py/ip/meta.t.json.output        |   45 +
 tests/py/ip/numgen.t.json             |   99 +
 tests/py/ip/numgen.t.json.output      |   82 +
 tests/py/ip/objects.t.json            |  185 ++
 tests/py/ip/objects.t.json.output     |   64 +
 tests/py/ip/redirect.t.json           |  610 +++++
 tests/py/ip/redirect.t.json.output    |  140 ++
 tests/py/ip/reject.t.json             |   94 +
 tests/py/ip/reject.t.json.output      |   24 +
 tests/py/ip/rt.t.json                 |   15 +
 tests/py/ip/sets.t.json               |  185 ++
 tests/py/ip/snat.t.json               |  161 ++
 tests/py/ip/snat.t.json.output        |   69 +
 tests/py/ip/tcp.t.json                |   59 +
 tests/py/ip/tcp.t.json.output         |   48 +
 tests/py/ip/tcpopt.t.json             |  390 +++
 tests/py/ip/tcpopt.t.json.output      |   15 +
 tests/py/ip6/dnat.t.json              |   75 +
 tests/py/ip6/dst.t.json               |  386 +++
 tests/py/ip6/dst.t.json.output        |   86 +
 tests/py/ip6/dup.t.json               |   46 +
 tests/py/ip6/ether.t.json             |  151 ++
 tests/py/ip6/ether.t.json.output      |   40 +
 tests/py/ip6/exthdr.t.json            |  172 ++
 tests/py/ip6/exthdr.t.json.output     |   56 +
 tests/py/ip6/flowtable.t.json         |   60 +
 tests/py/ip6/frag.t.json              |  624 +++++
 tests/py/ip6/frag.t.json.output       |  114 +
 tests/py/ip6/hbh.t.json               |  386 +++
 tests/py/ip6/hbh.t.json.output        |   66 +
 tests/py/ip6/icmpv6.t.json            | 1257 ++++++++++
 tests/py/ip6/icmpv6.t.json.output     |   92 +
 tests/py/ip6/ip6.t.json               | 1613 +++++++++++++
 tests/py/ip6/ip6.t.json.output        |  361 +++
 tests/py/ip6/map.t.json               |   38 +
 tests/py/ip6/map.t.json.output        |   38 +
 tests/py/ip6/masquerade.t.json        |  405 ++++
 tests/py/ip6/masquerade.t.json.output |   94 +
 tests/py/ip6/meta.t.json              |   99 +
 tests/py/ip6/meta.t.json.output       |   45 +
 tests/py/ip6/mh.t.json                |  781 ++++++
 tests/py/ip6/mh.t.json.output         |   86 +
 tests/py/ip6/redirect.t.json          |  576 +++++
 tests/py/ip6/redirect.t.json.output   |  140 ++
 tests/py/ip6/reject.t.json            |   84 +
 tests/py/ip6/reject.t.json.output     |   24 +
 tests/py/ip6/rt.t.json                |  717 ++++++
 tests/py/ip6/rt.t.json.output         |   86 +
 tests/py/ip6/rt0.t.json               |   15 +
 tests/py/ip6/sets.t.json              |   91 +
 tests/py/ip6/snat.t.json              |   52 +
 tests/py/ip6/srh.t.json               |  183 ++
 tests/py/ip6/srh.t.json.output        |   21 +
 tests/py/ip6/tcpopt.t.json            |  390 +++
 tests/py/ip6/tcpopt.t.json.output     |   15 +
 tests/py/ip6/vmap.t.json              | 1037 ++++++++
 tests/py/ip6/vmap.t.json.output       |  336 +++
 tests/py/nft-test.py                  |  381 ++-
 170 files changed, 40010 insertions(+), 158 deletions(-)
 create mode 100644 include/json.h
 create mode 100644 src/json.c
 create mode 100644 src/parser_json.c
 create mode 100644 tests/py/any/ct.t.json
 create mode 100644 tests/py/any/ct.t.json.output
 create mode 100644 tests/py/any/dup.t.json
 create mode 100644 tests/py/any/fwd.t.json
 create mode 100644 tests/py/any/fwd.t.json.output
 create mode 100644 tests/py/any/limit.t.json
 create mode 100644 tests/py/any/log.t.json
 create mode 100644 tests/py/any/log.t.json.output
 create mode 100644 tests/py/any/meta.t.json
 create mode 100644 tests/py/any/meta.t.json.output
 create mode 100644 tests/py/any/queue.t.json
 create mode 100644 tests/py/any/queue.t.json.output
 create mode 100644 tests/py/any/quota.t.json
 create mode 100644 tests/py/any/rawpayload.t.json
 create mode 100644 tests/py/any/rawpayload.t.json.output
 create mode 100644 tests/py/any/rt.t.json
 create mode 100644 tests/py/arp/arp.t.json
 create mode 100644 tests/py/arp/arp.t.json.output
 create mode 100644 tests/py/bridge/ether.t.json
 create mode 100644 tests/py/bridge/ether.t.json.output
 create mode 100644 tests/py/bridge/icmpX.t.json
 create mode 100644 tests/py/bridge/icmpX.t.json.output
 create mode 100644 tests/py/bridge/meta.t.json
 create mode 100644 tests/py/bridge/reject.t.json
 create mode 100644 tests/py/bridge/reject.t.json.output
 create mode 100644 tests/py/bridge/vlan.t.json
 create mode 100644 tests/py/inet/ah.t.json
 create mode 100644 tests/py/inet/comp.t.json
 create mode 100644 tests/py/inet/comp.t.json.output
 create mode 100644 tests/py/inet/ct.t.json
 create mode 100644 tests/py/inet/ct.t.json.output
 create mode 100644 tests/py/inet/dccp.t.json
 create mode 100644 tests/py/inet/dccp.t.json.output
 create mode 100644 tests/py/inet/esp.t.json
 create mode 100644 tests/py/inet/ether-ip.t.json
 create mode 100644 tests/py/inet/ether-ip.t.json.output
 create mode 100644 tests/py/inet/ether.t.json
 create mode 100644 tests/py/inet/ether.t.json.output
 create mode 100644 tests/py/inet/fib.t.json
 create mode 100644 tests/py/inet/fib.t.json.output
 create mode 100644 tests/py/inet/icmpX.t.json
 create mode 100644 tests/py/inet/icmpX.t.json.output
 create mode 100644 tests/py/inet/ip.t.json
 create mode 100644 tests/py/inet/ip_tcp.t.json
 create mode 100644 tests/py/inet/ip_tcp.t.json.output
 create mode 100644 tests/py/inet/map.t.json
 create mode 100644 tests/py/inet/map.t.json.output
 create mode 100644 tests/py/inet/meta.t.json
 create mode 100644 tests/py/inet/meta.t.json.output
 create mode 100644 tests/py/inet/reject.t.json
 create mode 100644 tests/py/inet/reject.t.json.output
 create mode 100644 tests/py/inet/rt.t.json
 create mode 100644 tests/py/inet/rt.t.json.output
 create mode 100644 tests/py/inet/sctp.t.json
 create mode 100644 tests/py/inet/tcp.t.json
 create mode 100644 tests/py/inet/tcp.t.json.output
 create mode 100644 tests/py/inet/tcpopt.t.json
 create mode 100644 tests/py/inet/tcpopt.t.json.output
 create mode 100644 tests/py/inet/udp.t.json
 create mode 100644 tests/py/inet/udplite.t.json
 create mode 100644 tests/py/ip/ct.t.json
 create mode 100644 tests/py/ip/ct.t.json.output
 create mode 100644 tests/py/ip/dnat.t.json
 create mode 100644 tests/py/ip/dnat.t.json.output
 create mode 100644 tests/py/ip/dup.t.json
 create mode 100644 tests/py/ip/ether.t.json
 create mode 100644 tests/py/ip/ether.t.json.output
 create mode 100644 tests/py/ip/flowtable.t.json
 create mode 100644 tests/py/ip/hash.t.json
 create mode 100644 tests/py/ip/icmp.t.json
 create mode 100644 tests/py/ip/icmp.t.json.output
 create mode 100644 tests/py/ip/ip.t.json
 create mode 100644 tests/py/ip/ip.t.json.output
 create mode 100644 tests/py/ip/ip_tcp.t.json
 create mode 100644 tests/py/ip/ip_tcp.t.json.output
 create mode 100644 tests/py/ip/masquerade.t.json
 create mode 100644 tests/py/ip/masquerade.t.json.output
 create mode 100644 tests/py/ip/meta.t.json
 create mode 100644 tests/py/ip/meta.t.json.output
 create mode 100644 tests/py/ip/numgen.t.json
 create mode 100644 tests/py/ip/numgen.t.json.output
 create mode 100644 tests/py/ip/objects.t.json
 create mode 100644 tests/py/ip/objects.t.json.output
 create mode 100644 tests/py/ip/redirect.t.json
 create mode 100644 tests/py/ip/redirect.t.json.output
 create mode 100644 tests/py/ip/reject.t.json
 create mode 100644 tests/py/ip/reject.t.json.output
 create mode 100644 tests/py/ip/rt.t.json
 create mode 100644 tests/py/ip/sets.t.json
 create mode 100644 tests/py/ip/snat.t.json
 create mode 100644 tests/py/ip/snat.t.json.output
 create mode 100644 tests/py/ip/tcp.t.json
 create mode 100644 tests/py/ip/tcp.t.json.output
 create mode 100644 tests/py/ip/tcpopt.t.json
 create mode 100644 tests/py/ip/tcpopt.t.json.output
 create mode 100644 tests/py/ip6/dnat.t.json
 create mode 100644 tests/py/ip6/dst.t.json
 create mode 100644 tests/py/ip6/dst.t.json.output
 create mode 100644 tests/py/ip6/dup.t.json
 create mode 100644 tests/py/ip6/ether.t.json
 create mode 100644 tests/py/ip6/ether.t.json.output
 create mode 100644 tests/py/ip6/exthdr.t.json
 create mode 100644 tests/py/ip6/exthdr.t.json.output
 create mode 100644 tests/py/ip6/flowtable.t.json
 create mode 100644 tests/py/ip6/frag.t.json
 create mode 100644 tests/py/ip6/frag.t.json.output
 create mode 100644 tests/py/ip6/hbh.t.json
 create mode 100644 tests/py/ip6/hbh.t.json.output
 create mode 100644 tests/py/ip6/icmpv6.t.json
 create mode 100644 tests/py/ip6/icmpv6.t.json.output
 create mode 100644 tests/py/ip6/ip6.t.json
 create mode 100644 tests/py/ip6/ip6.t.json.output
 create mode 100644 tests/py/ip6/map.t.json
 create mode 100644 tests/py/ip6/map.t.json.output
 create mode 100644 tests/py/ip6/masquerade.t.json
 create mode 100644 tests/py/ip6/masquerade.t.json.output
 create mode 100644 tests/py/ip6/meta.t.json
 create mode 100644 tests/py/ip6/meta.t.json.output
 create mode 100644 tests/py/ip6/mh.t.json
 create mode 100644 tests/py/ip6/mh.t.json.output
 create mode 100644 tests/py/ip6/redirect.t.json
 create mode 100644 tests/py/ip6/redirect.t.json.output
 create mode 100644 tests/py/ip6/reject.t.json
 create mode 100644 tests/py/ip6/reject.t.json.output
 create mode 100644 tests/py/ip6/rt.t.json
 create mode 100644 tests/py/ip6/rt.t.json.output
 create mode 100644 tests/py/ip6/rt0.t.json
 create mode 100644 tests/py/ip6/sets.t.json
 create mode 100644 tests/py/ip6/snat.t.json
 create mode 100644 tests/py/ip6/srh.t.json
 create mode 100644 tests/py/ip6/srh.t.json.output
 create mode 100644 tests/py/ip6/tcpopt.t.json
 create mode 100644 tests/py/ip6/tcpopt.t.json.output
 create mode 100644 tests/py/ip6/vmap.t.json
 create mode 100644 tests/py/ip6/vmap.t.json.output