mbox series

[nft,v2,00/14] libnftables: JSON support

Message ID 20180505125606.1909-1-phil@nwl.cc
Headers show
Series libnftables: JSON support | expand

Message

Phil Sutter May 5, 2018, 12:55 p.m. UTC 
This series adds JSON input and output support to libnftables via
libjansson.

The first five patches prepare the existing code for the actual
implementation which follows in patches 6 and 7. Patches 8 and 9 extend
the simple Nftables Python class in py/nftables.py. The remaining ones
deal with Python testsuite in tests/py: After a bit of cleanup, the last
patch finally extends the testsuite to cover JSON, at least for positive
rule tests.

JSON output support is relatively trivial: If enabled, do_command_list()
redirects to a new function do_command_list_json() which resides in
src/json.c and mimicks the former's behaviour. For translating the
elements of an nftables ruleset into JSON format, the various '_ops'
structs are extended by a new callback which returns a libjansson json_t
object. After the translation is finished, the resulting JSON object is
dumped as a string to output_fp.

To enable JSON output, libnftables API is extended by two new functions:

* nft_ctx_output_get_json() and
* nft_ctx_output_set_json()

these toggle JSON output in the same manner as other libnftables
configuration getter/setters do.

JSON input support is implemented as an alternative parser to the
existing lex/yacc-based one and resides in src/parser_json.c. It's two
entry functions are:

* nft_parse_json_buffer() and
* nft_parse_json_filename(),

which are called from the generic nft_run_cmd_from_*() functions
libnftables API exports iff JSON output has been enabled. Note that this
is rather a performance optimization: If JSON parsing fails early, i.e.
the given buffer or file doesn't look like JSON input at all, regular
syntax parsing is performed as fallback. So out of all combinations of
JSON or standard input or output, only JSON input and standard output
is not possible, but practical significance of that is probably
negligible.

Please note that this series of patches is not fully complete yet. The
(known) missing pieces are:

* JSON format documentation,
* echo and monitor support,
* a dedicated (but simple) test suite to increase test coverage.

The latter two of those are already WiP, for the first one I'm a bit
undecided in which form this should be done - dedicated man page,
README, new wiki article?

Heads-up for reviewers: In order to enforce constraints on which
expression types are allowed in a certain place, parser_json makes use
of context flags which are set by some routines (mainly
json_parse_*_expr()) and then checked in json_parse_expr() (and
elsewhere). This whole thing is so ugly it might burn your eyeballs and
possibly everything behind them. I'm more than happy for any suggestion
for improving this. You have been warned.

Changes since v1:
- Fix patches 6, 7 and patch 14 (former patch 13).
- New patch 13.

Phil Sutter (14):
  include/linux: Add required NFT_CT_MAX macro
  libnftables: Put bison parsing into dedicated functions
  libnftables: Make some arrays globally accessible
  libnftables: Make some functions globally accessible
  libnftables: Introduce a few helper functions
  libnftables: Implement JSON output support
  libnftables: Implement JSON parser
  py: Add getter/setter for echo output option
  py: Add JSON support to nftables Class
  tests/py: Reduce indenting level in nft-test.py
  tests/py: Simplify parsing of 'set' lines
  tests/py: Don't read expected payload for each table
  tests/py: Highlight offending parts in differences warnings
  tests/py: Support testing JSON input and output as well

 configure.ac                          |   14 +-
 include/ct.h                          |    4 +
 include/datatype.h                    |    5 +
 include/expression.h                  |    5 +
 include/fib.h                         |    3 +
 include/gmputil.h                     |    1 +
 include/json.h                        |  181 ++
 include/linux/netfilter/nf_tables.h   |    2 +
 include/meta.h                        |    4 +
 include/nftables.h                    |    3 +
 include/nftables/libnftables.h        |    2 +
 include/rt.h                          |    2 +
 include/rule.h                        |    5 +
 include/statement.h                   |   10 +
 include/tcpopt.h                      |    4 +
 py/nftables.py                        |   70 +-
 src/Makefile.am                       |    5 +
 src/ct.c                              |   50 +-
 src/datatype.c                        |   11 +-
 src/expression.c                      |   17 +-
 src/exthdr.c                          |    2 +
 src/fib.c                             |    3 +-
 src/hash.c                            |    1 +
 src/json.c                            | 1571 +++++++++++++
 src/libnftables.c                     |  124 +-
 src/main.c                            |   11 +-
 src/meta.c                            |   11 +-
 src/numgen.c                          |    1 +
 src/parser_json.c                     | 3141 +++++++++++++++++++++++++
 src/payload.c                         |    3 +
 src/rt.c                              |    4 +-
 src/rule.c                            |   10 +-
 src/statement.c                       |   42 +-
 src/tcpopt.c                          |    2 +-
 tests/py/any/ct.t.json                | 1352 +++++++++++
 tests/py/any/ct.t.json.output         |  575 +++++
 tests/py/any/dup.t.json               |   30 +
 tests/py/any/fwd.t.json               |   32 +
 tests/py/any/fwd.t.json.output        |   25 +
 tests/py/any/limit.t.json             |  374 +++
 tests/py/any/log.t.json               |  169 ++
 tests/py/any/log.t.json.output        |   16 +
 tests/py/any/meta.t.json              | 2418 +++++++++++++++++++
 tests/py/any/meta.t.json.output       |  577 +++++
 tests/py/any/queue.t.json             |   86 +
 tests/py/any/queue.t.json.output      |    9 +
 tests/py/any/quota.t.json             |  136 ++
 tests/py/any/rawpayload.t.json        |  157 ++
 tests/py/any/rawpayload.t.json.output |  104 +
 tests/py/any/rt.t.json                |   14 +
 tests/py/arp/arp.t.json               |  877 +++++++
 tests/py/arp/arp.t.json.output        |  148 ++
 tests/py/bridge/ether.t.json          |  180 ++
 tests/py/bridge/ether.t.json.output   |   77 +
 tests/py/bridge/icmpX.t.json          |   82 +
 tests/py/bridge/icmpX.t.json.output   |   52 +
 tests/py/bridge/meta.t.json           |   23 +
 tests/py/bridge/reject.t.json         |  221 ++
 tests/py/bridge/reject.t.json.output  |  260 ++
 tests/py/bridge/vlan.t.json           |  468 ++++
 tests/py/inet/ah.t.json               |  557 +++++
 tests/py/inet/comp.t.json             |  316 +++
 tests/py/inet/comp.t.json.output      |  166 ++
 tests/py/inet/ct.t.json               |   39 +
 tests/py/inet/ct.t.json.output        |   16 +
 tests/py/inet/dccp.t.json             |  340 +++
 tests/py/inet/dccp.t.json.output      |   17 +
 tests/py/inet/esp.t.json              |  256 ++
 tests/py/inet/ether-ip.t.json         |   85 +
 tests/py/inet/ether-ip.t.json.output  |   40 +
 tests/py/inet/ether.t.json            |   84 +
 tests/py/inet/ether.t.json.output     |   29 +
 tests/py/inet/fib.t.json              |  128 +
 tests/py/inet/fib.t.json.output       |   39 +
 tests/py/inet/icmpX.t.json            |  116 +
 tests/py/inet/icmpX.t.json.output     |   86 +
 tests/py/inet/ip.t.json               |   41 +
 tests/py/inet/ip_tcp.t.json           |  158 ++
 tests/py/inet/ip_tcp.t.json.output    |  132 ++
 tests/py/inet/map.t.json              |   66 +
 tests/py/inet/map.t.json.output       |   66 +
 tests/py/inet/meta.t.json             |  198 ++
 tests/py/inet/meta.t.json.output      |   35 +
 tests/py/inet/reject.t.json           |  240 ++
 tests/py/inet/reject.t.json.output    |  224 ++
 tests/py/inet/rt.t.json               |   56 +
 tests/py/inet/rt.t.json.output        |   23 +
 tests/py/inet/sctp.t.json             |  594 +++++
 tests/py/inet/tcp.t.json              | 1552 ++++++++++++
 tests/py/inet/tcp.t.json.output       |  134 ++
 tests/py/inet/tcpopt.t.json           |  418 ++++
 tests/py/inet/tcpopt.t.json.output    |   30 +
 tests/py/inet/udp.t.json              |  730 ++++++
 tests/py/inet/udplite.t.json          |  526 +++++
 tests/py/ip/ct.t.json                 |  213 ++
 tests/py/ip/ct.t.json.output          |   25 +
 tests/py/ip/dnat.t.json               |  255 ++
 tests/py/ip/dnat.t.json.output        |   65 +
 tests/py/ip/dup.t.json                |   46 +
 tests/py/ip/ether.t.json              |  151 ++
 tests/py/ip/ether.t.json.output       |   40 +
 tests/py/ip/flowtable.t.json          |   23 +
 tests/py/ip/hash.t.json               |  230 ++
 tests/py/ip/icmp.t.json               | 1382 +++++++++++
 tests/py/ip/icmp.t.json.output        |   85 +
 tests/py/ip/ip.t.json                 | 1781 ++++++++++++++
 tests/py/ip/ip.t.json.output          |  226 ++
 tests/py/ip/ip_tcp.t.json             |   60 +
 tests/py/ip/ip_tcp.t.json.output      |   49 +
 tests/py/ip/masquerade.t.json         |  411 ++++
 tests/py/ip/masquerade.t.json.output  |  118 +
 tests/py/ip/meta.t.json               |   99 +
 tests/py/ip/meta.t.json.output        |   45 +
 tests/py/ip/numgen.t.json             |   99 +
 tests/py/ip/numgen.t.json.output      |   82 +
 tests/py/ip/objects.t.json            |  185 ++
 tests/py/ip/objects.t.json.output     |   64 +
 tests/py/ip/redirect.t.json           |  610 +++++
 tests/py/ip/redirect.t.json.output    |  140 ++
 tests/py/ip/reject.t.json             |   94 +
 tests/py/ip/reject.t.json.output      |   24 +
 tests/py/ip/rt.t.json                 |   15 +
 tests/py/ip/sets.t.json               |  185 ++
 tests/py/ip/snat.t.json               |  161 ++
 tests/py/ip/snat.t.json.output        |   69 +
 tests/py/ip/tcp.t.json                |   59 +
 tests/py/ip/tcp.t.json.output         |   48 +
 tests/py/ip/tcpopt.t.json             |  390 +++
 tests/py/ip/tcpopt.t.json.output      |   15 +
 tests/py/ip6/dnat.t.json              |   75 +
 tests/py/ip6/dst.t.json               |  386 +++
 tests/py/ip6/dst.t.json.output        |   86 +
 tests/py/ip6/dup.t.json               |   46 +
 tests/py/ip6/ether.t.json             |  151 ++
 tests/py/ip6/ether.t.json.output      |   40 +
 tests/py/ip6/exthdr.t.json            |  172 ++
 tests/py/ip6/exthdr.t.json.output     |   56 +
 tests/py/ip6/flowtable.t.json         |   60 +
 tests/py/ip6/frag.t.json              |  624 +++++
 tests/py/ip6/frag.t.json.output       |  114 +
 tests/py/ip6/hbh.t.json               |  386 +++
 tests/py/ip6/hbh.t.json.output        |   66 +
 tests/py/ip6/icmpv6.t.json            | 1257 ++++++++++
 tests/py/ip6/icmpv6.t.json.output     |   92 +
 tests/py/ip6/ip6.t.json               | 1613 +++++++++++++
 tests/py/ip6/ip6.t.json.output        |  361 +++
 tests/py/ip6/map.t.json               |   38 +
 tests/py/ip6/map.t.json.output        |   38 +
 tests/py/ip6/masquerade.t.json        |  405 ++++
 tests/py/ip6/masquerade.t.json.output |   94 +
 tests/py/ip6/meta.t.json              |   99 +
 tests/py/ip6/meta.t.json.output       |   45 +
 tests/py/ip6/mh.t.json                |  781 ++++++
 tests/py/ip6/mh.t.json.output         |   86 +
 tests/py/ip6/redirect.t.json          |  576 +++++
 tests/py/ip6/redirect.t.json.output   |  140 ++
 tests/py/ip6/reject.t.json            |   84 +
 tests/py/ip6/reject.t.json.output     |   24 +
 tests/py/ip6/rt.t.json                |  717 ++++++
 tests/py/ip6/rt.t.json.output         |   86 +
 tests/py/ip6/rt0.t.json               |   15 +
 tests/py/ip6/sets.t.json              |   91 +
 tests/py/ip6/snat.t.json              |   52 +
 tests/py/ip6/srh.t.json               |  183 ++
 tests/py/ip6/srh.t.json.output        |   21 +
 tests/py/ip6/tcpopt.t.json            |  390 +++
 tests/py/ip6/tcpopt.t.json.output     |   15 +
 tests/py/ip6/vmap.t.json              | 1037 ++++++++
 tests/py/ip6/vmap.t.json.output       |  336 +++
 tests/py/nft-test.py                  |  381 ++-
 170 files changed, 40028 insertions(+), 158 deletions(-)
 create mode 100644 include/json.h
 create mode 100644 src/json.c
 create mode 100644 src/parser_json.c
 create mode 100644 tests/py/any/ct.t.json
 create mode 100644 tests/py/any/ct.t.json.output
 create mode 100644 tests/py/any/dup.t.json
 create mode 100644 tests/py/any/fwd.t.json
 create mode 100644 tests/py/any/fwd.t.json.output
 create mode 100644 tests/py/any/limit.t.json
 create mode 100644 tests/py/any/log.t.json
 create mode 100644 tests/py/any/log.t.json.output
 create mode 100644 tests/py/any/meta.t.json
 create mode 100644 tests/py/any/meta.t.json.output
 create mode 100644 tests/py/any/queue.t.json
 create mode 100644 tests/py/any/queue.t.json.output
 create mode 100644 tests/py/any/quota.t.json
 create mode 100644 tests/py/any/rawpayload.t.json
 create mode 100644 tests/py/any/rawpayload.t.json.output
 create mode 100644 tests/py/any/rt.t.json
 create mode 100644 tests/py/arp/arp.t.json
 create mode 100644 tests/py/arp/arp.t.json.output
 create mode 100644 tests/py/bridge/ether.t.json
 create mode 100644 tests/py/bridge/ether.t.json.output
 create mode 100644 tests/py/bridge/icmpX.t.json
 create mode 100644 tests/py/bridge/icmpX.t.json.output
 create mode 100644 tests/py/bridge/meta.t.json
 create mode 100644 tests/py/bridge/reject.t.json
 create mode 100644 tests/py/bridge/reject.t.json.output
 create mode 100644 tests/py/bridge/vlan.t.json
 create mode 100644 tests/py/inet/ah.t.json
 create mode 100644 tests/py/inet/comp.t.json
 create mode 100644 tests/py/inet/comp.t.json.output
 create mode 100644 tests/py/inet/ct.t.json
 create mode 100644 tests/py/inet/ct.t.json.output
 create mode 100644 tests/py/inet/dccp.t.json
 create mode 100644 tests/py/inet/dccp.t.json.output
 create mode 100644 tests/py/inet/esp.t.json
 create mode 100644 tests/py/inet/ether-ip.t.json
 create mode 100644 tests/py/inet/ether-ip.t.json.output
 create mode 100644 tests/py/inet/ether.t.json
 create mode 100644 tests/py/inet/ether.t.json.output
 create mode 100644 tests/py/inet/fib.t.json
 create mode 100644 tests/py/inet/fib.t.json.output
 create mode 100644 tests/py/inet/icmpX.t.json
 create mode 100644 tests/py/inet/icmpX.t.json.output
 create mode 100644 tests/py/inet/ip.t.json
 create mode 100644 tests/py/inet/ip_tcp.t.json
 create mode 100644 tests/py/inet/ip_tcp.t.json.output
 create mode 100644 tests/py/inet/map.t.json
 create mode 100644 tests/py/inet/map.t.json.output
 create mode 100644 tests/py/inet/meta.t.json
 create mode 100644 tests/py/inet/meta.t.json.output
 create mode 100644 tests/py/inet/reject.t.json
 create mode 100644 tests/py/inet/reject.t.json.output
 create mode 100644 tests/py/inet/rt.t.json
 create mode 100644 tests/py/inet/rt.t.json.output
 create mode 100644 tests/py/inet/sctp.t.json
 create mode 100644 tests/py/inet/tcp.t.json
 create mode 100644 tests/py/inet/tcp.t.json.output
 create mode 100644 tests/py/inet/tcpopt.t.json
 create mode 100644 tests/py/inet/tcpopt.t.json.output
 create mode 100644 tests/py/inet/udp.t.json
 create mode 100644 tests/py/inet/udplite.t.json
 create mode 100644 tests/py/ip/ct.t.json
 create mode 100644 tests/py/ip/ct.t.json.output
 create mode 100644 tests/py/ip/dnat.t.json
 create mode 100644 tests/py/ip/dnat.t.json.output
 create mode 100644 tests/py/ip/dup.t.json
 create mode 100644 tests/py/ip/ether.t.json
 create mode 100644 tests/py/ip/ether.t.json.output
 create mode 100644 tests/py/ip/flowtable.t.json
 create mode 100644 tests/py/ip/hash.t.json
 create mode 100644 tests/py/ip/icmp.t.json
 create mode 100644 tests/py/ip/icmp.t.json.output
 create mode 100644 tests/py/ip/ip.t.json
 create mode 100644 tests/py/ip/ip.t.json.output
 create mode 100644 tests/py/ip/ip_tcp.t.json
 create mode 100644 tests/py/ip/ip_tcp.t.json.output
 create mode 100644 tests/py/ip/masquerade.t.json
 create mode 100644 tests/py/ip/masquerade.t.json.output
 create mode 100644 tests/py/ip/meta.t.json
 create mode 100644 tests/py/ip/meta.t.json.output
 create mode 100644 tests/py/ip/numgen.t.json
 create mode 100644 tests/py/ip/numgen.t.json.output
 create mode 100644 tests/py/ip/objects.t.json
 create mode 100644 tests/py/ip/objects.t.json.output
 create mode 100644 tests/py/ip/redirect.t.json
 create mode 100644 tests/py/ip/redirect.t.json.output
 create mode 100644 tests/py/ip/reject.t.json
 create mode 100644 tests/py/ip/reject.t.json.output
 create mode 100644 tests/py/ip/rt.t.json
 create mode 100644 tests/py/ip/sets.t.json
 create mode 100644 tests/py/ip/snat.t.json
 create mode 100644 tests/py/ip/snat.t.json.output
 create mode 100644 tests/py/ip/tcp.t.json
 create mode 100644 tests/py/ip/tcp.t.json.output
 create mode 100644 tests/py/ip/tcpopt.t.json
 create mode 100644 tests/py/ip/tcpopt.t.json.output
 create mode 100644 tests/py/ip6/dnat.t.json
 create mode 100644 tests/py/ip6/dst.t.json
 create mode 100644 tests/py/ip6/dst.t.json.output
 create mode 100644 tests/py/ip6/dup.t.json
 create mode 100644 tests/py/ip6/ether.t.json
 create mode 100644 tests/py/ip6/ether.t.json.output
 create mode 100644 tests/py/ip6/exthdr.t.json
 create mode 100644 tests/py/ip6/exthdr.t.json.output
 create mode 100644 tests/py/ip6/flowtable.t.json
 create mode 100644 tests/py/ip6/frag.t.json
 create mode 100644 tests/py/ip6/frag.t.json.output
 create mode 100644 tests/py/ip6/hbh.t.json
 create mode 100644 tests/py/ip6/hbh.t.json.output
 create mode 100644 tests/py/ip6/icmpv6.t.json
 create mode 100644 tests/py/ip6/icmpv6.t.json.output
 create mode 100644 tests/py/ip6/ip6.t.json
 create mode 100644 tests/py/ip6/ip6.t.json.output
 create mode 100644 tests/py/ip6/map.t.json
 create mode 100644 tests/py/ip6/map.t.json.output
 create mode 100644 tests/py/ip6/masquerade.t.json
 create mode 100644 tests/py/ip6/masquerade.t.json.output
 create mode 100644 tests/py/ip6/meta.t.json
 create mode 100644 tests/py/ip6/meta.t.json.output
 create mode 100644 tests/py/ip6/mh.t.json
 create mode 100644 tests/py/ip6/mh.t.json.output
 create mode 100644 tests/py/ip6/redirect.t.json
 create mode 100644 tests/py/ip6/redirect.t.json.output
 create mode 100644 tests/py/ip6/reject.t.json
 create mode 100644 tests/py/ip6/reject.t.json.output
 create mode 100644 tests/py/ip6/rt.t.json
 create mode 100644 tests/py/ip6/rt.t.json.output
 create mode 100644 tests/py/ip6/rt0.t.json
 create mode 100644 tests/py/ip6/sets.t.json
 create mode 100644 tests/py/ip6/snat.t.json
 create mode 100644 tests/py/ip6/srh.t.json
 create mode 100644 tests/py/ip6/srh.t.json.output
 create mode 100644 tests/py/ip6/tcpopt.t.json
 create mode 100644 tests/py/ip6/tcpopt.t.json.output
 create mode 100644 tests/py/ip6/vmap.t.json
 create mode 100644 tests/py/ip6/vmap.t.json.output

Comments

Pablo Neira Ayuso May 11, 2018, 10:16 a.m. UTC | #1 
On Sat, May 05, 2018 at 02:55:52PM +0200, Phil Sutter wrote:
> This series adds JSON input and output support to libnftables via
> libjansson.

Series applied, thanks Phil.

Except:

 [nft PATCH v3 11/14] tests/py: Simplify parsing of 'set' lines

which seems to clash with some recent update. Please, resend this one.

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Phil Sutter May 11, 2018, 12:15 p.m. UTC | #2 
Hi,

On Fri, May 11, 2018 at 12:16:07PM +0200, Pablo Neira Ayuso wrote:
> On Sat, May 05, 2018 at 02:55:52PM +0200, Phil Sutter wrote:
> > This series adds JSON input and output support to libnftables via
> > libjansson.
> 
> Series applied, thanks Phil.
> 
> Except:
> 
>  [nft PATCH v3 11/14] tests/py: Simplify parsing of 'set' lines
> 
> which seems to clash with some recent update. Please, resend this one.

With the addition of timeout support to set definitions, the
simplification that patch achieves is no longer easily possible. Since
it doesn't provide any additional functionality otherwise, let's just
drop it.

Cheers, Phil
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html