[ovs-dev,v3,00/18] python: add flow parsing library

While troubleshooting or developing new features in OVS, a considerable
amount of time is spent analyzing flows (whether that's Openflow flows
or datapath flows). Currently, OVS has tools to dump flows with
different levels of verbosity as well as to filter flows prior to
dumping them, e.g: 'ovs-ofctl dump-flows', 'ovs-appctl
dpctl/dump-flows', etc.

The output of these commands is considered stable so it should be
possible to write more layered tools that enable advanced flow analysis.
However, the way flows are formatted into strings is not trivial to
parse.

This series introduces the a flow parsing library capable of parsing
both Openflow and DPIF flows.

The library is based on generic key-value and list parsers and a number
of common decoders. Based on that, an Openflow flow parser and a DPIF
flow parser are introduced by defining a way to decode each possible
field and action they might contain.

The library has the following features:
- Parsed key-value pairs keep some metadata that refer to the original
  strings they were extracted from. That way the flows can be printed
  and formatted in flexible ways.
- It includes a basic flow filtering mechanism. A filter can be defined
  combining logical (||, &&, !), arithmetical (<, >, =) or mask (~=)
  operations
- It supports IPAddress and Ethernet masking (based on netaddr)
- The decoder to use for each key (match or action) is set explicitly to
  avoid expensive runtime type-guessing.
- The decoders to use for Openflow fields is automatically generated
  based on meta-flow.h
- Additional dependencies:
  - netaddr: For IP and Ethernet Address management.
  - pyparsing: For filtering syntax.
  - pytest: For unit tests.

One key goal challenge of including this library is avoiding diversion
between the C code that prints/parses the flows and the python parsing
code. To that effect, the series introduces the following mechanisms:
- Decoding information of openflow fields is automatically generated
  based on meta-flow.h
- The calls to ovs-ofctl made from tests/ofp-actions.at are wrapped by a
  python script that also runs the python parsers. If an exception is
  raised by the python code (meaning it was not capable of parsing the
  flow string), the test will fail
- The calls to the test-odp made from tests/odp.at are wrapped by a
  python script that also runs the python parsers. If an exception is
  raised by the python code (meaning it was not capable of parsing the
  flow string), the test will fail.
- A python unit test testsuite ensures python code works and it's easy
  to add more flow examples to it
- A dependency check is introduced. The python parsing code mainly
  depends on lib/ofp-actions.c and lib/odp-util.c. This series stores
  the md5sum of such files and adds a build target that ensures the
  files have not been changed. That way, anyone who modifies those files
  will get a warning the fist time they build the project. Dependency
  digests are easily updated using a string so hopefully this warning
  would not be too inconvenient.

Library usage
-------------
>>> from ovs.flows.ofp import OFPFlow
>>> flow = OFPFlow("cookie=0x2b32ab4d, table=41, n_packets=11, n_bytes=462, priority=33,ip,reg15=0x2/0x2,nw_src=10.128.0.2/24 actions=move:NXM_OF_TCP_DST[]->NXM_NX_XXREG0[32..47],ct(table=16,zone=NXM_NX_REG13[0..15],nat)")
>>> flow.info
{'cookie': 724740941, 'table': 41, 'n_packets': 11, 'n_bytes': 462}
>>> flow.match
{'priority': 33, 'ip': True, 'reg15': Mask32('0x2/0x2'), 'nw_src': IPMask('10.128.0.2/24')}
>>> flow.actions
[{'move': {'src': {'field': 'NXM_OF_TCP_DST'}, 'dst': {'field': 'NXM_NX_XXREG0', 'start': 32, 'end': 47}}}, {'ct': {'table': 16, 'zone': {'field': 'NXM_NX_REG13', 'start': 0, 'end': 15}, 'nat': True}}]
>>> from ovs.flows.filter import OFFilter
>>> filt = OFFilter("nw_src ~= 10.128.0.10 and (table = 42 or n_packets > 0)")
>>> filt.evaluate(flow)
True

V2 -> V3:
- Simplified KV and list decoding code (Mark Michelson's suggestion)
- Fixed typos
- Added missing files to FLAKE8_PYFILES
- Go back to a simplified ipv4/6 regexp for ip-port range extraction.
  Also added specific unit test for ip-port range decoding.
- Adapt ofp encap() action decoder to support new header types: mpls and mpls_mc
  (the need for change was detected by patch 13)

V1 -> V2:
- list/kv parsers: changed the API to accept the string to parse in the
  constructor.
- list/kv parsers: allow re.split() to return less than 3 elements
  (enables support for python 3.6).
- decoders: add a more accurate IPv6 regexp and remove confusing max()
  in IPMask and EthMask.
- odp/ofp flows: remove the *Factory class and implement caching of
  decoders using class variables.
- odp/ofp flows: homogenize names of functions and made them static.
- moved pytest unit tests from a build target to a testsuite and added
  their requirements to python/test_requirements.txt.
- Formatting fixes and missing dots (lots of them!).

RFC -> V1:
- filters: created a class to represent the filtering result. That way
  we can store more information such as what key actually triggered the
  match. This enables functionality such as highlighting of keys based
  on an expression
- Formatted python code according to flake8 requirements
- Split ofp actions in ofp_act.py
- drop ofparse utility (will send a RFC to the mail list soon)
- Moved the initialization of the decoders objects to a factory so they
  are cached. This significantly decreases parsing time of large dumps.

Adrian Moreno (18):
  python: add generic Key-Value parser
  python: add mask, ip and eth decoders
  python: add list parser
  build-aux: split extract-ofp-fields
  build-aux: generate ofp field decoders
  python: add flow base class
  python: introduce OpenFlow Flow parsing
  python: add ovs datapath flow parsing
  python: add flow filtering syntax
  python: add a json encoder to flow fields
  tests: wrap ovs-ofctl calls to test python parser
  tests: Wrap test-odp to also run python parsers
  python: detect changes in flow formatting code
  python: introduce unit tests
  python: add unit tests for ListParser
  python: add unit tests for openflow parsing
  python: add unit tests to datapath parsing
  python: add unit tests for filtering engine

 .github/workflows/build-and-test.yml    |   3 +
 .gitignore                              |   1 +
 Documentation/intro/install/general.rst |   4 +
 Makefile.am                             |   3 +-
 build-aux/automake.mk                   |   6 +-
 build-aux/extract-ofp-fields            | 706 ++++++---------------
 build-aux/gen_ofp_field_decoders        |  69 +++
 python/.gitignore                       |   1 +
 python/automake.mk                      |  49 +-
 python/build/extract_ofp_fields.py      | 421 +++++++++++++
 python/build/flow-parse-deps.py         | 106 ++++
 python/ovs/flows/__init__.py            |   0
 python/ovs/flows/decoders.py            | 538 ++++++++++++++++
 python/ovs/flows/deps.py                |   5 +
 python/ovs/flows/filter.py              | 261 ++++++++
 python/ovs/flows/flow.py                | 125 ++++
 python/ovs/flows/kv.py                  | 314 ++++++++++
 python/ovs/flows/list.py                | 121 ++++
 python/ovs/flows/odp.py                 | 783 ++++++++++++++++++++++++
 python/ovs/flows/ofp.py                 | 428 +++++++++++++
 python/ovs/flows/ofp_act.py             | 306 +++++++++
 python/ovs/tests/test_decoders.py       | 130 ++++
 python/ovs/tests/test_filter.py         | 221 +++++++
 python/ovs/tests/test_kv.py             |  76 +++
 python/ovs/tests/test_list.py           |  66 ++
 python/ovs/tests/test_odp.py            | 527 ++++++++++++++++
 python/ovs/tests/test_ofp.py            | 534 ++++++++++++++++
 python/setup.py                         |   4 +-
 python/test_requirements.txt            |   3 +
 tests/atlocal.in                        |  19 +
 tests/automake.mk                       |   6 +
 tests/odp.at                            |  36 +-
 tests/ofp-actions.at                    |  46 +-
 tests/ovs-test-dpparse.py               |  82 +++
 tests/ovs-test-ofparse.py               | 107 ++++
 tests/pytest.at                         |   7 +
 tests/testsuite.at                      |   1 +
 37 files changed, 5550 insertions(+), 565 deletions(-)
 create mode 100755 build-aux/gen_ofp_field_decoders
 create mode 100644 python/build/extract_ofp_fields.py
 create mode 100755 python/build/flow-parse-deps.py
 create mode 100644 python/ovs/flows/__init__.py
 create mode 100644 python/ovs/flows/decoders.py
 create mode 100644 python/ovs/flows/deps.py
 create mode 100644 python/ovs/flows/filter.py
 create mode 100644 python/ovs/flows/flow.py
 create mode 100644 python/ovs/flows/kv.py
 create mode 100644 python/ovs/flows/list.py
 create mode 100644 python/ovs/flows/odp.py
 create mode 100644 python/ovs/flows/ofp.py
 create mode 100644 python/ovs/flows/ofp_act.py
 create mode 100644 python/ovs/tests/test_decoders.py
 create mode 100644 python/ovs/tests/test_filter.py
 create mode 100644 python/ovs/tests/test_kv.py
 create mode 100644 python/ovs/tests/test_list.py
 create mode 100644 python/ovs/tests/test_odp.py
 create mode 100644 python/ovs/tests/test_ofp.py
 create mode 100644 python/test_requirements.txt
 create mode 100755 tests/ovs-test-dpparse.py
 create mode 100755 tests/ovs-test-ofparse.py
 create mode 100644 tests/pytest.at