[ovs-dev,v1,00/18] lflow incremental processing

From: Numan Siddique <numans@ovn.org>

This patch series adds incremental processing in the lflow engine
node to handle changes to northd and other engine nodes.
Changed related to load balancers and NAT are mainly handled in
this patch series.

This patch series can also be found here - https://github.com/numansiddique/ovn/tree/northd_lflow_ip/v1

Prior to this patch series, most of the changes to northd engine
resulted in full recomputation of logical flows.  This series
aims to improve the performance of ovn-northd by adding the I-P
support.  In order to add this support, some of the northd engine
node data (from struct ovn_datapath) is split and moved over to
new engine nodes - mainly related to load balancers, NAT and ACLs.

objdep mgr is used for resource (ovn_port, ovn_datapath,
ovn_lb_datapaths) to logical flow references.

Below are the scale testing results done with these patches applied
using ovn-heater.  The test ran the scenario  -
ocp-500-density-heavy.yml [1].

With all the lflow I-P patches applied, the resuts are:

-------------------------------------------------------------------------------------------------------------------------------------------------------
                        Min (s)         Median (s)      90%ile (s)      99%ile (s)      Max (s)         Mean (s)        Total (s)       Count   Failed
-------------------------------------------------------------------------------------------------------------------------------------------------------
Iteration Total         0.136883        1.129016        1.192001        1.204167        1.212728        0.665017        83.127099       125     0
Namespace.add_ports     0.005216        0.005736        0.007034        0.015486        0.018978        0.006211        0.776373        125     0
WorkerNode.bind_port    0.035030        0.046082        0.052469        0.058293        0.060311        0.045973        11.493259       250     0
WorkerNode.ping_port    0.005057        0.006727        1.047692        1.069253        1.071336        0.266896        66.724094       250     0
-------------------------------------------------------------------------------------------------------------------------------------------------------

The results with the present main are:

-------------------------------------------------------------------------------------------------------------------------------------------------------
                        Min (s)         Median (s)      90%ile (s)      99%ile (s)      Max (s)         Mean (s)        Total (s)       Count   Failed
-------------------------------------------------------------------------------------------------------------------------------------------------------
Iteration Total         0.135491        2.223805        3.311270        3.339078        3.345346        1.729172        216.146495      125     0
Namespace.add_ports     0.005380        0.005744        0.006819        0.018773        0.020800        0.006292        0.786532        125     0
WorkerNode.bind_port    0.034179        0.046055        0.053488        0.058801        0.071043        0.046117        11.529311       250     0
WorkerNode.ping_port    0.004956        0.006952        3.086952        3.191743        3.192807        0.791544        197.886026      250     0
-------------------------------------------------------------------------------------------------------------------------------------------------------

[1] - https://github.com/ovn-org/ovn-heater/blob/main/test-scenarios/ocp-500-density-heavy.yml

Numan Siddique (18):
  northd: Refactor the northd change tracking.
  northd: Track ovn_datapaths in northd engine track data.
  tests: Add a couple of tests in ovn-northd for I-P.
  northd: Move router ports SB PB options sync to sync_to_sb_pb node.
  northd: Add a new engine 'lr-nat' to manage lr NAT data.
  northd: Add a new engine 'lr-lb-nat-data' to manage routers' lb data.
  northd:  Generate logical router's LB and NAT flows using
    lr_lbnat_data.
  northd: Don't commit dhcp response flows in the conntrack.
  northd: Add a new node ls_lbacls.
  northd: Refactor lflow management into a separate module.
  northd: Use lflow_ref when adding all logical flows.
  northd:  Move ovn_lb_datapaths from lib to northd module.
  northd: Handle lb changes in lflow engine.
  northd: Add lr_lb_nat_data handler for lflow engine node.
  northd: Add ls_lbacls handler for lflow engine node.
  northd:  Add I-P for NB_Global and SB_Global.
  northd: Add a noop handler for northd SB mac binding.
  northd: Add northd change handler for sync_to_sb_lb node.

 lib/lb.c                   |   96 -
 lib/lb.h                   |   57 -
 lib/objdep.h               |    2 +
 lib/ovn-util.c             |   17 +-
 lib/ovn-util.h             |    2 +-
 lib/stopwatch-names.h      |    3 +
 northd/aging.c             |   21 +-
 northd/automake.mk         |   12 +-
 northd/en-global-config.c  |  588 ++++
 northd/en-global-config.h  |   65 +
 northd/en-lflow.c          |  118 +-
 northd/en-lflow.h          |    8 +
 northd/en-lr-lb-nat-data.c |  666 ++++
 northd/en-lr-lb-nat-data.h |  103 +
 northd/en-lr-nat.c         |  480 +++
 northd/en-lr-nat.h         |  134 +
 northd/en-ls-lb-acls.c     |  530 +++
 northd/en-ls-lb-acls.h     |   92 +
 northd/en-northd.c         |   67 +-
 northd/en-northd.h         |    2 +-
 northd/en-port-group.h     |    3 +
 northd/en-sync-sb.c        |  513 ++-
 northd/inc-proc-northd.c   |   79 +-
 northd/lflow-mgr.c         | 1156 +++++++
 northd/lflow-mgr.h         |  195 ++
 northd/northd.c            | 6473 ++++++++++++++++--------------------
 northd/northd.h            |  546 ++-
 northd/ovn-northd.c        |    4 +
 tests/ovn-northd.at        |  858 ++++-
 29 files changed, 8804 insertions(+), 4086 deletions(-)
 create mode 100644 northd/en-global-config.c
 create mode 100644 northd/en-global-config.h
 create mode 100644 northd/en-lr-lb-nat-data.c
 create mode 100644 northd/en-lr-lb-nat-data.h
 create mode 100644 northd/en-lr-nat.c
 create mode 100644 northd/en-lr-nat.h
 create mode 100644 northd/en-ls-lb-acls.c
 create mode 100644 northd/en-ls-lb-acls.h
 create mode 100644 northd/lflow-mgr.c
 create mode 100644 northd/lflow-mgr.h