From patchwork Tue Oct 24 00:42:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Numan Siddique X-Patchwork-Id: 1854073 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SDtZy510Jz23jn for ; Tue, 24 Oct 2023 11:43:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 157D683C02; Tue, 24 Oct 2023 00:43:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 157D683C02 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A5Bhx2wE-1Ao; Tue, 24 Oct 2023 00:43:17 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 04B8D83BF5; Tue, 24 Oct 2023 00:43:16 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 04B8D83BF5 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CE7C0C0071; Tue, 24 Oct 2023 00:43:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 415B6C0032 for ; Tue, 24 Oct 2023 00:43:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0F89D612BC for ; Tue, 24 Oct 2023 00:43:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 0F89D612BC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eNkf3v8Dq4-W for ; Tue, 24 Oct 2023 00:43:04 +0000 (UTC) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0937661088 for ; Tue, 24 Oct 2023 00:43:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 0937661088 Received: by mail.gandi.net (Postfix) with ESMTPSA id 7197820002; Tue, 24 Oct 2023 00:42:58 +0000 (UTC) From: numans@ovn.org To: dev@openvswitch.org Date: Mon, 23 Oct 2023 20:42:44 -0400 Message-ID: <20231024004244.4133452-1-numans@ovn.org> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-GND-Sasl: numans@ovn.org Subject: [ovs-dev] [PATCH ovn v1 00/18] lflow incremental processing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Numan Siddique This patch series adds incremental processing in the lflow engine node to handle changes to northd and other engine nodes. Changed related to load balancers and NAT are mainly handled in this patch series. This patch series can also be found here - https://github.com/numansiddique/ovn/tree/northd_lflow_ip/v1 Prior to this patch series, most of the changes to northd engine resulted in full recomputation of logical flows. This series aims to improve the performance of ovn-northd by adding the I-P support. In order to add this support, some of the northd engine node data (from struct ovn_datapath) is split and moved over to new engine nodes - mainly related to load balancers, NAT and ACLs. objdep mgr is used for resource (ovn_port, ovn_datapath, ovn_lb_datapaths) to logical flow references. Below are the scale testing results done with these patches applied using ovn-heater. The test ran the scenario - ocp-500-density-heavy.yml [1]. With all the lflow I-P patches applied, the resuts are: ------------------------------------------------------------------------------------------------------------------------------------------------------- Min (s) Median (s) 90%ile (s) 99%ile (s) Max (s) Mean (s) Total (s) Count Failed ------------------------------------------------------------------------------------------------------------------------------------------------------- Iteration Total 0.136883 1.129016 1.192001 1.204167 1.212728 0.665017 83.127099 125 0 Namespace.add_ports 0.005216 0.005736 0.007034 0.015486 0.018978 0.006211 0.776373 125 0 WorkerNode.bind_port 0.035030 0.046082 0.052469 0.058293 0.060311 0.045973 11.493259 250 0 WorkerNode.ping_port 0.005057 0.006727 1.047692 1.069253 1.071336 0.266896 66.724094 250 0 ------------------------------------------------------------------------------------------------------------------------------------------------------- The results with the present main are: ------------------------------------------------------------------------------------------------------------------------------------------------------- Min (s) Median (s) 90%ile (s) 99%ile (s) Max (s) Mean (s) Total (s) Count Failed ------------------------------------------------------------------------------------------------------------------------------------------------------- Iteration Total 0.135491 2.223805 3.311270 3.339078 3.345346 1.729172 216.146495 125 0 Namespace.add_ports 0.005380 0.005744 0.006819 0.018773 0.020800 0.006292 0.786532 125 0 WorkerNode.bind_port 0.034179 0.046055 0.053488 0.058801 0.071043 0.046117 11.529311 250 0 WorkerNode.ping_port 0.004956 0.006952 3.086952 3.191743 3.192807 0.791544 197.886026 250 0 ------------------------------------------------------------------------------------------------------------------------------------------------------- [1] - https://github.com/ovn-org/ovn-heater/blob/main/test-scenarios/ocp-500-density-heavy.yml Numan Siddique (18): northd: Refactor the northd change tracking. northd: Track ovn_datapaths in northd engine track data. tests: Add a couple of tests in ovn-northd for I-P. northd: Move router ports SB PB options sync to sync_to_sb_pb node. northd: Add a new engine 'lr-nat' to manage lr NAT data. northd: Add a new engine 'lr-lb-nat-data' to manage routers' lb data. northd: Generate logical router's LB and NAT flows using lr_lbnat_data. northd: Don't commit dhcp response flows in the conntrack. northd: Add a new node ls_lbacls. northd: Refactor lflow management into a separate module. northd: Use lflow_ref when adding all logical flows. northd: Move ovn_lb_datapaths from lib to northd module. northd: Handle lb changes in lflow engine. northd: Add lr_lb_nat_data handler for lflow engine node. northd: Add ls_lbacls handler for lflow engine node. northd: Add I-P for NB_Global and SB_Global. northd: Add a noop handler for northd SB mac binding. northd: Add northd change handler for sync_to_sb_lb node. lib/lb.c | 96 - lib/lb.h | 57 - lib/objdep.h | 2 + lib/ovn-util.c | 17 +- lib/ovn-util.h | 2 +- lib/stopwatch-names.h | 3 + northd/aging.c | 21 +- northd/automake.mk | 12 +- northd/en-global-config.c | 588 ++++ northd/en-global-config.h | 65 + northd/en-lflow.c | 118 +- northd/en-lflow.h | 8 + northd/en-lr-lb-nat-data.c | 666 ++++ northd/en-lr-lb-nat-data.h | 103 + northd/en-lr-nat.c | 480 +++ northd/en-lr-nat.h | 134 + northd/en-ls-lb-acls.c | 530 +++ northd/en-ls-lb-acls.h | 92 + northd/en-northd.c | 67 +- northd/en-northd.h | 2 +- northd/en-port-group.h | 3 + northd/en-sync-sb.c | 513 ++- northd/inc-proc-northd.c | 79 +- northd/lflow-mgr.c | 1156 +++++++ northd/lflow-mgr.h | 195 ++ northd/northd.c | 6473 ++++++++++++++++-------------------- northd/northd.h | 546 ++- northd/ovn-northd.c | 4 + tests/ovn-northd.at | 858 ++++- 29 files changed, 8804 insertions(+), 4086 deletions(-) create mode 100644 northd/en-global-config.c create mode 100644 northd/en-global-config.h create mode 100644 northd/en-lr-lb-nat-data.c create mode 100644 northd/en-lr-lb-nat-data.h create mode 100644 northd/en-lr-nat.c create mode 100644 northd/en-lr-nat.h create mode 100644 northd/en-ls-lb-acls.c create mode 100644 northd/en-ls-lb-acls.h create mode 100644 northd/lflow-mgr.c create mode 100644 northd/lflow-mgr.h