From patchwork Mon Aug 13 17:47:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 957071 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HPQcO75M"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41q39t2kvtz9sBJ for ; Tue, 14 Aug 2018 03:48:33 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id B2FAFCD2; Mon, 13 Aug 2018 17:48:31 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 6A70B9D7 for ; Mon, 13 Aug 2018 17:48:30 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C5E41765 for ; Mon, 13 Aug 2018 17:48:29 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id y10-v6so7993926pfn.8 for ; Mon, 13 Aug 2018 10:48:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=Sfl5IHf288QHXp0FGiKeG76igD5tbw566ylrFgwOV/M=; b=HPQcO75MnRF5BzINU+Rw9XVQaBOEbwOTBfkp7U+yDDEG0VxMpfHTcoXltaZ/lYa+mr 9SRZUmVA0VJPE6enqRTYkeO6sJdb7lscRfx2TXe8FI7eKgbzxTePwFBx4+ZAOoDoKYfw XboolEhxJm59RpdA1RkcAEp8h5cyUbTvP5m1E+DF8y61ncAQ8h8zmzJ6eKBuiu9CPWba KKmNkEyf1CYSisOJOQsHcUqqDrVyIG8P8Vc+CvBcMzTJ4eP4bIyYJXe+SZtX03D0swes WA43MCT6QJ9onsoeAWhiZtNGn3ZPQ2mHNosMRuTEYaXV2NJfHR9li9+8q28pY0Tt6Nhj /FmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=Sfl5IHf288QHXp0FGiKeG76igD5tbw566ylrFgwOV/M=; b=SiLX9zGffH+o4LqRH76WJNwpHAcmOxNEOV678/JPTQdd/xnM6OXPyzHYOFBtjJ6a2x LWljBsI4JZMaWV1ysZ1uYwKf7pnEXvHw5jtlvdIQwjXcydXNb277J2UYfsMvAhu+7SvM 2VmFIISYwQARsWcg3HJxkAkxBzOdJ60e4j0UNQY2SENIGNbL8Ip8J6u5+ApjsZfCLwiw Ya/AqUf/F1ail07R3O9N8uurDpkX06/XRRxXGOMUGChQGussrZUHuXCDFrIJASi4dVqI RkCFN6NxvLw+OmlTvRQZCcHdUWDSrKe5x5fso33edR0xtgFzG3IOz2IrnuPn2QWN5hHK zm4w== X-Gm-Message-State: AOUpUlGcIUfswLMx9MSMJsIfmamb9R7q0QDpUMO+LmOrpHS1wK4e1pTA oQ3mluZIiMiK2AK+l0e7sMCC+XXQ X-Google-Smtp-Source: AA+uWPwf9CV8BW2BrQYiTzMD2oIIlTuNroN5EBacQwYUm0yimzgMjQr3s1ssOuaU7F6h+/yvseHSrA== X-Received: by 2002:a62:13ca:: with SMTP id 71-v6mr20226274pft.34.1534182509076; Mon, 13 Aug 2018 10:48:29 -0700 (PDT) Received: from localhost.localdomain.localdomain ([216.113.160.70]) by smtp.gmail.com with ESMTPSA id d22-v6sm38068664pfk.69.2018.08.13.10.48.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Aug 2018 10:48:28 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Mon, 13 Aug 2018 10:47:59 -0700 Message-Id: <1534182499-75914-1-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v5 00/20] ovn-controller incremental processing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org ovn-controller currently recomputes everything when there are any changes of input, which leads to high CPU usages and slow in end-to-end flow enforcement in response to changes. It even wastes CPU to recompute flows for unrelated inputs such as pinctrl events. This patch series implements incremental processing in ovn-controller to solve above problems. There has been a similar attempt of solve the problem earlier but was reverted (see commit: 926c34fd). This patch series takes a different approach with an incremental processing engine, to make the dependencies clear and easier to maintain. For performance test results please find them in commit messages. The major change from v3 to v5 is better dependency maintenance. Firstly, it forces explicit dependency exposure for OVSDB tables used by any engine nodes, thanks for Ben's patches that supported passing individual tables. Since we are not passing IDL directly to the engine, missing dependency will either make compiling failed or result in assert failure in the first run. Secondly, it forces implicit dependency exposure for OVSDB table references through change tracking implementation, so that any referenced row change will cause the referencing node change, which will trigger proper change handling in the engine. Because of this capability, in change handler of port-binding, the logic is simplied a lot so that we don't have to re- consider multicast_group when port-binding is processed. Instead, change handler of multicast_group will automatically be triggered, and it will be incrementally processed, too. Some more notes: * In engine node any global variable access should be wrapped as engine node input. One example is mff_ovn_geneve, which is implemented as a separate engine node (an input node). However, currently there is no way to force this. So it depends on developers' awareness and code reviews. (Currently I don't see any global variables directly accessed from engine.) * Although dependencies between nodes are clear, the correctness of implementing change handler within an engine node relies on developers only - there is no mechanism yet to guarantee the outcome of change handler implementation is equivalent to full recompute, i.e. the run() function. * There can be further optimization when tables being referenced is not accessed, so that a change may not have to be handled by a specific engine node. This may be achieved by defining different versions of tables with different access permissions to referenced tables, as demonstrated by Ben's RFC: https://mail.openvswitch.org/pipermail/ovs-dev/2018-June/348454.html. However, it seems not necessary at this point for most frequent use cases that we are targeting for performance improvement. For example, port-binding is referenced by multicast-group, but we do need to access port-binding from multicast-group in flow-output engine node, and both of them are incrementally processed. There is no other table references for the incremental processing we have in this patch series (port-binding, multicast-group, logical-flow, address-set, port-group). If we do have such use cases in the future, we may need to implement change tracking for different versions of the same table. --- v4 -> v5: * Fix the rebasing issue (compiling failure) in patch 6/20. v3 -> v4: * Rebased on master, and also reorganized patches by squashing some of them. * Forced explicit dependency exposure for OVSDB tables used by any engine node, thanks for Ben's patches that supported passing individual tables. * Forced implicit dependency exposure for OVSDB table references through change tracking. * Expose mff_ovn_geneve as a separate engine node. * Added Jakub's incremental test cases in the series, and added new tests for port-group incremental processing. * Added test result of address-set i-p from Mark Michelson in commit message. * Some general improvements in IDL (the first 3 patches) Han Zhou (18): ovsdb-idl.c: Remove a misleading comment for change tracking. ovsdb-idl.c: Update conditions when handling change tracking list. ovsdb-idlc.in: Support more interfaces for passing pointers of individual tables. ovsdb-idl.c: Track changes for table references. ovn-controller: Incremental processing engine ovn-controller: Track OVSDB changes ovn-controller: Initial use of incremental engine. ovn-controller: Incremental logical flow processing ovn-controller: runtime_data change handler for SB port-binding ovn-controller: port-binding incremental processing for physical flows ovn-controller: incremental processing for multicast group changes ovsdb-idl: Tracking - preserve data for deleted rows. ovn-controller: Split addr_sets from runtime_data. ovn-controller: Maintain resource references for logical flows. ovn-controller: Incremental processing for address-set changes. ovn-controller: Split port_groups from runtime_data. ovn-controller: Incremental processing for port-group changes. ovn-performance.at: Test port group incremental processing. Jakub Sitnicki (2): coverage: Add command for reading counter value. ovn: Test for full logical flow processing in ovn-controller. include/ovn/actions.h | 3 + include/ovn/expr.h | 5 +- lib/coverage-unixctl.man | 2 + lib/coverage.c | 42 ++ lib/ovsdb-idl-provider.h | 2 + lib/ovsdb-idl.c | 75 +- ovn/controller/bfd.c | 4 +- ovn/controller/binding.c | 108 ++- ovn/controller/binding.h | 7 + ovn/controller/encaps.c | 12 +- ovn/controller/lflow.c | 428 +++++++++++- ovn/controller/lflow.h | 100 ++- ovn/controller/ofctrl.c | 261 +++++-- ovn/controller/ofctrl.h | 33 +- ovn/controller/ovn-controller.c | 1454 +++++++++++++++++++++++++++++++++------ ovn/controller/physical.c | 194 ++++-- ovn/controller/physical.h | 19 +- ovn/lib/actions.c | 11 +- ovn/lib/automake.mk | 4 +- ovn/lib/expr.c | 21 +- ovn/lib/extend-table.c | 60 +- ovn/lib/extend-table.h | 16 +- ovn/lib/inc-proc-eng.c | 201 ++++++ ovn/lib/inc-proc-eng.h | 240 +++++++ ovn/utilities/ovn-trace.c | 2 +- ovsdb/ovsdb-idlc.in | 25 + tests/automake.mk | 3 +- tests/ovn-performance.at | 420 +++++++++++ tests/ovn.at | 74 ++ tests/test-ovn.c | 7 +- tests/testsuite.at | 1 + 31 files changed, 3382 insertions(+), 452 deletions(-) create mode 100644 ovn/lib/inc-proc-eng.c create mode 100644 ovn/lib/inc-proc-eng.h create mode 100644 tests/ovn-performance.at