From patchwork Fri Dec 2 11:18:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1711369 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=nfEW3PZM; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NNr6J4pYCz23nC for ; Fri, 2 Dec 2022 22:18:30 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 03DA2415F2; Fri, 2 Dec 2022 11:18:28 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 03DA2415F2 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=nfEW3PZM X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X4vhCpTVxFRs; Fri, 2 Dec 2022 11:18:26 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 61CFF41511; Fri, 2 Dec 2022 11:18:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 61CFF41511 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 237D2C0033; Fri, 2 Dec 2022 11:18:25 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 631F3C002D for ; Fri, 2 Dec 2022 11:18:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3125140106 for ; Fri, 2 Dec 2022 11:18:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3125140106 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=nfEW3PZM X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ruh4GbZatHtD for ; Fri, 2 Dec 2022 11:18:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 9CCE9400D2 Received: from smtp-relay-canonical-1.canonical.com (smtp-relay-canonical-1.canonical.com [185.125.188.121]) by smtp2.osuosl.org (Postfix) with ESMTPS id 9CCE9400D2 for ; Fri, 2 Dec 2022 11:18:21 +0000 (UTC) Received: from frode-threadripper.. (ti0189a330-0102.bb.online.no [88.91.31.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-1.canonical.com (Postfix) with ESMTPSA id 604BA41F52; Fri, 2 Dec 2022 11:18:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1669979898; bh=+gsnk6Hyz2VR/D+5CGZA9ivE1mR6ov03xdVf9jnHVcY=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=nfEW3PZMUzzZXcxuiRPV+yORISl/qyYA+a+YF5qHMiISK4O9WYqFdxBFsHC4Pj66F iUy6LKnNFgSMh1nJ5fpA/Sv/NSfSI2Sacvg0DIYDFzMSh5m3tAt0fYrin5JS/KzVHt pSl7a4Vy0Gb9oU2b7SM2IpQ03bqpyVYYOXc1+4AZaUa8MxDEtIFnAlNMCQ50KZq4DB cKD99KksT/ts03iKIaggcE8TN0mNYfoAf0QwlwpQLZ7DiP9CCbBeVIIpffD3TvPR9C e6o97wqLtIXS8D6XvB48PhGnfgVbjsQVeBY1vCDhrPzhReuUuH9e+VrnDDCHoJCGhp yvwtTRe6sYuCw== From: Frode Nordahl To: dev@openvswitch.org Date: Fri, 2 Dec 2022 12:18:17 +0100 Message-Id: <20221202111817.204676-1-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 Cc: Dumitru Ceara Subject: [ovs-dev] [PATCH ovn] docs: Extend upgrade documentation. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" As uncovered during the OVSCON'22 open discussion forum on upgrades, there are some challenges in upgrading from older versions of OVN. Document version requirements for performing a controller first rolling upgrade. Add a section about how to perform a fail-safe upgrade for deployments that want to upgrade beyond a supported version span. Reported-at: https://bugs.launchpad.net/bugs/1940043 Signed-off-by: Frode Nordahl --- Documentation/intro/install/ovn-upgrades.rst | 84 ++++++++++++++++++-- 1 file changed, 78 insertions(+), 6 deletions(-) diff --git a/Documentation/intro/install/ovn-upgrades.rst b/Documentation/intro/install/ovn-upgrades.rst index 4c131987e..a27fa3285 100644 --- a/Documentation/intro/install/ovn-upgrades.rst +++ b/Documentation/intro/install/ovn-upgrades.rst @@ -27,7 +27,12 @@ OVN Upgrades Since OVN is a distributed system, special consideration must be given to the process used to upgrade OVN across a deployment. This document discusses -the recommended upgrade process. +the two recommended `Upgrade procedures`_, `Rolling upgrade`_ and `Fail-safe +upgrade`_. + +Which one to choose depends on whether you are running a version of OVN that is +within range of upstream support for upgrades to the version of OVN you want to +upgrade to. Release Notes ------------- @@ -43,8 +48,69 @@ upgraded together, partly for convenience. OVN is included in OVS releases so it's easiest to upgrade them together. OVN may also make use of new features of OVS only available in that release. +Upgrade procedures +------------------ + +Rolling upgrade +~~~~~~~~~~~~~~~ + +In order to successfully perform a rolling upgrade, the ovn-controller process +needs to understand the structure of the database for the version you are +upgrading from and to simultaneously. + +To avoid buildup of complexity and technical debt we limit the span of versions +supported for a rolling upgrade on `Long-term Support Releases`_ (LTS), and it +should always be possible to upgrade from the previous LTS version to the next. + +The first LTS version of OVN was 22.03. If you want to upgrade between other +versions, you can use the `Fail-safe upgrade`_ procedure. + +1. `Upgrade ovn-controller`_ + +2. `Upgrade OVN Databases and ovn-northd`_ + +3. `Upgrade OVN Integration`_ + +Fail-safe upgrade +~~~~~~~~~~~~~~~~~ + +When upgrading between a span of versions that is not supported, you may be at +risk for the new ovn-controller process not understanding the structure of the +old database, which may lead to data plane downtime for running instances. + +To avoid this there is a fail safe approach, which involves making the +ovn-controller process refrain from making changes to the local flow state when +a version mismatch between the ovn-controller and ovn-northd is detected. + +1. Upgrade to the most recent point release or package version available for + the major version of OVN you are upgrading from. + +2. Enable the version pinning feature in the ovn-controller by setting the + ``external_ids:ovn-match-northd-version`` flag to 'true' as documented in + the `ovn-controller man page`_. + +3. If the version of OVN you are upgrading from does not have the version + pinning check in the incremental processing engine, you can manually change + the northd_internal_version to ensure the controllers go into fail-safe mode + before processing changes induced by the upgrade. + + $ sudo ovn-sbctl set sb-global . options:northd_internal_version="foo" + +4. `Upgrade OVN Databases and ovn-northd`_ + +5. `Upgrade ovn-controller`_ + +6. `Upgrade OVN Integration`_ + +Steps +----- + +This section documents individual steps in a upgrade procedure in no particular +order. For information on ordering of the steps, please refer to the `Upgrade +procedures`_ section. + Upgrade ovn-controller ----------------------- +~~~~~~~~~~~~~~~~~~~~~~ You should start by upgrading ovn-controller on each host it's running on. First, you upgrade the OVS and OVN packages. Then, restart the @@ -57,7 +123,7 @@ or with systemd:: $ sudo systemd restart ovn-controller Upgrade OVN Databases and ovn-northd ------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The OVN databases and ovn-northd should be upgraded next. Since ovn-controller has already been upgraded, it will be ready to operate on any new functionality @@ -76,7 +142,7 @@ or if you're using a Linux distribution with systemd:: $ sudo systemctl restart ovn-northd Schema Change -^^^^^^^^^^^^^ ++++++++++++++ During database upgrading, if there is schema change, the DB file will be converted to the new schema automatically, if the schema change is backward @@ -103,8 +169,8 @@ of known impactible schema changes and how to fix when error encountered. $ ovn-sbctl chassis-del -Upgrading OVN Integration -------------------------- +Upgrade OVN Integration +~~~~~~~~~~~~~~~~~~~~~~~ Lastly, you may also want to upgrade integration with OVN that you may be using. For example, this could be the OpenStack Neutron driver or @@ -113,3 +179,9 @@ ovn-kubernetes. OVN's northbound database schema is a backwards compatible interface, so you should be able to safely complete an OVN upgrade before upgrading any integration in use. + +.. LINKS +.. _Long-term Support Releases: + ../../internals/release-process.html#long-term-support-releases +.. _ovn-controller man page: + https://www.ovn.org/support/dist-docs/ovn-controller.8.html