| Message ID | 20221202111817.204676-1-frode.nordahl@canonical.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [ovs-dev] docs: Extend upgrade documentation. | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | success | github build: passed |
| ovsrobot/github-robot-_ovn-kubernetes | success | github build: passed |
On 12/2/22 12:18, Frode Nordahl wrote: > As uncovered during the OVSCON'22 open discussion forum on > upgrades, there are some challenges in upgrading from older > versions of OVN. > > Document version requirements for performing a controller first > rolling upgrade. > > Add a section about how to perform a fail-safe upgrade for > deployments that want to upgrade beyond a supported version span. > > Reported-at: https://bugs.launchpad.net/bugs/1940043 > Signed-off-by: Frode Nordahl <frode.nordahl@canonical.com> > --- Hi Frode, Thanks a lot for taking the time to document this! It makes sense to me. I just have a couple of minor comments below. > Documentation/intro/install/ovn-upgrades.rst | 84 ++++++++++++++++++-- > 1 file changed, 78 insertions(+), 6 deletions(-) > > diff --git a/Documentation/intro/install/ovn-upgrades.rst b/Documentation/intro/install/ovn-upgrades.rst > index 4c131987e..a27fa3285 100644 > --- a/Documentation/intro/install/ovn-upgrades.rst > +++ b/Documentation/intro/install/ovn-upgrades.rst > @@ -27,7 +27,12 @@ OVN Upgrades > > Since OVN is a distributed system, special consideration must be given to > the process used to upgrade OVN across a deployment. This document discusses > -the recommended upgrade process. > +the two recommended `Upgrade procedures`_, `Rolling upgrade`_ and `Fail-safe > +upgrade`_. > + > +Which one to choose depends on whether you are running a version of OVN that is > +within range of upstream support for upgrades to the version of OVN you want to > +upgrade to. > > Release Notes > ------------- > @@ -43,8 +48,69 @@ upgraded together, partly for convenience. OVN is included in OVS releases > so it's easiest to upgrade them together. OVN may also make use of new > features of OVS only available in that release. > > +Upgrade procedures > +------------------ > + > +Rolling upgrade > +~~~~~~~~~~~~~~~ > + > +In order to successfully perform a rolling upgrade, the ovn-controller process > +needs to understand the structure of the database for the version you are > +upgrading from and to simultaneously. > + > +To avoid buildup of complexity and technical debt we limit the span of versions > +supported for a rolling upgrade on `Long-term Support Releases`_ (LTS), and it > +should always be possible to upgrade from the previous LTS version to the next. > + > +The first LTS version of OVN was 22.03. If you want to upgrade between other > +versions, you can use the `Fail-safe upgrade`_ procedure. > + Does it make sense to move this paragraph just after the 3 point upgrade sequence below? > +1. `Upgrade ovn-controller`_ > + > +2. `Upgrade OVN Databases and ovn-northd`_ > + > +3. `Upgrade OVN Integration`_ > + If we add it here it's more clear that the 3 steps are how a rolling upgrade should be performed. I found it a bit confusing in the current form as I was initially thinking that the 3 steps correspond to the "Fail-safe upgrade" procedure. > +Fail-safe upgrade > +~~~~~~~~~~~~~~~~~ > + > +When upgrading between a span of versions that is not supported, you may be at > +risk for the new ovn-controller process not understanding the structure of the > +old database, which may lead to data plane downtime for running instances. > + > +To avoid this there is a fail safe approach, which involves making the > +ovn-controller process refrain from making changes to the local flow state when > +a version mismatch between the ovn-controller and ovn-northd is detected. > + > +1. Upgrade to the most recent point release or package version available for > + the major version of OVN you are upgrading from. > + > +2. Enable the version pinning feature in the ovn-controller by setting the > + ``external_ids:ovn-match-northd-version`` flag to 'true' as documented in > + the `ovn-controller man page`_. > + > +3. If the version of OVN you are upgrading from does not have the version > + pinning check in the incremental processing engine, you can manually change I'm guessing you refer to this one, right? https://github.com/ovn-org/ovn/commit/c2eeb2c98ea860dbbc7eee5e9bae8a65769b0da3 Should we link to it then? > + the northd_internal_version to ensure the controllers go into fail-safe mode > + before processing changes induced by the upgrade. > + > + $ sudo ovn-sbctl set sb-global . options:northd_internal_version="foo" > + > +4. `Upgrade OVN Databases and ovn-northd`_ > + > +5. `Upgrade ovn-controller`_ > + > +6. `Upgrade OVN Integration`_ > + > +Steps > +----- > + > +This section documents individual steps in a upgrade procedure in no particular > +order. For information on ordering of the steps, please refer to the `Upgrade > +procedures`_ section. > + > Upgrade ovn-controller > ----------------------- > +~~~~~~~~~~~~~~~~~~~~~~ > > You should start by upgrading ovn-controller on each host it's running on. > First, you upgrade the OVS and OVN packages. Then, restart the > @@ -57,7 +123,7 @@ or with systemd:: > $ sudo systemd restart ovn-controller > > Upgrade OVN Databases and ovn-northd > ------------------------------------- > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > The OVN databases and ovn-northd should be upgraded next. Since ovn-controller > has already been upgraded, it will be ready to operate on any new functionality > @@ -76,7 +142,7 @@ or if you're using a Linux distribution with systemd:: > $ sudo systemctl restart ovn-northd > > Schema Change > -^^^^^^^^^^^^^ > ++++++++++++++ > > During database upgrading, if there is schema change, the DB file will be > converted to the new schema automatically, if the schema change is backward > @@ -103,8 +169,8 @@ of known impactible schema changes and how to fix when error encountered. > $ ovn-sbctl chassis-del <chassis> > > > -Upgrading OVN Integration > -------------------------- > +Upgrade OVN Integration > +~~~~~~~~~~~~~~~~~~~~~~~ > > Lastly, you may also want to upgrade integration with OVN that you may be > using. For example, this could be the OpenStack Neutron driver or > @@ -113,3 +179,9 @@ ovn-kubernetes. > OVN's northbound database schema is a backwards compatible interface, so > you should be able to safely complete an OVN upgrade before upgrading > any integration in use. > + > +.. LINKS > +.. _Long-term Support Releases: > + ../../internals/release-process.html#long-term-support-releases > +.. _ovn-controller man page: > + https://www.ovn.org/support/dist-docs/ovn-controller.8.html\ Thanks, Dumitru
diff --git a/Documentation/intro/install/ovn-upgrades.rst b/Documentation/intro/install/ovn-upgrades.rst index 4c131987e..a27fa3285 100644 --- a/Documentation/intro/install/ovn-upgrades.rst +++ b/Documentation/intro/install/ovn-upgrades.rst @@ -27,7 +27,12 @@ OVN Upgrades Since OVN is a distributed system, special consideration must be given to the process used to upgrade OVN across a deployment. This document discusses -the recommended upgrade process. +the two recommended `Upgrade procedures`_, `Rolling upgrade`_ and `Fail-safe +upgrade`_. + +Which one to choose depends on whether you are running a version of OVN that is +within range of upstream support for upgrades to the version of OVN you want to +upgrade to. Release Notes ------------- @@ -43,8 +48,69 @@ upgraded together, partly for convenience. OVN is included in OVS releases so it's easiest to upgrade them together. OVN may also make use of new features of OVS only available in that release. +Upgrade procedures +------------------ + +Rolling upgrade +~~~~~~~~~~~~~~~ + +In order to successfully perform a rolling upgrade, the ovn-controller process +needs to understand the structure of the database for the version you are +upgrading from and to simultaneously. + +To avoid buildup of complexity and technical debt we limit the span of versions +supported for a rolling upgrade on `Long-term Support Releases`_ (LTS), and it +should always be possible to upgrade from the previous LTS version to the next. + +The first LTS version of OVN was 22.03. If you want to upgrade between other +versions, you can use the `Fail-safe upgrade`_ procedure. + +1. `Upgrade ovn-controller`_ + +2. `Upgrade OVN Databases and ovn-northd`_ + +3. `Upgrade OVN Integration`_ + +Fail-safe upgrade +~~~~~~~~~~~~~~~~~ + +When upgrading between a span of versions that is not supported, you may be at +risk for the new ovn-controller process not understanding the structure of the +old database, which may lead to data plane downtime for running instances. + +To avoid this there is a fail safe approach, which involves making the +ovn-controller process refrain from making changes to the local flow state when +a version mismatch between the ovn-controller and ovn-northd is detected. + +1. Upgrade to the most recent point release or package version available for + the major version of OVN you are upgrading from. + +2. Enable the version pinning feature in the ovn-controller by setting the + ``external_ids:ovn-match-northd-version`` flag to 'true' as documented in + the `ovn-controller man page`_. + +3. If the version of OVN you are upgrading from does not have the version + pinning check in the incremental processing engine, you can manually change + the northd_internal_version to ensure the controllers go into fail-safe mode + before processing changes induced by the upgrade. + + $ sudo ovn-sbctl set sb-global . options:northd_internal_version="foo" + +4. `Upgrade OVN Databases and ovn-northd`_ + +5. `Upgrade ovn-controller`_ + +6. `Upgrade OVN Integration`_ + +Steps +----- + +This section documents individual steps in a upgrade procedure in no particular +order. For information on ordering of the steps, please refer to the `Upgrade +procedures`_ section. + Upgrade ovn-controller ----------------------- +~~~~~~~~~~~~~~~~~~~~~~ You should start by upgrading ovn-controller on each host it's running on. First, you upgrade the OVS and OVN packages. Then, restart the @@ -57,7 +123,7 @@ or with systemd:: $ sudo systemd restart ovn-controller Upgrade OVN Databases and ovn-northd ------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The OVN databases and ovn-northd should be upgraded next. Since ovn-controller has already been upgraded, it will be ready to operate on any new functionality @@ -76,7 +142,7 @@ or if you're using a Linux distribution with systemd:: $ sudo systemctl restart ovn-northd Schema Change -^^^^^^^^^^^^^ ++++++++++++++ During database upgrading, if there is schema change, the DB file will be converted to the new schema automatically, if the schema change is backward @@ -103,8 +169,8 @@ of known impactible schema changes and how to fix when error encountered. $ ovn-sbctl chassis-del <chassis> -Upgrading OVN Integration -------------------------- +Upgrade OVN Integration +~~~~~~~~~~~~~~~~~~~~~~~ Lastly, you may also want to upgrade integration with OVN that you may be using. For example, this could be the OpenStack Neutron driver or @@ -113,3 +179,9 @@ ovn-kubernetes. OVN's northbound database schema is a backwards compatible interface, so you should be able to safely complete an OVN upgrade before upgrading any integration in use. + +.. LINKS +.. _Long-term Support Releases: + ../../internals/release-process.html#long-term-support-releases +.. _ovn-controller man page: + https://www.ovn.org/support/dist-docs/ovn-controller.8.html
As uncovered during the OVSCON'22 open discussion forum on upgrades, there are some challenges in upgrading from older versions of OVN. Document version requirements for performing a controller first rolling upgrade. Add a section about how to perform a fail-safe upgrade for deployments that want to upgrade beyond a supported version span. Reported-at: https://bugs.launchpad.net/bugs/1940043 Signed-off-by: Frode Nordahl <frode.nordahl@canonical.com> --- Documentation/intro/install/ovn-upgrades.rst | 84 ++++++++++++++++++-- 1 file changed, 78 insertions(+), 6 deletions(-)