| Message ID | 20161207192814.27799-1-russell@ovn.org |
|---|---|
| State | Changes Requested |
| Headers | show |
On Wed, Dec 07, 2016 at 02:28:13PM -0500, Russell Bryant wrote:
> Signed-off-by: Russell Bryant <russell@ovn.org>
This procedure isn't what I expected. It recommends:
1. Upgrade SB/NB databases and northd.
2. Upgrade ovn-controller.
3. Upgrade integration.
I think that this is likely to cause problems in the common case,
because the new northd is likely to try to start using features that are
not yet available on the hypervisors, such as new logical match fields
and actions.
I expected:
1. Upgrade ovn-controller.
2. Upgrade SB/NB databases and northd.
3. Upgrade integration.
(I don't think that it matters when the databases are upgraded, though,
as long as they're upgraded before northd.)
On Wed, Dec 21, 2016 at 5:06 PM, Ben Pfaff <blp@ovn.org> wrote: > On Wed, Dec 07, 2016 at 02:28:13PM -0500, Russell Bryant wrote: > > Signed-off-by: Russell Bryant <russell@ovn.org> > > This procedure isn't what I expected. It recommends: > > 1. Upgrade SB/NB databases and northd. > 2. Upgrade ovn-controller. > 3. Upgrade integration. > > I think that this is likely to cause problems in the common case, > because the new northd is likely to try to start using features that are > not yet available on the hypervisors, such as new logical match fields > and actions. > > I expected: > > 1. Upgrade ovn-controller. > 2. Upgrade SB/NB databases and northd. > 3. Upgrade integration. > > (I don't think that it matters when the databases are upgraded, though, > as long as they're upgraded before northd.) > Thanks for the feedback! There's a possible problem in this case, too. ovn-controller could try to start using new fields in the southbound database that aren't there yet. Maybe it should be: 1. Upgrade SB/NB databases. 2. Upgrade ovn-controller. 3. Upgrade ovn-northd. 4. Upgrade integration.
On Wed, Dec 21, 2016 at 10:56:35PM -0500, Russell Bryant wrote: > On Wed, Dec 21, 2016 at 5:06 PM, Ben Pfaff <blp@ovn.org> wrote: > > > On Wed, Dec 07, 2016 at 02:28:13PM -0500, Russell Bryant wrote: > > > Signed-off-by: Russell Bryant <russell@ovn.org> > > > > This procedure isn't what I expected. It recommends: > > > > 1. Upgrade SB/NB databases and northd. > > 2. Upgrade ovn-controller. > > 3. Upgrade integration. > > > > I think that this is likely to cause problems in the common case, > > because the new northd is likely to try to start using features that are > > not yet available on the hypervisors, such as new logical match fields > > and actions. > > > > I expected: > > > > 1. Upgrade ovn-controller. > > 2. Upgrade SB/NB databases and northd. > > 3. Upgrade integration. > > > > (I don't think that it matters when the databases are upgraded, though, > > as long as they're upgraded before northd.) > > > > Thanks for the feedback! > > There's a possible problem in this case, too. ovn-controller could try to > start using new fields in the southbound database that aren't there yet. I guess that I have always considered that ovn-controller needs to tolerate some version-related variation in the database schema. This is a familiar problem, since ovs-vsctl and (to a lesser extent) ovs-vswitchd also need to tolerate similar variation. Different kinds of tolerance must be considered. Missing columns (because they were added in a later database schema) are one of the easiest, and the IDL takes care of that for us; the column will just appear empty. Over in OVS, we normally design the schema so that an empty column yields the default new behavior anyhow. The related_datapaths column that had been proposed for optimizing conditional monitoring was a tougher case for upgrade. With this proposal, an empty column or a missing one would break ovn-controller, since it wouldn't get all of the rows that it needed. Even if the column was present, it still wouldn't help if ovn-northd was too old to populate it correctly. I was about to propose that ovn-controller needed to figure out not just whether the column was present but whether ovn-northd was new enough and properly populating it, when I came up with the approach we're now using that doesn't need this column at all. So, upgrading the databases earlier cannot hurt, but I doubt that it solves many problems. > Maybe it should be: > > 1. Upgrade SB/NB databases. > 2. Upgrade ovn-controller. > 3. Upgrade ovn-northd. > 4. Upgrade integration. This ordering seems fine to me. Thank you for writing down the upgrade procedure. (Have we written down the installation procedure?)
Oops, I forgot to follow up on this ... On Thu, Dec 22, 2016 at 12:21 AM, Ben Pfaff <blp@ovn.org> wrote: > On Wed, Dec 21, 2016 at 10:56:35PM -0500, Russell Bryant wrote: > > On Wed, Dec 21, 2016 at 5:06 PM, Ben Pfaff <blp@ovn.org> wrote: > > > > > On Wed, Dec 07, 2016 at 02:28:13PM -0500, Russell Bryant wrote: > > > > Signed-off-by: Russell Bryant <russell@ovn.org> > > > > > > This procedure isn't what I expected. It recommends: > > > > > > 1. Upgrade SB/NB databases and northd. > > > 2. Upgrade ovn-controller. > > > 3. Upgrade integration. > > > > > > I think that this is likely to cause problems in the common case, > > > because the new northd is likely to try to start using features that > are > > > not yet available on the hypervisors, such as new logical match fields > > > and actions. > > > > > > I expected: > > > > > > 1. Upgrade ovn-controller. > > > 2. Upgrade SB/NB databases and northd. > > > 3. Upgrade integration. > > > > > > (I don't think that it matters when the databases are upgraded, though, > > > as long as they're upgraded before northd.) > > > > > > > Thanks for the feedback! > > > > There's a possible problem in this case, too. ovn-controller could try > to > > start using new fields in the southbound database that aren't there yet. > > I guess that I have always considered that ovn-controller needs to > tolerate some version-related variation in the database schema. This is > a familiar problem, since ovs-vsctl and (to a lesser extent) > ovs-vswitchd also need to tolerate similar variation. > > Different kinds of tolerance must be considered. Missing columns > (because they were added in a later database schema) are one of the > easiest, and the IDL takes care of that for us; the column will just > appear empty. Over in OVS, we normally design the schema so that an > empty column yields the default new behavior anyhow. > > The related_datapaths column that had been proposed for optimizing > conditional monitoring was a tougher case for upgrade. With this > proposal, an empty column or a missing one would break ovn-controller, > since it wouldn't get all of the rows that it needed. Even if the > column was present, it still wouldn't help if ovn-northd was too old to > populate it correctly. I was about to propose that ovn-controller > needed to figure out not just whether the column was present but whether > ovn-northd was new enough and properly populating it, when I came up > with the approach we're now using that doesn't need this column at all. > > So, upgrading the databases earlier cannot hurt, but I doubt that it > solves many problems. > Thanks for the info. > > > Maybe it should be: > > > > 1. Upgrade SB/NB databases. > > 2. Upgrade ovn-controller. > > 3. Upgrade ovn-northd. > > 4. Upgrade integration. > > This ordering seems fine to me. > so we could do the above, or based on your comments: 1. Upgrade ovn-controller. 2. Upgrade databases. 3. Upgrade ovn-northd. 4. Upgrade integration. This one is a little easier to document since 2&3 can be done with a single command, so I'll just document it that way. > Thank you for writing down the upgrade procedure. (Have we written down > the installation procedure?) > No, I don't think so. We have options well documented, but not an OVN install guide. It tends to be covered along with integration guides, though. For example: https://github.com/openvswitch/ovn-kubernetes or http://docs.openstack.org/developer/networking-ovn/install.html but I think a standalone install guide would be good to have, too.
diff --git a/ovn/automake.mk b/ovn/automake.mk index 3bcc1d5..6905061 100644 --- a/ovn/automake.mk +++ b/ovn/automake.mk @@ -73,7 +73,8 @@ DISTCLEANFILES += ovn/ovn-architecture.7 EXTRA_DIST += \ ovn/TODO.rst \ ovn/CONTAINERS.OpenStack.rst \ - ovn/OVN-GW-HA.rst + ovn/OVN-GW-HA.rst \ + ovn/docs/upgrades.rst # Version checking for ovn-nb.ovsschema. ALL_LOCAL += ovn/ovn-nb.ovsschema.stamp diff --git a/ovn/docs/upgrades.rst b/ovn/docs/upgrades.rst new file mode 100644 index 0000000..b5f8bb0 --- /dev/null +++ b/ovn/docs/upgrades.rst @@ -0,0 +1,89 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +============ +OVN Upgrades +============ + +Since OVN is a distributed system, special consideration must be given to +the process used to upgrade OVN across a deployment. This document discusses +the recommended upgrade process. + +Release Notes +------------- + +You should always check the OVS and OVN release notes (NEWS file) for any +release specific notes on upgrades. + +OVS +--- + +OVN depends on and is included with OVS. It's expected that OVS and OVN are +upgraded together, partly for convenience. OVN is included in OVS releases +so it's easiest to upgrade them together. OVN may also make use of new +features of OVS only available in that release. + +Upgrade OVN Databases and ovn-northd +------------------------------------ + +The OVN databases and ovn-northd should be upgraded first. The OVN database +schemas are versioned and are only changed in a backwards compatible way. +This lets you upgrade the database while still allowing older clients of the +database to continue operating as you do a rolling upgrade through the +environment. + +Upgrading the OVN packages installs everything needed for an upgrade. The only +step required after upgrading the packages is to restart ovn-northd, which +automatically restarts the databases and upgrades the database schema, as well. + +You may perform this restart using the ovn-ctl script:: + + $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_northd + +or if you're using a Linux distribution with systemd:: + + $ sudo systemctl restart ovn-northd + +Upgrade ovn-controller +---------------------- + +Next you should upgrade ovn-controller on each host its running on. +First, you upgrade the OVS and OVN packages. Then, restart the +ovn-controller service. You can restart with ovn-ctl:: + + $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_controller + +or with systemd:: + + $ sudo systemd restart ovn-controller + +Upgrading OVN Integration +------------------------- + +Lastly, you may also want to upgrade integration with OVN that you may be +using. For example, this could be the OpenStack Neutron driver or +ovn-kubernetes. + +OVN's northbound database schema is a backwards compatible interface, so +you should be able to safely complete an OVN upgrade before upgrading +any integration in use.
Signed-off-by: Russell Bryant <russell@ovn.org> --- ovn/automake.mk | 3 +- ovn/docs/upgrades.rst | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) create mode 100644 ovn/docs/upgrades.rst