From patchwork Sun Oct 30 13:30:09 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stephen Finucane X-Patchwork-Id: 688947 X-Patchwork-Delegate: rbryant@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3t6JR50JHxz9t1T for ; Mon, 31 Oct 2016 00:35:44 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="key not found in DNS" (0-bit key; unprotected) header.d=that.guru header.i=@that.guru header.b=HbutkajQ; dkim-atps=neutral Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 0051210605; Sun, 30 Oct 2016 06:35:32 -0700 (PDT) X-Original-To: dev@openvswitch.org Delivered-To: dev@openvswitch.org Received: from mx3v3.cudamail.com (mx3.cudamail.com [64.34.241.5]) by archives.nicira.com (Postfix) with ESMTPS id 2AB73105CC for ; Sun, 30 Oct 2016 06:35:30 -0700 (PDT) Received: from bar6.cudamail.com (localhost [127.0.0.1]) by mx3v3.cudamail.com (Postfix) with ESMTPS id B7C5E164068 for ; Sun, 30 Oct 2016 07:35:29 -0600 (MDT) X-ASG-Debug-ID: 1477834527-0b323720422290f0001-byXFYA Received: from mx3-pf3.cudamail.com ([192.168.14.3]) by bar6.cudamail.com with ESMTP id ceTECtqoocsDwmdq (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 30 Oct 2016 07:35:27 -0600 (MDT) X-Barracuda-Envelope-From: stephen@that.guru X-Barracuda-RBL-Trusted-Forwarder: 192.168.14.3 Received: from unknown (HELO nov-007-i631.relay.mailchannels.net) (46.232.183.185) by mx3-pf3.cudamail.com with ESMTPS (DHE-RSA-AES256-SHA encrypted); 30 Oct 2016 13:35:25 -0000 Received-SPF: none (mx3-pf3.cudamail.com: domain at that.guru does not designate permitted sender hosts) X-Barracuda-Apparent-Source-IP: 46.232.183.185 X-Barracuda-RBL-IP: 46.232.183.185 X-Sender-Id: mxroute|x-authuser|stephen@that.guru Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 957431BD34B for ; Sun, 30 Oct 2016 13:35:19 +0000 (UTC) Received: from one.mxroute.com (ip-10-107-69-155.us-west-2.compute.internal [10.107.69.155]) by relay.mailchannels.net (Postfix) with ESMTPA id 00E6A1BC07F for ; Sun, 30 Oct 2016 13:35:18 +0000 (UTC) X-Sender-Id: mxroute|x-authuser|stephen@that.guru Received: from one.mxroute.com ([TEMPUNAVAIL]. [10.104.207.55]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384) by 0.0.0.0:2500 (trex/5.7.8); Sun, 30 Oct 2016 13:35:19 +0000 X-MC-Relay: Neutral X-MailChannels-SenderId: mxroute|x-authuser|stephen@that.guru X-MailChannels-Auth-Id: mxroute X-MC-Loop-Signature: 1477834519401:3492521228 X-MC-Ingress-Time: 1477834519400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=that.guru; s=default; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=dG23px54uxgttjeulBT4xQUGtvOtavf4laM6R7oc8eI=; b=HbutkajQgwblVLykj6QC0YKsIK WYLSLPdUaDlW1JI/zZ1O+vGwd1LUqQKcH9zQVXMpVuXoywVqdOfx7B0A2oI0IIh9fVKiIEI0SNPr2 cy2pgRDm2VHWrFEHrWrUL0Y+u2gAjAr0EdjWPyqLTvIBZf9EJkkcX4KXXIKlXd0h7duK85d7CweHq vC23hmd4PWZVtTRpE6sO65z4oun2QZg4R8k42EnA+efrf6pRDwd/7AIQOm7xPP49U8ozN6m748tLF y/XnXoc/aoHz+OB0bTwC7ZEcraSA6qoXxUEtJ3z1L/HAK4NfnFWjbkGvKr42maZaEbVk6lGKiagcC pBY/FLrg==; X-CudaMail-Envelope-Sender: stephen@that.guru From: Stephen Finucane To: dev@openvswitch.org X-CudaMail-MID: CM-V3-1029004477 X-CudaMail-DTE: 103016 X-CudaMail-Originating-IP: 46.232.183.185 Date: Sun, 30 Oct 2016 13:30:09 +0000 X-ASG-Orig-Subj: [##CM-V3-1029004477##][PATCH 23/23] doc: Convert vswitchd/INTERNALS to rST Message-Id: <1477834209-11414-24-git-send-email-stephen@that.guru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1477834209-11414-1-git-send-email-stephen@that.guru> References: <1477834209-11414-1-git-send-email-stephen@that.guru> X-OutGoing-Spam-Status: No, score=-9.2 X-AuthUser: stephen@that.guru X-GBUdb-Analysis: 0, 46.232.183.185, Ugly c=0.071429 p=0 Source Normal X-MessageSniffer-Rules: 0-0-0-32767-c X-Barracuda-Connect: UNKNOWN[192.168.14.3] X-Barracuda-Start-Time: 1477834527 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://web.cudamail.com:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at cudamail.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.10 X-Barracuda-Spam-Status: No, SCORE=1.10 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=4.0 tests=BSF_SC0_MV0713, BSF_SC5_MJ1963, DKIM_SIGNED, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.34168 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC0_MV0713 Custom rule MV0713 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Subject: [ovs-dev] [PATCH 23/23] doc: Convert vswitchd/INTERNALS to rST X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" Signed-off-by: Stephen Finucane --- lib/mac-learning.c | 3 +- lib/mac-learning.h | 2 +- vswitchd/INTERNALS | 239 ------------------------------------------------ vswitchd/INTERNALS.rst | 244 +++++++++++++++++++++++++++++++++++++++++++++++++ vswitchd/automake.mk | 2 +- 5 files changed, 248 insertions(+), 242 deletions(-) delete mode 100644 vswitchd/INTERNALS create mode 100644 vswitchd/INTERNALS.rst diff --git a/lib/mac-learning.c b/lib/mac-learning.c index 5509f22..57b81f4 100644 --- a/lib/mac-learning.c +++ b/lib/mac-learning.c @@ -411,7 +411,8 @@ update_learning_table__(struct mac_learning *ml, struct eth_addr src, * packet was received over a non-bond interface and refrain from * learning from gratuitous ARP packets that arrive over bond * interfaces for this entry while the lock is in effect. See - * vswitchd/INTERNALS for more in-depth discussion on this topic. */ + * vswitchd/INTERNALS.rst for more in-depth discussion on this + * topic. */ if (!is_bond) { mac_entry_set_grat_arp_lock(mac); } else if (mac_entry_is_grat_arp_locked(mac)) { diff --git a/lib/mac-learning.h b/lib/mac-learning.h index d380690..e427815 100644 --- a/lib/mac-learning.h +++ b/lib/mac-learning.h @@ -47,7 +47,7 @@ * Second, the implementation has the ability to "lock" a MAC table entry * updated by a gratuitous ARP. This is a simple feature but the rationale for * it is complicated. Please refer to the description of SLB bonding in - * vswitchd/INTERNALS for an explanation. + * vswitchd/INTERNALS.rst for an explanation. * * Third, the implementation expires entries that are idle for longer than a * configurable amount of time. This is implemented by keeping all of the diff --git a/vswitchd/INTERNALS b/vswitchd/INTERNALS deleted file mode 100644 index 994353d..0000000 --- a/vswitchd/INTERNALS +++ /dev/null @@ -1,239 +0,0 @@ - ======================== - ovs-vswitchd Internals - ======================== - -This document describes some of the internals of the ovs-vswitchd -process. It is not complete. It tends to be updated on demand, so if -you have questions about the vswitchd implementation, ask them and -perhaps we'll add some appropriate documentation here. - -Most of the ovs-vswitchd implementation is in vswitchd/bridge.c, so -code references below should be assumed to refer to that file except -as otherwise specified. - -Bonding -======= - -Bonding allows two or more interfaces (the "slaves") to share network -traffic. From a high-level point of view, bonded interfaces act like -a single port, but they have the bandwidth of multiple network -devices, e.g. two 1 GB physical interfaces act like a single 2 GB -interface. Bonds also increase robustness: the bonded port does not -go down as long as at least one of its slaves is up. - -In vswitchd, a bond always has at least two slaves (and may have -more). If a configuration error, etc. would cause a bond to have only -one slave, the port becomes an ordinary port, not a bonded port, and -none of the special features of bonded ports described in this section -apply. - -There are many forms of bonding of which ovs-vswitchd implements only -a few. The most complex bond ovs-vswitchd implements is called -"source load balancing" or SLB bonding. SLB bonding divides traffic -among the slaves based on the Ethernet source address. This is useful -only if the traffic over the bond has multiple Ethernet source -addresses, for example if network traffic from multiple VMs are -multiplexed over the bond. - -Enabling and Disabling Slaves ------------------------------ - -When a bond is created, a slave is initially enabled or disabled based -on whether carrier is detected on the NIC (see iface_create()). After -that, a slave is disabled if its carrier goes down for a period of -time longer than the downdelay, and it is enabled if carrier comes up -for longer than the updelay (see bond_link_status_update()). There is -one exception where the updelay is skipped: if no slaves at all are -currently enabled, then the first slave on which carrier comes up is -enabled immediately. - -The updelay should be set to a time longer than the STP forwarding -delay of the physical switch to which the bond port is connected (if -STP is enabled on that switch). Otherwise, the slave will be enabled, -and load may be shifted to it, before the physical switch starts -forwarding packets on that port, which can cause some data to be -"blackholed" for a time. The exception for a single enabled slave -does not cause any problem in this regard because when no slaves are -enabled all output packets are blackholed anyway. - -When a slave becomes disabled, the vswitch immediately chooses a new -output port for traffic that was destined for that slave (see -bond_enable_slave()). It also sends a "gratuitous learning packet", -specifically a RARP, on the bond port (on the newly chosen slave) for -each MAC address that the vswitch has learned on a port other than the -bond (see bond_send_learning_packets()), to teach the physical switch -that the new slave should be used in place of the one that is now -disabled. (This behavior probably makes sense only for a vswitch that -has only one port (the bond) connected to a physical switch; vswitchd -should probably provide a way to disable or configure it in other -scenarios.) - -Bond Packet Input ------------------ - -Bonding accepts unicast packets on any bond slave. This can -occasionally cause packet duplication for the first few packets sent -to a given MAC, if the physical switch attached to the bond is -flooding packets to that MAC because it has not yet learned the -correct slave for that MAC. - -Bonding only accepts multicast (and broadcast) packets on a single -bond slave (the "active slave") at any given time. Multicast packets -received on other slaves are dropped. Otherwise, every multicast -packet would be duplicated, once for every bond slave, because the -physical switch attached to the bond will flood those packets. - -Bonding also drops received packets when the vswitch has learned that -the packet's MAC is on a port other than the bond port itself. This is -because it is likely that the vswitch itself sent the packet out the -bond port on a different slave and is now receiving the packet back. -This occurs when the packet is multicast or the physical switch has not -yet learned the MAC and is flooding it. However, the vswitch makes an -exception to this rule for broadcast ARP replies, which indicate that -the MAC has moved to another switch, probably due to VM migration. -(ARP replies are normally unicast, so this exception does not match -normal ARP replies. It will match the learning packets sent on bond -fail-over.) - -The active slave is simply the first slave to be enabled after the -bond is created (see bond_choose_active_iface()). If the active slave -is disabled, then a new active slave is chosen among the slaves that -remain active. Currently due to the way that configuration works, -this tends to be the remaining slave whose interface name is first -alphabetically, but this is by no means guaranteed. - -Bond Packet Output ------------------- - -When a packet is sent out a bond port, the bond slave actually used is -selected based on the packet's source MAC and VLAN tag (see -choose_output_iface()). In particular, the source MAC and VLAN tag -are hashed into one of 256 values, and that value is looked up in a -hash table (the "bond hash") kept in the "bond_hash" member of struct -port. The hash table entry identifies a bond slave. If no bond slave -has yet been chosen for that hash table entry, vswitchd chooses one -arbitrarily. - -Every 10 seconds, vswitchd rebalances the bond slaves (see -bond_rebalance_port()). To rebalance, vswitchd examines the -statistics for the number of bytes transmitted by each slave over -approximately the past minute, with data sent more recently weighted -more heavily than data sent less recently. It considers each of the -slaves in order from most-loaded to least-loaded. If highly loaded -slave H is significantly more heavily loaded than the least-loaded -slave L, and slave H carries at least two hashes, then vswitchd shifts -one of H's hashes to L. However, vswitchd will only shift a hash from -H to L if it will decrease the ratio of the load between H and L by at -least 0.1. - -Currently, "significantly more loaded" means that H must carry at -least 1 Mbps more traffic, and that traffic must be at least 3% -greater than L's. - -Bond Balance Modes ------------------- - -Each bond balancing mode has different considerations, described -below. - -LACP Bonding ------------- - -LACP bonding requires the remote switch to implement LACP, but it is -otherwise very simple in that, after LACP negotiation is complete, -there is no need for special handling of received packets. - -Several of the physical switches that support LACP block all traffic -for ports that are configured to use LACP, until LACP is negotiated with -the host. When configuring a LACP bond on a OVS host (eg: XenServer), -this means that there will be an interruption of the network connectivity -between the time the ports on the physical switch and the bond on the OVS -host are configured. The interruption may be relatively long, if different -people are responsible for managing the switches and the OVS host. - -Such network connectivity failure can be avoided if LACP can be configured -on the OVS host before configuring the physical switch, and having -the OVS host fall back to a bond mode (active-backup) till the physical -switch LACP configuration is complete. An option "lacp-fallback-ab" exists to -provide such behavior on openvswitch. - -Active Backup Bonding ---------------------- - -Active Backup bonds send all traffic out one "active" slave until that -slave becomes unavailable. Since they are significantly less -complicated than SLB bonds, they are preferred when LACP is not an -option. Additionally, they are the only bond mode which supports -attaching each slave to a different upstream switch. - -SLB Bonding ------------ - -SLB bonding allows a limited form of load balancing without the remote -switch's knowledge or cooperation. The basics of SLB are simple. SLB -assigns each source MAC+VLAN pair to a link and transmits all packets -from that MAC+VLAN through that link. Learning in the remote switch -causes it to send packets to that MAC+VLAN through the same link. - -SLB bonding has the following complications: - - 0. When the remote switch has not learned the MAC for the - destination of a unicast packet and hence floods the packet to - all of the links on the SLB bond, Open vSwitch will forward - duplicate packets, one per link, to each other switch port. - - Open vSwitch does not solve this problem. - - 1. When the remote switch receives a multicast or broadcast packet - from a port not on the SLB bond, it will forward it to all of - the links in the SLB bond. This would cause packet duplication - if not handled specially. - - Open vSwitch avoids packet duplication by accepting multicast - and broadcast packets on only the active slave, and dropping - multicast and broadcast packets on all other slaves. - - 2. When Open vSwitch forwards a multicast or broadcast packet to a - link in the SLB bond other than the active slave, the remote - switch will forward it to all of the other links in the SLB - bond, including the active slave. Without special handling, - this would mean that Open vSwitch would forward a second copy of - the packet to each switch port (other than the bond), including - the port that originated the packet. - - Open vSwitch deals with this case by dropping packets received - on any SLB bonded link that have a source MAC+VLAN that has been - learned on any other port. (This means that SLB as implemented - in Open vSwitch relies critically on MAC learning. Notably, SLB - is incompatible with the "flood_vlans" feature.) - - 3. Suppose that a MAC+VLAN moves to an SLB bond from another port - (e.g. when a VM is migrated from this hypervisor to a different - one). Without additional special handling, Open vSwitch will - not notice until the MAC learning entry expires, up to 60 - seconds later as a consequence of rule #2. - - Open vSwitch avoids a 60-second delay by listening for - gratuitous ARPs, which VMs commonly emit upon migration. As an - exception to rule #2, a gratuitous ARP received on an SLB bond - is not dropped and updates the MAC learning table in the usual - way. (If a move does not trigger a gratuitous ARP, or if the - gratuitous ARP is lost in the network, then a 60-second delay - still occurs.) - - 4. Suppose that a MAC+VLAN moves from an SLB bond to another port - (e.g. when a VM is migrated from a different hypervisor to this - one), that the MAC+VLAN emits a gratuitous ARP, and that Open - vSwitch forwards that gratuitous ARP to a link in the SLB bond - other than the active slave. The remote switch will forward the - gratuitous ARP to all of the other links in the SLB bond, - including the active slave. Without additional special - handling, this would mean that Open vSwitch would learn that the - MAC+VLAN was located on the SLB bond, as a consequence of rule - #3. - - Open vSwitch avoids this problem by "locking" the MAC learning - table entry for a MAC+VLAN from which a gratuitous ARP was - received from a non-SLB bond port. For 5 seconds, a locked MAC - learning table entry will not be updated based on a gratuitous - ARP received on a SLB bond. diff --git a/vswitchd/INTERNALS.rst b/vswitchd/INTERNALS.rst new file mode 100644 index 0000000..95c00f2 --- /dev/null +++ b/vswitchd/INTERNALS.rst @@ -0,0 +1,244 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +====================== +ovs-vswitchd Internals +====================== + +This document describes some of the internals of the ovs-vswitchd process. It +is not complete. It tends to be updated on demand, so if you have questions +about the vswitchd implementation, ask them and perhaps we'll add some +appropriate documentation here. + +Most of the ovs-vswitchd implementation is in ``vswitchd/bridge.c``, so code +references below should be assumed to refer to that file except as otherwise +specified. + +Bonding +------- + +Bonding allows two or more interfaces (the "slaves") to share network traffic. +From a high-level point of view, bonded interfaces act like a single port, but +they have the bandwidth of multiple network devices, e.g. two 1 GB physical +interfaces act like a single 2 GB interface. Bonds also increase robustness: +the bonded port does not go down as long as at least one of its slaves is up. + +In vswitchd, a bond always has at least two slaves (and may have more). If a +configuration error, etc. would cause a bond to have only one slave, the port +becomes an ordinary port, not a bonded port, and none of the special features +of bonded ports described in this section apply. + +There are many forms of bonding of which ovs-vswitchd implements only a few. +The most complex bond ovs-vswitchd implements is called "source load balancing" +or SLB bonding. SLB bonding divides traffic among the slaves based on the +Ethernet source address. This is useful only if the traffic over the bond has +multiple Ethernet source addresses, for example if network traffic from +multiple VMs are multiplexed over the bond. + +Enabling and Disabling Slaves +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When a bond is created, a slave is initially enabled or disabled based on +whether carrier is detected on the NIC (see ``iface_create()``). After that, a +slave is disabled if its carrier goes down for a period of time longer than the +downdelay, and it is enabled if carrier comes up for longer than the updelay +(see ``bond_link_status_update()``). There is one exception where the updelay +is skipped: if no slaves at all are currently enabled, then the first slave on +which carrier comes up is enabled immediately. + +The updelay should be set to a time longer than the STP forwarding delay of the +physical switch to which the bond port is connected (if STP is enabled on that +switch). Otherwise, the slave will be enabled, and load may be shifted to it, +before the physical switch starts forwarding packets on that port, which can +cause some data to be "blackholed" for a time. The exception for a single +enabled slave does not cause any problem in this regard because when no slaves +are enabled all output packets are blackholed anyway. + +When a slave becomes disabled, the vswitch immediately chooses a new output +port for traffic that was destined for that slave (see +``bond_enable_slave()``). It also sends a "gratuitous learning packet", +specifically a RARP, on the bond port (on the newly chosen slave) for each MAC +address that the vswitch has learned on a port other than the bond (see +``bond_send_learning_packets()``), to teach the physical switch that the new +slave should be used in place of the one that is now disabled. (This behavior +probably makes sense only for a vswitch that has only one port (the bond) +connected to a physical switch; vswitchd should probably provide a way to +disable or configure it in other scenarios.) + +Bond Packet Input +~~~~~~~~~~~~~~~~~ + +Bonding accepts unicast packets on any bond slave. This can occasionally cause +packet duplication for the first few packets sent to a given MAC, if the +physical switch attached to the bond is flooding packets to that MAC because it +has not yet learned the correct slave for that MAC. + +Bonding only accepts multicast (and broadcast) packets on a single bond slave +(the "active slave") at any given time. Multicast packets received on other +slaves are dropped. Otherwise, every multicast packet would be duplicated, +once for every bond slave, because the physical switch attached to the bond +will flood those packets. + +Bonding also drops received packets when the vswitch has learned that the +packet's MAC is on a port other than the bond port itself. This is because it +is likely that the vswitch itself sent the packet out the bond port on a +different slave and is now receiving the packet back. This occurs when the +packet is multicast or the physical switch has not yet learned the MAC and is +flooding it. However, the vswitch makes an exception to this rule for +broadcast ARP replies, which indicate that the MAC has moved to another switch, +probably due to VM migration. (ARP replies are normally unicast, so this +exception does not match normal ARP replies. It will match the learning +packets sent on bond fail-over.) + +The active slave is simply the first slave to be enabled after the bond is +created (see ``bond_choose_active_iface()``). If the active slave is disabled, +then a new active slave is chosen among the slaves that remain active. +Currently due to the way that configuration works, this tends to be the +remaining slave whose interface name is first alphabetically, but this is by no +means guaranteed. + +Bond Packet Output +~~~~~~~~~~~~~~~~~~ + +When a packet is sent out a bond port, the bond slave actually used is selected +based on the packet's source MAC and VLAN tag (see ``choose_output_iface()``). +In particular, the source MAC and VLAN tag are hashed into one of 256 values, +and that value is looked up in a hash table (the "bond hash") kept in the +``bond_hash`` member of struct port. The hash table entry identifies a bond +slave. If no bond slave has yet been chosen for that hash table entry, +vswitchd chooses one arbitrarily. + +Every 10 seconds, vswitchd rebalances the bond slaves (see +``bond_rebalance_port()``). To rebalance, vswitchd examines the statistics for +the number of bytes transmitted by each slave over approximately the past +minute, with data sent more recently weighted more heavily than data sent less +recently. It considers each of the slaves in order from most-loaded to +least-loaded. If highly loaded slave H is significantly more heavily loaded +than the least-loaded slave L, and slave H carries at least two hashes, then +vswitchd shifts one of H's hashes to L. However, vswitchd will only shift a +hash from H to L if it will decrease the ratio of the load between H and L by +at least 0.1. + +Currently, "significantly more loaded" means that H must carry at least 1 Mbps +more traffic, and that traffic must be at least 3% greater than L's. + +Bond Balance Modes +~~~~~~~~~~~~~~~~~~ + +Each bond balancing mode has different considerations, described below. + +LACP Bonding +++++++++++++ + +LACP bonding requires the remote switch to implement LACP, but it is otherwise +very simple in that, after LACP negotiation is complete, there is no need for +special handling of received packets. + +Several of the physical switches that support LACP block all traffic for ports +that are configured to use LACP, until LACP is negotiated with the host. When +configuring a LACP bond on a OVS host (eg: XenServer), this means that there +will be an interruption of the network connectivity between the time the ports +on the physical switch and the bond on the OVS host are configured. The +interruption may be relatively long, if different people are responsible for +managing the switches and the OVS host. + +Such network connectivity failure can be avoided if LACP can be configured on +the OVS host before configuring the physical switch, and having the OVS host +fall back to a bond mode (active-backup) till the physical switch LACP +configuration is complete. An option "lacp-fallback-ab" exists to provide such +behavior on openvswitch. + +Active Backup Bonding ++++++++++++++++++++++ + +Active Backup bonds send all traffic out one "active" slave until that slave +becomes unavailable. Since they are significantly less complicated than SLB +bonds, they are preferred when LACP is not an option. Additionally, they are +the only bond mode which supports attaching each slave to a different upstream +switch. + +SLB Bonding ++++++++++++ + +SLB bonding allows a limited form of load balancing without the remote switch's +knowledge or cooperation. The basics of SLB are simple. SLB assigns each +source MAC+VLAN pair to a link and transmits all packets from that MAC+VLAN +through that link. Learning in the remote switch causes it to send packets to +that MAC+VLAN through the same link. + +SLB bonding has the following complications: + +0. When the remote switch has not learned the MAC for the destination of a + unicast packet and hence floods the packet to all of the links on the SLB + bond, Open vSwitch will forward duplicate packets, one per link, to each + other switch port. + + Open vSwitch does not solve this problem. + +1. When the remote switch receives a multicast or broadcast packet from a port + not on the SLB bond, it will forward it to all of the links in the SLB bond. + This would cause packet duplication if not handled specially. + + Open vSwitch avoids packet duplication by accepting multicast and broadcast + packets on only the active slave, and dropping multicast and broadcast + packets on all other slaves. + +2. When Open vSwitch forwards a multicast or broadcast packet to a link in the + SLB bond other than the active slave, the remote switch will forward it to + all of the other links in the SLB bond, including the active slave. Without + special handling, this would mean that Open vSwitch would forward a second + copy of the packet to each switch port (other than the bond), including the + port that originated the packet. + + Open vSwitch deals with this case by dropping packets received on any SLB + bonded link that have a source MAC+VLAN that has been learned on any other + port. (This means that SLB as implemented in Open vSwitch relies critically + on MAC learning. Notably, SLB is incompatible with the "flood_vlans" + feature.) + +3. Suppose that a MAC+VLAN moves to an SLB bond from another port (e.g. when a + VM is migrated from this hypervisor to a different one). Without additional + special handling, Open vSwitch will not notice until the MAC learning entry + expires, up to 60 seconds later as a consequence of rule #2. + + Open vSwitch avoids a 60-second delay by listening for gratuitous ARPs, + which VMs commonly emit upon migration. As an exception to rule #2, a + gratuitous ARP received on an SLB bond is not dropped and updates the MAC + learning table in the usual way. (If a move does not trigger a gratuitous + ARP, or if the gratuitous ARP is lost in the network, then a 60-second delay + still occurs.) + +4. Suppose that a MAC+VLAN moves from an SLB bond to another port (e.g. when a + VM is migrated from a different hypervisor to this one), that the MAC+VLAN + emits a gratuitous ARP, and that Open vSwitch forwards that gratuitous ARP + to a link in the SLB bond other than the active slave. The remote switch + will forward the gratuitous ARP to all of the other links in the SLB bond, + including the active slave. Without additional special handling, this would + mean that Open vSwitch would learn that the MAC+VLAN was located on the SLB + bond, as a consequence of rule #3. + + Open vSwitch avoids this problem by "locking" the MAC learning table entry + for a MAC+VLAN from which a gratuitous ARP was received from a non-SLB bond + port. For 5 seconds, a locked MAC learning table entry will not be updated + based on a gratuitous ARP received on a SLB bond. + diff --git a/vswitchd/automake.mk b/vswitchd/automake.mk index 8d7f3ea..94a0272 100644 --- a/vswitchd/automake.mk +++ b/vswitchd/automake.mk @@ -16,7 +16,7 @@ vswitchd_ovs_vswitchd_LDADD = \ lib/libsflow.la \ lib/libopenvswitch.la vswitchd_ovs_vswitchd_LDFLAGS = $(AM_LDFLAGS) $(DPDK_vswitchd_LDFLAGS) -EXTRA_DIST += vswitchd/INTERNALS +EXTRA_DIST += vswitchd/INTERNALS.rst MAN_ROOTS += vswitchd/ovs-vswitchd.8.in # vswitch schema and IDL