From patchwork Mon Jun 27 01:34:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Darrell Ball X-Patchwork-Id: 640770 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3rdBLV1bjLz9t19 for ; Mon, 27 Jun 2016 11:34:26 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=pjWDVWM/; dkim-atps=neutral Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 010F010B2F; Sun, 26 Jun 2016 18:34:17 -0700 (PDT) X-Original-To: dev@openvswitch.com Delivered-To: dev@openvswitch.com Received: from mail-pf0-f196.google.com (mail-pf0-f196.google.com [209.85.192.196]) by archives.nicira.com (Postfix) with ESMTPS id 9E59E10B0A for ; Sun, 26 Jun 2016 18:34:13 -0700 (PDT) Received: by mail-pf0-f196.google.com with SMTP id 66so14434210pfy.1 for ; Sun, 26 Jun 2016 18:34:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:date:message-id:in-reply-to:references; bh=HHtBOA8Kypx07ERH9jWkNZcnTphGhYY1oOEEwLA9Xe4=; b=pjWDVWM/VVZHCJrNK4Vha4/SnYp0++Hb+Wkndc48D0MfUiShsWTQgiluZx8XYHNPE9 w8alKQ0U1VXHNSVwKAoVSYoIOQyrfur6SkAAfEItDCP0hkB6+SoRZcpupMUppvHH39n7 kJ9oIhkKScacBCe0sMUFKlgrtilL07Tha8zjIpobYqFTE4Foish2E3F2jx1w9m5V9hE5 Fvcmr5IU0DLqMGyYR32ilynaJWOFN4YgsVxN4NtrZFB6EZAsVu2/9X1dkr2dfsJhcJGC tWu0fJyczIRHmbUy1otEGifwXrU8MM9f2+njJq1Y5ZPbUdd+CFsrdRBDqDHKDtt9zRaV 9q6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=HHtBOA8Kypx07ERH9jWkNZcnTphGhYY1oOEEwLA9Xe4=; b=Es9Ju29m4XZ1wfEFX9QLpW7dp4BafAAxDXiggnzgiVduK/vGSxtS74ed/j5lIubUUG ZYvi4darHeVSmTsC4/gmHIW7ZRHBQcKtMz+PF0rypISHnOx1LOtwRVbieUeQI6C1puXk HTalON43lsGSLhB0EwyJokXxGhI76UD6wmWU9zjgfrBZL6MSuQcuOeJau+HOQWpQsEEw HK2H7L7//hkQRhZnHAouBLh2P60rEJmTHih6A+L3xwSR4iC0nH5RstNd0IFXN6Df22K6 ZhBvDdhj1/Z/6Pi6uxDiyHlN9QRqnz+5J8hWfj7HAUkh1odNvgiUtFp5xIXkFzxj0mEf TDXw== X-Gm-Message-State: ALyK8tJrPxACg4fxAWUtXbgeHzCL0OyzA1BHRZQfKiIbLFOvonCt+jRE36dUOFeZRi8Nlw== X-Received: by 10.98.54.198 with SMTP id d189mr29164735pfa.39.1466991252731; Sun, 26 Jun 2016 18:34:12 -0700 (PDT) Received: from localhost.localdomain (c-24-4-8-103.hsd1.ca.comcast.net. [24.4.8.103]) by smtp.gmail.com with ESMTPSA id e9sm1555460pfg.2.2016.06.26.18.34.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 26 Jun 2016 18:34:12 -0700 (PDT) From: Darrell Ball To: dlu998@gmail.com, dev@openvswitch.com Date: Sun, 26 Jun 2016 18:34:02 -0700 Message-Id: <1466991242-104128-3-git-send-email-dlu998@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1466991242-104128-1-git-send-email-dlu998@gmail.com> References: <1466991242-104128-1-git-send-email-dlu998@gmail.com> Subject: [ovs-dev] [patch_v3 2/2] ovn: Reformat some ovn design documentation X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" Some design micro-details (e.g.) register assignments) that may change over time were moved from the ovn-architecture.7.xml document to the OVN-DESIGN.md document. A table is added to summarize and quantify MFF logical metadata and register usage. The OVN-DESIGN.md file was tested using the following markdown parsers: https://jbt.github.io/markdown-editor/ http://dillinger.io/ The new flags register usage is also documented in the new OVN-DESIGN.md document. Signed-off-by: Darrell Ball --- ovn/OVN-DESIGN.md | 185 ++++++++++++++++++++++++++++++++++++++++++++ ovn/automake.mk | 1 + ovn/ovn-architecture.7.xml | 188 --------------------------------------------- 3 files changed, 186 insertions(+), 188 deletions(-) create mode 100644 ovn/OVN-DESIGN.md diff --git a/ovn/OVN-DESIGN.md b/ovn/OVN-DESIGN.md new file mode 100644 index 0000000..ec3f0c4 --- /dev/null +++ b/ovn/OVN-DESIGN.md @@ -0,0 +1,185 @@ +OVN register and metadata usage: +------------------------------- + +logical datapath field: + +A field that denotes the logical datapath through which a packet is being +processed. +*Keep the following in sync with MFF_LOG_DATAPATH in* +*ovn/lib/logical-fields.h.* +OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls +'metadata' to store the logical datapath. (This field is passed across +tunnels as part of the tunnel key.) + + +logical input port field: + +A field that denotes the logical port from which the packet entered the +logical datapath. +*Keep the following in sync with MFF_LOG_INPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 6. + +Geneve and STT tunnels pass this field as part of the tunnel key. +Although VXLAN tunnels do not explicitly carry a logical input port, +OVN only uses VXLAN to communicate with gateways that from OVN's +perspective consist of only a single logical port, so that OVN can set +the logical input port field to this one on ingress to the OVN logical +pipeline. + + +logical output port field: + +A field that denotes the logical port from which the packet will leave +the logical datapath. This is initialized to 0 at the beginning of the +logical ingress pipeline. +*Keep the following in sync with MFF_LOG_OUTPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 7. + +Geneve and STT tunnels pass this field as part of the tunnel key. VXLAN +tunnels do not transmit the logical output port field. + + +conntrack zone field for logical ports: + +A field that denotes the connection tracking zone for logical ports. +The value only has local significance and is not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. OVN stores this in Nicira extension register number 5. + + +conntrack zone fields for Gateway router: + +Fields that denote the connection tracking zones for Gateway routers. +These values only have local significance (only on chassis that have +Gateway routers instantiated) and is not meaningful between +chassis. OVN stores the zone information for DNATting in Nicira +extension register number 3 and zone information for SNATing in Nicira +extension register number 4. + + +flags field: + +Scratchpad flags that denote the pipeline state between tables. The +values only have local significance and are not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 2. + + +VLAN ID: + +The VLAN ID is used as an interface between OVN and containers nested +inside a VM (see Life Cycle of a container interface inside a VM, in +ovn-architecture.7.xml, for more information). + + +The following table summarizes the register and metadata usage for OVN. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.*: + +``` + + Register/Metadata Usage Bits (Used/Total) + ---------------- --------- -------------- + MFF_METADATA logical datapath 24/64 + MFF_REG0 generic scratch 32/32 + MFF_REG1 generic scratch 32/32 + MFF_REG2 flags 1/32 + MFF_REG3 conntrack dnat zone gateway 16/32 + MFF_REG4 conntrack snat zone gateway 16/32 + MFF_REG5 conntrack zone logical ports 16/32 + MFF_REG6 logical input port 15/32 + MFF_REG7 logical output port 16/32 + +``` + +OVN Tunnel Encapsulations: +------------------------- + +tunnel key: + +When OVN encapsulates a packet in Geneve or another tunnel, it attaches +extra data to it to allow the receiving OVN instance to process it +correctly. This takes different forms depending on the particular +encapsulation, but in each case we refer to it here as the 'tunnel key'. + +OVN supports three types of IP tunnel encapsulations used in the IP mesh +connecting HVs, physical switches, appliances and/or servers. OVN passes +packet context information across tunnels. + +OVN annotates logical network packets that it sends from one hypervisor to +another with the following three pieces of metadata, which are encoded in an +encapsulation-specific fashion. + + +24-bit logical datapath identifier: +This is derived from the tunnel_key column in the OVN Southbound +Datapath_Binding table. + +15-bit logical ingress port identifier: +ID 0 is reserved for internal use within OVN. IDs 1 through 32767, inclusive, +may be assigned to logical ports (see the tunnel_key column in the OVN +Southbound Port_Binding table). + +16-bit logical egress port identifier: +IDs 0 through 32767 have the same meaning as for logical ingress ports. IDs +32768 through 65535, inclusive, may be assigned to logical multicast groups +(see the tunnel_key column in the OVN Southbound Multicast_Group table). + +For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT +encapsulations, for the following reasons: + +Only STT and Geneve support the large amounts of metadata (over 32 bits per +packet) that OVN uses (as described above). + +STT and Geneve use randomized UDP or TCP source ports that allows efficient +distribution among multiple paths in environments that use ECMP in their +underlay. + +NICs are available to offload STT and Geneve encapsulation and decapsulation. + +Due to its flexibility, the preferred encapsulation between hypervisors is +Geneve. For Geneve encapsulation, OVN transmits the logical datapath +identifier in the Geneve VNI. + +*Keep the following in sync with ovn/controller/physical.h.* +OVN transmits the logical ingress and logical egress ports in a TLV with +class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to +LSB: + +``` + + 1 15 16 + ----------------------------------------- + | | | | + | resv | ingress port | egress port | + | | | | + ----------------------------------------- + ``` + +Environments whose NICs lack Geneve offload may prefer STT encapsulation +for performance reasons. For STT encapsulation, OVN encodes all three +pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB +to LSB: + +``` + + 9 15 16 24 + ------------------------------------------------------ + | | | | | + | resv | ingress port | egress port | datapath | + | | | | | + ------------------------------------------------------ +``` + +For connecting to gateways, in addition to Geneve and STT, OVN supports +VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. +Currently, gateways have a feature set that matches the capabilities as +defined by the VTEP schema, so fewer bits of metadata are necessary. In +the future, gateways that do not support encapsulations with large amounts +of metadata may continue to have a reduced feature set. + diff --git a/ovn/automake.mk b/ovn/automake.mk index f3f40e5..39277de 100644 --- a/ovn/automake.mk +++ b/ovn/automake.mk @@ -73,6 +73,7 @@ DISTCLEANFILES += ovn/ovn-architecture.7 EXTRA_DIST += \ ovn/TODO \ ovn/CONTAINERS.OpenStack.md \ + ovn/OVN-DESIGN.md \ ovn/OVN-GW-HA.md # Version checking for ovn-nb.ovsschema. diff --git a/ovn/ovn-architecture.7.xml b/ovn/ovn-architecture.7.xml index 72786bc..e023818 100644 --- a/ovn/ovn-architecture.7.xml +++ b/ovn/ovn-architecture.7.xml @@ -609,95 +609,6 @@

- This section mentions several data and metadata fields, for clarity - summarized here: -

- -
-
tunnel key
-
- When OVN encapsulates a packet in Geneve or another tunnel, it attaches - extra data to it to allow the receiving OVN instance to process it - correctly. This takes different forms depending on the particular - encapsulation, but in each case we refer to it here as the ``tunnel - key.'' See Tunnel Encapsulations, below, for details. -
- -
logical datapath field
-
- A field that denotes the logical datapath through which a packet is being - processed. - - OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls - ``metadata'' to store the logical datapath. (This field is passed across - tunnels as part of the tunnel key.) -
- -
logical input port field
-
-

- A field that denotes the logical port from which the packet - entered the logical datapath. - - OVN stores this in Nicira extension register number 6. -

- -

- Geneve and STT tunnels pass this field as part of the tunnel key. - Although VXLAN tunnels do not explicitly carry a logical input port, - OVN only uses VXLAN to communicate with gateways that from OVN's - perspective consist of only a single logical port, so that OVN can set - the logical input port field to this one on ingress to the OVN logical - pipeline. -

-
- -
logical output port field
-
-

- A field that denotes the logical port from which the packet will - leave the logical datapath. This is initialized to 0 at the - beginning of the logical ingress pipeline. - - OVN stores this in Nicira extension register number 7. -

- -

- Geneve and STT tunnels pass this field as part of the tunnel key. - VXLAN tunnels do not transmit the logical output port field. -

-
- -
conntrack zone field for logical ports
-
- A field that denotes the connection tracking zone for logical ports. - The value only has local significance and is not meaningful between - chassis. This is initialized to 0 at the beginning of the logical - ingress pipeline. OVN stores this in Nicira extension register number 5. -
- -
conntrack zone fields for Gateway router
-
- Fields that denote the connection tracking zones for Gateway routers. - These values only have local significance (only on chassis that have - Gateway routers instantiated) and is not meaningful between - chassis. OVN stores the zone information for DNATting in Nicira - extension register number 3 and zone information for SNATing in Nicira - extension register number 4. -
- -
VLAN ID
-
- The VLAN ID is used as an interface between OVN and containers nested - inside a VM (see Life Cycle of a container interface inside a - VM, above, for more information). -
-
- -

Initially, a VM or container on the ingress hypervisor sends a packet on a port attached to the OVN integration bridge. Then:

@@ -998,103 +909,4 @@ entries and the Logical_Switch tunnel keys. - -

Design Decisions

- -

Tunnel Encapsulations

- -

- OVN annotates logical network packets that it sends from one hypervisor to - another with the following three pieces of metadata, which are encoded in - an encapsulation-specific fashion: -

- -
    -
  • - 24-bit logical datapath identifier, from the tunnel_key - column in the OVN Southbound Datapath_Binding table. -
    • - -
  • - 15-bit logical ingress port identifier. ID 0 is reserved for internal - use within OVN. IDs 1 through 32767, inclusive, may be assigned to - logical ports (see the tunnel_key column in the OVN - Southbound Port_Binding table). -
    • - -
  • - 16-bit logical egress port identifier. IDs 0 through 32767 have the same - meaning as for logical ingress ports. IDs 32768 through 65535, - inclusive, may be assigned to logical multicast groups (see the - tunnel_key column in the OVN Southbound - Multicast_Group table). -
    • -
- -

- For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT - encapsulations, for the following reasons: -

- -
    -
  • - Only STT and Geneve support the large amounts of metadata (over 32 bits - per packet) that OVN uses (as described above). -
    • - -
  • - STT and Geneve use randomized UDP or TCP source ports that allows - efficient distribution among multiple paths in environments that use ECMP - in their underlay. -
    • - -
  • - NICs are available to offload STT and Geneve encapsulation and - decapsulation. -
    • -
- -

- Due to its flexibility, the preferred encapsulation between hypervisors is - Geneve. For Geneve encapsulation, OVN transmits the logical datapath - identifier in the Geneve VNI. - - - OVN transmits the logical ingress and logical egress ports in a TLV with - class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to - LSB: -

- - -
- - - -
- - -

- Environments whose NICs lack Geneve offload may prefer STT encapsulation - for performance reasons. For STT encapsulation, OVN encodes all three - pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB - to LSB: -

- - -
- - - - -
- - -

- For connecting to gateways, in addition to Geneve and STT, OVN supports - VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. - Currently, gateways have a feature set that matches the capabilities as - defined by the VTEP schema, so fewer bits of metadata are necessary. In - the future, gateways that do not support encapsulations with large amounts - of metadata may continue to have a reduced feature set. -