From patchwork Sat Jul 23 20:18:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Darrell Ball X-Patchwork-Id: 652013 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3rxf3Y6Y8cz9t0p for ; Sun, 24 Jul 2016 06:18:33 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=DMZZgtdK; dkim-atps=neutral Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 6B5F810930; Sat, 23 Jul 2016 13:18:17 -0700 (PDT) X-Original-To: dev@openvswitch.com Delivered-To: dev@openvswitch.com Received: from mail-pa0-f65.google.com (mail-pa0-f65.google.com [209.85.220.65]) by archives.nicira.com (Postfix) with ESMTPS id 4890510677 for ; Sat, 23 Jul 2016 13:18:15 -0700 (PDT) Received: by mail-pa0-f65.google.com with SMTP id ez1so8842932pab.3 for ; Sat, 23 Jul 2016 13:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:date:message-id:in-reply-to:references; bh=dBi5rk3V6268P1uJpTMnqUigQYkSZHPPvlGQHrhO9iw=; b=DMZZgtdK3bT6ZSlrdcD2/P8CjVx8J1z1EFLr4QgPcIup/zktvBtFbftTmEdqnCK+4L ebacyZj0c3Xar1kD1gnDxGOHKnBu5q8C9GXhud1p0wIIKXzOAHFm+nA/uLVWxBLnv7iD NEhdCoT5yR5/1rOCaaArF1+J3Kj9PmsdijLlAetKXNc+K5cTaI4zdM/747aZ0yELU4xf pYfgN2NOrD7zd1J3tNDU1P9r4eupjcQnzQDrD6/KKUZcTtT+5YuV9nM+eOke4gRdUa4N VT4+n6oJ4tP103TTbF+PEFpDC7Nut6+r9D8wreVEP/hPlH23zWuD3eGMk+Bkl2vVJhp8 aTNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=dBi5rk3V6268P1uJpTMnqUigQYkSZHPPvlGQHrhO9iw=; b=EhvoHKR4W7C+AsDBEw7bbe8eDC+m7rNXm0VvbML9nuIRZIOLPn3ESzzV8cdQJuEiMM w39WLUjV/sAfeJA2Gl5WI5EfDBS6hYR1h/h2caacyhxoaH7AbuAI7dHvsKa4LXmoy7H/ Z0EFCADHBQhS2oOUlNacTKCieOGVkTmQB/Y6lJc15PKd+QFliSz8l1NPipjK81Jd2mmc 929m2Cd2/OBe+QV+DmvvYydzFAeIVfK0nX4wdVBauUYQoEeCxvp9IhIHnhWxcT7hsK/J P8L6iCLq1zHAlA+Fu/rd3NrS0bhfUJx5QdHR+2rynVP5GQB73kehsFx9OIK4/9OLzJ9t C93Q== X-Gm-Message-State: AEkooutYf4JeAZxxDgfAbayAgdojsQ7zOl/qy5AuXMWnEDTbAelqsqtYKMrLYFmlouPNgQ== X-Received: by 10.66.62.168 with SMTP id z8mr17121959par.152.1469305094415; Sat, 23 Jul 2016 13:18:14 -0700 (PDT) Received: from localhost.localdomain (c-24-4-8-103.hsd1.ca.comcast.net. [24.4.8.103]) by smtp.gmail.com with ESMTPSA id u7sm25952752paz.45.2016.07.23.13.18.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 23 Jul 2016 13:18:13 -0700 (PDT) From: Darrell Ball To: dlu998@gmail.com, dev@openvswitch.com, jpettit@ovn.org Date: Sat, 23 Jul 2016 13:18:02 -0700 Message-Id: <1469305082-58994-3-git-send-email-dlu998@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1469305082-58994-1-git-send-email-dlu998@gmail.com> References: <1469305082-58994-1-git-send-email-dlu998@gmail.com> Subject: [ovs-dev] [patch_v6 2/2] ovn: Reformat some ovn design documentation. X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" Some design micro-details (e.g.) register assignments) that may change over time were moved from the ovn-architecture.7.xml document to the OVN-DESIGN.md document. A table is added to summarize and quantify MFF logical metadata and register usage. The OVN-DESIGN.md file was tested using the following markdown parsers: https://jbt.github.io/markdown-editor/ http://dillinger.io/ The new flags register usage is also documented in the new OVN-DESIGN.md document. Signed-off-by: Darrell Ball --- ovn/OVN-DESIGN.md | 199 +++++++++++++++++++++++++++++++++++++++++++++ ovn/automake.mk | 1 + ovn/ovn-architecture.7.xml | 193 ------------------------------------------- 3 files changed, 200 insertions(+), 193 deletions(-) create mode 100644 ovn/OVN-DESIGN.md diff --git a/ovn/OVN-DESIGN.md b/ovn/OVN-DESIGN.md new file mode 100644 index 0000000..2162c65 --- /dev/null +++ b/ovn/OVN-DESIGN.md @@ -0,0 +1,199 @@ +OVN register and metadata usage: +------------------------------- + +logical datapath field: + +A field that denotes the logical datapath through which a packet is being +processed. +*Keep the following in sync with MFF_LOG_DATAPATH in* +*ovn/lib/logical-fields.h.* +OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls +'metadata' to store the logical datapath. (This field is passed across +tunnels as part of the tunnel key.) + + +logical input port field: + +A field that denotes the logical port from which the packet entered the +logical datapath. +*Keep the following in sync with MFF_LOG_INPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 14. + +Geneve and STT tunnels pass this field as part of the tunnel key. +Although VXLAN tunnels do not explicitly carry a logical input port, +OVN only uses VXLAN to communicate with gateways that from OVN's +perspective consist of only a single logical port, so that OVN can set +the logical input port field to this one on ingress to the OVN logical +pipeline. + + +logical output port field: + +A field that denotes the logical port from which the packet will leave +the logical datapath. This is initialized to 0 at the beginning of the +logical ingress pipeline. +*Keep the following in sync with MFF_LOG_OUTPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 15. + +Geneve and STT tunnels pass this field as part of the tunnel key. VXLAN +tunnels do not transmit the logical output port field. + + +conntrack zone field for logical ports: + +A field that denotes the connection tracking zone for logical ports. +The value only has local significance and is not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. +*Keep the following in sync with MFF_LOG_CT_ZONE in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 13. + + +conntrack zone fields for Gateway router: + +Fields that denote the connection tracking zones for Gateway routers. +These values only have local significance (only on chassis that have +Gateway routers instantiated) and is not meaningful between +chassis. +*Keep the following in sync with MFF_LOG_DNAT_ZONE and* +*MFF_LOG_DNAT_ZONE inovn/lib/logical-fields.h.* +OVN stores the zone information for DNATting in Nicira +extension register number 11 and zone information for SNATing in Nicira +extension register number 12. + + +flags field: + +Scratchpad flags that denote the pipeline state between tables. The +values only have local significance and are not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 10. + + +VLAN ID: + +The VLAN ID is used as an interface between OVN and containers nested +inside a VM (see Life Cycle of a container interface inside a VM, in +ovn-architecture.7.xml, for more information). + + +The following table summarizes the register and metadata usage for OVN. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.*: + +``` + + Register/Metadata Usage Bits (Used/Total) + ---------------- --------- -------------- + MFF_METADATA logical datapath 24/64 + MFF_REG0 generic scratch 32/32 + MFF_REG1 generic scratch 32/32 + MFF_REG2 generic scratch 32/32 + MFF_REG3 generic scratch 32/32 + MFF_REG4 generic scratch 32/32 + MFF_REG5 generic scratch 32/32 + MFF_REG6 generic scratch 32/32 + MFF_REG7 generic scratch 32/32 + MFF_REG8 unused 0/32 + MFF_REG9 unused 0/32 + MFF_REG10 flags 1/32 + MFF_REG11 conntrack dnat zone gateway 16/32 + MFF_REG12 conntrack snat zone gateway 16/32 + MFF_REG13 conntrack zone logical ports 16/32 + MFF_REG14 logical input port 15/32 + MFF_REG15 logical output port 16/32 + +``` + +OVN Tunnel Encapsulations: +------------------------- + +tunnel key: + +When OVN encapsulates a packet in Geneve or another tunnel, it attaches +extra data to it to allow the receiving OVN instance to process it +correctly. This takes different forms depending on the particular +encapsulation, but in each case we refer to it here as the 'tunnel key'. + +OVN supports three types of IP tunnel encapsulations used in the IP mesh +connecting HVs, physical switches, appliances and/or servers. OVN passes +packet context information across tunnels. + +OVN annotates logical network packets that it sends from one hypervisor to +another with the following three pieces of metadata, which are encoded in an +encapsulation-specific fashion. + + +24-bit logical datapath identifier: +This is derived from the tunnel_key column in the OVN Southbound +Datapath_Binding table. + +15-bit logical ingress port identifier: +ID 0 is reserved for internal use within OVN. IDs 1 through 32767, inclusive, +may be assigned to logical ports (see the tunnel_key column in the OVN +Southbound Port_Binding table). + +16-bit logical egress port identifier: +IDs 0 through 32767 have the same meaning as for logical ingress ports. IDs +32768 through 65535, inclusive, may be assigned to logical multicast groups +(see the tunnel_key column in the OVN Southbound Multicast_Group table). + +For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT +encapsulations, for the following reasons: + +Only STT and Geneve support the large amounts of metadata (over 32 bits per +packet) that OVN uses (as described above). + +STT and Geneve use randomized UDP or TCP source ports that allows efficient +distribution among multiple paths in environments that use ECMP in their +underlay. + +NICs are available to offload STT and Geneve encapsulation and decapsulation. + +Due to its flexibility, the preferred encapsulation between hypervisors is +Geneve. For Geneve encapsulation, OVN transmits the logical datapath +identifier in the Geneve VNI. + +*Keep the following in sync with ovn/controller/physical.h.* +OVN transmits the logical ingress and logical egress ports in a TLV with +class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to +LSB: + +``` + + 1 15 16 + ----------------------------------------- + | | | | + | resv | ingress port | egress port | + | | | | + ----------------------------------------- + ``` + +Environments whose NICs lack Geneve offload may prefer STT encapsulation +for performance reasons. For STT encapsulation, OVN encodes all three +pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB +to LSB: + +``` + + 9 15 16 24 + ------------------------------------------------------ + | | | | | + | resv | ingress port | egress port | datapath | + | | | | | + ------------------------------------------------------ +``` + +For connecting to gateways, in addition to Geneve and STT, OVN supports +VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. +Currently, gateways have a feature set that matches the capabilities as +defined by the VTEP schema, so fewer bits of metadata are necessary. In +the future, gateways that do not support encapsulations with large amounts +of metadata may continue to have a reduced feature set. + diff --git a/ovn/automake.mk b/ovn/automake.mk index f3f40e5..39277de 100644 --- a/ovn/automake.mk +++ b/ovn/automake.mk @@ -73,6 +73,7 @@ DISTCLEANFILES += ovn/ovn-architecture.7 EXTRA_DIST += \ ovn/TODO \ ovn/CONTAINERS.OpenStack.md \ + ovn/OVN-DESIGN.md \ ovn/OVN-GW-HA.md # Version checking for ovn-nb.ovsschema. diff --git a/ovn/ovn-architecture.7.xml b/ovn/ovn-architecture.7.xml index ead4feb..e023818 100644 --- a/ovn/ovn-architecture.7.xml +++ b/ovn/ovn-architecture.7.xml @@ -609,100 +609,6 @@

- This section mentions several data and metadata fields, for clarity - summarized here: -

- -
-
tunnel key
-
- When OVN encapsulates a packet in Geneve or another tunnel, it attaches - extra data to it to allow the receiving OVN instance to process it - correctly. This takes different forms depending on the particular - encapsulation, but in each case we refer to it here as the ``tunnel - key.'' See Tunnel Encapsulations, below, for details. -
- -
logical datapath field
-
- A field that denotes the logical datapath through which a packet is being - processed. - - OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls - ``metadata'' to store the logical datapath. (This field is passed across - tunnels as part of the tunnel key.) -
- -
logical input port field
-
-

- A field that denotes the logical port from which the packet - entered the logical datapath. - - OVN stores this in Nicira extension register number 14. -

- -

- Geneve and STT tunnels pass this field as part of the tunnel key. - Although VXLAN tunnels do not explicitly carry a logical input port, - OVN only uses VXLAN to communicate with gateways that from OVN's - perspective consist of only a single logical port, so that OVN can set - the logical input port field to this one on ingress to the OVN logical - pipeline. -

-
- -
logical output port field
-
-

- A field that denotes the logical port from which the packet will - leave the logical datapath. This is initialized to 0 at the - beginning of the logical ingress pipeline. - - OVN stores this in Nicira extension register number 15. -

- -

- Geneve and STT tunnels pass this field as part of the tunnel key. - VXLAN tunnels do not transmit the logical output port field. -

-
- -
conntrack zone field for logical ports
-
- A field that denotes the connection tracking zone for logical ports. - The value only has local significance and is not meaningful between - chassis. This is initialized to 0 at the beginning of the logical - - ingress pipeline. OVN stores this in Nicira extension register - number 13. -
- -
conntrack zone fields for Gateway router
-
- Fields that denote the connection tracking zones for Gateway routers. - These values only have local significance (only on chassis that have - Gateway routers instantiated) and is not meaningful between - chassis. OVN stores the zone information for DNATting in Nicira - - extension register number 11 and zone information for SNATing in Nicira - extension register number 12. -
- -
VLAN ID
-
- The VLAN ID is used as an interface between OVN and containers nested - inside a VM (see Life Cycle of a container interface inside a - VM, above, for more information). -
-
- -

Initially, a VM or container on the ingress hypervisor sends a packet on a port attached to the OVN integration bridge. Then:

@@ -1003,103 +909,4 @@ entries and the Logical_Switch tunnel keys. - -

Design Decisions

- -

Tunnel Encapsulations

- -

- OVN annotates logical network packets that it sends from one hypervisor to - another with the following three pieces of metadata, which are encoded in - an encapsulation-specific fashion: -

- -
    -
  • - 24-bit logical datapath identifier, from the tunnel_key - column in the OVN Southbound Datapath_Binding table. -
    • - -
  • - 15-bit logical ingress port identifier. ID 0 is reserved for internal - use within OVN. IDs 1 through 32767, inclusive, may be assigned to - logical ports (see the tunnel_key column in the OVN - Southbound Port_Binding table). -
    • - -
  • - 16-bit logical egress port identifier. IDs 0 through 32767 have the same - meaning as for logical ingress ports. IDs 32768 through 65535, - inclusive, may be assigned to logical multicast groups (see the - tunnel_key column in the OVN Southbound - Multicast_Group table). -
    • -
- -

- For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT - encapsulations, for the following reasons: -

- -
    -
  • - Only STT and Geneve support the large amounts of metadata (over 32 bits - per packet) that OVN uses (as described above). -
    • - -
  • - STT and Geneve use randomized UDP or TCP source ports that allows - efficient distribution among multiple paths in environments that use ECMP - in their underlay. -
    • - -
  • - NICs are available to offload STT and Geneve encapsulation and - decapsulation. -
    • -
- -

- Due to its flexibility, the preferred encapsulation between hypervisors is - Geneve. For Geneve encapsulation, OVN transmits the logical datapath - identifier in the Geneve VNI. - - - OVN transmits the logical ingress and logical egress ports in a TLV with - class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to - LSB: -

- - -
- - - -
- - -

- Environments whose NICs lack Geneve offload may prefer STT encapsulation - for performance reasons. For STT encapsulation, OVN encodes all three - pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB - to LSB: -

- - -
- - - - -
- - -

- For connecting to gateways, in addition to Geneve and STT, OVN supports - VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. - Currently, gateways have a feature set that matches the capabilities as - defined by the VTEP schema, so fewer bits of metadata are necessary. In - the future, gateways that do not support encapsulations with large amounts - of metadata may continue to have a reduced feature set. -