diff mbox series

[ovs-dev,v3] doc: Add "Representors" topic document

Message ID 1550425041-24749-1-git-send-email-ophirmu@mellanox.com
State Accepted
Delegated to: Ian Stokes
Headers show
Series [ovs-dev,v3] doc: Add "Representors" topic document | expand

Commit Message

Ophir Munk Feb. 17, 2019, 5:37 p.m. UTC
This details how to configure representors ports.

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
---
v1:
First version
v2:
Following patch reviews.
https://patchwork.ozlabs.org/patch/1039515/
Mainly add how to create representors with Intel and Mellanox NICs.
v3:
Fix Checkpatch complains regarding too long lines but still leave long printouts

 Documentation/topics/dpdk/phy.rst | 139 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)

Comments

0-day Robot Feb. 17, 2019, 6:15 p.m. UTC | #1
Bleep bloop.  Greetings Ophir Munk, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


checkpatch:
WARNING: Line is 82 characters long (recommended limit is 79)
#104 FILE: Documentation/topics/dpdk/phy.rst:305:
    05:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

WARNING: Line is 82 characters long (recommended limit is 79)
#105 FILE: Documentation/topics/dpdk/phy.rst:306:
    05:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

WARNING: Line is 82 characters long (recommended limit is 79)
#114 FILE: Documentation/topics/dpdk/phy.rst:315:
    05:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

WARNING: Line is 82 characters long (recommended limit is 79)
#115 FILE: Documentation/topics/dpdk/phy.rst:316:
    05:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

WARNING: Line is 99 characters long (recommended limit is 79)
#116 FILE: Documentation/topics/dpdk/phy.rst:317:
    05:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

WARNING: Line is 99 characters long (recommended limit is 79)
#117 FILE: Documentation/topics/dpdk/phy.rst:318:
    05:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

Lines checked: 166, Warnings: 6, Errors: 0


Please check this out.  If you feel there has been an error, please email aconole@bytheb.org

Thanks,
0-day Robot
Stokes, Ian Feb. 18, 2019, 6:05 p.m. UTC | #2
On 2/17/2019 5:37 PM, Ophir Munk wrote:
> This details how to configure representors ports.
> 
> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> ---
> v1:
> First version
> v2:
> Following patch reviews.
> https://patchwork.ozlabs.org/patch/1039515/
> Mainly add how to create representors with Intel and Mellanox NICs.
> v3:
> Fix Checkpatch complains regarding too long lines but still leave long printouts
> 
>   Documentation/topics/dpdk/phy.rst | 139 ++++++++++++++++++++++++++++++++++++++
>   1 file changed, 139 insertions(+)
> 

Thanks for the patch and incorporating the ixgbe/i40e examples Ophir, 
I've pushed to master and 2.11.

Ian
Flavio Leitner Feb. 19, 2019, 4:22 p.m. UTC | #3
On Mon, Feb 18, 2019 at 06:05:54PM +0000, Ian Stokes wrote:
> On 2/17/2019 5:37 PM, Ophir Munk wrote:
> > This details how to configure representors ports.
> > 
> > Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> > ---
> > v1:
> > First version
> > v2:
> > Following patch reviews.
> > https://patchwork.ozlabs.org/patch/1039515/
> > Mainly add how to create representors with Intel and Mellanox NICs.
> > v3:
> > Fix Checkpatch complains regarding too long lines but still leave long printouts
> > 
> >   Documentation/topics/dpdk/phy.rst | 139 ++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 139 insertions(+)
> > 
> 
> Thanks for the patch and incorporating the ixgbe/i40e examples Ophir, I've
> pushed to master and 2.11.
 
Thanks Ophir, nice doc.

fbl
diff mbox series

Patch

diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst
index 1470623..93d74df 100644
--- a/Documentation/topics/dpdk/phy.rst
+++ b/Documentation/topics/dpdk/phy.rst
@@ -219,6 +219,145 @@  For more information please refer to the `DPDK Port Hotplug Framework`__.
 
 __ http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug
 
+.. _representors:
+
+Representors
+------------
+
+DPDK representors enable configuring a phy port to a guest (VM) machine.
+
+OVS resides in the hypervisor which has one or more physical interfaces also
+known as the physical functions (PFs). If a PF supports SR-IOV it can be used
+to enable communication with the VMs via Virtual Functions (VFs).
+The VFs are virtual PCIe devices created from the physical Ethernet controller.
+
+DPDK models a physical interface as a rte device on top of which an eth
+device is created.
+DPDK (version 18.xx) introduced the representors eth devices.
+A representor device represents the VF eth device (VM side) on the hypervisor
+side and operates on top of a PF.
+Representors are multi devices created on top of one PF.
+
+For more information, refer to the `DPDK documentation`__.
+
+__ https://doc.dpdk.org/guides-18.11/prog_guide/switch_representation.html
+
+Prior to port representors there was a one-to-one relationship between the PF
+and the eth device. With port representors the relationship becomes one PF to
+many eth devices.
+In case of two representors ports, when one of the ports is closed - the PCI
+bus cannot be detached until the second representor port is closed as well.
+
+.. _representors-configuration:
+
+When configuring a PF-based port, OVS traditionally assigns the device PCI
+address in devargs. For an existing bridge called ``br0`` and PCI address
+``0000:08:00.0`` an ``add-port`` command is written as::
+
+    $ ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk \
+       options:dpdk-devargs=0000:08:00.0
+
+When configuring a VF-based port, DPDK uses an extended devargs syntax which
+has the following format::
+
+    BDBF,representor=[<representor id>]
+
+This syntax shows that a representor is an enumerated eth device (with
+a representor ID) which uses the PF PCI address.
+The following commands add representors 3 and 5 using PCI device address
+``0000:08:00.0``::
+
+    $ ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \
+       options:dpdk-devargs=0000:08:00.0,representor=[3]
+
+    $ ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk \
+       options:dpdk-devargs=0000:08:00.0,representor=[5]
+
+.. important::
+
+   Representors ports are configured prior to OVS invocation and independently
+   of it, or by other means as well. Please consult a NIC vendor instructions
+   on how to establish representors.
+
+.. _multi-dev-configuration:
+
+**Intel NICs ixgbe and i40e**
+
+In the following example we create one representor on PF address
+``0000:05:00.0``. Once the NIC is bounded to a DPDK compatible PMD the
+representor is created::
+
+    # echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
+
+**Mellanox NICs ConnectX-4, ConnectX-5 and ConnectX-6**
+
+In the following example we create two representors on PF address
+``0000:05:00.0`` and net device name ``enp3s0f0``.
+
+- Ensure SR-IOV is enabled on the system.
+
+Enable IOMMU in Linux by adding ``intel_iommu=on`` to kernel parameters, for
+example, using GRUB (see /etc/grub/grub.conf).
+
+- Verify the PF PCI address prior to representors creation::
+
+    # lspci | grep Mellanox
+    05:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
+    05:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
+
+- Create the two VFs on the compute node::
+
+    # echo 2 > /sys/class/net/enp3s0f0/device/sriov_numvfs
+
+ Verify the VFs creation::
+
+    # lspci | grep Mellanox
+    05:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
+    05:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
+    05:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
+    05:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
+
+- Unbind the relevant VFs 0000:05:00.2..0000:05:00.3::
+
+    # echo 0000:05:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
+    # echo 0000:05:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
+
+- Change e-switch mode.
+
+The Mellanox NIC has an e-switch on it. Change the e-switch mode from
+legacy to switchdev using the PF PCI address::
+
+    # sudo devlink dev eswitch set pci/0000:05:00.0 mode switchdev
+
+This will create the VF representors network devices in the host OS.
+
+- After setting the PF to switchdev mode bind back the relevant VFs::
+
+    # echo 0000:05:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
+    # echo 0000:05:00.3 > /sys/bus/pci/drivers/mlx5_core/bind
+
+- Restart Open vSwitch
+
+To verify representors correct configuration, execute::
+
+    $ ovs-vsctl show
+
+and make sure no errors are indicated.
+
+.. _vendor_configuration:
+
+Port representors are an example of multi devices. There are NICs which support
+multi devices by other methods than representors for which a generic devargs
+syntax is used. The generic syntax is based on the device mac address::
+
+    class=eth,mac=<MAC address>
+
+For example, the following command adds a port to a bridge called ``br0`` using
+an eth device whose mac address is ``00:11:22:33:44:55``::
+
+    $ ovs-vsctl add-port br0 dpdk-mac -- set Interface dpdk-mac type=dpdk \
+       options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55"
+
 Jumbo Frames
 ------------