| Message ID | 1549843305-5034-1-git-send-email-ophirmu@mellanox.com |
|---|---|
| State | Superseded |
| Delegated to: | Ian Stokes |
| Headers | show |
| Series | [ovs-dev,v1] doc: Add "Representors" topic document | expand |
On 11.02.2019 3:01, Ophir Munk wrote: > This details how to configure representors ports. > > Signed-off-by: Ophir Munk <ophirmu@mellanox.com> > --- > Documentation/topics/dpdk/phy.rst | 80 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 80 insertions(+) > > diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst > index 1470623..3792fde 100644 > --- a/Documentation/topics/dpdk/phy.rst > +++ b/Documentation/topics/dpdk/phy.rst > @@ -219,6 +219,86 @@ For more information please refer to the `DPDK Port Hotplug Framework`__. > > __ http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug > > +.. _representors: > + > +Representors > +------------ > + > +DPDK representors enable configuring a phy port to a guest (VM) machine. > + > +OVS resides in the hypervisor which has one or more physical interfaces also > +known as the physical functions (PFs). If a PF supports SR-IOV it can be used > +to enable communication with the VMs via Virtual Functions (VFs). > +The VFs are virtual PCIe devices created from the physical Ethernet controller. > + > +DPDK models a physical interface as a rte device on top of which an eth > +device is created. > +DPDK (version 18.xx) introduced the representors eth devices. > +A representor device represents the VF eth device (VM side) on the hypervisor > +side and operates on top of a PF. > +Representors are multi devices created on top of one PF. > + > +For more information, refer to the `DPDK documentation`__. > + > +__ https://doc.dpdk.org/guides-18.11/prog_guide/switch_representation.html > + > +Prior to port representors there was a one-to-one relationship between the PF > +and the eth device. With port representors the relationship becomes one PF to > +many eth devices. > +In case of two representors ports, when one of the ports is closed - the PCI > +bus cannot be detached until the second representor port is closed as well. > + > +.. _representors-configuration: > + > +When configuring a PF-based port, OVS traditionally assigns the device PCI > +address in devargs. For an existing bridge called ``br0`` and PCI address > +``0000:08:00.0`` an ``add-port`` command is written as:: > + > + $ ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk \ > + options:dpdk-devargs=0000:08:00.0 > + > +When configuring a VF-based port, DPDK uses an extended devargs syntax which > +has the following format:: > + > + BDBF,representor=[<representor id>] > + > +This syntax shows that a representor is an enumerated eth device (with > +a representor ID) which uses the PF PCI address. > +The following commands add representors 3 and 5 using PCI device address > +``0000:08:00.0``:: > + > + $ ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ > + options:dpdk-devargs=0000:08:00.0,representor=[3] > + > + $ ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk \ > + options:dpdk-devargs=0000:08:00.0,representor=[5] > + > +.. important:: > + > + Representors ports are configured prior to OVS invocation and independently > + of it, or by other means as well. Please consult a NIC vendor instructions > + on how to establish representors. It'll be good to have configuration example for at least one commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs where the process described. What do you think ? Ian, maybe you could add some example, since you have already tried it in practice? > To verify their correct configuration, > + execute:: > + > + $ ovs-vsctl show > + > + and make sure no errors are indicated. > + > +.. _multi-dev-configuration: > + > + > +Port representors are an example of multi devices. There are NICs which support > +multi devices by other methods than representors for which a generic devargs > +syntax is used. The generic syntax is based on the device mac address:: > + > + class=eth,mac=<MAC address> > + > +For example, the following command adds a port to a bridge called ``br0`` using > +an eth device whose mac address is ``00:11:22:33:44:55``:: > + > + $ ovs-vsctl add-port br0 dpdk-mac -- set Interface dpdk-mac type=dpdk \ > + options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55" > + > Jumbo Frames > ------------ > >
On 2/12/2019 1:15 PM, Ilya Maximets wrote: > On 11.02.2019 3:01, Ophir Munk wrote: >> This details how to configure representors ports. >> >> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> >> --- >> Documentation/topics/dpdk/phy.rst | 80 +++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 80 insertions(+) >> >> diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst >> index 1470623..3792fde 100644 >> --- a/Documentation/topics/dpdk/phy.rst >> +++ b/Documentation/topics/dpdk/phy.rst >> @@ -219,6 +219,86 @@ For more information please refer to the `DPDK Port Hotplug Framework`__. >> >> __ http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug >> >> +.. _representors: >> + >> +Representors >> +------------ >> + >> +DPDK representors enable configuring a phy port to a guest (VM) machine. >> + >> +OVS resides in the hypervisor which has one or more physical interfaces also >> +known as the physical functions (PFs). If a PF supports SR-IOV it can be used >> +to enable communication with the VMs via Virtual Functions (VFs). >> +The VFs are virtual PCIe devices created from the physical Ethernet controller. >> + >> +DPDK models a physical interface as a rte device on top of which an eth >> +device is created. >> +DPDK (version 18.xx) introduced the representors eth devices. >> +A representor device represents the VF eth device (VM side) on the hypervisor >> +side and operates on top of a PF. >> +Representors are multi devices created on top of one PF. >> + >> +For more information, refer to the `DPDK documentation`__. >> + >> +__ https://doc.dpdk.org/guides-18.11/prog_guide/switch_representation.html >> + >> +Prior to port representors there was a one-to-one relationship between the PF >> +and the eth device. With port representors the relationship becomes one PF to >> +many eth devices. >> +In case of two representors ports, when one of the ports is closed - the PCI >> +bus cannot be detached until the second representor port is closed as well. >> + >> +.. _representors-configuration: >> + >> +When configuring a PF-based port, OVS traditionally assigns the device PCI >> +address in devargs. For an existing bridge called ``br0`` and PCI address >> +``0000:08:00.0`` an ``add-port`` command is written as:: >> + >> + $ ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk \ >> + options:dpdk-devargs=0000:08:00.0 >> + >> +When configuring a VF-based port, DPDK uses an extended devargs syntax which >> +has the following format:: >> + >> + BDBF,representor=[<representor id>] >> + >> +This syntax shows that a representor is an enumerated eth device (with >> +a representor ID) which uses the PF PCI address. >> +The following commands add representors 3 and 5 using PCI device address >> +``0000:08:00.0``:: >> + >> + $ ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ >> + options:dpdk-devargs=0000:08:00.0,representor=[3] >> + >> + $ ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk \ >> + options:dpdk-devargs=0000:08:00.0,representor=[5] >> + >> +.. important:: >> + >> + Representors ports are configured prior to OVS invocation and independently >> + of it, or by other means as well. Please consult a NIC vendor instructions >> + on how to establish representors. > > It'll be good to have configuration example for at least one commonly used NIC > (ixgbe/i40e ?). Or maybe a link to the docs where the process described. > > What do you think ? > Ian, maybe you could add some example, since you have already tried it in practice? > Good call, I'll draw up an incremental and post here, if acceptable we can roll it into the same patch. Ian >> To verify their correct configuration, >> + execute:: >> + >> + $ ovs-vsctl show >> + >> + and make sure no errors are indicated. >> + >> +.. _multi-dev-configuration: >> + >> + >> +Port representors are an example of multi devices. There are NICs which support >> +multi devices by other methods than representors for which a generic devargs >> +syntax is used. The generic syntax is based on the device mac address:: >> + >> + class=eth,mac=<MAC address> >> + >> +For example, the following command adds a port to a bridge called ``br0`` using >> +an eth device whose mac address is ``00:11:22:33:44:55``:: >> + >> + $ ovs-vsctl add-port br0 dpdk-mac -- set Interface dpdk-mac type=dpdk \ >> + options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55" >> + >> Jumbo Frames >> ------------ >> >>
> -----Original Message----- > From: Ian Stokes <ian.stokes@intel.com> > Sent: Tuesday, February 12, 2019 7:17 PM > To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk > <ophirmu@mellanox.com>; ovs-dev@openvswitch.org > Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor > <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas > Monjalon <thomas@monjalon.net> > Subject: Re: [PATCH v1] doc: Add "Representors" topic document > > On 2/12/2019 1:15 PM, Ilya Maximets wrote: > > On 11.02.2019 3:01, Ophir Munk wrote: > >> This details how to configure representors ports. > >> > >> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> > >> --- > >> Documentation/topics/dpdk/phy.rst | 80 > +++++++++++++++++++++++++++++++++++++++ > >> 1 file changed, 80 insertions(+) > >> > >> diff --git a/Documentation/topics/dpdk/phy.rst > >> b/Documentation/topics/dpdk/phy.rst > >> index 1470623..3792fde 100644 > >> --- a/Documentation/topics/dpdk/phy.rst > >> +++ b/Documentation/topics/dpdk/phy.rst > >> @@ -219,6 +219,86 @@ For more information please refer to the `DPDK > Port Hotplug Framework`__. > >> ... > >> + > >> +Representors > >> +------------ ..... > >> +This syntax shows that a representor is an enumerated eth device > >> +(with a representor ID) which uses the PF PCI address. > >> +The following commands add representors 3 and 5 using PCI device > >> +address > >> +``0000:08:00.0``:: > >> + > >> + $ ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 > type=dpdk \ > >> + options:dpdk-devargs=0000:08:00.0,representor=[3] > >> + > >> + $ ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 > type=dpdk \ > >> + options:dpdk-devargs=0000:08:00.0,representor=[5] > >> + > >> +.. important:: > >> + > >> + Representors ports are configured prior to OVS invocation and > independently > >> + of it, or by other means as well. Please consult a NIC vendor > instructions > >> + on how to establish representors. > > > > It'll be good to have configuration example for at least one commonly > > used NIC (ixgbe/i40e ?). Or maybe a link to the docs where the process > described. > > > > What do you think ? > > Ian, maybe you could add some example, since you have already tried it in > practice? > > > > Good call, I'll draw up an incremental and post here, if acceptable we can roll > it into the same patch. > In addition to Ian drawing - please find a link which details how to create VFs for ConnectX-4 5 or 6 NICs: https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html section "Create Compute virtual functions". I will send an updated v2 with this reference. > Ian > >> To verify their correct configuration, > >> + execute:: > >> + > >> + $ ovs-vsctl show > >> + > >> + and make sure no errors are indicated. > >> + > >> +.. _multi-dev-configuration: > >> + > >> + > >> +Port representors are an example of multi devices. There are NICs > >> +which support multi devices by other methods than representors for > >> +which a generic devargs syntax is used. The generic syntax is based on > the device mac address:: > >> + > >> + class=eth,mac=<MAC address> > >> + > >> +For example, the following command adds a port to a bridge called > >> +``br0`` using an eth device whose mac address is ``00:11:22:33:44:55``:: > >> + > >> + $ ovs-vsctl add-port br0 dpdk-mac -- set Interface dpdk-mac > type=dpdk \ > >> + options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55" > >> + > >> Jumbo Frames > >> ------------ > >> > >>
> -----Original Message----- > > -----Original Message----- > > From: Ian Stokes <ian.stokes@intel.com> > > Sent: Tuesday, February 12, 2019 7:17 PM > > To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk > > <ophirmu@mellanox.com>; ovs-dev@openvswitch.org > > Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor > > <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas > > Monjalon <thomas@monjalon.net> > > Subject: Re: [PATCH v1] doc: Add "Representors" topic document > > > > On 2/12/2019 1:15 PM, Ilya Maximets wrote: > > > On 11.02.2019 3:01, Ophir Munk wrote: > > >> This details how to configure representors ports. > > >> > > >> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> > > >> --- > > >> Documentation/topics/dpdk/phy.rst | 80 > > +++++++++++++++++++++++++++++++++++++++ > > >> 1 file changed, 80 insertions(+) > > >> ... > > >> + > > >> +Representors > > >> +------------ > ..... > > >> + > > >> +.. important:: > > >> + > > >> + Representors ports are configured prior to OVS invocation and > > independently > > >> + of it, or by other means as well. Please consult a NIC vendor > > instructions > > >> + on how to establish representors. > > > > > > It'll be good to have configuration example for at least one > > > commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs where > > > the process > > described. > > > > > > What do you think ? > > > Ian, maybe you could add some example, since you have already tried > > > it in > > practice? > > > > > > > Good call, I'll draw up an incremental and post here, if acceptable we > > can roll it into the same patch. > > > > In addition to Ian drawing - please find a link which details how to create VFs > for ConnectX-4 5 or 6 NICs: > https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html > section "Create Compute virtual functions". > I will send an updated v2 with this reference. > > > Ian After giving it more thought - IMHO it's better to add a new document under howto/ where NIC vendors could specify their configuration for representors setup. We can then refer from this document to the new one. Ilay, Ian - what do you think?
On 2/13/2019 8:14 AM, Ophir Munk wrote: > > >> -----Original Message----- >>> -----Original Message----- >>> From: Ian Stokes <ian.stokes@intel.com> >>> Sent: Tuesday, February 12, 2019 7:17 PM >>> To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk >>> <ophirmu@mellanox.com>; ovs-dev@openvswitch.org >>> Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor >>> <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas >>> Monjalon <thomas@monjalon.net> >>> Subject: Re: [PATCH v1] doc: Add "Representors" topic document >>> >>> On 2/12/2019 1:15 PM, Ilya Maximets wrote: >>>> On 11.02.2019 3:01, Ophir Munk wrote: >>>>> This details how to configure representors ports. >>>>> >>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> >>>>> --- >>>>> Documentation/topics/dpdk/phy.rst | 80 >>> +++++++++++++++++++++++++++++++++++++++ >>>>> 1 file changed, 80 insertions(+) >>>>> > ... >>>>> + >>>>> +Representors >>>>> +------------ >> ..... >>>>> + >>>>> +.. important:: >>>>> + >>>>> + Representors ports are configured prior to OVS invocation and >>> independently >>>>> + of it, or by other means as well. Please consult a NIC vendor >>> instructions >>>>> + on how to establish representors. >>>> >>>> It'll be good to have configuration example for at least one >>>> commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs where >>>> the process >>> described. >>>> >>>> What do you think ? >>>> Ian, maybe you could add some example, since you have already tried >>>> it in >>> practice? >>>> >>> >>> Good call, I'll draw up an incremental and post here, if acceptable we >>> can roll it into the same patch. >>> >> >> In addition to Ian drawing - please find a link which details how to create VFs >> for ConnectX-4 5 or 6 NICs: >> https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html >> section "Create Compute virtual functions". >> I will send an updated v2 with this reference. >> >>> Ian > > After giving it more thought - IMHO it's better to add a new document under howto/ where NIC vendors could specify their configuration for representors setup. > We can then refer from this document to the new one. > Ilay, Ian - what do you think? > Hmmm, for ixgbe/i40e that are bound to a dpdk compatible driver it's just a case of creating the vf with something similar to below echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs And then adding it as described above ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep0 type=dpdk \ options:dpdk-devargs=0000:05:00.0,representor=[0] I'm not sure that it would warrant it's own document for those devices, how does it compare to the mellanox creation of vfs? Is that more complex or different? Ian
On 13.02.2019 19:39, Ian Stokes wrote: > On 2/13/2019 8:14 AM, Ophir Munk wrote: >> >> >>> -----Original Message----- >>>> -----Original Message----- >>>> From: Ian Stokes <ian.stokes@intel.com> >>>> Sent: Tuesday, February 12, 2019 7:17 PM >>>> To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk >>>> <ophirmu@mellanox.com>; ovs-dev@openvswitch.org >>>> Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor >>>> <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas >>>> Monjalon <thomas@monjalon.net> >>>> Subject: Re: [PATCH v1] doc: Add "Representors" topic document >>>> >>>> On 2/12/2019 1:15 PM, Ilya Maximets wrote: >>>>> On 11.02.2019 3:01, Ophir Munk wrote: >>>>>> This details how to configure representors ports. >>>>>> >>>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> >>>>>> --- >>>>>> Documentation/topics/dpdk/phy.rst | 80 >>>> +++++++++++++++++++++++++++++++++++++++ >>>>>> 1 file changed, 80 insertions(+) >>>>>> >> ... >>>>>> + >>>>>> +Representors >>>>>> +------------ >>> ..... >>>>>> + >>>>>> +.. important:: >>>>>> + >>>>>> + Representors ports are configured prior to OVS invocation and >>>> independently >>>>>> + of it, or by other means as well. Please consult a NIC vendor >>>> instructions >>>>>> + on how to establish representors. >>>>> >>>>> It'll be good to have configuration example for at least one >>>>> commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs where >>>>> the process >>>> described. >>>>> >>>>> What do you think ? >>>>> Ian, maybe you could add some example, since you have already tried >>>>> it in >>>> practice? >>>>> >>>> >>>> Good call, I'll draw up an incremental and post here, if acceptable we >>>> can roll it into the same patch. >>>> >>> >>> In addition to Ian drawing - please find a link which details how to create VFs >>> for ConnectX-4 5 or 6 NICs: >>> https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html >>> section "Create Compute virtual functions". >>> I will send an updated v2 with this reference. >>> >>>> Ian >> >> After giving it more thought - IMHO it's better to add a new document under howto/ where NIC vendors could specify their configuration for representors setup. >> We can then refer from this document to the new one. >> Ilay, Ian - what do you think? >> > > Hmmm, for ixgbe/i40e that are bound to a dpdk compatible driver it's just a case of creating the vf with something similar to below > > echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs > > And then adding it as described above > > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep0 type=dpdk \ > options:dpdk-devargs=0000:05:00.0,representor=[0] > > I'm not sure that it would warrant it's own document for those devices, how does it compare to the mellanox creation of vfs? Is that more complex or different? I know that to make representors appear as linux interfaces (for using them in OVS kernel datapath, for example) you need to turn the PF into "switchdev" mode. Probably, the same needed with DPDK. Anyway, The instructions for most of NICs, I guess, should be "just echo number of VFs to 'max_vfs' sysfs knob". We may add this information right here and make a remark that some NICs may require additional steps. We may also point to the openstack ovs-offload guide, first parts of it describes how to configure representors with Mellanox NICs. IMHO, this amount of information should be enough and it does not deserve a separate manual.
On Thu, Feb 14, 2019 at 01:47:27PM +0300, Ilya Maximets wrote: > On 13.02.2019 19:39, Ian Stokes wrote: > > On 2/13/2019 8:14 AM, Ophir Munk wrote: > >> > >> > >>> -----Original Message----- > >>>> -----Original Message----- > >>>> From: Ian Stokes <ian.stokes@intel.com> > >>>> Sent: Tuesday, February 12, 2019 7:17 PM > >>>> To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk > >>>> <ophirmu@mellanox.com>; ovs-dev@openvswitch.org > >>>> Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor > >>>> <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas > >>>> Monjalon <thomas@monjalon.net> > >>>> Subject: Re: [PATCH v1] doc: Add "Representors" topic document > >>>> > >>>> On 2/12/2019 1:15 PM, Ilya Maximets wrote: > >>>>> On 11.02.2019 3:01, Ophir Munk wrote: > >>>>>> This details how to configure representors ports. > >>>>>> > >>>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> > >>>>>> --- > >>>>>> Documentation/topics/dpdk/phy.rst | 80 > >>>> +++++++++++++++++++++++++++++++++++++++ > >>>>>> 1 file changed, 80 insertions(+) > >>>>>> > >> ... > >>>>>> + > >>>>>> +Representors > >>>>>> +------------ > >>> ..... > >>>>>> + > >>>>>> +.. important:: > >>>>>> + > >>>>>> + Representors ports are configured prior to OVS invocation and > >>>> independently > >>>>>> + of it, or by other means as well. Please consult a NIC vendor > >>>> instructions > >>>>>> + on how to establish representors. > >>>>> > >>>>> It'll be good to have configuration example for at least one > >>>>> commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs where > >>>>> the process > >>>> described. > >>>>> > >>>>> What do you think ? > >>>>> Ian, maybe you could add some example, since you have already tried > >>>>> it in > >>>> practice? > >>>>> > >>>> > >>>> Good call, I'll draw up an incremental and post here, if acceptable we > >>>> can roll it into the same patch. > >>>> > >>> > >>> In addition to Ian drawing - please find a link which details how to create VFs > >>> for ConnectX-4 5 or 6 NICs: > >>> https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html > >>> section "Create Compute virtual functions". > >>> I will send an updated v2 with this reference. > >>> > >>>> Ian > >> > >> After giving it more thought - IMHO it's better to add a new document under howto/ where NIC vendors could specify their configuration for representors setup. > >> We can then refer from this document to the new one. > >> Ilay, Ian - what do you think? > >> > > > > Hmmm, for ixgbe/i40e that are bound to a dpdk compatible driver it's just a case of creating the vf with something similar to below > > > > echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs > > > > And then adding it as described above > > > > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep0 type=dpdk \ > > options:dpdk-devargs=0000:05:00.0,representor=[0] > > > > I'm not sure that it would warrant it's own document for those devices, how does it compare to the mellanox creation of vfs? Is that more complex or different? > > I know that to make representors appear as linux interfaces (for using them > in OVS kernel datapath, for example) you need to turn the PF into "switchdev" > mode. Probably, the same needed with DPDK. > > Anyway, The instructions for most of NICs, I guess, should be "just echo number > of VFs to 'max_vfs' sysfs knob". We may add this information right here and > make a remark that some NICs may require additional steps. We may also point > to the openstack ovs-offload guide, first parts of it describes how to configure > representors with Mellanox NICs. > IMHO, this amount of information should be enough and it does not deserve a > separate manual. I agree it would be very useful to have instructions on how to put the card in the correct state to use the representors. However, I'd avoid pointing to other guides outside of OVS because we don't control them, so they can change and move and our side will break. Also that usually they include more instruction/details than actually is needed to make it work, which can be confusing. I think people building OSP, for instance (but could be any other project), would look after OVS documentation on how to build their side and not vice-versa :-). fbl
> -----Original Message----- > From: Ilya Maximets <i.maximets@samsung.com> > Sent: Thursday, February 14, 2019 12:47 PM > To: Ian Stokes <ian.stokes@intel.com>; Ophir Munk > <ophirmu@mellanox.com>; ovs-dev@openvswitch.org > Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor > <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas > Monjalon <thomas@monjalon.net> > Subject: Re: [PATCH v1] doc: Add "Representors" topic document > > On 13.02.2019 19:39, Ian Stokes wrote: > > On 2/13/2019 8:14 AM, Ophir Munk wrote: > >> > >> > >>> -----Original Message----- > >>>> -----Original Message----- > >>>> From: Ian Stokes <ian.stokes@intel.com> > >>>> Sent: Tuesday, February 12, 2019 7:17 PM > >>>> To: Ilya Maximets <i.maximets@samsung.com>; Ophir Munk > >>>> <ophirmu@mellanox.com>; ovs-dev@openvswitch.org > >>>> Cc: Olga Shern <olgas@mellanox.com>; Kevin Traynor > >>>> <ktraynor@redhat.com>; Asaf Penso <asafp@mellanox.com>; Thomas > >>>> Monjalon <thomas@monjalon.net> > >>>> Subject: Re: [PATCH v1] doc: Add "Representors" topic document > >>>> > >>>> On 2/12/2019 1:15 PM, Ilya Maximets wrote: > >>>>> On 11.02.2019 3:01, Ophir Munk wrote: > >>>>>> This details how to configure representors ports. > >>>>>> > >>>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com> > >>>>>> --- > >>>>>> Documentation/topics/dpdk/phy.rst | 80 > >>>> +++++++++++++++++++++++++++++++++++++++ > >>>>>> 1 file changed, 80 insertions(+) > >>>>>> > >> ... > >>>>>> + > >>>>>> +Representors > >>>>>> +------------ > >>> ..... > >>>>>> + > >>>>>> +.. important:: > >>>>>> + > >>>>>> + Representors ports are configured prior to OVS invocation and > >>>> independently > >>>>>> + of it, or by other means as well. Please consult a NIC vendor > >>>> instructions > >>>>>> + on how to establish representors. > >>>>> > >>>>> It'll be good to have configuration example for at least one > >>>>> commonly used NIC (ixgbe/i40e ?). Or maybe a link to the docs > >>>>> where the process > >>>> described. > >>>>> > >>>>> What do you think ? > >>>>> Ian, maybe you could add some example, since you have already > >>>>> tried it in > >>>> practice? > >>>>> > >>>> > >>>> Good call, I'll draw up an incremental and post here, if acceptable > >>>> we can roll it into the same patch. > >>>> > >>> > >>> In addition to Ian drawing - please find a link which details how to > >>> create VFs for ConnectX-4 5 or 6 NICs: > >>> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd > >>> ocs.openstack.org%2Fneutron%2Frocky%2Fadmin%2Fconfig-ovs- > offload.htm > >>> > l&data=02%7C01%7Cophirmu%40mellanox.com%7C53cb5ae93dbd423a > 481608 > >>> > d69269d466%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C6368573 > 80568 > >>> > 209851&sdata=Uz60DuaXDBOk7%2BkSuEG6iGk4qAOg3OfkYBj2BbUKK5 > k%3D&am > >>> p;reserved=0 section "Create Compute virtual functions". > >>> I will send an updated v2 with this reference. > >>> > >>>> Ian > >> > >> After giving it more thought - IMHO it's better to add a new document > under howto/ where NIC vendors could specify their configuration for > representors setup. > >> We can then refer from this document to the new one. > >> Ilay, Ian - what do you think? > >> > > > > Hmmm, for ixgbe/i40e that are bound to a dpdk compatible driver it's > > just a case of creating the vf with something similar to below > > > > echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs > > > > And then adding it as described above > > > > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep0 type=dpdk > > \ options:dpdk-devargs=0000:05:00.0,representor=[0] > > > > I'm not sure that it would warrant it's own document for those devices, > how does it compare to the mellanox creation of vfs? Is that more complex > or different? > > I know that to make representors appear as linux interfaces (for using them > in OVS kernel datapath, for example) you need to turn the PF into > "switchdev" > mode. Probably, the same needed with DPDK. > Exactly. For Mellanox NICs we should explicitly turn the PF into "switchdev". Those are additional commands, not mentioned here. > Anyway, The instructions for most of NICs, I guess, should be "just echo > number of VFs to 'max_vfs' sysfs knob". We may add this information right > here and make a remark that some NICs may require additional steps. Agreed. I will add them in v2. > We > may also point to the openstack ovs-offload guide, first parts of it describes > how to configure representors with Mellanox NICs. Agreed. In v2 I will add the link to the openstack ovs-offload guilde. > IMHO, this amount of information should be enough and it does not deserve > a separate manual. Agreed. Many thanks for the prompt review.
> -----Original Message----- > From: Flavio Leitner <fbl@sysclose.org> > Sent: Thursday, February 14, 2019 2:56 PM > To: Ilya Maximets <i.maximets@samsung.com> > Cc: Ian Stokes <ian.stokes@intel.com>; Ophir Munk > <ophirmu@mellanox.com>; ovs-dev@openvswitch.org; Thomas Monjalon > <thomas@monjalon.net> > Subject: Re: [ovs-dev] [PATCH v1] doc: Add "Representors" topic document > > On Thu, Feb 14, 2019 at 01:47:27PM +0300, Ilya Maximets wrote: > > On 13.02.2019 19:39, Ian Stokes wrote: > > > On 2/13/2019 8:14 AM, Ophir Munk wrote: > > >> > > > > Anyway, The instructions for most of NICs, I guess, should be "just > > echo number of VFs to 'max_vfs' sysfs knob". We may add this > > information right here and make a remark that some NICs may require > > additional steps. We may also point to the openstack ovs-offload > > guide, first parts of it describes how to configure representors with > Mellanox NICs. > > IMHO, this amount of information should be enough and it does not > > deserve a separate manual. > > I agree it would be very useful to have instructions on how to put the card in > the correct state to use the representors. However, I'd avoid pointing to > other guides outside of OVS because we don't control them, so they can > change and move and our side will break. Also that usually they include more > instruction/details than actually is needed to make it work, which can be > confusing. > > I think people building OSP, for instance (but could be any other project), > would look after OVS documentation on how to build their side and not vice- > versa :-). > > fbl > I can drop the link to the external documents. The openstack document for example refers to ovs-kernel which may be confused with ovs-dpdk. To avoid confusion I will add all the information inside the OVS document here.
diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst index 1470623..3792fde 100644 --- a/Documentation/topics/dpdk/phy.rst +++ b/Documentation/topics/dpdk/phy.rst @@ -219,6 +219,86 @@ For more information please refer to the `DPDK Port Hotplug Framework`__. __ http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug +.. _representors: + +Representors +------------ + +DPDK representors enable configuring a phy port to a guest (VM) machine. + +OVS resides in the hypervisor which has one or more physical interfaces also +known as the physical functions (PFs). If a PF supports SR-IOV it can be used +to enable communication with the VMs via Virtual Functions (VFs). +The VFs are virtual PCIe devices created from the physical Ethernet controller. + +DPDK models a physical interface as a rte device on top of which an eth +device is created. +DPDK (version 18.xx) introduced the representors eth devices. +A representor device represents the VF eth device (VM side) on the hypervisor +side and operates on top of a PF. +Representors are multi devices created on top of one PF. + +For more information, refer to the `DPDK documentation`__. + +__ https://doc.dpdk.org/guides-18.11/prog_guide/switch_representation.html + +Prior to port representors there was a one-to-one relationship between the PF +and the eth device. With port representors the relationship becomes one PF to +many eth devices. +In case of two representors ports, when one of the ports is closed - the PCI +bus cannot be detached until the second representor port is closed as well. + +.. _representors-configuration: + +When configuring a PF-based port, OVS traditionally assigns the device PCI +address in devargs. For an existing bridge called ``br0`` and PCI address +``0000:08:00.0`` an ``add-port`` command is written as:: + + $ ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk \ + options:dpdk-devargs=0000:08:00.0 + +When configuring a VF-based port, DPDK uses an extended devargs syntax which +has the following format:: + + BDBF,representor=[<representor id>] + +This syntax shows that a representor is an enumerated eth device (with +a representor ID) which uses the PF PCI address. +The following commands add representors 3 and 5 using PCI device address +``0000:08:00.0``:: + + $ ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ + options:dpdk-devargs=0000:08:00.0,representor=[3] + + $ ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk \ + options:dpdk-devargs=0000:08:00.0,representor=[5] + +.. important:: + + Representors ports are configured prior to OVS invocation and independently + of it, or by other means as well. Please consult a NIC vendor instructions + on how to establish representors. To verify their correct configuration, + execute:: + + $ ovs-vsctl show + + and make sure no errors are indicated. + +.. _multi-dev-configuration: + + +Port representors are an example of multi devices. There are NICs which support +multi devices by other methods than representors for which a generic devargs +syntax is used. The generic syntax is based on the device mac address:: + + class=eth,mac=<MAC address> + +For example, the following command adds a port to a bridge called ``br0`` using +an eth device whose mac address is ``00:11:22:33:44:55``:: + + $ ovs-vsctl add-port br0 dpdk-mac -- set Interface dpdk-mac type=dpdk \ + options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55" + Jumbo Frames ------------
This details how to configure representors ports. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> --- Documentation/topics/dpdk/phy.rst | 80 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+)