[ovs-dev,RFC] netdev-dpdk: add af_xdp support.
diff mbox series

Message ID 1559074364-7635-1-git-send-email-u9012063@gmail.com
State Rejected
Headers show
Series
  • [ovs-dev,RFC] netdev-dpdk: add af_xdp support.
Related show

Commit Message

William Tu May 28, 2019, 8:12 p.m. UTC
The patch adds support for using DPDK's AF_XDP PMD.
Base on DPDK 19.08-rc0:
  commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3
  Author: Thomas Monjalon <thomas@monjalon.net>
  Date:   Tue May 14 18:04:37 2019 +0200

      version: 19.08-rc0

An AF_XDP PMD can be created by:

  $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
       options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0

  $ ovs-vsctl show
  Port "afxdp-p0"
    Interface "afxdp-p0"
      type: dpdk
      options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}

The performance of physical device loopback shows around 7Mpps with 64B pkt.

  $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
       actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"

Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.

Signed-off-by: William Tu <u9012063@gmail.com>
---
 lib/netdev-dpdk.c | 38 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 36 insertions(+), 2 deletions(-)

Comments

Ilya Maximets May 29, 2019, 1:15 p.m. UTC | #1
On 28.05.2019 23:12, William Tu wrote:
> The patch adds support for using DPDK's AF_XDP PMD.
> Base on DPDK 19.08-rc0:
>   commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3
>   Author: Thomas Monjalon <thomas@monjalon.net>
>   Date:   Tue May 14 18:04:37 2019 +0200
> 
>       version: 19.08-rc0
> 
> An AF_XDP PMD can be created by:
> 
>   $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
>        options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0

I'm confused. I didn't try, but I thought that this should work out-of-the-box
without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe()
that we're using right now are the same function with different way of passing
arguments. Have you tried to just use above command without this patch applied?
What happens in this case?

Another option you may try is:

ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
          options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"

so DPDK will not have to guess the bus type.

> 
>   $ ovs-vsctl show
>   Port "afxdp-p0"
>     Interface "afxdp-p0"
>       type: dpdk
>       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> 
> The performance of physical device loopback shows around 7Mpps with 64B pkt.
> 
>   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
>        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> 
> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.

You may use 'dpdk-latest' OVS branch. The proper fix exists there.
In general, patches that intended to work only with latest DPDK that
not yet supported on master should be made on top of 'dpdk-latest'
branch and have [PATCH dpdk-latest] subject prefix.

Best regards, Ilya Maximets.
William Tu May 29, 2019, 5:51 p.m. UTC | #2
Hi Ilya,

Thanks for your feedback.

On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote:
>
> On 28.05.2019 23:12, William Tu wrote:
> > The patch adds support for using DPDK's AF_XDP PMD.
> > Base on DPDK 19.08-rc0:
> >   commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3
> >   Author: Thomas Monjalon <thomas@monjalon.net>
> >   Date:   Tue May 14 18:04:37 2019 +0200
> >
> >       version: 19.08-rc0
> >
> > An AF_XDP PMD can be created by:
> >
> >   $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
> >        options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0
>
> I'm confused. I didn't try, but I thought that this should work out-of-the-box
> without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe()
> that we're using right now are the same function with different way of passing
> arguments. Have you tried to just use above command without this patch applied?
> What happens in this case?
>
> Another option you may try is:
>
> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
>           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
>
It actually works! When adding the device:
2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
0, core id: 11 created.
2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
threads on numa node 0
2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
not supported on port 0

However , when removing the device, some errors show up.
ovs-vsctl del-port br0 afxdp-p0
2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
interface afxdp-p0 on port 1
2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
0, core id: 11 destroyed.
2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
device (net_af_xdp)
2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
on primary process
2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached

I also measured the performance, it works fine and shows around 7Mpps.

> so DPDK will not have to guess the bus type.
>
> >
> >   $ ovs-vsctl show
> >   Port "afxdp-p0"
> >     Interface "afxdp-p0"
> >       type: dpdk
> >       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> >
> > The performance of physical device loopback shows around 7Mpps with 64B pkt.
> >
> >   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
> >        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> >
> > Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
>
> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
> In general, patches that intended to work only with latest DPDK that
> not yet supported on master should be made on top of 'dpdk-latest'
> branch and have [PATCH dpdk-latest] subject prefix.

Got it, thank you.
William

>
> Best regards, Ilya Maximets.
William Tu May 29, 2019, 5:57 p.m. UTC | #3
On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote:
>
> Hi Ilya,
>
> Thanks for your feedback.
>
> On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote:
> >
> > On 28.05.2019 23:12, William Tu wrote:
> > > The patch adds support for using DPDK's AF_XDP PMD.
> > > Base on DPDK 19.08-rc0:
> > >   commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3
> > >   Author: Thomas Monjalon <thomas@monjalon.net>
> > >   Date:   Tue May 14 18:04:37 2019 +0200
> > >
> > >       version: 19.08-rc0
> > >
> > > An AF_XDP PMD can be created by:
> > >
> > >   $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
> > >        options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0
> >
> > I'm confused. I didn't try, but I thought that this should work out-of-the-box
> > without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe()
> > that we're using right now are the same function with different way of passing
> > arguments. Have you tried to just use above command without this patch applied?
> > What happens in this case?

So without my patch, if I do:
ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
      options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0

It also works:
2019-05-29T17:53:39.981Z|00044|netdev_dpdk|INFO|Device
'net_af_xdp,iface=enp2s0,queue=0' attached to DPDK

ovs-vsctl show
Port "afxdp-p0"
            Interface "afxdp-p0"
                type: dpdk
                options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}

And the only problem is when removing the device
ovs-vsctl del-port afxdp-p0
2019-05-29T17:54:27.671Z|00055|dpdk|ERR|EAL: Driver cannot detach the
device (net_af_xdp)
2019-05-29T17:54:27.671Z|00056|dpdk|ERR|EAL: Failed to detach device
on primary process
2019-05-29T17:54:27.671Z|00057|netdev_dpdk|ERR|Device
'net_af_xdp,iface=enp2s0,queue=0' can not be detached

Thanks,
William
> >
> > Another option you may try is:
> >
> > ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
> >           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
> >
> It actually works! When adding the device:
> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
> 0, core id: 11 created.
> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
> threads on numa node 0
> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
> not supported on port 0
>
> However , when removing the device, some errors show up.
> ovs-vsctl del-port br0 afxdp-p0
> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
> interface afxdp-p0 on port 1
> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
> 0, core id: 11 destroyed.
> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
> device (net_af_xdp)
> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
> on primary process
> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
>
> I also measured the performance, it works fine and shows around 7Mpps.
>
> > so DPDK will not have to guess the bus type.
> >
> > >
> > >   $ ovs-vsctl show
> > >   Port "afxdp-p0"
> > >     Interface "afxdp-p0"
> > >       type: dpdk
> > >       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> > >
> > > The performance of physical device loopback shows around 7Mpps with 64B pkt.
> > >
> > >   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
> > >        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> > >
> > > Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
> >
> > You may use 'dpdk-latest' OVS branch. The proper fix exists there.
> > In general, patches that intended to work only with latest DPDK that
> > not yet supported on master should be made on top of 'dpdk-latest'
> > branch and have [PATCH dpdk-latest] subject prefix.
>
> Got it, thank you.
> William
>
> >
> > Best regards, Ilya Maximets.
Ilya Maximets May 30, 2019, 7:47 a.m. UTC | #4
On 29.05.2019 20:57, William Tu wrote:
> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote:
>>
>> Hi Ilya,
>>
>> Thanks for your feedback.
>>
>> On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote:
>>>
>>> On 28.05.2019 23:12, William Tu wrote:
>>>> The patch adds support for using DPDK's AF_XDP PMD.
>>>> Base on DPDK 19.08-rc0:
>>>>   commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3
>>>>   Author: Thomas Monjalon <thomas@monjalon.net>
>>>>   Date:   Tue May 14 18:04:37 2019 +0200
>>>>
>>>>       version: 19.08-rc0
>>>>
>>>> An AF_XDP PMD can be created by:
>>>>
>>>>   $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
>>>>        options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0
>>>
>>> I'm confused. I didn't try, but I thought that this should work out-of-the-box
>>> without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe()
>>> that we're using right now are the same function with different way of passing
>>> arguments. Have you tried to just use above command without this patch applied?
>>> What happens in this case?
> 
> So without my patch, if I do:
> ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \
>       options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0
> 
> It also works:
> 2019-05-29T17:53:39.981Z|00044|netdev_dpdk|INFO|Device
> 'net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
> 
> ovs-vsctl show
> Port "afxdp-p0"
>             Interface "afxdp-p0"
>                 type: dpdk
>                 options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> 
> And the only problem is when removing the device
> ovs-vsctl del-port afxdp-p0
> 2019-05-29T17:54:27.671Z|00055|dpdk|ERR|EAL: Driver cannot detach the
> device (net_af_xdp)
> 2019-05-29T17:54:27.671Z|00056|dpdk|ERR|EAL: Failed to detach device
> on primary process
> 2019-05-29T17:54:27.671Z|00057|netdev_dpdk|ERR|Device
> 'net_af_xdp,iface=enp2s0,queue=0' can not be detached
> 
> Thanks,
> William

Thanks for the logs. It looks like a bug in DPDK.
Does rte_eal_hotplug_remove() works without issues?
Unfortunately, I have no setup to test myself.

You may also use following patch to get more information about detaching
failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html

Another possible issue is that you're using driver base name (net_af_xdp)
as a name of DPDK device. Maybe this causes some issues.
Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".

Best regards, Ilya Maximets.

>>>
>>> Another option you may try is:
>>>
>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
>>>           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
>>>
>> It actually works! When adding the device:
>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 11 created.
>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
>> threads on numa node 0
>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
>> not supported on port 0
>>
>> However , when removing the device, some errors show up.
>> ovs-vsctl del-port br0 afxdp-p0
>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
>> interface afxdp-p0 on port 1
>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 11 destroyed.
>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
>> device (net_af_xdp)
>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
>> on primary process
>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
>>
>> I also measured the performance, it works fine and shows around 7Mpps.
>>
>>> so DPDK will not have to guess the bus type.
>>>
>>>>
>>>>   $ ovs-vsctl show
>>>>   Port "afxdp-p0"
>>>>     Interface "afxdp-p0"
>>>>       type: dpdk
>>>>       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
>>>>
>>>> The performance of physical device loopback shows around 7Mpps with 64B pkt.
>>>>
>>>>   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
>>>>        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
>>>>
>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
>>>
>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
>>> In general, patches that intended to work only with latest DPDK that
>>> not yet supported on master should be made on top of 'dpdk-latest'
>>> branch and have [PATCH dpdk-latest] subject prefix.
>>
>> Got it, thank you.
>> William
>>
>>>
>>> Best regards, Ilya Maximets.
> 
>
William Tu May 30, 2019, 10:08 p.m. UTC | #5
Hi Ilya,

Thanks for the comments.

On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote:
>
> On 29.05.2019 20:57, William Tu wrote:
> > On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote:
> >>
>
> Thanks for the logs. It looks like a bug in DPDK.
> Does rte_eal_hotplug_remove() works without issues?
> Unfortunately, I have no setup to test myself.

Right, no issue. reason below:

>
> You may also use following patch to get more information about detaching
> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html

Thanks, -1 is returned
2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device
'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation
not permitted.

The root cause is at
rte_dev_remove
  local_dev_remove
    ret = dev->bus->unplug(dev);
      rte_pmd_af_xdp_remove()

At  rte_pmd_af_xdp_remove()
/* find the ethdev entry */
  eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
  if (eth_dev == NULL)
    return -1;  // failed here

And the reason rte_eth_dev_allocated find no device is that
at _rte_eth_dev_allocated
the rte_eth_devices[i].data is an empty string.

And the reason is that at rte_eth_dev_close
it calls
    if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) {
        /* new behaviour: send event + reset state + free all data */
        rte_eth_dev_release_port(dev);

the rte_eth_dev_release_port clear the dev->data->name
        memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));

>
> Another possible issue is that you're using driver base name (net_af_xdp)
> as a name of DPDK device. Maybe this causes some issues.
> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".

Yes, I use the above comment.

Regards,
William

>
> Best regards, Ilya Maximets.
>
> >>>
> >>> Another option you may try is:
> >>>
> >>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
> >>>           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
> >>>
> >> It actually works! When adding the device:
> >> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
> >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
> >> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
> >> 0, core id: 11 created.
> >> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
> >> threads on numa node 0
> >> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
> >> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
> >> not supported on port 0
> >>
> >> However , when removing the device, some errors show up.
> >> ovs-vsctl del-port br0 afxdp-p0
> >> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
> >> interface afxdp-p0 on port 1
> >> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
> >> 0, core id: 11 destroyed.
> >> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
> >> device (net_af_xdp)
> >> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
> >> on primary process
> >> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
> >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
> >>
> >> I also measured the performance, it works fine and shows around 7Mpps.
> >>
> >>> so DPDK will not have to guess the bus type.
> >>>
> >>>>
> >>>>   $ ovs-vsctl show
> >>>>   Port "afxdp-p0"
> >>>>     Interface "afxdp-p0"
> >>>>       type: dpdk
> >>>>       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> >>>>
> >>>> The performance of physical device loopback shows around 7Mpps with 64B pkt.
> >>>>
> >>>>   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
> >>>>        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> >>>>
> >>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
> >>>
> >>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
> >>> In general, patches that intended to work only with latest DPDK that
> >>> not yet supported on master should be made on top of 'dpdk-latest'
> >>> branch and have [PATCH dpdk-latest] subject prefix.
> >>
> >> Got it, thank you.
> >> William
> >>
> >>>
> >>> Best regards, Ilya Maximets.
> >
> >
Ilya Maximets May 31, 2019, 3:58 p.m. UTC | #6
On 31.05.2019 1:08, William Tu wrote:
> Hi Ilya,
> 
> Thanks for the comments.
> 
> On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote:
>>
>> On 29.05.2019 20:57, William Tu wrote:
>>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote:
>>>>
>>
>> Thanks for the logs. It looks like a bug in DPDK.
>> Does rte_eal_hotplug_remove() works without issues?
>> Unfortunately, I have no setup to test myself.
> 
> Right, no issue. reason below:
> 
>>
>> You may also use following patch to get more information about detaching
>> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html
> 
> Thanks, -1 is returned
> 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device
> 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation
> not permitted.
> 
> The root cause is at
> rte_dev_remove
>   local_dev_remove
>     ret = dev->bus->unplug(dev);
>       rte_pmd_af_xdp_remove()
> 
> At  rte_pmd_af_xdp_remove()
> /* find the ethdev entry */
>   eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
>   if (eth_dev == NULL)
>     return -1;  // failed here

Thanks for detailed investigation. So, it's just a DPDK bug.
I think that we need to simply replace 'return -1' with 'return 0' here.

The life-cycle of a DPDK port is following:

    rte_dev_probe()     <-- attaches device to dpdk bus.
    rte_eth_dev_configure()
    rte_eth_dev_start()
    rte_eth_dev_stop()
    rte_eth_dev_close() <-- closes the eth device.
    rte_dev_remove()    <-- detaches device from dpdk bus.

af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all
the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach
device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove()
that tries to free already freed resources and fails.
I think that rte_pmd_af_xdp_remove() should just ignore this case
reporting successful removal, otherwise device will never be detached
from vdev bus reporting failure of driver->remove() callback.

Will you prepare the patch for this?
If you wish, I could prepare it myself on Monday.

Best regards, Ilya Maximets.

> 
> And the reason rte_eth_dev_allocated find no device is that
> at _rte_eth_dev_allocated
> the rte_eth_devices[i].data is an empty string.
> 
> And the reason is that at rte_eth_dev_close
> it calls
>     if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) {
>         /* new behaviour: send event + reset state + free all data */
>         rte_eth_dev_release_port(dev);
> 
> the rte_eth_dev_release_port clear the dev->data->name
>         memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> 
>>
>> Another possible issue is that you're using driver base name (net_af_xdp)
>> as a name of DPDK device. Maybe this causes some issues.
>> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".
> 
> Yes, I use the above comment.
> 
> Regards,
> William
> 
>>
>> Best regards, Ilya Maximets.
>>
>>>>>
>>>>> Another option you may try is:
>>>>>
>>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
>>>>>           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
>>>>>
>>>> It actually works! When adding the device:
>>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
>>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
>>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
>>>> 0, core id: 11 created.
>>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
>>>> threads on numa node 0
>>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
>>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
>>>> not supported on port 0
>>>>
>>>> However , when removing the device, some errors show up.
>>>> ovs-vsctl del-port br0 afxdp-p0
>>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
>>>> interface afxdp-p0 on port 1
>>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
>>>> 0, core id: 11 destroyed.
>>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
>>>> device (net_af_xdp)
>>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
>>>> on primary process
>>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
>>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
>>>>
>>>> I also measured the performance, it works fine and shows around 7Mpps.
>>>>
>>>>> so DPDK will not have to guess the bus type.
>>>>>
>>>>>>
>>>>>>   $ ovs-vsctl show
>>>>>>   Port "afxdp-p0"
>>>>>>     Interface "afxdp-p0"
>>>>>>       type: dpdk
>>>>>>       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
>>>>>>
>>>>>> The performance of physical device loopback shows around 7Mpps with 64B pkt.
>>>>>>
>>>>>>   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
>>>>>>        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
>>>>>>
>>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
>>>>>
>>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
>>>>> In general, patches that intended to work only with latest DPDK that
>>>>> not yet supported on master should be made on top of 'dpdk-latest'
>>>>> branch and have [PATCH dpdk-latest] subject prefix.
>>>>
>>>> Got it, thank you.
>>>> William
>>>>
>>>>>
>>>>> Best regards, Ilya Maximets.
>>>
>>>
> 
>
William Tu May 31, 2019, 4:19 p.m. UTC | #7
On Fri, May 31, 2019 at 8:58 AM Ilya Maximets <i.maximets@samsung.com> wrote:
>
> On 31.05.2019 1:08, William Tu wrote:
> > Hi Ilya,
> >
> > Thanks for the comments.
> >
> > On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote:
> >>
> >> On 29.05.2019 20:57, William Tu wrote:
> >>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote:
> >>>>
> >>
> >> Thanks for the logs. It looks like a bug in DPDK.
> >> Does rte_eal_hotplug_remove() works without issues?
> >> Unfortunately, I have no setup to test myself.
> >
> > Right, no issue. reason below:
> >
> >>
> >> You may also use following patch to get more information about detaching
> >> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html
> >
> > Thanks, -1 is returned
> > 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device
> > 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation
> > not permitted.
> >
> > The root cause is at
> > rte_dev_remove
> >   local_dev_remove
> >     ret = dev->bus->unplug(dev);
> >       rte_pmd_af_xdp_remove()
> >
> > At  rte_pmd_af_xdp_remove()
> > /* find the ethdev entry */
> >   eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
> >   if (eth_dev == NULL)
> >     return -1;  // failed here
>
> Thanks for detailed investigation. So, it's just a DPDK bug.
> I think that we need to simply replace 'return -1' with 'return 0' here.
>
> The life-cycle of a DPDK port is following:
>
>     rte_dev_probe()     <-- attaches device to dpdk bus.
>     rte_eth_dev_configure()
>     rte_eth_dev_start()
>     rte_eth_dev_stop()
>     rte_eth_dev_close() <-- closes the eth device.
>     rte_dev_remove()    <-- detaches device from dpdk bus.
>

Thanks for the explanation. Much clear to me now.

> af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all
> the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach
> device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove()
> that tries to free already freed resources and fails.
> I think that rte_pmd_af_xdp_remove() should just ignore this case
> reporting successful removal, otherwise device will never be detached
> from vdev bus reporting failure of driver->remove() callback.
>
> Will you prepare the patch for this?

Yes, I will test it and send out patch to dpdk-dev.

Regards,
William

> If you wish, I could prepare it myself on Monday.
>
> Best regards, Ilya Maximets.
>
> >
> > And the reason rte_eth_dev_allocated find no device is that
> > at _rte_eth_dev_allocated
> > the rte_eth_devices[i].data is an empty string.
> >
> > And the reason is that at rte_eth_dev_close
> > it calls
> >     if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) {
> >         /* new behaviour: send event + reset state + free all data */
> >         rte_eth_dev_release_port(dev);
> >
> > the rte_eth_dev_release_port clear the dev->data->name
> >         memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> >
> >>
> >> Another possible issue is that you're using driver base name (net_af_xdp)
> >> as a name of DPDK device. Maybe this causes some issues.
> >> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".
> >
> > Yes, I use the above comment.
> >
> > Regards,
> > William
> >
> >>
> >> Best regards, Ilya Maximets.
> >>
> >>>>>
> >>>>> Another option you may try is:
> >>>>>
> >>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
> >>>>>           options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
> >>>>>
> >>>> It actually works! When adding the device:
> >>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
> >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
> >>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
> >>>> 0, core id: 11 created.
> >>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
> >>>> threads on numa node 0
> >>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped
> >>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
> >>>> not supported on port 0
> >>>>
> >>>> However , when removing the device, some errors show up.
> >>>> ovs-vsctl del-port br0 afxdp-p0
> >>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
> >>>> interface afxdp-p0 on port 1
> >>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
> >>>> 0, core id: 11 destroyed.
> >>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
> >>>> device (net_af_xdp)
> >>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
> >>>> on primary process
> >>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
> >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
> >>>>
> >>>> I also measured the performance, it works fine and shows around 7Mpps.
> >>>>
> >>>>> so DPDK will not have to guess the bus type.
> >>>>>
> >>>>>>
> >>>>>>   $ ovs-vsctl show
> >>>>>>   Port "afxdp-p0"
> >>>>>>     Interface "afxdp-p0"
> >>>>>>       type: dpdk
> >>>>>>       options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> >>>>>>
> >>>>>> The performance of physical device loopback shows around 7Mpps with 64B pkt.
> >>>>>>
> >>>>>>   $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
> >>>>>>        actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> >>>>>>
> >>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
> >>>>>
> >>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
> >>>>> In general, patches that intended to work only with latest DPDK that
> >>>>> not yet supported on master should be made on top of 'dpdk-latest'
> >>>>> branch and have [PATCH dpdk-latest] subject prefix.
> >>>>
> >>>> Got it, thank you.
> >>>> William
> >>>>
> >>>>>
> >>>>> Best regards, Ilya Maximets.
> >>>
> >>>
> >
> >

Patch
diff mbox series

diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index 47153dc60148..99697bc896ca 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -380,6 +380,8 @@  struct netdev_dpdk {
 
         /* If true, device was attached by rte_eth_dev_attach(). */
         bool attached;
+        /* If true, device was hop plugged by rte_eal_hotplug_add(). */
+        bool hotplugged;
         /* If true, rte_eth_dev_start() was successfully called */
         bool started;
         struct eth_addr hwaddr;
@@ -1376,6 +1378,14 @@  netdev_dpdk_destruct(struct netdev *netdev)
     rte_eth_dev_stop(dev->port_id);
     dev->started = false;
 
+    if (dev->hotplugged) {
+        int ret;
+        ret = rte_eal_hotplug_remove("vdev", "net_af_xdp");
+        if (ret) {
+            VLOG_ERR("%s unplugged error %d", __func__, ret);
+        }
+    }
+
     if (dev->attached) {
         /* Retrieve eth device data before closing it.
          * FIXME: avoid direct access to DPDK internal array rte_eth_devices.
@@ -1648,6 +1658,26 @@  netdev_dpdk_lookup_by_port_id(dpdk_port_t port_id)
 }
 
 static dpdk_port_t
+netdev_dpdk_get_af_xdp_port(const char *vdev_args)
+{
+    const char vdev_name[] = "net_af_xdp";
+    dpdk_port_t port_id;
+
+    if (rte_eal_hotplug_add("vdev", vdev_name, vdev_args) < 0) {
+        VLOG_WARN("Cannot hotplug AF_XDP device: %s\n", vdev_args);
+        return DPDK_ETH_PORT_ID_INVALID;
+    }
+
+    if (rte_eth_dev_get_port_by_name(vdev_name, &port_id) != 0) {
+        rte_eal_hotplug_remove("vdev", vdev_name);
+        VLOG_WARN("Cannot get AF_XDP port_id, device: %s\n", vdev_args);
+        return DPDK_ETH_PORT_ID_INVALID;
+    }
+
+    return port_id;
+}
+
+static dpdk_port_t
 netdev_dpdk_get_port_by_mac(const char *mac_str)
 {
     dpdk_port_t port_id;
@@ -1705,7 +1735,11 @@  netdev_dpdk_process_devargs(struct netdev_dpdk *dev,
 {
     dpdk_port_t new_port_id;
 
-    if (strncmp(devargs, "class=eth,mac=", 14) == 0) {
+    if (strncmp(devargs, "net_af_xdp", 10) == 0) {
+        new_port_id = netdev_dpdk_get_af_xdp_port(&devargs[11]);
+        dev->hotplugged = true;
+        VLOG_INFO("Device '%s' hotplugged to DPDK", devargs);
+    } else if (strncmp(devargs, "class=eth,mac=", 14) == 0) {
         new_port_id = netdev_dpdk_get_port_by_mac(&devargs[14]);
     } else {
         new_port_id = netdev_dpdk_get_port_by_devargs(devargs);
@@ -2031,7 +2065,7 @@  netdev_dpdk_policer_pkt_handle(struct rte_meter_srtcm *meter,
     uint32_t pkt_len = rte_pktmbuf_pkt_len(pkt) - sizeof(struct ether_hdr);
 
     return rte_meter_srtcm_color_blind_check(meter, profile, time, pkt_len) ==
-                                             e_RTE_METER_GREEN;
+                                             0; /* e_RTE_METER_GREEN */
 }
 
 static int