| Message ID | 1559074364-7635-1-git-send-email-u9012063@gmail.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | [ovs-dev,RFC] netdev-dpdk: add af_xdp support. | expand |
On 28.05.2019 23:12, William Tu wrote: > The patch adds support for using DPDK's AF_XDP PMD. > Base on DPDK 19.08-rc0: > commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3 > Author: Thomas Monjalon <thomas@monjalon.net> > Date: Tue May 14 18:04:37 2019 +0200 > > version: 19.08-rc0 > > An AF_XDP PMD can be created by: > > $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ > options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 I'm confused. I didn't try, but I thought that this should work out-of-the-box without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe() that we're using right now are the same function with different way of passing arguments. Have you tried to just use above command without this patch applied? What happens in this case? Another option you may try is: ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" so DPDK will not have to guess the bus type. > > $ ovs-vsctl show > Port "afxdp-p0" > Interface "afxdp-p0" > type: dpdk > options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > > The performance of physical device loopback shows around 7Mpps with 64B pkt. > > $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ > actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" > > Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. You may use 'dpdk-latest' OVS branch. The proper fix exists there. In general, patches that intended to work only with latest DPDK that not yet supported on master should be made on top of 'dpdk-latest' branch and have [PATCH dpdk-latest] subject prefix. Best regards, Ilya Maximets.
Hi Ilya, Thanks for your feedback. On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote: > > On 28.05.2019 23:12, William Tu wrote: > > The patch adds support for using DPDK's AF_XDP PMD. > > Base on DPDK 19.08-rc0: > > commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3 > > Author: Thomas Monjalon <thomas@monjalon.net> > > Date: Tue May 14 18:04:37 2019 +0200 > > > > version: 19.08-rc0 > > > > An AF_XDP PMD can be created by: > > > > $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ > > options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 > > I'm confused. I didn't try, but I thought that this should work out-of-the-box > without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe() > that we're using right now are the same function with different way of passing > arguments. Have you tried to just use above command without this patch applied? > What happens in this case? > > Another option you may try is: > > ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ > options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" > It actually works! When adding the device: 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 11 created. 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd threads on numa node 0 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is not supported on port 0 However , when removing the device, some errors show up. ovs-vsctl del-port br0 afxdp-p0 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted interface afxdp-p0 on port 1 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 11 destroyed. 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the device (net_af_xdp) 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device on primary process 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached I also measured the performance, it works fine and shows around 7Mpps. > so DPDK will not have to guess the bus type. > > > > > $ ovs-vsctl show > > Port "afxdp-p0" > > Interface "afxdp-p0" > > type: dpdk > > options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > > > > The performance of physical device loopback shows around 7Mpps with 64B pkt. > > > > $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ > > actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" > > > > Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. > > You may use 'dpdk-latest' OVS branch. The proper fix exists there. > In general, patches that intended to work only with latest DPDK that > not yet supported on master should be made on top of 'dpdk-latest' > branch and have [PATCH dpdk-latest] subject prefix. Got it, thank you. William > > Best regards, Ilya Maximets.
On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote: > > Hi Ilya, > > Thanks for your feedback. > > On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote: > > > > On 28.05.2019 23:12, William Tu wrote: > > > The patch adds support for using DPDK's AF_XDP PMD. > > > Base on DPDK 19.08-rc0: > > > commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3 > > > Author: Thomas Monjalon <thomas@monjalon.net> > > > Date: Tue May 14 18:04:37 2019 +0200 > > > > > > version: 19.08-rc0 > > > > > > An AF_XDP PMD can be created by: > > > > > > $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ > > > options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 > > > > I'm confused. I didn't try, but I thought that this should work out-of-the-box > > without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe() > > that we're using right now are the same function with different way of passing > > arguments. Have you tried to just use above command without this patch applied? > > What happens in this case? So without my patch, if I do: ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 It also works: 2019-05-29T17:53:39.981Z|00044|netdev_dpdk|INFO|Device 'net_af_xdp,iface=enp2s0,queue=0' attached to DPDK ovs-vsctl show Port "afxdp-p0" Interface "afxdp-p0" type: dpdk options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} And the only problem is when removing the device ovs-vsctl del-port afxdp-p0 2019-05-29T17:54:27.671Z|00055|dpdk|ERR|EAL: Driver cannot detach the device (net_af_xdp) 2019-05-29T17:54:27.671Z|00056|dpdk|ERR|EAL: Failed to detach device on primary process 2019-05-29T17:54:27.671Z|00057|netdev_dpdk|ERR|Device 'net_af_xdp,iface=enp2s0,queue=0' can not be detached Thanks, William > > > > Another option you may try is: > > > > ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ > > options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" > > > It actually works! When adding the device: > 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device > 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK > 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: > 0, core id: 11 created. > 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd > threads on numa node 0 > 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped > 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is > not supported on port 0 > > However , when removing the device, some errors show up. > ovs-vsctl del-port br0 afxdp-p0 > 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted > interface afxdp-p0 on port 1 > 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: > 0, core id: 11 destroyed. > 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the > device (net_af_xdp) > 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device > on primary process > 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device > 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached > > I also measured the performance, it works fine and shows around 7Mpps. > > > so DPDK will not have to guess the bus type. > > > > > > > > $ ovs-vsctl show > > > Port "afxdp-p0" > > > Interface "afxdp-p0" > > > type: dpdk > > > options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > > > > > > The performance of physical device loopback shows around 7Mpps with 64B pkt. > > > > > > $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ > > > actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" > > > > > > Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. > > > > You may use 'dpdk-latest' OVS branch. The proper fix exists there. > > In general, patches that intended to work only with latest DPDK that > > not yet supported on master should be made on top of 'dpdk-latest' > > branch and have [PATCH dpdk-latest] subject prefix. > > Got it, thank you. > William > > > > > Best regards, Ilya Maximets.
On 29.05.2019 20:57, William Tu wrote: > On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote: >> >> Hi Ilya, >> >> Thanks for your feedback. >> >> On Wed, May 29, 2019 at 6:15 AM Ilya Maximets <i.maximets@samsung.com> wrote: >>> >>> On 28.05.2019 23:12, William Tu wrote: >>>> The patch adds support for using DPDK's AF_XDP PMD. >>>> Base on DPDK 19.08-rc0: >>>> commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3 >>>> Author: Thomas Monjalon <thomas@monjalon.net> >>>> Date: Tue May 14 18:04:37 2019 +0200 >>>> >>>> version: 19.08-rc0 >>>> >>>> An AF_XDP PMD can be created by: >>>> >>>> $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ >>>> options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 >>> >>> I'm confused. I didn't try, but I thought that this should work out-of-the-box >>> without any changes to OVS. Basically, your rte_eal_hotplug_add() and rte_dev_probe() >>> that we're using right now are the same function with different way of passing >>> arguments. Have you tried to just use above command without this patch applied? >>> What happens in this case? > > So without my patch, if I do: > ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ > options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 > > It also works: > 2019-05-29T17:53:39.981Z|00044|netdev_dpdk|INFO|Device > 'net_af_xdp,iface=enp2s0,queue=0' attached to DPDK > > ovs-vsctl show > Port "afxdp-p0" > Interface "afxdp-p0" > type: dpdk > options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > > And the only problem is when removing the device > ovs-vsctl del-port afxdp-p0 > 2019-05-29T17:54:27.671Z|00055|dpdk|ERR|EAL: Driver cannot detach the > device (net_af_xdp) > 2019-05-29T17:54:27.671Z|00056|dpdk|ERR|EAL: Failed to detach device > on primary process > 2019-05-29T17:54:27.671Z|00057|netdev_dpdk|ERR|Device > 'net_af_xdp,iface=enp2s0,queue=0' can not be detached > > Thanks, > William Thanks for the logs. It looks like a bug in DPDK. Does rte_eal_hotplug_remove() works without issues? Unfortunately, I have no setup to test myself. You may also use following patch to get more information about detaching failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html Another possible issue is that you're using driver base name (net_af_xdp) as a name of DPDK device. Maybe this causes some issues. Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0". Best regards, Ilya Maximets. >>> >>> Another option you may try is: >>> >>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ >>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" >>> >> It actually works! When adding the device: >> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK >> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: >> 0, core id: 11 created. >> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd >> threads on numa node 0 >> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped >> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is >> not supported on port 0 >> >> However , when removing the device, some errors show up. >> ovs-vsctl del-port br0 afxdp-p0 >> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted >> interface afxdp-p0 on port 1 >> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: >> 0, core id: 11 destroyed. >> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the >> device (net_af_xdp) >> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device >> on primary process >> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached >> >> I also measured the performance, it works fine and shows around 7Mpps. >> >>> so DPDK will not have to guess the bus type. >>> >>>> >>>> $ ovs-vsctl show >>>> Port "afxdp-p0" >>>> Interface "afxdp-p0" >>>> type: dpdk >>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} >>>> >>>> The performance of physical device loopback shows around 7Mpps with 64B pkt. >>>> >>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ >>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" >>>> >>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. >>> >>> You may use 'dpdk-latest' OVS branch. The proper fix exists there. >>> In general, patches that intended to work only with latest DPDK that >>> not yet supported on master should be made on top of 'dpdk-latest' >>> branch and have [PATCH dpdk-latest] subject prefix. >> >> Got it, thank you. >> William >> >>> >>> Best regards, Ilya Maximets. > >
Hi Ilya, Thanks for the comments. On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote: > > On 29.05.2019 20:57, William Tu wrote: > > On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote: > >> > > Thanks for the logs. It looks like a bug in DPDK. > Does rte_eal_hotplug_remove() works without issues? > Unfortunately, I have no setup to test myself. Right, no issue. reason below: > > You may also use following patch to get more information about detaching > failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html Thanks, -1 is returned 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation not permitted. The root cause is at rte_dev_remove local_dev_remove ret = dev->bus->unplug(dev); rte_pmd_af_xdp_remove() At rte_pmd_af_xdp_remove() /* find the ethdev entry */ eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev)); if (eth_dev == NULL) return -1; // failed here And the reason rte_eth_dev_allocated find no device is that at _rte_eth_dev_allocated the rte_eth_devices[i].data is an empty string. And the reason is that at rte_eth_dev_close it calls if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) { /* new behaviour: send event + reset state + free all data */ rte_eth_dev_release_port(dev); the rte_eth_dev_release_port clear the dev->data->name memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data)); > > Another possible issue is that you're using driver base name (net_af_xdp) > as a name of DPDK device. Maybe this causes some issues. > Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0". Yes, I use the above comment. Regards, William > > Best regards, Ilya Maximets. > > >>> > >>> Another option you may try is: > >>> > >>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ > >>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" > >>> > >> It actually works! When adding the device: > >> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device > >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK > >> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: > >> 0, core id: 11 created. > >> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd > >> threads on numa node 0 > >> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped > >> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is > >> not supported on port 0 > >> > >> However , when removing the device, some errors show up. > >> ovs-vsctl del-port br0 afxdp-p0 > >> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted > >> interface afxdp-p0 on port 1 > >> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: > >> 0, core id: 11 destroyed. > >> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the > >> device (net_af_xdp) > >> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device > >> on primary process > >> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device > >> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached > >> > >> I also measured the performance, it works fine and shows around 7Mpps. > >> > >>> so DPDK will not have to guess the bus type. > >>> > >>>> > >>>> $ ovs-vsctl show > >>>> Port "afxdp-p0" > >>>> Interface "afxdp-p0" > >>>> type: dpdk > >>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > >>>> > >>>> The performance of physical device loopback shows around 7Mpps with 64B pkt. > >>>> > >>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ > >>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" > >>>> > >>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. > >>> > >>> You may use 'dpdk-latest' OVS branch. The proper fix exists there. > >>> In general, patches that intended to work only with latest DPDK that > >>> not yet supported on master should be made on top of 'dpdk-latest' > >>> branch and have [PATCH dpdk-latest] subject prefix. > >> > >> Got it, thank you. > >> William > >> > >>> > >>> Best regards, Ilya Maximets. > > > >
On 31.05.2019 1:08, William Tu wrote: > Hi Ilya, > > Thanks for the comments. > > On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote: >> >> On 29.05.2019 20:57, William Tu wrote: >>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote: >>>> >> >> Thanks for the logs. It looks like a bug in DPDK. >> Does rte_eal_hotplug_remove() works without issues? >> Unfortunately, I have no setup to test myself. > > Right, no issue. reason below: > >> >> You may also use following patch to get more information about detaching >> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html > > Thanks, -1 is returned > 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device > 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation > not permitted. > > The root cause is at > rte_dev_remove > local_dev_remove > ret = dev->bus->unplug(dev); > rte_pmd_af_xdp_remove() > > At rte_pmd_af_xdp_remove() > /* find the ethdev entry */ > eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev)); > if (eth_dev == NULL) > return -1; // failed here Thanks for detailed investigation. So, it's just a DPDK bug. I think that we need to simply replace 'return -1' with 'return 0' here. The life-cycle of a DPDK port is following: rte_dev_probe() <-- attaches device to dpdk bus. rte_eth_dev_configure() rte_eth_dev_start() rte_eth_dev_stop() rte_eth_dev_close() <-- closes the eth device. rte_dev_remove() <-- detaches device from dpdk bus. af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove() that tries to free already freed resources and fails. I think that rte_pmd_af_xdp_remove() should just ignore this case reporting successful removal, otherwise device will never be detached from vdev bus reporting failure of driver->remove() callback. Will you prepare the patch for this? If you wish, I could prepare it myself on Monday. Best regards, Ilya Maximets. > > And the reason rte_eth_dev_allocated find no device is that > at _rte_eth_dev_allocated > the rte_eth_devices[i].data is an empty string. > > And the reason is that at rte_eth_dev_close > it calls > if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) { > /* new behaviour: send event + reset state + free all data */ > rte_eth_dev_release_port(dev); > > the rte_eth_dev_release_port clear the dev->data->name > memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data)); > >> >> Another possible issue is that you're using driver base name (net_af_xdp) >> as a name of DPDK device. Maybe this causes some issues. >> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0". > > Yes, I use the above comment. > > Regards, > William > >> >> Best regards, Ilya Maximets. >> >>>>> >>>>> Another option you may try is: >>>>> >>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ >>>>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" >>>>> >>>> It actually works! When adding the device: >>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK >>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: >>>> 0, core id: 11 created. >>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd >>>> threads on numa node 0 >>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped >>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is >>>> not supported on port 0 >>>> >>>> However , when removing the device, some errors show up. >>>> ovs-vsctl del-port br0 afxdp-p0 >>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted >>>> interface afxdp-p0 on port 1 >>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: >>>> 0, core id: 11 destroyed. >>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the >>>> device (net_af_xdp) >>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device >>>> on primary process >>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached >>>> >>>> I also measured the performance, it works fine and shows around 7Mpps. >>>> >>>>> so DPDK will not have to guess the bus type. >>>>> >>>>>> >>>>>> $ ovs-vsctl show >>>>>> Port "afxdp-p0" >>>>>> Interface "afxdp-p0" >>>>>> type: dpdk >>>>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} >>>>>> >>>>>> The performance of physical device loopback shows around 7Mpps with 64B pkt. >>>>>> >>>>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ >>>>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" >>>>>> >>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. >>>>> >>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there. >>>>> In general, patches that intended to work only with latest DPDK that >>>>> not yet supported on master should be made on top of 'dpdk-latest' >>>>> branch and have [PATCH dpdk-latest] subject prefix. >>>> >>>> Got it, thank you. >>>> William >>>> >>>>> >>>>> Best regards, Ilya Maximets. >>> >>> > >
On Fri, May 31, 2019 at 8:58 AM Ilya Maximets <i.maximets@samsung.com> wrote: > > On 31.05.2019 1:08, William Tu wrote: > > Hi Ilya, > > > > Thanks for the comments. > > > > On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maximets@samsung.com> wrote: > >> > >> On 29.05.2019 20:57, William Tu wrote: > >>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012063@gmail.com> wrote: > >>>> > >> > >> Thanks for the logs. It looks like a bug in DPDK. > >> Does rte_eal_hotplug_remove() works without issues? > >> Unfortunately, I have no setup to test myself. > > > > Right, no issue. reason below: > > > >> > >> You may also use following patch to get more information about detaching > >> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html > > > > Thanks, -1 is returned > > 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device > > 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation > > not permitted. > > > > The root cause is at > > rte_dev_remove > > local_dev_remove > > ret = dev->bus->unplug(dev); > > rte_pmd_af_xdp_remove() > > > > At rte_pmd_af_xdp_remove() > > /* find the ethdev entry */ > > eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev)); > > if (eth_dev == NULL) > > return -1; // failed here > > Thanks for detailed investigation. So, it's just a DPDK bug. > I think that we need to simply replace 'return -1' with 'return 0' here. > > The life-cycle of a DPDK port is following: > > rte_dev_probe() <-- attaches device to dpdk bus. > rte_eth_dev_configure() > rte_eth_dev_start() > rte_eth_dev_stop() > rte_eth_dev_close() <-- closes the eth device. > rte_dev_remove() <-- detaches device from dpdk bus. > Thanks for the explanation. Much clear to me now. > af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all > the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach > device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove() > that tries to free already freed resources and fails. > I think that rte_pmd_af_xdp_remove() should just ignore this case > reporting successful removal, otherwise device will never be detached > from vdev bus reporting failure of driver->remove() callback. > > Will you prepare the patch for this? Yes, I will test it and send out patch to dpdk-dev. Regards, William > If you wish, I could prepare it myself on Monday. > > Best regards, Ilya Maximets. > > > > > And the reason rte_eth_dev_allocated find no device is that > > at _rte_eth_dev_allocated > > the rte_eth_devices[i].data is an empty string. > > > > And the reason is that at rte_eth_dev_close > > it calls > > if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) { > > /* new behaviour: send event + reset state + free all data */ > > rte_eth_dev_release_port(dev); > > > > the rte_eth_dev_release_port clear the dev->data->name > > memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data)); > > > >> > >> Another possible issue is that you're using driver base name (net_af_xdp) > >> as a name of DPDK device. Maybe this causes some issues. > >> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0". > > > > Yes, I use the above comment. > > > > Regards, > > William > > > >> > >> Best regards, Ilya Maximets. > >> > >>>>> > >>>>> Another option you may try is: > >>>>> > >>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ > >>>>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" > >>>>> > >>>> It actually works! When adding the device: > >>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device > >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK > >>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: > >>>> 0, core id: 11 created. > >>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd > >>>> threads on numa node 0 > >>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already stopped > >>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is > >>>> not supported on port 0 > >>>> > >>>> However , when removing the device, some errors show up. > >>>> ovs-vsctl del-port br0 afxdp-p0 > >>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted > >>>> interface afxdp-p0 on port 1 > >>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: > >>>> 0, core id: 11 destroyed. > >>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the > >>>> device (net_af_xdp) > >>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device > >>>> on primary process > >>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device > >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached > >>>> > >>>> I also measured the performance, it works fine and shows around 7Mpps. > >>>> > >>>>> so DPDK will not have to guess the bus type. > >>>>> > >>>>>> > >>>>>> $ ovs-vsctl show > >>>>>> Port "afxdp-p0" > >>>>>> Interface "afxdp-p0" > >>>>>> type: dpdk > >>>>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} > >>>>>> > >>>>>> The performance of physical device loopback shows around 7Mpps with 64B pkt. > >>>>>> > >>>>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ > >>>>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" > >>>>>> > >>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. > >>>>> > >>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there. > >>>>> In general, patches that intended to work only with latest DPDK that > >>>>> not yet supported on master should be made on top of 'dpdk-latest' > >>>>> branch and have [PATCH dpdk-latest] subject prefix. > >>>> > >>>> Got it, thank you. > >>>> William > >>>> > >>>>> > >>>>> Best regards, Ilya Maximets. > >>> > >>> > > > >
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 47153dc60148..99697bc896ca 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -380,6 +380,8 @@ struct netdev_dpdk { /* If true, device was attached by rte_eth_dev_attach(). */ bool attached; + /* If true, device was hop plugged by rte_eal_hotplug_add(). */ + bool hotplugged; /* If true, rte_eth_dev_start() was successfully called */ bool started; struct eth_addr hwaddr; @@ -1376,6 +1378,14 @@ netdev_dpdk_destruct(struct netdev *netdev) rte_eth_dev_stop(dev->port_id); dev->started = false; + if (dev->hotplugged) { + int ret; + ret = rte_eal_hotplug_remove("vdev", "net_af_xdp"); + if (ret) { + VLOG_ERR("%s unplugged error %d", __func__, ret); + } + } + if (dev->attached) { /* Retrieve eth device data before closing it. * FIXME: avoid direct access to DPDK internal array rte_eth_devices. @@ -1648,6 +1658,26 @@ netdev_dpdk_lookup_by_port_id(dpdk_port_t port_id) } static dpdk_port_t +netdev_dpdk_get_af_xdp_port(const char *vdev_args) +{ + const char vdev_name[] = "net_af_xdp"; + dpdk_port_t port_id; + + if (rte_eal_hotplug_add("vdev", vdev_name, vdev_args) < 0) { + VLOG_WARN("Cannot hotplug AF_XDP device: %s\n", vdev_args); + return DPDK_ETH_PORT_ID_INVALID; + } + + if (rte_eth_dev_get_port_by_name(vdev_name, &port_id) != 0) { + rte_eal_hotplug_remove("vdev", vdev_name); + VLOG_WARN("Cannot get AF_XDP port_id, device: %s\n", vdev_args); + return DPDK_ETH_PORT_ID_INVALID; + } + + return port_id; +} + +static dpdk_port_t netdev_dpdk_get_port_by_mac(const char *mac_str) { dpdk_port_t port_id; @@ -1705,7 +1735,11 @@ netdev_dpdk_process_devargs(struct netdev_dpdk *dev, { dpdk_port_t new_port_id; - if (strncmp(devargs, "class=eth,mac=", 14) == 0) { + if (strncmp(devargs, "net_af_xdp", 10) == 0) { + new_port_id = netdev_dpdk_get_af_xdp_port(&devargs[11]); + dev->hotplugged = true; + VLOG_INFO("Device '%s' hotplugged to DPDK", devargs); + } else if (strncmp(devargs, "class=eth,mac=", 14) == 0) { new_port_id = netdev_dpdk_get_port_by_mac(&devargs[14]); } else { new_port_id = netdev_dpdk_get_port_by_devargs(devargs); @@ -2031,7 +2065,7 @@ netdev_dpdk_policer_pkt_handle(struct rte_meter_srtcm *meter, uint32_t pkt_len = rte_pktmbuf_pkt_len(pkt) - sizeof(struct ether_hdr); return rte_meter_srtcm_color_blind_check(meter, profile, time, pkt_len) == - e_RTE_METER_GREEN; + 0; /* e_RTE_METER_GREEN */ } static int
The patch adds support for using DPDK's AF_XDP PMD. Base on DPDK 19.08-rc0: commit c6a53a6bcd0ae52afe0a6d58b059af577c9408b3 Author: Thomas Monjalon <thomas@monjalon.net> Date: Tue May 14 18:04:37 2019 +0200 version: 19.08-rc0 An AF_XDP PMD can be created by: $ ovs-vsctl add-port br0 afxdp-p0 -- set int afxdp-p0 type=dpdk \ options:dpdk-devargs=net_af_xdp,iface=enp2s0,queue=0 $ ovs-vsctl show Port "afxdp-p0" Interface "afxdp-p0" type: dpdk options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} The performance of physical device loopback shows around 7Mpps with 64B pkt. $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. Signed-off-by: William Tu <u9012063@gmail.com> --- lib/netdev-dpdk.c | 38 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-)