diff mbox series

[ovs-dev,v6] system-dpdk: Test with mlx5 devices.

Message ID 20240110100436.601350-1-david.marchand@redhat.com
State Accepted
Delegated to: Kevin Traynor
Headers show
Series [ovs-dev,v6] system-dpdk: Test with mlx5 devices. | expand

Checks

Context Check Description
ovsrobot/apply-robot success apply and check: success
ovsrobot/github-robot-_Build_and_Test success github build: passed
ovsrobot/intel-ovs-compilation success test: success

Commit Message

David Marchand Jan. 10, 2024, 10:04 a.m. UTC 
The DPDK unit test only runs if vfio or igb_uio kernel modules are loaded:
on systems with only mlx5, this test is always skipped.

Besides, the test tries to grab the first device listed by dpdk-devbind.py,
regardless of the PCI device status regarding kmod binding.

Remove dependency on this DPDK script and use a minimal script that
reads PCI sysfs.

This script is not perfect, as one can imagine PCI devices bound to
vfio-pci for virtual machines.
Plus, this script only tries to take over vfio-pci devices. mlx5 devices
can't be taken over blindly as it could mean losing connectivity to the
machine if the netdev was in use for this system.

For those two reasons, add a new environment variable DPDK_PCI_ADDR for
testers to select the PCI device of their liking.
For consistency and grep, the temporary file PCI_ADDR is renamed
to DPDK_PCI_ADDR.

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since v5:
- rebased,
- moved the script to the python scripts list in automake.mk,
- bumped copyright date,

Changes since v4:
- separated from the original series,
- rebased,
- dropped mlx5 devices from the discovery script,
- documented DPDK_PCI_ADDR env variable,

Changes since v3:
- fixed nit from Maxime,

Changes since v2:
- sorted logs alphabetically,

---
 Documentation/topics/testing.rst | 11 ++++++---
 tests/automake.mk                |  1 +
 tests/system-dpdk-find-device.py | 39 ++++++++++++++++++++++++++++++++
 tests/system-dpdk-macros.at      | 10 ++------
 tests/system-dpdk.at             | 14 ++++++------
 5 files changed, 57 insertions(+), 18 deletions(-)
 create mode 100755 tests/system-dpdk-find-device.py

Comments

Kevin Traynor Jan. 10, 2024, 1:19 p.m. UTC | #1 
On 10/01/2024 10:04, David Marchand wrote:
> The DPDK unit test only runs if vfio or igb_uio kernel modules are loaded:
> on systems with only mlx5, this test is always skipped.
> 
> Besides, the test tries to grab the first device listed by dpdk-devbind.py,
> regardless of the PCI device status regarding kmod binding.
> 
> Remove dependency on this DPDK script and use a minimal script that
> reads PCI sysfs.
> 
> This script is not perfect, as one can imagine PCI devices bound to
> vfio-pci for virtual machines.
> Plus, this script only tries to take over vfio-pci devices. mlx5 devices
> can't be taken over blindly as it could mean losing connectivity to the
> machine if the netdev was in use for this system.
> 
> For those two reasons, add a new environment variable DPDK_PCI_ADDR for
> testers to select the PCI device of their liking.
> For consistency and grep, the temporary file PCI_ADDR is renamed
> to DPDK_PCI_ADDR.
> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Acked-by: Eelco Chaudron <echaudro@redhat.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Applied. Thanks David, Maxime, Eelco and Ilya.
diff mbox series

Patch

diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
index 5f6940b84d..fb9b3e77b1 100644
--- a/Documentation/topics/testing.rst
+++ b/Documentation/topics/testing.rst
@@ -343,15 +343,20 @@  To see a list of all the available tests, run::
 
 These tests support a `DPDK supported NIC`_. The tests operate on a wider set of
 environments, for instance, when a virtual port is used.
-They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
 Moreover you need to have root privileges to load the required modules and to bind
-the NIC to the DPDK-compatible driver.
+a PCI device to the DPDK-compatible driver.
 
 .. _DPDK supported NIC: https://core.dpdk.org/supported/#nics
 
+The phy test will skip if no suitable PCI device is found.
+It is possible to select which PCI device is used for this test by setting the
+DPDK_PCI_ADDR environment variable, which is especially useful when testing
+with a mlx5 device::
+
+    # DPDK_PCI_ADDR=0000:82:00.0 make check-dpdk
+
 All tests are skipped if no hugepages are configured. User must look into the DPDK
 manual to figure out how to `Configure hugepages`_.
-The phy test will skip if no compatible physical device is available.
 
 .. _Configure hugepages: https://doc.dpdk.org/guides-22.11/linux_gsg/sys_reqs.html
 
diff --git a/tests/automake.mk b/tests/automake.mk
index 2ae0aeecaf..10c9fbb01f 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -520,6 +520,7 @@  CHECK_PYFILES = \
 	tests/flowgen.py \
 	tests/genpkts.py \
 	tests/ovsdb-monitor-sort.py \
+	tests/system-dpdk-find-device.py \
 	tests/test-daemon.py \
 	tests/test-dpparse.py \
 	tests/test-json.py \
diff --git a/tests/system-dpdk-find-device.py b/tests/system-dpdk-find-device.py
new file mode 100755
index 0000000000..ced74e7f31
--- /dev/null
+++ b/tests/system-dpdk-find-device.py
@@ -0,0 +1,39 @@ 
+#!/usr/bin/env python3
+# Copyright (c) 2024 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from pathlib import Path
+import os
+import sys
+
+# The tester might want to select a PCI device, if so, trust it.
+if 'DPDK_PCI_ADDR' in os.environ:
+    print(os.environ['DPDK_PCI_ADDR'])
+    sys.exit(0)
+
+for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
+    class_path = device / 'class'
+    # Only consider Network class devices
+    if class_path.read_text().strip() != '0x020000':
+        continue
+    kmod_path = device / 'driver' / 'module'
+    kmod_name = kmod_path.resolve().name
+    # Only care about devices bound to vfio_pci or igb_uio.
+    if kmod_name not in ['vfio_pci', 'igb_uio']:
+        continue
+    print(device.resolve().name)
+    sys.exit(0)
+
+sys.exit(1)
diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
index dcdfa55741..3b5a3512d4 100644
--- a/tests/system-dpdk-macros.at
+++ b/tests/system-dpdk-macros.at
@@ -19,14 +19,8 @@  m4_define([OVS_DPDK_PRE_PHY_SKIP],
   [dnl Perform the precheck
    OVS_DPDK_PRE_CHECK()
 
-   dnl Check if VFIO or UIO driver is loaded
-   AT_SKIP_IF([ ! (lsmod | grep -E "igb_uio|vfio") ], [], [stdout])
-
-   dnl Find PCI address candidate, skip if there is no DPDK-compatible NIC
-   AT_CHECK([$DPDK_DIR/usertools/dpdk-devbind.py -s | head -n +4 | tail -1], [], [stdout])
-   AT_CHECK([cat stdout | cut -d" " -s -f1 > PCI_ADDR])
-   AT_SKIP_IF([ ! test -s PCI_ADDR ])
-
+   dnl Check if a device is available for DPDK
+   AT_SKIP_IF([ ! $abs_top_srcdir/tests/system-dpdk-find-device.py > DPDK_PCI_ADDR ])
 ])
 
 
diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index fab3dcbeaf..1c97bf7772 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -63,7 +63,7 @@  OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 sleep 2
 
@@ -240,7 +240,7 @@  OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and add policer
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set interface phy0 ingress_policing_rate=10000 ingress_policing_burst=1000])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 sleep 2
@@ -380,7 +380,7 @@  OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and add egress policer
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 OVS_WAIT_UNTIL([ovs-vsctl set port phy0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cir=1250000 other-config:cbs=2048])
 AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show phy0], [], [stdout])
 sleep 2
@@ -509,7 +509,7 @@  dnl First set MTU to its default value and confirm that value, then increase the
 
 dnl Add userspace bridge and attach it to OVS with default MTU value
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
 dnl Check default MTU value in the datapath
@@ -546,7 +546,7 @@  dnl First set an increased MTU value and confirm that value, then decrease the M
 
 dnl Add userspace bridge and attach it to OVS and modify MTU value
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=9000])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
@@ -665,7 +665,7 @@  OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and set MTU value to max upper bound
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=9702])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
@@ -703,7 +703,7 @@  OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and set MTU value to min lower bound
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=68])
 AT_CHECK([ovs-vsctl show], [], [stdout])