From patchwork Tue May 25 19:27:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wilder X-Patchwork-Id: 1483739 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=R3uxXpH4; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FqPHX67Xyz9sCD for ; Wed, 26 May 2021 05:27:52 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 23719405D6; Tue, 25 May 2021 19:27:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MxGOaut-M6dJ; Tue, 25 May 2021 19:27:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTP id 2C9F4405B1; Tue, 25 May 2021 19:27:48 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F3CFDC0019; Tue, 25 May 2021 19:27:47 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 67954C000E for ; Tue, 25 May 2021 19:27:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 4ABD3401D3 for ; Tue, 25 May 2021 19:27:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=ibm.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gm2oB0zNDIrK for ; Tue, 25 May 2021 19:27:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by smtp2.osuosl.org (Postfix) with ESMTPS id 690E8401B4 for ; Tue, 25 May 2021 19:27:45 +0000 (UTC) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 14PJ4WO5060524; Tue, 25 May 2021 15:27:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : mime-version; s=pp1; bh=yoE7RV+ecbvPv9f31FyVeOebX4AaROJRn7/M6QtEUwY=; b=R3uxXpH4trqROlQM6hkuOf9Fa4RPif4/k+eLbFy0+nwV8DgXEwyTfdAcEn0o8ey5CvBw EJ+hIgtYBZWfhrG5jzESjtt8QMMs0uMdO+5igl81xHAmu1zU9UTmZEi85McMwScZ6Pk7 uTWKqwgrlOmFULRRSvRf1SC7th19xsKjWrecfIRhVdu++6rlYYD42xe2vhZAi38mM5Qb 1AiPDXMRxHDDMPn9KJzChMS/84biQ6+NO4ApuIZwfyV+Km1zo122LgW6b1W4CfzAKVq2 IbHpGpKWbsxi8vEiPuOGIIcwgSFRYtNZ2HgE126jFJHMwx9Ey3T/qO0EY80JK+jSgO7Z MA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 38s65kt5aq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 15:27:44 -0400 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 14PJ4r0T062793; Tue, 25 May 2021 15:27:44 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 38s65kt5aa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 15:27:44 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 14PJDaPd024651; Tue, 25 May 2021 19:27:43 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma03dal.us.ibm.com with ESMTP id 38s1v9u8yd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 19:27:43 +0000 Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 14PJRgJb25952656 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 25 May 2021 19:27:42 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 595EDAE05F; Tue, 25 May 2021 19:27:42 +0000 (GMT) Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 41BDFAE062; Tue, 25 May 2021 19:27:40 +0000 (GMT) Received: from oc8377887825.ibm.com (unknown [9.160.106.8]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 25 May 2021 19:27:40 +0000 (GMT) From: David Wilder To: ovs-dev@openvswitch.org Date: Tue, 25 May 2021 12:27:23 -0700 Message-Id: <20210525192724.20021-2-dwilder@us.ibm.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20210525192724.20021-1-dwilder@us.ibm.com> References: <20210525192724.20021-1-dwilder@us.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: -pLo_kxp8-6czywV6ZpuEn6jw47dlvFt X-Proofpoint-GUID: 8fg9mwu2hX24vsvEpqjQTbH8SKe9JZAD X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-05-25_08:2021-05-25, 2021-05-25 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 bulkscore=0 priorityscore=1501 spamscore=0 suspectscore=0 lowpriorityscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2105250117 Cc: daniele.di.proietto@gmail.com, wilder@us.ibm.com, i.maximets@ovn.org Subject: [ovs-dev] [PATCH v5 1/2] ovs-numa: Support non-contiguous numa nodes and offline CPU cores X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This change removes the assumption that numa nodes and cores are numbered contiguously in linux. This change is required to support some Power systems. An additional check has been added to verify that cores are online, offline cores result in non-contiguously numbered cores. I manually verified that all the exported functions defined in ovs-numa.h function correctly with these changes on Power and amd64 systems. Tests for dpif-netdev and pmd that define numa systems using the --dummy-numa option have been modified to use both contiguous and non-contiguous numbered nodes. Signed-off-by: David Wilder Reviewed-by: David Christensen Tested-by: Emma Finn --- lib/ovs-numa.c | 50 ++++++++++++++++++++++++++++++++++---------------- tests/dpif-netdev.at | 12 ++++++------ tests/pmd.at | 23 ++++++++++++++++++++--- 3 files changed, 60 insertions(+), 25 deletions(-) diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 6d0a685..8c7d25b 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -42,16 +42,18 @@ VLOG_DEFINE_THIS_MODULE(ovs_numa); * This module stores the affinity information of numa nodes and cpu cores. * It also provides functions to bookkeep the pin of threads on cpu cores. * - * It is assumed that the numa node ids and cpu core ids all start from 0 and - * range continuously. So, for example, if 'ovs_numa_get_n_cores()' returns N, - * user can assume core ids from 0 to N-1 are all valid and there is a - * 'struct cpu_core' for each id. + * It is assumed that the numa node ids and cpu core ids all start from 0. + * There is no guarantee that node and cpu ids are numbered consecutively + * (this is a change from earlier version of the code). So, for example, + * if two nodes exist with ids 0 and 8, 'ovs_numa_get_n_nodes()' will + * return 2, no assumption of node numbering should be made. * * NOTE, this module should only be used by the main thread. * - * NOTE, the assumption above will fail when cpu hotplug is used. In that - * case ovs-numa will not function correctly. For now, add a TODO entry - * for addressing it in the future. + * NOTE, if cpu hotplug is used 'all_numa_nodes' and 'all_cpu_cores' must be + * invalidated when ever the system topology changes. Support for detecting + * topology changes has not been included. For now, add a TODO entry for + * addressing it in the future. * * TODO: Fix ovs-numa when cpu hotplug is used. */ @@ -130,15 +132,14 @@ insert_new_cpu_core(struct numa_node *n, unsigned core_id) * - "0,0,0,0": four cores on numa socket 0. * - "0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1": 16 cores on two numa sockets. * - "0,0,0,0,1,1,1,1": 8 cores on two numa sockets. - * - * The different numa ids must be consecutives or the function will abort. */ + * - "0,0,0,0,8,8,8,8": 8 cores on two numa sockets, non-contiguous. + */ static void discover_numa_and_core_dummy(void) { char *conf = xstrdup(dummy_config); char *id, *saveptr = NULL; unsigned i = 0; - long max_numa_id = 0; for (id = strtok_r(conf, ",", &saveptr); id; id = strtok_r(NULL, ",", &saveptr)) { @@ -152,8 +153,6 @@ discover_numa_and_core_dummy(void) continue; } - max_numa_id = MAX(max_numa_id, numa_id); - hnode = hmap_first_with_hash(&all_numa_nodes, hash_int(numa_id, 0)); if (hnode) { @@ -169,10 +168,27 @@ discover_numa_and_core_dummy(void) free(conf); - if (max_numa_id + 1 != hmap_count(&all_numa_nodes)) { - ovs_fatal(0, "dummy numa contains non consecutive numa ids"); +} + +#ifdef __linux__ +/* Check if a cpu is detected and online */ +static int +cpu_detected(unsigned int core_id) +{ + char path[PATH_MAX]; + int len = snprintf(path, sizeof(path), + "/sys/devices/system/cpu/cpu%d/topology/core_id", + core_id); + if (len <= 0 || (unsigned) len >= sizeof(path)) { + return 0; + } + if (access(path, F_OK) != 0) { + return 0; } + + return 1; } +#endif /* __linux__ */ /* Discovers all numa nodes and the corresponding cpu cores. * Constructs the 'struct numa_node' and 'struct cpu_core'. */ @@ -219,7 +235,9 @@ discover_numa_and_core(void) unsigned core_id; core_id = strtoul(subdir->d_name + 3, NULL, 10); - insert_new_cpu_core(n, core_id); + if (cpu_detected(core_id)) { + insert_new_cpu_core(n, core_id); + } } } closedir(dir); @@ -229,7 +247,7 @@ discover_numa_and_core(void) } free(path); - if (!dir || !numa_supported) { + if (!numa_supported) { break; } } diff --git a/tests/dpif-netdev.at b/tests/dpif-netdev.at index 57cae38..8d88626 100644 --- a/tests/dpif-netdev.at +++ b/tests/dpif-netdev.at @@ -98,7 +98,7 @@ m4_define([DPIF_NETDEV_DUMMY_IFACE], fail-mode=secure -- \ add-port br1 p2 -- set interface p2 type=$1 options:stream=unix:$OVS_RUNDIR/p0.sock ofport_request=2 -- \ add-port br1 p8 -- set interface p8 ofport_request=8 type=$1 --], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 action=normal]) @@ -130,7 +130,7 @@ m4_define([DPIF_NETDEV_MISS_FLOW_INSTALL], -- set interface p1 type=$1 options:pstream=punix:$OVS_RUNDIR/p0.sock \ -- set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 action=normal]) @@ -172,7 +172,7 @@ m4_define([DPIF_NETDEV_FLOW_PUT_MODIFY], add-port br0 p2 -- set interface p2 type=$1 ofport_request=2 options:pstream=punix:$OVS_RUNDIR/p2.sock -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg]) # Add a flow that directs some packets received on p1 to p2 and the @@ -226,7 +226,7 @@ m4_define([DPIF_NETDEV_MISS_FLOW_DUMP], -- set interface p1 type=$1 options:pstream=punix:$OVS_RUNDIR/p0.sock \ -- set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl upcall/disable-ufid], [0], [Datapath dumping tersely using UFID disabled ], []) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) @@ -384,7 +384,7 @@ m4_define([DPIF_NETDEV_FLOW_HW_OFFLOAD], set interface p1 type=$1 ofport_request=1 options:pstream=punix:$OVS_RUNDIR/p1.sock options:ifindex=1100 -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg netdev_dummy:file:dbg]) AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:hw-offload=true]) @@ -447,7 +447,7 @@ m4_define([DPIF_NETDEV_FLOW_HW_OFFLOAD_OFFSETS], set interface p1 type=$1 ofport_request=1 options:pcap=p1.pcap options:ifindex=1101 -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg netdev_dummy:file:dbg]) AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:hw-offload=true]) diff --git a/tests/pmd.at b/tests/pmd.at index cc5371d..3e6d6bb 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -584,7 +584,7 @@ AT_CLEANUP AT_SETUP([PMD - rxq affinity - NUMA]) OVS_VSWITCHD_START( - [], [], [], [--dummy-numa 0,0,0,1,1]) + [], [], [], [--dummy-numa 0,0,0,1,1,8,8]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 actions=controller]) @@ -601,21 +601,38 @@ p1 1 0 2 AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:3,1:4"]) -dnl We moved the queues to different numa node. Expecting threads on +dnl We moved the queues to different contiguous numa node. Expecting threads on dnl NUMA node 1 to be created. AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl p1 0 1 3 p1 1 1 4 ]) +AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:5,1:6"]) + +dnl We moved the queues to different non-contiguous numa node. Expecting threads on +dnl NUMA node 8 to be created. +AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl +p1 0 8 5 +p1 1 8 6 +]) + AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:3,1:1"]) -dnl Queues splitted between NUMA nodes. +dnl Queues splitted between contiguous NUMA nodes. AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl p1 0 1 3 p1 1 0 1 ]) +AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:5,1:1"]) + +dnl Queues splitted between non-contiguous NUMA nodes. +AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl +p1 0 8 5 +p1 1 0 1 +]) + AT_CHECK([ovs-vsctl remove Interface p1 other_config pmd-rxq-affinity]) dnl We removed the rxq-affinity request. dpif-netdev should assign queues From patchwork Tue May 25 19:27:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wilder X-Patchwork-Id: 1483740 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=MG5NG9Po; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FqPHb1CPvz9sCD for ; Wed, 26 May 2021 05:27:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 9EEC683D41; Tue, 25 May 2021 19:27:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XBHvSEw4eMuB; Tue, 25 May 2021 19:27:51 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTP id DD73883D32; Tue, 25 May 2021 19:27:49 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CC416C0019; Tue, 25 May 2021 19:27:49 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id C4310C0023 for ; Tue, 25 May 2021 19:27:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 8C2C383D26 for ; Tue, 25 May 2021 19:27:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Gpd8PyLj_t8 for ; Tue, 25 May 2021 19:27:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by smtp1.osuosl.org (Postfix) with ESMTPS id DD42983CF6 for ; Tue, 25 May 2021 19:27:47 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 14PJBfnX022136; Tue, 25 May 2021 15:27:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=R7smvfFGLBdQyH0oTxdOysM/bUD52QI/e3fdr7QiJJ8=; b=MG5NG9PotuDEVpkoff8oRgirasQghEMQBk1EDZlY06EKxmN5MLA6dUPu/Gof/Vh0yi8B 7QeabmeKbNzHy0rzhfBEXLRoOlKfg9IfE9X6Nb7Dnlb02DgRgu1lpxX8ii0rpBPsO2lm m6Mm0TKRHLxtd5jEX8va3QKoqEidDdK11Q/rWIdPwz5drKirZXqaynZ7kJ0hYO9DWLXO /5KYBE2GJnYUFuhW62n88P9pe9R66Ofs5soDe+TYEI8O4fq5KKp3Cn1OSAZaOxVyBpSc +VkzkMd0HB9P2gPASADJVO5Z/8YhrYTDv4M2v/+Jh5s1yWLu9RInQHQFV7prknm0bNiH Xg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 38s4jk4x8b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 15:27:47 -0400 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 14PJD5En028408; Tue, 25 May 2021 15:27:46 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 38s4jk4x7r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 15:27:46 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 14PJC02o030159; Tue, 25 May 2021 19:27:45 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03wdc.us.ibm.com with ESMTP id 38s1qd2qny-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 May 2021 19:27:45 +0000 Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 14PJRjeD18416050 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 25 May 2021 19:27:45 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0BECEAE05C; Tue, 25 May 2021 19:27:45 +0000 (GMT) Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A6C17AE060; Tue, 25 May 2021 19:27:42 +0000 (GMT) Received: from oc8377887825.ibm.com (unknown [9.160.106.8]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 25 May 2021 19:27:42 +0000 (GMT) From: David Wilder To: ovs-dev@openvswitch.org Date: Tue, 25 May 2021 12:27:24 -0700 Message-Id: <20210525192724.20021-3-dwilder@us.ibm.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20210525192724.20021-1-dwilder@us.ibm.com> References: <20210525192724.20021-1-dwilder@us.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 1nVpYD1-mVWdUGOwNAad0cV0HZLVDfHu X-Proofpoint-ORIG-GUID: NEVDQtkG6jgTtI0zUOWdAtv0tDqSmIId X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-05-25_08:2021-05-25, 2021-05-25 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 suspectscore=0 adultscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 spamscore=0 phishscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2105250117 Cc: daniele.di.proietto@gmail.com, wilder@us.ibm.com, i.maximets@ovn.org Subject: [ovs-dev] [PATCH v5 2/2] ovs-numa: Dpdk options with non-contiguous nodes X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If not supplied by the user --socket-mem and --socket-limit EAL options are set to a default value based system topology. The assumption that numa nodes must be numbered contiguously is removed by this change. These options can be seen in the ovs-vswitchd.log. For example: a system containing only numa nodes 0 and 8 will generate the following: EAL ARGS: ovs-vswitchd --socket-mem 1024,0,0,0,0,0,0,0,1024 \ --socket-limit 1024,0,0,0,0,0,0,0,1024 -l 0 Signed-off-by: David Wilder Reviewed-by: David Christensen Tested-by: Emma Finn --- lib/dpdk.c | 33 ++++++++++++++++++++++++--------- lib/ovs-numa.c | 1 - lib/ovs-numa.h | 2 ++ 3 files changed, 26 insertions(+), 10 deletions(-) diff --git a/lib/dpdk.c b/lib/dpdk.c index 3195403..6e4a2e3 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -133,18 +133,33 @@ static char * construct_dpdk_socket_mem(void) { const char *def_value = "1024"; - int numa, numa_nodes = ovs_numa_get_n_numas(); - struct ds dpdk_socket_mem = DS_EMPTY_INITIALIZER; + struct ovs_numa_dump *dump; + int last_node = 0; + const struct ovs_numa_info_numa *node; - if (numa_nodes == 0 || numa_nodes == OVS_NUMA_UNSPEC) { - numa_nodes = 1; - } + /* Build a list of all numa nodes with at least one core */ + dump = ovs_numa_dump_n_cores_per_numa(1); + struct ds dpdk_socket_mem = DS_EMPTY_INITIALIZER; - ds_put_cstr(&dpdk_socket_mem, def_value); - for (numa = 1; numa < numa_nodes; ++numa) { - ds_put_format(&dpdk_socket_mem, ",%s", def_value); + FOR_EACH_NUMA_ON_DUMP(node, dump) { + while (node->numa_id > last_node && + node->numa_id != OVS_NUMA_UNSPEC && + node->numa_id <= MAX_NUMA_NODES){ + if (last_node == 0) { + ds_put_format(&dpdk_socket_mem, "%s", "0"); + } else { + ds_put_format(&dpdk_socket_mem, ",%s", "0"); + } + last_node++; + } + if (node->numa_id == 0) { + ds_put_format(&dpdk_socket_mem, "%s", def_value); + } else { + ds_put_format(&dpdk_socket_mem, ",%s", def_value); + } + last_node++; } - + ovs_numa_dump_destroy(dump); return ds_cstr(&dpdk_socket_mem); } diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 8c7d25b..b825ecb 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -58,7 +58,6 @@ VLOG_DEFINE_THIS_MODULE(ovs_numa); * TODO: Fix ovs-numa when cpu hotplug is used. */ -#define MAX_NUMA_NODES 128 /* numa node. */ struct numa_node { diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index 8f2ea34..ecc251a 100644 --- a/lib/ovs-numa.h +++ b/lib/ovs-numa.h @@ -26,6 +26,8 @@ #define OVS_CORE_UNSPEC INT_MAX #define OVS_NUMA_UNSPEC INT_MAX +#define MAX_NUMA_NODES 128 + /* Dump of a list of 'struct ovs_numa_info'. */ struct ovs_numa_dump { struct hmap cores;