From patchwork Wed Feb 5 21:06:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wilder X-Patchwork-Id: 1234013 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=us.ibm.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48CYzc5THYz9sRK for ; Thu, 6 Feb 2020 08:07:24 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7632C85D54; Wed, 5 Feb 2020 21:07:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FDMrIj8WHiOw; Wed, 5 Feb 2020 21:07:21 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 23CF785EF0; Wed, 5 Feb 2020 21:07:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 016A4C1D85; Wed, 5 Feb 2020 21:07:21 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id C5403C0174 for ; Wed, 5 Feb 2020 21:07:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B34E585EC6 for ; Wed, 5 Feb 2020 21:07:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0Eg9ktmKKFiy for ; Wed, 5 Feb 2020 21:07:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 6083F85E83 for ; Wed, 5 Feb 2020 21:07:16 +0000 (UTC) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 015L6TVR131509; Wed, 5 Feb 2020 16:07:15 -0500 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xyhmmqa9k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 16:07:15 -0500 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 015L6vAo132877; Wed, 5 Feb 2020 16:07:14 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xyhmmqa99-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 16:07:14 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 015L4wuW004827; Wed, 5 Feb 2020 21:07:13 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma01dal.us.ibm.com with ESMTP id 2xykc91kjf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 21:07:13 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 015L791710224064 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 5 Feb 2020 21:07:10 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D0A046E059; Wed, 5 Feb 2020 21:07:09 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6C75C6E04E; Wed, 5 Feb 2020 21:07:08 +0000 (GMT) Received: from oc8377887825.ibm.com (unknown [9.80.224.136]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTP; Wed, 5 Feb 2020 21:07:08 +0000 (GMT) From: David Wilder To: ovs-dev@openvswitch.org Date: Wed, 5 Feb 2020 13:06:44 -0800 Message-Id: <20200205210645.11256-2-dwilder@us.ibm.com> X-Mailer: git-send-email 2.23.0.162.gf1d4a28 In-Reply-To: <20200205210645.11256-1-dwilder@us.ibm.com> References: <20200205210645.11256-1-dwilder@us.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-02-05_06:2020-02-04, 2020-02-05 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 impostorscore=0 mlxscore=0 phishscore=0 malwarescore=0 priorityscore=1501 mlxlogscore=999 suspectscore=4 bulkscore=0 adultscore=0 spamscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002050161 Cc: daniele.di.proietto@gmail.com, wilder@us.ibm.com, i.maximets@ovn.org, drc@ibm.com Subject: [ovs-dev] [PATCH v1 1/2] ovs-numa: Non-consecutively numa nodes and cores X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This change removes the assumption that numa nodes and cores are numbered consecutively in linux. This change is required to support some Power systems. An additional check has been added to verify that cores are online, offline cores result in non-consecutively numbered cores. I manually verified that all the exported functions defined in ovs-numa.h function correctly with these changes on Power and amd64 systems. Tests for dpif-netdev and pmd that define numa systems using the --dummy-numa option have been modified to use non-consecutive numbered nodes. Signed-off-by: David Wilder --- lib/ovs-numa.c | 40 +++++++++++++++++++++++++++------------- tests/dpif-netdev.at | 12 ++++++------ tests/pmd.at | 20 ++++++++++---------- 3 files changed, 43 insertions(+), 29 deletions(-) diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 6d0a68522..da4754094 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -42,16 +42,18 @@ VLOG_DEFINE_THIS_MODULE(ovs_numa); * This module stores the affinity information of numa nodes and cpu cores. * It also provides functions to bookkeep the pin of threads on cpu cores. * - * It is assumed that the numa node ids and cpu core ids all start from 0 and - * range continuously. So, for example, if 'ovs_numa_get_n_cores()' returns N, - * user can assume core ids from 0 to N-1 are all valid and there is a - * 'struct cpu_core' for each id. + * It is assumed that the numa node ids and cpu core ids all start from 0. + * There is no guarantee that node and cpu ids are numbered consecutively + * (this is a change from earlier version of the code). So, for example, + * if two nodes exist with ids 0 and 8, 'ovs_numa_get_n_nodes()' will + * return 2, no assumption of node numbering should be made. * * NOTE, this module should only be used by the main thread. * - * NOTE, the assumption above will fail when cpu hotplug is used. In that - * case ovs-numa will not function correctly. For now, add a TODO entry - * for addressing it in the future. + * NOTE, if cpu hotplug is used 'all_numa_nodes' and 'all_cpu_cores' must be + * invalidated when ever the system topology changes. Support for detecting + * topology changes has not been included. For now, add a TODO entry for + * addressing it in the future. * * TODO: Fix ovs-numa when cpu hotplug is used. */ @@ -169,11 +171,23 @@ discover_numa_and_core_dummy(void) free(conf); - if (max_numa_id + 1 != hmap_count(&all_numa_nodes)) { - ovs_fatal(0, "dummy numa contains non consecutive numa ids"); - } } +#ifdef __linux__ +/* Check if a cpu is detected and online */ +static int +cpu_detected(unsigned int core_id) +{ + char *path; + + path = xasprintf("/sys/devices/system/cpu/cpu%d/topology/core_id", core_id); + if (access(path, F_OK) != 0) + return 0; + + return 1; +} +#endif /* __linux__ */ + /* Discovers all numa nodes and the corresponding cpu cores. * Constructs the 'struct numa_node' and 'struct cpu_core'. */ static void @@ -219,7 +233,8 @@ discover_numa_and_core(void) unsigned core_id; core_id = strtoul(subdir->d_name + 3, NULL, 10); - insert_new_cpu_core(n, core_id); + if (cpu_detected(core_id)) + insert_new_cpu_core(n, core_id); } } closedir(dir); @@ -229,9 +244,8 @@ discover_numa_and_core(void) } free(path); - if (!dir || !numa_supported) { + if (!numa_supported) break; - } } #endif /* __linux__ */ } diff --git a/tests/dpif-netdev.at b/tests/dpif-netdev.at index 0aeb4e788..3a1a58276 100644 --- a/tests/dpif-netdev.at +++ b/tests/dpif-netdev.at @@ -94,7 +94,7 @@ m4_define([DPIF_NETDEV_DUMMY_IFACE], fail-mode=secure -- \ add-port br1 p2 -- set interface p2 type=$1 options:stream=unix:$OVS_RUNDIR/p0.sock ofport_request=2 -- \ add-port br1 p8 -- set interface p8 ofport_request=8 type=$1 --], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 action=normal]) @@ -126,7 +126,7 @@ m4_define([DPIF_NETDEV_MISS_FLOW_INSTALL], -- set interface p1 type=$1 options:pstream=punix:$OVS_RUNDIR/p0.sock \ -- set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 action=normal]) @@ -168,7 +168,7 @@ m4_define([DPIF_NETDEV_FLOW_PUT_MODIFY], add-port br0 p2 -- set interface p2 type=$1 ofport_request=2 options:pstream=punix:$OVS_RUNDIR/p2.sock -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg]) # Add a flow that directs some packets received on p1 to p2 and the @@ -222,7 +222,7 @@ m4_define([DPIF_NETDEV_MISS_FLOW_DUMP], -- set interface p1 type=$1 options:pstream=punix:$OVS_RUNDIR/p0.sock \ -- set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl upcall/disable-ufid], [0], [Datapath dumping tersely using UFID disabled ], []) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) @@ -374,7 +374,7 @@ m4_define([DPIF_NETDEV_FLOW_HW_OFFLOAD], set interface p1 type=$1 ofport_request=1 options:pstream=punix:$OVS_RUNDIR/p1.sock options:ifindex=1 -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg netdev_dummy:file:dbg]) AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:hw-offload=true]) @@ -437,7 +437,7 @@ m4_define([DPIF_NETDEV_FLOW_HW_OFFLOAD_OFFSETS], set interface p1 type=$1 ofport_request=1 options:pcap=p1.pcap options:ifindex=1 -- \ set bridge br0 datapath-type=dummy \ other-config:datapath-id=1234 fail-mode=secure], [], [], - [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,1,1,1,1"], [])]) + [m4_if([$1], [dummy-pmd], [--dummy-numa="0,0,0,0,8,8,8,8"], [])]) AT_CHECK([ovs-appctl vlog/set dpif:file:dbg dpif_netdev:file:dbg netdev_dummy:file:dbg]) AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:hw-offload=true]) diff --git a/tests/pmd.at b/tests/pmd.at index 5b612f88f..94e7a0a11 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -362,7 +362,7 @@ OVS_VSWITCHD_START( [add-port br0 p1 -- set Interface p1 type=dummy-pmd ofport_request=1 options:n_rxq=2 -- \ add-port br0 p2 -- set Interface p2 type=dummy-pmd ofport_request=2 options:n_rxq=2 -- \ set Open_vSwitch . other_config:pmd-cpu-mask=3 -], [], [], [--dummy-numa 0,1]) +], [], [], [--dummy-numa 0,8]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 action=controller]) @@ -399,13 +399,13 @@ NXT_PACKET_IN2 (xid=0x0): cookie=0x0 total_len=106 in_port=2 (via action) data_l icmp,vlan_tci=0x0000,dl_src=50:54:00:00:00:09,dl_dst=50:54:00:00:00:0a,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0 icmp_csum:13fc ]) -AT_CHECK([ovs-vsctl set Interface p2 options:numa_id=1]) +AT_CHECK([ovs-vsctl set Interface p2 options:numa_id=8]) AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl p1 0 0 0 p1 1 0 0 -p2 0 1 1 -p2 1 1 1 +p2 0 8 1 +p2 1 8 1 ]) AT_CHECK([ovs-ofctl monitor br0 65534 invalid_ttl --detach --no-chdir --pidfile 2> ofctl_monitor.log]) @@ -584,7 +584,7 @@ AT_CLEANUP AT_SETUP([PMD - rxq affinity - NUMA]) OVS_VSWITCHD_START( - [], [], [], [--dummy-numa 0,0,0,1,1]) + [], [], [], [--dummy-numa 0,0,0,8,8]) AT_CHECK([ovs-appctl vlog/set dpif:dbg dpif_netdev:dbg]) AT_CHECK([ovs-ofctl add-flow br0 actions=controller]) @@ -604,15 +604,15 @@ AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:3,1:4"]) dnl We moved the queues to different numa node. Expecting threads on dnl NUMA node 1 to be created. AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl -p1 0 1 3 -p1 1 1 4 +p1 0 8 3 +p1 1 8 4 ]) AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity="0:3,1:1"]) dnl Queues splitted between NUMA nodes. AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show], [0], [dnl -p1 0 1 3 +p1 0 8 3 p1 1 0 1 ]) @@ -628,10 +628,10 @@ p1 1 AT_CHECK([ovs-vsctl set Interface p1 other_config:pmd-rxq-affinity='0:3']) -dnl We explicitly requesting NUMA node 1 for queue 0. +dnl We explicitly requesting NUMA node 8 for queue 0. dnl Queue 1 should be polled by thread from NUMA node 0. AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | parse_pmd_rxq_show | cut -f 1,2,3 -d ' '], [0], [dnl -p1 0 1 +p1 0 8 p1 1 0 ]) From patchwork Wed Feb 5 21:06:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wilder X-Patchwork-Id: 1234012 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=us.ibm.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48CYzc2B5Zz9sPK for ; Thu, 6 Feb 2020 08:07:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0D1272152F; Wed, 5 Feb 2020 21:07:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kqOqKxu5p2bo; Wed, 5 Feb 2020 21:07:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 3CDE720335; Wed, 5 Feb 2020 21:07:20 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 320D8C1D83; Wed, 5 Feb 2020 21:07:20 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 91AD5C0175 for ; Wed, 5 Feb 2020 21:07:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 79022207A6 for ; Wed, 5 Feb 2020 21:07:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nmfy8m3f57V1 for ; Wed, 5 Feb 2020 21:07:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by silver.osuosl.org (Postfix) with ESMTPS id A3D422079A for ; Wed, 5 Feb 2020 21:07:17 +0000 (UTC) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 015L6SUC131401; Wed, 5 Feb 2020 16:07:16 -0500 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xyhmmqaae-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 16:07:16 -0500 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 015L6sq0132712; Wed, 5 Feb 2020 16:07:16 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xyhmmqa9r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 16:07:16 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 015L4vgx004824; Wed, 5 Feb 2020 21:07:14 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma01dal.us.ibm.com with ESMTP id 2xykc91kjs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 21:07:14 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 015L7BnC8454432 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 5 Feb 2020 21:07:11 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B64AC6E054; Wed, 5 Feb 2020 21:07:11 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 580CC6E053; Wed, 5 Feb 2020 21:07:10 +0000 (GMT) Received: from oc8377887825.ibm.com (unknown [9.80.224.136]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTP; Wed, 5 Feb 2020 21:07:10 +0000 (GMT) From: David Wilder To: ovs-dev@openvswitch.org Date: Wed, 5 Feb 2020 13:06:45 -0800 Message-Id: <20200205210645.11256-3-dwilder@us.ibm.com> X-Mailer: git-send-email 2.23.0.162.gf1d4a28 In-Reply-To: <20200205210645.11256-1-dwilder@us.ibm.com> References: <20200205210645.11256-1-dwilder@us.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-02-05_06:2020-02-04, 2020-02-05 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 impostorscore=0 mlxscore=0 phishscore=0 malwarescore=0 priorityscore=1501 mlxlogscore=999 suspectscore=1 bulkscore=0 adultscore=0 spamscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002050161 Cc: daniele.di.proietto@gmail.com, wilder@us.ibm.com, i.maximets@ovn.org, drc@ibm.com Subject: [ovs-dev] [PATCH v1 2/2] ovs-numa: Dpdk options with non-consecutive nodes. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If not supplied by the user --socket-mem and --socket-limit EAL options are set to a default value based system topology. The assumption that numa nodes must be numbered consecutive is removed by this change. These options can be seen in the ovs-vswitchd.log. For example: a system containing only numa nodes 0 and 8 will generate the following: EAL ARGS: ovs-vswitchd --socket-mem 1024,0,0,0,0,0,0,0,1024 \ --socket-limit 1024,0,0,0,0,0,0,0,1024 -l 0 Signed-off-by: David Wilder --- lib/dpdk.c | 25 +++++++++++++++++-------- lib/ovs-numa.c | 1 - lib/ovs-numa.h | 2 ++ 3 files changed, 19 insertions(+), 9 deletions(-) diff --git a/lib/dpdk.c b/lib/dpdk.c index 37ea2973c..954643018 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -134,18 +134,27 @@ static char * construct_dpdk_socket_mem(void) { const char *def_value = "1024"; - int numa, numa_nodes = ovs_numa_get_n_numas(); + struct ovs_numa_dump *dump; + int last_node = 0; + + /* Build a list of all numa nodes with at least one core */ + dump = ovs_numa_dump_n_cores_per_numa(1); struct ds dpdk_socket_mem = DS_EMPTY_INITIALIZER; + ds_put_cstr(&dpdk_socket_mem, def_value); - if (numa_nodes == 0 || numa_nodes == OVS_NUMA_UNSPEC) { - numa_nodes = 1; - } + const struct ovs_numa_info_numa *node; - ds_put_cstr(&dpdk_socket_mem, def_value); - for (numa = 1; numa < numa_nodes; ++numa) { - ds_put_format(&dpdk_socket_mem, ",%s", def_value); + FOR_EACH_NUMA_ON_DUMP(node, dump) { + while (node->numa_id > last_node+1 && + node->numa_id != OVS_NUMA_UNSPEC && + node->numa_id <= MAX_NUMA_NODES){ + ds_put_format(&dpdk_socket_mem, ",%s", "0"); + ++last_node; + } + if (node->numa_id != 0) + ds_put_format(&dpdk_socket_mem, ",%s", def_value); } - + ovs_numa_dump_destroy(dump); return ds_cstr(&dpdk_socket_mem); } diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index da4754094..5b00739fa 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -58,7 +58,6 @@ VLOG_DEFINE_THIS_MODULE(ovs_numa); * TODO: Fix ovs-numa when cpu hotplug is used. */ -#define MAX_NUMA_NODES 128 /* numa node. */ struct numa_node { diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index 8f2ea3430..ecc251a7f 100644 --- a/lib/ovs-numa.h +++ b/lib/ovs-numa.h @@ -26,6 +26,8 @@ #define OVS_CORE_UNSPEC INT_MAX #define OVS_NUMA_UNSPEC INT_MAX +#define MAX_NUMA_NODES 128 + /* Dump of a list of 'struct ovs_numa_info'. */ struct ovs_numa_dump { struct hmap cores;