From patchwork Thu Feb 6 20:38:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wilder X-Patchwork-Id: 1234570 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=us.ibm.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48D9JB5vP5z9sRK for ; Fri, 7 Feb 2020 07:38:50 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 4474F86AFF; Thu, 6 Feb 2020 20:38:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y590UDVisILV; Thu, 6 Feb 2020 20:38:48 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by whitealder.osuosl.org (Postfix) with ESMTP id E147D86A1E; Thu, 6 Feb 2020 20:38:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CAFF6C0881; Thu, 6 Feb 2020 20:38:39 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8BB41C013E for ; Thu, 6 Feb 2020 20:38:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8591522053 for ; Thu, 6 Feb 2020 20:38:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VgVMzI9cSIUF for ; Thu, 6 Feb 2020 20:38:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by silver.osuosl.org (Postfix) with ESMTPS id 5CCFF22011 for ; Thu, 6 Feb 2020 20:38:36 +0000 (UTC) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 016KcKN9072758; Thu, 6 Feb 2020 15:38:35 -0500 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 2y0m793m1m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Feb 2020 15:38:35 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 016KcYo8073321; Thu, 6 Feb 2020 15:38:34 -0500 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 2y0m793m1b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Feb 2020 15:38:34 -0500 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 016KZ4ai007882; Thu, 6 Feb 2020 20:38:33 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma03dal.us.ibm.com with ESMTP id 2xykc9uvtr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Feb 2020 20:38:33 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 016KcW3A54723060 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Feb 2020 20:38:32 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2BC54AC05B; Thu, 6 Feb 2020 20:38:32 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A4070AC05E; Thu, 6 Feb 2020 20:38:31 +0000 (GMT) Received: from oc8377887825.ibm.com (unknown [9.70.82.120]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Thu, 6 Feb 2020 20:38:31 +0000 (GMT) From: David Wilder To: ovs-dev@openvswitch.org Date: Thu, 6 Feb 2020 12:38:17 -0800 Message-Id: <20200206203817.21680-3-dwilder@us.ibm.com> X-Mailer: git-send-email 2.23.0.162.gf1d4a28 In-Reply-To: <20200206203817.21680-1-dwilder@us.ibm.com> References: <20200206203817.21680-1-dwilder@us.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-02-06_04:2020-02-06, 2020-02-06 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 spamscore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 phishscore=0 mlxscore=0 clxscore=1015 suspectscore=1 mlxlogscore=999 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002060150 Cc: daniele.di.proietto@gmail.com, wilder@us.ibm.com, i.maximets@ovn.org, drc@ibm.com Subject: [ovs-dev] [PATCH v2 2/2] ovs-numa: Dpdk options with non-consecutive nodes. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If not supplied by the user --socket-mem and --socket-limit EAL options are set to a default value based system topology. The assumption that numa nodes must be numbered consecutive is removed by this change. These options can be seen in the ovs-vswitchd.log. For example: a system containing only numa nodes 0 and 8 will generate the following: EAL ARGS: ovs-vswitchd --socket-mem 1024,0,0,0,0,0,0,0,1024 \ --socket-limit 1024,0,0,0,0,0,0,0,1024 -l 0 Changes for v2: 0-day Robot suggested changes. Signed-off-by: David Wilder --- lib/dpdk.c | 26 ++++++++++++++++++-------- lib/ovs-numa.c | 1 - lib/ovs-numa.h | 2 ++ 3 files changed, 20 insertions(+), 9 deletions(-) diff --git a/lib/dpdk.c b/lib/dpdk.c index 37ea2973c..5af6bb49b 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -134,18 +134,28 @@ static char * construct_dpdk_socket_mem(void) { const char *def_value = "1024"; - int numa, numa_nodes = ovs_numa_get_n_numas(); + struct ovs_numa_dump *dump; + int last_node = 0; + + /* Build a list of all numa nodes with at least one core */ + dump = ovs_numa_dump_n_cores_per_numa(1); struct ds dpdk_socket_mem = DS_EMPTY_INITIALIZER; + ds_put_cstr(&dpdk_socket_mem, def_value); - if (numa_nodes == 0 || numa_nodes == OVS_NUMA_UNSPEC) { - numa_nodes = 1; - } + const struct ovs_numa_info_numa *node; - ds_put_cstr(&dpdk_socket_mem, def_value); - for (numa = 1; numa < numa_nodes; ++numa) { - ds_put_format(&dpdk_socket_mem, ",%s", def_value); + FOR_EACH_NUMA_ON_DUMP(node, dump) { + while (node->numa_id > last_node+1 && + node->numa_id != OVS_NUMA_UNSPEC && + node->numa_id <= MAX_NUMA_NODES){ + ds_put_format(&dpdk_socket_mem, ",%s", "0"); + ++last_node; + } + if (node->numa_id != 0) { + ds_put_format(&dpdk_socket_mem, ",%s", def_value); + } } - + ovs_numa_dump_destroy(dump); return ds_cstr(&dpdk_socket_mem); } diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 5153ed27e..8a76e6b5e 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -58,7 +58,6 @@ VLOG_DEFINE_THIS_MODULE(ovs_numa); * TODO: Fix ovs-numa when cpu hotplug is used. */ -#define MAX_NUMA_NODES 128 /* numa node. */ struct numa_node { diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index 8f2ea3430..ecc251a7f 100644 --- a/lib/ovs-numa.h +++ b/lib/ovs-numa.h @@ -26,6 +26,8 @@ #define OVS_CORE_UNSPEC INT_MAX #define OVS_NUMA_UNSPEC INT_MAX +#define MAX_NUMA_NODES 128 + /* Dump of a list of 'struct ovs_numa_info'. */ struct ovs_numa_dump { struct hmap cores;