From patchwork Wed Oct 23 21:08:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1182484 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="u0fd2I+0"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46z30X2jB5z9sNw for ; Thu, 24 Oct 2019 08:09:32 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 42560CD9; Wed, 23 Oct 2019 21:09:30 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 498DCC91 for ; Wed, 23 Oct 2019 21:09:29 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id E3D358A0 for ; Wed, 23 Oct 2019 21:09:25 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id u20so10715050plq.4 for ; Wed, 23 Oct 2019 14:09:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=TxdmHjBG5FbTPClRbJ6VGm0lErwi4hoZenc57GVYs+c=; b=u0fd2I+0PsAgj0deZCymDn82XExZStuRSMgCH2xfUgdG95UEtJhXPBPfJp0sdSMXXQ 2NkwVQ3yt/p5D5cBD465oPUshbxtuQxxjMKvz3sYr/01CnMCS8j3ft+940t0IFKYuJZX i6LZ/OIOHqSE2cBFTpcKmJAYPl8wG7Kx4Px3Apb4/NebGnE3EcWOGmad7WYjxE2TtdZl AwQfG/7t7DZiFq5VUV1MQx45OLaoFMV7psYSfgFYHjUx9/AsIVasSNztga7PIkvSTQTk hljH2i/JWeJv53qVCQ2ILF4OiR4Krgra26ybCYg+UmUu3pCYvSOugqIn+R8ctz2lfDwd ZVgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=TxdmHjBG5FbTPClRbJ6VGm0lErwi4hoZenc57GVYs+c=; b=qqsZdwgwqwdraFYhtk+SfNGf0vwX/o4ZsCQGcbnIMgLqFjU2jZ3yM9vjqPfFcLMfzH /tnqpDBrEsX+7Fby9j8QgUE8z4e5r0CSZpoRmYITRUfDkYz4z62qFt4tRe/Fz6vraCfg mlNH9668GPZfG3KBc7S+C+DZa9M/hrtniwkn4XzZGcJan0sJRS3ETvw6EKk71K3BMy2x rYs5WGHox3FB8YIXi55XBjsSVWR9yj9EBZ0C2XkRTqcQ5Jzc70jGTfSeacgVVho88y9R nCTsfvgyfmx+LDzLjYTwe2KGL8sGfeDSo4rfokEs0fMekJ49rVQh5N5WN56GwdWhGtzu GKiQ== X-Gm-Message-State: APjAAAWjaOdX35MIQv8A8NUUqaTPNyFbW1NPK4aJiEBZ9WFEXesy9zvE d7JZPiQ03kfzB6fOjMHT/8cdcnU9Ipk= X-Google-Smtp-Source: APXvYqynl6xzb3YHd+Cxb5vIxFPjbxaw8L4JhZmarfFnt65LeA5Yc6iaCkiW8v0wBCVv8MB5ZwERVw== X-Received: by 2002:a17:902:d915:: with SMTP id c21mr11907316plz.264.1571864965166; Wed, 23 Oct 2019 14:09:25 -0700 (PDT) Received: from sc9-mailhost2.vmware.com ([66.170.99.95]) by smtp.gmail.com with ESMTPSA id m68sm24217246pfb.122.2019.10.23.14.09.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 23 Oct 2019 14:09:24 -0700 (PDT) From: William Tu To: dev@openvswitch.org Date: Wed, 23 Oct 2019 14:08:34 -0700 Message-Id: <1571864914-64829-1-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCHv5] netdev-linux: Detect numa node id. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org The patch detects the numa node id from the name of the netdev, by reading the '/sys/class/net//device/numa_node'. If not available, ex: virtual device, or any error happens, return numa id 0. Currently only the afxdp netdev type uses it, other linux netdev types are disabled due to no use case. Signed-off-by: William Tu Acked-by: Eelco Chaudron --- v5: Feedbacks from Ilya - reafactor the error handling - add mutex lock - Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/601947245 v4: Feedbacks from Eelco - Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/599308893 v3: Feedbacks from Ilya and Eelco - update doc, afxdp.rst - fix coding style - fix limit of numa node max, by using ovs_numa_numa_id_is_valid - move the function to netdev-linux - cache the result of numa_id - add security check for netdev name - use fscanf instead of read and convert to int - revise some error message content v2: address feedback from Eelco fix memory leak of xaspintf log using INFO instead of WARN --- Documentation/intro/install/afxdp.rst | 1 - lib/netdev-afxdp.c | 11 ------ lib/netdev-linux-private.h | 2 ++ lib/netdev-linux.c | 63 ++++++++++++++++++++++++++++++++++- 4 files changed, 64 insertions(+), 13 deletions(-) diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index 820e9d993d8f..6c00c4ad1356 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -309,7 +309,6 @@ Below is a script using namespaces and veth peer:: Limitations/Known Issues ------------------------ -#. Device's numa ID is always 0, need a way to find numa id from a netdev. #. No QoS support because AF_XDP netdev by-pass the Linux TC layer. A possible work-around is to use OpenFlow meter action. #. Most of the tests are done using i40e single port. Multiple ports and diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index 8eb270c150e8..cfd93fab9f45 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -543,17 +543,6 @@ out: return err; } -int -netdev_afxdp_get_numa_id(const struct netdev *netdev) -{ - /* FIXME: Get netdev's PCIe device ID, then find - * its NUMA node id. - */ - VLOG_INFO("FIXME: Device %s always use numa id 0.", - netdev_get_name(netdev)); - return 0; -} - static void xsk_remove_xdp_program(uint32_t ifindex, int xdpmode) { diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index a350be151147..c8f2be47b10b 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -96,6 +96,8 @@ struct netdev_linux { /* LAG information. */ bool is_lag_master; /* True if the netdev is a LAG master. */ + int numa_id; /* NUMA node id. */ + #ifdef HAVE_AF_XDP /* AF_XDP information. */ struct xsk_socket_info **xsks; diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index f4819237370a..4dd05493b238 100644 --- a/lib/netdev-linux.c +++ b/lib/netdev-linux.c @@ -236,6 +236,7 @@ enum { VALID_VPORT_STAT_ERROR = 1 << 5, VALID_DRVINFO = 1 << 6, VALID_FEATURES = 1 << 7, + VALID_NUMA_ID = 1 << 8, }; struct linux_lag_slave { @@ -1391,6 +1392,66 @@ netdev_linux_tap_batch_send(struct netdev *netdev_, return 0; } +static int +netdev_linux_get_numa_id__(const struct netdev *netdev_) +{ + struct netdev_linux *netdev = netdev_linux_cast(netdev_); + char *numa_node_path; + const char *name; + int node_id; + FILE *stream; + + if (netdev->cache_valid & VALID_NUMA_ID) { + return netdev->numa_id; + } + + netdev->numa_id = 0; + netdev->cache_valid |= VALID_NUMA_ID; + + name = netdev_get_name(netdev_); + if (strpbrk(name, "/\\")) { + VLOG_ERR_RL(&rl, "\"%s\" is not a valid name for a port. " + "A valid name must not include '/' or '\\'." + "Using numa_id 0", name); + return 0; + } + + numa_node_path = xasprintf("/sys/class/net/%s/device/numa_node", name); + + stream = fopen(numa_node_path, "r"); + if (!stream) { + /* Virtual device does not have this info. */ + VLOG_INFO_RL(&rl, "%s: Can't open '%s': %s, using numa_id 0", + name, numa_node_path, ovs_strerror(errno)); + free(numa_node_path); + return 0; + } + + if (fscanf(stream, "%d", &node_id) != 1 + || !ovs_numa_numa_id_is_valid(node_id)) { + VLOG_WARN_RL(&rl, "%s: Can't detect NUMA node, using numa_id 0", name); + node_id = 0; + } + + netdev->numa_id = node_id; + fclose(stream); + free(numa_node_path); + return node_id; +} + +static int OVS_UNUSED +netdev_linux_get_numa_id(const struct netdev *netdev_) +{ + struct netdev_linux *netdev = netdev_linux_cast(netdev_); + int numa_id; + + ovs_mutex_lock(&netdev->mutex); + numa_id = netdev_linux_get_numa_id__(netdev_); + ovs_mutex_unlock(&netdev->mutex); + + return numa_id; +} + /* Sends 'batch' on 'netdev'. Returns 0 if successful, otherwise a positive * errno value. Returns EAGAIN without blocking if the packet cannot be queued * immediately. Returns EMSGSIZE if a partial packet was transmitted or if @@ -3298,7 +3359,7 @@ const struct netdev_class netdev_afxdp_class = { .set_config = netdev_afxdp_set_config, .get_config = netdev_afxdp_get_config, .reconfigure = netdev_afxdp_reconfigure, - .get_numa_id = netdev_afxdp_get_numa_id, + .get_numa_id = netdev_linux_get_numa_id, .send = netdev_afxdp_batch_send, .rxq_construct = netdev_afxdp_rxq_construct, .rxq_destruct = netdev_afxdp_rxq_destruct,