From patchwork Tue Dec 17 22:27:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi-Hung Wei X-Patchwork-Id: 1211789 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="u+p4u2Dc"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47ct7y2KFjz9sR4 for ; Wed, 18 Dec 2019 09:28:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8B63886274; Tue, 17 Dec 2019 22:28:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wyBfU6Nr-x3k; Tue, 17 Dec 2019 22:28:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 84F958462E; Tue, 17 Dec 2019 22:28:11 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6EB70C1AE8; Tue, 17 Dec 2019 22:28:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 39641C077D for ; Tue, 17 Dec 2019 22:28:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 23B3F8458C for ; Tue, 17 Dec 2019 22:28:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8bJ8H_XjiK79 for ; Tue, 17 Dec 2019 22:28:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 4CBE68462E for ; Tue, 17 Dec 2019 22:28:09 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id q8so21311pfh.7 for ; Tue, 17 Dec 2019 14:28:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=mJGjNuWlh74RR74Q4/4IMs+xqOHyOqIxwNBsxyK3zC8=; b=u+p4u2DcdEzedUFa+2KrERcvIKr5s5E6Cb0ipJ8AxYvvuHaDgwHRO+/AtXN79/EJPV tj/yRGbiqPYKBwb3TNBca28VHHf0lBJAAoWqDDNBCNhHiYtDFlOL2mdUjkP+yXEqIPAn qIFLVV1g/SRZbmKvLAnoAr0puZbS+FBDz0XJAY+PPS/WzWjh2HUtXVXLbj1fpZXrt8M+ nhTU7VCTkLDzMV65qf191KUOnrwyVIuq2XRZzwRbbg8QSgyEk8ZUdZPBbgwd8/9L1sqE nh08goVHGDBvAQT0wb4bSzgTpBKG9uJUORurHTu/+RfeuT3q207ja5t+RxvSym7U2jIx 6mjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=mJGjNuWlh74RR74Q4/4IMs+xqOHyOqIxwNBsxyK3zC8=; b=V63NfA3FB4FLl1MQDtCoFxf59E8FMY3HtcX821gCggiiwVQlJFl+iLCrzSqvHfox5E zMwnCnZ/ih3orDagmHS4Nxh2P1a3i0oIhG+LH7AQOfM7ZdjdBUnFm4jlu2PUwyF/UmBr mbYchPz4I9I+JwNIdp/IR+Mqb94AzYyxGXiv6tk5FjEWsNMT3Cez3hW5+/cGLW1UJ+kj JfPBbWeKiOTKWim49mllTUJfdyJofSbL6BNGEdqb8iE56OExSBgDsuLfKp1aa+yBUP9s L4TFAIX4ZYqzmu3WVsOQehv3fhfdjNT5MoiO4yDuXHOo6wQGTrJ1XB0TBqrTm6jd6hWp f3Ow== X-Gm-Message-State: APjAAAVl7/4tcebXBAaZHUtXIHEl+ZAL6o9QMwlO/gbtcJTy7r9d/FrI 0fYh44e7K/2jvgv11qiMfUuqidsn X-Google-Smtp-Source: APXvYqxeQ+imTMINHy9YDeBxXJtmVZEvz5FhrMNr8sY0nWCmVuxeaWEySqORN17qVGxu9VW0/tsYrQ== X-Received: by 2002:a63:1204:: with SMTP id h4mr10356245pgl.288.1576621688065; Tue, 17 Dec 2019 14:28:08 -0800 (PST) Received: from bionic.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e2sm19442pfh.84.2019.12.17.14.28.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2019 14:28:07 -0800 (PST) From: Yi-Hung Wei To: dev@openvswitch.org, i.maximets@ovn.org, echaudro@redhat.com, u9012063@gmail.com Date: Tue, 17 Dec 2019 14:27:53 -0800 Message-Id: <20191217222754.86810-1-yihung.wei@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [ovs-dev] [PATCH v7 1/2] netdev-linux: Detect numa node id. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: William Tu The patch detects the numa node id from the name of the netdev, by reading the '/sys/class/net//device/numa_node'. If not available, ex: virtual device, or any error happens, return numa id 0. Currently only the afxdp netdev type uses it, other linux netdev types are disabled due to no use case. Signed-off-by: William Tu Acked-by: Eelco Chaudron --- v7: - Add numa aware memory allocation for afxdp related memory in the following patch. - Travis CI: https://travis-ci.org/YiHungWei/ovs/builds/626403984 v6: Feedbacks from Ilya - add thread safety check at netdev_linux_get_numa_id__, and pass netdev_linux - preserve numa cache on netlink updates - Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/605634673 v5: Feedbacks from Ilya - reafactor the error handling - add mutex lock - Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/601947245 v4: Feedbacks from Eelco - Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/599308893 v3: Feedbacks from Ilya and Eelco - update doc, afxdp.rst - fix coding style - fix limit of numa node max, by using ovs_numa_numa_id_is_valid - move the function to netdev-linux - cache the result of numa_id - add security check for netdev name - use fscanf instead of read and convert to int - revise some error message content v2: address feedback from Eelco fix memory leak of xaspintf log using INFO instead of WARN --- Documentation/intro/install/afxdp.rst | 1 - lib/netdev-afxdp.c | 11 ----- lib/netdev-afxdp.h | 1 - lib/netdev-linux-private.h | 2 + lib/netdev-linux.c | 67 +++++++++++++++++++++++++-- 5 files changed, 66 insertions(+), 16 deletions(-) diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index 15e3c918f942..7b0736c96114 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -327,7 +327,6 @@ Below is a script using namespaces and veth peer:: Limitations/Known Issues ------------------------ -#. Device's numa ID is always 0, need a way to find numa id from a netdev. #. No QoS support because AF_XDP netdev by-pass the Linux TC layer. A possible work-around is to use OpenFlow meter action. #. Most of the tests are done using i40e single port. Multiple ports and diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index ca2dfd005b9f..4c1f9c68270a 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -695,17 +695,6 @@ out: return err; } -int -netdev_afxdp_get_numa_id(const struct netdev *netdev) -{ - /* FIXME: Get netdev's PCIe device ID, then find - * its NUMA node id. - */ - VLOG_INFO("FIXME: Device %s always use numa id 0.", - netdev_get_name(netdev)); - return 0; -} - static void xsk_remove_xdp_program(uint32_t ifindex, enum afxdp_mode mode) { diff --git a/lib/netdev-afxdp.h b/lib/netdev-afxdp.h index 8188bc669526..246a4b62fb57 100644 --- a/lib/netdev-afxdp.h +++ b/lib/netdev-afxdp.h @@ -60,7 +60,6 @@ int netdev_afxdp_batch_send(struct netdev *netdev_, int qid, int netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, char **errp); int netdev_afxdp_get_config(const struct netdev *netdev, struct smap *args); -int netdev_afxdp_get_numa_id(const struct netdev *netdev); int netdev_afxdp_get_stats(const struct netdev *netdev_, struct netdev_stats *stats); int netdev_afxdp_get_custom_stats(const struct netdev *netdev, diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index 8873caa9d412..4d66b0858222 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -96,6 +96,8 @@ struct netdev_linux { /* LAG information. */ bool is_lag_master; /* True if the netdev is a LAG master. */ + int numa_id; /* NUMA node id. */ + #ifdef HAVE_AF_XDP /* AF_XDP information. */ struct xsk_socket_info **xsks; diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index f8e59bacfb13..e32df970fe4b 100644 --- a/lib/netdev-linux.c +++ b/lib/netdev-linux.c @@ -236,6 +236,7 @@ enum { VALID_VPORT_STAT_ERROR = 1 << 5, VALID_DRVINFO = 1 << 6, VALID_FEATURES = 1 << 7, + VALID_NUMA_ID = 1 << 8, }; struct linux_lag_slave { @@ -820,9 +821,9 @@ netdev_linux_update__(struct netdev_linux *dev, { if (rtnetlink_type_is_rtnlgrp_link(change->nlmsg_type)) { if (change->nlmsg_type == RTM_NEWLINK) { - /* Keep drv-info, and ip addresses. */ + /* Keep drv-info, ip addresses, and NUMA id. */ netdev_linux_changed(dev, change->ifi_flags, - VALID_DRVINFO | VALID_IN); + VALID_DRVINFO | VALID_IN | VALID_NUMA_ID); /* Update netdev from rtnl-change msg. */ if (change->mtu) { @@ -1391,6 +1392,66 @@ netdev_linux_tap_batch_send(struct netdev *netdev_, return 0; } +static int +netdev_linux_get_numa_id__(struct netdev_linux *netdev) + OVS_REQUIRES(netdev->mutex) +{ + char *numa_node_path; + const char *name; + int node_id; + FILE *stream; + + if (netdev->cache_valid & VALID_NUMA_ID) { + return netdev->numa_id; + } + + netdev->numa_id = 0; + netdev->cache_valid |= VALID_NUMA_ID; + + name = netdev_get_name(&netdev->up); + if (strpbrk(name, "/\\")) { + VLOG_ERR_RL(&rl, "\"%s\" is not a valid name for a port. " + "A valid name must not include '/' or '\\'." + "Using numa_id 0", name); + return 0; + } + + numa_node_path = xasprintf("/sys/class/net/%s/device/numa_node", name); + + stream = fopen(numa_node_path, "r"); + if (!stream) { + /* Virtual device does not have this info. */ + VLOG_INFO_RL(&rl, "%s: Can't open '%s': %s, using numa_id 0", + name, numa_node_path, ovs_strerror(errno)); + free(numa_node_path); + return 0; + } + + if (fscanf(stream, "%d", &node_id) != 1 + || !ovs_numa_numa_id_is_valid(node_id)) { + VLOG_WARN_RL(&rl, "%s: Can't detect NUMA node, using numa_id 0", name); + node_id = 0; + } + + netdev->numa_id = node_id; + fclose(stream); + free(numa_node_path); + return node_id; +} + +static int OVS_UNUSED +netdev_linux_get_numa_id(const struct netdev *netdev_) +{ + struct netdev_linux *netdev = netdev_linux_cast(netdev_); + int numa_id; + + ovs_mutex_lock(&netdev->mutex); + numa_id = netdev_linux_get_numa_id__(netdev); + ovs_mutex_unlock(&netdev->mutex); + + return numa_id; +} + /* Sends 'batch' on 'netdev'. Returns 0 if successful, otherwise a positive * errno value. Returns EAGAIN without blocking if the packet cannot be queued * immediately. Returns EMSGSIZE if a partial packet was transmitted or if @@ -3308,7 +3369,7 @@ const struct netdev_class netdev_afxdp_class = { .set_config = netdev_afxdp_set_config, .get_config = netdev_afxdp_get_config, .reconfigure = netdev_afxdp_reconfigure, - .get_numa_id = netdev_afxdp_get_numa_id, + .get_numa_id = netdev_linux_get_numa_id, .send = netdev_afxdp_batch_send, .rxq_construct = netdev_afxdp_rxq_construct, .rxq_destruct = netdev_afxdp_rxq_destruct, From patchwork Tue Dec 17 22:27:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi-Hung Wei X-Patchwork-Id: 1211790 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ZUOG1FVo"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47ct830rzBz9sR4 for ; Wed, 18 Dec 2019 09:28:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5A37F84AF4; Tue, 17 Dec 2019 22:28:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id irjAe-D33gNM; Tue, 17 Dec 2019 22:28:14 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8A051862FC; Tue, 17 Dec 2019 22:28:13 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8278CC1AE8; Tue, 17 Dec 2019 22:28:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 677A1C1AE8 for ; Tue, 17 Dec 2019 22:28:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 40ED285B0D for ; Tue, 17 Dec 2019 22:28:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6wMmFnGbI3AT for ; Tue, 17 Dec 2019 22:28:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E1E6E8458C for ; Tue, 17 Dec 2019 22:28:10 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id y8so4715pll.13 for ; Tue, 17 Dec 2019 14:28:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=X8WNyHRZblK0BrmHiKr9rVr/k1saOcEIUz90oUnyP8g=; b=ZUOG1FVo28xNUEA4JqlFD0oVt2jZw2CLXNhcFm+t4gP3bMxX4hH5cYP/ZhkoykbCAl iz5vSYFypj+bNZ1nOP6k3ypZ068AiZhozyrwDyWdf47qv195Ah+dU2QqR92p7uk5K77G zL2AH59Xzb3AqXdknaiKxJvGvx50KTEqYYdK9Up5ekszkXtemFyogLuRzFYHYurNqJvC z5qE0Drrzg+VkpkC3ZbDx+hhS7/6JuCabArKJUu/RwbQIUq1yM3d+35padt7uNTbsSwS N9PhKxcDe7l6WSi9AuKgBC2rT1vem0E4/t6S9rNXY6qTtQhmcFGVA58J1GOBv18qVEGR W+Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=X8WNyHRZblK0BrmHiKr9rVr/k1saOcEIUz90oUnyP8g=; b=N7bv6rKXvas3aNidzBeHKeeKaIPzbERw7k+JUtm1EVy4GEDk+svxa4poSp47HRHvgD HEGVCvf2oVKmwh31eSON9oj8vHICUrYmnQMj52UAguskZzBY0yW1JItyvHGu8Q0l3Ump kAqCKrwM8XuLdy/WbU0kd0lBm0GJViYTkyZQnGLbti1KVWY9dr4y2/tKh1e+OQ7kJ8R9 u9hXZTO5E7Wcjtw7TWfm9fY5JSt3n24/L9IxK19Pn7VYT/Ank8QZXo/5jUjGvsEPr69u ENwXvZXqxOQ1VpNSvJNyCgVXwY0q9cPAfIGkLOA0eu8RQzaWqSJzdF+Nh6nUOVRAayIS y5Rw== X-Gm-Message-State: APjAAAWZ4JQvHaMWi1lQWBfKl/bB8uoS3PyOKKav3odFlFgzWLpEEti8 H1oNBZkvezWm8DY0j0LZI1a5Ab2B X-Google-Smtp-Source: APXvYqyqy/SY+wewGUZw2RhWtdMpF/EoHDSQR41NzWzUL3k2axc/ztnpfPUbYPzs2P6DcMrYSdClKg== X-Received: by 2002:a17:90a:28a1:: with SMTP id f30mr9245813pjd.77.1576621689943; Tue, 17 Dec 2019 14:28:09 -0800 (PST) Received: from bionic.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e2sm19442pfh.84.2019.12.17.14.28.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2019 14:28:08 -0800 (PST) From: Yi-Hung Wei To: dev@openvswitch.org, i.maximets@ovn.org, echaudro@redhat.com, u9012063@gmail.com Date: Tue, 17 Dec 2019 14:27:54 -0800 Message-Id: <20191217222754.86810-2-yihung.wei@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191217222754.86810-1-yihung.wei@gmail.com> References: <20191217222754.86810-1-yihung.wei@gmail.com> Subject: [ovs-dev] [PATCH v7 2/2] netdev-afxdp: NUMA-aware memory allocation for XSK related memory X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Currently, the AF_XDP socket (XSK) related memory are allocated by main thread in the main thread's NUMA domain. With the patch that detects netdev-linux's NUMA node id, the PMD thread of AF_XDP port will be run on the AF_XDP netdev's NUMA domain. If the net device's NUMA domain is different from the main thread's NUMA domain, we will have two cross-NUMA memory accesses (netdev <-> memory, memory <-> CPU). This patch addresses the aforementioned issue by allocating the memory in the net device's NUMA domain. Signed-off-by: Yi-Hung Wei --- Documentation/intro/install/afxdp.rst | 2 +- acinclude.m4 | 5 ++++- include/sparse/automake.mk | 1 + include/sparse/numa.h | 27 +++++++++++++++++++++++++++ lib/netdev-afxdp.c | 21 ++++++++++++++++++--- 5 files changed, 51 insertions(+), 5 deletions(-) create mode 100644 include/sparse/numa.h diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index 7b0736c96114..c4685fa7ebac 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -164,7 +164,7 @@ If a test case fails, check the log at:: Setup AF_XDP netdev ------------------- -Before running OVS with AF_XDP, make sure the libbpf and libelf are +Before running OVS with AF_XDP, make sure the libbpf, libelf, and libnuma are set-up right:: ldd vswitchd/ovs-vswitchd diff --git a/acinclude.m4 b/acinclude.m4 index 542637ac8cb8..73ed11d701aa 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -286,9 +286,12 @@ AC_DEFUN([OVS_CHECK_LINUX_AF_XDP], [ AC_CHECK_FUNCS([pthread_spin_lock], [], [AC_MSG_ERROR([unable to find pthread_spin_lock for AF_XDP support])]) + AC_CHECK_LIB(numa, numa_alloc_onnode, [], + [AC_MSG_ERROR([unable to find libnuma for AF_XDP support])]) + AC_DEFINE([HAVE_AF_XDP], [1], [Define to 1 if AF_XDP support is available and enabled.]) - LIBBPF_LDADD=" -lbpf -lelf" + LIBBPF_LDADD=" -lbpf -lelf -lnuma" AC_SUBST([LIBBPF_LDADD]) AC_CHECK_DECL([xsk_ring_prod__needs_wakeup], [ diff --git a/include/sparse/automake.mk b/include/sparse/automake.mk index 073631e8c082..974ad3fe55f7 100644 --- a/include/sparse/automake.mk +++ b/include/sparse/automake.mk @@ -5,6 +5,7 @@ noinst_HEADERS += \ include/sparse/bits/floatn.h \ include/sparse/assert.h \ include/sparse/math.h \ + include/sparse/numa.h \ include/sparse/netinet/in.h \ include/sparse/netinet/ip6.h \ include/sparse/netpacket/packet.h \ diff --git a/include/sparse/numa.h b/include/sparse/numa.h new file mode 100644 index 000000000000..3691a0eaf729 --- /dev/null +++ b/include/sparse/numa.h @@ -0,0 +1,27 @@ +/* + * Copyright (c) 2019 Nicira, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef __CHECKER__ +#error "Use this header only with sparse. It is not a correct implementation." +#endif + +/* Avoid sparse warning: non-ANSI function declaration of function" */ +#define numa_get_membind_compat() numa_get_membind_compat(void) +#define numa_get_interleave_mask_compat() numa_get_interleave_mask_compat(void) +#define numa_get_run_node_mask_compat() numa_get_run_node_mask_compat(void) + +/* Get actual definitions for us to annotate and build on. */ +#include_next diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index 4c1f9c68270a..2d6f739b4b67 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -469,7 +470,7 @@ xsk_configure_all(struct netdev *netdev) { struct netdev_linux *dev = netdev_linux_cast(netdev); int i, ifindex, n_rxq, n_txq; - int qid = 0; + int qid = 0, err = 0; ifindex = linux_get_ifindex(netdev_get_name(netdev)); @@ -477,6 +478,14 @@ xsk_configure_all(struct netdev *netdev) ovs_assert(dev->tx_locks == NULL); n_rxq = netdev_n_rxq(netdev); + + /* Allocate all the xsk related memory in the netdev's NUMA domain. */ + struct bitmask *old_bm = numa_get_membind(); + struct bitmask *new_bm = numa_allocate_nodemask(); + netdev_get_numa_id(netdev); + numa_bitmask_setbit(new_bm, dev->numa_id); + numa_set_membind(new_bm); + dev->xsks = xcalloc(n_rxq, sizeof *dev->xsks); if (dev->xdp_mode == OVS_AF_XDP_MODE_BEST_EFFORT) { @@ -518,11 +527,17 @@ xsk_configure_all(struct netdev *netdev) ovs_spin_init(&dev->tx_locks[i]); } - return 0; + goto out; err: xsk_destroy_all(netdev); - return EINVAL; + err = EINVAL; + +out: + numa_set_membind(old_bm); + numa_bitmask_free(old_bm); + numa_bitmask_free(new_bm); + return err; } static void