From patchwork Mon Sep 14 17:24:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363828 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=BAdvLttg; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4BqtZN3KFcz9sSP for ; Tue, 15 Sep 2020 03:26:36 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726273AbgINR0a (ORCPT ); Mon, 14 Sep 2020 13:26:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726234AbgINR0G (ORCPT ); Mon, 14 Sep 2020 13:26:06 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D453FC06174A for ; Mon, 14 Sep 2020 10:26:05 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 141so443697ybe.15 for ; Mon, 14 Sep 2020 10:26:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=3ZPCz6yGDMmfeZzFk+B+8dWhjbeTPlsxcJtM23X/JKE=; b=BAdvLttg53K+0G/jIPWu6h/QnW146GRRuLkr7FTksnweapdwEYMkFaU8GVx9pWCGQE VXHhR3MQA58yzSYcZztKEupITHMJP+9IvGrbUpRpv1QyByyGbPJ2WLJ424hSvHTFFwTn S2QRKj3MCdjv0NXT1LXTIFvOWVXwBfZnfYoi/WQGZ0z9FYVIs32zmG6nqW/eIFilu4xx KuiveIZF3RvgrCS7weIlFTEFXRl/Fj5srdO/Je2CH5eXiSxjRftcn/5AcRLbNGrh+xKB rlnH5aTFKr6UiRaeP8YLq2zLEgk4JVyZSAIlvY+ydV5cJNd4bq0cNTm5qKkecTeaWzgH 6IXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=3ZPCz6yGDMmfeZzFk+B+8dWhjbeTPlsxcJtM23X/JKE=; b=rw+yYYNrgKzvsvyuxyARbhx8miWYcFDFEw5dcVnfmcH0WQv0BIaUiYbgFwcRVhnnid xrMcTpqGDTXfSOpLttJFJ08X6uYL5wZ+gNOxnL5WvbYxmFV8LWy+2pFbhDW5xD9rJcgM q/m7AvDNuD9t3ukW/S3HTeAwDNT6I+QWpyf39grdizx87D8BDyd/RD7JGE/iZ763ZyW9 TIGF8hcpkv6NZjEOa79Q/xvsiWu+0myF67zX0SUXe6w0U15TErv3Dv09a1NHUz5Zj9gU XxTzUw1zTU8xjZ8b6TADGacsoZxeD832X7uS+Jn0qhZQBZt4R5vYclN02mVF1ztum77W 9wRA== X-Gm-Message-State: AOAM530gKd3nkhSSYmny5667Apo9O80rh/+ulLrk+MwpfhhSAh2Vxbx0 1I1EIMkh0+BfUYIYBNNFftUue+Yp2MQ= X-Google-Smtp-Source: ABdhPJxzrl0QtuCTAgds/ZKT7wxiRw6ekJPWHkUEQs+FUaMv9lrm2jVL6YknU/4VxmmXhfV9hlE/MRN6xgw= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a25:701:: with SMTP id 1mr22265173ybh.415.1600104364995; Mon, 14 Sep 2020 10:26:04 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:48 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-2-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 1/6] net: implement threaded-able napi poll loop support From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Paolo Abeni This patch allows running each napi poll loop inside its own kernel thread. The rx mode can be enabled per napi instance via the newly addded napi_set_threaded() api; the requested kthread will be created on demand and shut down on device stop. Once that threaded mode is enabled and the kthread is started, napi_schedule() will wake-up such thread instead of scheduling the softirq. The threaded poll loop behaves quite likely the net_rx_action, but it does not have to manipulate local irqs and uses an explicit scheduling point based on netdev_budget. Signed-off-by: Paolo Abeni Signed-off-by: Hannes Frederic Sowa Signed-off-by: Wei Wang --- include/linux/netdevice.h | 5 ++ net/core/dev.c | 113 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 157e0242e9ee..6797eb356e2e 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -348,6 +348,7 @@ struct napi_struct { struct list_head dev_list; struct hlist_node napi_hash_node; unsigned int napi_id; + struct task_struct *thread; }; enum { @@ -358,6 +359,7 @@ enum { NAPI_STATE_LISTED, /* NAPI added to system lists */ NAPI_STATE_NO_BUSY_POLL,/* Do not add in napi_hash, no busy polling */ NAPI_STATE_IN_BUSY_POLL,/* sk_busy_loop() owns this NAPI */ + NAPI_STATE_THREADED, /* The poll is performed inside its own thread*/ }; enum { @@ -368,6 +370,7 @@ enum { NAPIF_STATE_LISTED = BIT(NAPI_STATE_LISTED), NAPIF_STATE_NO_BUSY_POLL = BIT(NAPI_STATE_NO_BUSY_POLL), NAPIF_STATE_IN_BUSY_POLL = BIT(NAPI_STATE_IN_BUSY_POLL), + NAPIF_STATE_THREADED = BIT(NAPI_STATE_THREADED), }; enum gro_result { @@ -489,6 +492,8 @@ static inline bool napi_complete(struct napi_struct *n) return napi_complete_done(n, 0); } +int napi_set_threaded(struct napi_struct *n, bool threded); + /** * napi_disable - prevent NAPI from scheduling * @n: NAPI context diff --git a/net/core/dev.c b/net/core/dev.c index 03624192862a..0fe4c531b682 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -91,6 +91,7 @@ #include #include #include +#include #include #include #include @@ -1486,9 +1487,19 @@ void netdev_notify_peers(struct net_device *dev) } EXPORT_SYMBOL(netdev_notify_peers); +static int napi_threaded_poll(void *data); + +static void napi_thread_start(struct napi_struct *n) +{ + if (test_bit(NAPI_STATE_THREADED, &n->state) && !n->thread) + n->thread = kthread_create(napi_threaded_poll, n, "%s-%d", + n->dev->name, n->napi_id); +} + static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) { const struct net_device_ops *ops = dev->netdev_ops; + struct napi_struct *n; int ret; ASSERT_RTNL(); @@ -1520,6 +1531,9 @@ static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) if (!ret && ops->ndo_open) ret = ops->ndo_open(dev); + list_for_each_entry(n, &dev->napi_list, dev_list) + napi_thread_start(n); + netpoll_poll_enable(dev); if (ret) @@ -1565,6 +1579,14 @@ int dev_open(struct net_device *dev, struct netlink_ext_ack *extack) } EXPORT_SYMBOL(dev_open); +static void napi_thread_stop(struct napi_struct *n) +{ + if (!n->thread) + return; + kthread_stop(n->thread); + n->thread = NULL; +} + static void __dev_close_many(struct list_head *head) { struct net_device *dev; @@ -1593,6 +1615,7 @@ static void __dev_close_many(struct list_head *head) list_for_each_entry(dev, head, close_list) { const struct net_device_ops *ops = dev->netdev_ops; + struct napi_struct *n; /* * Call the device specific close. This cannot fail. @@ -1604,6 +1627,9 @@ static void __dev_close_many(struct list_head *head) if (ops->ndo_stop) ops->ndo_stop(dev); + list_for_each_entry(n, &dev->napi_list, dev_list) + napi_thread_stop(n); + dev->flags &= ~IFF_UP; netpoll_poll_enable(dev); } @@ -4240,6 +4266,11 @@ int gro_normal_batch __read_mostly = 8; static inline void ____napi_schedule(struct softnet_data *sd, struct napi_struct *napi) { + if (napi->thread) { + wake_up_process(napi->thread); + return; + } + list_add_tail(&napi->poll_list, &sd->poll_list); __raise_softirq_irqoff(NET_RX_SOFTIRQ); } @@ -6590,6 +6621,30 @@ static void init_gro_hash(struct napi_struct *napi) napi->gro_bitmask = 0; } +int napi_set_threaded(struct napi_struct *n, bool threaded) +{ + ASSERT_RTNL(); + + if (n->dev->flags & IFF_UP) + return -EBUSY; + + if (threaded == !!test_bit(NAPI_STATE_THREADED, &n->state)) + return 0; + if (threaded) + set_bit(NAPI_STATE_THREADED, &n->state); + else + clear_bit(NAPI_STATE_THREADED, &n->state); + + /* if the device is initializing, nothing todo */ + if (test_bit(__LINK_STATE_START, &n->dev->state)) + return 0; + + napi_thread_stop(n); + napi_thread_start(n); + return 0; +} +EXPORT_SYMBOL(napi_set_threaded); + void netif_napi_add(struct net_device *dev, struct napi_struct *napi, int (*poll)(struct napi_struct *, int), int weight) { @@ -6730,6 +6785,64 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) return work; } +static int napi_thread_wait(struct napi_struct *napi) +{ + set_current_state(TASK_INTERRUPTIBLE); + + while (!kthread_should_stop() && !napi_disable_pending(napi)) { + if (test_bit(NAPI_STATE_SCHED, &napi->state)) { + __set_current_state(TASK_RUNNING); + return 0; + } + + schedule(); + set_current_state(TASK_INTERRUPTIBLE); + } + __set_current_state(TASK_RUNNING); + return -1; +} + +static int napi_threaded_poll(void *data) +{ + struct napi_struct *napi = data; + + while (!napi_thread_wait(napi)) { + struct list_head dummy_repoll; + int budget = netdev_budget; + unsigned long time_limit; + bool again = true; + + INIT_LIST_HEAD(&dummy_repoll); + local_bh_disable(); + time_limit = jiffies + 2; + do { + /* ensure that the poll list is not empty */ + if (list_empty(&dummy_repoll)) + list_add(&napi->poll_list, &dummy_repoll); + + budget -= napi_poll(napi, &dummy_repoll); + if (unlikely(budget <= 0 || + time_after_eq(jiffies, time_limit))) { + cond_resched(); + + /* refresh the budget */ + budget = netdev_budget; + __kfree_skb_flush(); + time_limit = jiffies + 2; + } + + if (napi_disable_pending(napi)) + again = false; + else if (!test_bit(NAPI_STATE_SCHED, &napi->state)) + again = false; + } while (again); + + __kfree_skb_flush(); + local_bh_enable(); + } + return 0; +} + static __latent_entropy void net_rx_action(struct softirq_action *h) { struct softnet_data *sd = this_cpu_ptr(&softnet_data); From patchwork Mon Sep 14 17:24:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363830 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=a5opZK4r; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Bqtb10nKnz9sSP for ; Tue, 15 Sep 2020 03:27:09 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726038AbgINR0r (ORCPT ); Mon, 14 Sep 2020 13:26:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726258AbgINR0N (ORCPT ); Mon, 14 Sep 2020 13:26:13 -0400 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F9B2C061788 for ; Mon, 14 Sep 2020 10:26:08 -0700 (PDT) Received: by mail-qk1-x74a.google.com with SMTP id r128so656354qkc.9 for ; Mon, 14 Sep 2020 10:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=RgCsCaqgYmgC1Mfvq5pQLQQYxiHHLMC+4ovtFO74twY=; b=a5opZK4rIQ7YDjDbYJ+2QVaMYI2KAbXkyKoDpN6EhfLpxLKrOpgJxGArBJKzRP9TgP fqWGwi7Zs16jcFSADA/YMarv6KYd7COZqI3/myEXOutxOInstfoWvosR7JDphj88/ZVs R9dDVAISE0Y8/kX0Jj1mR52jbLDLnIA/oJHnUYT5t/pjL/abM7O3Pen3yuWEfweHxAEe WDKDlZnXLjFRxhAsUMh0GLkoW+xD9dScICQoBCB8itkqQOqHQdxel0TbxX7QR3ijj5Qc 4imAOfPNrmteUL40AYJNW5VQ6YZln6xUuSv6mEzntBDZXIcCWT9y9dkgY+3wnmvJY7/t gsvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=RgCsCaqgYmgC1Mfvq5pQLQQYxiHHLMC+4ovtFO74twY=; b=bTzw3+/47jUU7L31vs782OUO5dE55bKEk9YRw2oQ8uKerEXrm3oKY+NFfOqmsOMU4G ymOHW+iKMyjVcOg8RVQA32Gh4QDIGMEp/l8Js4CJElfr2fVxUpm6JI6ec65GAWnbmx6Z XkBpnIzDxqvTgONdT24SOZJZkSnOe5ZLy1XSMLJaYgiYLgZfuqqVpFHZMvd07CaZb2Yg O2QOpfELBtDE0JcwI4Apf1N0asDa2JGJuG2ZSugUIC+mKQSOyZhxM0b1FfGO4xR873ms O1ctnMX4bxX7xbnImAhPxeQUnZv+x9mbBoqVWVDKHbeGzDJWbSKTYbFx6LF81faOFMXI DQZg== X-Gm-Message-State: AOAM530HZiA6ICWBv1y5hHFEvL2WVmHBrhUwiNe6RXVwj0YJmmeNz70O JQIod3LoNw0nCxglcgtyvVDxWudXLxo= X-Google-Smtp-Source: ABdhPJzSc5vtgLiMS/r8RgBBjpYhyUOUivM35PDt0VyyHE24tGqdkgNAw14ZgJccRp+aJMqLyaPxT1elDS0= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a0c:c390:: with SMTP id o16mr14569352qvi.0.1600104367439; Mon, 14 Sep 2020 10:26:07 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:49 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-3-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 2/6] net: add sysfs attribute to control napi threaded mode From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Paolo Abeni this patch adds a new sysfs attribute to the network device class. Said attribute is a bitmask that allows controlling the threaded mode for all the napi instances of the given network device. The threaded mode can be switched only if related network device is down. Signed-off-by: Paolo Abeni Signed-off-by: Hannes Frederic Sowa Signed-off-by: Wei Wang --- include/linux/netdevice.h | 1 + net/core/net-sysfs.c | 102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 6797eb356e2e..37941d6a911d 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1965,6 +1965,7 @@ struct net_device { spinlock_t addr_list_lock; unsigned char name_assign_type; bool uc_promisc; + bool napi_threaded; struct netdev_hw_addr_list uc; struct netdev_hw_addr_list mc; struct netdev_hw_addr_list dev_addrs; diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index efec66fa78b7..0172457a1bfe 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -538,6 +538,107 @@ static ssize_t phys_switch_id_show(struct device *dev, } static DEVICE_ATTR_RO(phys_switch_id); +unsigned long *__alloc_thread_bitmap(struct net_device *netdev, int *bits) +{ + struct napi_struct *n; + + *bits = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) + (*bits)++; + + return kmalloc_array(BITS_TO_LONGS(*bits), sizeof(unsigned long), + GFP_ATOMIC | __GFP_ZERO); +} + +static ssize_t threaded_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct net_device *netdev = to_net_dev(dev); + struct napi_struct *n; + unsigned long *bmap; + size_t count = 0; + int i, bits; + + if (!rtnl_trylock()) + return restart_syscall(); + + if (!dev_isalive(netdev)) + goto unlock; + + bmap = __alloc_thread_bitmap(netdev, &bits); + if (!bmap) { + count = -ENOMEM; + goto unlock; + } + + i = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) { + if (test_bit(NAPI_STATE_THREADED, &n->state)) + set_bit(i, bmap); + i++; + } + + count = bitmap_print_to_pagebuf(true, buf, bmap, bits); + kfree(bmap); + +unlock: + rtnl_unlock(); + + return count; +} + +static ssize_t threaded_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + struct net_device *netdev = to_net_dev(dev); + struct napi_struct *n; + unsigned long *bmap; + int i, bits; + size_t ret; + + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + if (!rtnl_trylock()) + return restart_syscall(); + + if (!dev_isalive(netdev)) { + ret = len; + goto unlock; + } + + if (netdev->flags & IFF_UP) { + ret = -EBUSY; + goto unlock; + } + + bmap = __alloc_thread_bitmap(netdev, &bits); + if (!bmap) { + ret = -ENOMEM; + goto unlock; + } + + ret = bitmap_parselist(buf, bmap, bits); + if (ret) + goto free_unlock; + + i = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) { + napi_set_threaded(n, test_bit(i, bmap)); + i++; + } + ret = len; + +free_unlock: + kfree(bmap); + +unlock: + rtnl_unlock(); + return ret; +} +static DEVICE_ATTR_RW(threaded); + static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_netdev_group.attr, &dev_attr_type.attr, @@ -570,6 +671,7 @@ static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_proto_down.attr, &dev_attr_carrier_up_count.attr, &dev_attr_carrier_down_count.attr, + &dev_attr_threaded.attr, NULL, }; ATTRIBUTE_GROUPS(net_class); From patchwork Mon Sep 14 17:24:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363831 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=lctbWoGO; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4BqtbS0n1jz9sSP for ; Tue, 15 Sep 2020 03:27:32 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726200AbgINR1R (ORCPT ); Mon, 14 Sep 2020 13:27:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726058AbgINR0U (ORCPT ); Mon, 14 Sep 2020 13:26:20 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AC9FC06178B for ; Mon, 14 Sep 2020 10:26:19 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id f4so280945qvw.15 for ; Mon, 14 Sep 2020 10:26:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BYFSDnsZ7CuyDxsknJLPnmb95P0RzbVeR4SkBJzlkVA=; b=lctbWoGOpEsl+BNavcOA1GcDL2htg6vhGxsTtZ7XDhsDTuA0MzKWBWZTTJ9aejTGN0 k7O14lk2EPG9I9PWSbbMGrIu9J29CltGhy/tYmKbvaFiFzVORxavfn4M2UsR96LdKhrG D2LNXtI8sVoUAtEsRWxpFKC3/Tnw4WiLyxpNaHL9qg2y3siZSG9ACXwMP0JP4PjCqRmE SDlPDpSK+/HeTyooUHMZBCUz/tjIhYk+sPiWYuAfLUvSLRWosYsCzi+AexNjplWiJ5Aq midNsks4mQc0ohlcohX2b4KcQey7DAi87tkokRf69rgjFPPyghYyTqXT04OK95DUuH2c 1rtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BYFSDnsZ7CuyDxsknJLPnmb95P0RzbVeR4SkBJzlkVA=; b=WBDQ9ChXWjNk5PT/ev3YZB2te4HKxWwJGVtMVlZBGgI2GTrII4RXgMoaX5oH6uYfdb N2jyOZbFVYaiYq85jHT7OpNMDililhr3BIVHSQXcwwYzMZLauEEbBo9iWrnqndwzneSt Ls6nqOeKl7NiGLJUnWgqr71aGhGt5dWE3ePLG7qi+CWpi1c6QoZWmRvOiar18u8sM//G kHoVI5zDiZCNDSPfuJEEETdNLEUcngQNVPBIz4YUWqfqXmkg3FNprw8PsSYelvjsDI4I Xzst9o2Ztj29UA4MntK53a/Z4ML3ynukriGHb1xtVt6TakYb2OR7BtBzAxl9R+Fe9V1m /4OA== X-Gm-Message-State: AOAM531h95W8Q9m6xVrek/Pig9ZS0UVo1meOliTuvec22BTYRmcy00OS t8I570p09ZXpzyN1h8g+XGNtCy896BI= X-Google-Smtp-Source: ABdhPJw+KfQCn3FQSFLrXBvQa9e6HjajYy/kMe2SX9WQJMBOcgaLQShXWkJ6goWbp0bov21Tz4K0L1oJO+M= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:ad4:5a53:: with SMTP id ej19mr14402811qvb.54.1600104376172; Mon, 14 Sep 2020 10:26:16 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:50 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-4-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 3/6] net: extract napi poll functionality to __napi_poll() From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Felix Fietkau This commit introduces a new function __napi_poll() which does the main logic of the existing napi_poll() function, and will be called by other functions in later commits. This idea and implementation is done by Felix Fietkau and is proposed as part of the patch to move napi work to work_queue context. This commit by itself is a code restructure. Signed-off-by: Felix Fietkau Signed-off-by: Wei Wang --- net/core/dev.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 0fe4c531b682..bc2a7681b239 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6719,15 +6719,10 @@ void __netif_napi_del(struct napi_struct *napi) } EXPORT_SYMBOL(__netif_napi_del); -static int napi_poll(struct napi_struct *n, struct list_head *repoll) +static int __napi_poll(struct napi_struct *n, bool *repoll) { - void *have; int work, weight; - list_del_init(&n->poll_list); - - have = netpoll_poll_lock(n); - weight = n->weight; /* This NAPI_STATE_SCHED test is for avoiding a race @@ -6747,7 +6742,7 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) n->poll, work, weight); if (likely(work < weight)) - goto out_unlock; + return work; /* Drivers must not modify the NAPI state if they * consume the entire weight. In such cases this code @@ -6756,7 +6751,7 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) */ if (unlikely(napi_disable_pending(n))) { napi_complete(n); - goto out_unlock; + return work; } if (n->gro_bitmask) { @@ -6768,6 +6763,26 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) gro_normal_list(n); + *repoll = true; + + return work; +} + +static int napi_poll(struct napi_struct *n, struct list_head *repoll) +{ + bool do_repoll = false; + void *have; + int work; + + list_del_init(&n->poll_list); + + have = netpoll_poll_lock(n); + + work = __napi_poll(n, &do_repoll); + + if (!do_repoll) + goto out_unlock; + /* Some drivers may have called napi_schedule * prior to exhausting their budget. */ From patchwork Mon Sep 14 17:24:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363832 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=O+YSkcEp; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Bqtbx64KDz9sTK for ; Tue, 15 Sep 2020 03:27:57 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726056AbgINR1k (ORCPT ); Mon, 14 Sep 2020 13:27:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726306AbgINR00 (ORCPT ); Mon, 14 Sep 2020 13:26:26 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1715EC06174A for ; Mon, 14 Sep 2020 10:26:26 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id e190so433325ybf.18 for ; Mon, 14 Sep 2020 10:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=LYZ1+juOAoa/emu0bMvEoA7YWrA+PmRXyRdY+YEq8Tw=; b=O+YSkcEpuAZmFQfPD/KixyL1MdeWGfam/Ed3HgnOtiD19nRsM+q4hIJNZ8iYDCMXO2 OJhS/UQ9yAQKswOrSJE1+layYfB8EHSM0LZCBEFM0qe7R+8cA8FCTTyqDQ6K5q7fmrvv 1cIgi5hBumdLaHswaWxlWz2ErltL0lngGk8vM/jEeqnUXFvU8ibUfY9kBg2qoPv2BT6H 6Z53xZdqI92Ng9VOmihDhUg9pELtMpnjrjKQZhx2lVOjGpoyCSz0q/GUaSHMMD8y6Ljx 2ee1aDgP9frXEOGveJv4p3HePBWaE7qa2C3lCG+tZl1myP6jlZ4w64dITo90N/JDZVQU NQ8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LYZ1+juOAoa/emu0bMvEoA7YWrA+PmRXyRdY+YEq8Tw=; b=rvfio7kuGitLYBEYmVJ50iG/dzYKWXdzWdBHpgVPyn9vB7c6QKhEdalklI6COMu7z1 dnThRUHh+BCzTWFJ0VhvsbL3zUV0YM2CMq+zxhuM2I4ZrX9iSj6aLzlxOMuV4EaNaABr NPUUs42QwfmfbdEDEoFomfMdHboDu1Sxm2R4CQ4pQWDkyq/hQ4LagGzMR1Ycv1bShyB3 3rEIkz/v7QSuEisEsV91vAw/2Tt5F/u+4LEm+96uSSeIbuHwvg709i/8hwpd2e8qYbf4 wjQTXJ35ciIlOabU6TUfcXNV6o/by6lvSfT4MZbSNafGi2zie4fkSiYllBz+TqKjZnsp f5Vg== X-Gm-Message-State: AOAM5326S/2RiJIhieq4MpqxWn2L3oviE0YgyKO/RIphMpsKATVYEKuX oB3aPdxzCPDk7ZZkxz8NpjFpDl/qBps= X-Google-Smtp-Source: ABdhPJzEDpRe07jBYHx33+e1peTFcjdq7HiVHjsAcCt51s3ColGJm2b58l6xOeggCjSPidNXsF0UTCaMdp8= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a25:f20d:: with SMTP id i13mr20998124ybe.152.1600104385278; Mon, 14 Sep 2020 10:26:25 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:51 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-5-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 4/6] net: modify kthread handler to use __napi_poll() From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Jakub Kicinski The current kthread handler calls napi_poll() and has to pass a dummy repoll list to the function, which seems redundent. The new proposed kthread handler calls the newly proposed __napi_poll(), and respects napi->weight as before. If repoll is needed, cond_resched() is called first to give other tasks a chance to run before repolling. This change is proposed by Jakub Kicinski on top of the previous patch. Signed-off-by: Jakub Kicinski Signed-off-by: Wei Wang --- net/core/dev.c | 62 +++++++++++++++++++------------------------------- 1 file changed, 24 insertions(+), 38 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index bc2a7681b239..be676c21bdc4 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6763,6 +6763,15 @@ static int __napi_poll(struct napi_struct *n, bool *repoll) gro_normal_list(n); + /* Some drivers may have called napi_schedule + * prior to exhausting their budget. + */ + if (unlikely(!list_empty(&n->poll_list))) { + pr_warn_once("%s: Budget exhausted after napi rescheduled\n", + n->dev ? n->dev->name : "backlog"); + return work; + } + *repoll = true; return work; @@ -6783,15 +6792,6 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) if (!do_repoll) goto out_unlock; - /* Some drivers may have called napi_schedule - * prior to exhausting their budget. - */ - if (unlikely(!list_empty(&n->poll_list))) { - pr_warn_once("%s: Budget exhausted after napi rescheduled\n", - n->dev ? n->dev->name : "backlog"); - goto out_unlock; - } - list_add_tail(&n->poll_list, repoll); out_unlock: @@ -6820,40 +6820,26 @@ static int napi_thread_wait(struct napi_struct *napi) static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; + void *have; while (!napi_thread_wait(napi)) { - struct list_head dummy_repoll; - int budget = netdev_budget; - unsigned long time_limit; - bool again = true; + for (;;) { + bool repoll = false; - INIT_LIST_HEAD(&dummy_repoll); - local_bh_disable(); - time_limit = jiffies + 2; - do { - /* ensure that the poll list is not empty */ - if (list_empty(&dummy_repoll)) - list_add(&napi->poll_list, &dummy_repoll); - - budget -= napi_poll(napi, &dummy_repoll); - if (unlikely(budget <= 0 || - time_after_eq(jiffies, time_limit))) { - cond_resched(); - - /* refresh the budget */ - budget = netdev_budget; - __kfree_skb_flush(); - time_limit = jiffies + 2; - } + local_bh_disable(); - if (napi_disable_pending(napi)) - again = false; - else if (!test_bit(NAPI_STATE_SCHED, &napi->state)) - again = false; - } while (again); + have = netpoll_poll_lock(napi); + __napi_poll(napi, &repoll); + netpoll_poll_unlock(have); - __kfree_skb_flush(); - local_bh_enable(); + __kfree_skb_flush(); + local_bh_enable(); + + if (!repoll) + break; + + cond_resched(); + } } return 0; } From patchwork Mon Sep 14 17:24:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363841 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=u6zTyYRq; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4BqtmF2BHKz9sSP for ; Tue, 15 Sep 2020 03:35:09 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726342AbgINRfG (ORCPT ); Mon, 14 Sep 2020 13:35:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725957AbgINR03 (ORCPT ); Mon, 14 Sep 2020 13:26:29 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D28CBC06174A for ; Mon, 14 Sep 2020 10:26:28 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id p187so451365ybg.14 for ; Mon, 14 Sep 2020 10:26:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=N7ud6SVmQsyeAo1FFabO4II1hmbm33EQsopeZ69GJts=; b=u6zTyYRqKGoaz6mBSCvkfnvI+0iG3USvJ/2orh5uqMIK9Ty23zTEg4Eepbmhfy+oN3 FIX918dyhMIQ524UpViV99xoLz5XC4LAEd8Pp6G7tHYfvfUT3wwSq+QXs82NCCSsijY+ xzptmcUEoU/vYjWdEVLadLcfbHNKDuKGy8A/oV0oqx1jqhXNpQOp4pfYLbk6NX6eJ4pl GpVPeyNbZD5iV3SWzvmGW1SGehwbLZ4YzI1T3rd4yW43lYmRPAgvWvSJ9/r/XjNrMonL XhkDelr4x+dLtgDew1ddbiGP2/+kPBDHoKg/uwW/6tN39AGzZbyC/+MTs1ZiAXeLw9jn LtnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=N7ud6SVmQsyeAo1FFabO4II1hmbm33EQsopeZ69GJts=; b=TVpN1H7yfgmpSvQonMPimnVwARYZD95Voi56COk3Rp5ftqiOnMpA4ahgEGpeAyNOr3 bQfa3Vm2KUbNRrlSBgscADwQugw8N027EGiSuYHlFoKCX1UYKodX+fXFZaJuiCmrnFJ2 DqoWu7yBUbHOoKHrnyW7jJWu4YnpeqA2qWKoOPDzkRLxlplg8IGFeF5C+G8539gL00Nu UHVW6s+t+pUN0lPVc4/Vdd5XSIuPEOKz7rr2IK08yLLUXsUsqldUXEW4S30iSpxxWIJv 0OoQ1BEpg+0WJAu6F6V57SrVOqArqeOpkNuVBn+A5Y8LXklzypHsl7njgfWcpcinUuOv l/Hg== X-Gm-Message-State: AOAM5332tbdp5HnxzbaG9fe/gAE5rqIUwlaf3RPi/rmElHb0rBWUj+VW FTh6Lt+zLEF4Bh8h6B4RwhrCh580zig= X-Google-Smtp-Source: ABdhPJyV8DDlE5ivVzjfIvgNKZ18IxkPOYGk0n9VAYweLKxZKilu6XpGHOJ9AdlIsYjx1HOSSJOt34Egi+U= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a25:ae9e:: with SMTP id b30mr7629296ybj.281.1600104388111; Mon, 14 Sep 2020 10:26:28 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:52 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-6-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 5/6] net: process RPS/RFS work in kthread context From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Paolo Abeni This patch adds the missing part to handle RFS/RPS in the napi thread handler and makes sure RPS/RFS works properly when using kthread to do napi poll. Signed-off-by: Paolo Abeni Signed-off-by: Wei Wang --- net/core/dev.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/net/core/dev.c b/net/core/dev.c index be676c21bdc4..ab8af727058b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6820,6 +6820,7 @@ static int napi_thread_wait(struct napi_struct *napi) static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; + struct softnet_data *sd; void *have; while (!napi_thread_wait(napi)) { @@ -6835,6 +6836,12 @@ static int napi_threaded_poll(void *data) __kfree_skb_flush(); local_bh_enable(); + sd = this_cpu_ptr(&softnet_data); + if (sd_has_rps_ipi_waiting(sd)) { + local_irq_disable(); + net_rps_action_and_irq_enable(sd); + } + if (!repoll) break; From patchwork Mon Sep 14 17:24:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 1363833 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ukPXw99f; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4BqtcT1snpz9sTR for ; Tue, 15 Sep 2020 03:28:25 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726094AbgINR17 (ORCPT ); Mon, 14 Sep 2020 13:27:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726353AbgINR0e (ORCPT ); Mon, 14 Sep 2020 13:26:34 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5DE0C061788 for ; Mon, 14 Sep 2020 10:26:32 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id hr13so499024pjb.3 for ; Mon, 14 Sep 2020 10:26:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=PQr51oLt7gFvXdGEi4RaoFMJAvJLCKvLDQkYyMQ+TnE=; b=ukPXw99fFce5QP/Wfa0z3Q4vl+10VKzHKklnQFvqoY4iIz+W4BJ7MKf8BlEEkxu2dn 2J1jBCybOoPXBnkqri7YFsAwoqX8lQERggtl10pCRV7UEXqsotC2UbPGXIvpn5clOgSX +u1TDy1kPwIdJJkTxqOvjpVveO/Vec7pH/DNpsANWAJyj5BsbIcUSRzc9WKdEruxTRrY g230U8qUlWTEyxP7UaVZ8TFkOL2edjyHs9od9WFqvMPpTI27M4NWYn+LQLVBdFpfpqNJ f5N1KSOwaToDrTMoh4Ea9rgMx6gVNHQMmM8i/zrkI/nL5gxzZtQJ9W1FsJPvljOUHQNR OB4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PQr51oLt7gFvXdGEi4RaoFMJAvJLCKvLDQkYyMQ+TnE=; b=X6kbyJMivy+ccR/TuAwLTy19gN6b/emhxKRdb7ZmqGOWI1apmOGyZLBtoXYDJpTQAO lQ10iQnU9VaA0F/FtyaEx7ewm7oGBc3iCG2x92usAVvDJF/zWy2gcaUz9hLMw9IgpU9Z O+Zp6uofCHT1huTHTlb8XHBdGwiRp0VtApVKZhwuCQv8dKQ6+B6eE1hk+kgMxKtP9pLx lkoIxJxQkuSC22av3bDCjRz08pQTQrE1NA7L6/W9b2nY273HXLziMXwlAmSz4oxRjAsy oVOo/4HqEkC9NCiEZCHAdKQPvoLiNXqAdq78wYNzf5yJBYwu0HKgdU8cStjqvcOncDTB YI8A== X-Gm-Message-State: AOAM532y+XjeHMB6zhSD9qBO+gtfbkQi5c+FMRh39WTNtKOmyzduY9Kz mxii+sGHYm2mveroP0MWVt8IVy2y9Ng= X-Google-Smtp-Source: ABdhPJy5d2Vp/j9Lr4okRSrVE8uSrBJKS+z9eiiFHItwr39nG+gSO+Ru8FEdpGtrDul1Q8DgiI92lMrPIVE= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a17:902:ab88:b029:d1:9be2:c683 with SMTP id f8-20020a170902ab88b02900d19be2c683mr15031435plr.24.1600104391252; Mon, 14 Sep 2020 10:26:31 -0700 (PDT) Date: Mon, 14 Sep 2020 10:24:53 -0700 In-Reply-To: <20200914172453.1833883-1-weiwan@google.com> Message-Id: <20200914172453.1833883-7-weiwan@google.com> Mime-Version: 1.0 References: <20200914172453.1833883-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [RFC PATCH net-next 6/6] net: improve napi threaded config From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Jakub Kicinski , Eric Dumazet , Paolo Abeni , Hannes Frederic Sowa , Felix Fietkau , Wei Wang Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit mainly addresses the threaded config to make the switch between softirq based and kthread based NAPI processing not require a device down/up. It also moves the kthread_create() call to the sysfs handler when user tries to enable "threaded" on napi, and properly handles the kthread_create() failure. This is because certain drivers do not have the napi created and linked to the dev when dev_open() is called. So the previous implementation does not work properly there. Signed-off-by: Wei Wang --- net/core/dev.c | 49 +++++++++++++++++++++++++------------------- net/core/net-sysfs.c | 9 +++----- 2 files changed, 31 insertions(+), 27 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index ab8af727058b..9f7df61f7c9a 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1489,17 +1489,24 @@ EXPORT_SYMBOL(netdev_notify_peers); static int napi_threaded_poll(void *data); -static void napi_thread_start(struct napi_struct *n) +static int napi_kthread_create(struct napi_struct *n) { - if (test_bit(NAPI_STATE_THREADED, &n->state) && !n->thread) - n->thread = kthread_create(napi_threaded_poll, n, "%s-%d", - n->dev->name, n->napi_id); + int err = 0; + + n->thread = kthread_create(napi_threaded_poll, n, "%s-%d", + n->dev->name, n->napi_id); + if (IS_ERR(n->thread)) { + err = PTR_ERR(n->thread); + pr_err("kthread_create failed with err %d\n", err); + n->thread = NULL; + } + + return err; } static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) { const struct net_device_ops *ops = dev->netdev_ops; - struct napi_struct *n; int ret; ASSERT_RTNL(); @@ -1531,9 +1538,6 @@ static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) if (!ret && ops->ndo_open) ret = ops->ndo_open(dev); - list_for_each_entry(n, &dev->napi_list, dev_list) - napi_thread_start(n); - netpoll_poll_enable(dev); if (ret) @@ -1584,6 +1588,7 @@ static void napi_thread_stop(struct napi_struct *n) if (!n->thread) return; kthread_stop(n->thread); + clear_bit(NAPI_STATE_THREADED, &n->state); n->thread = NULL; } @@ -4266,7 +4271,7 @@ int gro_normal_batch __read_mostly = 8; static inline void ____napi_schedule(struct softnet_data *sd, struct napi_struct *napi) { - if (napi->thread) { + if (test_bit(NAPI_STATE_THREADED, &napi->state)) { wake_up_process(napi->thread); return; } @@ -6623,25 +6628,25 @@ static void init_gro_hash(struct napi_struct *napi) int napi_set_threaded(struct napi_struct *n, bool threaded) { - ASSERT_RTNL(); + int err = 0; - if (n->dev->flags & IFF_UP) - return -EBUSY; + ASSERT_RTNL(); if (threaded == !!test_bit(NAPI_STATE_THREADED, &n->state)) return 0; - if (threaded) + if (threaded) { + if (!n->thread) { + err = napi_kthread_create(n); + if (err) + goto out; + } set_bit(NAPI_STATE_THREADED, &n->state); - else + } else { clear_bit(NAPI_STATE_THREADED, &n->state); + } - /* if the device is initializing, nothing todo */ - if (test_bit(__LINK_STATE_START, &n->dev->state)) - return 0; - - napi_thread_stop(n); - napi_thread_start(n); - return 0; +out: + return err; } EXPORT_SYMBOL(napi_set_threaded); @@ -6686,6 +6691,7 @@ void napi_disable(struct napi_struct *n) msleep(1); hrtimer_cancel(&n->timer); + napi_thread_stop(n); clear_bit(NAPI_STATE_DISABLE, &n->state); } @@ -6806,6 +6812,7 @@ static int napi_thread_wait(struct napi_struct *napi) while (!kthread_should_stop() && !napi_disable_pending(napi)) { if (test_bit(NAPI_STATE_SCHED, &napi->state)) { + WARN_ON(!list_empty(&napi->poll_list)); __set_current_state(TASK_RUNNING); return 0; } diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index 0172457a1bfe..48b7582a0372 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -608,11 +608,6 @@ static ssize_t threaded_store(struct device *dev, goto unlock; } - if (netdev->flags & IFF_UP) { - ret = -EBUSY; - goto unlock; - } - bmap = __alloc_thread_bitmap(netdev, &bits); if (!bmap) { ret = -ENOMEM; @@ -625,7 +620,9 @@ static ssize_t threaded_store(struct device *dev, i = 0; list_for_each_entry(n, &netdev->napi_list, dev_list) { - napi_set_threaded(n, test_bit(i, bmap)); + ret = napi_set_threaded(n, test_bit(i, bmap)); + if (ret) + goto free_unlock; i++; } ret = len;