From patchwork Thu Mar 23 21:37:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander H Duyck X-Patchwork-Id: 742956 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3vq0LL2cZWz9s7j for ; Fri, 24 Mar 2017 08:39:06 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="K4GGaDHf"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756853AbdCWViH (ORCPT ); Thu, 23 Mar 2017 17:38:07 -0400 Received: from mail-io0-f193.google.com ([209.85.223.193]:36582 "EHLO mail-io0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753401AbdCWViE (ORCPT ); Thu, 23 Mar 2017 17:38:04 -0400 Received: by mail-io0-f193.google.com with SMTP id 68so923556ioh.3; Thu, 23 Mar 2017 14:37:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=s6O0eb6Cspuv5jYqqIpZB4Ay8+5UxaPvueuB3JPsrLs=; b=K4GGaDHfyBjsPt2uZuzcWU00ozuyEu4xdknHO7g6QeXDqjrDNaRkeRLeHUNkGYw2ot qke1c54Jn7tvDGczRYgGaUW07Eu6xLr9wFnln+vzJ1GHQCNDZi525nJOLWJ3vF+bQJM0 GM8pOAsWak6o0XiKna5EEmE26IZMzNJxtd1znYIh3WGHKqIZ8Dsfcm+xZwXYlgiHI1Ad jJksTKAzSZc6q48RM4b/gAM104ft0asPbml5Z3dVR/CccEeV1N+JUY0POR7eRqVByADV KUVDUyYy+MBtGLKX6G2gRkQTnnfHoeOvi4aUjsLIxPiRkWcX+cACyT+PeDZzH3NIfCLP cp2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=s6O0eb6Cspuv5jYqqIpZB4Ay8+5UxaPvueuB3JPsrLs=; b=sRUie0lvQMZqnOJ1cO29Dms9E3CfS5afY8qVGWHESmrew4JqelaxLKfp9b6HSQHKZh hJoHJ3bbxb57QLRIq6/9K4p+DuG4cOpbmz1Gu2oe/a4732liwk9vThLQaPG8vbRhn39A 9kRUeF3cS+eAZ9r3cjmnbeo0oXAsPIWOXsZaRYOukBX5yVNlkMF/2sBhofpUs9t/RxZJ L9vbzIFMAqk5sy8375V+Zqgg11vuCXGx+HGOltEyGipNSymnEI4VsI4tHTTXzxzpgiiE h/I4WEUF3akmtZR2CjXmDkx5xtJp5N0fPg3QKTHOuUQr7zWmqmNoQVtuObCYdCa/G9dn 7XxQ== X-Gm-Message-State: AFeK/H1kY2VWuRNJK/ytinQDt7LMNOM8BpAOB58xB64Arpy6BvybASUJy5KqF4Nv8CzzKw== X-Received: by 10.107.162.73 with SMTP id l70mr5433782ioe.184.1490305077438; Thu, 23 Mar 2017 14:37:57 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id h11sm144950ioa.43.2017.03.23.14.37.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Mar 2017 14:37:56 -0700 (PDT) Subject: [net-next PATCH v2 7/8] epoll: Add busy poll support to epoll with socket fds. From: Alexander Duyck To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sridhar.samudrala@intel.com, edumazet@google.com, davem@davemloft.net, linux-api@vger.kernel.org Date: Thu, 23 Mar 2017 14:37:55 -0700 Message-ID: <20170323213755.12615.6599.stgit@localhost.localdomain> In-Reply-To: <20170323211820.12615.88907.stgit@localhost.localdomain> References: <20170323211820.12615.88907.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Sridhar Samudrala This patch adds busy poll support to epoll. The implementation is meant to be opportunistic in that it will take the NAPI ID from the last socket that is added to the ready list that contains a valid NAPI ID and it will use that for busy polling until the ready list goes empty. Once the ready list goes empty the NAPI ID is reset and busy polling is disabled until a new socket is added to the ready list. In addition when we insert a new socket into the epoll we record the NAPI ID and assume we are going to receive events on it. If that doesn't occur it will be evicted as the active NAPI ID and we will resume normal behavior. An application can use SO_INCOMING_CPU or SO_REUSEPORT_ATTACH_C/EBPF socket options to spread the incoming connections to specific worker threads based on the incoming queue. This enables epoll for each worker thread to have only sockets that receive packets from a single queue. So when an application calls epoll_wait() and there are no events available to report, busy polling is done on the associated queue to pull the packets. Signed-off-by: Sridhar Samudrala Signed-off-by: Alexander Duyck --- fs/eventpoll.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 341251421ced..5420767c9b68 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -42,6 +42,7 @@ #include #include #include +#include /* * LOCKING: @@ -224,6 +225,11 @@ struct eventpoll { /* used to optimize loop detection check */ int visited; struct list_head visited_list_link; + +#ifdef CONFIG_NET_RX_BUSY_POLL + /* used to track busy poll napi_id */ + unsigned int napi_id; +#endif }; /* Wait structure used by the poll hooks */ @@ -384,6 +390,77 @@ static inline int ep_events_available(struct eventpoll *ep) return !list_empty(&ep->rdllist) || ep->ovflist != EP_UNACTIVE_PTR; } +#ifdef CONFIG_NET_RX_BUSY_POLL +static bool ep_busy_loop_end(void *p, unsigned long start_time) +{ + struct eventpoll *ep = p; + + return ep_events_available(ep) || busy_loop_timeout(start_time); +} +#endif /* CONFIG_NET_RX_BUSY_POLL */ + +/* + * Busy poll if globally on and supporting sockets found && no events, + * busy loop will return if need_resched or ep_events_available. + * + * we must do our busy polling with irqs enabled + */ +static void ep_busy_loop(struct eventpoll *ep, int nonblock) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + unsigned int napi_id = READ_ONCE(ep->napi_id); + + if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) + napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep); +#endif +} + +static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + if (ep->napi_id) + ep->napi_id = 0; +#endif +} + +/* + * Set epoll busy poll NAPI ID from sk. + */ +static inline void ep_set_busy_poll_napi_id(struct epitem *epi) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + struct eventpoll *ep; + unsigned int napi_id; + struct socket *sock; + struct sock *sk; + int err; + + if (!net_busy_loop_on()) + return; + + sock = sock_from_file(epi->ffd.file, &err); + if (!sock) + return; + + sk = sock->sk; + if (!sk) + return; + + napi_id = READ_ONCE(sk->sk_napi_id); + ep = epi->ep; + + /* Non-NAPI IDs can be rejected + * or + * Nothing to do if we already have this ID + */ + if (napi_id < MIN_NAPI_ID || napi_id == ep->napi_id) + return; + + /* record NAPI ID for use in next busy poll */ + ep->napi_id = napi_id; +#endif +} + /** * ep_call_nested - Perform a bound (possibly) nested call, by checking * that the recursion limit is not exceeded, and that @@ -1022,6 +1099,8 @@ static int ep_poll_callback(wait_queue_t *wait, unsigned mode, int sync, void *k spin_lock_irqsave(&ep->lock, flags); + ep_set_busy_poll_napi_id(epi); + /* * If the event mask does not contain any poll(2) event, we consider the * descriptor to be disabled. This condition is likely the effect of the @@ -1363,6 +1442,9 @@ static int ep_insert(struct eventpoll *ep, struct epoll_event *event, /* We have to drop the new item inside our item list to keep track of it */ spin_lock_irqsave(&ep->lock, flags); + /* record NAPI ID of new item if present */ + ep_set_busy_poll_napi_id(epi); + /* If the file is already "ready" we drop it inside the ready list */ if ((revents & event->events) && !ep_is_linked(&epi->rdllink)) { list_add_tail(&epi->rdllink, &ep->rdllist); @@ -1637,10 +1719,21 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } fetch_events: + + if (!ep_events_available(ep)) + ep_busy_loop(ep, timed_out); + spin_lock_irqsave(&ep->lock, flags); if (!ep_events_available(ep)) { /* + * Busy poll timed out. Drop NAPI ID for now, we can add + * it back in when we have moved a socket with a valid NAPI + * ID onto the ready list. + */ + ep_reset_busy_poll_napi_id(ep); + + /* * We don't have any available event to return to the caller. * We need to sleep here, and we will be wake up by * ep_poll_callback() when events will become available.