From patchwork Wed Jun 21 21:18:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Willem de Bruijn X-Patchwork-Id: 779124 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3wtHd92n7Mz9s2s for ; Thu, 22 Jun 2017 07:18:37 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="XAirnsCI"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752112AbdFUVSe (ORCPT ); Wed, 21 Jun 2017 17:18:34 -0400 Received: from mail-qt0-f195.google.com ([209.85.216.195]:35632 "EHLO mail-qt0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751818AbdFUVSc (ORCPT ); Wed, 21 Jun 2017 17:18:32 -0400 Received: by mail-qt0-f195.google.com with SMTP id x58so29081104qtc.2; Wed, 21 Jun 2017 14:18:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sg93nq3X06OnL9LoLJa7EUYokeCbFog/l5E7YyxMpe8=; b=XAirnsCI/NmTvObX/+bz2KiF2+NYm90spQLQ+HhRDL4oVYaYAEWSQHK0nGuZmIdApZ MUD5+oveE0Y02Gq7vRLXul97rjGGOjDw5p0lX3oZLBmuBlTJXrS9+hSbdn4zOlnFqDem 0UPKtlICl0HnFjhvEAtLJu0NJsJ+t5vfEm857y6OSZmSOGu5bheKhzw+HyAcmJIsv3HD AoGJmjbDhgymq+gq10LlJj8kSlVKkIlJSwuCThKTtUaeIkI9XAKbgTHCyKPAm7WcHL8i J5PuCIAefJSDFBD4x1SKCLiQgVS+q0Wdcha/LM00Uvz0I8WAScI1B29NVKOBvgFQQ7g4 MRKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sg93nq3X06OnL9LoLJa7EUYokeCbFog/l5E7YyxMpe8=; b=cEdKjjvMZaF5HZifXsDmrMooV7bLVibg0sbxjHDyLL7oqanzX8TO9x8nRoBXO5rEHq rezocd7u7KLHJBtu/wbWDMTuV7THWmF6B7Zj5EA/88+HMUSb4ZrstkPJpW3OINF58mFg bmZiEzf9IUnZGWZ+DgY6xXh2QguEcwOszAX9tppzEWcEdaBbryqr7qAFSaHghFiwKYzk meAuIbzbdoSUagpcH9buZ7omBmB9ajhgqkwoPGQjoYSsp9uSJ7Dhc0d74oWCgoTkmf1Y cZ8XrNBPcCRvkoI+jZN47YQPzMsgrrRnI/c4LBshVh4tYeefIyfVD60MqzGaJ147DA6O PMRA== X-Gm-Message-State: AKS2vOxti5HyMFF55dLpegt3b+oR4ATtFmn2KJkF/KSmYpfSN6ERIm10 kRBfNMYhIuUFBzTlrCI= X-Received: by 10.200.51.172 with SMTP id c41mr4211631qtb.71.1498079910883; Wed, 21 Jun 2017 14:18:30 -0700 (PDT) Received: from willemb1.nyc.corp.google.com ([100.101.212.72]) by smtp.gmail.com with ESMTPSA id r7sm2122755qtb.4.2017.06.21.14.18.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 21 Jun 2017 14:18:30 -0700 (PDT) From: Willem de Bruijn To: netdev@vger.kernel.org Cc: davem@davemloft.net, linux-api@vger.kernel.org, Willem de Bruijn Subject: [PATCH net-next v3 09/13] tcp: enable MSG_ZEROCOPY Date: Wed, 21 Jun 2017 17:18:12 -0400 Message-Id: <20170621211816.53837-10-willemdebruijn.kernel@gmail.com> X-Mailer: git-send-email 2.13.1.611.g7e3b11ae1-goog In-Reply-To: <20170621211816.53837-1-willemdebruijn.kernel@gmail.com> References: <20170621211816.53837-1-willemdebruijn.kernel@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Willem de Bruijn Enable support for MSG_ZEROCOPY to the TCP stack. TSO and GSO are both supported. Only data sent to remote destinations is sent without copying. Packets looped onto a local destination have their payload copied to avoid unbounded latency. Tested: A 10x TCP_STREAM between two hosts showed a reduction in netserver process cycles by up to 70%, depending on packet size. Systemwide, savings are of course much less pronounced, at up to 20% best case. msg_zerocopy.sh 4 tcp: without zerocopy tx=121792 (7600 MB) txc=0 zc=n rx=60458 (7600 MB) with zerocopy tx=286257 (17863 MB) txc=286257 zc=y rx=140022 (17863 MB) This test opens a pair of sockets over veth, one one calls send with 64KB and optionally MSG_ZEROCOPY and on the other reads the initial bytes. The receiver truncates, so this is strictly an upper bound on what is achievable. It is more representative of sending data out of a physical NIC (when payload is not touched, either). Signed-off-by: Willem de Bruijn --- net/ipv4/tcp.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 058f509ca98e..02a91ac8d71b 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1149,6 +1149,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) struct tcp_sock *tp = tcp_sk(sk); struct sk_buff *skb; struct sockcm_cookie sockc; + struct ubuf_info *uarg = NULL; int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; bool process_backlog = false; @@ -1158,6 +1159,26 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) lock_sock(sk); flags = msg->msg_flags; + + if (flags & MSG_ZEROCOPY && size) { + if (sk->sk_state != TCP_ESTABLISHED) { + err = -EINVAL; + goto out_err; + } + + skb = tcp_send_head(sk) ? tcp_write_queue_tail(sk) : NULL; + uarg = sock_zerocopy_realloc(sk, size, skb_zcopy(skb)); + if (!uarg) { + err = -ENOBUFS; + goto out_err; + } + + /* skb may be freed in main loop, keep extra ref on uarg */ + sock_zerocopy_get(uarg); + if (!(sk_check_csum_caps(sk) && sk->sk_route_caps & NETIF_F_SG)) + uarg->zerocopy = 0; + } + if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect)) { err = tcp_sendmsg_fastopen(sk, msg, &copied_syn, size); if (err == -EINPROGRESS && copied_syn > 0) @@ -1281,7 +1302,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) err = skb_add_data_nocache(sk, skb, &msg->msg_iter, copy); if (err) goto do_fault; - } else { + } else if (!uarg || !uarg->zerocopy) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1319,6 +1340,13 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; + } else { + err = skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg); + if (err == -EMSGSIZE || err == -EEXIST) + goto new_segment; + if (err < 0) + goto do_error; + copy = err; } if (!copied) @@ -1365,6 +1393,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); } out_nopush: + sock_zerocopy_put(uarg); release_sock(sk); return copied + copied_syn; @@ -1382,6 +1411,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) if (copied + copied_syn) goto out; out_err: + sock_zerocopy_put_abort(uarg); err = sk_stream_error(sk, flags, err); /* make sure we wake any epoll edge trigger waiter */ if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 &&