From patchwork Thu Apr 26 17:42:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Willem de Bruijn X-Patchwork-Id: 905282 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="dDGt7eUb"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40X4CN0P0Sz9ry1 for ; Fri, 27 Apr 2018 03:42:40 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932240AbeDZRmi (ORCPT ); Thu, 26 Apr 2018 13:42:38 -0400 Received: from mail-qk0-f196.google.com ([209.85.220.196]:46025 "EHLO mail-qk0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932120AbeDZRmd (ORCPT ); Thu, 26 Apr 2018 13:42:33 -0400 Received: by mail-qk0-f196.google.com with SMTP id x22so8427180qkb.12 for ; Thu, 26 Apr 2018 10:42:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9xyO/PLneMT3TJwQoecAhs9L3I0k6Ixm6+eCSxmGGDc=; b=dDGt7eUbjBy90b8U3DpgSs6CL/hGPhUaiEi6S38e988Ix/Fc+OybXULbsX0iPvGMC8 kmLZBHuNg0G86V6xQdRRrbCyvdzqNFaLAsg8CjOLKTBjI0yAegjRi0AQAl/XMqMfHGUi YuqQbXtGSr9uKhZS4LDoOsGSP2RkA9/t5G9W8J0Ybt5DFIeHeMlaL6KNg5ncKyuY5bYy t4jB5MIEkuwYGHpKcQj9grglAGbnEZJ6lkKCafbRjptDNWvb9VkCIPe7iIZMmjBSgvc7 n/klPRIraWJ7taZxutk4V49+h9XeWu57VPsg55SFcCpa3Cet46eGuDvYtnGAPl6I7OSH pUIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9xyO/PLneMT3TJwQoecAhs9L3I0k6Ixm6+eCSxmGGDc=; b=I+EQO/HMDVSgx3sFduqByfkHvVQnDlKSg33cU2wYdr+7Y2c4hly0Gjs85JwO3CvNKM Pb+ropgE7N8721+EpXYfGqKb+FLJn5wF8EXjc5AHORlKbmyRCru26J6ha/PvmHR7juoV DsAlaqm2/CRcugtHlGE13teZkfOeehQcllRqBDqeOtAyRTQWXeMZgaTF6JJwm2rLi559 TRveGlF+djNjU0Zp+jOMfPBzi1W3tSGz68sO0xVtniVi4BpAE9zXv+XY57TED4QoOSkV yi8R/vq5T6qOkrmpLIhstkXrCYiXNzkFdcwAoOIYbYeAuSB1V5kChe46zx3V1LhuFRRR yc3g== X-Gm-Message-State: ALQs6tC/UK4PNjSSXlktEOTrU6poaGyC/dGgeMFTvF1nWPlRRa7reNQ1 IwuEM2iuSKiwKynLhLGElPPOPXHq X-Google-Smtp-Source: AB8JxZrvUS+5RKiTYLP7lwZnYopXzY8XLEmTbfP/SqBOjWev8nQNdkHboAEw4sURDxtt6ziMx0P7nw== X-Received: by 10.55.11.65 with SMTP id 62mr35712713qkl.287.1524764551063; Thu, 26 Apr 2018 10:42:31 -0700 (PDT) Received: from willemb1.nyc.corp.google.com ([2620:0:1003:315:3fa1:a34c:1128:1d39]) by smtp.gmail.com with ESMTPSA id o14-v6sm12927515qta.23.2018.04.26.10.42.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Apr 2018 10:42:30 -0700 (PDT) From: Willem de Bruijn To: netdev@vger.kernel.org Cc: davem@davemloft.net, alexander.duyck@gmail.com, Willem de Bruijn Subject: [PATCH net-next v2 05/11] udp: paged allocation with gso Date: Thu, 26 Apr 2018 13:42:19 -0400 Message-Id: <20180426174225.246388-6-willemdebruijn.kernel@gmail.com> X-Mailer: git-send-email 2.17.0.484.g0c8726318c-goog In-Reply-To: <20180426174225.246388-1-willemdebruijn.kernel@gmail.com> References: <20180426174225.246388-1-willemdebruijn.kernel@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Willem de Bruijn When sending large datagrams that are later segmented, store data in page frags to avoid copying from linear in skb_segment. Signed-off-by: Willem de Bruijn --- net/ipv4/ip_output.c | 15 +++++++++++---- net/ipv6/ip6_output.c | 19 ++++++++++++++----- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index da4abbee10f7..f2338e40c37d 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -878,11 +878,13 @@ static int __ip_append_data(struct sock *sk, struct rtable *rt = (struct rtable *)cork->dst; unsigned int wmem_alloc_delta = 0; u32 tskey = 0; + bool paged; skb = skb_peek_tail(queue); exthdrlen = !skb ? rt->dst.header_len : 0; mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; + paged = !!cork->gso_size; if (cork->tx_flags & SKBTX_ANY_SW_TSTAMP && sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) @@ -934,6 +936,7 @@ static int __ip_append_data(struct sock *sk, unsigned int fraglen; unsigned int fraggap; unsigned int alloclen; + unsigned int pagedlen = 0; struct sk_buff *skb_prev; alloc_new_skb: skb_prev = skb; @@ -954,8 +957,12 @@ static int __ip_append_data(struct sock *sk, if ((flags & MSG_MORE) && !(rt->dst.dev->features&NETIF_F_SG)) alloclen = mtu; - else + else if (!paged) alloclen = fraglen; + else { + alloclen = min_t(int, fraglen, MAX_HEADER); + pagedlen = fraglen - alloclen; + } alloclen += exthdrlen; @@ -999,7 +1006,7 @@ static int __ip_append_data(struct sock *sk, /* * Find where to start putting bytes. */ - data = skb_put(skb, fraglen + exthdrlen); + data = skb_put(skb, fraglen + exthdrlen - pagedlen); skb_set_network_header(skb, exthdrlen); skb->transport_header = (skb->network_header + fragheaderlen); @@ -1015,7 +1022,7 @@ static int __ip_append_data(struct sock *sk, pskb_trim_unique(skb_prev, maxfraglen); } - copy = datalen - transhdrlen - fraggap; + copy = datalen - transhdrlen - fraggap - pagedlen; if (copy > 0 && getfrag(from, data + transhdrlen, offset, copy, fraggap, skb) < 0) { err = -EFAULT; kfree_skb(skb); @@ -1023,7 +1030,7 @@ static int __ip_append_data(struct sock *sk, } offset += copy; - length -= datalen - fraggap; + length -= copy + transhdrlen; transhdrlen = 0; exthdrlen = 0; csummode = CHECKSUM_NONE; diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index a1c4a78132d2..dfd8af41824e 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1276,6 +1276,7 @@ static int __ip6_append_data(struct sock *sk, int csummode = CHECKSUM_NONE; unsigned int maxnonfragsize, headersize; unsigned int wmem_alloc_delta = 0; + bool paged; skb = skb_peek_tail(queue); if (!skb) { @@ -1283,6 +1284,7 @@ static int __ip6_append_data(struct sock *sk, dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len; } + paged = !!cork->gso_size; mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize; orig_mtu = mtu; @@ -1374,6 +1376,7 @@ static int __ip6_append_data(struct sock *sk, unsigned int fraglen; unsigned int fraggap; unsigned int alloclen; + unsigned int pagedlen = 0; alloc_new_skb: /* There's no room in the current skb */ if (skb) @@ -1396,11 +1399,17 @@ static int __ip6_append_data(struct sock *sk, if (datalen > (cork->length <= mtu && !(cork->flags & IPCORK_ALLFRAG) ? mtu : maxfraglen) - fragheaderlen) datalen = maxfraglen - fragheaderlen - rt->dst.trailer_len; + fraglen = datalen + fragheaderlen; + if ((flags & MSG_MORE) && !(rt->dst.dev->features&NETIF_F_SG)) alloclen = mtu; - else - alloclen = datalen + fragheaderlen; + else if (!paged) + alloclen = fraglen; + else { + alloclen = min_t(int, fraglen, MAX_HEADER); + pagedlen = fraglen - alloclen; + } alloclen += dst_exthdrlen; @@ -1422,7 +1431,7 @@ static int __ip6_append_data(struct sock *sk, */ alloclen += sizeof(struct frag_hdr); - copy = datalen - transhdrlen - fraggap; + copy = datalen - transhdrlen - fraggap - pagedlen; if (copy < 0) { err = -EINVAL; goto error; @@ -1461,7 +1470,7 @@ static int __ip6_append_data(struct sock *sk, /* * Find where to start putting bytes */ - data = skb_put(skb, fraglen); + data = skb_put(skb, fraglen - pagedlen); skb_set_network_header(skb, exthdrlen); data += fragheaderlen; skb->transport_header = (skb->network_header + @@ -1484,7 +1493,7 @@ static int __ip6_append_data(struct sock *sk, } offset += copy; - length -= datalen - fraggap; + length -= copy + transhdrlen; transhdrlen = 0; exthdrlen = 0; dst_exthdrlen = 0;