From patchwork Sat Mar 31 20:16:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 893884 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="KsH0hdnS"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40D8s91rRGz9s3B for ; Sun, 1 Apr 2018 06:16:45 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751434AbeCaUQj (ORCPT ); Sat, 31 Mar 2018 16:16:39 -0400 Received: from mail-pf0-f194.google.com ([209.85.192.194]:43278 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750903AbeCaUQg (ORCPT ); Sat, 31 Mar 2018 16:16:36 -0400 Received: by mail-pf0-f194.google.com with SMTP id j2so7477923pff.10 for ; Sat, 31 Mar 2018 13:16:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=28XY1b0Pp+O2GAyskqKT17K+NOJxE+5nRZL6VmesUdU=; b=KsH0hdnSz2He7ahn/yWMv7CKN6xAnOPoM9Agqj3+qHkKsGOScuHx+pr1h1kWgBqVGJ YG5lbKm0dwWXyM7fIc2JR8qc3mzk4erxKgMwnj0Q3/D7n4ivfmqiF8eJkt78GS3mOghu Rkeidq9ogYanFH2MdTEUrxbDguOe3jSYd9yn9lOndG13AgOoO8E6N37hk86tb8hUE/hL UfIycx0iNPp9zv+81OWDPPg11n13SZ1cWeesSdbF+Q2DGx3VMCn8sPTfzYvHMwSfb+ER zxUaR5JEX241WzRlxPSnE+AOZFC3+Vq0nTvJOipPkkB4X9rd/hTa5zycIDXlMt0IPwJT o8xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=28XY1b0Pp+O2GAyskqKT17K+NOJxE+5nRZL6VmesUdU=; b=V+OWd8jXHAA7nj80PQeWUFxDU+jE8ml0dPiH8IflrlORE5J7AEa65uTojPdYvOcJ1A V99DEuyd9Kk25Zzt0kfARiOtT0MTWl+HTO+eaRFNCmZEsVia9j4d68ap9107XpXSVD6Q BK8AmF16pRzwDpnDhUxzlRw/1jeUe2Vh12G8fr8yww0CXU8SJyjykwVc05ElOdxNr14N +kInGSaJXIjb1p7gs3wrtVnnpr89R63Pd0fnGuC7cHrs2Ms3v1ejbuoglRa04BE/PiAW OLKmpXH02pqXQY+id7HRuOp19OKLSF4yEwyLKos1nPDDJW52aG/2BtzyTOA6oIRBYJO3 o7bQ== X-Gm-Message-State: AElRT7EKrxOlDZbc/fOdfBNEnDCzHD020+qU0N1UYsi0zcUtBqEQ+3bF o3FBdyl86tpadmKQlAV1wrJ/cg== X-Google-Smtp-Source: AIpwx488uLsyKwVB17lZnBHuJjnts0sX/rQinnctHvR+6GNLMd15K4gN3aJshsH4y3eY9hcimTcajg== X-Received: by 2002:a17:902:728f:: with SMTP id d15-v6mr3995589pll.227.1522527395947; Sat, 31 Mar 2018 13:16:35 -0700 (PDT) Received: from localhost ([2620:15c:2c4:201:f5a:7eca:440a:3ead]) by smtp.gmail.com with ESMTPSA id o66sm24248504pfg.90.2018.03.31.13.16.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 31 Mar 2018 13:16:34 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 1/2] ipv4: factorize sk_wmem_alloc updates done by __ip_append_data() Date: Sat, 31 Mar 2018 13:16:25 -0700 Message-Id: <20180331201626.186789-2-edumazet@google.com> X-Mailer: git-send-email 2.17.0.rc1.321.gba9d0f2565-goog In-Reply-To: <20180331201626.186789-1-edumazet@google.com> References: <20180331201626.186789-1-edumazet@google.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org While testing my inet defrag changes, I found that the senders could spend ~20% of cpu cycles in skb_set_owner_w() updating sk->sk_wmem_alloc for every fragment they cook. The solution to this problem is to use alloc_skb() instead of sock_wmalloc() and manually perform a single sk_wmem_alloc change. Similar change for IPv6 is provided in following patch. Signed-off-by: Eric Dumazet --- net/ipv4/ip_output.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 66340ab750e69ff5775f7996192839a24ddc6e65..94cacae76aca41e6e7feb7575c7999a414145c49 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -876,6 +876,7 @@ static int __ip_append_data(struct sock *sk, unsigned int maxfraglen, fragheaderlen, maxnonfragsize; int csummode = CHECKSUM_NONE; struct rtable *rt = (struct rtable *)cork->dst; + unsigned int wmem_alloc_delta = 0; u32 tskey = 0; skb = skb_peek_tail(queue); @@ -971,11 +972,10 @@ static int __ip_append_data(struct sock *sk, (flags & MSG_DONTWAIT), &err); } else { skb = NULL; - if (refcount_read(&sk->sk_wmem_alloc) <= + if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <= 2 * sk->sk_sndbuf) - skb = sock_wmalloc(sk, - alloclen + hh_len + 15, 1, - sk->sk_allocation); + skb = alloc_skb(alloclen + hh_len + 15, + sk->sk_allocation); if (unlikely(!skb)) err = -ENOBUFS; } @@ -1033,6 +1033,11 @@ static int __ip_append_data(struct sock *sk, /* * Put the packet on the pending queue. */ + if (!skb->destructor) { + skb->destructor = sock_wfree; + skb->sk = sk; + wmem_alloc_delta += skb->truesize; + } __skb_queue_tail(queue, skb); continue; } @@ -1079,12 +1084,13 @@ static int __ip_append_data(struct sock *sk, skb->len += copy; skb->data_len += copy; skb->truesize += copy; - refcount_add(copy, &sk->sk_wmem_alloc); + wmem_alloc_delta += copy; } offset += copy; length -= copy; } + refcount_add(wmem_alloc_delta, &sk->sk_wmem_alloc); return 0; error_efault: @@ -1092,6 +1098,7 @@ static int __ip_append_data(struct sock *sk, error: cork->length -= length; IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); + refcount_add(wmem_alloc_delta, &sk->sk_wmem_alloc); return err; } From patchwork Sat Mar 31 20:16:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 893885 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="LjDx4ncE"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40D8sB5837z9s3K for ; Sun, 1 Apr 2018 06:16:46 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751787AbeCaUQm (ORCPT ); Sat, 31 Mar 2018 16:16:42 -0400 Received: from mail-pf0-f194.google.com ([209.85.192.194]:34496 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751606AbeCaUQj (ORCPT ); Sat, 31 Mar 2018 16:16:39 -0400 Received: by mail-pf0-f194.google.com with SMTP id q9so7497385pff.1 for ; Sat, 31 Mar 2018 13:16:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=D6e1e6ORk/a+uD0CQok3LfN1P73mbRn+dldSytMpLHo=; b=LjDx4ncENtQz512/eSGFZ2QwTb/8k5MDFcZZv0QhvwhC8NEXhvpZvEv4Ga2EI2dNx5 2l6qfTzv/ebPl8eRFCnWgWJ1KQOyFoUe/z5k3Kmm4Xvu+MzRLEpOUeP0ILv3gbhdNDhg uCrygIK5ECZcKU8d9wdfhX2T9bi2sWT1tSECPhXiNMLfB9276GjrJdS88O32Ua5bIWmn R/O6i6C57doFeOhtuvfnOPaZyfgyQ4u9vuLbfcytyBRmlSysN5N8NoiMxGB+EqoiDLnq 8UMrkjTL66CVXP6QM+spRDotlDiAGFJANG5FHo80wsJVu41ZRZJo4tTyt5guYFqs9Zln joyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=D6e1e6ORk/a+uD0CQok3LfN1P73mbRn+dldSytMpLHo=; b=nfQr9rPpXOGwRGCJ1I97yYz5LXAysC9mzeTELDXVw9L5NgkCBkbhsDkT/Y1/goYHaF g7NOaABdpF+dsuW4m3LK4/phbb5EIzdE9l15kVB9a3TqrNzgASgEjXkFDaSif7QOvNcf +C4879mBMkMFq53Qe6f/WVCiC7TA0hspEQFFghOYpDLXhwKhlxShIsJeaKXCBXeVmQ/n 5h5OHK2Ur5SNaIKulYlIrdPLCKtELWTg3ZOK2odK8gyvqQWuVKZ06VcraOaIHLo8dz9G VGdSboYTLzrKqpJBZvEQU2EG2k/VO+E9JghuQAGRXQFIAYKf3icrtNSZ1y2q7GygJjp9 HuRA== X-Gm-Message-State: AElRT7EKtouHq4UEQdRmdaYh017w/XMatFp1f7WhQmoM4yUGMX05ey7U zRI3wl2EPNUSyPX8Jbg7L0QIpKE7Wf4= X-Google-Smtp-Source: AIpwx48cyij+lGQfJqeDb8PyZrAqnNZX1aO8WeMaTj7wLSpGLdwtaslFHIs40m+qYaJlMkT2Jl+p6g== X-Received: by 2002:a17:902:8f97:: with SMTP id z23-v6mr4152033plo.162.1522527398673; Sat, 31 Mar 2018 13:16:38 -0700 (PDT) Received: from localhost ([2620:15c:2c4:201:f5a:7eca:440a:3ead]) by smtp.gmail.com with ESMTPSA id v18sm19244953pgb.23.2018.03.31.13.16.37 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 31 Mar 2018 13:16:37 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 2/2] ipv6: factorize sk_wmem_alloc updates done by __ip6_append_data() Date: Sat, 31 Mar 2018 13:16:26 -0700 Message-Id: <20180331201626.186789-3-edumazet@google.com> X-Mailer: git-send-email 2.17.0.rc1.321.gba9d0f2565-goog In-Reply-To: <20180331201626.186789-1-edumazet@google.com> References: <20180331201626.186789-1-edumazet@google.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org While testing my inet defrag changes, I found that the senders could spend ~20% of cpu cycles in skb_set_owner_w() updating sk->sk_wmem_alloc for every fragment they cook, competing with TX completion of prior skbs possibly happening on another cpus. The solution to this problem is to use alloc_skb() instead of sock_wmalloc() and manually perform a single sk_wmem_alloc change. Signed-off-by: Eric Dumazet --- net/ipv6/ip6_output.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 2c7f09c3c39ed8a1e85a967e105ff3cc30dce5b9..323d7a354ffb6f75e2a948dea63a8018ed0e057f 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1259,6 +1259,7 @@ static int __ip6_append_data(struct sock *sk, struct ipv6_txoptions *opt = v6_cork->opt; int csummode = CHECKSUM_NONE; unsigned int maxnonfragsize, headersize; + unsigned int wmem_alloc_delta = 0; skb = skb_peek_tail(queue); if (!skb) { @@ -1411,11 +1412,10 @@ static int __ip6_append_data(struct sock *sk, (flags & MSG_DONTWAIT), &err); } else { skb = NULL; - if (refcount_read(&sk->sk_wmem_alloc) <= + if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <= 2 * sk->sk_sndbuf) - skb = sock_wmalloc(sk, - alloclen + hh_len, 1, - sk->sk_allocation); + skb = alloc_skb(alloclen + hh_len, + sk->sk_allocation); if (unlikely(!skb)) err = -ENOBUFS; } @@ -1474,6 +1474,11 @@ static int __ip6_append_data(struct sock *sk, /* * Put the packet on the pending queue */ + if (!skb->destructor) { + skb->destructor = sock_wfree; + skb->sk = sk; + wmem_alloc_delta += skb->truesize; + } __skb_queue_tail(queue, skb); continue; } @@ -1520,12 +1525,13 @@ static int __ip6_append_data(struct sock *sk, skb->len += copy; skb->data_len += copy; skb->truesize += copy; - refcount_add(copy, &sk->sk_wmem_alloc); + wmem_alloc_delta += copy; } offset += copy; length -= copy; } + refcount_add(wmem_alloc_delta, &sk->sk_wmem_alloc); return 0; error_efault: @@ -1533,6 +1539,7 @@ static int __ip6_append_data(struct sock *sk, error: cork->length -= length; IP6_INC_STATS(sock_net(sk), rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS); + refcount_add(wmem_alloc_delta, &sk->sk_wmem_alloc); return err; }