From patchwork Mon Sep 22 23:50:44 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 392189 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 686B01400EA for ; Tue, 23 Sep 2014 09:51:13 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755353AbaIVXvI (ORCPT ); Mon, 22 Sep 2014 19:51:08 -0400 Received: from mail-oi0-f74.google.com ([209.85.218.74]:62370 "EHLO mail-oi0-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755342AbaIVXvF (ORCPT ); Mon, 22 Sep 2014 19:51:05 -0400 Received: by mail-oi0-f74.google.com with SMTP id u20so1278872oif.1 for ; Mon, 22 Sep 2014 16:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=G53rWdHc8GzBeyoa+rTke0QD3AJ75TFY1ci15/f6g7o=; b=dkpGqlI96IWHblmBhB7wuWgoRihYTrPUBFpqsoWbqKfWyR1d084bC5gFhjqF8nXHNX gzE8KYGW7tvIz7fZd6VK3ZvZKlaYAyndcKrZi5GuKsrwqjv/tV31737JbJFvOZSwwLdt 7+Zhv5pEiDU3lyBqDBJrdNrXMFpilGEaJd14OmEL3gOOubVCodNP+UQ8r8Mgt/XyyCYR lmE/wmVZnc/H3VwqVlBuwpq0vMfcrQ/mQr7CymBBbakZHn1lbjAIlwoV9whlIzmeRsMI rPF51Pxqh7vE3sfWzDVrHVOgZFK4a4bu8M1fDCEt7o15Eabpy6byYvko3SF19DUmIYEn dHZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=G53rWdHc8GzBeyoa+rTke0QD3AJ75TFY1ci15/f6g7o=; b=QuT0frBp89hNXg7MKEL/V9Fon+64iLVREyImKNFHgTgi92Zsw7bvhJPaPxH8gySkOv eWW1ClpgKe8k4WKk+EdVcXHVJAu+REoYKRGuY2sROW2RT5fK9eOMo+Y1jxHO4gGbiGrX UEDkRwaa6Vv9IdK0+WJuh+wgxpb/JF35MsM7BZIXG1JDKhXCnIyapsjWvIIGAPu+n2lH 00Hxs/lx1Ke9tgX4BctjMYUsh/cR0Up2bFIN0f7mnbyrW9UKrHcGqZNCQsSMiitF+Shp C4Gi3wxYKIZ5vtsUtls1jHAJJUA37KVfqno2JWYS/RAuMG9+V3Ziya/rkEYWbm5b7i/v ewFw== X-Gm-Message-State: ALoCoQnnuSraFvdZzPaiyz5pNDOK2dEM6wRkM1JgOYxlRzlGefbJOwHTdl5x1jMsCReY2KEGBm5p X-Received: by 10.182.56.200 with SMTP id c8mr22669568obq.7.1411429864430; Mon, 22 Sep 2014 16:51:04 -0700 (PDT) Received: from corpmail-nozzle1-2.hot.corp.google.com ([100.108.1.103]) by gmr-mx.google.com with ESMTPS id l45si567684yha.2.2014.09.22.16.51.03 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Sep 2014 16:51:04 -0700 (PDT) Received: from manihi.mtv.corp.google.com ([172.17.131.143]) by corpmail-nozzle1-2.hot.corp.google.com with ESMTP id WBfOX9kK.1; Mon, 22 Sep 2014 16:51:04 -0700 Received: by manihi.mtv.corp.google.com (Postfix, from userid 160623) id 0C496A0A1F; Mon, 22 Sep 2014 16:51:03 -0700 (PDT) From: Eric Dumazet To: "David S. Miller" Cc: netdev@vger.kernel.org, Yuchung Cheng , Neal Cardwell , Eric Dumazet Subject: [PATCH net-next 3/3] tcp: better TCP_SKB_CB layout to reduce cache line misses Date: Mon, 22 Sep 2014 16:50:44 -0700 Message-Id: <1411429844-11099-4-git-send-email-edumazet@google.com> X-Mailer: git-send-email 2.1.0.rc2.206.gedb03e5 In-Reply-To: <1411429844-11099-1-git-send-email-edumazet@google.com> References: <1411429844-11099-1-git-send-email-edumazet@google.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org TCP maintains lists of skb in write queue, and in receive queues (in order and out of order queues) Scanning these lists both in input and output path usually requires access to skb->next, TCP_SKB_CB(skb)->seq, and TCP_SKB_CB(skb)->end_seq These fields are currently in two different cache lines, meaning we waste lot of memory bandwidth when these queues are big and flows have either packet drops or packet reorders. We can move TCP_SKB_CB(skb)->header at the end of TCP_SKB_CB, because this header is not used in fast path. This allows TCP to search much faster in the skb lists. Even with regular flows, we save one cache line miss in fast path. Signed-off-by: Eric Dumazet --- include/net/tcp.h | 12 ++++++------ net/ipv4/tcp_ipv4.c | 7 +++++++ net/ipv6/tcp_ipv6.c | 7 +++++++ 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index a4201ef216e8..4dc6641ee990 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -696,12 +696,6 @@ static inline u32 tcp_skb_timestamp(const struct sk_buff *skb) * If this grows please adjust skbuff.h:skbuff->cb[xxx] size appropriately. */ struct tcp_skb_cb { - union { - struct inet_skb_parm h4; -#if IS_ENABLED(CONFIG_IPV6) - struct inet6_skb_parm h6; -#endif - } header; /* For incoming frames */ __u32 seq; /* Starting sequence number */ __u32 end_seq; /* SEQ + FIN + SYN + datalen */ __u32 tcp_tw_isn; /* isn chosen by tcp_timewait_state_process() */ @@ -720,6 +714,12 @@ struct tcp_skb_cb { __u8 ip_dsfield; /* IPv4 tos or IPv6 dsfield */ /* 1 byte hole */ __u32 ack_seq; /* Sequence number ACK'd */ + union { + struct inet_skb_parm h4; +#if IS_ENABLED(CONFIG_IPV6) + struct inet6_skb_parm h6; +#endif + } header; /* For incoming frames */ }; #define TCP_SKB_CB(__skb) ((struct tcp_skb_cb *)&((__skb)->cb[0])) diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 28ab90382c01..70c4a21f6f45 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1636,6 +1636,13 @@ int tcp_v4_rcv(struct sk_buff *skb) th = tcp_hdr(skb); iph = ip_hdr(skb); + /* This is tricky : We move IPCB at its correct location into TCP_SKB_CB() + * barrier() makes sure compiler wont play fool^Waliasing games. + */ + memmove(&TCP_SKB_CB(skb)->header.h4, IPCB(skb), + sizeof(struct inet_skb_parm)); + barrier(); + TCP_SKB_CB(skb)->seq = ntohl(th->seq); TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin + skb->len - th->doff * 4); diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 9400b4326f22..132bac137aed 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -1412,6 +1412,13 @@ static int tcp_v6_rcv(struct sk_buff *skb) th = tcp_hdr(skb); hdr = ipv6_hdr(skb); + /* This is tricky : We move IPCB at its correct location into TCP_SKB_CB() + * barrier() makes sure compiler wont play fool^Waliasing games. + */ + memmove(&TCP_SKB_CB(skb)->header.h6, IP6CB(skb), + sizeof(struct inet6_skb_parm)); + barrier(); + TCP_SKB_CB(skb)->seq = ntohl(th->seq); TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin + skb->len - th->doff*4);