From patchwork Tue Aug 7 12:19:56 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 175613 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id EB23C2C0086 for ; Tue, 7 Aug 2012 22:20:07 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753801Ab2HGMUE (ORCPT ); Tue, 7 Aug 2012 08:20:04 -0400 Received: from mail-bk0-f46.google.com ([209.85.214.46]:40574 "EHLO mail-bk0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752548Ab2HGMUC (ORCPT ); Tue, 7 Aug 2012 08:20:02 -0400 Received: by bkwj10 with SMTP id j10so1419029bkw.19 for ; Tue, 07 Aug 2012 05:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:from:to:cc:content-type:date:message-id:mime-version :x-mailer:content-transfer-encoding; bh=FbM80HjkxyfnLtkZvtA/qQx/jLFfVkXoWgpmD1Aj764=; b=Gz3cGwu/tMp8G4xr0l+7wHe/12VdnQmcLMc+/qrxhKVjThrn92QISNL/5ACeeAPBQ9 L+He+2l8A0t/PCOYGOxJ4fg7Bt3brES3SwVC5RXGGWjQtlbDclnsqEgtzURcoUHhZI2h 60ddB2PWm9XEA8AD6tk5AD2vAhi7VOM2eVTDgAFI2WbWJGir4SOUPIt2Af3KIwrZG/2A PwotKRm1kgKiM5YDqbdBO1SkcFx+ln3evPJnRr6DIYKaqdiK9sLPH6Wam8Os7YN/Ceq/ oOqV41WjVCEPAmEGCvsvV6RIvz2DofqqEk6/+YgEjHCWdp/G8P/HTd96lhXQ9Jx5CKPQ x9jg== Received: by 10.204.148.66 with SMTP id o2mr5540014bkv.43.1344342001245; Tue, 07 Aug 2012 05:20:01 -0700 (PDT) Received: from [172.28.91.2] ([74.125.122.49]) by mx.google.com with ESMTPS id ht18sm8774941bkc.16.2012.08.07.05.19.58 (version=SSLv3 cipher=OTHER); Tue, 07 Aug 2012 05:19:59 -0700 (PDT) Subject: [PATCH net-next] net: output path optimizations From: Eric Dumazet To: David Miller Cc: netdev Date: Tue, 07 Aug 2012 14:19:56 +0200 Message-ID: <1344341996.28967.87.camel@edumazet-glaptop> Mime-Version: 1.0 X-Mailer: Evolution 2.28.3 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Eric Dumazet 1) Avoid dirtying neighbour's confirmed field. TCP workloads hits this cache line for each incoming ACK. Lets write n->confirmed only if there is a jiffie change. 2) Optimize neigh_hh_output() for the common Ethernet case, were hh_len is less than 16 bytes. Replace the memcpy() call by two inlined 64bit load/stores on x86_64. Bench results using udpflood test, with -C option (MSG_CONFIRM flag added to sendto(), to reproduce the n->confirmed dirtying on UDP) 24 threads doing 1.000.000 UDP sendto() on dummy device, 4 runs. before : 2.247s, 2.235s, 2.247s, 2.318s after : 1.884s, 1.905s, 1.891s, 1.895s Signed-off-by: Eric Dumazet --- include/net/dst.h | 10 +++++++--- include/net/neighbour.h | 14 +++++++++----- 2 files changed, 16 insertions(+), 8 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/net/dst.h b/include/net/dst.h index baf5978..77f52f7 100644 --- a/include/net/dst.h +++ b/include/net/dst.h @@ -396,11 +396,15 @@ static inline void dst_confirm(struct dst_entry *dst) static inline int dst_neigh_output(struct dst_entry *dst, struct neighbour *n, struct sk_buff *skb) { - struct hh_cache *hh; + const struct hh_cache *hh; + + if (dst->pending_confirm) { + unsigned long now = jiffies; - if (unlikely(dst->pending_confirm)) { - n->confirmed = jiffies; dst->pending_confirm = 0; + /* avoid dirtying neighbour */ + if (n->confirmed != now) + n->confirmed = now; } hh = &n->hh; diff --git a/include/net/neighbour.h b/include/net/neighbour.h index 344d898..0dab173 100644 --- a/include/net/neighbour.h +++ b/include/net/neighbour.h @@ -334,18 +334,22 @@ static inline int neigh_hh_bridge(struct hh_cache *hh, struct sk_buff *skb) } #endif -static inline int neigh_hh_output(struct hh_cache *hh, struct sk_buff *skb) +static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb) { unsigned int seq; int hh_len; do { - int hh_alen; - seq = read_seqbegin(&hh->hh_lock); hh_len = hh->hh_len; - hh_alen = HH_DATA_ALIGN(hh_len); - memcpy(skb->data - hh_alen, hh->hh_data, hh_alen); + if (likely(hh_len <= HH_DATA_MOD)) { + /* this is inlined by gcc */ + memcpy(skb->data - HH_DATA_MOD, hh->hh_data, HH_DATA_MOD); + } else { + int hh_alen = HH_DATA_ALIGN(hh_len); + + memcpy(skb->data - hh_alen, hh->hh_data, hh_alen); + } } while (read_seqretry(&hh->hh_lock, seq)); skb_push(skb, hh_len);