From patchwork Fri Mar 20 23:29:43 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jarek Poplawski X-Patchwork-Id: 24783 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by ozlabs.org (Postfix) with ESMTP id CB686DDE0E for ; Sat, 21 Mar 2009 10:31:22 +1100 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1763466AbZCTXaT (ORCPT ); Fri, 20 Mar 2009 19:30:19 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1763460AbZCTXaS (ORCPT ); Fri, 20 Mar 2009 19:30:18 -0400 Received: from fk-out-0910.google.com ([209.85.128.191]:51431 "EHLO fk-out-0910.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1763429AbZCTXaN (ORCPT ); Fri, 20 Mar 2009 19:30:13 -0400 Received: by fk-out-0910.google.com with SMTP id 18so481378fkq.5 for ; Fri, 20 Mar 2009 16:30:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject :message-id:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=gtLrYz3YY7IVSbmfEpDxE6knCGLr+gDOG6+P8Kv6ERA=; b=mzkxMUd5XFovWccu0f01ecr59Yva8jvdI9ydyaKUJBMwa4heQHZvaZswYezONHBBrb wHLyBN24q+1yfjk//yCDUByYQz29Sc0nuNCkvLlZMthOPkXS3F8vnUUeZ8VBVwVi0x7d qrIm5PnKYWVsp+UHxfGwtCRnxpUI8GP91Oar8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:mime-version:content-type :content-disposition:in-reply-to:user-agent; b=cB28K+wvcTl4TqpttVST3pEl04w1GGB1zH8li+tH272IreXdJICRS8XCkKKrxrjGXk aHztFIP+G01VDrHnXq8oG7QsarNmo8NwhCoegtCWUvqHHN60cDUxPtlfoyED2oSEiJ6C fIdqUYH58CKRh5ucbUUnmEg2cBa9tlhdd5LE8= Received: by 10.223.108.74 with SMTP id e10mr3745168fap.35.1237591809207; Fri, 20 Mar 2009 16:30:09 -0700 (PDT) Received: from ami.dom.local ([79.162.190.84]) by mx.google.com with ESMTPS id 12sm4750222fks.5.2009.03.20.16.30.07 (version=SSLv3 cipher=RC4-MD5); Fri, 20 Mar 2009 16:30:08 -0700 (PDT) Date: Sat, 21 Mar 2009 00:29:43 +0100 From: Jarek Poplawski To: Vernon Mauery Cc: Eric Dumazet , netdev , LKML , rt-users Subject: Re: High contention on the sk_buff_head.lock Message-ID: <20090320232943.GA3024@ami.dom.local> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <49C156E8.1090306@us.ibm.com> User-Agent: Mutt/1.5.18 (2008-05-17) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Vernon Mauery wrote, On 03/18/2009 09:17 PM: ... > This patch does seem to reduce the number of contentions by about 10%. That is > a good start (and a good catch on the cacheline bounces). But, like I mentioned > above, this lock still has 2 orders of magnitude greater contention than the > next lock, so even a large decrease like 10% makes little difference in the > overall contention characteristics. > > So we will have to do something more. Whether it needs to be more complex or > not is still up in the air. Batched enqueueing/dequeueing are just two options > and the former would be a *lot* less complex than the latter. > > If anyone else has any ideas they have been holding back, now would be a great > time to get them out in the open. I think there would be interesting to check another idea around this contention: not all contenders are equal here. One thread is doing qdisc_run() and owning the transmit queue (even after releasing the TX lock). So if it waits for the qdisc lock the NIC, if not multiqueue, is idle. Probably some handicap like in the patch below could make some difference in throughput; alas I didn't test it. Jarek P. --- net/core/dev.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/core/dev.c b/net/core/dev.c index f112970..d5ad808 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1852,7 +1852,11 @@ gso: if (q->enqueue) { spinlock_t *root_lock = qdisc_lock(q); - spin_lock(root_lock); + while (!spin_trylock(root_lock)) { + do { + cpu_relax(); + } while (spin_is_locked(root_lock)); + } if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) { kfree_skb(skb);