From patchwork Wed May 14 14:17:48 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 348793 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id A0F2D14008B for ; Thu, 15 May 2014 00:18:27 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753071AbaENOSY (ORCPT ); Wed, 14 May 2014 10:18:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51078 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752525AbaENOSW (ORCPT ); Wed, 14 May 2014 10:18:22 -0400 Received: from int-mx12.intmail.prod.int.phx2.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s4EEHnfK014976 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 May 2014 10:17:50 -0400 Received: from dragon.localdomain (ovpn-116-95.ams2.redhat.com [10.36.116.95]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s4EEHmII018144; Wed, 14 May 2014 10:17:49 -0400 Received: from [127.0.0.1] (localhost [IPv6:::1]) by dragon.localdomain (Postfix) with ESMTP id D3AA7E40748; Wed, 14 May 2014 16:17:48 +0200 (CEST) From: Jesper Dangaard Brouer Subject: [net-next PATCH 2/5] ixgbe: increase default TX ring buffer to 1024 To: Jesper Dangaard Brouer , netdev@vger.kernel.org Cc: Alexander Duyck , Jeff Kirsher , Daniel Borkmann , Florian Westphal , "David S. Miller" , Stephen Hemminger , "Paul E. McKenney" , Robert Olsson , Ben Greear , John Fastabend , danieltt@kth.se, zhouzhouyi@gmail.com Date: Wed, 14 May 2014 16:17:48 +0200 Message-ID: <20140514141748.20309.83121.stgit@dragon> In-Reply-To: <20140514141545.20309.28343.stgit@dragon> References: <20140514141545.20309.28343.stgit@dragon> User-Agent: StGIT/0.14.3 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Using pktgen I'm seeing the ixgbe driver "push-back", due TX ring running full. Thus, the TX ring is artificially limiting pktgen. Diagnose via "ethtool -S", look for "tx_restart_queue" or "tx_busy" counters. Increasing the TX ring buffer should be done carefully, as it comes at a higher memory cost, which can also negatively influence performance. E.g. ring buffer array of struct ixgbe_tx_buffer (current size 48bytes) increase from 512*48=24576bytes to 1024*48=49152bytes which is larger than the L1 data cache (32KB on my E5-2630), thus increasing the L1->L2 cache-references. Adjusting the TX ring buffer (TXSZ) measured over 10 sec with ifpps (single CPU performance, ixgbe 10Gbit/s, E5-2630) * cmd: ethtool -G eth8 tx $TXSZ * 3,930,065 pps -- TXSZ= 512 * 5,312,249 pps -- TXSZ= 768 * 5,362,722 pps -- TXSZ=1024 * 5,361,390 pps -- TXSZ=1536 * 5,362,439 pps -- TXSZ=2048 * 5,359,744 pps -- TXSZ=4096 Choosing size 1024 because for the next optimizations 768 is not enough. Notice after commit 6f25cd47d (pktgen: fix xmit test for BQL enabled devices) pktgen uses netif_xmit_frozen_or_drv_stopped() and ignores the BQL "stack" pause (QUEUE_STATE_STACK_XOFF) flag. This allow us to put more pressure on the TX ring buffers. It is the ixgbe_maybe_stop_tx() call that stops the transmits, and pktgen respecting this in the call to netif_xmit_frozen_or_drv_stopped(txq). Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h index c688c8a..bf078fe 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h @@ -63,7 +63,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt /* TX/RX descriptor defines */ -#define IXGBE_DEFAULT_TXD 512 +#define IXGBE_DEFAULT_TXD 1024 #define IXGBE_DEFAULT_TX_WORK 256 #define IXGBE_MAX_TXD 4096 #define IXGBE_MIN_TXD 64