From patchwork Wed Mar 15 09:25:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirsher, Jeffrey T" X-Patchwork-Id: 739065 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3vjmSp5M26z9s0g for ; Wed, 15 Mar 2017 20:26:34 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="key not found in DNS" (0-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="oTg2311w"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752674AbdCOJ02 (ORCPT ); Wed, 15 Mar 2017 05:26:28 -0400 Received: from mga11.intel.com ([192.55.52.93]:44457 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752400AbdCOJZ5 (ORCPT ); Wed, 15 Mar 2017 05:25:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1489569956; x=1521105956; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bQhkn2FOrqCGcQEwKR91rbJRMooLSwZ+4is6Y1bIRHE=; b=oTg2311wI7bUcGIuXQcruWDmvhyrzte+LsVmegWwxR+TkKnWb8nfxFXw Q2MB93FukxrcnJi/27Yw9vXPASubBQ==; Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Mar 2017 02:25:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,168,1486454400"; d="scan'208";a="236349799" Received: from jshanks-mobl5.amr.corp.intel.com (HELO jtkirshe-DESK.amr.corp.intel.com.com) ([10.254.185.115]) by fmsmga004.fm.intel.com with ESMTP; 15 Mar 2017 02:25:55 -0700 From: Jeff Kirsher To: davem@davemloft.net Cc: Jacob Keller , netdev@vger.kernel.org, nhorman@redhat.com, sassmann@redhat.com, jogreene@redhat.com, Harshitha Ramamurthy , Carolyn Wyborny , Jeff Kirsher Subject: [net-next v2 05/12] i40e: don't add more vectors to num_lan_msix than number of CPUs Date: Wed, 15 Mar 2017 02:25:38 -0700 Message-Id: <20170315092545.63654-6-jeffrey.t.kirsher@intel.com> X-Mailer: git-send-email 2.12.0 In-Reply-To: <20170315092545.63654-1-jeffrey.t.kirsher@intel.com> References: <20170315092545.63654-1-jeffrey.t.kirsher@intel.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Jacob Keller This is a solution to avoid adding too many queues to num_lan_msix. A recent refactor of queue pairs accidentally added all remaining vectors to the num_lan_msix which can have adverse performance issues, due to enabling more queues than the number of CPU cores. This patch removes the old calculation, and replaces it with a simple algorithm. 1) add queue pairs up to num_online_cpus(), but capped at half of total vectors 2) then add alternative features such as flow directory and similar 3) finally, add the remaining vectors back to queue pairs, but capped such that the total number of queue pairs does not exceed num_online_cpus(). Change-ID: I668abf67d5011a1248866daba8885f4ff00cb8d9 Signed-off-by: Jacob Keller Signed-off-by: Harshitha Ramamurthy Signed-off-by: Carolyn Wyborny Tested-by: Andrew Bowers Signed-off-by: Jeff Kirsher --- drivers/net/ethernet/intel/i40e/i40e_main.c | 30 ++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 3d7f179af6be..cb678ed7a2ad 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -7819,6 +7819,7 @@ static int i40e_reserve_msix_vectors(struct i40e_pf *pf, int vectors) static int i40e_init_msix(struct i40e_pf *pf) { struct i40e_hw *hw = &pf->hw; + int cpus, extra_vectors; int vectors_left; int v_budget, i; int v_actual; @@ -7854,10 +7855,16 @@ static int i40e_init_msix(struct i40e_pf *pf) vectors_left--; } - /* reserve vectors for the main PF traffic queues */ - pf->num_lan_msix = min_t(int, num_online_cpus(), vectors_left); + /* reserve some vectors for the main PF traffic queues. Initially we + * only reserve at most 50% of the available vectors, in the case that + * the number of online CPUs is large. This ensures that we can enable + * extra features as well. Once we've enabled the other features, we + * will use any remaining vectors to reach as close as we can to the + * number of online CPUs. + */ + cpus = num_online_cpus(); + pf->num_lan_msix = min_t(int, cpus, vectors_left / 2); vectors_left -= pf->num_lan_msix; - v_budget += pf->num_lan_msix; /* reserve one vector for sideband flow director */ if (pf->flags & I40E_FLAG_FD_SB_ENABLED) { @@ -7920,6 +7927,23 @@ static int i40e_init_msix(struct i40e_pf *pf) } } + /* On systems with a large number of SMP cores, we previously limited + * the number of vectors for num_lan_msix to be at most 50% of the + * available vectors, to allow for other features. Now, we add back + * the remaining vectors. However, we ensure that the total + * num_lan_msix will not exceed num_online_cpus(). To do this, we + * calculate the number of vectors we can add without going over the + * cap of CPUs. For systems with a small number of CPUs this will be + * zero. + */ + extra_vectors = min_t(int, cpus - pf->num_lan_msix, vectors_left); + pf->num_lan_msix += extra_vectors; + vectors_left -= extra_vectors; + + WARN(vectors_left < 0, + "Calculation of remaining vectors underflowed. This is an accounting bug when determining total MSI-X vectors.\n"); + + v_budget += pf->num_lan_msix; pf->msix_entries = kcalloc(v_budget, sizeof(struct msix_entry), GFP_KERNEL); if (!pf->msix_entries)