From patchwork Tue Apr 16 17:30:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anirudh Venkataramanan X-Patchwork-Id: 1086481 X-Patchwork-Delegate: jeffrey.t.kirsher@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44kC8n601Zz9s3l for ; Wed, 17 Apr 2019 03:31:37 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 23926870EA; Tue, 16 Apr 2019 17:31:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7QVu6ukE74s; Tue, 16 Apr 2019 17:31:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id D085A870B3; Tue, 16 Apr 2019 17:31:30 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id B53A11C1626 for ; Tue, 16 Apr 2019 17:31:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B320F85F0F for ; Tue, 16 Apr 2019 17:31:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t8XHQVuDIl1Z for ; Tue, 16 Apr 2019 17:31:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by fraxinus.osuosl.org (Postfix) with ESMTPS id CD7BE85E8C for ; Tue, 16 Apr 2019 17:31:27 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Apr 2019 10:31:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,358,1549958400"; d="scan'208";a="151390494" Received: from shasta.jf.intel.com ([10.166.244.191]) by orsmga002.jf.intel.com with ESMTP; 16 Apr 2019 10:31:25 -0700 From: Anirudh Venkataramanan To: intel-wired-lan@lists.osuosl.org Date: Tue, 16 Apr 2019 10:30:52 -0700 Message-Id: <20190416173052.28928-16-anirudh.venkataramanan@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190416173052.28928-1-anirudh.venkataramanan@intel.com> References: <20190416173052.28928-1-anirudh.venkataramanan@intel.com> MIME-Version: 1.0 Subject: [Intel-wired-lan] [PATCH S20 15/15] ice: Use GLINT_DYN_CTL to disable VF's interrupts X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" From: Brett Creeley Currently in ice_free_vf_res() we are writing to the VFINT_DYN_CTLN register in the PF's function space to disable all VF's interrupts. This is incorrect because this register is only for use in the VF's function space. This becomes obvious when seeing that the valid indices used for the VFINT_DYN_CTLN register is from 0-63, which is the maximum number of interrupts for a VF (not including the OICR interrupt). Fix this by writing to the GLINT_DYN_CTL register for each VF. We can do this because we keep track of each VF's first_vector_idx inside of the PF's function space and the number of interrupts given to each VF. Also in ice_free_vfs() we were disabling Rx/Tx queues after calling pci_disable_sriov(). One part of disabling the Tx queues causes the PF driver to trigger a software interrupt, which causes the VF's napi routine to run. This doesn't currently work because pci_disable_sriov() causes iavf_remove() to be called which disables interrupts. Fix this by disabling Rx/Tx queues prior to pci_disable_sriov(). Signed-off-by: Brett Creeley Signed-off-by: Anirudh Venkataramanan Tested-by: Andrew Bowers --- .../net/ethernet/intel/ice/ice_virtchnl_pf.c | 41 ++++++------------- 1 file changed, 13 insertions(+), 28 deletions(-) diff --git a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c index 0f79cf0e4ee8..9c6d9c95f4f6 100644 --- a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c +++ b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c @@ -140,18 +140,6 @@ static void ice_vc_notify_vf_link_state(struct ice_vf *vf) sizeof(pfe), NULL); } -/** - * ice_get_vf_vector - get VF interrupt vector register offset - * @vf_msix: number of MSIx vector per VF on a PF - * @vf_id: VF identifier - * @i: index of MSIx vector - */ -static u32 ice_get_vf_vector(int vf_msix, int vf_id, int i) -{ - return ((i == 0) ? VFINT_DYN_CTLN(vf_id) : - VFINT_DYN_CTLN(((vf_msix - 1) * (vf_id)) + (i - 1))); -} - /** * ice_free_vf_res - Free a VF's resources * @vf: pointer to the VF info @@ -159,7 +147,7 @@ static u32 ice_get_vf_vector(int vf_msix, int vf_id, int i) static void ice_free_vf_res(struct ice_vf *vf) { struct ice_pf *pf = vf->pf; - int i, pf_vf_msix; + int i, last_vector_idx; /* First, disable VF's configuration API to prevent OS from * accessing the VF's VSI after it's freed or invalidated. @@ -174,13 +162,10 @@ static void ice_free_vf_res(struct ice_vf *vf) vf->num_mac = 0; } - pf_vf_msix = pf->num_vf_msix; + last_vector_idx = vf->first_vector_idx + pf->num_vf_msix - 1; /* Disable interrupts so that VF starts in a known state */ - for (i = 0; i < pf_vf_msix; i++) { - u32 reg_idx; - - reg_idx = ice_get_vf_vector(pf_vf_msix, vf->vf_id, i); - wr32(&pf->hw, reg_idx, VFINT_DYN_CTLN_CLEARPBA_M); + for (i = vf->first_vector_idx; i <= last_vector_idx; i++) { + wr32(&pf->hw, GLINT_DYN_CTL(i), GLINT_DYN_CTL_CLEARPBA_M); ice_flush(&pf->hw); } /* reset some of the state variables keeping track of the resources */ @@ -281,15 +266,6 @@ void ice_free_vfs(struct ice_pf *pf) while (test_and_set_bit(__ICE_VF_DIS, pf->state)) usleep_range(1000, 2000); - /* Disable IOV before freeing resources. This lets any VF drivers - * running in the host get themselves cleaned up before we yank - * the carpet out from underneath their feet. - */ - if (!pci_vfs_assigned(pf->pdev)) - pci_disable_sriov(pf->pdev); - else - dev_warn(&pf->pdev->dev, "VFs are assigned - not disabling SR-IOV\n"); - /* Avoid wait time by stopping all VFs at the same time */ for (i = 0; i < pf->num_alloc_vfs; i++) { struct ice_vsi *vsi; @@ -305,6 +281,15 @@ void ice_free_vfs(struct ice_pf *pf) clear_bit(ICE_VF_STATE_ENA, pf->vf[i].vf_states); } + /* Disable IOV before freeing resources. This lets any VF drivers + * running in the host get themselves cleaned up before we yank + * the carpet out from underneath their feet. + */ + if (!pci_vfs_assigned(pf->pdev)) + pci_disable_sriov(pf->pdev); + else + dev_warn(&pf->pdev->dev, "VFs are assigned - not disabling SR-IOV\n"); + tmp = pf->num_alloc_vfs; pf->num_vf_qps = 0; pf->num_alloc_vfs = 0;