Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/750540/?format=api
{ "id": 750540, "url": "http://patchwork.ozlabs.org/api/patches/750540/?format=api", "web_url": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/20170413084555.6962-9-alice.michael@intel.com/", "project": { "id": 46, "url": "http://patchwork.ozlabs.org/api/projects/46/?format=api", "name": "Intel Wired Ethernet development", "link_name": "intel-wired-lan", "list_id": "intel-wired-lan.osuosl.org", "list_email": "intel-wired-lan@osuosl.org", "web_url": "", "scm_url": "", "webscm_url": "", "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20170413084555.6962-9-alice.michael@intel.com>", "list_archive_url": null, "date": "2017-04-13T08:45:52", "name": "[next,S70,09/12] i40e: reset all VFs in parallel when rebuilding PF", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": false, "hash": "808910e9b0a383f4872bcf538ba2e05eed6eaf6b", "submitter": { "id": 71123, "url": "http://patchwork.ozlabs.org/api/people/71123/?format=api", "name": "Michael, Alice", "email": "alice.michael@intel.com" }, "delegate": { "id": 68, "url": "http://patchwork.ozlabs.org/api/users/68/?format=api", "username": "jtkirshe", "first_name": "Jeff", "last_name": "Kirsher", "email": "jeffrey.t.kirsher@intel.com" }, "mbox": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/20170413084555.6962-9-alice.michael@intel.com/mbox/", "series": [], "comments": "http://patchwork.ozlabs.org/api/patches/750540/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/750540/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<intel-wired-lan-bounces@lists.osuosl.org>", "X-Original-To": [ "incoming@patchwork.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Delivered-To": [ "patchwork-incoming@bilbo.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Received": [ "from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3w3mvh03D3z9sDG\n\tfor <incoming@patchwork.ozlabs.org>;\n\tFri, 14 Apr 2017 02:48:48 +1000 (AEST)", "from localhost (localhost [127.0.0.1])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id 94D768A67A;\n\tThu, 13 Apr 2017 16:48:46 +0000 (UTC)", "from hemlock.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id 74WomsPZ+gZZ; Thu, 13 Apr 2017 16:48:42 +0000 (UTC)", "from ash.osuosl.org (ash.osuosl.org [140.211.166.34])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id 3D8628A69F;\n\tThu, 13 Apr 2017 16:48:38 +0000 (UTC)", "from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138])\n\tby ash.osuosl.org (Postfix) with ESMTP id D127F1CE48F\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 13 Apr 2017 16:48:33 +0000 (UTC)", "from localhost (localhost [127.0.0.1])\n\tby whitealder.osuosl.org (Postfix) with ESMTP id A695A89BF0\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 13 Apr 2017 16:48:33 +0000 (UTC)", "from whitealder.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id 7oXV4ttjeaWG for <intel-wired-lan@lists.osuosl.org>;\n\tThu, 13 Apr 2017 16:48:32 +0000 (UTC)", "from mga02.intel.com (mga02.intel.com [134.134.136.20])\n\tby whitealder.osuosl.org (Postfix) with ESMTPS id A54EB89BEC\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 13 Apr 2017 16:48:32 +0000 (UTC)", "from orsmga002.jf.intel.com ([10.7.209.21])\n\tby orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t13 Apr 2017 09:48:29 -0700", "from unknown (HELO localhost.jf.intel.com) ([10.166.16.121])\n\tby orsmga002.jf.intel.com with ESMTP; 13 Apr 2017 09:48:29 -0700" ], "X-Virus-Scanned": [ "amavisd-new at osuosl.org", "amavisd-new at osuosl.org" ], "X-Greylist": "domain auto-whitelisted by SQLgrey-1.7.6", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.37,195,1488873600\"; d=\"scan'208\";a=\"73711334\"", "From": "Alice Michael <alice.michael@intel.com>", "To": "alice.michael@intel.com,\n\tintel-wired-lan@lists.osuosl.org", "Date": "Thu, 13 Apr 2017 04:45:52 -0400", "Message-Id": "<20170413084555.6962-9-alice.michael@intel.com>", "X-Mailer": "git-send-email 2.9.3", "In-Reply-To": "<20170413084555.6962-1-alice.michael@intel.com>", "References": "<20170413084555.6962-1-alice.michael@intel.com>", "Subject": "[Intel-wired-lan] [next PATCH S70 09/12] i40e: reset all VFs in\n\tparallel when rebuilding PF", "X-BeenThere": "intel-wired-lan@lists.osuosl.org", "X-Mailman-Version": "2.1.18-1", "Precedence": "list", "List-Id": "Intel Wired Ethernet Linux Kernel Driver Development\n\t<intel-wired-lan.lists.osuosl.org>", "List-Unsubscribe": "<http://lists.osuosl.org/mailman/options/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@lists.osuosl.org?subject=unsubscribe>", "List-Archive": "<http://lists.osuosl.org/pipermail/intel-wired-lan/>", "List-Post": "<mailto:intel-wired-lan@lists.osuosl.org>", "List-Help": "<mailto:intel-wired-lan-request@lists.osuosl.org?subject=help>", "List-Subscribe": "<http://lists.osuosl.org/mailman/listinfo/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@lists.osuosl.org?subject=subscribe>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "7bit", "Errors-To": "intel-wired-lan-bounces@lists.osuosl.org", "Sender": "\"Intel-wired-lan\" <intel-wired-lan-bounces@lists.osuosl.org>" }, "content": "From: Jacob Keller <jacob.e.keller@intel.com>\n\nWhen there are a lot of active VFs, it can take multiple seconds to\nfinish resetting all of them during certain flows., which can cause some\nVFs to fail to wait long enough for the reset to occur. The user might\nsee messages like \"Never saw reset\" or \"Reset never finished\" and the VF\ndriver will stop functioning properly.\n\nThe naive solution would be to simply increase the wait timer. We can\nget much more clever. Notice that i40e_reset_vf is run in a serialized\nfashion, and includes lots of delays.\n\nThere are two prominent delays which take most of the time. First, when\nwe begin resetting VFs, we have multiple 10ms delays which accrue\nbecause we reset each VF in a serial fashion. These delays accumulate to\nalmost 4 seconds when handling the maximum number of VFs (128).\n\nSecondly, there is a massive 50ms delay for each time we disable queues\non a VSI. This delay is necessary to allow HW to finish disabling queues\nbefore we restore functionality. However, just like with the first case,\nwe are paying the cost for each VF, rather than disabling all VFs and\nwaiting once.\n\nBoth of these can be fixed, but required some previous refactoring to\nhandle the special case. First, we will need the\ni40e_vsi_wait_queues_disabled function which was previously DCB\nspecific. Second, we will need to implement our own\ni40e_vsi_stop_rings_no_wait function which will handle the stopping of\nrings without the delays.\n\nFinally, implement an i40e_reset_all_vfs function, which will first\nstart the reset of all VFs, and pay the wait cost all at once, rather\nthan serially waiting for each VF before we start processing then next\none. After the VF has been reset, we'll disable all the VF queues, and\nthen wait for them to disable. Again, we'll organize the flow such that\nwe pay the wait cost only once.\n\nFinally, after we've disabled queues we'll go ahead and begin restoring\nVF functionality. The result is reducing the wait time by a large factor\nand ensuring that VFs do not timeout when waiting in the VF driver.\n\nSigned-off-by: Jacob Keller <jacob.e.keller@intel.com>\nChange-ID: Ia6e8cf8d98131b78aec89db78afb8d905c9b12be\n---\nTesting-hints:\n \"time echo 1 > /sys/class/net/<enpX>/device/reset\" will trigger\n a request for a reset in the driver. I used the \"time\" command to\n measure how long it took for the reset to finish.\n\n drivers/net/ethernet/intel/i40e/i40e.h | 2 +\n drivers/net/ethernet/intel/i40e/i40e_main.c | 32 +++++--\n drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c | 100 +++++++++++++++++++++\n drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h | 1 +\n 4 files changed, 129 insertions(+), 6 deletions(-)", "diff": "diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h\nindex a33a733..d35fec3 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e.h\n+++ b/drivers/net/ethernet/intel/i40e/i40e.h\n@@ -903,6 +903,8 @@ void i40e_notify_client_of_vf_msg(struct i40e_vsi *vsi, u32 vf_id,\n \n int i40e_vsi_start_rings(struct i40e_vsi *vsi);\n void i40e_vsi_stop_rings(struct i40e_vsi *vsi);\n+void i40e_vsi_stop_rings_no_wait(struct i40e_vsi *vsi);\n+int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi);\n int i40e_reconfig_rss_queues(struct i40e_pf *pf, int queue_count);\n struct i40e_veb *i40e_veb_setup(struct i40e_pf *pf, u16 flags, u16 uplink_seid,\n \t\t\t\tu16 downlink_seid, u8 enabled_tc);\ndiff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c\nindex ebf2ad4..7ceb256 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_main.c\n+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c\n@@ -4159,6 +4159,29 @@ void i40e_vsi_stop_rings(struct i40e_vsi *vsi)\n }\n \n /**\n+ * i40e_vsi_stop_rings_no_wait - Stop a VSI's rings and do not delay\n+ * @vsi: the VSI being shutdown\n+ *\n+ * This function stops all the rings for a VSI but does not delay to verify\n+ * that rings have been disabled. It is expected that the caller is shutting\n+ * down multiple VSIs at once and will delay together for all the VSIs after\n+ * initiating the shutdown. This is particularly useful for shutting down lots\n+ * of VFs together. Otherwise, a large delay can be incurred while configuring\n+ * each VSI in serial.\n+ **/\n+void i40e_vsi_stop_rings_no_wait(struct i40e_vsi *vsi)\n+{\n+\tstruct i40e_pf *pf = vsi->back;\n+\tint i, pf_q;\n+\n+\tpf_q = vsi->base_queue;\n+\tfor (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {\n+\t\ti40e_control_tx_q(pf, pf_q, false);\n+\t\ti40e_control_rx_q(pf, pf_q, false);\n+\t}\n+}\n+\n+/**\n * i40e_vsi_free_irq - Free the irq association with the OS\n * @vsi: the VSI being configured\n **/\n@@ -4488,14 +4511,13 @@ static void i40e_pf_unquiesce_all_vsi(struct i40e_pf *pf)\n \t}\n }\n \n-#ifdef CONFIG_I40E_DCB\n /**\n * i40e_vsi_wait_queues_disabled - Wait for VSI's queues to be disabled\n * @vsi: the VSI being configured\n *\n * Wait until all queues on a given VSI have been disabled.\n **/\n-static int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)\n+int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)\n {\n \tstruct i40e_pf *pf = vsi->back;\n \tint i, pf_q, ret;\n@@ -4523,6 +4545,7 @@ static int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)\n \treturn 0;\n }\n \n+#ifdef CONFIG_I40E_DCB\n /**\n * i40e_pf_wait_queues_disabled - Wait for all queues of PF VSIs to be disabled\n * @pf: the PF\n@@ -7194,10 +7217,7 @@ static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired)\n \tif (!lock_acquired)\n \t\trtnl_unlock();\n \n-\tif (pf->num_alloc_vfs) {\n-\t\tfor (v = 0; v < pf->num_alloc_vfs; v++)\n-\t\t\ti40e_reset_vf(&pf->vf[v], true);\n-\t}\n+\ti40e_reset_all_vfs(pf, true);\n \n \t/* tell the firmware that we're starting */\n \ti40e_send_version(pf);\ndiff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c\nindex a09c2b3..423068a 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c\n+++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c\n@@ -1076,6 +1076,106 @@ void i40e_reset_vf(struct i40e_vf *vf, bool flr)\n }\n \n /**\n+ * i40e_reset_all_vfs\n+ * @pf: pointer to the PF structure\n+ * @flr: VFLR was issued or not\n+ *\n+ * Reset all allocated VFs in one go. First, tell the hardware to reset each\n+ * VF, then do all the waiting in one chunk, and finally finish restoring each\n+ * VF after the wait. This is useful during PF routines which need to reset\n+ * all VFs, as otherwise it must perform these resets in a serialized fashion.\n+ **/\n+void i40e_reset_all_vfs(struct i40e_pf *pf, bool flr)\n+{\n+\tstruct i40e_hw *hw = &pf->hw;\n+\tstruct i40e_vf *vf;\n+\tint i, v;\n+\tu32 reg;\n+\n+\t/* If we don't have any VFs, then there is nothing to reset */\n+\tif (!pf->num_alloc_vfs)\n+\t\treturn;\n+\n+\t/* If VFs have been disabled, there is no need to reset */\n+\tif (test_and_set_bit(__I40E_VF_DISABLE, &pf->state))\n+\t\treturn;\n+\n+\t/* Begin reset on all VFs at once */\n+\tfor (v = 0; v < pf->num_alloc_vfs; v++)\n+\t\ti40e_trigger_vf_reset(&pf->vf[v], flr);\n+\n+\t/* HW requires some time to make sure it can flush the FIFO for a VF\n+\t * when it resets it. Poll the VPGEN_VFRSTAT register for each VF in\n+\t * sequence to make sure that it has completed. We'll keep track of\n+\t * the VFs using a simple iterator that increments once that VF has\n+\t * finished resetting.\n+\t */\n+\tfor (i = 0, v = 0; i < 10 && v < pf->num_alloc_vfs; i++) {\n+\t\tusleep_range(10000, 20000);\n+\n+\t\t/* Check each VF in sequence, beginning with the VF to fail\n+\t\t * the previous check.\n+\t\t */\n+\t\twhile (v < pf->num_alloc_vfs) {\n+\t\t\tvf = &pf->vf[v];\n+\t\t\treg = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_id));\n+\t\t\tif (!(reg & I40E_VPGEN_VFRSTAT_VFRD_MASK))\n+\t\t\t\tbreak;\n+\n+\t\t\t/* If the current VF has finished resetting, move on\n+\t\t\t * to the next VF in sequence.\n+\t\t\t */\n+\t\t\tv++;\n+\t\t}\n+\t}\n+\n+\tif (flr)\n+\t\tusleep_range(10000, 20000);\n+\n+\t/* Display a warning if at least one VF didn't manage to reset in\n+\t * time, but continue on with the operation.\n+\t */\n+\tif (v < pf->num_alloc_vfs)\n+\t\tdev_err(&pf->pdev->dev, \"VF reset check timeout on VF %d\\n\",\n+\t\t\tpf->vf[v].vf_id);\n+\tusleep_range(10000, 20000);\n+\n+\t/* Begin disabling all the rings associated with VFs, but do not wait\n+\t * between each VF.\n+\t */\n+\tfor (v = 0; v < pf->num_alloc_vfs; v++) {\n+\t\t/* On initial reset, we don't have any queues to disable */\n+\t\tif (pf->vf[v].lan_vsi_idx == 0)\n+\t\t\tcontinue;\n+\n+\t\ti40e_vsi_stop_rings_no_wait(pf->vsi[pf->vf[v].lan_vsi_idx]);\n+\t}\n+\n+\t/* Now that we've notified HW to disable all of the VF rings, wait\n+\t * until they finish.\n+\t */\n+\tfor (v = 0; v < pf->num_alloc_vfs; v++) {\n+\t\t/* On initial reset, we don't have any queues to disable */\n+\t\tif (pf->vf[v].lan_vsi_idx == 0)\n+\t\t\tcontinue;\n+\n+\t\ti40e_vsi_wait_queues_disabled(pf->vsi[pf->vf[v].lan_vsi_idx]);\n+\t}\n+\n+\t/* Hw may need up to 50ms to finish disabling the RX queues. We\n+\t * minimize the wait by delaying only once for all VFs.\n+\t */\n+\tmdelay(50);\n+\n+\t/* Finish the reset on each VF */\n+\tfor (v = 0; v < pf->num_alloc_vfs; v++)\n+\t\ti40e_cleanup_reset_vf(&pf->vf[v]);\n+\n+\ti40e_flush(hw);\n+\tclear_bit(__I40E_VF_DISABLE, &pf->state);\n+}\n+\n+/**\n * i40e_free_vfs\n * @pf: pointer to the PF structure\n *\ndiff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h\nindex 3e1f8f6..f76d234 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h\n+++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h\n@@ -130,6 +130,7 @@ int i40e_vc_process_vf_msg(struct i40e_pf *pf, s16 vf_id, u32 v_opcode,\n \t\t\t u32 v_retval, u8 *msg, u16 msglen);\n int i40e_vc_process_vflr_event(struct i40e_pf *pf);\n void i40e_reset_vf(struct i40e_vf *vf, bool flr);\n+void i40e_reset_all_vfs(struct i40e_pf *pf, bool flr);\n void i40e_vc_notify_vf_reset(struct i40e_vf *vf);\n \n /* VF configuration related iplink handlers */\n", "prefixes": [ "next", "S70", "09/12" ] }