From patchwork Thu Mar 29 18:13:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Manlunas, Felix" X-Patchwork-Id: 892914 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=cavium.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=CAVIUMNETWORKS.onmicrosoft.com header.i=@CAVIUMNETWORKS.onmicrosoft.com header.b="lymiYjtV"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40BtCw3q2Jz9s1s for ; Fri, 30 Mar 2018 05:13:32 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754244AbeC2SNa (ORCPT ); Thu, 29 Mar 2018 14:13:30 -0400 Received: from mail-dm3nam03on0085.outbound.protection.outlook.com ([104.47.41.85]:61664 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752201AbeC2SN1 (ORCPT ); Thu, 29 Mar 2018 14:13:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=slJ1bQ9/oEaEho53npOPqqJPQYajdj/QIi4QX5OB1CE=; b=lymiYjtVf4o6CfLoCTjCvt2mgNqbwi3CplE99lSum9g21qlB6spMMx9tuUj86u57eK188ey6n5uRL2LkVgux5UCtsh8QuFrJjHLYOg8NYiwxckfCiGEoTRGNRbWsocwmAgvQyCU2AedQRJIb/2E4vu33QUDX//PLVW9FnbC0o08= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Felix.Manlunas@cavium.com; Received: from localhost (50.233.148.156) by DM6PR07MB4220.namprd07.prod.outlook.com (2603:10b6:5:bd::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.609.10; Thu, 29 Mar 2018 18:13:25 +0000 Date: Thu, 29 Mar 2018 11:13:22 -0700 From: Felix Manlunas To: davem@davemloft.net Cc: netdev@vger.kernel.org, raghu.vatsavayi@cavium.com, derek.chickles@cavium.com, satananda.burla@cavium.com, felix.manlunas@cavium.com Subject: [PATCH V2 net-next] liquidio: prevent rx queues from getting stalled Message-ID: <20180329181322.GA9939@felix-thinkpad.cavium.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [50.233.148.156] X-ClientProxiedBy: CO2PR06CA0059.namprd06.prod.outlook.com (2603:10b6:104:3::17) To DM6PR07MB4220.namprd07.prod.outlook.com (2603:10b6:5:bd::29) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fd29d347-79a7-4375-86cf-08d595a0c40a X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DM6PR07MB4220; X-Microsoft-Exchange-Diagnostics: 1; DM6PR07MB4220; 3:nwwZ0QvlgntP+/DiTXeptQRt6SWnVaoq92HDppn5uCdem0l+n6q4bIyrgw9g9VvMxflSSY9qTtDvJZ3l2LddjxD/gYQ17UhyixAHk73LqVbfxrnt8ZZHmrnO9kpirNobyU0nS2t0M96b5LVQDJdjaEdvoJIslB2PzYypU+i0s+9nvIwmONhoZ0MSM42etOHFg5K3H/8TGi4cA0jQebdJR84eYxEHK7ybjE+IHIz1Fx7C0WHYW5RveRBe7951TSki; 25:0VU9AM9tNdb/to4XXAaLTecqLBtMq7d3Fq0t/2P+Drw9glpYyftFzOP6oYpzSVnr0K/yFsY8eHiXCBPUn6HsIBrttMYuJyWpiGJcxPB5SssnmtpqKT+6smJXAgoM8u5j1p9uMK4MdqV6IV6hTC2N/dPSAG3ANMWDnM4oCm6gdWNjdMDNMIC7O4Bp34qM7c+pPXlQ9fBfdRHl2mGbU7z0+X8txbqNMD7HPsmTjIVmM9RjDOd7H841od99fPjpfXEIab56lgmKSe/UnLWkxntu/+V1pFxvCV/H5cMrW8KsDHhPxrNv4DfO9lFzUAiys3TS1JQFq76KHNNxglRL8+Lk2Q==; 31:JV3iPy3eOnmYLRjbKa5mfiQbgfxFpAzR4A00joRPJgbU+40+kj6ESjzpphO9L6ZDeMTfKgY0/wbAGFyZ8WkOKSjo5R1N6Huxnq5JlgdbA2DDgR+QYVDN+26jG55pbd2WqBowDxxJHaefqWGITJhROqTDh3pNUfxUU98kt1xIprRlPAM/XzscADitJU52hGHBTnUAQgYvRGsxWWPeBNV4j/h7MvjNeUHnnhGT13mtlqU= X-MS-TrafficTypeDiagnostic: DM6PR07MB4220: X-Microsoft-Exchange-Diagnostics: 1; DM6PR07MB4220; 20:duoZYdWq92Kr6b5EDrVh5oq9nnuhULGUKO0kk3haeiAmAgWbAiDzV7LKvBGBhwOq/ppbesDkWd6Ttizuubv9p8XN+Oyvmcneectg9/j0dA8CvsCV6qcwZb/x51sUyfD739EZVLAvtl8cItq3c0+EJK6CMUgFmGvExyvhDPAR5T0aCbeuXyt/DjjjfqqLydr6c2P9/wG2m+rrm3+cgxdpGVKiiLY35ioivAVD7qiC2hBPHe0vbLCmuXZsZrlFZBqd0jBWPW7Jj04XW+BD0jeYcqJZXBzpELe5JoBiZl4TeNxtXIS5qQvV1GPWcK0eyvC8hlq4rJhu1nJGIqJzsVYwi1zgmIvevuFaXVg1pnyD8iUW/BfOldXOcyAzxum2WH0Q2Bgf6kYJl3R5g6XtUdu3/OSkadD2RUxCyfBxTah09L6bO2E69uUdq6mOglXv4DAtgR9GAZkgy5yN1VLrInWnO31vtir/3EswlB1DW6zETv8Ozd+ageoaZwNPo3GoIob5; 4:mqcHXQWSvwbOdYjCtb04KAQFZCciU6XH3N5O5KT+/6BotFF9EmO7/iE2W0KhodjnX4tpV/1vQGnLGX3w29aS4331uC2fik5igWO7mxTtHtJcX07GvQltW84pTBsPG18/XwyJfhH0YT5vNz1yG9LfshPmPF5SzywgGo/uxuqh21tyxdo3Bd2tljQudfIkLmCsRYbVvbNqC9ECjoLEvYTLWqjLvkKVVXRNjDxR+YqtpN9jl3ARozWTAWcouIhZZC/Yq1JwGO7Pll59VrJJt1R25g== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231221)(944501327)(52105095)(6041310)(20161123560045)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(6072148)(201708071742011); SRVR:DM6PR07MB4220; BCL:0; PCL:0; RULEID:; SRVR:DM6PR07MB4220; X-Forefront-PRVS: 0626C21B10 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(366004)(39860400002)(346002)(39380400002)(376002)(396003)(54534003)(199004)(189003)(33656002)(81156014)(5660300001)(107886003)(186003)(6486002)(68736007)(50466002)(6916009)(8676002)(2361001)(2351001)(486005)(26005)(486005)(76506005)(81166006)(53936002)(16526019)(6666003)(72206003)(25786009)(478600001)(52116002)(6496006)(6116002)(1076002)(386003)(16586007)(8936002)(58126008)(956004)(86362001)(316002)(47776003)(3846002)(305945005)(7736002)(59450400001)(2906002)(66066001)(4326008)(106356001)(476003)(105586002)(97736004)(23726003)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR07MB4220; H:localhost; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: 1; DM6PR07MB4220; 23:Gwuau3ywzk85ucV6lac864ZmH2WjTJiQw+03CTeDOjAXViSCW5XDznjr65gJjYS5Sj3b6ZXgogShgBgpiyEGiS9L32OX/tmm2kynZJlS5C2AOzBpIJuYCgZazHs8cz3eFvxB5H65a6WbxZwzOa0GXgrqoO7KWGM93ec9mgBO3WfVkpamaq7Rm6S97ufleLOcx27wa5EyVnF6rjli7Zmz/CpSpJqziy7QWvMFN6OSVYvoUAxYtIkNtk3Z4Jir0FdnEoWHkB4MOOVj6U1mhDIgXSgGcLZzPPPNxxbymolBnwlBJQKKAJCI2viKWsUYE2n2n2V/nbl1y0xLhZ/qumWMwx4P7K+RdmcBj9M2PKkI0HK6ihbBfWtLYwcmwqcxoKLPvaln2yrWlL9mo16+DZpBROhzUkXc7mEA3bW3+UF8XPovWJDCfRE1MtinTECkhi4kRnz3Vk21CvY0F3yAjiSIebDc55AiH06O+OzdDQbDNdWobsplpYYrovl5+KDVXXLFQEtK6kbpHmikKdhFUV0udsxZF2uhgFCXJFg55H4mX+oiRfGuHmo8SZyYQAEZ8XZlVYM6hBzHoFAKNMQFV0N6Vv7ta2AzBEuzvzLaGzUv7/qnrngA97cO0C42gRY/MVYb1OM1SbPWZlARArV5z7LW7W5IXcUf8X12ou001vaifLydRpRHq8HbRjoH7jj0t37GkM+W9MWRNNTLbliTzmk4dW6vdq2B4u2F4qiKzIXsWiJMZENpMNyRgnj70uT8N8ZHp6J3E2KeEWCSuu3t1Avd88BnKD85t9rj84ST/xS/N9KBH6ATbU+J/Wjgs6pLiK0u1ds2BU+gPRD2J8cmJVza2fZ/NcoSj0VivJl4BWd2rqN/F1jqTcV7+57Y3eBQ6XwScKckgcUKY34kZU3+5bv0IdhkeqLc2HzK13NhLSgz817e686n1Y+YBPhwEdFPxw2geg0IUSxyomW3hxpb8z8CQedCm6XDfL+u5vqZrSz2Qw3Xvz7y64+EFM7uatv8v+tTcdnKiDQQ6ADo9s9ydf/hZBCpxDytrpxG5xMyrTxUsQxcwmmo1Y5xARuymvRGJXTp/e5CShQZFevTN2msmOzVPOeZuvueNjUo5m1hkGM9/AXuz3HJlPevesDaAdr23f+dAL5VYx0RiIyU5/5SFrHpV4uF+stv+XMStBXCoNQjHQK0SlHDPSEc3pW8Yvx6j8vtu81LCE4/EkBCMxUZzIE1wamFoFT/gh3ooPaZZ/wIZx5ZqQDZZqmNL20sbpfshTTqM2XZkJU/ZQFN/mlkqaK9Ww== X-Microsoft-Antispam-Message-Info: drzxY6CvHgVFDE0R1Ay7ynXWlef8TQo0eZ0wPNrWyspnlsoSsY691lmCx82kpv+vzLzrqjZEAc5A9/3GUTodYGP2jGrryI3AzsIEjTamyEL1hpiNSmhJficKIZlQwxf3roqHMW5TwzgZmn+TFjU+TaNQmBUhyhck4VCjdyQZgMCjquGMOnTZdiZGqJrxnVsA X-Microsoft-Exchange-Diagnostics: 1; DM6PR07MB4220; 6:aOYY7/bqXGmJKODPaBFpt8b5ZcG5VmkUPcTFDxEPBTxHgPC9M/5+r378mN5rjiW9ltj7zkVzhg0xkI1vmkvcYXJu+2ZLjcEc3djbWiXTf73aZG11Dmjg+aw1qDaRDVYgrbwmZqQZBCa5Z8Y00w6rbVxK03ZT1FbnBgaTWhxwBu9klOMu3/dBr4V7MtQ4sIh/rZ7uoyDBnGcSdxlsVLNx+SCEvdN1RJXtmOfbyNhOzzwFiOFDuPHEQDPMTeGKIontZBBJJ+CnA1cXlhOCv4nE9Qm4jgx3fJWwiVgoj53cnz11QqWaYcQaBFw77BRG+unZOc2MLw6wOiU10U4DsgGPEj/7wXDmlwdU24pGkuS16vnbbf4prkc4N8c7cH0caCH/VhzQCVjuimZGDRBDy8mkyENulwzFmO3X7UKns22S9beODoJS9E8WYSVqhjbuN5Gnm7O72KSB479/piWDqKh5mQ==; 5:RA5bi5yeWmOV3+I+RX3MWnm/J1cWUmAxaukeDGMCP/c0WtsUvfPlvw9VfIzEGIrx+KghxO1hFqk+a+/XFrOoO6wSxoU9TeT3sKNf6APTgbmCq8xV28E7nCK/yKU5/WUHsrxucuVfbC9wQDkR/53WwXPBJB3S1Ui0HV5NpYvHJNw=; 24:WWEuoiHhmBVcr9Ffu2G1XIPIc7zYpt/oarUhoy8ainspn8XDtDTOdymvT+V2aSKUNEgWZI8dS2hz1QM5NmagmFfY4pdPIcyc5v3ungPys4Y= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM6PR07MB4220; 7:atK7XL9QlB65RwZH/QR6P40LymVch4fgVdxJeq5LE6ksh/2MePQfVvaBQ0mI/tOJxoBI2utqbwMBMtc8wwNrrgbyoA+VsTJwJGh40qr/m4cdwTGD4iSj1suyDbwPxB/ZCQO/hhhSL27Q1PrYipgBjniEAwt0r+8Fn/STWl+6hLVsEJ/SDMPlU9ixWt+cooTmIoqvuCK1wmsjD/y/Kuo3mAzMi9X1U7GcQqBAtE2W1pL6bUs2xgNACjopruq57Dia X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Mar 2018 18:13:25.7717 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fd29d347-79a7-4375-86cf-08d595a0c40a X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR07MB4220 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Raghu Vatsavayi This commit has fix for RX traffic issues when we stress test the driver with continuous ifconfig up/down under very high traffic conditions. Reason for the issue is that, in existing liquidio_stop function NAPI is disabled even before actual FW/HW interface is brought down via send_rx_ctrl_cmd(lio, 0). Between time frame of NAPI disable and actual interface down in firmware, firmware continuously enqueues rx traffic to host. When interrupt happens for new packets, host irq handler fails in scheduling NAPI as the NAPI is already disabled. After "ifconfig up", Host re-enables NAPI but cannot schedule it until it receives another Rx interrupt. Host never receives Rx interrupt as it never cleared the Rx interrupt it received during interface down operation. NIC Rx interrupt gets cleared only when Host processes queue and clears the queue counts. Above anomaly leads to other issues like packet overflow in FW/HW queues, backpressure. Fix: This commit fixes this issue by disabling NAPI only after informing firmware to stop queueing packets to host via send_rx_ctrl_cmd(lio, 0). send_rx_ctrl_cmd is not visible in the patch as it is already there in the code. The DOWN command also waits for any pending packets to be processed by NAPI so that the deadlock will not occur. Signed-off-by: Raghu Vatsavayi Acked-by: Derek Chickles Signed-off-by: Felix Manlunas --- Patch Change Log: V1 -> V2: Revised the subject line and patch description to add clarity. No change in the actual code. drivers/net/ethernet/cavium/liquidio/lio_core.c | 23 ++++++++++++++++++++ drivers/net/ethernet/cavium/liquidio/lio_main.c | 25 +++++++++++++--------- drivers/net/ethernet/cavium/liquidio/lio_vf_main.c | 23 ++++++++++++-------- .../net/ethernet/cavium/liquidio/octeon_network.h | 1 + 4 files changed, 53 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/cavium/liquidio/lio_core.c b/drivers/net/ethernet/cavium/liquidio/lio_core.c index 73e70e0..2a94eee 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_core.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_core.c @@ -1146,3 +1146,26 @@ int liquidio_change_mtu(struct net_device *netdev, int new_mtu) octeon_free_soft_command(oct, sc); return 0; } + +int lio_wait_for_clean_oq(struct octeon_device *oct) +{ + int retry = 100, pending_pkts = 0; + int idx; + + do { + pending_pkts = 0; + + for (idx = 0; idx < MAX_OCTEON_OUTPUT_QUEUES(oct); idx++) { + if (!(oct->io_qmask.oq & BIT_ULL(idx))) + continue; + pending_pkts += + atomic_read(&oct->droq[idx]->pkts_pending); + } + + if (pending_pkts > 0) + schedule_timeout_uninterruptible(1); + + } while (retry-- && pending_pkts); + + return pending_pkts; +} diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c index 43c5ba0..603a144 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_main.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c @@ -2084,16 +2084,6 @@ static int liquidio_stop(struct net_device *netdev) struct octeon_device *oct = lio->oct_dev; struct napi_struct *napi, *n; - if (oct->props[lio->ifidx].napi_enabled) { - list_for_each_entry_safe(napi, n, &netdev->napi_list, dev_list) - napi_disable(napi); - - oct->props[lio->ifidx].napi_enabled = 0; - - if (OCTEON_CN23XX_PF(oct)) - oct->droq[0]->ops.poll_mode = 0; - } - ifstate_reset(lio, LIO_IFSTATE_RUNNING); netif_tx_disable(netdev); @@ -2119,6 +2109,21 @@ static int liquidio_stop(struct net_device *netdev) lio->ptp_clock = NULL; } + /* Wait for any pending Rx descriptors */ + if (lio_wait_for_clean_oq(oct)) + netif_info(lio, rx_err, lio->netdev, + "Proceeding with stop interface after partial RX desc processing\n"); + + if (oct->props[lio->ifidx].napi_enabled == 1) { + list_for_each_entry_safe(napi, n, &netdev->napi_list, dev_list) + napi_disable(napi); + + oct->props[lio->ifidx].napi_enabled = 0; + + if (OCTEON_CN23XX_PF(oct)) + oct->droq[0]->ops.poll_mode = 0; + } + dev_info(&oct->pci_dev->dev, "%s interface is stopped\n", netdev->name); return 0; diff --git a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c index dc62698..f92dfa4 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c @@ -1138,15 +1138,6 @@ static int liquidio_stop(struct net_device *netdev) /* tell Octeon to stop forwarding packets to host */ send_rx_ctrl_cmd(lio, 0); - if (oct->props[lio->ifidx].napi_enabled) { - list_for_each_entry_safe(napi, n, &netdev->napi_list, dev_list) - napi_disable(napi); - - oct->props[lio->ifidx].napi_enabled = 0; - - oct->droq[0]->ops.poll_mode = 0; - } - netif_info(lio, ifdown, lio->netdev, "Stopping interface!\n"); /* Inform that netif carrier is down */ lio->intf_open = 0; @@ -1159,6 +1150,20 @@ static int liquidio_stop(struct net_device *netdev) stop_txqs(netdev); + /* Wait for any pending Rx descriptors */ + if (lio_wait_for_clean_oq(oct)) + netif_info(lio, rx_err, lio->netdev, + "Proceeding with stop interface after partial RX desc processing\n"); + + if (oct->props[lio->ifidx].napi_enabled == 1) { + list_for_each_entry_safe(napi, n, &netdev->napi_list, dev_list) + napi_disable(napi); + + oct->props[lio->ifidx].napi_enabled = 0; + + oct->droq[0]->ops.poll_mode = 0; + } + dev_info(&oct->pci_dev->dev, "%s interface is stopped\n", netdev->name); return 0; diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_network.h b/drivers/net/ethernet/cavium/liquidio/octeon_network.h index 8782206..4069710 100644 --- a/drivers/net/ethernet/cavium/liquidio/octeon_network.h +++ b/drivers/net/ethernet/cavium/liquidio/octeon_network.h @@ -190,6 +190,7 @@ irqreturn_t liquidio_msix_intr_handler(int irq __attribute__((unused)), int octeon_setup_interrupt(struct octeon_device *oct, u32 num_ioqs); +int lio_wait_for_clean_oq(struct octeon_device *oct); /** * \brief Register ethtool operations * @param netdev pointer to network device