From patchwork Mon Mar 5 15:16:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?R=C3=B3bert_Mulik?= X-Patchwork-Id: 881541 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=ericsson.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="CvyOv2Yo"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="U68L0Cwp"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zw3RY1775z9sZS for ; Tue, 6 Mar 2018 02:17:12 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 0C37ED74; Mon, 5 Mar 2018 15:17:06 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 067B3D25 for ; Mon, 5 Mar 2018 15:17:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from sessmg22.ericsson.net (sessmg22.ericsson.net [193.180.251.58]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C56A12C3 for ; Mon, 5 Mar 2018 15:17:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=ericsson.com; s=mailgw201801; c=relaxed/simple; q=dns/txt; i=@ericsson.com; t=1520263021; h=From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=SInvAoEX7lASt5B8Q7EGdUQsILCvYQv567FhBXoiwhk=; b=CvyOv2YojOJMV7uZjqCFHiDhR8B1qPzzY7P6DYC2f3JaIqgTVyP8ao8/4M543EN/ nl6riUIcmuJd0endaiFt6jq3Z6Fkkv87bv+8ZyvIILpHoQcUSCOeMcKzC4H3Dt5O uSdMFqx705DGTDeZ1FVLVK6N5XaIt1ZfyOH70ePON7c=; X-AuditID: c1b4fb3a-35fff700000067b4-e8-5a9d5f6dbfd5 Received: from ESESSHC023.ericsson.se (Unknown_Domain [153.88.183.87]) by sessmg22.ericsson.net (Symantec Mail Security) with SMTP id 7D.23.26548.D6F5D9A5; Mon, 5 Mar 2018 16:17:01 +0100 (CET) Received: from ESESBMR503.ericsson.se (153.88.183.135) by ESESSHC023.ericsson.se (153.88.183.87) with Microsoft SMTP Server (TLS) id 14.3.352.0; Mon, 5 Mar 2018 16:17:01 +0100 Received: from ESESBMB501.ericsson.se (153.88.183.168) by ESESBMR503.ericsson.se (153.88.183.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1034.26; Mon, 5 Mar 2018 16:17:01 +0100 Received: from EUR01-HE1-obe.outbound.protection.outlook.com (153.88.183.157) by ESESBMB501.ericsson.se (153.88.183.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1034.26 via Frontend Transport; Mon, 5 Mar 2018 16:17:01 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=SInvAoEX7lASt5B8Q7EGdUQsILCvYQv567FhBXoiwhk=; b=U68L0Cwp/84fVczEZY2KhPC0DTMpPo38kuAdNDa2QIcHSTpoa6pBtSYR0ygBBSf8fbGYrY6z7Khz2HzTBv9Q/N5Zna1+Q3/HvRT7B/tKQmI5vxXWAuCRej3uf/hjNembZb2+xtKZx4qeejtZW76jal/MhY8bu4gXt9VZg2NMCAc= Received: from HE1PR07MB1212.eurprd07.prod.outlook.com (10.163.178.30) by HE1PR07MB1129.eurprd07.prod.outlook.com (10.163.177.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.567.6; Mon, 5 Mar 2018 15:16:59 +0000 Received: from HE1PR07MB1212.eurprd07.prod.outlook.com ([fe80::65d6:54c:6727:fcbf]) by HE1PR07MB1212.eurprd07.prod.outlook.com ([fe80::65d6:54c:6727:fcbf%5]) with mapi id 15.20.0567.011; Mon, 5 Mar 2018 15:16:59 +0000 From: =?utf-8?q?R=C3=B3bert_Mulik?= To: "dev@openvswitch.org" Thread-Topic: [PATCH v6] Configurable Link State Change (LSC) detection mode Thread-Index: AdO0kn+5477Y7LQ2THe2vdQd18vxqw== Date: Mon, 5 Mar 2018 15:16:59 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [89.135.192.225] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR07MB1129; 7:XHyM/xp1I6OIfcNFYhuG5sf4eZCD0BpFFcIbtoUCDWZ9kS08zMd+NVvsfokQQZR3Ov7sxKP4T6Ph3ghS7iWdXMLRWyhAk7OExqYzuefEX38tJdELUnKjE1tgelQ3i7Q31M/T6xpfQ2M3yjTfg3b9gWHIpcIPNIu1zZ3GWbJm0XtOPyprzwykHVxQIUMTYStAiU32gbm58ZXQucqLPuXzEpz4aKK8gCMn64UQkGI4+NjJTvQY2yo3vSspr96ASVA5 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: b8a80738-4885-4bb9-d806-08d582ac2438 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603307)(7153060)(7193020); SRVR:HE1PR07MB1129; x-ms-traffictypediagnostic: HE1PR07MB1129: authentication-results: spf=none (sender IP is ) smtp.mailfrom=robert.mulik@ericsson.com; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(37575265505322); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040501)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3231220)(944501244)(52105095)(3002001)(6041288)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(6072148)(201708071742011); SRVR:HE1PR07MB1129; BCL:0; PCL:0; RULEID:; SRVR:HE1PR07MB1129; x-forefront-prvs: 06022AA85F x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39380400002)(39860400002)(376002)(346002)(366004)(396003)(189003)(199004)(81166006)(2906002)(5660300001)(85182001)(68736007)(81156014)(1730700003)(2501003)(186003)(33656002)(7696005)(2351001)(316002)(99286004)(102836004)(25786009)(9686003)(6506007)(86362001)(3280700002)(74316002)(66066001)(8936002)(14454004)(8676002)(478600001)(3660700001)(6116002)(3846002)(305945005)(105586002)(85202003)(7736002)(106356001)(5250100002)(5890100001)(53936002)(6436002)(55016002)(6916009)(26005)(5640700003)(97736004)(2900100001)(59450400001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR07MB1129; H:HE1PR07MB1212.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: KVOpVTmXl+EPin/qZGK6sstF2SIC7Xt0z4ROxLi468qhgG6FFEaztpJnsCyhoHM8abAVNY7VjFSyhDeA6aLRG2BmutL1yMJOTwcmeSAl1R/1F+6XyHQReUHTDpewmaGggDhZ3pbOOb4Hp4KqD8NnGTAnDMLnLaKsWkwcz1gk5ffPZhyZ4/Uam0dbJQHu+Lo5GAdbMtSRJ249KH1y5NUVXlWkh+p4foS6pZR/Qrz3j9T4ku/nVvFxPnpCOkzS/IMgwIlfOMtul6FAXhxTDhj7cGnvTspnT7iwl2sfNFlwzdxFEitMktHQRQpW1VigW6TP7hiCKiagNe1fw771DTbW2g== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: b8a80738-4885-4bb9-d806-08d582ac2438 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Mar 2018 15:16:59.4472 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR07MB1129 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrBIsWRmVeSWpSXmKPExsUyM2J7uG5u/Nwog6/bzSyOnt7D7MDo8ezm f8YAxigum5TUnMyy1CJ9uwSujIkHrzAXXAiuuP3LpIFxT2AXIweHhICJxI7fvl2MXBxCAocZ Jda96WLpYuQEcjYzSsx+VwuR+MYosXN6AyuEs4RJordxDxNIFYvAJyaJc30REImJTBKPXrWw QDj3GSXe333NClLFJuAkcfPnETBbREBf4lDPWbAdwgLuEu1LHjBDxH0kJmyczgxyk4iAnsSs bx4QC1QkjjZdBmvlFYiR+Dr7DpjNKCAm8f3UGrAjmAXEJW49mQ9mSwgISCzZc54ZwhaVePn4 HyuErSQx49UtKFtW4tL8bkYIewuTxN3ZZRC2rsSHqVOhen0lppxfxAbyi4TACUaJzS3ToBq0 JDofLWaCOCJG4sH9tVAN2RI/p35ihISplcS8M7EQvUuZJSZMWsYGUSMj0f52EzNE4hSrxKvD G1gmMOrNQvLELKB+ZgFNifW79CHCihJTuh+yzwL7X1Di5MwnLAsYWVYxihanFhfnphsZ6aUW ZSYXF+fn6eWllmxiBKaHg1t+W+1gPPjc8RCjAAejEg9vuvfcKCHWxLLiytxDjBIczEoivHWR QCHelMTKqtSi/Pii0pzU4kOM0hwsSuK8TmkWUUIC6YklqdmpqQWpRTBZJg5OqQZGi9vPJ3bL zdlbxl+VpMCdcOe22zurcs2yQ2V7doixzlx6f+mjKSKb4x9XPfXe2llVIOp+sXXt9qh9zHvE z4YtbDjyefUOb8XIN++aLqRelk/QLXsf7JOQd1PJbfm1RSKHt9lNeKoh+jT0q+o668UHnvP7 u15n00rmXqZj9bROeckR9T2T+P/dUmIpzkg01GIuKk4EAM/LvsELAwAA X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v6] Configurable Link State Change (LSC) detection mode X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is possible to change LSC detection mode to polling or interrupt mode for DPDK interfaces. The default is polling mode. To set interrupt mode, option dpdk-lsc-interrupt has to be set to true. In polling mode more processor time is needed, since the OVS repeatedly reads the link state with a short period. It can lead to packet loss for certain systems. In interrupt mode the hardware itself triggers an interrupt when link state change happens, so less processing time needs for the OVS. For detailed description and usage see the dpdk install documentation. Signed-off-by: Robert Mulik --- v5 -> v6: - DPDK install documentation updated. - Status of lsc_interrupt_mode of DPDK interfaces can be read by command ovs-appctl dpif/show. - It was suggested to check if the HW supports interrupt mode, but it is not possible to do without DPDK code change, so it is skipped from this patch. --- Documentation/intro/install/dpdk.rst | 33 +++++++++++++++++++++++++++++++++ lib/netdev-dpdk.c | 24 ++++++++++++++++++++++-- vswitchd/vswitch.xml | 17 +++++++++++++++++ 3 files changed, 72 insertions(+), 2 deletions(-) -- 1.9.1 diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst index ed358d5..eb1bc7b 100644 --- a/Documentation/intro/install/dpdk.rst +++ b/Documentation/intro/install/dpdk.rst @@ -628,6 +628,39 @@ The average number of packets per output batch can be checked in PMD stats:: $ ovs-appctl dpif-netdev/pmd-stats-show +Link State Change (LSC) detection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two methods to get the information when Link State Change (LSC) +happens on a network interface: by polling or interrupt. + +With the polling method, the main process checks the link state on a +fixed interval. This fixed interval determines how fast a link change is +detected. + +If interrupts are used to get LSC information, the hardware itself triggers +an interrupt when link state change happens, the interrupt thread wakes up +from sleep, updates the information, and goes back to sleep mode. When no link +state change happens (most of the time), the thread remains in sleep mode and +doesn`t use processor time at all. The disadvantage of this method is that +when an interrupt happens, the processor has to handle it immediately, so it +puts the currently running process to background, handles the interrupt, and +takes the background process back. + +Note that not all PMD drivers support LSC interrupts. + +The default configuration is polling mode. To set interrupt mode, option +``dpdk-lsc-interrupt`` has to be set to ``true``. + +Command to set interrupt mode for a specific interface:: + $ ovs-vsctl set interface options:dpdk-lsc-interrupt=true + +Command to set polling mode for a specific interface:: + $ ovs-vsctl set interface options:dpdk-lsc-interrupt=false + +Command to remove ``dpdk-lsc-interrupt`` option:: + $ ovs-vsctl remove interface options dpdk-lsc-interrupt + Limitations ------------ diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 94fb163..e2794e8 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -433,6 +433,12 @@ struct netdev_dpdk { /* DPDK-ETH hardware offload features, * from the enum set 'dpdk_hw_ol_features' */ uint32_t hw_ol_features; + + /* Properties for link state change detection mode. + * If lsc_interrupt_mode is set to false, poll mode is used, + * otherwise interrupt mode is used. */ + bool requested_lsc_interrupt_mode; + bool lsc_interrupt_mode; ); PADDED_MEMBERS(CACHE_LINE_SIZE, @@ -686,12 +692,14 @@ dpdk_watchdog(void *dummy OVS_UNUSED) } static int -dpdk_eth_dev_queue_setup(struct netdev_dpdk *dev, int n_rxq, int n_txq) +dpdk_eth_dev_port_config(struct netdev_dpdk *dev, int n_rxq, int n_txq) { int diag = 0; int i; struct rte_eth_conf conf = port_conf; + conf.intr_conf.lsc = dev->lsc_interrupt_mode; + /* For some NICs (e.g. Niantic), scatter_rx mode needs to be explicitly * enabled. */ if (dev->mtu > ETHER_MTU) { @@ -801,7 +809,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) n_rxq = MIN(info.max_rx_queues, dev->up.n_rxq); n_txq = MIN(info.max_tx_queues, dev->up.n_txq); - diag = dpdk_eth_dev_queue_setup(dev, n_rxq, n_txq); + diag = dpdk_eth_dev_port_config(dev, n_rxq, n_txq); if (diag) { VLOG_ERR("Interface %s(rxq:%d txq:%d) configure error: %s", dev->up.name, n_rxq, n_txq, rte_strerror(-diag)); @@ -897,6 +905,7 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->flags = 0; dev->requested_mtu = ETHER_MTU; dev->max_packet_len = MTU_TO_FRAME_LEN(dev->mtu); + dev->requested_lsc_interrupt_mode = 0; ovsrcu_index_init(&dev->vid, -1); dev->vhost_reconfigured = false; dev->attached = false; @@ -1320,6 +1329,8 @@ netdev_dpdk_get_config(const struct netdev *netdev, struct smap *args) } else { smap_add(args, "rx_csum_offload", "false"); } + smap_add(args, "lsc_interrupt_mode", + dev->lsc_interrupt_mode?"true":"false"); } ovs_mutex_unlock(&dev->mutex); @@ -1520,6 +1531,12 @@ netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args, goto out; } + bool lsc_interrupt_mode = smap_get_bool(args, "dpdk-lsc-interrupt", false); + if (dev->requested_lsc_interrupt_mode != lsc_interrupt_mode) { + dev->requested_lsc_interrupt_mode = lsc_interrupt_mode; + netdev_request_reconfigure(netdev); + } + rx_fc_en = smap_get_bool(args, "rx-flow-ctrl", false); tx_fc_en = smap_get_bool(args, "tx-flow-ctrl", false); autoneg = smap_get_bool(args, "flow-ctrl-autoneg", false); @@ -3546,6 +3563,7 @@ netdev_dpdk_reconfigure(struct netdev *netdev) if (netdev->n_txq == dev->requested_n_txq && netdev->n_rxq == dev->requested_n_rxq && dev->mtu == dev->requested_mtu + && dev->lsc_interrupt_mode == dev->requested_lsc_interrupt_mode && dev->rxq_size == dev->requested_rxq_size && dev->txq_size == dev->requested_txq_size && dev->socket_id == dev->requested_socket_id) { @@ -3561,6 +3579,8 @@ netdev_dpdk_reconfigure(struct netdev *netdev) goto out; } + dev->lsc_interrupt_mode = dev->requested_lsc_interrupt_mode; + netdev->n_txq = dev->requested_n_txq; netdev->n_rxq = dev->requested_n_rxq; diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 0c6a43d..3c9e637 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -3631,6 +3631,23 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \ + + +

+ Set this value to true to configure interrupt mode for + Link State Change (LSC) detection instead of poll mode for the DPDK + interface. +

+

+ If this value is not set, poll mode is configured. +

+

+ This parameter has an effect only on netdev dpdk interfaces. +

+ + + The overall purpose of these columns is described under Common Columns at the beginning of this document.