From patchwork Fri Feb 2 14:05:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?R=C3=B3bert_Mulik?= X-Patchwork-Id: 868585 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="I3tEsxcO"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="lrFSxUO5"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zXzKC74PTz9t0m for ; Sat, 3 Feb 2018 01:05:35 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 15DA4D20; Fri, 2 Feb 2018 14:05:33 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 53A47D10 for ; Fri, 2 Feb 2018 14:05:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DD967108 for ; Fri, 2 Feb 2018 14:05:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=ericsson.com; s=mailgw201801; c=relaxed/simple; q=dns/txt; i=@ericsson.com; t=1517580328; h=From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=T79+VK61NfRr5dt6GMR+/GS9gWMdl9pYTLaAR8zfLoM=; b=I3tEsxcOiam3jCaY8sHowBInxTmY+/fT4i7D7qV+7nBvs2di4pQ4dpdNnB/psTFz UJEuc4s705HJcED15d6jvl7xC7VsIuseBR2c17GTH4bMaEaa81B3WWcDogEJKtii y/W5EWW1P6bQed+txo62TrRvLlBExDG50P4ydY99dtE=; X-AuditID: c1b4fb30-799639c000004778-30-5a7470282fcf Received: from ESESSHC020.ericsson.se (Unknown_Domain [153.88.183.78]) by sesbmg22.ericsson.net (Symantec Mail Security) with SMTP id BD.A5.18296.820747A5; Fri, 2 Feb 2018 15:05:28 +0100 (CET) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.78) with Microsoft SMTP Server (TLS) id 14.3.352.0; Fri, 2 Feb 2018 15:05:17 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=T79+VK61NfRr5dt6GMR+/GS9gWMdl9pYTLaAR8zfLoM=; b=lrFSxUO5tkwwi8uKQpgm0BHcq0kHPPpOcU6OI9/5AzX41hPQ5M45+UjBGMO/RTDjy4mumSpSFfdCGf6U9wMpSSMk3SIrVI3LJpXPfOwmK97ZXdJuOh/+Axbz2vsYnBPM5LBdBNMGjVjTBXJCd2v2sF4Dlt0meRS35wdxDWVMfFM= Received: from HE1PR07MB1212.eurprd07.prod.outlook.com (10.163.178.30) by HE1PR07MB1099.eurprd07.prod.outlook.com (10.163.177.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.464.6; Fri, 2 Feb 2018 14:05:16 +0000 Received: from HE1PR07MB1212.eurprd07.prod.outlook.com ([fe80::5c9f:996f:a148:b7a7]) by HE1PR07MB1212.eurprd07.prod.outlook.com ([fe80::5c9f:996f:a148:b7a7%13]) with mapi id 15.20.0464.008; Fri, 2 Feb 2018 14:05:16 +0000 From: =?utf-8?q?R=C3=B3bert_Mulik?= To: "dev@openvswitch.org" Thread-Topic: [ovs-dev, v4] netdev-dpdk: Configurable Link State Change (LSC) detection mode Thread-Index: AdOcLoPsb8+NXBRCT7Shf/zFxDGM8w== Date: Fri, 2 Feb 2018 14:05:15 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=robert.mulik@ericsson.com; x-originating-ip: [89.135.192.225] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR07MB1099; 7:fP7WqJqq/WhS/mzgCO+uLr2SIIX1QNgTbFIGOrhh0SsYRbUhYuUX7MB3hf+wz+W9Rb6q4xx26vZVGXmi3DSAwP3VaArkALA5yD0+HDazLflrM3VAWb/osLK7pHfHi60/faXwxYhZbka9RFEfKmHarVhBQJX7nDPiZDwq7LZRgnWCyj+ZeJ0U9jUpBlCIdzsE5yVIfpKVGlwOLu1gVOcOyXHAAmZ9F40CqL/hv99wCyK1lXOfFi6VooIEwP8ajVXS x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: de180aa6-f2a1-4ff6-3501-08d56a45fc4e x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020); SRVR:HE1PR07MB1099; x-ms-traffictypediagnostic: HE1PR07MB1099: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(37575265505322)(7411616537696)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231101)(2400082)(944501161)(6041288)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(6072148)(201708071742011); SRVR:HE1PR07MB1099; BCL:0; PCL:0; RULEID:; SRVR:HE1PR07MB1099; x-forefront-prvs: 05715BE7FD x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39380400002)(39860400002)(376002)(346002)(396003)(366004)(189003)(199004)(1730700003)(2351001)(74316002)(66066001)(86362001)(59450400001)(6506007)(7696005)(102836004)(305945005)(5660300001)(7736002)(6916009)(26005)(316002)(6436002)(186003)(5640700003)(9686003)(2906002)(2900100001)(85182001)(99286004)(6116002)(53936002)(85202003)(3846002)(55016002)(81166006)(575784001)(25786009)(33656002)(5890100001)(14454004)(105586002)(2501003)(5250100002)(3660700001)(478600001)(81156014)(8936002)(97736004)(106356001)(8676002)(3280700002)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR07MB1099; H:HE1PR07MB1212.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: s1OX0PL59hbjeFDMVsr1EsWufoPnLs7pF0x9pq3UtX1kYOpY7WHi+u5TZXVJD5duqNti8kypng9nddxH1CCbDA== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: de180aa6-f2a1-4ff6-3501-08d56a45fc4e X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Feb 2018 14:05:15.8720 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR07MB1099 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA02SXUhTYRjHec85m8fl6HV+PWhdNMPKclPZRRchGRWVWJKENk1betChTtmZ 0rJQDItcRWIOnYEShmVqXzNnKNbM/DaQWTi/TQ3DLvSmQpltexd093v+z//5enlZWrLFBLNq jY7TalS5UqGIqU3uOBuxv0CnjGyYwof7hrvoo+jUyuQ2SkBK0ZFMLlddxGnlMZdF2T0rlVRB WcnVMccTqhTNF1cgbxawAr6slnlVIBErwb0IjPUOIQn6ESxtNVKugMEbFNxuKUckY6Rgrcbm sS0iuDW6TbmaCfExmPzzUeBifywH691RxsV++CK0zzxiiJ4GlvEemrAMZquW3X4G74WuVrOb xTgVtl8tuHsiHAi/hlrcTOMgsC/VU2RxDI1dn2nCAbD6zSEg/lSYn2v16FKo+WEXEN4N4/UG 9wWAzRTccawLSUIG7ZU/EeF4qDP3eRHTAILKlTYvkggH22y/pyAHljs3PBOU0G3upEhBDQ2G lmmniXUGu8CwFkn0CQE0mizutSWYg6bWcs+0YmieeUk9QIdM/11ncpbT+AC8eCcn8h54aFjw MrkfxhcGa5eYBsQ0owCe46/kZUVHyzitOoPn8zUyDad7jZx/4oN5M9KCVr/HWhFmkdRH3KvU KSUCVRGvz7MiYGmpv/htvFMSZ6r01zhtfrq2MJfjrSiEZaRB4sHTYqUEZ6l0XA7HFXDaf1mK 9Q4uRca4Z+ttEVNxA3z/YLq5IQkkwzZLLH9u+nzdzonxTZ+o8o43kBpzbzF9xFhdHe4ojEtI PLiZYh/6fbN05H7o87kdi3b7p26DWfHU2zRUe6ldrg/t3Bd4/MyNqq+25eSSxKz4mBRFqG+Y vilMN3bC733Ihd60JIXxZMZP8fVJ9WMpw2erosJpLa/6C1ZDnGYPAwAA X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [ovs-dev, v4] netdev-dpdk: Configurable Link State Change (LSC) detection mode X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is possible to change LSC detection mode to polling or interrupt mode for DPDK interfaces. The default is polling mode. To set interrupt mode, option dpdk-lsc-interrupt has to be set to true. In polling mode more processor time is needed, since the OVS repeatedly reads the link state with a short period. It can lead to packet loss for certain systems. In interrupt mode the hardware itself triggers an interrupt when link state change happens, so less processing time needs for the OVS. It is not possible to enable the interrupt mode on all hardware. For detailed description and usage see the dpdk install documentation. Signed-off-by: Robert Mulik Reviewed-by: Ilya Maximets Reviewed-by: Ian Stokes Reviewed-by: Eelco Chaudron --- Documentation/intro/install/dpdk.rst | 48 ++++++++++++++++++++++++++++++++++++ lib/netdev-dpdk.c | 42 ++++++++++++++++++++++++++++--- lib/netdev-dpdk.h | 2 ++ vswitchd/bridge.c | 8 ++++++ vswitchd/vswitch.xml | 45 +++++++++++++++++++++++++++++++++ 5 files changed, 142 insertions(+), 3 deletions(-) -- 1.9.1 diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst index ed358d5..14a6684 100644 --- a/Documentation/intro/install/dpdk.rst +++ b/Documentation/intro/install/dpdk.rst @@ -628,6 +628,54 @@ The average number of packets per output batch can be checked in PMD stats:: $ ovs-appctl dpif-netdev/pmd-stats-show +Link State Change (LSC) detection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two methods to get the information when Link State Change (LSC) +happens on a network interface: by polling or interrupt. + +With polling method, a process is running in the background and repeatedly +reads the link state with a short period. It continuously needs processor time +and between 2 reading periods it can`t see the link state change, therefore +the reaction time depends on the polling period. With higher rate, more +processor time is needed. Another problem with the poll mode is that on some +hardware a polling cycle takes too much time, which (in the end) leads to +packet loss for certain systems. + +If interrupts are used to get LSC information, the hardware itself triggers +an interrupt when link state change happens, the thread wakes up from sleep, +updates the information, and goes back to sleep mode. When no link state +change happens (most of the time), the thread remains in sleep mode and +doesn`t use processor time at all. The disadvantage of this method is that +when interrupt happens, the processor has to handle it immediately, so it +puts the currently running process to background, handles the interrupt, and +takes the background process back. Another disadvantage is that some hardware +can`t be configured to generate LSC interrupts. + +The default configuration is polling mode. To set interrupt mode, option +dpdk-lsc-interrupt has to be set to true. + +Global settings + +Command to set interrupt mode for all interfaces: +ovs-vsctl set Open_vSwitch . other_config:dpdk-lsc-interrupt=true + +Command to set polling mode for all interfaces: +ovs-vsctl set Open_vSwitch . other_config:dpdk-lsc-interrupt=false +or: +ovs-vsctl remove Open_vSwitch . other_config dpdk-lsc-interrupt + +Interface specific settings (override global settings) + +Command to set interrupt mode for a specific interface: +ovs-vsctl set interface options:dpdk-lsc-interrupt=true + +Command to set polling mode for a specific interface: +ovs-vsctl set interface options:dpdk-lsc-interrupt=false + +Command to reset to globally defined mode for a specific interface: +ovs-vsctl remove interface options dpdk-lsc-interrupt + Limitations ------------ diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 94fb163..73d0d4b 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -148,7 +148,7 @@ typedef uint16_t dpdk_port_t; #define VHOST_ENQ_RETRY_NUM 8 #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ) -static const struct rte_eth_conf port_conf = { +static struct rte_eth_conf port_conf = { .rxmode = { .mq_mode = ETH_MQ_RX_RSS, .split_hdr_size = 0, @@ -167,6 +167,10 @@ static const struct rte_eth_conf port_conf = { .txmode = { .mq_mode = ETH_MQ_TX_NONE, }, + .intr_conf = { + /* LSC interrupt mode disabled, polling mode used. */ + .lsc = 0, + }, }; /* @@ -433,6 +437,12 @@ struct netdev_dpdk { /* DPDK-ETH hardware offload features, * from the enum set 'dpdk_hw_ol_features' */ uint32_t hw_ol_features; + + /* Properties for link state change detection mode. + * If lsc_interrupt_mode is set to false, poll mode is used, + * otherwise interrupt mode is used. */ + bool requested_lsc_interrupt_mode; + bool lsc_interrupt_mode; ); PADDED_MEMBERS(CACHE_LINE_SIZE, @@ -459,6 +469,18 @@ int netdev_dpdk_get_vid(const struct netdev_dpdk *dev); struct ingress_policer * netdev_dpdk_get_ingress_policer(const struct netdev_dpdk *dev); +void +netdev_dpdk_set_default_lsc_detect_mode(bool enabled) +{ + port_conf.intr_conf.lsc = enabled; +} + +bool +netdev_dpdk_get_def_lsc_int_mode_enabled(void) +{ + return port_conf.intr_conf.lsc; +} + static bool is_dpdk_class(const struct netdev_class *class) { @@ -686,12 +708,14 @@ dpdk_watchdog(void *dummy OVS_UNUSED) } static int -dpdk_eth_dev_queue_setup(struct netdev_dpdk *dev, int n_rxq, int n_txq) +dpdk_eth_dev_port_config(struct netdev_dpdk *dev, int n_rxq, int n_txq) { int diag = 0; int i; struct rte_eth_conf conf = port_conf; + conf.intr_conf.lsc = dev->lsc_interrupt_mode; + /* For some NICs (e.g. Niantic), scatter_rx mode needs to be explicitly * enabled. */ if (dev->mtu > ETHER_MTU) { @@ -801,7 +825,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) n_rxq = MIN(info.max_rx_queues, dev->up.n_rxq); n_txq = MIN(info.max_tx_queues, dev->up.n_txq); - diag = dpdk_eth_dev_queue_setup(dev, n_rxq, n_txq); + diag = dpdk_eth_dev_port_config(dev, n_rxq, n_txq); if (diag) { VLOG_ERR("Interface %s(rxq:%d txq:%d) configure error: %s", dev->up.name, n_rxq, n_txq, rte_strerror(-diag)); @@ -897,6 +921,8 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->flags = 0; dev->requested_mtu = ETHER_MTU; dev->max_packet_len = MTU_TO_FRAME_LEN(dev->mtu); + dev->requested_lsc_interrupt_mode = + netdev_dpdk_get_def_lsc_int_mode_enabled(); ovsrcu_index_init(&dev->vid, -1); dev->vhost_reconfigured = false; dev->attached = false; @@ -1520,6 +1546,13 @@ netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args, goto out; } + bool lsc_interrupt_mode = smap_get_bool(args, "dpdk-lsc-interrupt", + netdev_dpdk_get_def_lsc_int_mode_enabled()); + if (dev->requested_lsc_interrupt_mode != lsc_interrupt_mode) { + dev->requested_lsc_interrupt_mode = lsc_interrupt_mode; + netdev_request_reconfigure(netdev); + } + rx_fc_en = smap_get_bool(args, "rx-flow-ctrl", false); tx_fc_en = smap_get_bool(args, "tx-flow-ctrl", false); autoneg = smap_get_bool(args, "flow-ctrl-autoneg", false); @@ -3546,6 +3579,7 @@ netdev_dpdk_reconfigure(struct netdev *netdev) if (netdev->n_txq == dev->requested_n_txq && netdev->n_rxq == dev->requested_n_rxq && dev->mtu == dev->requested_mtu + && dev->lsc_interrupt_mode == dev->requested_lsc_interrupt_mode && dev->rxq_size == dev->requested_rxq_size && dev->txq_size == dev->requested_txq_size && dev->socket_id == dev->requested_socket_id) { @@ -3561,6 +3595,8 @@ netdev_dpdk_reconfigure(struct netdev *netdev) goto out; } + dev->lsc_interrupt_mode = dev->requested_lsc_interrupt_mode; + netdev->n_txq = dev->requested_n_txq; netdev->n_rxq = dev->requested_n_rxq; diff --git a/lib/netdev-dpdk.h b/lib/netdev-dpdk.h index b7d02a7..8eb51bc 100644 --- a/lib/netdev-dpdk.h +++ b/lib/netdev-dpdk.h @@ -27,6 +27,8 @@ struct dp_packet; void netdev_dpdk_register(void); void free_dpdk_buf(struct dp_packet *); +void netdev_dpdk_set_def_lsc_int_mode_enabled(bool enabled); +bool netdev_dpdk_get_def_lsc_int_mode_enabled(void); #else diff --git a/vswitchd/bridge.c b/vswitchd/bridge.c index f44f950..87608a5 100644 --- a/vswitchd/bridge.c +++ b/vswitchd/bridge.c @@ -68,6 +68,9 @@ #include "lib/vswitch-idl.h" #include "xenserver.h" #include "vlan-bitmap.h" +#ifdef DPDK_NETDEV +#include "./lib/netdev-provider.h" +#endif VLOG_DEFINE_THIS_MODULE(bridge); @@ -601,6 +604,11 @@ bridge_reconfigure(const struct ovsrec_open_vswitch *ovs_cfg) ofproto_set_vlan_limit(smap_get_int(&ovs_cfg->other_config, "vlan-limit", LEGACY_MAX_VLAN_HEADERS)); +#ifdef DPDK_NETDEV + netdev_dpdk_set_def_lsc_int_mode_enabled( + smap_get_bool(&ovs_cfg->other_config, "dpdk-lsc-interrupt", false)); +#endif + ofproto_set_threads( smap_get_int(&ovs_cfg->other_config, "n-handler-threads", 0), smap_get_int(&ovs_cfg->other_config, "n-revalidator-threads", 0)); diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 0c6a43d..91a9003 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -320,6 +320,29 @@

+ +

+ Set this value to true to configure interrupt mode for + Link State Change (LSC) detection instead of poll mode for DPDK + interfaces. +

+

+ The default value is false. +

+

+ If this value is false at startup, poll mode is used for + all netdev dpdk interfaces. +

+

+ This value can be overridden for a specific interface in the options + section of that interface. +

+

+ This parameter has an effect only on netdev dpdk interfaces. +

+ +

@@ -3631,6 +3654,28 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \ + + +

+ Set this value to true to configure interrupt mode for + Link State Change (LSC) detection instead of poll mode for the DPDK + interface. +

+

+ If this value is not set, the value is taken from the global + settings. +

+

+ If this value is set, the global LSC interrupt settings are + overridden. +

+

+ This parameter has an effect only on netdev dpdk interfaces. +

+ + + The overall purpose of these columns is described under Common Columns at the beginning of this document.