From patchwork Thu Feb 15 14:44:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?R=C3=B3bert_Mulik?= X-Patchwork-Id: 873916 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="EEPunaY4"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ericsson.com header.i=@ericsson.com header.b="Tb0rb7uF"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zhzZJ6mVwz9sRm for ; Fri, 16 Feb 2018 01:44:40 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id D94F910D6; Thu, 15 Feb 2018 14:44:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 810F51073 for ; Thu, 15 Feb 2018 14:44:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9F31DE4 for ; Thu, 15 Feb 2018 14:44:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=ericsson.com; s=mailgw201801; c=relaxed/simple; q=dns/txt; i=@ericsson.com; t=1518705873; h=From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=iQppRLHIwOgTTU8BWnawGP3aDqvMDzrUa76fTb0MgBs=; b=EEPunaY4Fi/pNyjdX6GO7+bLU8gcbOZu7W3+i+jUnQrHIfLVo/+ZDTYozko/1Qyo CjL4rooqjJN0MsuErRsq13GLukp0PITVr7sDtoDmftGn5zTI3G0apct65pauRx83 GIk4/ymrr6x4qhuKyJjpHtsRXnDijK9ZKZuSmFrFxGY=; X-AuditID: c1b4fb30-399ff70000004778-76-5a859cd10a81 Received: from ESESSHC001.ericsson.se (Unknown_Domain [153.88.183.21]) by sesbmg22.ericsson.net (Symantec Mail Security) with SMTP id 0C.FE.18296.1DC958A5; Thu, 15 Feb 2018 15:44:33 +0100 (CET) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.21) with Microsoft SMTP Server (TLS) id 14.3.352.0; Thu, 15 Feb 2018 15:44:33 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=iQppRLHIwOgTTU8BWnawGP3aDqvMDzrUa76fTb0MgBs=; b=Tb0rb7uFWZv4D3zeQqqCCNTLdHOcT5sInB92OBPhky+1lJ5XLaGkChPJKTFaOI8doX4Z+6bqNXhdyn1zeBrtJT8qa4mDSuxQqRGsLBvvJL9XGYrPUXOg4JsQY5xnc77LKIKBdZtye2XEGtCLX49VfzQhKjnGJfP3+gx9PcYMdO8= Received: from AM3PR07MB1202.eurprd07.prod.outlook.com (10.163.188.28) by AM3PR07MB1201.eurprd07.prod.outlook.com (10.163.188.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.527.6; Thu, 15 Feb 2018 14:44:32 +0000 Received: from AM3PR07MB1202.eurprd07.prod.outlook.com ([fe80::7d22:2aad:204c:dfc1]) by AM3PR07MB1202.eurprd07.prod.outlook.com ([fe80::7d22:2aad:204c:dfc1%7]) with mapi id 15.20.0527.006; Thu, 15 Feb 2018 14:44:32 +0000 From: =?utf-8?q?R=C3=B3bert_Mulik?= To: "dev@openvswitch.org" Thread-Topic: [PATCH v5] Configurable Link State Change (LSC) detection mode Thread-Index: AdOma0uzjvztF7Q/SCCJ7FckAguo0w== Date: Thu, 15 Feb 2018 14:44:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [89.135.192.225] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM3PR07MB1201; 7:St7klpe6zHo/dwM1gVgccogHRhVsru5rAI9bk3fCYuHeyK3lknr6nUptVjfOLiK/C0KsxsyVix+Yvw9bWdXlbKjytQgts2ZcS17KWpki06lFIjCq39HMiol3kw+YpMSDb15TmzOHCShBP0xgaXROBydlC25pDpDbTtgdJYb2BpeB+OEHGlaza1Up7tvvQQS6ilTMluNMzq3jUas3zDcHQrb3gSrcR4UeRAidHPOT5HUZjHw/AyREBwIHjTK2+rbc x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 515af30c-788f-468b-9512-08d574829ffa x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020); SRVR:AM3PR07MB1201; x-ms-traffictypediagnostic: AM3PR07MB1201: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(37575265505322); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001026)(6040501)(2401047)(5005006)(8121501046)(10201501046)(3231101)(944501161)(93006095)(93001095)(3002001)(6041288)(20161123560045)(20161123562045)(20161123564045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM3PR07MB1201; BCL:0; PCL:0; RULEID:; SRVR:AM3PR07MB1201; x-forefront-prvs: 058441C12A x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(376002)(39860400002)(39380400002)(346002)(396003)(366004)(199004)(189003)(106356001)(74316002)(3660700001)(316002)(7736002)(2351001)(1730700003)(81166006)(8676002)(81156014)(2900100001)(5640700003)(6506007)(26005)(186003)(6436002)(305945005)(68736007)(59450400001)(102836004)(85202003)(33656002)(8936002)(6916009)(6116002)(14454004)(97736004)(7696005)(66066001)(86362001)(85182001)(5660300001)(105586002)(2501003)(5250100002)(5890100001)(3846002)(6346003)(2906002)(53936002)(9686003)(99286004)(25786009)(3280700002)(55016002)(478600001)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM3PR07MB1201; H:AM3PR07MB1202.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=robert.mulik@ericsson.com; x-microsoft-antispam-message-info: U79Xi/8YUnDZ2KRmIN3o2t0dxctzu4KhdcH09Xfkr7NuLxMFAukaabBlAvg9XjcP+0dLlDtMSiW7gr4e2c2sCQ== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 515af30c-788f-468b-9512-08d574829ffa X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Feb 2018 14:44:31.9771 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR07MB1201 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrOIsWRmVeSWpSXmKPExsUyM2K7qO7FOa1RBvPemlgcPb2H2YHR49nN /4wBjFFcNimpOZllqUX6dglcGd8+TmYu2OZX8fXySqYGxi0+XYycHBICJhLt6zexg9hCAocZ JTY+Duxi5AKyTzBK/Hv/kwXEYRH4xCRxZt4PZoiqiUwS37vNIaoeMUo8P7OZCSTBJuAkcfPn EVYQW0RAX+JQz1kWEFtYwF3i/LerUHEfifmLdrBA2HoSNxf1gfWyCKhKXG7cxQhi8wrESEz+ PB/MZhQQk/h+ag1YDbOAuMStJ/OZIM4WkFiy5zwzhC0q8fLxP1YIW0lixqtbULasxKX53Ywg h0oIbGGSODttIjtEQk9i68S3jBC2r8T650fYIIqAfp60dwZUt5bEn+1LgbZxANnZEnPbVCFM K4l5Z2IhKmYzS5xfogthy0g8OreXBcJuZZPY9ZsFElipEsvXtkL9EiPx4P5aqJurJbbN+cU4 gVFnFpLXZgFtYBbQlFi/Sx8irCgxpfsh+yxwqAhKnJz5hGUBI8sqRtHi1OKk3HQjI73Uoszk 4uL8PL281JJNjMDkcHDLb4MdjC+fOx5iFOBgVOLhjc1sjRJiTSwrrsw9xCjBwawkwvsxCijE m5JYWZValB9fVJqTWnyIUZqDRUmc96Qnb5SQQHpiSWp2ampBahFMlomDU6qBcWnQI+Yr3nMT K627Cmwj5q8IveHyQfn3jYu/DW9yp1V+PZHLEHzOKe6XqUX4x7KNW3pE3Wc5cBYFzYhhuRd1 OJhr0pmXlxVmXf7mL+HOXJ3v+GBJ6DSfi0b3Et9XnLL883i3qKfe/fg5mht1P8qUKKm4V7M9 Y3JxZtOO3XJY0LuhUowt0kdQiaU4I9FQi7moOBEAZrmHcgoDAAA= X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v5] Configurable Link State Change (LSC) detection mode X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is possible to change LSC detection mode to polling or interrupt mode for DPDK interfaces. The default is polling mode. To set interrupt mode, option dpdk-lsc-interrupt has to be set to true. In polling mode more processor time is needed, since the OVS repeatedly reads the link state with a short period. It can lead to packet loss for certain systems. In interrupt mode the hardware itself triggers an interrupt when link state change happens, so less processing time needs for the OVS. For detailed description and usage see the dpdk install documentation. Signed-off-by: Robert Mulik --- Documentation/intro/install/dpdk.rst | 35 +++++++++++++++++++++++++++++++++++ lib/netdev-dpdk.c | 22 ++++++++++++++++++++-- vswitchd/vswitch.xml | 17 +++++++++++++++++ 3 files changed, 72 insertions(+), 2 deletions(-) -- 1.9.1 diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst index ed358d5..e3872e7 100644 --- a/Documentation/intro/install/dpdk.rst +++ b/Documentation/intro/install/dpdk.rst @@ -628,6 +628,41 @@ The average number of packets per output batch can be checked in PMD stats:: $ ovs-appctl dpif-netdev/pmd-stats-show +Link State Change (LSC) detection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two methods to get the information when Link State Change (LSC) +happens on a network interface: by polling or interrupt. + +With the polling method, the main process checks the link state on a +fixed interval. This fixed interval determines how fast a link change is +detected. Another problem with the poll mode is that on some hardware a +polling cycle takes too much time, which (in the end) leads to packet loss +for certain systems. + +If interrupts are used to get LSC information, the hardware itself triggers +an interrupt when link state change happens, the interrupt thread wakes up +from sleep, updates the information, and goes back to sleep mode. When no link +state change happens (most of the time), the thread remains in sleep mode and +doesn`t use processor time at all. The disadvantage of this method is that +when an interrupt happens, the processor has to handle it immediately, so it +puts the currently running process to background, handles the interrupt, and +takes the background process back. + +Note that not all PMD drivers support LSC interrupts. + +The default configuration is polling mode. To set interrupt mode, option +``dpdk-lsc-interrupt`` has to be set to ``true``. + +Command to set interrupt mode for a specific interface:: + $ ovs-vsctl set interface options:dpdk-lsc-interrupt=true + +Command to set polling mode for a specific interface:: + $ ovs-vsctl set interface options:dpdk-lsc-interrupt=false + +Command to remove ``dpdk-lsc-interrupt`` option:: + $ ovs-vsctl remove interface options dpdk-lsc-interrupt + Limitations ------------ diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 94fb163..d092ef1 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -433,6 +433,12 @@ struct netdev_dpdk { /* DPDK-ETH hardware offload features, * from the enum set 'dpdk_hw_ol_features' */ uint32_t hw_ol_features; + + /* Properties for link state change detection mode. + * If lsc_interrupt_mode is set to false, poll mode is used, + * otherwise interrupt mode is used. */ + bool requested_lsc_interrupt_mode; + bool lsc_interrupt_mode; ); PADDED_MEMBERS(CACHE_LINE_SIZE, @@ -686,12 +692,14 @@ dpdk_watchdog(void *dummy OVS_UNUSED) } static int -dpdk_eth_dev_queue_setup(struct netdev_dpdk *dev, int n_rxq, int n_txq) +dpdk_eth_dev_port_config(struct netdev_dpdk *dev, int n_rxq, int n_txq) { int diag = 0; int i; struct rte_eth_conf conf = port_conf; + conf.intr_conf.lsc = dev->lsc_interrupt_mode; + /* For some NICs (e.g. Niantic), scatter_rx mode needs to be explicitly * enabled. */ if (dev->mtu > ETHER_MTU) { @@ -801,7 +809,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) n_rxq = MIN(info.max_rx_queues, dev->up.n_rxq); n_txq = MIN(info.max_tx_queues, dev->up.n_txq); - diag = dpdk_eth_dev_queue_setup(dev, n_rxq, n_txq); + diag = dpdk_eth_dev_port_config(dev, n_rxq, n_txq); if (diag) { VLOG_ERR("Interface %s(rxq:%d txq:%d) configure error: %s", dev->up.name, n_rxq, n_txq, rte_strerror(-diag)); @@ -897,6 +905,7 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->flags = 0; dev->requested_mtu = ETHER_MTU; dev->max_packet_len = MTU_TO_FRAME_LEN(dev->mtu); + dev->requested_lsc_interrupt_mode = 0; ovsrcu_index_init(&dev->vid, -1); dev->vhost_reconfigured = false; dev->attached = false; @@ -1520,6 +1529,12 @@ netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args, goto out; } + bool lsc_interrupt_mode = smap_get_bool(args, "dpdk-lsc-interrupt", false); + if (dev->requested_lsc_interrupt_mode != lsc_interrupt_mode) { + dev->requested_lsc_interrupt_mode = lsc_interrupt_mode; + netdev_request_reconfigure(netdev); + } + rx_fc_en = smap_get_bool(args, "rx-flow-ctrl", false); tx_fc_en = smap_get_bool(args, "tx-flow-ctrl", false); autoneg = smap_get_bool(args, "flow-ctrl-autoneg", false); @@ -3546,6 +3561,7 @@ netdev_dpdk_reconfigure(struct netdev *netdev) if (netdev->n_txq == dev->requested_n_txq && netdev->n_rxq == dev->requested_n_rxq && dev->mtu == dev->requested_mtu + && dev->lsc_interrupt_mode == dev->requested_lsc_interrupt_mode && dev->rxq_size == dev->requested_rxq_size && dev->txq_size == dev->requested_txq_size && dev->socket_id == dev->requested_socket_id) { @@ -3561,6 +3577,8 @@ netdev_dpdk_reconfigure(struct netdev *netdev) goto out; } + dev->lsc_interrupt_mode = dev->requested_lsc_interrupt_mode; + netdev->n_txq = dev->requested_n_txq; netdev->n_rxq = dev->requested_n_rxq; diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 0c6a43d..3c9e637 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -3631,6 +3631,23 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \ + + +

+ Set this value to true to configure interrupt mode for + Link State Change (LSC) detection instead of poll mode for the DPDK + interface. +

+

+ If this value is not set, poll mode is configured. +

+

+ This parameter has an effect only on netdev dpdk interfaces. +

+ + + The overall purpose of these columns is described under Common Columns at the beginning of this document.