From patchwork Tue Mar 27 07:54:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shahaf Shuler X-Patchwork-Id: 891411 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b="fv9NZjKP"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 409Nc92pztz9ryG for ; Tue, 27 Mar 2018 18:55:52 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 3AB68DE1; Tue, 27 Mar 2018 07:55:47 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 76DADCE8 for ; Tue, 27 Mar 2018 07:55:46 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0088.outbound.protection.outlook.com [104.47.2.88]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C2C2A51F for ; Tue, 27 Mar 2018 07:55:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=09si+3qNclY7IUnsqBlJiR6qMuF9KvQLoRW9P5iRJRs=; b=fv9NZjKPebSjo7yeRZwvnh65H+pytAXQstuTCte7LIfSowuif1X/R+Uxs3DrZ9UxxZtyQDPV0DRKNPU0zdqyESvRaH4zqtwdj+IKx88asgwi55hCx1mObPx8DSKvTMJpRB9bQ88u9ZbYcm2L9sXD16M7/BidKQsmzaBiY96Jme8= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; Received: from mellanox.com (141.226.120.58) by AM0PR05MB4418.eurprd05.prod.outlook.com (2603:10a6:208:5a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.609.10; Tue, 27 Mar 2018 07:55:40 +0000 From: Shahaf Shuler To: fc@napatech.com, ian.stokes@intel.com Date: Tue, 27 Mar 2018 10:54:41 +0300 Message-Id: X-Mailer: git-send-email 2.12.0 In-Reply-To: <1517209188-16608-2-git-send-email-yliu@fridaylinux.org> References: <1517209188-16608-2-git-send-email-yliu@fridaylinux.org> MIME-Version: 1.0 X-Originating-IP: [141.226.120.58] X-ClientProxiedBy: DB6PR1001CA0041.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:4:55::27) To AM0PR05MB4418.eurprd05.prod.outlook.com (2603:10a6:208:5a::15) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 07aeff71-d7d8-413e-d384-08d593b82319 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:AM0PR05MB4418; X-Microsoft-Exchange-Diagnostics: 1; AM0PR05MB4418; 3:5fQe1U4SbzoxFi3ZgkawAXWUC/WmDPG2sV7uDL15ozXDGbMVUGmQROBfzCHfXCiS4qV+WX/yPP21R1Xs1x+m5vwu8U5hVsoQK1q16IguTPVsiKsF2KaNs1tt7BhYES2cE28F6o8XDEG4QJzL/SRo0RmeWCQWlzqWfKNO0dNadFVlaxL6BBLLt2rry+hUrjk9vWpNKxVKZ43dpkVvfEvoRMAQRT6WCJvkGppDXmfr5A0jFv/fYK6RPMN7vZ1pg/D/; 25:w2FlRSSUwoq/psBUeQZBwqKf1aegTV3o0Uyx6u9yI/XUjCJmcIuVW7KF1qK9EWvp4pvhL1hxVZ/RYxmfHR/2hMzF9s6VCcheGQoJ5dXrywh/06g5vDMnj/tCzYhxivckLQBsM27ZNL2Ts5Ku29wzIR8uxBa1N6nf0nRrmGgmEVufcN8WSfmrnUPnTK14R1tB/g+9c0kaghAhJc+V34LNsUjN/uiSeNNze2IksmM3p6H/Weh+T/3Xg17r+mZqKJCQrJJmxvMOlW2q5aB8LRsQ7LNeSsbGEfynFks7eVD0JUMMeuInbC2oHMPujzaQS+HIqaz+OjV52VWpGd1nA9pSiw==; 31:W8vPBzuZosMKvmWxpo/LPuU55k49x2n1veCMp/CEbiuuEp/lateURdtuV8waIA64d2tb7rK7/5QIVkeO0dKqEJ9kGaKZe7Ii20yIw8e0lomicSImdDFbH5qXXnq7xMoryQZn9zH88OwZIOLnlxt/6T83CiP1hIJtzBiDsxmb7pb0YNSUAcBC0QDWCD9uCUiTDI1iDtFaj00DS7KINfj4XSygVgRXfw0dGYCVio+45Ww= X-MS-TrafficTypeDiagnostic: AM0PR05MB4418: X-Microsoft-Exchange-Diagnostics: 1; AM0PR05MB4418; 20:Y0vXWgpmGEnHTfcNu5tHD5/R1oS0IRzLZxNzivABGmiq8xdoO7juBW6S5PLbGfUV66KPob4RDSVHuGCzK26NuQhvyx6OmA2cUSaBWEXXpslZUwaV3bTv1cYU1Wh8zv1RabyODNPrv6YTr3wE5rIEducWXGZKlwwHVinZYtWlVHhzEL68RXDGhqSNGo5ryt8buFgIn21JRY5qkf4oP+ptxCQibvIoh5JYR0ymouGOcQAVejXemmp0DQP+hECTvEkwYkZYjatz9Ro6rJ1jy6LCNFWNeuhqdI5pUCnyFqKv+mBxo8jZubP7Ov/xVzas26upYWx7AwFfe0CFWOcTAyFS2Ynp0a2oOvXZLBvmfYjLt1QT198+3/Wdc3Ua209baAkPxJDkAll01uf6p5ksOp+hVoxiFZeT3xUOcnHoJjYozUAO1qRvWQ9QkL1Aa0u25pilWwuTCAMf3kShyRiKORHdxsls9DCgCuHVOQrm0Qd9UpLDLKnH8y+xSuFyNJ/yTB3E; 4:8eOmCZDtgLa9TWs5+ARirbF1cgheOCEOMDTlA4XvIpPWU5tEY6EV2XrFNBlxb73Pc9C+Do6rl9NRPEs5muQ2x3cQF4eFWlr1/a5dc+Aw4kRRtBHRmAUPeEzYEKgdL+ekpJ2dNXjTOrCfnuo7S1dS7/VdKdLOT70+4bPobYenYlkGdaAJunhp42wcqap8VV/numkGFqH0KwyJDH2aqw34cCPM7551fRW4VTnjyagkbh1KhNNe5EBGOtdGYLWnZr+a+pof53tn32vplYJ27ZmTBw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3002001)(3231221)(944501327)(52105095)(6055026)(6041310)(20161123564045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(6072148)(201708071742011); SRVR:AM0PR05MB4418; BCL:0; PCL:0; RULEID:; SRVR:AM0PR05MB4418; X-Forefront-PRVS: 0624A2429E X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(39860400002)(366004)(39380400002)(376002)(396003)(199004)(189003)(50226002)(7736002)(68736007)(3846002)(36756003)(6116002)(7696005)(50466002)(51416003)(21086003)(52116002)(59450400001)(386003)(76176011)(16586007)(97736004)(8936002)(486005)(956004)(486005)(478600001)(305945005)(39060400002)(8676002)(25786009)(4326008)(48376002)(186003)(86362001)(26005)(316002)(106356001)(69596002)(16526019)(81156014)(55016002)(2906002)(81166006)(66066001)(105586002)(5660300001)(11346002)(2616005)(53936002)(47776003)(476003)(446003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR05MB4418; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM0PR05MB4418; 23:Ug9Uolxwdj3CfI+FdSSHbKTjJ3Nk8ewVfA41WENo+?= 32ckSpmSY6VdOjLx/zsJ8zDF0abiU7vOjIEpMhKeEM+PoH4EW9fKuhu9HrKaFZfXFTxfYVE6SvIggo2WbliJUoTsB7B79/gMeUCld1aNEZ4MADg5cOAEGxfNAcR8kflUrdWGL/Kh8EYviECYJvNsp1YdvQUKDS/YOn0PZzVq+yXCU0KrbcYy7Y60SyAQJ8E5v5kX3Gb0ka/381GNfEGXMdn/mT+43TNVCtjLpoirioMkdQEuSqLoJc6Sn7CCNIOS76Cjo6R+HJB4Sds3vn3uD+kZsvjAC+V/2BV4qdiKwdSj6qhvM5k/NC8XGeXRzjC5v8jNLqo7LmIl60Rlr79iKYGEm5lKtR5MQgGyQkYQODgkHc3UuuOykD9TvkIU8WWtj/wioBHjcDBGK8YfEt9AF991a7w0eFsui0rFx2Y8SaVtTTjjnriUqQ3D8f3aLfB3yh4rsm0sd/ovOafOENmyiYWvJHQbvvPQFwdUf5rIWT6q+971EGVo8F/DYWWEEnxYn4dA1SDoQClkXTPuRvmiynZIqCjPZreCAJQ1IVcErg5ky3JxN9UCVY8/fHN26SG1O8YcXiuTZCEze8RsyYdpLpwW6FScaH3BSnYrc96Eebzb/7rFvAhqIUqJpYm2S0qxgo9qi+H5V6iOqqtBfkJE7of30g7d3fHrY+kSPwSDk4oWThd2wTzY1FsGQCNrgnDMMTq5MsEsj0F6kVQYVw7YlnDLTYEnN5lARnxqLGzCBz0Ta5Q0/ddkEqJchcIjpxJLwTOWi410WzvNMs9zP/6nK029p+EIZ9tLOvPrz3yG9SHgUdURRaloNFnw0Kh9lWDVJmcGP9f4IJ8+Dj1ilm0gQVJx5ztntxg8fNlMAV1gKXElLDxHZcS/lX41XJtf/67Kiw/dZSJpDYkycLVILlJ7HYJptKacIFtrPNfq9YO2h7v0nzXWQFCzu4i6JXyVuqqn7cHhb010k1Kw07hFHxJAmliJpeZ8T3C4IbRERyHSBmtyLshSd01xZZ6b85c0wV18oaIa2KrM88hhmhszIvoUlRwyZDqQVV05FGjaX8hgt4m5ioBtWd7XqcYjRFoYSNosjCZnnicHHcoQ0Mzvo8/Aui1iWmDzllJfFlriHcZr9aRLlbTXuZTbP8ktCHlMHDRmarmUklmprmG/0eA4IJ8L+hl4kZl437+/O2qVoueh4GTZA== X-Microsoft-Antispam-Message-Info: B+MPHYxWvRfRHKlUDtDsjVWOiE+qYt+JVh4BfwCSFOUfduW2TBF0W1y+BN2ekOwD9sFJPdLq2Q+vca+85aczQlHQ01Q8v02n8MbpkxN7lgXC2lUt3zXI9z7CtGTbFfJgtzLTUQesx8wXWtUeW1tAS22ZebXtDn5WLRAPb5QgYqengQsNY8ET+7/wYo63omUe X-Microsoft-Exchange-Diagnostics: 1; AM0PR05MB4418; 6:a9ve8QP4ere5qEG7gYK2MNUIO1VY9EwmOgUm19BIfKFDW3oNByO51kuAP8XnaIUUYVdHBYTexxj5YdDkp4UdNO/Ka7OsBc0sM8n8fb9v58oFStexAKRQPanf0Co/qAdR/x8BYdnJwxM/ZHyikQwGKwI+hePWc79tmUOPJf1jTsN98+9PDF+uRfqRYQYRnBKL6FwJ08hzDAj6vE/Ig2cy/Q004dQK12kOrm3yxsRpOAD5TMQgZexODTPqlvsXqhYbbe35ayRmHz2iTfdFsgktVtW8B4qzFd8lki6qFVXuWGiCU8ID7BvP1ZlCljm8Xva2/CBwbMcjAXEIEeuJFqL6+TLe2/QIN+B3xXE1qy76ybo3r8nytFXA8/y7GO6SvuWnBWH0rbwq9Dm5YzDHwUleYuz0mZ7XcF4Ti1LieVOavF6APsb7UyXaR79OzuGlRywp3FMZI25uwq7GdOzP/ah+KA==; 5:EH6TWA4ce6jfN/U6Mr8sWo73zmbgzR4r4ycfCdCXiDbX8KcLdtmB2kIIy7s2uH1tD7SX1o7bkTCH/rjGy1gCw5a8eJvtZm+MM6+RkDrfhkWNwx/cWKipVNgYwxi5qK5Sw2mHbyHGluZbKETpu+71m9J5dOXchrGCcFIIiAKlN10=; 24:noZxMczcPMz3KeYMjJFZZYC8RczSv+SzmAP0Stlwq7oj+noahiqNrW5LCqsMZwQwxpP2qYsU6ntJ48d7/UMYDU/wLgzIWdOdmvdKJVlaSTk= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM0PR05MB4418; 7:yaysA+RhQFa1BA+IqAsPDElrLCAZ45PgHizu57oF6Bj8Mj/bYlKhMxOh2oq3uVuQgw96pN0+czjH+wlvZAFWQSdSPAbXb8Pf0TqvGfsAHxT5a8fh87JB8hLIEchb63kgQOcdb7RNihL/GXUzVQArCicVybSd1cQG3rMZszE0pzpO7y6yolVq/TEFKubD2ylFOEL9VNE8V5EuiImqhmVapBcff7AodV2VFC1x6WVmzwaAE+4ojgWAA0YE0fZLjewZ X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2018 07:55:40.6985 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 07aeff71-d7d8-413e-d384-08d593b82319 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB4418 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: simon.horman@netronome.com, ovs-dev@openvswitch.org Subject: [ovs-dev] [PATCH v8 0/6] OVS-DPDK flow offload with rte_flow X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Hi, Here is a joint work from Mellanox and Napatech, to enable the flow hw offload with the DPDK generic flow interface (rte_flow). The basic idea is to associate the flow with a mark id (a unit32_t number). Later, we then get the flow directly from the mark id, which could bypass some heavy CPU operations, including but not limiting to mini flow extract, emc lookup, dpcls lookup, etc. The association is done with CMAP in patch 1. The CPU workload bypassing is done in patch 2. The flow offload is done in patch 3, which mainly does two things: - translate the ovs match to DPDK rte flow patterns - bind those patterns with a RSS + MARK action. Patch 5 makes the offload work happen in another thread, for leaving the datapath as light as possible. A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1 million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than 260% performance boost. Note that it's disabled by default, which can be enabled by: $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true v8: - enhanced documentation with more details on supported protocols - fixed VLOG to start with capital letter - fixed compilation issues - fixed coding style - addressed the rest of Ian's comments v7: - fixed wrong hash for mark_to_flow that has been refactored in v6 - set the rss_conf for rss action to NULL, to workaround a mlx5 change in DPDK v17.11. Note that it will obey the rss settings OVS-DPDK has set in the beginning. Thus, nothing should be effected. v6: - fixed a sparse warning - added documentation - used hash_int to compute mark to flow hash - added more comments - added lock for pot lookup - rebased on top of the latest code v5: - fixed an issue that it took too long if we do flow add/remove repeatedly. - removed an unused mutex lock - turned most of the log level to DBG - rebased on top of the latest code v4: - use RSS action instead of QUEUE action with MARK - make it work with multiple queue (see patch 1) - rebased on top of latest code v3: - The mark and id association is done with array instead of CMAP. - Added a thread to do hw offload operations - Removed macros completely - dropped the patch to set FDIR_CONF, which is a workround some Intel NICs. - Added a debug patch to show all flow patterns we have created. - Misc fixes v2: - workaround the queue action issue - fixed the tcp_flags being skipped issue, which also fixed the build warnings - fixed l2 patterns for Intel nic - Converted some macros to functions - did not hardcode the max number of flow/action - rebased on top of the latest code Thanks. --- Finn Christensen (1): netdev-dpdk: implement flow offload with rte flow Yuanhan Liu (5): dpif-netdev: associate flow with a mark id dpif-netdev: retrieve flow directly from the flow mark netdev-dpdk: add debug for rte flow patterns dpif-netdev: do hw flow offload in a thread Documentation: document ovs-dpdk flow offload Documentation/howto/dpdk.rst | 22 ++ NEWS | 3 +- lib/dp-packet.h | 13 + lib/dpif-netdev.c | 497 ++++++++++++++++++++++++- lib/flow.c | 155 ++++++-- lib/flow.h | 1 + lib/netdev-dpdk.c | 740 +++++++++++++++++++++++++++++++++++++- lib/netdev.h | 6 + 8 files changed, 1397 insertions(+), 40 deletions(-)