From patchwork Tue Nov 20 16:19:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1000597 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="OQfs6ffJ"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42zrWP3P9Kz9s1c for ; Wed, 21 Nov 2018 03:19:29 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 6D5B1AA6; Tue, 20 Nov 2018 16:19:25 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 13F1D8BF for ; Tue, 20 Nov 2018 16:19:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout2.w1.samsung.com (mailout2.w1.samsung.com [210.118.77.12]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C4E827C3 for ; Tue, 20 Nov 2018 16:19:22 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout2.w1.samsung.com (KnoxPortal) with ESMTP id 20181120161920euoutp029df0fdf6309b1e7485ea6cb6582278f5~o4XQDEI-V2182021820euoutp02C for ; Tue, 20 Nov 2018 16:19:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.w1.samsung.com 20181120161920euoutp029df0fdf6309b1e7485ea6cb6582278f5~o4XQDEI-V2182021820euoutp02C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1542730760; bh=XlWlLNEK1UAlbjiiBAq4gqx3as78TxPZGkm1gFLXybw=; h=From:To:Cc:Subject:Date:References:From; b=OQfs6ffJdisSPOyx0VuVm+oIOvZKxKVzP5vK8G3XCuUgXZnT5YRFQCp0rMBqTpstf +wKeHVoNHR44QJIfJ6zIz6wuX0s2jZp79qcQZsw8o/CHUK0I4kTf743y5oZ7ADrNIf a2ntMTTw81cMTGiUGXm0QEb3p8wfaR165Ao4hEag= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20181120161920eucas1p212e2a2a337f9942055441de6603565f9~o4XPgx2lL0630506305eucas1p2f; Tue, 20 Nov 2018 16:19:20 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id DB.B6.04294.70434FB5; Tue, 20 Nov 2018 16:19:19 +0000 (GMT) Received: from eusmtrp1.samsung.com (unknown [182.198.249.138]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20181120161919eucas1p1bb4fc6e779f27639b31a66beb786db33~o4XO0DnH-2612726127eucas1p1l; Tue, 20 Nov 2018 16:19:19 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp1.samsung.com (KnoxPortal) with ESMTP id 20181120161919eusmtrp16ab664a89deb6c26282378ae14db2578~o4XOlaDne0774907749eusmtrp1b; Tue, 20 Nov 2018 16:19:19 +0000 (GMT) X-AuditID: cbfec7f4-84fff700000010c6-23-5bf43407baa4 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id AC.4E.04284.70434FB5; Tue, 20 Nov 2018 16:19:19 +0000 (GMT) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20181120161918eusmtip1e61bc717ae9e5a9f288344601cea756c~o4XOKheLB2194021940eusmtip1_; Tue, 20 Nov 2018 16:19:18 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org, Ian Stokes Date: Tue, 20 Nov 2018 19:19:13 +0300 Message-Id: <20181120161913.26634-1-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCIsWRmVeSWpSXmKPExsWy7djP87rsJl+iDZZtNbe40v6T3WLjw7Os Frf397BYrD30gd1i7qfnjA6sHov3vGTyeHbzP6PH82s9LB7v911l8+jbsooxgDWKyyYlNSez LLVI3y6BK2PJ1eWMBU0RFd82/2JtYGx06GLk5JAQMJF41baaDcQWEljBKHHqfXYXIxeQ/YVR onvqcyYI5zOjxJ453WwwHTe/nWCDSCxnlJi/7R0zhPODUaJ301dWkCo2AR2JU6uPMILYIgKW El3Xu8BsZoESiYd/T4LVCAtYS/RvmMkMYrMIqEpsvvcdrIYXKH7ozzYWiG3yEqs3HABbICFw gE1ieudvoCIOIMdFoumqFUSNsMSr41vYIWwZidOTe6B66yXut7xkhOjtYJSYfugfE0TCXmLL 63PsIHOYBTQl1u/Shwg7Skz/cZ8JYjyfxI23ghAn80lM2jadGSLMK9HRJgRRrSLx++ByZghb SuLmu89QF3hI3D12lhESorESy9b3skxglJuFsGsBI+MqRvHU0uLc9NRio7zUcr3ixNzi0rx0 veT83E2MwLg//e/4lx2Mu/4kHWIU4GBU4uFNEP4SLcSaWFZcmXuIUYKDWUmE95wWUIg3JbGy KrUoP76oNCe1+BCjNAeLkjhvNcODaCGB9MSS1OzU1ILUIpgsEwenVANjxZ591t7XBEL6rqlW tbX9y1dgj9ztfnNZXFzoBbaqORMi/I9Gnk5evJfnZ7eyESfTxPXmgXZ8i5J5a2btzt3M+2OR bmj2QZFfzw3C8p+mqM1OWnJu5+/X8QuOsgkd857cdNk3PlB68tqeo20/ZCMalx3+PfdVzu5L 8hM1rFac2KuwRezAxsmMSizFGYmGWsxFxYkA+l031vcCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrDLMWRmVeSWpSXmKPExsVy+t/xu7rsJl+iDf51c1hcaf/JbrHx4VlW i9v7e1gs1h76wG4x99NzRgdWj8V7XjJ5PLv5n9Hj+bUeFo/3+66yefRtWcUYwBqlZ1OUX1qS qpCRX1xiqxRtaGGkZ2hpoWdkYqlnaGwea2VkqqRvZ5OSmpNZllqkb5egl7Hk6nLGgqaIim+b f7E2MDY6dDFyckgImEjc/HaCrYuRi0NIYCmjxPkJ79ghElISP35dYIWwhSX+XOuCKvrGKNE+ YTcbSIJNQEfi1OojjCC2iIC1xLrvG1lAbGaBMok33cfBaoSB4v0bZjKD2CwCqhKb730Hq+cF ih/6s40FYoG8xOoNB5gnMPIsYGRYxSiSWlqcm55bbKhXnJhbXJqXrpecn7uJERhy24793LyD 8dLG4EOMAhyMSjy8CcJfooVYE8uKK3MPMUpwMCuJ8J7TAgrxpiRWVqUW5ccXleakFh9iNAVa PpFZSjQ5HxgPeSXxhqaG5haWhubG5sZmFkrivOcNKqOEBNITS1KzU1MLUotg+pg4OKUaGGfc +MeSu+NTfnOx7nl2KzujSXLGIqFlFx3s3y7e+0C1YN5p20Bt/oq+jqs2An0n9z67nNdTFKCR evSwkuILxqmVB+LD5nc7K/k9OvthVej1U3/CLa9aOGbeyqvZqWylIMgrLHXi44ZP760TCxRP m7joveA8e0TCpmCT1uqpP7pdHnzR9mxyVGIpzkg01GIuKk4EADNioC1PAgAA X-CMS-MailID: 20181120161919eucas1p1bb4fc6e779f27639b31a66beb786db33 X-Msg-Generator: CA X-RootMTR: 20181120161919eucas1p1bb4fc6e779f27639b31a66beb786db33 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20181120161919eucas1p1bb4fc6e779f27639b31a66beb786db33 References: X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v2] dpif-netdev: Per-port configurable EMC. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Conditional EMC insert helps a lot in scenarios with high numbers of parallel flows, but in current implementation this option affects all the threads and ports at once. There are scenarios where we have different number of flows on different ports. For example, if one of the VMs encapsulates traffic using additional headers, it will receive large number of flows but only few flows will come out of this VM. In this scenario it's much faster to use EMC instead of classifier for traffic from the VM, but it's better to disable EMC for the traffic which flows to VM. To handle above issue introduced 'emc-enable' configurable to enable/disable EMC on a per-port basis. Ex.: ovs-vsctl set interface dpdk0 other_config:emc-enable=false EMC probability kept as is and it works for all the ports with 'emc-enable=true'. Signed-off-by: Ilya Maximets --- It's been a while since the first version. It's available here: https://patchwork.ozlabs.org/patch/800277/ Version 2: * The main concern was about backward compatibility. Also, there is no real profit having the per-port probability value. So, per-port probability switched to per-port 'emc-enable' configurable. Global probability kept back and can be used without any changes. Documentation/topics/dpdk/bridge.rst | 4 ++ NEWS | 4 +- lib/dpif-netdev.c | 79 +++++++++++++++++++++++++--- vswitchd/vswitch.xml | 19 +++++++ 4 files changed, 97 insertions(+), 9 deletions(-) diff --git a/Documentation/topics/dpdk/bridge.rst b/Documentation/topics/dpdk/bridge.rst index df74c02ad..b8a7f61e4 100644 --- a/Documentation/topics/dpdk/bridge.rst +++ b/Documentation/topics/dpdk/bridge.rst @@ -101,6 +101,10 @@ observed with pmd stats:: For certain traffic profiles with many parallel flows, it's recommended to set ``N`` to '0' to achieve higher forwarding performance. +It is also possible to enable/disable EMC on per-port basis using:: + + $ ovs-vsctl set interface other_config:emc-enable={true,false} + For more information on the EMC refer to :doc:`/intro/install/dpdk` . diff --git a/NEWS b/NEWS index 02402d1a4..bb7a0738e 100644 --- a/NEWS +++ b/NEWS @@ -9,9 +9,11 @@ Post-v2.10.0 - ovn: * New support for IPSEC encrypted tunnels between hypervisors. * ovn-ctl: allow passing user:group ids to the OVN daemons. - - DPDK: + - Userspace datapath: * Add option for simple round-robin based Rxq to PMD assignment. It can be set with pmd-rxq-assign. + * Added new per-port configurable option to manage EMC: + 'other_config:emc-enable'. - Add 'symmetric_l3' hash function. - OVS now honors 'updelay' and 'downdelay' for bonds with LACP configured. - ovs-vswitchd: diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 1564db9c6..404d46184 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -460,6 +460,7 @@ struct dp_netdev_port { unsigned n_rxq; /* Number of elements in 'rxqs' */ unsigned *txq_used; /* Number of threads that use each tx queue. */ struct ovs_mutex txq_used_mutex; + bool emc_enabled; /* If true EMC will be used. */ char *type; /* Port type as requested by user. */ char *rxq_affinity_list; /* Requested affinity of rx queues. */ }; @@ -574,6 +575,7 @@ static void dp_netdev_actions_free(struct dp_netdev_actions *); struct polled_queue { struct dp_netdev_rxq *rxq; odp_port_t port_no; + bool emc_enabled; }; /* Contained by struct dp_netdev_pmd_thread's 'poll_list' member. */ @@ -603,6 +605,8 @@ struct dp_netdev_pmd_thread_ctx { long long now; /* RX queue from which last packet was received. */ struct dp_netdev_rxq *last_rxq; + /* EMC insertion probability context for the current processing cycle. */ + uint32_t emc_insert_min; }; /* PMD: Poll modes drivers. PMD accesses devices via polling to eliminate @@ -1779,6 +1783,7 @@ port_create(const char *devname, const char *type, port->netdev = netdev; port->type = xstrdup(type); port->sf = sf; + port->emc_enabled = true; port->need_reconfigure = true; ovs_mutex_init(&port->txq_used_mutex); @@ -2811,9 +2816,7 @@ emc_probabilistic_insert(struct dp_netdev_pmd_thread *pmd, * default the value is UINT32_MAX / 100 which yields an insertion * probability of 1/100 ie. 1% */ - uint32_t min; - - atomic_read_relaxed(&pmd->dp->emc_insert_min, &min); + uint32_t min = pmd->ctx.emc_insert_min; if (min && random_uint32() <= min) { emc_insert(&(pmd->flow_cache).emc_cache, key, flow); @@ -3679,7 +3682,8 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute) ovs_mutex_lock(&dp->non_pmd_mutex); } - /* Update current time in PMD context. */ + /* Update current time in PMD context. We don't care about EMC insertion + * probability, because we are on a slow path. */ pmd_thread_ctx_time_update(pmd); /* The action processing expects the RSS hash to be valid, because @@ -3775,7 +3779,7 @@ dpif_netdev_set_config(struct dpif *dpif, const struct smap *other_config) if (insert_min != cur_min) { atomic_store_relaxed(&dp->emc_insert_min, insert_min); if (insert_min == 0) { - VLOG_INFO("EMC has been disabled"); + VLOG_INFO("EMC insertion probability changed to zero"); } else { VLOG_INFO("EMC insertion probability changed to 1/%llu (~%.2f%%)", insert_prob, (100 / (float)insert_prob)); @@ -3881,6 +3885,27 @@ exit: return error; } +/* Returns 'true' if one of the 'port's RX queues exists in 'poll_list' + * of given PMD thread. */ +static bool +dpif_netdev_pmd_polls_port(struct dp_netdev_pmd_thread *pmd, + struct dp_netdev_port *port) + OVS_EXCLUDED(pmd->port_mutex) +{ + struct rxq_poll *poll; + bool found = false; + + ovs_mutex_lock(&pmd->port_mutex); + HMAP_FOR_EACH (poll, node, &pmd->poll_list) { + if (port == poll->rxq->port) { + found = true; + break; + } + } + ovs_mutex_unlock(&pmd->port_mutex); + return found; +} + /* Changes the affinity of port's rx queues. The changes are actually applied * in dpif_netdev_run(). */ static int @@ -3891,10 +3916,33 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, struct dp_netdev_port *port; int error = 0; const char *affinity_list = smap_get(cfg, "pmd-rxq-affinity"); + bool emc_enabled = smap_get_bool(cfg, "emc-enable", true); ovs_mutex_lock(&dp->port_mutex); error = get_port_by_number(dp, port_no, &port); - if (error || !netdev_is_pmd(port->netdev) + if (error) { + goto unlock; + } + + if (emc_enabled != port->emc_enabled) { + struct dp_netdev_pmd_thread *pmd; + + port->emc_enabled = emc_enabled; + /* Mark for reload all the threads that polls this port and request + * for reconfiguration for the actual reloading of threads. */ + CMAP_FOR_EACH (pmd, node, &dp->poll_threads) { + if (dpif_netdev_pmd_polls_port(pmd, port)) { + pmd->need_reload = true; + } + } + dp_netdev_request_reconfigure(dp); + + VLOG_INFO("%s: EMC has been %s", netdev_get_name(port->netdev), + (emc_enabled) ? "enabled" : "disabled"); + } + + /* Checking for RXq affinity changes. */ + if (!netdev_is_pmd(port->netdev) || nullable_string_is_equal(affinity_list, port->rxq_affinity_list)) { goto unlock; } @@ -4798,6 +4846,13 @@ dpif_netdev_run(struct dpif *dpif) if (!netdev_is_pmd(port->netdev)) { int i; + if (port->emc_enabled) { + atomic_read_relaxed(&dp->emc_insert_min, + &non_pmd->ctx.emc_insert_min); + } else { + non_pmd->ctx.emc_insert_min = 0; + } + for (i = 0; i < port->n_rxq; i++) { if (dp_netdev_process_rxq_port(non_pmd, &port->rxqs[i], @@ -4945,6 +5000,7 @@ pmd_load_queues_and_ports(struct dp_netdev_pmd_thread *pmd, HMAP_FOR_EACH (poll, node, &pmd->poll_list) { poll_list[i].rxq = poll->rxq; poll_list[i].port_no = poll->rxq->port->port_no; + poll_list[i].emc_enabled = poll->rxq->port->emc_enabled; i++; } @@ -5007,6 +5063,14 @@ reload: pmd_perf_start_iteration(s); for (i = 0; i < poll_cnt; i++) { + + if (poll_list[i].emc_enabled) { + atomic_read_relaxed(&pmd->dp->emc_insert_min, + &pmd->ctx.emc_insert_min); + } else { + pmd->ctx.emc_insert_min = 0; + } + process_packets = dp_netdev_process_rxq_port(pmd, poll_list[i].rxq, poll_list[i].port_no); @@ -5948,7 +6012,7 @@ dfc_processing(struct dp_netdev_pmd_thread *pmd, struct dfc_cache *cache = &pmd->flow_cache; struct dp_packet *packet; const size_t cnt = dp_packet_batch_size(packets_); - uint32_t cur_min; + uint32_t cur_min = pmd->ctx.emc_insert_min; int i; uint16_t tcp_flags; bool smc_enable_db; @@ -5956,7 +6020,6 @@ dfc_processing(struct dp_netdev_pmd_thread *pmd, bool batch_enable = true; atomic_read_relaxed(&pmd->dp->smc_enable_db, &smc_enable_db); - atomic_read_relaxed(&pmd->dp->emc_insert_min, &cur_min); pmd_perf_update_counter(&pmd->perf_stats, md_is_valid ? PMD_STAT_RECIRC : PMD_STAT_RECV, cnt); diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 21609107d..21328d1b5 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -3060,6 +3060,25 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \ + +

+ These settings controls behaviour of EMC lookups/insertions for packets + received from the interface. +

+ + +

+ Specifies if Exact Match Cache (EMC) should be used while processing + packets received from this interface. + If true, will have effect on this interface. +

+

+ Defaults to true. +

+ + +

The MTU (maximum transmission unit) is the largest amount of data