From patchwork Thu Aug 10 15:38:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 800239 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xSskL0fWqz9sN5 for ; Fri, 11 Aug 2017 01:39:06 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id D0FF08D9; Thu, 10 Aug 2017 15:38:24 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 1F05C8D7 for ; Thu, 10 Aug 2017 15:38:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C2CCC350 for ; Thu, 10 Aug 2017 15:38:23 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20170810153820euoutp017cbf8b63baa28c79dae5dfe48f77f4aa~ZhjIt_dsQ0436604366euoutp01Z; Thu, 10 Aug 2017 15:38:20 +0000 (GMT) Received: from eusmges3.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170810153820eucas1p1b6260793cc186a904da0f57b2273714f~ZhjIGH1PN3033930339eucas1p1h; Thu, 10 Aug 2017 15:38:20 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges3.samsung.com (EUCPMTA) with SMTP id 14.04.12867.BED7C895; Thu, 10 Aug 2017 16:38:19 +0100 (BST) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170810153819eucas1p265fee29cb618c8ebceb1af5bb35b7b9f~ZhjHZ4DI-0432204322eucas1p2h; Thu, 10 Aug 2017 15:38:19 +0000 (GMT) X-AuditID: cbfec7f2-f793b6d000003243-8a-598c7deb4d07 Received: from eusync4.samsung.com ( [203.254.199.214]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id E7.14.18832.BED7C895; Thu, 10 Aug 2017 16:38:19 +0100 (BST) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OUH005CN7FL8NB0@eusync4.samsung.com>; Thu, 10 Aug 2017 16:38:19 +0100 (BST) From: Ilya Maximets To: ovs-dev@openvswitch.org, Bhanuprakash Bodireddy Date: Thu, 10 Aug 2017 18:38:03 +0300 Message-id: <1502379486-1568-2-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.7.4 In-reply-to: <1502379486-1568-1-git-send-email-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrLIsWRmVeSWpSXmKPExsWy7djPc7qva3siDW6cM7ZY/YvTYuczZYtX kxsYLVr6ZzJbPH+xkNnizpWfbBbTPt9mt7jS/pPdYu2hD+wWcz89Z3Tg8li85yWTx7Ob/xk9 nl/rYfF4v+8qm0ffllWMHu/mv2ULYIvisklJzcksSy3St0vgyvj0/SlbwUe/iinnf7A0MLZa dDFyckgImEgsnT6NFcIWk7hwbz1bFyMXh5DAUkaJzQ/fMEM4nxkl2n9sYILpODZtLwtEYhmj xPmlZ6Gqmpkk9i39zwJSxSagI3Fq9RFGEFtEIELiwfdZ7CBFzALbmCSWPJ4ElhAWsJG417kJ qJuDg0VAVWLC61AQk1fAVeLtDReIZXISN891MoPYnAJuEhP3TARbLCHQzy6xdc0EdpB6CQFZ iU0HmCHqXSQO7HgM9Y6wxKvjW9ghbBmJzo6DTBC9zYwSDasuMUI4ExglvjQvh3rNXuLUzatg NrMAn8SkbdOZIRbwSnS0CUGUeEi8eTyDDcJ2lJi0ZgoTxPMzGSXeL+tnmsAos4CRYRWjSGpp cW56arGxXnFibnFpXrpecn7uJkZgpJ/+d/zTDsavJ6wOMQpwMCrx8HIId0cKsSaWFVfmHmKU 4GBWEuHtqOyJFOJNSaysSi3Kjy8qzUktPsQozcGiJM5rG9UWKSSQnliSmp2aWpBaBJNl4uCU amDcsIaRZeWs/DfebHpqm8V9txwo3qoibfepeFeLVE6KvKjoTrU6dRFW7dqopMUm5ic5FrTX XT5z/ttt8W2rT0jcUV6YFjRpI5e/84kDEksWTOPfZpT7+3zx/elh1inGXHpihQ7VBkrL2hPZ 1QUbN5Q9qvlj8erG1lcpLKEzYq/nnSuI7+ruylNiKc5INNRiLipOBACyFFeZ8AIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrNLMWRmVeSWpSXmKPExsVy+t/xa7qva3siDXZ3ilqs/sVpsfOZssWr yQ2MFi39M5ktnr9YyGxx58pPNotpn2+zW1xp/8lusfbQB3aLuZ+eMzpweSze85LJ49nN/4we z6/1sHi833eVzaNvyypGj3fz37IFsEW52WSkJqakFimk5iXnp2TmpdsqhYa46VooKeQl5qba KkXo+oYEKSmUJeaUAnlGBmjAwTnAPVhJ3y7BLePT96dsBR/9Kqac/8HSwNhq0cXIySEhYCJx bNpeFghbTOLCvfVsXYxcHEICSxgl5m28xAjhtDJJHFn+gRWkik1AR+LU6iOMILaIQIREy5z1 YEXMAtuYJB5f+AQ2SljARuJe5ybmLkYODhYBVYkJr0NBTF4BV4m3N1wglslJ3DzXyQxicwq4 SUzcMxGsUwioZOW5x0wTGHkXMDKsYhRJLS3OTc8tNtQrTswtLs1L10vOz93ECAz4bcd+bt7B eGlj8CFGAQ5GJR7eBNHuSCHWxLLiytxDjBIczEoivB2VPZFCvCmJlVWpRfnxRaU5qcWHGE2B TprILCWanA+MxrySeEMTQ3NLQyNjCwtzIyMlcV71y02RQgLpiSWp2ampBalFMH1MHJxSDYz1 RleSqt7UcznxRYr8+9r+YqVxZhtLk8Pcna+OcIfLlp9qSjp+48PsWzaxu/iv3+HTfLVi7xwP w8l1pfd/bemRuv3y/AZOm43yYckGltXWnP+eHahcqGSf3iIirfCJtzPg958Ly1VEOlldOrI3 xn9YqvEwXOaIL4Po/pcn52x0fn1OpHBx9QslluKMREMt5qLiRAC2aSRnjgIAAA== X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170810153819eucas1p265fee29cb618c8ebceb1af5bb35b7b9f X-Msg-Generator: CA X-Sender-IP: 182.198.249.179 X-Local-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G+yCvOyEseyghOyekBtMZWFkaW5nIEVuZ2luZWVy?= X-Global-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G1NhbXN1bmcgRWxlY3Ryb25pY3MbTGVhZGluZyBFbmdpbmVlcg==?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-CMS-RootMailID: 20170810153819eucas1p265fee29cb618c8ebceb1af5bb35b7b9f X-RootMTR: 20170810153819eucas1p265fee29cb618c8ebceb1af5bb35b7b9f References: <1502379486-1568-1-git-send-email-i.maximets@samsung.com> Cc: Heetae Ahn , Ilya Maximets Subject: [ovs-dev] [PATCH v3 1/4] dpif-netdev: Output packet batching. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org While processing incoming batch of packets they are scattered across many per-flow batches and sent separately. This becomes an issue while using more than a few flows. For example if we have balanced-tcp OvS bonding with 2 ports there will be 256 datapath internal flows for each dp_hash pattern. This will lead to scattering of a single recieved batch across all of that 256 per-flow batches and invoking send for each packet separately. This behaviour greatly degrades overall performance of netdev_send because of inability to use advantages of vectorized transmit functions. But the half (if 2 ports in bonding) of datapath flows will have the same output actions. This means that we can collect them in a single place back and send at once using single call to netdev_send. This patch introduces per-port packet batch for output packets for that purpose. 'output_pkts' batch is thread local and located in send port cache. Signed-off-by: Ilya Maximets --- lib/dpif-netdev.c | 104 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 82 insertions(+), 22 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index e2cd931..a2a25be 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -502,6 +502,7 @@ struct tx_port { int qid; long long last_used; struct hmap_node node; + struct dp_packet_batch output_pkts; }; /* PMD: Poll modes drivers. PMD accesses devices via polling to eliminate @@ -633,9 +634,10 @@ static void dp_netdev_execute_actions(struct dp_netdev_pmd_thread *pmd, size_t actions_len, long long now); static void dp_netdev_input(struct dp_netdev_pmd_thread *, - struct dp_packet_batch *, odp_port_t port_no); + struct dp_packet_batch *, odp_port_t port_no, + long long now); static void dp_netdev_recirculate(struct dp_netdev_pmd_thread *, - struct dp_packet_batch *); + struct dp_packet_batch *, long long now); static void dp_netdev_disable_upcall(struct dp_netdev *); static void dp_netdev_pmd_reload_done(struct dp_netdev_pmd_thread *pmd); @@ -667,6 +669,9 @@ static void dp_netdev_add_rxq_to_pmd(struct dp_netdev_pmd_thread *pmd, static void dp_netdev_del_rxq_from_pmd(struct dp_netdev_pmd_thread *pmd, struct rxq_poll *poll) OVS_REQUIRES(pmd->port_mutex); +static void +dp_netdev_pmd_flush_output_packets(struct dp_netdev_pmd_thread *pmd, + long long now); static void reconfigure_datapath(struct dp_netdev *dp) OVS_REQUIRES(dp->port_mutex); static bool dp_netdev_pmd_try_ref(struct dp_netdev_pmd_thread *pmd); @@ -2809,6 +2814,7 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute) struct dp_netdev *dp = get_dp_netdev(dpif); struct dp_netdev_pmd_thread *pmd; struct dp_packet_batch pp; + long long now = time_msec(); if (dp_packet_size(execute->packet) < ETH_HEADER_LEN || dp_packet_size(execute->packet) > UINT16_MAX) { @@ -2851,8 +2857,8 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute) dp_packet_batch_init_packet(&pp, execute->packet); dp_netdev_execute_actions(pmd, &pp, false, execute->flow, - execute->actions, execute->actions_len, - time_msec()); + execute->actions, execute->actions_len, now); + dp_netdev_pmd_flush_output_packets(pmd, now); if (pmd->core_id == NON_PMD_CORE_ID) { ovs_mutex_unlock(&dp->non_pmd_mutex); @@ -3101,6 +3107,37 @@ cycles_count_intermediate(struct dp_netdev_pmd_thread *pmd, non_atomic_ullong_add(&pmd->cycles.n[type], interval); } +static void +dp_netdev_pmd_flush_output_on_port(struct dp_netdev_pmd_thread *pmd, + struct tx_port *p, long long now) +{ + int tx_qid; + bool dynamic_txqs; + + dynamic_txqs = p->port->dynamic_txqs; + if (dynamic_txqs) { + tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p, now); + } else { + tx_qid = pmd->static_tx_qid; + } + + netdev_send(p->port->netdev, tx_qid, &p->output_pkts, true, dynamic_txqs); + dp_packet_batch_init(&p->output_pkts); +} + +static void +dp_netdev_pmd_flush_output_packets(struct dp_netdev_pmd_thread *pmd, + long long now) +{ + struct tx_port *p; + + HMAP_FOR_EACH (p, node, &pmd->send_port_cache) { + if (!dp_packet_batch_is_empty(&p->output_pkts)) { + dp_netdev_pmd_flush_output_on_port(pmd, p, now); + } + } +} + static int dp_netdev_process_rxq_port(struct dp_netdev_pmd_thread *pmd, struct netdev_rxq *rx, @@ -3113,10 +3150,13 @@ dp_netdev_process_rxq_port(struct dp_netdev_pmd_thread *pmd, dp_packet_batch_init(&batch); error = netdev_rxq_recv(rx, &batch); if (!error) { + long long now = time_msec(); + *recirc_depth_get() = 0; batch_cnt = batch.count; - dp_netdev_input(pmd, &batch, port_no); + dp_netdev_input(pmd, &batch, port_no, now); + dp_netdev_pmd_flush_output_packets(pmd, now); } else if (error != EAGAIN && error != EOPNOTSUPP) { static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); @@ -4481,6 +4521,7 @@ dp_netdev_add_port_tx_to_pmd(struct dp_netdev_pmd_thread *pmd, tx->port = port; tx->qid = -1; + dp_packet_batch_init(&tx->output_pkts); hmap_insert(&pmd->tx_ports, &tx->node, hash_port_no(tx->port->port_no)); pmd->need_reload = true; @@ -4901,7 +4942,8 @@ fast_path_processing(struct dp_netdev_pmd_thread *pmd, static void dp_netdev_input__(struct dp_netdev_pmd_thread *pmd, struct dp_packet_batch *packets, - bool md_is_valid, odp_port_t port_no) + bool md_is_valid, odp_port_t port_no, + long long now) { int cnt = packets->count; #if !defined(__CHECKER__) && !defined(_WIN32) @@ -4913,7 +4955,6 @@ dp_netdev_input__(struct dp_netdev_pmd_thread *pmd, OVS_ALIGNED_VAR(CACHE_LINE_SIZE) struct netdev_flow_key keys[PKT_ARRAY_SIZE]; struct packet_batch_per_flow batches[PKT_ARRAY_SIZE]; - long long now = time_msec(); size_t n_batches; odp_port_t in_port; @@ -4949,16 +4990,16 @@ dp_netdev_input__(struct dp_netdev_pmd_thread *pmd, static void dp_netdev_input(struct dp_netdev_pmd_thread *pmd, struct dp_packet_batch *packets, - odp_port_t port_no) + odp_port_t port_no, long long now) { - dp_netdev_input__(pmd, packets, false, port_no); + dp_netdev_input__(pmd, packets, false, port_no, now); } static void dp_netdev_recirculate(struct dp_netdev_pmd_thread *pmd, - struct dp_packet_batch *packets) + struct dp_packet_batch *packets, long long now) { - dp_netdev_input__(pmd, packets, true, 0); + dp_netdev_input__(pmd, packets, true, 0, now); } struct dp_netdev_execute_aux { @@ -5136,18 +5177,37 @@ dp_execute_cb(void *aux_, struct dp_packet_batch *packets_, case OVS_ACTION_ATTR_OUTPUT: p = pmd_send_port_cache_lookup(pmd, nl_attr_get_odp_port(a)); if (OVS_LIKELY(p)) { - int tx_qid; - bool dynamic_txqs; + struct dp_packet *packet; + struct dp_packet_batch out; - dynamic_txqs = p->port->dynamic_txqs; - if (dynamic_txqs) { - tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p, now); - } else { - tx_qid = pmd->static_tx_qid; + if (!may_steal) { + dp_packet_batch_clone(&out, packets_); + dp_packet_batch_reset_cutlen(packets_); + packets_ = &out; + } + dp_packet_batch_apply_cutlen(packets_); + +#ifdef DPDK_NETDEV + if (OVS_UNLIKELY(!dp_packet_batch_is_empty(&p->output_pkts) + && packets_->packets[0]->source + != p->output_pkts.packets[0]->source)) { + /* XXX: netdev-dpdk assumes that all packets in a single + * outptut batch has the same source. Flush here to + * avoid memory access issues. */ + dp_netdev_pmd_flush_output_on_port(pmd, p, now); + } +#endif + + if (OVS_UNLIKELY(dp_packet_batch_size(&p->output_pkts) + + dp_packet_batch_size(packets_) > NETDEV_MAX_BURST)) { + /* Some packets was generated while input batch processing. + * Flush here to avoid overflow. */ + dp_netdev_pmd_flush_output_on_port(pmd, p, now); } - netdev_send(p->port->netdev, tx_qid, packets_, may_steal, - dynamic_txqs); + DP_PACKET_BATCH_FOR_EACH (packet, packets_) { + dp_packet_batch_add(&p->output_pkts, packet); + } return; } break; @@ -5188,7 +5248,7 @@ dp_execute_cb(void *aux_, struct dp_packet_batch *packets_, } (*depth)++; - dp_netdev_recirculate(pmd, packets_); + dp_netdev_recirculate(pmd, packets_, now); (*depth)--; return; } @@ -5253,7 +5313,7 @@ dp_execute_cb(void *aux_, struct dp_packet_batch *packets_, } (*depth)++; - dp_netdev_recirculate(pmd, packets_); + dp_netdev_recirculate(pmd, packets_, now); (*depth)--; return;