From patchwork Tue Aug 8 17:06:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Bodireddy, Bhanuprakash" X-Patchwork-Id: 799372 X-Patchwork-Delegate: dlu998@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xRh4c0m9cz9s7F for ; Wed, 9 Aug 2017 03:20:48 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 912EEB30; Tue, 8 Aug 2017 17:19:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 890CCB30 for ; Tue, 8 Aug 2017 17:19:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 94CD141E for ; Tue, 8 Aug 2017 17:17:35 +0000 (UTC) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2017 10:15:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,344,1498546800"; d="scan'208";a="297454197" Received: from silpixa00393942.ir.intel.com (HELO silpixa00393942.ger.corp.intel.com) ([10.237.223.42]) by fmsmga004.fm.intel.com with ESMTP; 08 Aug 2017 10:15:28 -0700 From: Bhanuprakash Bodireddy To: dev@openvswitch.org Date: Tue, 8 Aug 2017 18:06:16 +0100 Message-Id: <1502211976-76937-6-git-send-email-bhanuprakash.bodireddy@intel.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> References: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 5/5] dpif-netdev: Flush the packets in intermediate queue. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Under low rate traffic conditions, there can be 2 issues. (1) Packets potentially can get stuck in the intermediate queue. (2) Latency of the packets can increase significantly due to buffering in intermediate queue. This commit handles the (1) issue by flushing the tx port queues using dp_netdev_flush_txq_ports() as part of PMD packet processing loop. Also this commit addresses issue (2) by flushing the tx queues after every rxq port processing. This reduces the latency with out impacting the forwarding throughput. MASTER -------- Pkt size min(ns) avg(ns) max(ns) 512 4,631 5,022 309,914 1024 5,545 5,749 104,294 1280 5,978 6,159 45,306 1518 6,419 6,774 946,850 MASTER + COMMIT ----------------- Pkt size min(ns) avg(ns) max(ns) 512 4,711 5,064 182,477 1024 5,601 5,888 701,654 1280 6,018 6,491 533,037 1518 6,467 6,734 312,471 PMDs can be teared down and spawned at runtime and so the rxq and txq mapping of the PMD threads can change. In few cases packets can get stuck in the queue due to reconfiguration and this commit helps flush the queues. Suggested-by: Eelco Chaudron Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2017-April/331039.html Signed-off-by: Bhanuprakash Bodireddy Signed-off-by: Antonio Fischetti Co-authored-by: Antonio Fischetti Signed-off-by: Markus Magnusson Co-authored-by: Markus Magnusson Acked-by: Eelco Chaudron --- lib/dpif-netdev.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index e2cd931..bfb9650 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -340,6 +340,7 @@ enum pmd_cycles_counter_type { }; #define XPS_TIMEOUT_MS 500LL +#define LAST_USED_QID_NONE -1 /* Contained by struct dp_netdev_port's 'rxqs' member. */ struct dp_netdev_rxq { @@ -500,7 +501,13 @@ struct rxq_poll { struct tx_port { struct dp_netdev_port *port; int qid; - long long last_used; + int last_used_qid; /* Last queue id where packets got + enqueued. */ + long long last_used; /* In case XPS is enabled, it contains the + * timestamp of the last time the port was + * used by the thread to send data. After + * XPS_TIMEOUT_MS elapses the qid will be + * marked as -1. */ struct hmap_node node; }; @@ -3101,6 +3108,25 @@ cycles_count_intermediate(struct dp_netdev_pmd_thread *pmd, non_atomic_ullong_add(&pmd->cycles.n[type], interval); } +static void +dp_netdev_flush_txq_ports(struct dp_netdev_pmd_thread *pmd) +{ + struct tx_port *cached_tx_port; + int tx_qid; + + HMAP_FOR_EACH (cached_tx_port, node, &pmd->send_port_cache) { + tx_qid = cached_tx_port->last_used_qid; + + if (tx_qid != LAST_USED_QID_NONE) { + netdev_txq_flush(cached_tx_port->port->netdev, tx_qid, + cached_tx_port->port->dynamic_txqs); + + /* Queue flushed and mark it empty. */ + cached_tx_port->last_used_qid = LAST_USED_QID_NONE; + } + } +} + static int dp_netdev_process_rxq_port(struct dp_netdev_pmd_thread *pmd, struct netdev_rxq *rx, @@ -3667,6 +3693,9 @@ dpif_netdev_run(struct dpif *dpif) dp_netdev_process_rxq_port(non_pmd, port->rxqs[i].rx, port->port_no); + + dp_netdev_flush_txq_ports(non_pmd); + cycles_count_intermediate(non_pmd, process_packets ? PMD_CYCLES_PROCESSING : PMD_CYCLES_IDLE); @@ -3854,6 +3883,8 @@ reload: process_packets = dp_netdev_process_rxq_port(pmd, poll_list[i].rx, poll_list[i].port_no); + dp_netdev_flush_txq_ports(pmd); + cycles_count_intermediate(pmd, process_packets ? PMD_CYCLES_PROCESSING : PMD_CYCLES_IDLE); @@ -3879,6 +3910,9 @@ reload: cycles_count_end(pmd, PMD_CYCLES_IDLE); + /* Flush the queues as part of reconfiguration logic. */ + dp_netdev_flush_txq_ports(pmd); + poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); exiting = latch_is_set(&pmd->exit_latch); /* Signal here to make sure the pmd finishes @@ -4481,6 +4515,7 @@ dp_netdev_add_port_tx_to_pmd(struct dp_netdev_pmd_thread *pmd, tx->port = port; tx->qid = -1; + tx->last_used_qid = LAST_USED_QID_NONE; hmap_insert(&pmd->tx_ports, &tx->node, hash_port_no(tx->port->port_no)); pmd->need_reload = true; @@ -5051,6 +5086,14 @@ dpif_netdev_xps_get_tx_qid(const struct dp_netdev_pmd_thread *pmd, dpif_netdev_xps_revalidate_pmd(pmd, now, false); + /* The tx queue can change in XPS case, make sure packets in previous + * queue is flushed properly. */ + if (tx->last_used_qid != LAST_USED_QID_NONE && + tx->qid != tx->last_used_qid) { + netdev_txq_flush(port->netdev, tx->last_used_qid, port->dynamic_txqs); + tx->last_used_qid = LAST_USED_QID_NONE; + } + VLOG_DBG("Core %d: New TX queue ID %d for port \'%s\'.", pmd->core_id, tx->qid, netdev_get_name(tx->port->netdev)); return min_qid; @@ -5146,6 +5189,13 @@ dp_execute_cb(void *aux_, struct dp_packet_batch *packets_, tx_qid = pmd->static_tx_qid; } + /* In case these packets gets buffered into an intermediate + * queue and XPS is enabled the flush function could find a + * different tx qid assigned to its thread. We keep track + * of the qid we're now using, that will trigger the flush + * function and will select the right queue to flush. */ + p->last_used_qid = tx_qid; + netdev_send(p->port->netdev, tx_qid, packets_, may_steal, dynamic_txqs); return;