From patchwork Tue Aug 8 17:06:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Bodireddy, Bhanuprakash" X-Patchwork-Id: 799371 X-Patchwork-Delegate: dlu998@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xRh3w0W0fz9s7C for ; Wed, 9 Aug 2017 03:20:12 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id A0AB3B4B; Tue, 8 Aug 2017 17:19:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 67280B30 for ; Tue, 8 Aug 2017 17:19:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 514D640A for ; Tue, 8 Aug 2017 17:17:35 +0000 (UTC) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2017 10:15:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,344,1498546800"; d="scan'208";a="297454193" Received: from silpixa00393942.ir.intel.com (HELO silpixa00393942.ger.corp.intel.com) ([10.237.223.42]) by fmsmga004.fm.intel.com with ESMTP; 08 Aug 2017 10:15:27 -0700 From: Bhanuprakash Bodireddy To: dev@openvswitch.org Date: Tue, 8 Aug 2017 18:06:15 +0100 Message-Id: <1502211976-76937-5-git-send-email-bhanuprakash.bodireddy@intel.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> References: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 4/5] netdev-dpdk: Enable intermediate queue for vHost User port. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This commit refactors the __netdev_dpdk_vhost_send() and enables intermediate queue where in the packets are buffered till the threshold 'INTERIM_QUEUE_BURST_THRESHOLD[32] is hit and eventually gets transmitted. This commit improves the throughput as reported below in simple Physical to virtual testcase with higher flows @10G Line rate. Num Flow Master Commit ======== ========= ========= 10 5945899 7833914 32 3872211 6530133 50 3283713 6618711 100 3132540 5857226 500 2964499 5273006 1000 2931952 5178038 Latency stats: MASTER ------- Pkt size min(ns) avg(ns) max(ns) 512 10,011 12,100 281,915 1024 7,870 9,313 193,116 1280 7,862 9,036 194,439 1518 8,215 9,417 204,782 MASTER + COMMIT --------------- Pkt size min(ns) avg(ns) max(ns) 512 10,492 13,655 281,538 1024 8,407 9,784 205,095 1280 8,399 9,750 194,888 1518 8,367 9,722 196,973 Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2017-May/332271.html [By Eelco Chaudron ] Signed-off-by: Bhanuprakash Bodireddy Signed-off-by: Antonio Fischetti Co-authored-by: Antonio Fischetti Acked-by: Eelco Chaudron --- lib/netdev-dpdk.c | 38 +++++++++++++++----------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index b4dc36a..9f43c9c 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -1858,16 +1858,21 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, struct rte_mbuf **cur_pkts = (struct rte_mbuf **) pkts; unsigned int total_pkts = cnt; unsigned int dropped = 0; - int i, retries = 0; + int i; qid = dev->tx_q[qid % netdev->n_txq].map; + struct dpdk_tx_queue *txq = &dev->tx_q[qid]; if (OVS_UNLIKELY(!is_vhost_running(dev) || qid < 0 || !(dev->flags & NETDEV_UP))) { rte_spinlock_lock(&dev->stats_lock); dev->stats.tx_dropped+= cnt; rte_spinlock_unlock(&dev->stats_lock); - goto out; + + for (i = 0; i < total_pkts; i++) { + dp_packet_delete(pkts[i]); + } + return; } rte_spinlock_lock(&dev->tx_q[qid].tx_lock); @@ -1877,34 +1882,21 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, cnt = netdev_dpdk_qos_run(dev, cur_pkts, cnt); dropped = total_pkts - cnt; - do { - int vhost_qid = qid * VIRTIO_QNUM + VIRTIO_RXQ; - unsigned int tx_pkts; - - tx_pkts = rte_vhost_enqueue_burst(netdev_dpdk_get_vid(dev), - vhost_qid, cur_pkts, cnt); - if (OVS_LIKELY(tx_pkts)) { - /* Packets have been sent.*/ - cnt -= tx_pkts; - /* Prepare for possible retry.*/ - cur_pkts = &cur_pkts[tx_pkts]; - } else { - /* No packets sent - do not retry.*/ - break; + int idx = 0; + while (idx < cnt) { + txq->vhost_burst_pkts[txq->vhost_pkt_cnt++] = pkts[idx++]; + + if (txq->vhost_pkt_cnt >= INTERIM_QUEUE_BURST_THRESHOLD) { + dropped += netdev_dpdk_vhost_tx_burst(dev, qid); } - } while (cnt && (retries++ <= VHOST_ENQ_RETRY_NUM)); + } rte_spinlock_unlock(&dev->tx_q[qid].tx_lock); rte_spinlock_lock(&dev->stats_lock); netdev_dpdk_vhost_update_tx_counters(&dev->stats, pkts, total_pkts, - cnt + dropped); + dropped); rte_spinlock_unlock(&dev->stats_lock); - -out: - for (i = 0; i < total_pkts - dropped; i++) { - dp_packet_delete(pkts[i]); - } } /* Enqueue packets in an intermediate queue and call the flush