From patchwork Tue Aug 8 17:06:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Bodireddy, Bhanuprakash" X-Patchwork-Id: 799369 X-Patchwork-Delegate: dlu998@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xRh2H32Ccz9s7C for ; Wed, 9 Aug 2017 03:18:46 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id DDD38AAE; Tue, 8 Aug 2017 17:17:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8992BA7F for ; Tue, 8 Aug 2017 17:17:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 23249445 for ; Tue, 8 Aug 2017 17:15:29 +0000 (UTC) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2017 10:15:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,344,1498546800"; d="scan'208";a="297454188" Received: from silpixa00393942.ir.intel.com (HELO silpixa00393942.ger.corp.intel.com) ([10.237.223.42]) by fmsmga004.fm.intel.com with ESMTP; 08 Aug 2017 10:15:26 -0700 From: Bhanuprakash Bodireddy To: dev@openvswitch.org Date: Tue, 8 Aug 2017 18:06:13 +0100 Message-Id: <1502211976-76937-3-git-send-email-bhanuprakash.bodireddy@intel.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> References: <1502211976-76937-1-git-send-email-bhanuprakash.bodireddy@intel.com> X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 2/5] netdev-dpdk: Add netdev_dpdk_vhost_txq_flush function. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Add netdev_dpdk_vhost_txq_flush(), that flushes packets on vHost User port queues. Also add netdev_dpdk_vhost_tx_burst() function that uses rte_vhost_enqueue_burst() to enqueue burst of packets on vHost User ports. Signed-off-by: Bhanuprakash Bodireddy Signed-off-by: Antonio Fischetti Co-authored-by: Antonio Fischetti Acked-by: Eelco Chaudron --- lib/netdev-dpdk.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 108 insertions(+), 4 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 50d6b29..d3892fe 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -327,12 +327,22 @@ struct dpdk_tx_queue { * pmd threads (see 'concurrent_txq'). */ int map; /* Mapping of configured vhost-user queues * to enabled by guest. */ - int dpdk_pkt_cnt; /* Number of buffered packets waiting to + union { + int dpdk_pkt_cnt; /* Number of buffered packets waiting to be sent on DPDK tx queue. */ - struct rte_mbuf *dpdk_burst_pkts[INTERIM_QUEUE_BURST_THRESHOLD]; + int vhost_pkt_cnt; /* Number of buffered packets waiting to + be sent on vhost port. */ + }; + + union { + struct rte_mbuf *dpdk_burst_pkts[INTERIM_QUEUE_BURST_THRESHOLD]; /* Intermediate queue where packets can * be buffered to amortize the cost of MMIO * writes. */ + struct dp_packet *vhost_burst_pkts[INTERIM_QUEUE_BURST_THRESHOLD]; + /* Intermediate queue where packets can + * be buffered for vhost ports. */ + }; }; /* dpdk has no way to remove dpdk ring ethernet devices @@ -1756,6 +1766,88 @@ netdev_dpdk_vhost_update_tx_counters(struct netdev_stats *stats, } } +static int +netdev_dpdk_vhost_tx_burst(struct netdev_dpdk *dev, int qid) +{ + struct dpdk_tx_queue *txq = &dev->tx_q[qid]; + struct rte_mbuf **cur_pkts = (struct rte_mbuf **)txq->vhost_burst_pkts; + + int tx_vid = netdev_dpdk_get_vid(dev); + int tx_qid = qid * VIRTIO_QNUM + VIRTIO_RXQ; + uint32_t sent = 0; + uint32_t retries = 0; + uint32_t sum, total_pkts; + + total_pkts = sum = txq->vhost_pkt_cnt; + do { + uint32_t ret; + ret = rte_vhost_enqueue_burst(tx_vid, tx_qid, &cur_pkts[sent], sum); + if (OVS_UNLIKELY(!ret)) { + /* No packets enqueued - do not retry. */ + break; + } else { + /* Packet have been sent. */ + sent += ret; + + /* 'sum' packet have to be retransmitted. */ + sum -= ret; + } + } while (sum && (retries++ < VHOST_ENQ_RETRY_NUM)); + + for (int i = 0; i < total_pkts; i++) { + dp_packet_delete(txq->vhost_burst_pkts[i]); + } + + /* Reset pkt count. */ + txq->vhost_pkt_cnt = 0; + + /* 'sum' refers to packets dropped. */ + return sum; +} + +/* Flush the txq if there are any packets available. */ +static int +netdev_dpdk_vhost_txq_flush(struct netdev *netdev, int qid, + bool concurrent_txq OVS_UNUSED) +{ + struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + struct dpdk_tx_queue *txq; + + qid = dev->tx_q[qid % netdev->n_txq].map; + + /* The qid may be disabled in the guest and has been set to + * OVS_VHOST_QUEUE_DISABLED. + */ + if (OVS_UNLIKELY(qid < 0)) { + return 0; + } + + txq = &dev->tx_q[qid]; + /* Increment the drop count and free the memory. */ + if (OVS_UNLIKELY(!is_vhost_running(dev) || + !(dev->flags & NETDEV_UP))) { + + if (txq->vhost_pkt_cnt) { + rte_spinlock_lock(&dev->stats_lock); + dev->stats.tx_dropped+= txq->vhost_pkt_cnt; + rte_spinlock_unlock(&dev->stats_lock); + + for (int i = 0; i < txq->vhost_pkt_cnt; i++) { + dp_packet_delete(txq->vhost_burst_pkts[i]); + } + txq->vhost_pkt_cnt = 0; + } + } + + if (OVS_LIKELY(txq->vhost_pkt_cnt)) { + rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + netdev_dpdk_vhost_tx_burst(dev, qid); + rte_spinlock_unlock(&dev->tx_q[qid].tx_lock); + } + + return 0; +} + static void __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, struct dp_packet **pkts, int cnt) @@ -2799,6 +2891,17 @@ vring_state_changed(int vid, uint16_t queue_id, int enable) if (enable) { dev->tx_q[qid].map = qid; } else { + /* If the queue is disabled in the guest, the corresponding qid + * map shall be set to OVS_VHOST_QUEUE_DISABLED(-2). + * + * The packets that were queued in 'qid' could be potentially + * stuck and needs to be dropped. + * + * XXX: The queues may be already disabled in the guest so + * flush function in this case only helps in updating stats + * and freeing memory. + */ + netdev_dpdk_vhost_txq_flush(&dev->up, qid, 0); dev->tx_q[qid].map = OVS_VHOST_QUEUE_DISABLED; } netdev_dpdk_remap_txqs(dev); @@ -3471,7 +3574,8 @@ static const struct netdev_class dpdk_vhost_class = NULL, netdev_dpdk_vhost_reconfigure, netdev_dpdk_vhost_rxq_recv, - NULL); + netdev_dpdk_vhost_txq_flush); + static const struct netdev_class dpdk_vhost_client_class = NETDEV_DPDK_CLASS( "dpdkvhostuserclient", @@ -3487,7 +3591,7 @@ static const struct netdev_class dpdk_vhost_client_class = NULL, netdev_dpdk_vhost_client_reconfigure, netdev_dpdk_vhost_rxq_recv, - NULL); + netdev_dpdk_vhost_txq_flush); void netdev_dpdk_register(void)