From patchwork Fri Apr 13 17:25:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 898029 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40N4RP66KVz9s0t for ; Sat, 14 Apr 2018 03:25:21 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 4AEE2CCE; Fri, 13 Apr 2018 17:25:19 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 711EFCBB for ; Fri, 13 Apr 2018 17:25:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8CAD9672 for ; Fri, 13 Apr 2018 17:25:16 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B40B426DBC7; Fri, 13 Apr 2018 17:25:15 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.117.255]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE40310F1BF0; Fri, 13 Apr 2018 17:25:13 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org Date: Fri, 13 Apr 2018 18:25:08 +0100 Message-Id: <1523640309-22024-1-git-send-email-ktraynor@redhat.com> In-Reply-To: <1523383980-14865-1-git-send-email-ktraynor@redhat.com> References: <1523383980-14865-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Fri, 13 Apr 2018 17:25:15 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Fri, 13 Apr 2018 17:25:15 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'ktraynor@redhat.com' RCPT:'' X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , mark.b.kavanagh81@gmail.com Subject: [ovs-dev] [branch 2.9 PATCH v3 1/2] netdev-dpdk: Free mempool only when no in-use mbufs. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org DPDK mempools are freed when they are no longer needed. This can happen when a port is removed or a port's mtu is reconfigured so that a new mempool is used. It is possible that an mbuf is attempted to be returned to a freed mempool from NIC Tx queues and this can lead to a segfault. In order to prevent this, only free mempools when they are not needed and have no in-use mbufs. As this might not be possible immediately, sweep the mempools anytime a port tries to get a mempool. Fixes: 8d38823bdf8b ("netdev-dpdk: fix memory leak") Cc: mark.b.kavanagh81@gmail.com Cc: Ilya Maximets Reported-by: Venkatesan Pradeep Signed-off-by: Kevin Traynor --- v3: Get number of entries on the mempool ring directly. v2: Add second call to rte_mempool_full() as it is not atomic so we can't trust one call to it. lib/netdev-dpdk.c | 58 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index c19cedc..d8fb222 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -590,8 +590,42 @@ dpdk_mp_create(int socket_id, int mtu) } +static int +dpdk_mp_full(const struct rte_mempool *mp) OVS_REQUIRES(dpdk_mp_mutex) +{ + unsigned ring_count; + /* This logic is needed because rte_mempool_full() is not guaranteed to + * be atomic and mbufs could be moved from mempool cache --> mempool ring + * during the call. However, as no mbufs will be taken from the mempool + * at this time, we can work around it by also checking the ring entries + * separately and ensuring that they have not changed. + */ + ring_count = rte_mempool_ops_get_count(mp); + if (rte_mempool_full(mp) && rte_mempool_ops_get_count(mp) == ring_count) { + return 1; + } + + return 0; +} + +/* Free unused mempools. */ +static void +dpdk_mp_sweep(void) OVS_REQUIRES(dpdk_mp_mutex) +{ + struct dpdk_mp *dmp, *next; + + LIST_FOR_EACH_SAFE (dmp, next, list_node, &dpdk_mp_list) { + if (!dmp->refcount && dpdk_mp_full(dmp->mp)) { + ovs_list_remove(&dmp->list_node); + rte_mempool_free(dmp->mp); + rte_free(dmp); + } + } +} + static struct dpdk_mp * dpdk_mp_get(int socket_id, int mtu) { struct dpdk_mp *dmp; + bool reuse = false; ovs_mutex_lock(&dpdk_mp_mutex); @@ -599,15 +633,18 @@ dpdk_mp_get(int socket_id, int mtu) if (dmp->socket_id == socket_id && dmp->mtu == mtu) { dmp->refcount++; - goto out; + reuse = true; + break; } } + /* Sweep mempools after reuse or before create. */ + dpdk_mp_sweep(); - dmp = dpdk_mp_create(socket_id, mtu); - if (dmp) { - ovs_list_push_back(&dpdk_mp_list, &dmp->list_node); + if (!reuse) { + dmp = dpdk_mp_create(socket_id, mtu); + if (dmp) { + ovs_list_push_back(&dpdk_mp_list, &dmp->list_node); + } } -out: - ovs_mutex_unlock(&dpdk_mp_mutex); @@ -615,5 +652,5 @@ out: } -/* Release an existing mempool. */ +/* Decrement reference to a mempool. */ static void dpdk_mp_put(struct dpdk_mp *dmp) @@ -625,10 +662,5 @@ dpdk_mp_put(struct dpdk_mp *dmp) ovs_mutex_lock(&dpdk_mp_mutex); ovs_assert(dmp->refcount); - - if (!--dmp->refcount) { - ovs_list_remove(&dmp->list_node); - rte_mempool_free(dmp->mp); - rte_free(dmp); - } + dmp->refcount--; ovs_mutex_unlock(&dpdk_mp_mutex); }