From patchwork Wed Oct 10 16:22:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Lam, Tiago" X-Patchwork-Id: 981985 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42VfcB2swSz9s3l for ; Thu, 11 Oct 2018 03:26:18 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 015EAA74; Wed, 10 Oct 2018 16:23:53 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id DB4B3A7C for ; Wed, 10 Oct 2018 16:23:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 791ECD0 for ; Wed, 10 Oct 2018 16:23:51 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Oct 2018 09:23:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,364,1534834800"; d="scan'208";a="81516410" Received: from silpixa00399125.ir.intel.com ([10.237.223.34]) by orsmga006.jf.intel.com with ESMTP; 10 Oct 2018 09:23:49 -0700 From: Tiago Lam To: ovs-dev@openvswitch.org Date: Wed, 10 Oct 2018 17:22:22 +0100 Message-Id: <1539188552-129083-5-git-send-email-tiago.lam@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1539188552-129083-1-git-send-email-tiago.lam@intel.com> References: <1539188552-129083-1-git-send-email-tiago.lam@intel.com> X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: fbl@sysclose.org, i.maximets@samsung.com Subject: [ovs-dev] [PATCH v11 04/14] netdev-dpdk: Serialise non-pmds mbufs' alloc/free. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org A new mutex, 'nonpmd_mp_mutex', has been introduced to serialise allocation and free operations by non-pmd threads on a given mempool. free_dpdk_buf() has been modified to make use of the introduced mutex. Signed-off-by: Tiago Lam Acked-by: Eelco Chaudron Acked-by: Flavio Leitner --- lib/netdev-dpdk.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index b6a22bd..f5044ca 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -322,6 +322,16 @@ static struct ovs_mutex dpdk_mp_mutex OVS_ACQ_AFTER(dpdk_mutex) static struct ovs_list dpdk_mp_list OVS_GUARDED_BY(dpdk_mp_mutex) = OVS_LIST_INITIALIZER(&dpdk_mp_list); +/* This mutex must be used by non pmd threads when allocating or freeing + * mbufs through mempools, when outside of the `non_pmd_mutex` mutex, in struct + * dp_netdev. + * The reason, as pointed out in the "Known Issues" section in DPDK's EAL docs, + * is that the implementation on which mempool is based off is non-preemptable. + * Since non-pmds may end up not being pinned this could lead to the preemption + * between non-pmds performing operations on the same mempool, which could lead + * to memory corruption. */ +static struct ovs_mutex nonpmd_mp_mutex = OVS_MUTEX_INITIALIZER; + struct dpdk_mp { struct rte_mempool *mp; int mtu; @@ -492,6 +502,8 @@ struct netdev_rxq_dpdk { dpdk_port_t port_id; }; +static bool dpdk_thread_is_pmd(void); + static void netdev_dpdk_destruct(struct netdev *netdev); static void netdev_dpdk_vhost_destruct(struct netdev *netdev); @@ -525,6 +537,12 @@ dpdk_buf_size(int mtu) NETDEV_DPDK_MBUF_ALIGN); } +static bool +dpdk_thread_is_pmd(void) +{ + return rte_lcore_id() != NON_PMD_CORE_ID; +} + /* Allocates an area of 'sz' bytes from DPDK. The memory is zero'ed. * * Unlike xmalloc(), this function can return NULL on failure. */ @@ -535,11 +553,20 @@ dpdk_rte_mzalloc(size_t sz) } void -free_dpdk_buf(struct dp_packet *p) +free_dpdk_buf(struct dp_packet *packet) { - struct rte_mbuf *pkt = (struct rte_mbuf *) p; + /* If non-pmd we need to lock on nonpmd_mp_mutex mutex */ + if (!dpdk_thread_is_pmd()) { + ovs_mutex_lock(&nonpmd_mp_mutex); + + rte_pktmbuf_free(&packet->mbuf); + + ovs_mutex_unlock(&nonpmd_mp_mutex); + + return; + } - rte_pktmbuf_free(pkt); + rte_pktmbuf_free(&packet->mbuf); } static void