From patchwork Mon Nov 18 13:33:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 1196779 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.b="hl/JB9xA"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Gqg04yYmz9sP4 for ; Tue, 19 Nov 2019 00:34:04 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726830AbfKRNeE (ORCPT ); Mon, 18 Nov 2019 08:34:04 -0500 Received: from mail.kernel.org ([198.145.29.99]:51708 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726575AbfKRNeD (ORCPT ); Mon, 18 Nov 2019 08:34:03 -0500 Received: from localhost.localdomain.com (unknown [77.139.212.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4433C2075E; Mon, 18 Nov 2019 13:34:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574084043; bh=CQ03LOxX3qfdQP4gLMGjQolBpXCyqwlZgmCwjn9Oe/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hl/JB9xAymZ+W0ZCAr8WzwkBqxcMWnxpUGHpkNZQC+Gm4ZlNClWUnAYWBprGNuR5E Lu66Gz8hzzPoXeAQakcpo1QUMJ9SMBUPn92oNMK1/XxoHob7597+KiUrVV3aqZ9JU8 fqg2BiPCfAByXWpnRXsEpKKEqrAJk4wmT6qT4tjU= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: davem@davemloft.net, ilias.apalodimas@linaro.org, brouer@redhat.com, lorenzo.bianconi@redhat.com, mcroce@redhat.com, jonathan.lemon@gmail.com Subject: [PATCH v4 net-next 1/3] net: mvneta: rely on page_pool_recycle_direct in mvneta_run_xdp Date: Mon, 18 Nov 2019 15:33:44 +0200 Message-Id: <5e6c462a912bcd6ba9284cd87272c5bff18fd4cf.1574083275.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: References: MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Rely on page_pool_recycle_direct and not on xdp_return_buff in mvneta_run_xdp. This is a preliminary patch to limit the dma sync len to the one strictly necessary Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 12e03b15f0ab..f7713c2c68e1 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2097,7 +2097,8 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, err = xdp_do_redirect(pp->dev, xdp, prog); if (err) { ret = MVNETA_XDP_DROPPED; - xdp_return_buff(xdp); + page_pool_recycle_direct(rxq->page_pool, + virt_to_head_page(xdp->data)); } else { ret = MVNETA_XDP_REDIR; } @@ -2106,7 +2107,8 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, case XDP_TX: ret = mvneta_xdp_xmit_back(pp, xdp); if (ret != MVNETA_XDP_TX) - xdp_return_buff(xdp); + page_pool_recycle_direct(rxq->page_pool, + virt_to_head_page(xdp->data)); break; default: bpf_warn_invalid_xdp_action(act); From patchwork Mon Nov 18 13:33:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 1196780 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.b="f08/5pmX"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Gqg45pxvz9sQp for ; Tue, 19 Nov 2019 00:34:08 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726992AbfKRNeH (ORCPT ); Mon, 18 Nov 2019 08:34:07 -0500 Received: from mail.kernel.org ([198.145.29.99]:51752 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726956AbfKRNeH (ORCPT ); Mon, 18 Nov 2019 08:34:07 -0500 Received: from localhost.localdomain.com (unknown [77.139.212.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 480B720748; Mon, 18 Nov 2019 13:34:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574084046; bh=9dEiXhhWnwrNC2N24xwfH2ZySDxmBMkQ3LMijcbL2hs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f08/5pmXGSAw4hNBC4K4SM5BkoNc07uHm2h8VfYSTh6InKD/WObjf7x9d1Sq7gYQa +64bBAwVHXtsxvxhlOmA2wzaQULF5mXa3NscX5NOkJo7lYvHsjAVSfhRvR0t1g9f+c Zk9Gj+RbxiyGIfUZTZnmNQysAgtVTc//aoik9l3Q= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: davem@davemloft.net, ilias.apalodimas@linaro.org, brouer@redhat.com, lorenzo.bianconi@redhat.com, mcroce@redhat.com, jonathan.lemon@gmail.com Subject: [PATCH v4 net-next 2/3] net: page_pool: add the possibility to sync DMA memory for device Date: Mon, 18 Nov 2019 15:33:45 +0200 Message-Id: <84b90677751f54c1c8d47f4036bce5999982379c.1574083275.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: References: MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce the following parameters in order to add the possibility to sync DMA memory for device before putting allocated pages in the page_pool caches: - PP_FLAG_DMA_SYNC_DEV: if set in page_pool_params flags, all pages that the driver gets from page_pool will be DMA-synced-for-device according to the length provided by the device driver. Please note DMA-sync-for-CPU is still device driver responsibility - offset: DMA address offset where the DMA engine starts copying rx data - max_len: maximum DMA memory size page_pool is allowed to flush. This is currently used in __page_pool_alloc_pages_slow routine when pages are allocated from page allocator These parameters are supposed to be set by device drivers. This optimization reduces the length of the DMA-sync-for-device. The optimization is valid because pages are initially DMA-synced-for-device as defined via max_len. At RX time, the driver will perform a DMA-sync-for-CPU on the memory for the packet length. What is important is the memory occupied by packet payload, because this is the area CPU is allowed to read and modify. As we don't track cache-lines written into by the CPU, simply use the packet payload length as dma_sync_size at page_pool recycle time. This also take into account any tail-extend. Tested-by: Matteo Croce Signed-off-by: Lorenzo Bianconi --- include/net/page_pool.h | 21 +++++++++++++++------ net/core/page_pool.c | 28 ++++++++++++++++++++++++++-- 2 files changed, 41 insertions(+), 8 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 1121faa99c12..6f684c3a3434 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -34,8 +34,15 @@ #include #include -#define PP_FLAG_DMA_MAP 1 /* Should page_pool do the DMA map/unmap */ -#define PP_FLAG_ALL PP_FLAG_DMA_MAP +#define PP_FLAG_DMA_MAP 1 /* Should page_pool do the DMA map/unmap */ +#define PP_FLAG_DMA_SYNC_DEV 2 /* if set all pages that the driver gets + * from page_pool will be + * DMA-synced-for-device according to the + * length provided by the device driver. + * Please note DMA-sync-for-CPU is still + * device driver responsibility + */ +#define PP_FLAG_ALL (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV) /* * Fast allocation side cache array/stack @@ -65,6 +72,8 @@ struct page_pool_params { int nid; /* Numa node id to allocate from pages from */ struct device *dev; /* device, for DMA pre-mapping purposes */ enum dma_data_direction dma_dir; /* DMA mapping direction */ + unsigned int max_len; /* max DMA sync memory size */ + unsigned int offset; /* DMA addr offset */ }; struct page_pool { @@ -149,8 +158,8 @@ static inline void page_pool_use_xdp_mem(struct page_pool *pool, #endif /* Never call this directly, use helpers below */ -void __page_pool_put_page(struct page_pool *pool, - struct page *page, bool allow_direct); +void __page_pool_put_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct); static inline void page_pool_put_page(struct page_pool *pool, struct page *page, bool allow_direct) @@ -159,14 +168,14 @@ static inline void page_pool_put_page(struct page_pool *pool, * allow registering MEM_TYPE_PAGE_POOL, but shield linker. */ #ifdef CONFIG_PAGE_POOL - __page_pool_put_page(pool, page, allow_direct); + __page_pool_put_page(pool, page, -1, allow_direct); #endif } /* Very limited use-cases allow recycle direct */ static inline void page_pool_recycle_direct(struct page_pool *pool, struct page *page) { - __page_pool_put_page(pool, page, true); + __page_pool_put_page(pool, page, -1, true); } /* Disconnects a page (from a page_pool). API users can have a need diff --git a/net/core/page_pool.c b/net/core/page_pool.c index dfc2501c35d9..4f9aed7bce5a 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -47,6 +47,13 @@ static int page_pool_init(struct page_pool *pool, (pool->p.dma_dir != DMA_BIDIRECTIONAL)) return -EINVAL; + /* In order to request DMA-sync-for-device the page needs to + * be mapped + */ + if ((pool->p.flags & PP_FLAG_DMA_SYNC_DEV) && + !(pool->p.flags & PP_FLAG_DMA_MAP)) + return -EINVAL; + if (ptr_ring_init(&pool->ring, ring_qsize, GFP_KERNEL) < 0) return -ENOMEM; @@ -115,6 +122,16 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) return page; } +static void page_pool_dma_sync_for_device(struct page_pool *pool, + struct page *page, + unsigned int dma_sync_size) +{ + dma_sync_size = min(dma_sync_size, pool->p.max_len); + dma_sync_single_range_for_device(pool->p.dev, page->dma_addr, + pool->p.offset, dma_sync_size, + pool->p.dma_dir); +} + /* slow path */ noinline static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, @@ -159,6 +176,9 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, } page->dma_addr = dma; + if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + page_pool_dma_sync_for_device(pool, page, pool->p.max_len); + skip_dma_map: /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; @@ -281,8 +301,8 @@ static bool __page_pool_recycle_direct(struct page *page, return true; } -void __page_pool_put_page(struct page_pool *pool, - struct page *page, bool allow_direct) +void __page_pool_put_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct) { /* This allocator is optimized for the XDP mode that uses * one-frame-per-page, but have fallbacks that act like the @@ -293,6 +313,10 @@ void __page_pool_put_page(struct page_pool *pool, if (likely(page_ref_count(page) == 1)) { /* Read barrier done in page_ref_count / READ_ONCE */ + if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + page_pool_dma_sync_for_device(pool, page, + dma_sync_size); + if (allow_direct && in_serving_softirq()) if (__page_pool_recycle_direct(page, pool)) return; From patchwork Mon Nov 18 13:33:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 1196781 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.b="S2RFkQI+"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Gqg90tlxz9sQp for ; Tue, 19 Nov 2019 00:34:12 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727022AbfKRNeL (ORCPT ); Mon, 18 Nov 2019 08:34:11 -0500 Received: from mail.kernel.org ([198.145.29.99]:51792 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726883AbfKRNeJ (ORCPT ); Mon, 18 Nov 2019 08:34:09 -0500 Received: from localhost.localdomain.com (unknown [77.139.212.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 537132075E; Mon, 18 Nov 2019 13:34:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574084049; bh=yk9UnCRQag+MRhAsMPjqi4p3IqpYTg+dCcYvgMZ8Cis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S2RFkQI++3a+oLpzNzrRYRV9+Rvq2vZB30Apa5u1QqdHpNpFmvOxaXrnJyglWV53v f3ZMxo/SwcvsXu9btFFpQz/6mMjcjAT0D5LIqJUYnmLUbpqn2H46S3DXjT6Aj/5BrH BqjVEf9SkxTYiz+QK74bCu/UCwr5UyHdAOyb+XEI= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: davem@davemloft.net, ilias.apalodimas@linaro.org, brouer@redhat.com, lorenzo.bianconi@redhat.com, mcroce@redhat.com, jonathan.lemon@gmail.com Subject: [PATCH v4 net-next 3/3] net: mvneta: get rid of huge dma sync in mvneta_rx_refill Date: Mon, 18 Nov 2019 15:33:46 +0200 Message-Id: <7bd772e5376af0c55e7319b7974439d4981aa167.1574083275.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: References: MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Get rid of costly dma_sync_single_for_device in mvneta_rx_refill since now the driver can let page_pool API to manage needed DMA sync with a proper size. - XDP_DROP DMA sync managed by mvneta driver: ~420Kpps - XDP_DROP DMA sync managed by page_pool API: ~585Kpps Tested-by: Matteo Croce Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index f7713c2c68e1..a06d109c9e80 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -1846,7 +1846,6 @@ static int mvneta_rx_refill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, gfp_t gfp_mask) { - enum dma_data_direction dma_dir; dma_addr_t phys_addr; struct page *page; @@ -1856,9 +1855,6 @@ static int mvneta_rx_refill(struct mvneta_port *pp, return -ENOMEM; phys_addr = page_pool_get_dma_addr(page) + pp->rx_offset_correction; - dma_dir = page_pool_get_dma_dir(rxq->page_pool); - dma_sync_single_for_device(pp->dev->dev.parent, phys_addr, - MVNETA_MAX_RX_BUF_SIZE, dma_dir); mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq); return 0; @@ -2097,8 +2093,10 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, err = xdp_do_redirect(pp->dev, xdp, prog); if (err) { ret = MVNETA_XDP_DROPPED; - page_pool_recycle_direct(rxq->page_pool, - virt_to_head_page(xdp->data)); + __page_pool_put_page(rxq->page_pool, + virt_to_head_page(xdp->data), + xdp->data_end - xdp->data_hard_start, + true); } else { ret = MVNETA_XDP_REDIR; } @@ -2107,8 +2105,10 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, case XDP_TX: ret = mvneta_xdp_xmit_back(pp, xdp); if (ret != MVNETA_XDP_TX) - page_pool_recycle_direct(rxq->page_pool, - virt_to_head_page(xdp->data)); + __page_pool_put_page(rxq->page_pool, + virt_to_head_page(xdp->data), + xdp->data_end - xdp->data_hard_start, + true); break; default: bpf_warn_invalid_xdp_action(act); @@ -2117,8 +2117,10 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, trace_xdp_exception(pp->dev, prog, act); /* fall through */ case XDP_DROP: - page_pool_recycle_direct(rxq->page_pool, - virt_to_head_page(xdp->data)); + __page_pool_put_page(rxq->page_pool, + virt_to_head_page(xdp->data), + xdp->data_end - xdp->data_hard_start, + true); ret = MVNETA_XDP_DROPPED; break; } @@ -3067,11 +3069,13 @@ static int mvneta_create_page_pool(struct mvneta_port *pp, struct bpf_prog *xdp_prog = READ_ONCE(pp->xdp_prog); struct page_pool_params pp_params = { .order = 0, - .flags = PP_FLAG_DMA_MAP, + .flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV, .pool_size = size, .nid = cpu_to_node(0), .dev = pp->dev->dev.parent, .dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE, + .offset = pp->rx_offset_correction, + .max_len = MVNETA_MAX_RX_BUF_SIZE, }; int err;