Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/567218/?format=api
{ "id": 567218, "url": "http://patchwork.ozlabs.org/api/patches/567218/?format=api", "web_url": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/1452732712-12248-13-git-send-email-joshua.a.hay@intel.com/", "project": { "id": 46, "url": "http://patchwork.ozlabs.org/api/projects/46/?format=api", "name": "Intel Wired Ethernet development", "link_name": "intel-wired-lan", "list_id": "intel-wired-lan.osuosl.org", "list_email": "intel-wired-lan@osuosl.org", "web_url": "", "scm_url": "", "webscm_url": "", "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<1452732712-12248-13-git-send-email-joshua.a.hay@intel.com>", "list_archive_url": null, "date": "2016-01-14T00:51:49", "name": "[v2,next,S27,12/15] i40e/i40evf: use pages correctly in rx", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": false, "hash": "1d6f01100c0f8e8a52b70043c994a5e544318a1d", "submitter": { "id": 19461, "url": "http://patchwork.ozlabs.org/api/people/19461/?format=api", "name": "Joshua Hay", "email": "joshua.a.hay@intel.com" }, "delegate": { "id": 68, "url": "http://patchwork.ozlabs.org/api/users/68/?format=api", "username": "jtkirshe", "first_name": "Jeff", "last_name": "Kirsher", "email": "jeffrey.t.kirsher@intel.com" }, "mbox": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/1452732712-12248-13-git-send-email-joshua.a.hay@intel.com/mbox/", "series": [], "comments": "http://patchwork.ozlabs.org/api/patches/567218/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/567218/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<intel-wired-lan-bounces@lists.osuosl.org>", "X-Original-To": [ "incoming@patchwork.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Delivered-To": [ "patchwork-incoming@bilbo.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Received": [ "from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137])\n\tby ozlabs.org (Postfix) with ESMTP id A3A15140B0E\n\tfor <incoming@patchwork.ozlabs.org>;\n\tThu, 14 Jan 2016 11:52:05 +1100 (AEDT)", "from localhost (localhost [127.0.0.1])\n\tby fraxinus.osuosl.org (Postfix) with ESMTP id 3B93EA596F;\n\tThu, 14 Jan 2016 00:52:05 +0000 (UTC)", "from fraxinus.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id JwPQf7TX3sYR; Thu, 14 Jan 2016 00:52:01 +0000 (UTC)", "from ash.osuosl.org (ash.osuosl.org [140.211.166.34])\n\tby fraxinus.osuosl.org (Postfix) with ESMTP id 0D720A5944;\n\tThu, 14 Jan 2016 00:52:01 +0000 (UTC)", "from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133])\n\tby ash.osuosl.org (Postfix) with ESMTP id 1FA721C0F4A\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 14 Jan 2016 00:51:55 +0000 (UTC)", "from localhost (localhost [127.0.0.1])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id 1B5538A7BC\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 14 Jan 2016 00:51:55 +0000 (UTC)", "from hemlock.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id Icdz1jh5Stza for <intel-wired-lan@lists.osuosl.org>;\n\tThu, 14 Jan 2016 00:51:54 +0000 (UTC)", "from mga03.intel.com (mga03.intel.com [134.134.136.65])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id 93AED932EE\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tThu, 14 Jan 2016 00:51:53 +0000 (UTC)", "from orsmga002.jf.intel.com ([10.7.209.21])\n\tby orsmga103.jf.intel.com with ESMTP; 13 Jan 2016 16:51:52 -0800", "from jahay1-mobl2.amr.corp.intel.com (HELO\n\tlocalhost.localdomain.localdomain) ([134.134.176.160])\n\tby orsmga002.jf.intel.com with ESMTP; 13 Jan 2016 16:51:52 -0800" ], "X-Virus-Scanned": [ "amavisd-new at osuosl.org", "amavisd-new at osuosl.org" ], "X-Greylist": "domain auto-whitelisted by SQLgrey-1.7.6", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.22,291,1449561600\"; d=\"scan'208\";a=\"890056927\"", "From": "Joshua Hay <joshua.a.hay@intel.com>", "To": "intel-wired-lan@lists.osuosl.org", "Date": "Wed, 13 Jan 2016 16:51:49 -0800", "Message-Id": "<1452732712-12248-13-git-send-email-joshua.a.hay@intel.com>", "X-Mailer": "git-send-email 2.1.0", "In-Reply-To": "<1452732712-12248-1-git-send-email-joshua.a.hay@intel.com>", "References": "<1452732712-12248-1-git-send-email-joshua.a.hay@intel.com>", "Subject": "[Intel-wired-lan] [v2 next PATCH S27 12/15] i40e/i40evf: use pages\n\tcorrectly in rx", "X-BeenThere": "intel-wired-lan@lists.osuosl.org", "X-Mailman-Version": "2.1.18-1", "Precedence": "list", "List-Id": "Intel Wired Ethernet Linux Kernel Driver Development\n\t<intel-wired-lan.lists.osuosl.org>", "List-Unsubscribe": "<http://lists.osuosl.org/mailman/options/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@lists.osuosl.org?subject=unsubscribe>", "List-Archive": "<http://lists.osuosl.org/pipermail/intel-wired-lan/>", "List-Post": "<mailto:intel-wired-lan@lists.osuosl.org>", "List-Help": "<mailto:intel-wired-lan-request@lists.osuosl.org?subject=help>", "List-Subscribe": "<http://lists.osuosl.org/mailman/listinfo/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@lists.osuosl.org?subject=subscribe>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "7bit", "Errors-To": "intel-wired-lan-bounces@lists.osuosl.org", "Sender": "\"Intel-wired-lan\" <intel-wired-lan-bounces@lists.osuosl.org>" }, "content": "From: Mitch Williams <mitch.a.williams@intel.com>\n\nRefactor the packet split rx code to properly use half-pages for\nreceives. The previous code was doing way more mapping and unmapping\nthan it needed to, and wasn't properly using half-pages.\n\nIncrement the page use count each time we give a half-page to an skb,\nknowing that the stack will probably process and release the page before\nwe need it again. Only free and reallocate pages if the count shows that\nboth half-pages are in use. Add counters to track reallocations and page\nreuse.\n\nSigned-off-by: Mitch Williams <mitch.a.williams@intel.com>\nChange-ID: I534b299196036b64be82b4861a0a4036310a8f22\n---\nTesting Hints: Hammer the receive path and make\nsure it a) works and b) doesn't leak memory.\n\n drivers/net/ethernet/intel/i40e/i40e_debugfs.c | 5 ++\n drivers/net/ethernet/intel/i40e/i40e_txrx.c | 118 ++++++++++++++++---------\n drivers/net/ethernet/intel/i40e/i40e_txrx.h | 2 +\n drivers/net/ethernet/intel/i40evf/i40e_txrx.c | 118 ++++++++++++++++---------\n drivers/net/ethernet/intel/i40evf/i40e_txrx.h | 2 +\n 5 files changed, 159 insertions(+), 86 deletions(-)", "diff": "diff --git a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c\nindex fcae3c8..bdac691 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c\n+++ b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c\n@@ -536,6 +536,11 @@ static void i40e_dbg_dump_vsi_seid(struct i40e_pf *pf, int seid)\n \t\t\t rx_ring->rx_stats.alloc_page_failed,\n \t\t\t rx_ring->rx_stats.alloc_buff_failed);\n \t\tdev_info(&pf->pdev->dev,\n+\t\t\t \" rx_rings[%i]: rx_stats: realloc_count = %lld, page_reuse_count = %lld\\n\",\n+\t\t\t i,\n+\t\t\t rx_ring->rx_stats.realloc_count,\n+\t\t\t rx_ring->rx_stats.page_reuse_count);\n+\t\tdev_info(&pf->pdev->dev,\n \t\t\t \" rx_rings[%i]: size = %i, dma = 0x%08lx\\n\",\n \t\t\t i, rx_ring->size,\n \t\t\t (unsigned long int)rx_ring->dma);\ndiff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c\nindex 9efd988..1ce572e 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c\n+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c\n@@ -1060,7 +1060,7 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring)\n \t\t\tif (rx_bi->page_dma) {\n \t\t\t\tdma_unmap_page(dev,\n \t\t\t\t\t rx_bi->page_dma,\n-\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t PAGE_SIZE,\n \t\t\t\t\t DMA_FROM_DEVICE);\n \t\t\t\trx_bi->page_dma = 0;\n \t\t\t}\n@@ -1203,6 +1203,7 @@ bool i40e_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)\n \tu16 i = rx_ring->next_to_use;\n \tunion i40e_rx_desc *rx_desc;\n \tstruct i40e_rx_buffer *bi;\n+\tconst int current_node = numa_node_id();\n \n \t/* do nothing if no valid netdev defined */\n \tif (!rx_ring->netdev || !cleaned_count)\n@@ -1214,39 +1215,50 @@ bool i40e_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)\n \n \t\tif (bi->skb) /* desc is in use */\n \t\t\tgoto no_buffers;\n+\n+\t/* If we've been moved to a different NUMA node, release the\n+\t * page so we can get a new one on the current node.\n+\t */\n+\t\tif (bi->page && page_to_nid(bi->page) != current_node) {\n+\t\t\tdma_unmap_page(rx_ring->dev,\n+\t\t\t\t bi->page_dma,\n+\t\t\t\t PAGE_SIZE,\n+\t\t\t\t DMA_FROM_DEVICE);\n+\t\t\t__free_page(bi->page);\n+\t\t\tbi->page = NULL;\n+\t\t\tbi->page_dma = 0;\n+\t\t\trx_ring->rx_stats.realloc_count++;\n+\t\t} else if (bi->page) {\n+\t\t\trx_ring->rx_stats.page_reuse_count++;\n+\t\t}\n+\n \t\tif (!bi->page) {\n \t\t\tbi->page = alloc_page(GFP_ATOMIC);\n \t\t\tif (!bi->page) {\n \t\t\t\trx_ring->rx_stats.alloc_page_failed++;\n \t\t\t\tgoto no_buffers;\n \t\t\t}\n-\t\t}\n-\n-\t\tif (!bi->page_dma) {\n-\t\t\t/* use a half page if we're re-using */\n-\t\t\tbi->page_offset ^= PAGE_SIZE / 2;\n \t\t\tbi->page_dma = dma_map_page(rx_ring->dev,\n \t\t\t\t\t\t bi->page,\n-\t\t\t\t\t\t bi->page_offset,\n-\t\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t\t 0,\n+\t\t\t\t\t\t PAGE_SIZE,\n \t\t\t\t\t\t DMA_FROM_DEVICE);\n-\t\t\tif (dma_mapping_error(rx_ring->dev,\n-\t\t\t\t\t bi->page_dma)) {\n+\t\t\tif (dma_mapping_error(rx_ring->dev, bi->page_dma)) {\n \t\t\t\trx_ring->rx_stats.alloc_page_failed++;\n+\t\t\t\t__free_page(bi->page);\n+\t\t\t\tbi->page = NULL;\n \t\t\t\tbi->page_dma = 0;\n+\t\t\t\tbi->page_offset = 0;\n \t\t\t\tgoto no_buffers;\n \t\t\t}\n+\t\t\tbi->page_offset = 0;\n \t\t}\n \n-\t\tdma_sync_single_range_for_device(rx_ring->dev,\n-\t\t\t\t\t\t rx_ring->rx_bi[0].dma,\n-\t\t\t\t\t\t i * rx_ring->rx_hdr_len,\n-\t\t\t\t\t\t rx_ring->rx_hdr_len,\n-\t\t\t\t\t\t DMA_FROM_DEVICE);\n \t\t/* Refresh the desc even if buffer_addrs didn't change\n \t\t * because each write-back erases this info.\n \t\t */\n-\t\trx_desc->read.pkt_addr = cpu_to_le64(bi->page_dma);\n+\t\trx_desc->read.pkt_addr =\n+\t\t\t\tcpu_to_le64(bi->page_dma + bi->page_offset);\n \t\trx_desc->read.hdr_addr = cpu_to_le64(bi->dma);\n \t\ti++;\n \t\tif (i == rx_ring->count)\n@@ -1527,7 +1539,6 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \tunsigned int total_rx_bytes = 0, total_rx_packets = 0;\n \tu16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;\n \tu16 cleaned_count = I40E_DESC_UNUSED(rx_ring);\n-\tconst int current_node = numa_mem_id();\n \tstruct i40e_vsi *vsi = rx_ring->vsi;\n \tu16 i = rx_ring->next_to_clean;\n \tunion i40e_rx_desc *rx_desc;\n@@ -1535,6 +1546,7 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \tbool failure = false;\n \tu8 rx_ptype;\n \tu64 qword;\n+\tu32 copysize;\n \n \tif (budget <= 0)\n \t\treturn 0;\n@@ -1565,6 +1577,12 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \t\t * DD bit is set.\n \t\t */\n \t\tdma_rmb();\n+\t\t/* sync header buffer for reading */\n+\t\tdma_sync_single_range_for_cpu(rx_ring->dev,\n+\t\t\t\t\t rx_ring->rx_bi[0].dma,\n+\t\t\t\t\t i * rx_ring->rx_hdr_len,\n+\t\t\t\t\t rx_ring->rx_hdr_len,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n \t\tif (i40e_rx_is_programming_status(qword)) {\n \t\t\ti40e_clean_programming_status(rx_ring, rx_desc);\n \t\t\tI40E_RX_INCREMENT(rx_ring, i);\n@@ -1606,9 +1624,16 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \n \t\trx_ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >>\n \t\t\t I40E_RXD_QW1_PTYPE_SHIFT;\n-\t\tprefetch(rx_bi->page);\n+\t\t/* sync half-page for reading */\n+\t\tdma_sync_single_range_for_cpu(rx_ring->dev,\n+\t\t\t\t\t rx_bi->page_dma,\n+\t\t\t\t\t rx_bi->page_offset,\n+\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n+\t\tprefetch(page_address(rx_bi->page) + rx_bi->page_offset);\n \t\trx_bi->skb = NULL;\n \t\tcleaned_count++;\n+\t\tcopysize = 0;\n \t\tif (rx_hbo || rx_sph) {\n \t\t\tint len;\n \n@@ -1619,38 +1644,45 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \t\t\tmemcpy(__skb_put(skb, len), rx_bi->hdr_buf, len);\n \t\t} else if (skb->len == 0) {\n \t\t\tint len;\n+\t\t\tunsigned char *va = page_address(rx_bi->page) +\n+\t\t\t\t\t rx_bi->page_offset;\n \n-\t\t\tlen = (rx_packet_len > skb_headlen(skb) ?\n-\t\t\t\tskb_headlen(skb) : rx_packet_len);\n-\t\t\tmemcpy(__skb_put(skb, len),\n-\t\t\t rx_bi->page + rx_bi->page_offset,\n-\t\t\t len);\n-\t\t\trx_bi->page_offset += len;\n+\t\t\tlen = min(rx_packet_len, rx_ring->rx_hdr_len);\n+\t\t\tmemcpy(__skb_put(skb, len), va, len);\n+\t\t\tcopysize = len;\n \t\t\trx_packet_len -= len;\n \t\t}\n-\n \t\t/* Get the rest of the data if this was a header split */\n \t\tif (rx_packet_len) {\n-\t\t\tskb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,\n-\t\t\t\t\t rx_bi->page,\n-\t\t\t\t\t rx_bi->page_offset,\n-\t\t\t\t\t rx_packet_len);\n-\n-\t\t\tskb->len += rx_packet_len;\n-\t\t\tskb->data_len += rx_packet_len;\n-\t\t\tskb->truesize += rx_packet_len;\n-\n-\t\t\tif ((page_count(rx_bi->page) == 1) &&\n-\t\t\t (page_to_nid(rx_bi->page) == current_node))\n-\t\t\t\tget_page(rx_bi->page);\n-\t\t\telse\n+\t\t\tskb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,\n+\t\t\t\t\trx_bi->page,\n+\t\t\t\t\trx_bi->page_offset + copysize,\n+\t\t\t\t\trx_packet_len, I40E_RXBUFFER_2048);\n+\n+\t\t\tget_page(rx_bi->page);\n+\t\t\t/* switch to the other half-page here; the allocation\n+\t\t\t * code programs the right addr into HW. If we haven't\n+\t\t\t * used this half-page, the address won't be changed,\n+\t\t\t * and HW can just use it next time through.\n+\t\t\t */\n+\t\t\trx_bi->page_offset ^= PAGE_SIZE / 2;\n+\t\t\t/* If the page count is more than 2, then both halves\n+\t\t\t * of the page are used and we need to free it. Do it\n+\t\t\t * here instead of in the alloc code. Otherwise one\n+\t\t\t * of the half-pages might be released between now and\n+\t\t\t * then, and we wouldn't know which one to use.\n+\t\t\t */\n+\t\t\tif (page_count(rx_bi->page) > 2) {\n+\t\t\t\tdma_unmap_page(rx_ring->dev,\n+\t\t\t\t\t rx_bi->page_dma,\n+\t\t\t\t\t PAGE_SIZE,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n+\t\t\t\t__free_page(rx_bi->page);\n \t\t\t\trx_bi->page = NULL;\n+\t\t\t\trx_bi->page_dma = 0;\n+\t\t\t\trx_ring->rx_stats.realloc_count++;\n+\t\t\t}\n \n-\t\t\tdma_unmap_page(rx_ring->dev,\n-\t\t\t\t rx_bi->page_dma,\n-\t\t\t\t PAGE_SIZE / 2,\n-\t\t\t\t DMA_FROM_DEVICE);\n-\t\t\trx_bi->page_dma = 0;\n \t\t}\n \t\tI40E_RX_INCREMENT(rx_ring, i);\n \ndiff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.h b/drivers/net/ethernet/intel/i40e/i40e_txrx.h\nindex 2e52889..e538b40 100644\n--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.h\n+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.h\n@@ -209,6 +209,8 @@ struct i40e_rx_queue_stats {\n \tu64 non_eop_descs;\n \tu64 alloc_page_failed;\n \tu64 alloc_buff_failed;\n+\tu64 page_reuse_count;\n+\tu64 realloc_count;\n };\n \n enum i40e_ring_state_t {\ndiff --git a/drivers/net/ethernet/intel/i40evf/i40e_txrx.c b/drivers/net/ethernet/intel/i40evf/i40e_txrx.c\nindex d3ec3b2..f971018 100644\n--- a/drivers/net/ethernet/intel/i40evf/i40e_txrx.c\n+++ b/drivers/net/ethernet/intel/i40evf/i40e_txrx.c\n@@ -532,7 +532,7 @@ void i40evf_clean_rx_ring(struct i40e_ring *rx_ring)\n \t\t\tif (rx_bi->page_dma) {\n \t\t\t\tdma_unmap_page(dev,\n \t\t\t\t\t rx_bi->page_dma,\n-\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t PAGE_SIZE,\n \t\t\t\t\t DMA_FROM_DEVICE);\n \t\t\t\trx_bi->page_dma = 0;\n \t\t\t}\n@@ -675,6 +675,7 @@ bool i40evf_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)\n \tu16 i = rx_ring->next_to_use;\n \tunion i40e_rx_desc *rx_desc;\n \tstruct i40e_rx_buffer *bi;\n+\tconst int current_node = numa_node_id();\n \n \t/* do nothing if no valid netdev defined */\n \tif (!rx_ring->netdev || !cleaned_count)\n@@ -686,39 +687,50 @@ bool i40evf_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)\n \n \t\tif (bi->skb) /* desc is in use */\n \t\t\tgoto no_buffers;\n+\n+\t/* If we've been moved to a different NUMA node, release the\n+\t * page so we can get a new one on the current node.\n+\t */\n+\t\tif (bi->page && page_to_nid(bi->page) != current_node) {\n+\t\t\tdma_unmap_page(rx_ring->dev,\n+\t\t\t\t bi->page_dma,\n+\t\t\t\t PAGE_SIZE,\n+\t\t\t\t DMA_FROM_DEVICE);\n+\t\t\t__free_page(bi->page);\n+\t\t\tbi->page = NULL;\n+\t\t\tbi->page_dma = 0;\n+\t\t\trx_ring->rx_stats.realloc_count++;\n+\t\t} else if (bi->page) {\n+\t\t\trx_ring->rx_stats.page_reuse_count++;\n+\t\t}\n+\n \t\tif (!bi->page) {\n \t\t\tbi->page = alloc_page(GFP_ATOMIC);\n \t\t\tif (!bi->page) {\n \t\t\t\trx_ring->rx_stats.alloc_page_failed++;\n \t\t\t\tgoto no_buffers;\n \t\t\t}\n-\t\t}\n-\n-\t\tif (!bi->page_dma) {\n-\t\t\t/* use a half page if we're re-using */\n-\t\t\tbi->page_offset ^= PAGE_SIZE / 2;\n \t\t\tbi->page_dma = dma_map_page(rx_ring->dev,\n \t\t\t\t\t\t bi->page,\n-\t\t\t\t\t\t bi->page_offset,\n-\t\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t\t 0,\n+\t\t\t\t\t\t PAGE_SIZE,\n \t\t\t\t\t\t DMA_FROM_DEVICE);\n-\t\t\tif (dma_mapping_error(rx_ring->dev,\n-\t\t\t\t\t bi->page_dma)) {\n+\t\t\tif (dma_mapping_error(rx_ring->dev, bi->page_dma)) {\n \t\t\t\trx_ring->rx_stats.alloc_page_failed++;\n+\t\t\t\t__free_page(bi->page);\n+\t\t\t\tbi->page = NULL;\n \t\t\t\tbi->page_dma = 0;\n+\t\t\t\tbi->page_offset = 0;\n \t\t\t\tgoto no_buffers;\n \t\t\t}\n+\t\t\tbi->page_offset = 0;\n \t\t}\n \n-\t\tdma_sync_single_range_for_device(rx_ring->dev,\n-\t\t\t\t\t\t rx_ring->rx_bi[0].dma,\n-\t\t\t\t\t\t i * rx_ring->rx_hdr_len,\n-\t\t\t\t\t\t rx_ring->rx_hdr_len,\n-\t\t\t\t\t\t DMA_FROM_DEVICE);\n \t\t/* Refresh the desc even if buffer_addrs didn't change\n \t\t * because each write-back erases this info.\n \t\t */\n-\t\trx_desc->read.pkt_addr = cpu_to_le64(bi->page_dma);\n+\t\trx_desc->read.pkt_addr =\n+\t\t\t\tcpu_to_le64(bi->page_dma + bi->page_offset);\n \t\trx_desc->read.hdr_addr = cpu_to_le64(bi->dma);\n \t\ti++;\n \t\tif (i == rx_ring->count)\n@@ -998,7 +1010,6 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \tunsigned int total_rx_bytes = 0, total_rx_packets = 0;\n \tu16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;\n \tu16 cleaned_count = I40E_DESC_UNUSED(rx_ring);\n-\tconst int current_node = numa_mem_id();\n \tstruct i40e_vsi *vsi = rx_ring->vsi;\n \tu16 i = rx_ring->next_to_clean;\n \tunion i40e_rx_desc *rx_desc;\n@@ -1006,6 +1017,7 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \tbool failure = false;\n \tu8 rx_ptype;\n \tu64 qword;\n+\tu32 copysize;\n \n \tdo {\n \t\tstruct i40e_rx_buffer *rx_bi;\n@@ -1033,6 +1045,12 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \t\t * DD bit is set.\n \t\t */\n \t\tdma_rmb();\n+\t\t/* sync header buffer for reading */\n+\t\tdma_sync_single_range_for_cpu(rx_ring->dev,\n+\t\t\t\t\t rx_ring->rx_bi[0].dma,\n+\t\t\t\t\t i * rx_ring->rx_hdr_len,\n+\t\t\t\t\t rx_ring->rx_hdr_len,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n \t\trx_bi = &rx_ring->rx_bi[i];\n \t\tskb = rx_bi->skb;\n \t\tif (likely(!skb)) {\n@@ -1069,9 +1087,16 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \n \t\trx_ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >>\n \t\t\t I40E_RXD_QW1_PTYPE_SHIFT;\n-\t\tprefetch(rx_bi->page);\n+\t\t/* sync half-page for reading */\n+\t\tdma_sync_single_range_for_cpu(rx_ring->dev,\n+\t\t\t\t\t rx_bi->page_dma,\n+\t\t\t\t\t rx_bi->page_offset,\n+\t\t\t\t\t PAGE_SIZE / 2,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n+\t\tprefetch(page_address(rx_bi->page) + rx_bi->page_offset);\n \t\trx_bi->skb = NULL;\n \t\tcleaned_count++;\n+\t\tcopysize = 0;\n \t\tif (rx_hbo || rx_sph) {\n \t\t\tint len;\n \n@@ -1082,38 +1107,45 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)\n \t\t\tmemcpy(__skb_put(skb, len), rx_bi->hdr_buf, len);\n \t\t} else if (skb->len == 0) {\n \t\t\tint len;\n+\t\t\tunsigned char *va = page_address(rx_bi->page) +\n+\t\t\t\t\t rx_bi->page_offset;\n \n-\t\t\tlen = (rx_packet_len > skb_headlen(skb) ?\n-\t\t\t\tskb_headlen(skb) : rx_packet_len);\n-\t\t\tmemcpy(__skb_put(skb, len),\n-\t\t\t rx_bi->page + rx_bi->page_offset,\n-\t\t\t len);\n-\t\t\trx_bi->page_offset += len;\n+\t\t\tlen = min(rx_packet_len, rx_ring->rx_hdr_len);\n+\t\t\tmemcpy(__skb_put(skb, len), va, len);\n+\t\t\tcopysize = len;\n \t\t\trx_packet_len -= len;\n \t\t}\n-\n \t\t/* Get the rest of the data if this was a header split */\n \t\tif (rx_packet_len) {\n-\t\t\tskb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,\n-\t\t\t\t\t rx_bi->page,\n-\t\t\t\t\t rx_bi->page_offset,\n-\t\t\t\t\t rx_packet_len);\n-\n-\t\t\tskb->len += rx_packet_len;\n-\t\t\tskb->data_len += rx_packet_len;\n-\t\t\tskb->truesize += rx_packet_len;\n-\n-\t\t\tif ((page_count(rx_bi->page) == 1) &&\n-\t\t\t (page_to_nid(rx_bi->page) == current_node))\n-\t\t\t\tget_page(rx_bi->page);\n-\t\t\telse\n+\t\t\tskb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,\n+\t\t\t\t\trx_bi->page,\n+\t\t\t\t\trx_bi->page_offset + copysize,\n+\t\t\t\t\trx_packet_len, I40E_RXBUFFER_2048);\n+\n+\t\t\tget_page(rx_bi->page);\n+\t\t\t/* switch to the other half-page here; the allocation\n+\t\t\t * code programs the right addr into HW. If we haven't\n+\t\t\t * used this half-page, the address won't be changed,\n+\t\t\t * and HW can just use it next time through.\n+\t\t\t */\n+\t\t\trx_bi->page_offset ^= PAGE_SIZE / 2;\n+\t\t\t/* If the page count is more than 2, then both halves\n+\t\t\t * of the page are used and we need to free it. Do it\n+\t\t\t * here instead of in the alloc code. Otherwise one\n+\t\t\t * of the half-pages might be released between now and\n+\t\t\t * then, and we wouldn't know which one to use.\n+\t\t\t */\n+\t\t\tif (page_count(rx_bi->page) > 2) {\n+\t\t\t\tdma_unmap_page(rx_ring->dev,\n+\t\t\t\t\t rx_bi->page_dma,\n+\t\t\t\t\t PAGE_SIZE,\n+\t\t\t\t\t DMA_FROM_DEVICE);\n+\t\t\t\t__free_page(rx_bi->page);\n \t\t\t\trx_bi->page = NULL;\n+\t\t\t\trx_bi->page_dma = 0;\n+\t\t\t\trx_ring->rx_stats.realloc_count++;\n+\t\t\t}\n \n-\t\t\tdma_unmap_page(rx_ring->dev,\n-\t\t\t\t rx_bi->page_dma,\n-\t\t\t\t PAGE_SIZE / 2,\n-\t\t\t\t DMA_FROM_DEVICE);\n-\t\t\trx_bi->page_dma = 0;\n \t\t}\n \t\tI40E_RX_INCREMENT(rx_ring, i);\n \ndiff --git a/drivers/net/ethernet/intel/i40evf/i40e_txrx.h b/drivers/net/ethernet/intel/i40evf/i40e_txrx.h\nindex 1438017..595cd8b 100644\n--- a/drivers/net/ethernet/intel/i40evf/i40e_txrx.h\n+++ b/drivers/net/ethernet/intel/i40evf/i40e_txrx.h\n@@ -208,6 +208,8 @@ struct i40e_rx_queue_stats {\n \tu64 non_eop_descs;\n \tu64 alloc_page_failed;\n \tu64 alloc_buff_failed;\n+\tu64 page_reuse_count;\n+\tu64 realloc_count;\n };\n \n enum i40e_ring_state_t {\n", "prefixes": [ "v2", "next", "S27", "12/15" ] }