Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/1122916/?format=api
{ "id": 1122916, "url": "http://patchwork.ozlabs.org/api/patches/1122916/?format=api", "web_url": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/20190626080711.634-2-anthony.l.nguyen@intel.com/", "project": { "id": 46, "url": "http://patchwork.ozlabs.org/api/projects/46/?format=api", "name": "Intel Wired Ethernet development", "link_name": "intel-wired-lan", "list_id": "intel-wired-lan.osuosl.org", "list_email": "intel-wired-lan@osuosl.org", "web_url": "", "scm_url": "", "webscm_url": "", "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20190626080711.634-2-anthony.l.nguyen@intel.com>", "list_archive_url": null, "date": "2019-06-26T08:07:11", "name": "[v2,2/2] ice: Add support for AF_XDP", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": false, "hash": "e8824e141263127341536df1d7a5eb798ffcd7dd", "submitter": { "id": 68875, "url": "http://patchwork.ozlabs.org/api/people/68875/?format=api", "name": "Tony Nguyen", "email": "anthony.l.nguyen@intel.com" }, "delegate": { "id": 68, "url": "http://patchwork.ozlabs.org/api/users/68/?format=api", "username": "jtkirshe", "first_name": "Jeff", "last_name": "Kirsher", "email": "jeffrey.t.kirsher@intel.com" }, "mbox": "http://patchwork.ozlabs.org/project/intel-wired-lan/patch/20190626080711.634-2-anthony.l.nguyen@intel.com/mbox/", "series": [ { "id": 116291, "url": "http://patchwork.ozlabs.org/api/series/116291/?format=api", "web_url": "http://patchwork.ozlabs.org/project/intel-wired-lan/list/?series=116291", "date": "2019-06-26T08:07:11", "name": "[v2,1/2] ice: Add support for XDP", "version": 2, "mbox": "http://patchwork.ozlabs.org/series/116291/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/patches/1122916/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/1122916/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<intel-wired-lan-bounces@osuosl.org>", "X-Original-To": [ "incoming@patchwork.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Delivered-To": [ "patchwork-incoming@bilbo.ozlabs.org", "intel-wired-lan@lists.osuosl.org" ], "Authentication-Results": [ "ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=osuosl.org\n\t(client-ip=140.211.166.133; helo=hemlock.osuosl.org;\n\tenvelope-from=intel-wired-lan-bounces@osuosl.org;\n\treceiver=<UNKNOWN>)", "ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=intel.com" ], "Received": [ "from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 45YpXT2B1Mz9s3C\n\tfor <incoming@patchwork.ozlabs.org>;\n\tThu, 27 Jun 2019 02:34:48 +1000 (AEST)", "from localhost (localhost [127.0.0.1])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id 6B37887D96;\n\tWed, 26 Jun 2019 16:34:46 +0000 (UTC)", "from hemlock.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id jULVSDX5NlWF; Wed, 26 Jun 2019 16:34:43 +0000 (UTC)", "from ash.osuosl.org (ash.osuosl.org [140.211.166.34])\n\tby hemlock.osuosl.org (Postfix) with ESMTP id A732987D6E;\n\tWed, 26 Jun 2019 16:34:43 +0000 (UTC)", "from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136])\n\tby ash.osuosl.org (Postfix) with ESMTP id CCDF51BF3B8\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tWed, 26 Jun 2019 16:34:40 +0000 (UTC)", "from localhost (localhost [127.0.0.1])\n\tby silver.osuosl.org (Postfix) with ESMTP id BF8B621574\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tWed, 26 Jun 2019 16:34:40 +0000 (UTC)", "from silver.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id O1jKCfh-kIcP for <intel-wired-lan@lists.osuosl.org>;\n\tWed, 26 Jun 2019 16:34:36 +0000 (UTC)", "from mga06.intel.com (mga06.intel.com [134.134.136.31])\n\tby silver.osuosl.org (Postfix) with ESMTPS id 48D182156B\n\tfor <intel-wired-lan@lists.osuosl.org>;\n\tWed, 26 Jun 2019 16:34:36 +0000 (UTC)", "from orsmga004.jf.intel.com ([10.7.209.38])\n\tby orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t26 Jun 2019 09:34:35 -0700", "from unknown (HELO localhost.jf.intel.com) ([10.166.244.174])\n\tby orsmga004.jf.intel.com with ESMTP; 26 Jun 2019 09:34:35 -0700" ], "X-Virus-Scanned": [ "amavisd-new at osuosl.org", "amavisd-new at osuosl.org" ], "X-Greylist": "domain auto-whitelisted by SQLgrey-1.7.6", "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.63,420,1557212400\"; d=\"scan'208\";a=\"313480905\"", "From": "Tony Nguyen <anthony.l.nguyen@intel.com>", "To": "intel-wired-lan@lists.osuosl.org", "Date": "Wed, 26 Jun 2019 01:07:11 -0700", "Message-Id": "<20190626080711.634-2-anthony.l.nguyen@intel.com>", "X-Mailer": "git-send-email 2.20.1", "In-Reply-To": "<20190626080711.634-1-anthony.l.nguyen@intel.com>", "References": "<20190626080711.634-1-anthony.l.nguyen@intel.com>", "MIME-Version": "1.0", "Subject": "[Intel-wired-lan] [PATCH v2 2/2] ice: Add support for AF_XDP", "X-BeenThere": "intel-wired-lan@osuosl.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "Intel Wired Ethernet Linux Kernel Driver Development\n\t<intel-wired-lan.osuosl.org>", "List-Unsubscribe": "<https://lists.osuosl.org/mailman/options/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@osuosl.org?subject=unsubscribe>", "List-Archive": "<http://lists.osuosl.org/pipermail/intel-wired-lan/>", "List-Post": "<mailto:intel-wired-lan@osuosl.org>", "List-Help": "<mailto:intel-wired-lan-request@osuosl.org?subject=help>", "List-Subscribe": "<https://lists.osuosl.org/mailman/listinfo/intel-wired-lan>, \n\t<mailto:intel-wired-lan-request@osuosl.org?subject=subscribe>", "Cc": "Krzysztof Kazimierczak <krzysztof.kazimierczak@intel.com>", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "7bit", "Errors-To": "intel-wired-lan-bounces@osuosl.org", "Sender": "\"Intel-wired-lan\" <intel-wired-lan-bounces@osuosl.org>" }, "content": "From: Krzysztof Kazimierczak <krzysztof.kazimierczak@intel.com>\n\nAdd zero copy AF_XDP support. This patch adds zero copy support for\nTx and Rx; code for zero copy is added to ice_xsk.h and ice_xsk.c.\n\nFor Tx, implement ndo_xsk_async_xmit. As with other drivers, reuse\nexisting XDP Tx queues for this task, since XDP_REDIRECT guarantees\nmutual exclusion between different NAPI contexts based on CPU ID. In\nturn, a netdev can XDP_REDIRECT to another netdev with a different\nNAPI context, since the operation is bound to a specific core and each\ncore has its own hardware ring.\n\nFor Rx, allocate frames as MEM_TYPE_ZERO_COPY on queues that AF_XDP is\nenabled.\n\nSigned-off-by: Krzysztof Kazimierczak <krzysztof.kazimierczak@intel.com>\nSigned-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>\n---\nv2: Fix build warning for 32 bit systems\n Remove unneeded casts to u8*\n\n drivers/net/ethernet/intel/ice/Makefile | 1 +\n drivers/net/ethernet/intel/ice/ice.h | 26 +\n drivers/net/ethernet/intel/ice/ice_ethtool.c | 7 +\n drivers/net/ethernet/intel/ice/ice_lib.c | 65 +-\n drivers/net/ethernet/intel/ice/ice_main.c | 13 +\n drivers/net/ethernet/intel/ice/ice_txrx.c | 84 +-\n drivers/net/ethernet/intel/ice/ice_txrx.h | 35 +-\n drivers/net/ethernet/intel/ice/ice_xsk.c | 941 +++++++++++++++++++\n drivers/net/ethernet/intel/ice/ice_xsk.h | 81 ++\n 9 files changed, 1223 insertions(+), 30 deletions(-)\n create mode 100644 drivers/net/ethernet/intel/ice/ice_xsk.c\n create mode 100644 drivers/net/ethernet/intel/ice/ice_xsk.h", "diff": "diff --git a/drivers/net/ethernet/intel/ice/Makefile b/drivers/net/ethernet/intel/ice/Makefile\nindex 2d140ba83781..54bad11e4736 100644\n--- a/drivers/net/ethernet/intel/ice/Makefile\n+++ b/drivers/net/ethernet/intel/ice/Makefile\n@@ -18,3 +18,4 @@ ice-y := ice_main.o\t\\\n \t ice_ethtool.o\n ice-$(CONFIG_PCI_IOV) += ice_virtchnl_pf.o ice_sriov.o\n ice-$(CONFIG_DCB) += ice_dcb.o ice_dcb_lib.o\n+ice-$(CONFIG_XDP_SOCKETS) += ice_xsk.o\ndiff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h\nindex 53adb93c6b61..3639cf8c75ac 100644\n--- a/drivers/net/ethernet/intel/ice/ice.h\n+++ b/drivers/net/ethernet/intel/ice/ice.h\n@@ -34,6 +34,8 @@\n #include <linux/bpf.h>\n #include <linux/avf/virtchnl.h>\n #include <net/ipv6.h>\n+#include <net/xdp_sock.h>\n+#include \"ice_xsk.h\"\n #include \"ice_devids.h\"\n #include \"ice_type.h\"\n #include \"ice_txrx.h\"\n@@ -308,6 +310,9 @@ struct ice_vsi {\n \tstruct ice_ring **xdp_rings;\t /* XDP ring array */\n \tu16 num_xdp_txq;\t\t /* Used XDP queues */\n \tu8 xdp_mapping_mode;\t\t /* ICE_MAP_MODE_[CONTIG|SCATTER] */\n+\tstruct xdp_umem **xsk_umems;\n+\tu16 num_xsk_umems_used;\n+\tu16 num_xsk_umems;\n } ____cacheline_internodealigned_in_smp;\n \n /* struct that defines an interrupt vector */\n@@ -449,6 +454,27 @@ static inline void ice_set_ring_xdp(struct ice_ring *ring)\n \tring->tx_buf[0].tx_flags |= ICE_TX_FLAGS_RING_XDP;\n }\n \n+/**\n+ * ice_xsk_umem - get XDP UMEM bound to a ring\n+ * @ring - ring to use\n+ *\n+ * Returns a pointer to xdp_umem structure if there is an UMEM present,\n+ * NULL otherwise.\n+ */\n+static inline struct xdp_umem *ice_xsk_umem(struct ice_ring *ring)\n+{\n+\tstruct xdp_umem **umems = ring->vsi->xsk_umems;\n+\tint qid = ring->q_index;\n+\n+\tif (ice_ring_is_xdp(ring))\n+\t\tqid -= ring->vsi->num_xdp_txq;\n+\n+\tif (!umems || !umems[qid] || !ice_is_xdp_ena_vsi(ring->vsi))\n+\t\treturn NULL;\n+\n+\treturn umems[qid];\n+}\n+\n /**\n * ice_find_vsi_by_type - Find and return VSI of a given type\n * @pf: PF to search for VSI\ndiff --git a/drivers/net/ethernet/intel/ice/ice_ethtool.c b/drivers/net/ethernet/intel/ice/ice_ethtool.c\nindex 2d9c184a2333..5c9f7a8bfd16 100644\n--- a/drivers/net/ethernet/intel/ice/ice_ethtool.c\n+++ b/drivers/net/ethernet/intel/ice/ice_ethtool.c\n@@ -2593,6 +2593,13 @@ ice_set_ringparam(struct net_device *netdev, struct ethtool_ringparam *ring)\n \t\treturn 0;\n \t}\n \n+\t/* If there is a AF_XDP UMEM attached to any of Rx rings,\n+\t * disallow changing the number of descriptors -- regardless\n+\t * if the netdev is running or not.\n+\t */\n+\tif (ice_xsk_any_rx_ring_ena(vsi))\n+\t\treturn -EBUSY;\n+\n \twhile (test_and_set_bit(__ICE_CFG_BUSY, pf->state)) {\n \t\ttimeout--;\n \t\tif (!timeout)\ndiff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c\nindex 09c6b9921ccd..cb998b97cde6 100644\n--- a/drivers/net/ethernet/intel/ice/ice_lib.c\n+++ b/drivers/net/ethernet/intel/ice/ice_lib.c\n@@ -13,14 +13,17 @@\n */\n static int ice_setup_rx_ctx(struct ice_ring *ring)\n {\n+\tint chain_len = ICE_MAX_CHAINED_RX_BUFS;\n \tstruct ice_vsi *vsi = ring->vsi;\n-\tstruct ice_hw *hw = &vsi->back->hw;\n \tu32 rxdid = ICE_RXDID_FLEX_NIC;\n \tstruct ice_rlan_ctx rlan_ctx;\n+\tstruct ice_hw *hw;\n \tu32 regval;\n \tu16 pf_q;\n \tint err;\n \n+\thw = &vsi->back->hw;\n+\n \t/* what is Rx queue number in global space of 2K Rx queues */\n \tpf_q = vsi->rxq_map[ring->q_index];\n \n@@ -34,10 +37,38 @@ static int ice_setup_rx_ctx(struct ice_ring *ring)\n \t\t\txdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,\n \t\t\t\t\t ring->q_index);\n \n-\t\terr = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,\n-\t\t\t\t\t\t MEM_TYPE_PAGE_SHARED, NULL);\n-\t\tif (err)\n-\t\t\treturn err;\n+\t\tring->xsk_umem = ice_xsk_umem(ring);\n+\t\tif (ring->xsk_umem) {\n+\t\t\txdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);\n+\n+\t\t\tring->rx_buf_len = ring->xsk_umem->chunk_size_nohr -\n+\t\t\t\t\t XDP_PACKET_HEADROOM;\n+\t\t\t/* For AF_XDP ZC, we disallow packets to span on\n+\t\t\t * multiple buffers, thus letting us skip that\n+\t\t\t * handling in the fast-path.\n+\t\t\t */\n+\t\t\tchain_len = 1;\n+\t\t\tring->zca.free = ice_zca_free;\n+\t\t\terr = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,\n+\t\t\t\t\t\t\t MEM_TYPE_ZERO_COPY,\n+\t\t\t\t\t\t\t &ring->zca);\n+\t\t\tif (err)\n+\t\t\t\treturn err;\n+\n+\t\t\tdev_info(&vsi->back->pdev->dev, \"Registered XDP mem model MEM_TYPE_ZERO_COPY on Rx ring %d\\n\",\n+\t\t\t\t ring->q_index);\n+\t\t} else {\n+\t\t\tif (!xdp_rxq_info_is_reg(&ring->xdp_rxq))\n+\t\t\t\txdp_rxq_info_reg(&ring->xdp_rxq,\n+\t\t\t\t\t\t ring->netdev,\n+\t\t\t\t\t\t ring->q_index);\n+\n+\t\t\terr = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,\n+\t\t\t\t\t\t\t MEM_TYPE_PAGE_SHARED,\n+\t\t\t\t\t\t\t NULL);\n+\t\t\tif (err)\n+\t\t\t\treturn err;\n+\t\t}\n \t}\n \t/* Receive Queue Base Address.\n \t * Indicates the starting address of the descriptor queue defined in\n@@ -77,7 +108,7 @@ static int ice_setup_rx_ctx(struct ice_ring *ring)\n \t * than 5 x DBUF\n \t */\n \trlan_ctx.rxmax = min_t(u16, vsi->max_frame,\n-\t\t\t ICE_MAX_CHAINED_RX_BUFS * ring->rx_buf_len);\n+\t\t\t chain_len * ring->rx_buf_len);\n \n \t/* Rx queue threshold in units of 64 */\n \trlan_ctx.lrxqthresh = 1;\n@@ -115,7 +146,15 @@ static int ice_setup_rx_ctx(struct ice_ring *ring)\n \t/* init queue specific tail register */\n \tring->tail = hw->hw_addr + QRX_TAIL(pf_q);\n \twritel(0, ring->tail);\n-\tice_alloc_rx_bufs(ring, ICE_DESC_UNUSED(ring));\n+\n+\terr = ring->xsk_umem ?\n+\t ice_alloc_rx_bufs_slow_zc(ring, ICE_DESC_UNUSED(ring)) :\n+\t ice_alloc_rx_bufs(ring, ICE_DESC_UNUSED(ring));\n+\tif (err)\n+\t\tdev_info(&vsi->back->pdev->dev,\n+\t\t\t \"Failed allocate some buffers on %sRx ring %d (pf_q %d)\\n\",\n+\t\t\t ring->xsk_umem ? \"UMEM enabled \" : \"\",\n+\t\t\t ring->q_index, pf_q);\n \n \treturn 0;\n }\n@@ -1770,7 +1809,17 @@ int ice_vsi_cfg_lan_txqs(struct ice_vsi *vsi)\n */\n int ice_vsi_cfg_xdp_txqs(struct ice_vsi *vsi)\n {\n-\treturn ice_vsi_cfg_txqs(vsi, vsi->xdp_rings, vsi->num_xdp_txq);\n+\tint ret;\n+\tint i;\n+\n+\tret = ice_vsi_cfg_txqs(vsi, vsi->xdp_rings, vsi->num_xdp_txq);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tfor (i = 0; i < vsi->num_xdp_txq; i++)\n+\t\tvsi->xdp_rings[i]->xsk_umem = ice_xsk_umem(vsi->xdp_rings[i]);\n+\n+\treturn ret;\n }\n \n /**\ndiff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c\nindex 5d14627a6ab6..5f45548f2940 100644\n--- a/drivers/net/ethernet/intel/ice/ice_main.c\n+++ b/drivers/net/ethernet/intel/ice/ice_main.c\n@@ -1520,6 +1520,7 @@ static int ice_xdp_alloc_setup_rings(struct ice_vsi *vsi)\n \t\tif (ice_setup_tx_ring(xdp_ring))\n \t\t\tgoto free_xdp_rings;\n \t\tice_set_ring_xdp(xdp_ring);\n+\t\txdp_ring->xsk_umem = ice_xsk_umem(xdp_ring);\n \t}\n \n \treturn 0;\n@@ -1755,6 +1756,14 @@ ice_xdp_setup_prog(struct ice_vsi *vsi, struct bpf_prog *prog,\n \tif (if_running)\n \t\tret = ice_up(vsi);\n \n+\tif (!ret && prog && vsi->xsk_umems)\n+\t\tice_for_each_rxq(vsi, i) {\n+\t\t\tstruct ice_ring *rx_ring = vsi->rx_rings[i];\n+\n+\t\t\tif (rx_ring->xsk_umem)\n+\t\t\t\tnapi_schedule(&rx_ring->q_vector->napi);\n+\t\t}\n+\n skip_setting_prog:\n \treturn ret;\n }\n@@ -1781,6 +1790,9 @@ static int ice_xdp(struct net_device *dev, struct netdev_bpf *xdp)\n \tcase XDP_QUERY_PROG:\n \t\txdp->prog_id = vsi->xdp_prog ? vsi->xdp_prog->aux->id : 0;\n \t\treturn 0;\n+\tcase XDP_SETUP_XSK_UMEM:\n+\t\treturn ice_xsk_umem_setup(vsi, xdp->xsk.umem,\n+\t\t\t\t\t xdp->xsk.queue_id);\n \tdefault:\n \t\tNL_SET_ERR_MSG_MOD(xdp->extack, \"Unknown XDP command\");\n \t\treturn -EINVAL;\n@@ -4731,4 +4743,5 @@ static const struct net_device_ops ice_netdev_ops = {\n \t.ndo_tx_timeout = ice_tx_timeout,\n \t.ndo_bpf = ice_xdp,\n \t.ndo_xdp_xmit = ice_xdp_xmit,\n+\t.ndo_xsk_async_xmit = ice_xsk_async_xmit,\n };\ndiff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c\nindex 0ed35cac8d60..d76237d82009 100644\n--- a/drivers/net/ethernet/intel/ice/ice_txrx.c\n+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c\n@@ -9,11 +9,15 @@\n #include <net/xdp.h>\n #include \"ice.h\"\n #include \"ice_dcb_lib.h\"\n+#include \"ice_xsk.h\"\n \n #define ICE_RX_HDR_SIZE\t\t256\n \n /* helper function for building cmd/type/offset */\n-static __le64\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+__le64\n ice_build_ctob(u64 td_cmd, u64 td_offset, unsigned int size, u64 td_tag)\n {\n \treturn cpu_to_le64(ICE_TX_DESC_DTYPE_DATA |\n@@ -67,6 +71,11 @@ void ice_clean_tx_ring(struct ice_ring *tx_ring)\n {\n \tu16 i;\n \n+\tif (ice_ring_is_xdp(tx_ring) && tx_ring->xsk_umem) {\n+\t\tice_xsk_clean_xdp_ring(tx_ring);\n+\t\tgoto tx_skip_free;\n+\t}\n+\n \t/* ring already cleared, nothing to do */\n \tif (!tx_ring->tx_buf)\n \t\treturn;\n@@ -75,6 +84,7 @@ void ice_clean_tx_ring(struct ice_ring *tx_ring)\n \tfor (i = 0; i < tx_ring->count; i++)\n \t\tice_unmap_and_free_tx_buf(tx_ring, &tx_ring->tx_buf[i]);\n \n+tx_skip_free:\n \tmemset(tx_ring->tx_buf, 0, sizeof(*tx_ring->tx_buf) * tx_ring->count);\n \n \t/* Zero out the descriptor ring */\n@@ -294,6 +304,11 @@ void ice_clean_rx_ring(struct ice_ring *rx_ring)\n \tif (!rx_ring->rx_buf)\n \t\treturn;\n \n+\tif (rx_ring->xsk_umem) {\n+\t\tice_xsk_clean_rx_ring(rx_ring);\n+\t\tgoto rx_skip_free;\n+\t}\n+\n \t/* Free all the Rx ring sk_buffs */\n \tfor (i = 0; i < rx_ring->count; i++) {\n \t\tstruct ice_rx_buf *rx_buf = &rx_ring->rx_buf[i];\n@@ -321,6 +336,7 @@ void ice_clean_rx_ring(struct ice_ring *rx_ring)\n \t\trx_buf->page_offset = 0;\n \t}\n \n+rx_skip_free:\n \tmemset(rx_ring->rx_buf, 0, sizeof(*rx_ring->rx_buf) * rx_ring->count);\n \n \t/* Zero out the descriptor ring */\n@@ -407,7 +423,10 @@ int ice_setup_rx_ring(struct ice_ring *rx_ring)\n * @rx_ring: ring to bump\n * @val: new head index\n */\n-static void ice_release_rx_desc(struct ice_ring *rx_ring, u32 val)\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+void ice_release_rx_desc(struct ice_ring *rx_ring, u32 val)\n {\n \trx_ring->next_to_use = val;\n \n@@ -440,7 +459,10 @@ static unsigned int ice_rx_offset(struct ice_ring *rx_ring)\n *\n * This function updates the XDP Tx ring tail register.\n */\n-static void ice_xdp_ring_update_tail(struct ice_ring *xdp_ring)\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+void ice_xdp_ring_update_tail(struct ice_ring *xdp_ring)\n {\n \t/* Force memory writes to complete before letting h/w\n \t * know there are new descriptors to fetch.\n@@ -455,8 +477,10 @@ static void ice_xdp_ring_update_tail(struct ice_ring *xdp_ring)\n * @size: packet data size\n * @xdp_ring: XDP ring for transmission\n */\n-static int\n-ice_xmit_xdp_ring(void *data, u16 size, struct ice_ring *xdp_ring)\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+int ice_xmit_xdp_ring(void *data, u16 size, struct ice_ring *xdp_ring)\n {\n \tu16 i = xdp_ring->next_to_use;\n \tstruct ice_tx_desc *tx_desc;\n@@ -603,8 +627,10 @@ ice_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,\n * should be called when a batch of packets has been processed in the\n * napi loop.\n */\n-static void\n-ice_finalize_xdp_rx(struct ice_ring *rx_ring, unsigned int xdp_res)\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+void ice_finalize_xdp_rx(struct ice_ring *rx_ring, unsigned int xdp_res)\n {\n \tif (xdp_res & ICE_XDP_REDIR)\n \t\txdp_do_flush_map();\n@@ -988,7 +1014,7 @@ static void ice_put_rx_buf(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf)\n }\n \n /**\n- * ice_cleanup_headers - Correct empty headers\n+ * ice_cleanup_hdrs - Correct empty headers\n * @skb: pointer to current skb being fixed\n *\n * Also address the case where we are pulling data in on pages only\n@@ -999,7 +1025,10 @@ static void ice_put_rx_buf(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf)\n *\n * Returns true if an error was encountered and skb was freed.\n */\n-static bool ice_cleanup_headers(struct sk_buff *skb)\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+bool ice_cleanup_hdrs(struct sk_buff *skb)\n {\n \t/* if eth_skb_pad returns an error the skb was freed */\n \tif (eth_skb_pad(skb))\n@@ -1018,11 +1047,13 @@ static bool ice_cleanup_headers(struct sk_buff *skb)\n * The status_error_len doesn't need to be shifted because it begins\n * at offset zero.\n */\n-static bool\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+bool\n ice_test_staterr(union ice_32b_rx_flex_desc *rx_desc, const u16 stat_err_bits)\n {\n-\treturn !!(rx_desc->wb.status_error0 &\n-\t\t cpu_to_le16(stat_err_bits));\n+\treturn !!(rx_desc->wb.status_error0 & cpu_to_le16(stat_err_bits));\n }\n \n /**\n@@ -1167,7 +1198,10 @@ ice_rx_csum(struct ice_vsi *vsi, struct sk_buff *skb,\n * order to populate the hash, checksum, VLAN, protocol, and\n * other fields within the skb.\n */\n-static void\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+void\n ice_process_skb_fields(struct ice_ring *rx_ring,\n \t\t union ice_32b_rx_flex_desc *rx_desc,\n \t\t struct sk_buff *skb, u8 ptype)\n@@ -1189,7 +1223,10 @@ ice_process_skb_fields(struct ice_ring *rx_ring,\n * This function sends the completed packet (via. skb) up the stack using\n * gro receive functions (with/without VLAN tag)\n */\n-static void\n+#ifndef CONFIG_XDP_SOCKETS\n+static\n+#endif\n+void\n ice_receive_skb(struct ice_ring *rx_ring, struct sk_buff *skb, u16 vlan_tag)\n {\n \tif ((rx_ring->netdev->features & NETIF_F_HW_VLAN_CTAG_RX) &&\n@@ -1330,7 +1367,7 @@ static int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)\n \t\t/* correct empty headers and pad skb if needed (to make valid\n \t\t * ethernet frame\n \t\t */\n-\t\tif (ice_cleanup_headers(skb)) {\n+\t\tif (ice_cleanup_hdrs(skb)) {\n \t\t\tskb = NULL;\n \t\t\tcontinue;\n \t\t}\n@@ -1683,9 +1720,14 @@ int ice_napi_poll(struct napi_struct *napi, int budget)\n \t/* Since the actual Tx work is minimal, we can give the Tx a larger\n \t * budget and be more aggressive about cleaning up the Tx descriptors.\n \t */\n-\tice_for_each_ring(ring, q_vector->tx)\n-\t\tif (!ice_clean_tx_irq(vsi, ring, budget))\n+\tice_for_each_ring(ring, q_vector->tx) {\n+\t\tbool wd = ring->xsk_umem ?\n+\t\t\t ice_clean_tx_irq_zc(vsi, ring) :\n+\t\t\t ice_clean_tx_irq(vsi, ring, budget);\n+\n+\t\tif (!wd)\n \t\t\tclean_complete = false;\n+\t}\n \n \t/* Handle case where we are called by netpoll with a budget of 0 */\n \tif (budget <= 0)\n@@ -1700,7 +1742,13 @@ int ice_napi_poll(struct napi_struct *napi, int budget)\n \tice_for_each_ring(ring, q_vector->rx) {\n \t\tint cleaned;\n \n-\t\tcleaned = ice_clean_rx_irq(ring, budget_per_ring);\n+\t\t/* A dedicated path for zero-copy allows making a single\n+\t\t * comparison in the irq context instead of many inside the\n+\t\t * ice_clean_rx_irq function and makes the codebase cleaner.\n+\t\t */\n+\t\tcleaned = ring->xsk_umem ?\n+\t\t\t ice_clean_rx_irq_zc(ring, budget_per_ring) :\n+\t\t\t ice_clean_rx_irq(ring, budget_per_ring);\n \t\twork_done += cleaned;\n \t\t/* if we clean as many as budgeted, we must not be done */\n \t\tif (cleaned >= budget_per_ring)\ndiff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h\nindex 355834b4abda..8f857e423739 100644\n--- a/drivers/net/ethernet/intel/ice/ice_txrx.h\n+++ b/drivers/net/ethernet/intel/ice/ice_txrx.h\n@@ -92,9 +92,17 @@ struct ice_tx_offload_params {\n struct ice_rx_buf {\n \tstruct sk_buff *skb;\n \tdma_addr_t dma;\n-\tstruct page *page;\n-\tunsigned int page_offset;\n-\tu16 pagecnt_bias;\n+\tunion {\n+\t\tstruct {\n+\t\t\tstruct page *page;\n+\t\t\tunsigned int page_offset;\n+\t\t\tu16 pagecnt_bias;\n+\t\t};\n+\t\tstruct {\n+\t\t\tvoid *addr;\n+\t\t\tu64 handle;\n+\t\t};\n+\t};\n };\n \n struct ice_q_stats {\n@@ -202,6 +210,8 @@ struct ice_ring {\n \n \tstruct rcu_head rcu;\t\t/* to avoid race on free */\n \tstruct bpf_prog *xdp_prog;\n+\tstruct xdp_umem *xsk_umem;\n+\tstruct zero_copy_allocator zca;\n \t/* CL3 - 3rd cacheline starts here */\n \tstruct xdp_rxq_info xdp_rxq;\n \t/* CLX - the below items are only accessed infrequently and should be\n@@ -244,6 +254,8 @@ struct ice_ring_container {\n #define ice_for_each_ring(pos, head) \\\n \tfor (pos = (head).ring; pos; pos = pos->next)\n \n+union ice_32b_rx_flex_desc;\n+\n bool ice_alloc_rx_bufs(struct ice_ring *rxr, u16 cleaned_count);\n netdev_tx_t ice_start_xmit(struct sk_buff *skb, struct net_device *netdev);\n void ice_clean_tx_ring(struct ice_ring *tx_ring);\n@@ -253,5 +265,20 @@ int ice_setup_rx_ring(struct ice_ring *rx_ring);\n void ice_free_tx_ring(struct ice_ring *tx_ring);\n void ice_free_rx_ring(struct ice_ring *rx_ring);\n int ice_napi_poll(struct napi_struct *napi, int budget);\n-\n+#ifdef CONFIG_XDP_SOCKETS\n+void ice_finalize_xdp_rx(struct ice_ring *rx_ring, unsigned int xdp_res);\n+void ice_release_rx_desc(struct ice_ring *rx_ring, u32 val);\n+bool\n+ice_test_staterr(union ice_32b_rx_flex_desc *rx_desc, const u16 stat_err_bits);\n+void\n+ice_process_skb_fields(struct ice_ring *rx_ring,\n+\t\t union ice_32b_rx_flex_desc *rx_desc,\n+\t\t struct sk_buff *skb, u8 ptype);\n+void\n+ice_receive_skb(struct ice_ring *rx_ring, struct sk_buff *skb, u16 vlan_tag);\n+bool ice_cleanup_hdrs(struct sk_buff *skb);\n+void ice_xdp_ring_update_tail(struct ice_ring *xdp_ring);\n+__le64 ice_build_ctob(u64 td_cmd, u64 td_offset, unsigned int size, u64 td_tag);\n+int ice_xmit_xdp_ring(void *data, u16 size, struct ice_ring *xdp_ring);\n+#endif /* CONFIG_XDP_SOCKETS */\n #endif /* _ICE_TXRX_H_ */\ndiff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c\nnew file mode 100644\nindex 000000000000..3e963311eac4\n--- /dev/null\n+++ b/drivers/net/ethernet/intel/ice/ice_xsk.c\n@@ -0,0 +1,941 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2019, Intel Corporation. */\n+\n+#include \"ice.h\"\n+#include \"ice_type.h\"\n+#include \"ice_xsk.h\"\n+#include \"ice_txrx.h\"\n+#include \"ice_lib.h\"\n+#include <linux/bpf_trace.h>\n+#include <net/xdp_sock.h>\n+#include <net/xdp.h>\n+\n+/**\n+ * ice_xsk_alloc_umems - allocate a UMEM region for an XDP socket\n+ * @vsi: VSI to allocate the UMEM on\n+ *\n+ * Returns 0 on success, negative on error\n+ */\n+static int ice_xsk_alloc_umems(struct ice_vsi *vsi)\n+{\n+\tif (vsi->xsk_umems)\n+\t\treturn 0;\n+\n+\tvsi->xsk_umems = kcalloc(vsi->num_xsk_umems, sizeof(*vsi->xsk_umems),\n+\t\t\t\t GFP_KERNEL);\n+\n+\tif (!vsi->xsk_umems) {\n+\t\tvsi->num_xsk_umems = 0;\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ice_xsk_add_umem - add a UMEM region for XDP sockets\n+ * @vsi: VSI to which the UMEM will be added\n+ * @umem: pointer to a requested UMEM region\n+ * @qid: queue ID\n+ *\n+ * Returns 0 on success, negative on error\n+ */\n+static int ice_xsk_add_umem(struct ice_vsi *vsi, struct xdp_umem *umem, u16 qid)\n+{\n+\tint err;\n+\n+\terr = ice_xsk_alloc_umems(vsi);\n+\tif (err)\n+\t\treturn err;\n+\n+\tvsi->xsk_umems[qid] = umem;\n+\tvsi->num_xsk_umems_used++;\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ice_xsk_remove_umem - Remove an UMEM for a certain ring/qid\n+ * @vsi: VSI from which the VSI will be removed\n+ * @qid: Ring/qid associated with the UMEM\n+ */\n+static void ice_xsk_remove_umem(struct ice_vsi *vsi, u16 qid)\n+{\n+\tvsi->xsk_umems[qid] = NULL;\n+\tvsi->num_xsk_umems_used--;\n+\n+\tif (vsi->num_xsk_umems_used == 0) {\n+\t\tkfree(vsi->xsk_umems);\n+\t\tvsi->xsk_umems = NULL;\n+\t\tvsi->num_xsk_umems = 0;\n+\t}\n+}\n+\n+/**\n+ * ice_xsk_umem_dma_map - DMA map UMEM region for XDP sockets\n+ * @vsi: VSI to map the UMEM region\n+ * @umem: UMEM to map\n+ *\n+ * Returns 0 on success, negative on error\n+ */\n+static int ice_xsk_umem_dma_map(struct ice_vsi *vsi, struct xdp_umem *umem)\n+{\n+\tstruct ice_pf *pf = vsi->back;\n+\tstruct device *dev;\n+\tunsigned int i;\n+\n+\tdev = &pf->pdev->dev;\n+\tfor (i = 0; i < umem->npgs; i++) {\n+\t\tdma_addr_t dma = dma_map_page_attrs(dev, umem->pgs[i], 0,\n+\t\t\t\t\t\t PAGE_SIZE,\n+\t\t\t\t\t\t DMA_BIDIRECTIONAL,\n+\t\t\t\t\t\t ICE_RX_DMA_ATTR);\n+\t\tif (dma_mapping_error(dev, dma)) {\n+\t\t\tdev_dbg(dev,\n+\t\t\t\t\"XSK UMEM DMA mapping error on page num %d\", i);\n+\t\t\tgoto out_unmap;\n+\t\t}\n+\n+\t\tumem->pages[i].dma = dma;\n+\t}\n+\n+\treturn 0;\n+\n+out_unmap:\n+\tfor (; i > 0; i--) {\n+\t\tdma_unmap_page_attrs(dev, umem->pages[i].dma, PAGE_SIZE,\n+\t\t\t\t DMA_BIDIRECTIONAL, ICE_RX_DMA_ATTR);\n+\t\tumem->pages[i].dma = 0;\n+\t}\n+\n+\treturn -EFAULT;\n+}\n+\n+/**\n+ * ice_xsk_umem_dma_unmap - DMA unmap UMEM region for XDP sockets\n+ * @vsi: VSI from which the UMEM will be unmapped\n+ * @umem: UMEM to unmap\n+ */\n+static void ice_xsk_umem_dma_unmap(struct ice_vsi *vsi, struct xdp_umem *umem)\n+{\n+\tstruct ice_pf *pf = vsi->back;\n+\tstruct device *dev;\n+\tunsigned int i;\n+\n+\tdev = &pf->pdev->dev;\n+\tfor (i = 0; i < umem->npgs; i++) {\n+\t\tdma_unmap_page_attrs(dev, umem->pages[i].dma, PAGE_SIZE,\n+\t\t\t\t DMA_BIDIRECTIONAL, ICE_RX_DMA_ATTR);\n+\n+\t\tumem->pages[i].dma = 0;\n+\t}\n+}\n+\n+/**\n+ * ice_xsk_umem_disable - disable a UMEM region\n+ * @vsi: Current VSI\n+ * @qid: queue ID\n+ *\n+ * Returns 0 on success, negative on failure\n+ */\n+static int ice_xsk_umem_disable(struct ice_vsi *vsi, u16 qid)\n+{\n+\tif (!vsi->xsk_umems || qid >= vsi->num_xsk_umems ||\n+\t !vsi->xsk_umems[qid])\n+\t\treturn -EINVAL;\n+\n+\tice_xsk_umem_dma_unmap(vsi, vsi->xsk_umems[qid]);\n+\tice_xsk_remove_umem(vsi, qid);\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ice_xsk_umem_enable - enable a UMEM region\n+ * @vsi: Current VSI\n+ * @umem: pointer to a requested UMEM region\n+ * @qid: queue ID\n+ *\n+ * Returns 0 on success, negative on failure\n+ */\n+static int\n+ice_xsk_umem_enable(struct ice_vsi *vsi, struct xdp_umem *umem, u16 qid)\n+{\n+\tstruct xdp_umem_fq_reuse *reuseq;\n+\tint err;\n+\n+\tif (vsi->type != ICE_VSI_PF)\n+\t\treturn -EINVAL;\n+\n+\tvsi->num_xsk_umems = min_t(u16, vsi->num_rxq, vsi->num_txq);\n+\tif (qid >= vsi->num_xsk_umems)\n+\t\treturn -EINVAL;\n+\n+\tif (vsi->xsk_umems && vsi->xsk_umems[qid])\n+\t\treturn -EBUSY;\n+\n+\treuseq = xsk_reuseq_prepare(vsi->rx_rings[0]->count);\n+\tif (!reuseq)\n+\t\treturn -ENOMEM;\n+\n+\txsk_reuseq_free(xsk_reuseq_swap(umem, reuseq));\n+\n+\terr = ice_xsk_umem_dma_map(vsi, umem);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = ice_xsk_add_umem(vsi, umem, qid);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ice_xsk_umem_setup - enable/disable a UMEM region depending on its state\n+ * @vsi: Current VSI\n+ * @umem: UMEM to enable/associate to a ring, NULL to disable\n+ * @qid: queue ID\n+ *\n+ * Returns 0 on success, negative on failure\n+ */\n+int ice_xsk_umem_setup(struct ice_vsi *vsi, struct xdp_umem *umem, u16 qid)\n+{\n+\tbool if_running, umem_present = !!umem;\n+\tint ret = 0, umem_failure = 0;\n+\n+\tif_running = netif_running(vsi->netdev) && ice_is_xdp_ena_vsi(vsi);\n+\n+\tif (if_running && !test_and_set_bit(__ICE_DOWN, vsi->state)) {\n+\t\tret = ice_down(vsi);\n+\t\tif (ret) {\n+\t\t\tnetdev_err(vsi->netdev, \"Could not bring down the interface for UMEM setup, error = %d\",\n+\t\t\t\t ret);\n+\t\t\tgoto xsk_umem_if_up;\n+\t\t}\n+\t}\n+\n+\tumem_failure = umem_present ? ice_xsk_umem_enable(vsi, umem, qid) :\n+\t\t\t\t ice_xsk_umem_disable(vsi, qid);\n+\n+xsk_umem_if_up:\n+\tif (if_running) {\n+\t\tret = ice_up(vsi);\n+\t\tif (!ret && umem_present)\n+\t\t\tnapi_schedule(&vsi->xdp_rings[qid]->q_vector->napi);\n+\t\telse if (ret)\n+\t\t\tnetdev_err(vsi->netdev, \"Could not bring down the interface for UMEM setup, error = %d\",\n+\t\t\t\t ret);\n+\t}\n+\n+\tif (umem_failure) {\n+\t\tnetdev_err(vsi->netdev, \"Could not %sable UMEM, error = %d\",\n+\t\t\t umem_present ? \"en\" : \"dis\", umem_failure);\n+\t\treturn umem_failure;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/**\n+ * ice_zca_free - Callback for MEM_TYPE_ZERO_COPY allocations\n+ * @zca: zero-cpoy allocator\n+ * @handle: Buffer handle\n+ */\n+void ice_zca_free(struct zero_copy_allocator *zca, unsigned long handle)\n+{\n+\tstruct ice_rx_buf *rx_buf;\n+\tstruct ice_ring *rx_ring;\n+\tstruct xdp_umem *umem;\n+\tu64 hr, mask;\n+\tu16 nta;\n+\n+\trx_ring = container_of(zca, struct ice_ring, zca);\n+\tumem = rx_ring->xsk_umem;\n+\thr = umem->headroom + XDP_PACKET_HEADROOM;\n+\n+\tmask = umem->chunk_mask;\n+\n+\tnta = rx_ring->next_to_alloc;\n+\trx_buf = &rx_ring->rx_buf[nta];\n+\n+\tnta++;\n+\trx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;\n+\n+\thandle &= mask;\n+\n+\trx_buf->dma = xdp_umem_get_dma(umem, handle);\n+\trx_buf->dma += hr;\n+\n+\trx_buf->addr = xdp_umem_get_data(umem, handle);\n+\trx_buf->addr += hr;\n+\n+\trx_buf->handle = (u64)handle + umem->headroom;\n+}\n+\n+/**\n+ * ice_alloc_buf_fast_zc - Retrieve buffer address from XDP umem\n+ * @rx_ring: ring with an xdp_umem bound to it\n+ * @rx_buf: buffer to which xsk page address will be assigned\n+ *\n+ * This function allocates an Rx buffer in the hot path.\n+ * The buffer can come from fill queue or recycle queue.\n+ *\n+ * Returns true if an assignment was successful, false if not.\n+ */\n+static __always_inline bool\n+ice_alloc_buf_fast_zc(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf)\n+{\n+\tstruct xdp_umem *umem = rx_ring->xsk_umem;\n+\tvoid *addr = rx_buf->addr;\n+\tu64 handle, hr;\n+\n+\tif (addr) {\n+\t\trx_ring->rx_stats.page_reuse_count++;\n+\t\treturn true;\n+\t}\n+\n+\tif (!xsk_umem_peek_addr(umem, &handle)) {\n+\t\trx_ring->rx_stats.alloc_page_failed++;\n+\t\treturn false;\n+\t}\n+\n+\thr = umem->headroom + XDP_PACKET_HEADROOM;\n+\n+\trx_buf->dma = xdp_umem_get_dma(umem, handle);\n+\trx_buf->dma += hr;\n+\trx_buf->addr = xdp_umem_get_data(umem, handle);\n+\trx_buf->addr += hr;\n+\trx_buf->handle = handle + umem->headroom;\n+\n+\txsk_umem_discard_addr(umem);\n+\treturn true;\n+}\n+\n+/**\n+ * ice_alloc_buf_slow_zc - Retrieve buffer address from XDP umem\n+ * @rx_ring: ring with an xdp_umem bound to it\n+ * @rx_buf: buffer to which xsk page address will be assigned\n+ *\n+ * This function allocates an Rx buffer in the slow path.\n+ * The buffer can come from fill queue or recycle queue.\n+ *\n+ * Returns true if an assignment was successful, false if not.\n+ */\n+static __always_inline bool\n+ice_alloc_buf_slow_zc(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf)\n+{\n+\tstruct xdp_umem *umem = rx_ring->xsk_umem;\n+\tu64 handle, headroom;\n+\n+\tif (!xsk_umem_peek_addr_rq(umem, &handle)) {\n+\t\trx_ring->rx_stats.alloc_page_failed++;\n+\t\treturn false;\n+\t}\n+\n+\thandle &= umem->chunk_mask;\n+\theadroom = umem->headroom + XDP_PACKET_HEADROOM;\n+\n+\trx_buf->dma = xdp_umem_get_dma(umem, handle) + headroom;\n+\trx_buf->addr = xdp_umem_get_data(umem, handle) + headroom;\n+\trx_buf->handle = handle + umem->headroom;\n+\n+\txsk_umem_discard_addr_rq(umem);\n+\treturn true;\n+}\n+\n+/**\n+ * ice_alloc_rx_bufs_zc - allocate a number of Rx buffers\n+ * @rx_ring: Rx ring\n+ * @count: The number of buffers to allocate\n+ * @rx_buf: Pointer to a buffer which will be allocated\n+ *\n+ * This function allocates a number of Rx buffers from the fill ring\n+ * or the internal recycle mechanism and places them on the Rx ring.\n+ *\n+ * Returns false if all allocations were successful, true if any fail.\n+ */\n+static bool\n+ice_alloc_rx_bufs_zc(struct ice_ring *rx_ring, int count,\n+\t\t bool alloc(struct ice_ring *rx_ring,\n+\t\t\t\tstruct ice_rx_buf *rx_buf))\n+{\n+\tunion ice_32b_rx_flex_desc *rx_desc;\n+\tu16 ntu = rx_ring->next_to_use;\n+\tstruct ice_rx_buf *rx_buf;\n+\tbool ret = false;\n+\n+\tif (!count)\n+\t\treturn ret;\n+\n+\trx_desc = ICE_RX_DESC(rx_ring, ntu);\n+\trx_buf = &rx_ring->rx_buf[ntu];\n+\n+\tdo {\n+\t\tif (!alloc(rx_ring, rx_buf)) {\n+\t\t\tret = true;\n+\t\t\tgoto no_bufs;\n+\t\t}\n+\n+\t\tdma_sync_single_range_for_device(rx_ring->dev, rx_buf->dma, 0,\n+\t\t\t\t\t\t rx_ring->rx_buf_len,\n+\t\t\t\t\t\t DMA_BIDIRECTIONAL);\n+\n+\t\trx_desc->read.pkt_addr = cpu_to_le64(rx_buf->dma);\n+\t\trx_desc->wb.status_error0 = 0;\n+\n+\t\trx_desc++;\n+\t\trx_buf++;\n+\t\tntu++;\n+\n+\t\tif (unlikely(ntu == rx_ring->count)) {\n+\t\t\trx_desc = ICE_RX_DESC(rx_ring, 0);\n+\t\t\trx_buf = rx_ring->rx_buf;\n+\t\t\tntu = 0;\n+\t\t}\n+\t} while (--count);\n+\n+no_bufs:\n+\tif (rx_ring->next_to_use != ntu)\n+\t\tice_release_rx_desc(rx_ring, ntu);\n+\n+\treturn ret;\n+}\n+\n+/**\n+ * ice_alloc_rx_bufs_fast_zc - allocate zero copy bufs in the hot path\n+ * @rx_ring: Rx ring\n+ * @count: number of bufs to allocate\n+ *\n+ * Returns false on success, true on failure.\n+ */\n+static bool ice_alloc_rx_bufs_fast_zc(struct ice_ring *rx_ring, u16 count)\n+{\n+\treturn ice_alloc_rx_bufs_zc(rx_ring, count,\n+\t\t\t\t ice_alloc_buf_fast_zc);\n+}\n+\n+/**\n+ * ice_alloc_rx_bufs_slow_zc - allocate zero copy bufs in the slow path\n+ * @rx_ring: Rx ring\n+ * @count: number of bufs to allocate\n+ *\n+ * Returns false on success, true on failure.\n+ */\n+bool ice_alloc_rx_bufs_slow_zc(struct ice_ring *rx_ring, u16 count)\n+{\n+\treturn ice_alloc_rx_bufs_zc(rx_ring, count,\n+\t\t\t\t ice_alloc_buf_slow_zc);\n+}\n+\n+/**\n+ * ice_bump_ntc - Bump the next_to_clean counter of an Rx ring\n+ * @rx_ring: Rx ring\n+ */\n+static void ice_bump_ntc(struct ice_ring *rx_ring)\n+{\n+\tint ntc = rx_ring->next_to_clean + 1;\n+\n+\tntc = (ntc < rx_ring->count) ? ntc : 0;\n+\trx_ring->next_to_clean = ntc;\n+\tprefetch(ICE_RX_DESC(rx_ring, ntc));\n+}\n+\n+/**\n+ * ice_get_rx_buf_zc - Fetch the current Rx buffer\n+ * @rx_ring: Rx ring\n+ * @size: size of a buffer\n+ *\n+ * This function returns the current, received Rx buffer and does\n+ * DMA synchronization.\n+ *\n+ * Returns a pointer to the received Rx buffer.\n+ */\n+static struct ice_rx_buf *ice_get_rx_buf_zc(struct ice_ring *rx_ring, int size)\n+{\n+\tstruct ice_rx_buf *rx_buf;\n+\n+\trx_buf = &rx_ring->rx_buf[rx_ring->next_to_clean];\n+\n+\tdma_sync_single_range_for_cpu(rx_ring->dev, rx_buf->dma, 0,\n+\t\t\t\t size, DMA_BIDIRECTIONAL);\n+\n+\treturn rx_buf;\n+}\n+\n+/**\n+ * ice_reuse_rx_buf_zc - reuse an Rx buffer\n+ * @rx_ring: Rx ring\n+ * @old_buf: The buffer to recycle\n+ *\n+ * This function recycles a finished Rx buffer, and places it on the recycle\n+ * queue (next_to_alloc).\n+ */\n+static void\n+ice_reuse_rx_buf_zc(struct ice_ring *rx_ring, struct ice_rx_buf *old_buf)\n+{\n+\tunsigned long mask = (unsigned long)rx_ring->xsk_umem->chunk_mask;\n+\tu64 hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;\n+\tu16 nta = rx_ring->next_to_alloc;\n+\tstruct ice_rx_buf *new_buf;\n+\n+\tnew_buf = &rx_ring->rx_buf[rx_ring->next_to_alloc];\n+\tnta++;\n+\trx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;\n+\n+\tnew_buf->dma = old_buf->dma & mask;\n+\tnew_buf->dma += hr;\n+\n+\tnew_buf->addr = (void *)((unsigned long)old_buf->addr & mask);\n+\tnew_buf->addr += hr;\n+\tnew_buf->handle = (old_buf->handle & mask);\n+\tnew_buf->handle += rx_ring->xsk_umem->headroom;\n+\n+\told_buf->addr = NULL;\n+}\n+\n+/**\n+ * ice_construct_skb_zc - Create an sk_buff from zero-copy buffer\n+ * @rx_ring: Rx ring\n+ * @rx_buf: zero-copy Rx buffer\n+ * @xdp: XDP buffer\n+ *\n+ * This function allocates a new skb from a zero-copy Rx buffer.\n+ *\n+ * Returns the skb on success, NULL on failure.\n+ */\n+static struct sk_buff *\n+ice_construct_skb_zc(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf,\n+\t\t struct xdp_buff *xdp)\n+{\n+\tu64 datasize_hard = xdp->data_end - xdp->data_hard_start;\n+\tu64 metasize = xdp->data - xdp->data_meta;\n+\tu64 datasize = xdp->data_end - xdp->data;\n+\tstruct sk_buff *skb;\n+\n+\tskb = __napi_alloc_skb(&rx_ring->q_vector->napi, datasize_hard,\n+\t\t\t GFP_ATOMIC | __GFP_NOWARN);\n+\tif (unlikely(!skb))\n+\t\treturn NULL;\n+\n+\tskb_reserve(skb, xdp->data - xdp->data_hard_start);\n+\tmemcpy(__skb_put(skb, datasize), xdp->data, datasize);\n+\tif (metasize)\n+\t\tskb_metadata_set(skb, metasize);\n+\n+\tice_reuse_rx_buf_zc(rx_ring, rx_buf);\n+\n+\treturn skb;\n+}\n+\n+/**\n+ * ice_run_xdp_zc - Executes an XDP program in zero-copy path\n+ * @rx_ring: Rx ring\n+ * @xdp: xdp_buff used as input to the XDP program\n+ * @xdp_prog: XDP program to run\n+ *\n+ * Returns any of ICE_XDP_{PASS, CONSUMED, TX, REDIR}\n+ */\n+static int\n+ice_run_xdp_zc(struct ice_ring *rx_ring, struct xdp_buff *xdp,\n+\t struct bpf_prog *xdp_prog)\n+{\n+\tu16 hard_size = xdp->data_end - xdp->data_hard_start;\n+\tint err, result = ICE_XDP_PASS;\n+\tstruct ice_ring *xdp_ring;\n+\tu32 act;\n+\n+\tact = bpf_prog_run_xdp(xdp_prog, xdp);\n+\txdp->handle += xdp->data - xdp->data_hard_start;\n+\tswitch (act) {\n+\tcase XDP_PASS:\n+\t\tbreak;\n+\tcase XDP_TX:\n+\t\txdp_ring = rx_ring->vsi->xdp_rings[rx_ring->q_index];\n+\t\tresult = ice_xmit_xdp_ring(xdp->data, hard_size, xdp_ring);\n+\t\tbreak;\n+\tcase XDP_REDIRECT:\n+\t\terr = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);\n+\t\tresult = !err ? ICE_XDP_REDIR : ICE_XDP_CONSUMED;\n+\t\tbreak;\n+\tdefault:\n+\t\tbpf_warn_invalid_xdp_action(act);\n+\t\t/* fallthrough -- not supported action */\n+\tcase XDP_ABORTED:\n+\t\ttrace_xdp_exception(rx_ring->netdev, xdp_prog, act);\n+\t\t/* fallthrough -- handle aborts by dropping frame */\n+\tcase XDP_DROP:\n+\t\tresult = ICE_XDP_CONSUMED;\n+\t\tbreak;\n+\t}\n+\n+\treturn result;\n+}\n+\n+/**\n+ * ice_clean_rx_irq_zc - consimes packets from the hardware ring\n+ * @rx_ring: AF_XDP Rx ring\n+ * @budget: NAPI budget\n+ *\n+ * Returns number of processed packets on success, remaining budget on failure.\n+ */\n+int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget)\n+{\n+\tunsigned int total_rx_bytes = 0, total_rx_packets = 0;\n+\tu16 cleaned_count = ICE_DESC_UNUSED(rx_ring);\n+\tunsigned int xdp_xmit = 0;\n+\tstruct bpf_prog *xdp_prog;\n+\tbool failure = false;\n+\tstruct xdp_buff xdp;\n+\n+\txdp.rxq = &rx_ring->xdp_rxq;\n+\n+\twhile (likely(total_rx_packets < (unsigned int)budget)) {\n+\t\tunion ice_32b_rx_flex_desc *rx_desc;\n+\t\tunsigned int size, xdp_res = 0;\n+\t\tstruct ice_rx_buf *rx_buf;\n+\t\tstruct sk_buff *skb;\n+\t\tu16 stat_err_bits;\n+\t\tu16 vlan_tag = 0;\n+\t\tu8 rx_ptype;\n+\n+\t\tif (cleaned_count >= ICE_RX_BUF_WRITE) {\n+\t\t\tfailure |= ice_alloc_rx_bufs_fast_zc(rx_ring,\n+\t\t\t\t\t\t\t cleaned_count);\n+\t\t\tcleaned_count = 0;\n+\t\t}\n+\n+\t\trx_desc = ICE_RX_DESC(rx_ring, rx_ring->next_to_clean);\n+\n+\t\tstat_err_bits = BIT(ICE_RX_FLEX_DESC_STATUS0_DD_S);\n+\t\tif (!ice_test_staterr(rx_desc, stat_err_bits))\n+\t\t\tbreak;\n+\n+\t\t/* This memory barrier is needed to keep us from reading\n+\t\t * any other fields out of the rx_desc until we have\n+\t\t * verified the descriptor has been written back.\n+\t\t */\n+\t\tdma_rmb();\n+\n+\t\tsize = le16_to_cpu(rx_desc->wb.pkt_len) &\n+\t\t\t\t ICE_RX_FLX_DESC_PKT_LEN_M;\n+\n+\t\tif (!size)\n+\t\t\tbreak;\n+\n+\t\trx_buf = ice_get_rx_buf_zc(rx_ring, size);\n+\n+\t\txdp.data = rx_buf->addr;\n+\t\txdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;\n+\t\txdp.data_end = xdp.data + size;\n+\t\txdp_set_data_meta_invalid(&xdp);\n+\t\txdp.handle = rx_buf->handle;\n+\n+\t\trcu_read_lock();\n+\t\txdp_prog = READ_ONCE(rx_ring->xdp_prog);\n+\t\tif (!xdp_prog) {\n+\t\t\trcu_read_unlock();\n+\t\t\tgoto xdp_pass_path;\n+\t\t}\n+\n+\t\txdp_res = ice_run_xdp_zc(rx_ring, &xdp, xdp_prog);\n+\t\trcu_read_unlock();\n+\t\tif (xdp_res) {\n+\t\t\tif (xdp_res & (ICE_XDP_TX | ICE_XDP_REDIR)) {\n+\t\t\t\txdp_xmit |= xdp_res;\n+\t\t\t\trx_buf->addr = NULL;\n+\t\t\t} else {\n+\t\t\t\tice_reuse_rx_buf_zc(rx_ring, rx_buf);\n+\t\t\t}\n+\n+\t\t\ttotal_rx_bytes += size;\n+\t\t\ttotal_rx_packets++;\n+\t\t\tcleaned_count++;\n+\n+\t\t\tice_bump_ntc(rx_ring);\n+\t\t\tcontinue;\n+\t\t}\n+\n+xdp_pass_path:\n+\t\t/* XDP_PASS path */\n+\t\tskb = ice_construct_skb_zc(rx_ring, rx_buf, &xdp);\n+\t\tif (!skb) {\n+\t\t\trx_ring->rx_stats.alloc_buf_failed++;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tcleaned_count++;\n+\t\tice_bump_ntc(rx_ring);\n+\n+\t\trx_ptype = le16_to_cpu(rx_desc->wb.ptype_flex_flags0) &\n+\t\t\t\t ICE_RX_FLEX_DESC_PTYPE_M;\n+\t\tstat_err_bits = BIT(ICE_RX_FLEX_DESC_STATUS0_L2TAG1P_S);\n+\t\tif (ice_test_staterr(rx_desc, stat_err_bits))\n+\t\t\tvlan_tag = le16_to_cpu(rx_desc->wb.l2tag1);\n+\n+\t\tif (ice_cleanup_hdrs(skb)) {\n+\t\t\tskb = NULL;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttotal_rx_bytes += skb->len;\n+\t\ttotal_rx_packets++;\n+\n+\t\tice_process_skb_fields(rx_ring, rx_desc, skb, rx_ptype);\n+\n+\t\tice_receive_skb(rx_ring, skb, vlan_tag);\n+\t}\n+\n+\tice_finalize_xdp_rx(rx_ring, xdp_xmit);\n+\n+\treturn failure ? budget : (int)total_rx_packets;\n+}\n+\n+/**\n+ * ice_xmit_zc - Completes AF_XDP entries, and cleans XDP entries\n+ * @xdp_ring: XDP Tx ring\n+ * @budget: used to determine if we are in netpoll\n+ *\n+ * Returns true if cleanup/transmission is done.\n+ */\n+static bool ice_xmit_zc(struct ice_ring *xdp_ring, int budget)\n+{\n+\tstruct ice_tx_desc *tx_desc = NULL;\n+\tbool work_done = true;\n+\tdma_addr_t dma;\n+\tu32 len;\n+\n+\twhile (likely(budget-- > 0)) {\n+\t\tstruct ice_tx_buf *tx_buf;\n+\n+\t\tif (!unlikely(ICE_DESC_UNUSED(xdp_ring))) {\n+\t\t\txdp_ring->tx_stats.tx_busy++;\n+\t\t\twork_done = false;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &dma, &len))\n+\t\t\tbreak;\n+\n+\t\tdma_sync_single_for_device(xdp_ring->dev, dma, len,\n+\t\t\t\t\t DMA_BIDIRECTIONAL);\n+\n+\t\ttx_buf = &xdp_ring->tx_buf[xdp_ring->next_to_use];\n+\t\ttx_buf->bytecount = len;\n+\n+\t\ttx_desc = ICE_TX_DESC(xdp_ring, xdp_ring->next_to_use);\n+\t\ttx_desc->buf_addr = cpu_to_le64(dma);\n+\t\ttx_desc->cmd_type_offset_bsz =\n+\t\t\tice_build_ctob(ICE_TX_DESC_CMD_EOP | ICE_TX_DESC_CMD_RS,\n+\t\t\t\t 0, len, 0);\n+\n+\t\txdp_ring->next_to_use++;\n+\n+\t\tif (xdp_ring->next_to_use == xdp_ring->count)\n+\t\t\txdp_ring->next_to_use = 0;\n+\t}\n+\n+\tif (tx_desc) {\n+\t\tice_xdp_ring_update_tail(xdp_ring);\n+\t\txsk_umem_consume_tx_done(xdp_ring->xsk_umem);\n+\t}\n+\n+\treturn !!budget && work_done;\n+}\n+\n+/**\n+ * ice_clean_xdp_tx_buf - Free and unmap XDP Tx buffer\n+ * @xdp_ring: XDP Tx ring\n+ * @tx_buf: Tx buffer to clean\n+ */\n+static void\n+ice_clean_xdp_tx_buf(struct ice_ring *xdp_ring, struct ice_tx_buf *tx_buf)\n+{\n+\txdp_return_frame((struct xdp_frame *)tx_buf->raw_buf);\n+\tdma_unmap_single(xdp_ring->dev, dma_unmap_addr(tx_buf, dma),\n+\t\t\t dma_unmap_len(tx_buf, len), DMA_TO_DEVICE);\n+\tdma_unmap_len_set(tx_buf, len, 0);\n+}\n+\n+/**\n+ * ice_clean_xdp_tx_irq - Completes AF_XDP entries, and cleans XDP entries\n+ * @vsi: The VSI that we care about\n+ * @xdp_ring: XDP Tx ring\n+ *\n+ * Returns true if cleanup/tranmission is done.\n+ */\n+bool ice_clean_tx_irq_zc(struct ice_vsi *vsi, struct ice_ring *xdp_ring)\n+{\n+\tu32 cleaned_frames, frames_to_xmit, xsk_frames = 0;\n+\tunsigned int i, ntc, ntu, budget, total_bytes = 0;\n+\tstruct xdp_umem *umem = xdp_ring->xsk_umem;\n+\tbool work_done = true, xmit_done;\n+\tstruct ice_tx_buf *tx_buf;\n+\n+\tbudget = vsi->work_lmt;\n+\n+\tntc = xdp_ring->next_to_clean;\n+\tntu = xdp_ring->next_to_use;\n+\n+\tif (ntu <= ntc)\n+\t\tntu += xdp_ring->count;\n+\tframes_to_xmit = ntu - ntc;\n+\n+\tif (frames_to_xmit == 0) {\n+\t\tgoto out_xmit;\n+\t} else if (frames_to_xmit > budget) {\n+\t\tcleaned_frames = budget;\n+\t\twork_done = false;\n+\t} else {\n+\t\tcleaned_frames = frames_to_xmit;\n+\t}\n+\n+\tfor (i = 0; i < cleaned_frames; i++) {\n+\t\tstruct ice_tx_desc *tx_desc = ICE_TX_DESC(xdp_ring, ntc);\n+\n+\t\ttx_buf = &xdp_ring->tx_buf[ntc];\n+\t\t/* if the descriptor isn't marked as done, stop processing */\n+\t\tif (!(tx_desc->cmd_type_offset_bsz &\n+\t\t cpu_to_le64(ICE_TX_DESC_DTYPE_DESC_DONE)))\n+\t\t\tbreak;\n+\n+\t\tif (tx_buf->raw_buf)\n+\t\t\tice_clean_xdp_tx_buf(xdp_ring, tx_buf);\n+\t\telse\n+\t\t\txsk_frames++;\n+\n+\t\ttx_buf->raw_buf = NULL;\n+\t\ttotal_bytes += tx_buf->bytecount;\n+\n+\t\tif (++ntc == xdp_ring->count)\n+\t\t\tntc = 0;\n+\t}\n+\n+\txdp_ring->next_to_clean += i;\n+\tif (unlikely(xdp_ring->next_to_clean >= xdp_ring->count))\n+\t\txdp_ring->next_to_clean -= xdp_ring->count;\n+\n+\tif (xsk_frames)\n+\t\txsk_umem_complete_tx(umem, xsk_frames);\n+\n+\tu64_stats_update_begin(&xdp_ring->syncp);\n+\txdp_ring->stats.bytes += total_bytes;\n+\txdp_ring->stats.pkts += cleaned_frames;\n+\tu64_stats_update_end(&xdp_ring->syncp);\n+\txdp_ring->q_vector->tx.total_bytes += total_bytes;\n+\txdp_ring->q_vector->tx.total_pkts += cleaned_frames;\n+\n+out_xmit:\n+\txmit_done = ice_xmit_zc(xdp_ring, budget);\n+\n+\treturn work_done && xmit_done;\n+}\n+\n+/**\n+ * ice_xsk_async_xmit - Implements ndo_xsk_async_xmit\n+ * @netdev: net_device\n+ * @queue_id: queue to wake up\n+ *\n+ * Returns negative on error, zero otherwise.\n+ */\n+int ice_xsk_async_xmit(struct net_device *netdev, u32 queue_id)\n+{\n+\tstruct ice_netdev_priv *np = netdev_priv(netdev);\n+\tstruct ice_q_vector *q_vector;\n+\tstruct ice_vsi *vsi = np->vsi;\n+\tstruct ice_ring *ring;\n+\n+\tif (test_bit(__ICE_DOWN, vsi->state))\n+\t\treturn -ENETDOWN;\n+\n+\tif (!ice_is_xdp_ena_vsi(vsi))\n+\t\treturn -ENXIO;\n+\n+\tif (queue_id >= vsi->num_txq)\n+\t\treturn -ENXIO;\n+\n+\tif (!vsi->xdp_rings[queue_id]->xsk_umem)\n+\t\treturn -ENXIO;\n+\n+\tring = vsi->xdp_rings[queue_id];\n+\n+\t/* The idea here is that if NAPI is running, mark a miss, so\n+\t * it will run again. If not, trigger an interrupt and\n+\t * schedule the NAPI from interrupt context. If NAPI would be\n+\t * scheduled here, the interrupt affinity would not be\n+\t * honored.\n+\t */\n+\tq_vector = ring->q_vector;\n+\tif (!napi_if_scheduled_mark_missed(&q_vector->napi))\n+\t\tice_trigger_sw_intr(&vsi->back->hw, q_vector);\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ice_xsk_any_rx_ring_ena - Checks if Rx rings have AF_XDP UMEM attached\n+ * @vsi: VSI to be checked\n+ *\n+ * Returns true if any of the Rx rings has an AF_XDP UMEM attached\n+ */\n+bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi)\n+{\n+\tint i;\n+\n+\tif (!vsi->xsk_umems)\n+\t\treturn false;\n+\n+\tfor (i = 0; i < vsi->num_xsk_umems; i++) {\n+\t\tif (vsi->xsk_umems[i])\n+\t\t\treturn true;\n+\t}\n+\n+\treturn false;\n+}\n+\n+/**\n+ * ice_xsk_clean_rx_ring - clean UMEM queues connected to a given Rx ring\n+ * @rx_ring: ring to be cleaned\n+ */\n+void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring)\n+{\n+\tu16 i;\n+\n+\tfor (i = 0; i < rx_ring->count; i++) {\n+\t\tstruct ice_rx_buf *rx_buf = &rx_ring->rx_buf[i];\n+\n+\t\tif (!rx_buf->addr)\n+\t\t\tcontinue;\n+\n+\t\txsk_umem_fq_reuse(rx_ring->xsk_umem, rx_buf->handle);\n+\t\trx_buf->addr = NULL;\n+\t}\n+}\n+\n+/**\n+ * ice_xsk_clean_xdp_ring - Clean the XDP Tx ring and its UMEM queues\n+ * @xdp_ring: XDP_Tx ring\n+ */\n+void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring)\n+{\n+\tu16 ntc = xdp_ring->next_to_clean, ntu = xdp_ring->next_to_use;\n+\tu32 xsk_frames = 0;\n+\n+\twhile (ntc != ntu) {\n+\t\tstruct ice_tx_buf *tx_buf = &xdp_ring->tx_buf[ntc];\n+\n+\t\tif (tx_buf->raw_buf)\n+\t\t\tice_clean_xdp_tx_buf(xdp_ring, tx_buf);\n+\t\telse\n+\t\t\txsk_frames++;\n+\n+\t\ttx_buf->raw_buf = NULL;\n+\n+\t\tntc++;\n+\t\tif (ntc >= xdp_ring->count)\n+\t\t\tntc = 0;\n+\t}\n+\n+\tif (xsk_frames)\n+\t\txsk_umem_complete_tx(xdp_ring->xsk_umem, xsk_frames);\n+}\ndiff --git a/drivers/net/ethernet/intel/ice/ice_xsk.h b/drivers/net/ethernet/intel/ice/ice_xsk.h\nnew file mode 100644\nindex 000000000000..3e70941f230d\n--- /dev/null\n+++ b/drivers/net/ethernet/intel/ice/ice_xsk.h\n@@ -0,0 +1,81 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Copyright (c) 2019, Intel Corporation. */\n+\n+#ifndef _ICE_XSK_H_\n+#define _ICE_XSK_H_\n+#include \"ice_txrx.h\"\n+#include \"ice.h\"\n+\n+struct ice_vsi;\n+\n+#ifdef CONFIG_XDP_SOCKETS\n+int ice_xsk_umem_setup(struct ice_vsi *vsi, struct xdp_umem *umem, u16 qid);\n+int ice_xsk_umem_query(struct ice_vsi *vsi, struct xdp_umem **umem, u16 qid);\n+void ice_zca_free(struct zero_copy_allocator *zca, unsigned long handle);\n+int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget);\n+bool ice_clean_tx_irq_zc(struct ice_vsi *vsi, struct ice_ring *tx_ring);\n+int ice_xsk_async_xmit(struct net_device *netdev, u32 queue_id);\n+bool ice_alloc_rx_bufs_slow_zc(struct ice_ring *rx_ring, u16 count);\n+bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi);\n+void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring);\n+void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring);\n+#else\n+static inline int\n+ice_xsk_umem_setup(struct ice_vsi __always_unused *vsi,\n+\t\t struct xdp_umem __always_unused *umem,\n+\t\t u16 __always_unused qid)\n+{\n+\treturn -ENOTSUPP;\n+}\n+\n+static inline int\n+ice_xsk_umem_query(struct ice_vsi __always_unused *vsi,\n+\t\t struct xdp_umem __always_unused **umem,\n+\t\t u16 __always_unused qid)\n+{\n+\treturn -ENOTSUPP;\n+}\n+\n+static inline void\n+ice_zca_free(struct zero_copy_allocator __always_unused *zca,\n+\t unsigned long __always_unused handle)\n+{\n+}\n+\n+static inline int\n+ice_clean_rx_irq_zc(struct ice_ring __always_unused *rx_ring,\n+\t\t int __always_unused budget)\n+{\n+\treturn 0;\n+}\n+\n+static inline bool\n+ice_clean_tx_irq_zc(struct ice_vsi __always_unused *vsi,\n+\t\t struct ice_ring __always_unused *tx_ring)\n+{\n+\treturn false;\n+}\n+\n+static inline int\n+ice_xsk_async_xmit(struct net_device __always_unused *netdev,\n+\t\t u32 __always_unused queue_id)\n+{\n+\treturn -ENOTSUPP;\n+}\n+\n+static inline bool\n+ice_alloc_rx_bufs_slow_zc(struct ice_ring __always_unused *rx_ring,\n+\t\t\t u16 __always_unused count)\n+{\n+\treturn false;\n+}\n+\n+static inline bool ice_xsk_any_rx_ring_ena(struct ice_vsi __always_unused *vsi)\n+{\n+\treturn false;\n+}\n+\n+#define ice_xsk_clean_rx_ring(rx_ring) do {} while (0)\n+#define ice_xsk_clean_xdp_ring(xdp_ring) do {} while (0)\n+#endif /* CONFIG_XDP_SOCKETS */\n+#endif /* !_ICE_XSK_H_ */\n", "prefixes": [ "v2", "2/2" ] }