Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/964078/?format=api
{ "id": 964078, "url": "http://patchwork.ozlabs.org/api/patches/964078/?format=api", "web_url": "http://patchwork.ozlabs.org/project/linux-pci/patch/20180830185352.3369-10-logang@deltatee.com/", "project": { "id": 28, "url": "http://patchwork.ozlabs.org/api/projects/28/?format=api", "name": "Linux PCI development", "link_name": "linux-pci", "list_id": "linux-pci.vger.kernel.org", "list_email": "linux-pci@vger.kernel.org", "web_url": null, "scm_url": null, "webscm_url": null, "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20180830185352.3369-10-logang@deltatee.com>", "list_archive_url": null, "date": "2018-08-30T18:53:48", "name": "[v5,09/13] nvme-pci: Use PCI p2pmem subsystem to manage the CMB", "commit_ref": null, "pull_url": null, "state": "changes-requested", "archived": false, "hash": "cbfab9da1e6badb76ef29c45c16650d1e940cb50", "submitter": { "id": 70191, "url": "http://patchwork.ozlabs.org/api/people/70191/?format=api", "name": "Logan Gunthorpe", "email": "logang@deltatee.com" }, "delegate": { "id": 6763, "url": "http://patchwork.ozlabs.org/api/users/6763/?format=api", "username": "bhelgaas", "first_name": "Bjorn", "last_name": "Helgaas", "email": "bhelgaas@google.com" }, "mbox": "http://patchwork.ozlabs.org/project/linux-pci/patch/20180830185352.3369-10-logang@deltatee.com/mbox/", "series": [ { "id": 63352, "url": "http://patchwork.ozlabs.org/api/series/63352/?format=api", "web_url": "http://patchwork.ozlabs.org/project/linux-pci/list/?series=63352", "date": "2018-08-30T18:53:41", "name": "Copy Offload in NVMe Fabrics with P2P PCI Memory", "version": 5, "mbox": "http://patchwork.ozlabs.org/series/63352/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/patches/964078/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/964078/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<linux-pci-owner@vger.kernel.org>", "X-Original-To": "incoming@patchwork.ozlabs.org", "Delivered-To": "patchwork-incoming@bilbo.ozlabs.org", "Authentication-Results": [ "ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)", "ozlabs.org; dmarc=none (p=none dis=none)\n\theader.from=deltatee.com" ], "Received": [ "from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 421Ws54PY2z9s3C\n\tfor <incoming@patchwork.ozlabs.org>;\n\tFri, 31 Aug 2018 04:55:21 +1000 (AEST)", "(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1728094AbeH3W6v (ORCPT <rfc822;incoming@patchwork.ozlabs.org>);\n\tThu, 30 Aug 2018 18:58:51 -0400", "from ale.deltatee.com ([207.54.116.67]:40062 \"EHLO\n\tale.deltatee.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1727173AbeH3W5l (ORCPT <rfc822;linux-pci@vger.kernel.org>);\n\tThu, 30 Aug 2018 18:57:41 -0400", "from cgy1-donard.priv.deltatee.com ([172.16.1.31])\n\tby ale.deltatee.com with esmtps\n\t(TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)\n\t(envelope-from <gunthorp@deltatee.com>)\n\tid 1fvS4t-0006Oo-Nc; Thu, 30 Aug 2018 12:54:04 -0600", "from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim\n\t4.89) (envelope-from <gunthorp@deltatee.com>)\n\tid 1fvS4p-0000tf-1C; Thu, 30 Aug 2018 12:53:55 -0600" ], "From": "Logan Gunthorpe <logang@deltatee.com>", "To": "linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,\n\tlinux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,\n\tlinux-nvdimm@lists.01.org, linux-block@vger.kernel.org", "Cc": "Stephen Bates <sbates@raithlin.com>, Christoph Hellwig <hch@lst.de>,\n\tKeith Busch <keith.busch@intel.com>, Sagi Grimberg <sagi@grimberg.me>,\n\tBjorn Helgaas <bhelgaas@google.com>, Jason Gunthorpe <jgg@mellanox.com>, \n\tMax Gurtovoy <maxg@mellanox.com>,\n\tDan Williams <dan.j.williams@intel.com>, =?utf-8?b?SsOpcsO0bWUgR2xp?=\n\t=?utf-8?q?sse?= <jglisse@redhat.com>,\n\tBenjamin Herrenschmidt <benh@kernel.crashing.org>, Alex Williamson\n\t<alex.williamson@redhat.com>, =?utf-8?q?Christian_K=C3=B6nig?=\n\t<christian.koenig@amd.com>, Logan Gunthorpe <logang@deltatee.com>", "Date": "Thu, 30 Aug 2018 12:53:48 -0600", "Message-Id": "<20180830185352.3369-10-logang@deltatee.com>", "X-Mailer": "git-send-email 2.11.0", "In-Reply-To": "<20180830185352.3369-1-logang@deltatee.com>", "References": "<20180830185352.3369-1-logang@deltatee.com>", "X-SA-Exim-Connect-IP": "172.16.1.31", "X-SA-Exim-Rcpt-To": "linux-nvme@lists.infradead.org, linux-nvdimm@lists.01.org,\n\tlinux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,\n\tlinux-rdma@vger.kernel.org, linux-block@vger.kernel.org,\n\tsbates@raithlin.com, hch@lst.de, sagi@grimberg.me,\n\tbhelgaas@google.com, jgg@mellanox.com, maxg@mellanox.com,\n\tkeith.busch@intel.com, dan.j.williams@intel.com,\n\tbenh@kernel.crashing.org, jglisse@redhat.com,\n\talex.williamson@redhat.com, christian.koenig@amd.com,\n\tlogang@deltatee.com", "X-SA-Exim-Mail-From": "gunthorp@deltatee.com", "X-Spam-Checker-Version": "SpamAssassin 3.4.1 (2015-04-28) on ale.deltatee.com", "X-Spam-Level": "", "X-Spam-Status": "No, score=-8.5 required=5.0 tests=ALL_TRUSTED,BAYES_00,\n\tGREYLIST_ISWHITE,MYRULES_FREE,MYRULES_NO_TEXT autolearn=ham\n\tautolearn_force=no version=3.4.1", "Subject": "[PATCH v5 09/13] nvme-pci: Use PCI p2pmem subsystem to manage the\n\tCMB", "X-SA-Exim-Version": "4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)", "X-SA-Exim-Scanned": "Yes (on ale.deltatee.com)", "Sender": "linux-pci-owner@vger.kernel.org", "Precedence": "bulk", "List-ID": "<linux-pci.vger.kernel.org>", "X-Mailing-List": "linux-pci@vger.kernel.org" }, "content": "Register the CMB buffer as p2pmem and use the appropriate allocation\nfunctions to create and destroy the IO submission queues.\n\nIf the CMB supports WDS and RDS, publish it for use as P2P memory\nby other devices.\n\nKernels without CONFIG_PCI_P2PDMA will also no longer support NVMe CMB.\nHowever, seeing the main use-cases for the CMB is P2P operations,\nthis seems like a reasonable dependency.\n\nWe drop the __iomem safety on the buffer seeing that, by convention, it's\nsafe to directly access memory mapped by memremap()/devm_memremap_pages().\nArchitectures where this is not safe will not be supported by memremap()\nand therefore will not be support PCI P2P and have no support for CMB.\n\nSigned-off-by: Logan Gunthorpe <logang@deltatee.com>\n---\n drivers/nvme/host/pci.c | 80 +++++++++++++++++++++++++++----------------------\n 1 file changed, 45 insertions(+), 35 deletions(-)", "diff": "diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c\nindex 1b9951d2067e..2902585c6ddf 100644\n--- a/drivers/nvme/host/pci.c\n+++ b/drivers/nvme/host/pci.c\n@@ -30,6 +30,7 @@\n #include <linux/types.h>\n #include <linux/io-64-nonatomic-lo-hi.h>\n #include <linux/sed-opal.h>\n+#include <linux/pci-p2pdma.h>\n \n #include \"nvme.h\"\n \n@@ -99,9 +100,8 @@ struct nvme_dev {\n \tstruct work_struct remove_work;\n \tstruct mutex shutdown_lock;\n \tbool subsystem;\n-\tvoid __iomem *cmb;\n-\tpci_bus_addr_t cmb_bus_addr;\n \tu64 cmb_size;\n+\tbool cmb_use_sqes;\n \tu32 cmbsz;\n \tu32 cmbloc;\n \tstruct nvme_ctrl ctrl;\n@@ -158,7 +158,7 @@ struct nvme_queue {\n \tstruct nvme_dev *dev;\n \tspinlock_t sq_lock;\n \tstruct nvme_command *sq_cmds;\n-\tstruct nvme_command __iomem *sq_cmds_io;\n+\tbool sq_cmds_is_io;\n \tspinlock_t cq_lock ____cacheline_aligned_in_smp;\n \tvolatile struct nvme_completion *cqes;\n \tstruct blk_mq_tags **tags;\n@@ -439,11 +439,8 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set)\n static void nvme_submit_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd)\n {\n \tspin_lock(&nvmeq->sq_lock);\n-\tif (nvmeq->sq_cmds_io)\n-\t\tmemcpy_toio(&nvmeq->sq_cmds_io[nvmeq->sq_tail], cmd,\n-\t\t\t\tsizeof(*cmd));\n-\telse\n-\t\tmemcpy(&nvmeq->sq_cmds[nvmeq->sq_tail], cmd, sizeof(*cmd));\n+\n+\tmemcpy(&nvmeq->sq_cmds[nvmeq->sq_tail], cmd, sizeof(*cmd));\n \n \tif (++nvmeq->sq_tail == nvmeq->q_depth)\n \t\tnvmeq->sq_tail = 0;\n@@ -1224,9 +1221,18 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)\n {\n \tdma_free_coherent(nvmeq->q_dmadev, CQ_SIZE(nvmeq->q_depth),\n \t\t\t\t(void *)nvmeq->cqes, nvmeq->cq_dma_addr);\n-\tif (nvmeq->sq_cmds)\n-\t\tdma_free_coherent(nvmeq->q_dmadev, SQ_SIZE(nvmeq->q_depth),\n-\t\t\t\t\tnvmeq->sq_cmds, nvmeq->sq_dma_addr);\n+\n+\tif (nvmeq->sq_cmds) {\n+\t\tif (nvmeq->sq_cmds_is_io)\n+\t\t\tpci_free_p2pmem(to_pci_dev(nvmeq->q_dmadev),\n+\t\t\t\t\tnvmeq->sq_cmds,\n+\t\t\t\t\tSQ_SIZE(nvmeq->q_depth));\n+\t\telse\n+\t\t\tdma_free_coherent(nvmeq->q_dmadev,\n+\t\t\t\t\t SQ_SIZE(nvmeq->q_depth),\n+\t\t\t\t\t nvmeq->sq_cmds,\n+\t\t\t\t\t nvmeq->sq_dma_addr);\n+\t}\n }\n \n static void nvme_free_queues(struct nvme_dev *dev, int lowest)\n@@ -1315,12 +1321,21 @@ static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues,\n static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,\n \t\t\t\tint qid, int depth)\n {\n-\t/* CMB SQEs will be mapped before creation */\n-\tif (qid && dev->cmb && use_cmb_sqes && (dev->cmbsz & NVME_CMBSZ_SQS))\n-\t\treturn 0;\n+\tstruct pci_dev *pdev = to_pci_dev(dev->dev);\n+\n+\tif (qid && dev->cmb_use_sqes && (dev->cmbsz & NVME_CMBSZ_SQS)) {\n+\t\tnvmeq->sq_cmds = pci_alloc_p2pmem(pdev, SQ_SIZE(depth));\n+\t\tnvmeq->sq_dma_addr = pci_p2pmem_virt_to_bus(pdev,\n+\t\t\t\t\t\tnvmeq->sq_cmds);\n+\t\tnvmeq->sq_cmds_is_io = true;\n+\t}\n+\n+\tif (!nvmeq->sq_cmds) {\n+\t\tnvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),\n+\t\t\t\t\t&nvmeq->sq_dma_addr, GFP_KERNEL);\n+\t\tnvmeq->sq_cmds_is_io = false;\n+\t}\n \n-\tnvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),\n-\t\t\t\t\t &nvmeq->sq_dma_addr, GFP_KERNEL);\n \tif (!nvmeq->sq_cmds)\n \t\treturn -ENOMEM;\n \treturn 0;\n@@ -1397,13 +1412,6 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid)\n \tint result;\n \ts16 vector;\n \n-\tif (dev->cmb && use_cmb_sqes && (dev->cmbsz & NVME_CMBSZ_SQS)) {\n-\t\tunsigned offset = (qid - 1) * roundup(SQ_SIZE(nvmeq->q_depth),\n-\t\t\t\t\t\t dev->ctrl.page_size);\n-\t\tnvmeq->sq_dma_addr = dev->cmb_bus_addr + offset;\n-\t\tnvmeq->sq_cmds_io = dev->cmb + offset;\n-\t}\n-\n \t/*\n \t * A queue's vector matches the queue identifier unless the controller\n \t * has only one vector available.\n@@ -1644,9 +1652,6 @@ static void nvme_map_cmb(struct nvme_dev *dev)\n \t\treturn;\n \tdev->cmbloc = readl(dev->bar + NVME_REG_CMBLOC);\n \n-\tif (!use_cmb_sqes)\n-\t\treturn;\n-\n \tsize = nvme_cmb_size_unit(dev) * nvme_cmb_size(dev);\n \toffset = nvme_cmb_size_unit(dev) * NVME_CMB_OFST(dev->cmbloc);\n \tbar = NVME_CMB_BIR(dev->cmbloc);\n@@ -1663,11 +1668,18 @@ static void nvme_map_cmb(struct nvme_dev *dev)\n \tif (size > bar_size - offset)\n \t\tsize = bar_size - offset;\n \n-\tdev->cmb = ioremap_wc(pci_resource_start(pdev, bar) + offset, size);\n-\tif (!dev->cmb)\n+\tif (pci_p2pdma_add_resource(pdev, bar, size, offset)) {\n+\t\tdev_warn(dev->ctrl.device,\n+\t\t\t \"failed to register the CMB\\n\");\n \t\treturn;\n-\tdev->cmb_bus_addr = pci_bus_address(pdev, bar) + offset;\n+\t}\n+\n \tdev->cmb_size = size;\n+\tdev->cmb_use_sqes = use_cmb_sqes && (dev->cmbsz & NVME_CMBSZ_SQS);\n+\n+\tif ((dev->cmbsz & (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS)) ==\n+\t\t\t(NVME_CMBSZ_WDS | NVME_CMBSZ_RDS))\n+\t\tpci_p2pmem_publish(pdev, true);\n \n \tif (sysfs_add_file_to_group(&dev->ctrl.device->kobj,\n \t\t\t\t &dev_attr_cmb.attr, NULL))\n@@ -1677,12 +1689,10 @@ static void nvme_map_cmb(struct nvme_dev *dev)\n \n static inline void nvme_release_cmb(struct nvme_dev *dev)\n {\n-\tif (dev->cmb) {\n-\t\tiounmap(dev->cmb);\n-\t\tdev->cmb = NULL;\n+\tif (dev->cmb_size) {\n \t\tsysfs_remove_file_from_group(&dev->ctrl.device->kobj,\n \t\t\t\t\t &dev_attr_cmb.attr, NULL);\n-\t\tdev->cmbsz = 0;\n+\t\tdev->cmb_size = 0;\n \t}\n }\n \n@@ -1881,13 +1891,13 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)\n \tif (nr_io_queues == 0)\n \t\treturn 0;\n \n-\tif (dev->cmb && (dev->cmbsz & NVME_CMBSZ_SQS)) {\n+\tif (dev->cmb_use_sqes) {\n \t\tresult = nvme_cmb_qdepth(dev, nr_io_queues,\n \t\t\t\tsizeof(struct nvme_command));\n \t\tif (result > 0)\n \t\t\tdev->q_depth = result;\n \t\telse\n-\t\t\tnvme_release_cmb(dev);\n+\t\t\tdev->cmb_use_sqes = false;\n \t}\n \n \tdo {\n", "prefixes": [ "v5", "09/13" ] }