From patchwork Wed Apr 25 11:24:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pankaj Gupta X-Patchwork-Id: 904138 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WHv93Bd5z9s0t for ; Wed, 25 Apr 2018 21:25:57 +1000 (AEST) Received: from localhost ([::1]:35822 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIYd-00012X-Hg for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:25:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33142) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIXM-0000B0-Dp for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIXK-00014W-SZ for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:36 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:46078 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIXK-000147-Mr for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:34 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 42EE2406C751; Wed, 25 Apr 2018 11:24:34 +0000 (UTC) Received: from dhcp201-121.englab.pnq.redhat.com (dhcp193-127.pnq.redhat.com [10.65.193.127]) by smtp.corp.redhat.com (Postfix) with ESMTP id EE1122023239; Wed, 25 Apr 2018 11:24:27 +0000 (UTC) From: Pankaj Gupta To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, qemu-devel@nongnu.org, linux-nvdimm@ml01.01.org, linux-mm@kvack.org Date: Wed, 25 Apr 2018 16:54:13 +0530 Message-Id: <20180425112415.12327-2-pagupta@redhat.com> In-Reply-To: <20180425112415.12327-1-pagupta@redhat.com> References: <20180425112415.12327-1-pagupta@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:24:34 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:24:34 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pagupta@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [RFC v2 1/2] virtio: add pmem driver X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, haozhong.zhang@intel.com, jack@suse.cz, xiaoguangrong.eric@gmail.com, riel@surriel.com, pagupta@redhat.com, niteshnarayanlal@hotmail.com, david@redhat.com, ross.zwisler@intel.com, lcapitulino@redhat.com, hch@infradead.org, mst@redhat.com, stefanha@redhat.com, imammedo@redhat.com, marcel@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, nilal@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds virtio-pmem driver for KVM guest. Guest reads the persistent memory range information from Qemu over VIRTIO and registers it on nvdimm_bus. It also creates a nd_region object with the persistent memory range information so that existing 'nvdimm/pmem' driver can reserve this into system memory map. This way 'virtio-pmem' driver uses existing functionality of pmem driver to register persistent memory compatible for DAX capable filesystems. This also provides function to perform guest flush over VIRTIO from 'pmem' driver when userspace performs flush on DAX memory range. Signed-off-by: Pankaj Gupta --- drivers/virtio/Kconfig | 12 ++++ drivers/virtio/Makefile | 1 + drivers/virtio/virtio_pmem.c | 118 +++++++++++++++++++++++++++++++++++++++ include/linux/libnvdimm.h | 4 ++ include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_pmem.h | 58 +++++++++++++++++++ 6 files changed, 194 insertions(+) create mode 100644 drivers/virtio/virtio_pmem.c create mode 100644 include/uapi/linux/virtio_pmem.h diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 3589764..879335d 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -42,6 +42,18 @@ config VIRTIO_PCI_LEGACY If unsure, say Y. +config VIRTIO_PMEM + tristate "Virtio pmem driver" + depends on VIRTIO + help + This driver adds persistent memory range to nd_region and registers + with nvdimm bus. NVDIMM 'pmem' driver later allocates a persistent + memory range on the memory information added by this driver. In addition + to this, 'virtio-pmem' driver also provides a paravirt flushing interface + from guest to host. + + If unsure, say M. + config VIRTIO_BALLOON tristate "Virtio balloon driver" depends on VIRTIO diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 3a2b5c5..cbe91c6 100644 --- a/drivers/virtio/Makefile +++ b/drivers/virtio/Makefile @@ -6,3 +6,4 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o +obj-$(CONFIG_VIRTIO_PMEM) += virtio_pmem.o diff --git a/drivers/virtio/virtio_pmem.c b/drivers/virtio/virtio_pmem.c new file mode 100644 index 0000000..0906d2d --- /dev/null +++ b/drivers/virtio/virtio_pmem.c @@ -0,0 +1,118 @@ +/* Virtio pmem Driver + * + * Discovers persitent memory range information + * from host and provides a virtio based flushing + * interface. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int init_vq(struct virtio_pmem *vpmem) +{ + struct virtqueue *vq; + + /* single vq */ + vpmem->req_vq = vq = virtio_find_single_vq(vpmem->vdev, + NULL, "flush_queue"); + + if (IS_ERR(vq)) + return PTR_ERR(vq); + + return 0; +}; + +static int virtio_pmem_probe(struct virtio_device *vdev) +{ + int err = 0; + struct resource res; + struct virtio_pmem *vpmem; + struct nvdimm_bus *nvdimm_bus; + struct nd_region_desc ndr_desc; + int nid = dev_to_node(&vdev->dev); + static struct nvdimm_bus_descriptor nd_desc; + + if (!vdev->config->get) { + dev_err(&vdev->dev, "%s failure: config disabled\n", + __func__); + return -EINVAL; + } + + vdev->priv = vpmem = devm_kzalloc(&vdev->dev, sizeof(*vpmem), + GFP_KERNEL); + if (!vpmem) { + err = -ENOMEM; + goto out; + } + + vpmem->vdev = vdev; + err = init_vq(vpmem); + if (err) + goto out; + + virtio_cread(vpmem->vdev, struct virtio_pmem_config, + start, &vpmem->start); + virtio_cread(vpmem->vdev, struct virtio_pmem_config, + size, &vpmem->size); + + res.start = vpmem->start; + res.end = vpmem->start + vpmem->size-1; + + memset(&nd_desc, 0, sizeof(nd_desc)); + nd_desc.provider_name = "virtio-pmem"; + nd_desc.module = THIS_MODULE; + nvdimm_bus = nvdimm_bus_register(&vdev->dev, &nd_desc); + + if (!nvdimm_bus) + goto out_nd; + dev_set_drvdata(&vdev->dev, nvdimm_bus); + + memset(&ndr_desc, 0, sizeof(ndr_desc)); + ndr_desc.res = &res; + ndr_desc.numa_node = nid; + set_bit(ND_REGION_PAGEMAP, &ndr_desc.flags); + set_bit(ND_REGION_VIRTIO, &ndr_desc.flags); + + if (!nvdimm_pmem_region_create(nvdimm_bus, &ndr_desc)) + goto out_nd; + + virtio_device_ready(vdev); + return 0; + +out_nd: + nvdimm_bus_unregister(nvdimm_bus); +out: + dev_err(&vdev->dev, "failed to register virtio pmem memory\n"); + vdev->config->del_vqs(vdev); + return err; +} + +static void virtio_pmem_remove(struct virtio_device *vdev) +{ + struct nvdimm_bus *nvdimm_bus = dev_get_drvdata(&vdev->dev); + + nvdimm_bus_unregister(nvdimm_bus); + vdev->config->del_vqs(vdev); +} + +static struct virtio_driver virtio_pmem_driver = { + .driver.name = KBUILD_MODNAME, + .driver.owner = THIS_MODULE, + .id_table = id_table, + .probe = virtio_pmem_probe, + .remove = virtio_pmem_remove, +}; + +module_virtio_driver(virtio_pmem_driver); +MODULE_DEVICE_TABLE(virtio, id_table); +MODULE_DESCRIPTION("Virtio pmem driver"); +MODULE_LICENSE("GPL"); diff --git a/include/linux/libnvdimm.h b/include/linux/libnvdimm.h index 097072c..b1b7f14 100644 --- a/include/linux/libnvdimm.h +++ b/include/linux/libnvdimm.h @@ -58,6 +58,10 @@ enum { * (ADR) */ ND_REGION_PERSIST_MEMCTRL = 2, + /* + * region flag indicating to use VIRTIO flush interface for pmem + */ + ND_REGION_VIRTIO = 3, /* mark newly adjusted resources as requiring a label update */ DPA_RESOURCE_ADJUSTED = 1 << 0, diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h index 6d5c3b2..5ebd049 100644 --- a/include/uapi/linux/virtio_ids.h +++ b/include/uapi/linux/virtio_ids.h @@ -43,5 +43,6 @@ #define VIRTIO_ID_INPUT 18 /* virtio input */ #define VIRTIO_ID_VSOCK 19 /* virtio vsock transport */ #define VIRTIO_ID_CRYPTO 20 /* virtio crypto */ +#define VIRTIO_ID_PMEM 21 /* virtio pmem */ #endif /* _LINUX_VIRTIO_IDS_H */ diff --git a/include/uapi/linux/virtio_pmem.h b/include/uapi/linux/virtio_pmem.h new file mode 100644 index 0000000..2ec27cb --- /dev/null +++ b/include/uapi/linux/virtio_pmem.h @@ -0,0 +1,58 @@ +/* Virtio pmem Driver + * + * Discovers persitent memory range information + * from host and provides a virtio based flushing + * interface. + */ + +#ifndef _LINUX_VIRTIO_PMEM_H +#define _LINUX_VIRTIO_PMEM_H + +#include +#include +#include +#include +#include + + +struct virtio_pmem_config { + + uint64_t start; + uint64_t size; +}; + +struct virtio_pmem { + + struct virtio_device *vdev; + struct virtqueue *req_vq; + + uint64_t start; + uint64_t size; +} __packed; + +static struct virtio_device_id id_table[] = { + { VIRTIO_ID_PMEM, VIRTIO_DEV_ANY_ID }, + { 0 }, +}; + +void virtio_pmem_flush(struct device *dev) +{ + struct scatterlist sg; + struct virtio_device *vdev = dev_to_virtio(dev->parent->parent); + struct virtio_pmem *vpmem = vdev->priv; + char *buf = "FLUSH"; + int err; + + sg_init_one(&sg, buf, sizeof(buf)); + + err = virtqueue_add_outbuf(vpmem->req_vq, &sg, 1, buf, GFP_KERNEL); + + if (err) { + dev_err(&vdev->dev, "failed to send command to virtio pmem device\n"); + return; + } + + virtqueue_kick(vpmem->req_vq); +}; + +#endif From patchwork Wed Apr 25 11:24:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pankaj Gupta X-Patchwork-Id: 904144 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ131yFrz9s0t for ; Wed, 25 Apr 2018 21:31:03 +1000 (AEST) Received: from localhost ([::1]:35851 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIdY-0005EJ-BU for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:31:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33190) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIXT-0000HJ-87 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIXR-0001Ab-2Z for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:46092 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIXQ-0001AF-Uc for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:24:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 849ED406C757; Wed, 25 Apr 2018 11:24:40 +0000 (UTC) Received: from dhcp201-121.englab.pnq.redhat.com (dhcp193-127.pnq.redhat.com [10.65.193.127]) by smtp.corp.redhat.com (Postfix) with ESMTP id AC3FE2023239; Wed, 25 Apr 2018 11:24:34 +0000 (UTC) From: Pankaj Gupta To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, qemu-devel@nongnu.org, linux-nvdimm@ml01.01.org, linux-mm@kvack.org Date: Wed, 25 Apr 2018 16:54:14 +0530 Message-Id: <20180425112415.12327-3-pagupta@redhat.com> In-Reply-To: <20180425112415.12327-1-pagupta@redhat.com> References: <20180425112415.12327-1-pagupta@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:24:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:24:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pagupta@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [RFC v2 2/2] pmem: device flush over VIRTIO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, haozhong.zhang@intel.com, jack@suse.cz, xiaoguangrong.eric@gmail.com, riel@surriel.com, pagupta@redhat.com, niteshnarayanlal@hotmail.com, david@redhat.com, ross.zwisler@intel.com, lcapitulino@redhat.com, hch@infradead.org, mst@redhat.com, stefanha@redhat.com, imammedo@redhat.com, marcel@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, nilal@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds functionality to perform flush from guest to hosy over VIRTIO when 'ND_REGION_VIRTIO'flag is set on nd_negion. Flag is set by 'virtio-pmem' driver. Signed-off-by: Pankaj Gupta --- drivers/nvdimm/region_devs.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c index a612be6..6c6454e 100644 --- a/drivers/nvdimm/region_devs.c +++ b/drivers/nvdimm/region_devs.c @@ -20,6 +20,7 @@ #include #include "nd-core.h" #include "nd.h" +#include /* * For readq() and writeq() on 32-bit builds, the hi-lo, lo-hi order is @@ -1074,6 +1075,12 @@ void nvdimm_flush(struct nd_region *nd_region) struct nd_region_data *ndrd = dev_get_drvdata(&nd_region->dev); int i, idx; + /* call PV device flush */ + if (test_bit(ND_REGION_VIRTIO, &nd_region->flags)) { + virtio_pmem_flush(&nd_region->dev); + return; + } + /* * Try to encourage some diversity in flush hint addresses * across cpus assuming a limited number of flush hints.