| Message ID | 1472804172-25542-5-git-send-email-jike.song@intel.com |
|---|---|
| State | New |
| Headers | show |
On 09/02/2016 03:16 AM, Jike Song wrote: > From: Kirti Wankhede <kwankhede@nvidia.com> > > Add file Documentation/vfio-mediated-device.txt that include details of > mediated device framework. > > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com> > Signed-off-by: Neo Jia <cjia@nvidia.com> > Signed-off-by: Jike Song <jike.song@intel.com> > --- > Documentation/vfio-mediated-device.txt | 203 +++++++++++++++++++++++++++++++++ > 1 file changed, 203 insertions(+) > create mode 100644 Documentation/vfio-mediated-device.txt > > diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt > new file mode 100644 > index 0000000..39bdcd9 > --- /dev/null > +++ b/Documentation/vfio-mediated-device.txt > @@ -0,0 +1,203 @@ > +VFIO Mediated devices [1] > +------------------------------------------------------------------------------- Many files under Documentation trim the ---- decorator to the length of the line above. Also, since you have no explicit copyright/license notice, your documentation is under GPLv2+ per the top level. Other files do this, and if you are okay with it, I won't complain; but if you intended something else, or even if you wanted to make it explicit rather than implicit, then you may want to copy the example of files that call out a quick blurb on copyright and licensing. > + > +There are more and more use cases/demands to virtualize the DMA devices which > +doesn't have SR_IOV capability built-in. To do this, drivers of different s/doesn't/don't/ > +devices had to develop their own management interface and set of APIs and then > +integrate it to user space software. We've identified common requirements and > +unified management interface for such devices to make user space software > +integration easier. > + > +The VFIO driver framework provides unified APIs for direct device access. It is > +an IOMMU/device agnostic framework for exposing direct device access to > +user space, in a secure, IOMMU protected environment. This framework is > +used for multiple devices like GPUs, network adapters and compute accelerators. > +With direct device access, virtual machines or user space applications have > +direct access of physical device. This framework is reused for mediated devices. > + > +Mediated core driver provides a common interface for mediated device management > +that can be used by drivers of different devices. This module provides a generic > +interface to create/destroy mediated device, add/remove it to mediated bus s/mediated/a mediated/ twice > +driver, add/remove device to IOMMU group. It also provides an interface to s/add/and add/ s/device to/a device to an/ > +register different types of bus drivers, for example, Mediated VFIO PCI driver > +is designed for mediated PCI devices and supports VFIO APIs. Similarly, driver s/driver/the driver/ > +can be designed to support any type of mediated device and added to this > +framework. Mediated bus driver add/delete mediated device to VFIO Group. Missing a verb and several articles, but I'm not sure what you meant. Maybe: A mediated bus driver can add/delete mediated devices to a VFIO Group. > + > +Below is the high level block diagram, with NVIDIA, Intel and IBM devices > +as examples, since these are the devices which are going to actively use > +this module as of now. > + > + > + > +Registration Interfaces > +------------------------------------------------------------------------------- > + Again, rather long separator, > +Mediated core driver provides two types of registration interfaces: > + > +1. Registration interface for mediated bus driver: > +------------------------------------------------- while this seems one byte short. I'll quit pointing it out. > + > + /** > + * struct mdev_driver [2] - Mediated device driver > + * @name: driver name > + * @probe: called when new device created > + * @remove: called when device removed > + * @driver: device driver structure No mention of online, offline. > + **/ > + struct mdev_driver { > + const char *name; > + int (*probe)(struct device *dev); > + void (*remove)(struct device *dev); > + int (*online)(struct device *dev); > + int (*offline)(struct device *dev); > + struct device_driver driver; > + }; > + > + ... > + > +For echo physical device, there is a mdev host device created, it shows in sysfs: > +/sys/bus/pci/devices/0000:05:00.0/mdev-host/ Did you mean 'For echoing a' (as in duplicating the device), or maybe 'For echoing to a' (as in duplicating data sent to the device)? Or is this a typo s/echo/each/?
On Fri, Sep 02, 2016 at 05:09:46PM -0500, Eric Blake wrote: > * PGP Signed by an unknown key > > On 09/02/2016 03:16 AM, Jike Song wrote: > > From: Kirti Wankhede <kwankhede@nvidia.com> > > > > Add file Documentation/vfio-mediated-device.txt that include details of > > mediated device framework. > > > > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com> > > Signed-off-by: Neo Jia <cjia@nvidia.com> > > Signed-off-by: Jike Song <jike.song@intel.com> > > --- > > Documentation/vfio-mediated-device.txt | 203 +++++++++++++++++++++++++++++++++ > > 1 file changed, 203 insertions(+) > > create mode 100644 Documentation/vfio-mediated-device.txt > > > > diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt > > new file mode 100644 > > index 0000000..39bdcd9 > > --- /dev/null > > +++ b/Documentation/vfio-mediated-device.txt > > @@ -0,0 +1,203 @@ > > +VFIO Mediated devices [1] > > +------------------------------------------------------------------------------- > > Many files under Documentation trim the ---- decorator to the length of > the line above. > > Also, since you have no explicit copyright/license notice, your > documentation is under GPLv2+ per the top level. Other files do this, > and if you are okay with it, I won't complain; but if you intended > something else, or even if you wanted to make it explicit rather than > implicit, then you may want to copy the example of files that call out a > quick blurb on copyright and licensing. > Hi Eric, Thanks for the review and really sorry about the extra email threads of this review, we already have one actively going on for a while starting from RFC to currently v7. http://www.spinics.net/lists/kvm/msg137208.html And the related latest v7 document is at: http://www.spinics.net/lists/kvm/msg137210.html We will address all your review comments there. Thanks, Neo > > + > > +There are more and more use cases/demands to virtualize the DMA devices which > > +doesn't have SR_IOV capability built-in. To do this, drivers of different > > s/doesn't/don't/ > > > +devices had to develop their own management interface and set of APIs and then > > +integrate it to user space software. We've identified common requirements and > > +unified management interface for such devices to make user space software > > +integration easier. > > + > > +The VFIO driver framework provides unified APIs for direct device access. It is > > +an IOMMU/device agnostic framework for exposing direct device access to > > +user space, in a secure, IOMMU protected environment. This framework is > > +used for multiple devices like GPUs, network adapters and compute accelerators. > > +With direct device access, virtual machines or user space applications have > > +direct access of physical device. This framework is reused for mediated devices. > > + > > +Mediated core driver provides a common interface for mediated device management > > +that can be used by drivers of different devices. This module provides a generic > > +interface to create/destroy mediated device, add/remove it to mediated bus > > s/mediated/a mediated/ twice > > > +driver, add/remove device to IOMMU group. It also provides an interface to > > s/add/and add/ > s/device to/a device to an/ > > > +register different types of bus drivers, for example, Mediated VFIO PCI driver > > +is designed for mediated PCI devices and supports VFIO APIs. Similarly, driver > > s/driver/the driver/ > > > +can be designed to support any type of mediated device and added to this > > +framework. Mediated bus driver add/delete mediated device to VFIO Group. > > Missing a verb and several articles, but I'm not sure what you meant. > Maybe: > > A mediated bus driver can add/delete mediated devices to a VFIO Group. > > > + > > +Below is the high level block diagram, with NVIDIA, Intel and IBM devices > > +as examples, since these are the devices which are going to actively use > > +this module as of now. > > + > > > + > > + > > +Registration Interfaces > > +------------------------------------------------------------------------------- > > + > > Again, rather long separator, > > > +Mediated core driver provides two types of registration interfaces: > > + > > +1. Registration interface for mediated bus driver: > > +------------------------------------------------- > > while this seems one byte short. I'll quit pointing it out. > > > + > > + /** > > + * struct mdev_driver [2] - Mediated device driver > > + * @name: driver name > > + * @probe: called when new device created > > + * @remove: called when device removed > > + * @driver: device driver structure > > No mention of online, offline. > > > + **/ > > + struct mdev_driver { > > + const char *name; > > + int (*probe)(struct device *dev); > > + void (*remove)(struct device *dev); > > + int (*online)(struct device *dev); > > + int (*offline)(struct device *dev); > > + struct device_driver driver; > > + }; > > + > > + > ... > > + > > +For echo physical device, there is a mdev host device created, it shows in sysfs: > > +/sys/bus/pci/devices/0000:05:00.0/mdev-host/ > > Did you mean 'For echoing a' (as in duplicating the device), or maybe > 'For echoing to a' (as in duplicating data sent to the device)? Or is > this a typo s/echo/each/? > > > -- > Eric Blake eblake redhat com +1-919-301-3266 > Libvirt virtualization library http://libvirt.org > > > * Unknown Key > * 0x2527436A
diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt new file mode 100644 index 0000000..39bdcd9 --- /dev/null +++ b/Documentation/vfio-mediated-device.txt @@ -0,0 +1,203 @@ +VFIO Mediated devices [1] +------------------------------------------------------------------------------- + +There are more and more use cases/demands to virtualize the DMA devices which +doesn't have SR_IOV capability built-in. To do this, drivers of different +devices had to develop their own management interface and set of APIs and then +integrate it to user space software. We've identified common requirements and +unified management interface for such devices to make user space software +integration easier. + +The VFIO driver framework provides unified APIs for direct device access. It is +an IOMMU/device agnostic framework for exposing direct device access to +user space, in a secure, IOMMU protected environment. This framework is +used for multiple devices like GPUs, network adapters and compute accelerators. +With direct device access, virtual machines or user space applications have +direct access of physical device. This framework is reused for mediated devices. + +Mediated core driver provides a common interface for mediated device management +that can be used by drivers of different devices. This module provides a generic +interface to create/destroy mediated device, add/remove it to mediated bus +driver, add/remove device to IOMMU group. It also provides an interface to +register different types of bus drivers, for example, Mediated VFIO PCI driver +is designed for mediated PCI devices and supports VFIO APIs. Similarly, driver +can be designed to support any type of mediated device and added to this +framework. Mediated bus driver add/delete mediated device to VFIO Group. + +Below is the high level block diagram, with NVIDIA, Intel and IBM devices +as examples, since these are the devices which are going to actively use +this module as of now. + + + +---------------+ + | | + | +-----------+ | + | | | | + | | | | + | | mdev | | mdev_register_driver() +---------------+ + | | bus | |<------------------------------+ | + | | driver | | | | + | | | +------------------------------>| vfio_mdev.ko |<-> VFIO user + | | | | probe()/remove() | | APIs + | | | | +---------------+ + | | | | + | +-----------+ | + | | + | MDEV CORE | + | MODULE | + | mdev.ko | + | | + | +-----------+ | mdev_register_host_device() +---------------+ + | | | +<------------------------------+ | + | | | | | nvidia.ko |<-> physical + | | | +------------------------------>+ | device + | | | | callbacks +---------------+ + | | | | + | | Physical | | mdev_register_host_device() +---------------+ + | | Device | |<------------------------------+ | + | | Interface | | | i915.ko |<-> physical + | | | +------------------------------>+ | device + | | | | callbacks +---------------+ + | | | | + | | | | mdev_register_host_device() +---------------+ + | | | +<------------------------------+ | + | | | | | ccw_device.ko |<-> physical + | | | +------------------------------>+ | device + | | | | callbacks +---------------+ + | +-----------+ | + +---------------+ + + +Registration Interfaces +------------------------------------------------------------------------------- + +Mediated core driver provides two types of registration interfaces: + +1. Registration interface for mediated bus driver: +------------------------------------------------- + + /** + * struct mdev_driver [2] - Mediated device driver + * @name: driver name + * @probe: called when new device created + * @remove: called when device removed + * @driver: device driver structure + **/ + struct mdev_driver { + const char *name; + int (*probe)(struct device *dev); + void (*remove)(struct device *dev); + int (*online)(struct device *dev); + int (*offline)(struct device *dev); + struct device_driver driver; + }; + + + +Mediated bus driver for mdev should use this interface to register and +unregister with core driver respectively: + +extern int mdev_register_driver(struct mdev_driver *drv, struct module *owner); +extern void mdev_unregister_driver(struct mdev_driver *drv); + +Mediated bus driver is responsible to add/delete mediated devices to/from VFIO +group when devices are bound and unbound to the driver. + +2. Physical device driver interface: +----------------------------------- +This interface [3] provides a set of APIs to manage physical device related work +in its driver. APIs are: + +* hdev_attr_groups: attribute groups of the mdev host device. +* mdev_attr_groups: attribute groups of the mediated device. +* supported_config: to provide supported configuration list by the driver. +* create: to allocate basic resources in driver for a mediated device. +* destroy: to free resources in driver when mediated device is destroyed. +* start: to start the mediated device +* stop: to stop the mediated device +* read : read emulation callback. +* write: write emulation callback. +* mmap: the mmap callback +* ioctl: the ioctl callback + +Drivers should use this interface to register and unregister device to mdev core +driver respectively: + + struct mdev_host *mdev_register_host_device(struct device *pdev, + const struct mdev_host_ops *ops); + void mdev_unregister_device(struct mdev_host *host); + + +Mediated device management interface via sysfs +------------------------------------------------------------------------------- +This is the interface that allows user space software, like libvirt, to query +and configure mediated device in a HW agnostic fashion. This management +interface provide flexibility to underlying physical device's driver to support +mediated device hotplug, multiple mediated devices per virtual machine, multiple +mediated devices from different physical devices, etc. + +For echo physical device, there is a mdev host device created, it shows in sysfs: +/sys/bus/pci/devices/0000:05:00.0/mdev-host/ +--------------------------------- + +* mdev_supported_types: (read only) + List the current supported mediated device types and its details. + +* mdev_create: (write only) + Create a mediated device on target physical device. + Input syntax: <UUID:params> + where, + UUID: mediated device's UUID + params: extra parameters required by driver + Example: + # echo "12345678-1234-1234-1234-123456789abc" > + /sys/bus/pci/devices/0000\:05\:00.0/mdev-host/mdev_create + +* mdev_destroy: (write only) + Destroy a mediated device on a target physical device. + Input syntax: <UUID> + where, + UUID: mediated device's UUID + Example: + # echo "12345678-1234-1234-1234-123456789abc" > + /sys/bus/pci/devices/0000\:05\:00.0/mdev-host/mdev_destroy + +Under mdev sysfs directory: +/sys/bus/pci/devices/0000:05:00.0/mdev-host/12345678-1234-1234-1234-123456789abc/ +---------------------------------- + +* online: (read/write) + This shows and sets the online status of mdev. + Input syntax: <0|1> + To set mdev online, simply echo "1" to the online file; "0" to + offline the mdev. + Example: + # echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/mdev-host/12345678-1234-1234-1234-123456789abc/online + # echo 0 > /sys/bus/pci/devices/0000\:05\:00.0/mdev-host/12345678-1234-1234-1234-123456789abc/online + + +Translation APIs for Mediated device +------------------------------------------------------------------------------ + +Below APIs are provided for user pfn to host pfn translation in VFIO driver: + +extern long vfio_pin_pages(struct mdev_device *mdev, unsigned long *user_pfn, + long npage, int prot, unsigned long *phys_pfn); + +extern long vfio_unpin_pages(struct mdev_device *mdev, unsigned long *pfn, + long npage); + +These functions call back into the backend IOMMU module using two callbacks of +struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these are +supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend +modules, such as PPC64 sPAPR module, they need to provide these two callback +functions. + +References +------------------------------------------------------------------------------- + +[1] See Documentation/vfio.txt for more information on VFIO. +[2] struct mdev_driver in include/linux/mdev.h +[3] struct mdev_host_ops in include/linux/mdev.h +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h +