| Message ID | 1551753295-30167-1-git-send-email-yi.y.sun@linux.intel.com |
|---|---|
| Headers | show |
| Series | intel_iommu: support scalable mode | expand |
On Tue, Mar 05, 2019 at 10:34:52AM +0800, Yi Sun wrote: > Intel vt-d rev3.0 [1] introduces a new translation mode called > 'scalable mode', which enables PASID-granular translations for > first level, second level, nested and pass-through modes. The > vt-d scalable mode is the key ingredient to enable Scalable I/O > Virtualization (Scalable IOV) [2] [3], which allows sharing a > device in minimal possible granularity (ADI - Assignable Device > Interface). As a result, previous Extended Context (ECS) mode > is deprecated (no production ever implements ECS). > > This patch set emulates a minimal capability set of VT-d scalable > mode, equivalent to what is available in VT-d legacy mode today: > 1. Scalable mode root entry, context entry and PASID table > 2. Seconds level translation under scalable mode > 3. Queued invalidation (with 256 bits descriptor) > 4. Pass-through mode > > Corresponding intel-iommu driver support will be included in > kernel 5.0: > https://www.spinics.net/lists/kernel/msg2985279.html > > We will add emulation of full scalable mode capability along with > guest iommu driver progress later, e.g.: > 1. First level translation > 2. Nested translation > 3. Per-PASID invalidation descriptors > 4. Page request services for handling recoverable faults > > To verify the patches, below cases were tested according to Peter Xu's > suggestions. > +---------+----------------------------------------------------------------+----------------------------------------------------------------+ > | | w/ Device Passthr | w/o Device Passthr | > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ Legacy vfio-pci? I've reviewed the whole series, I would assume that the maintainer might still test it a bit before a pull but again even before that I would really like to double confirm this series won't break anything. Thanks,
On Tue, Mar 05, 2019 at 11:09:34AM +0800, Peter Xu wrote: > On Tue, Mar 05, 2019 at 10:34:52AM +0800, Yi Sun wrote: > > Intel vt-d rev3.0 [1] introduces a new translation mode called > > 'scalable mode', which enables PASID-granular translations for > > first level, second level, nested and pass-through modes. The > > vt-d scalable mode is the key ingredient to enable Scalable I/O > > Virtualization (Scalable IOV) [2] [3], which allows sharing a > > device in minimal possible granularity (ADI - Assignable Device > > Interface). As a result, previous Extended Context (ECS) mode > > is deprecated (no production ever implements ECS). > > > > This patch set emulates a minimal capability set of VT-d scalable > > mode, equivalent to what is available in VT-d legacy mode today: > > 1. Scalable mode root entry, context entry and PASID table > > 2. Seconds level translation under scalable mode > > 3. Queued invalidation (with 256 bits descriptor) > > 4. Pass-through mode > > > > Corresponding intel-iommu driver support will be included in > > kernel 5.0: > > https://www.spinics.net/lists/kernel/msg2985279.html > > > > We will add emulation of full scalable mode capability along with > > guest iommu driver progress later, e.g.: > > 1. First level translation > > 2. Nested translation > > 3. Per-PASID invalidation descriptors > > 4. Page request services for handling recoverable faults > > > > To verify the patches, below cases were tested according to Peter Xu's > > suggestions. > > +---------+----------------------------------------------------------------+----------------------------------------------------------------+ > > | | w/ Device Passthr | w/o Device Passthr | > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > Legacy vfio-pci? > > I've reviewed the whole series, I would assume that the maintainer > might still test it a bit before a pull but again even before that I > would really like to double confirm this series won't break anything. > > Thanks, If that's me FYI I don't normally test with vfio either. Yes I'd appreciate testing with that from contributor. > -- > Peter Xu
On 19-03-05 11:09:34, Peter Xu wrote: > On Tue, Mar 05, 2019 at 10:34:52AM +0800, Yi Sun wrote: > > Intel vt-d rev3.0 [1] introduces a new translation mode called > > 'scalable mode', which enables PASID-granular translations for > > first level, second level, nested and pass-through modes. The > > vt-d scalable mode is the key ingredient to enable Scalable I/O > > Virtualization (Scalable IOV) [2] [3], which allows sharing a > > device in minimal possible granularity (ADI - Assignable Device > > Interface). As a result, previous Extended Context (ECS) mode > > is deprecated (no production ever implements ECS). > > > > This patch set emulates a minimal capability set of VT-d scalable > > mode, equivalent to what is available in VT-d legacy mode today: > > 1. Scalable mode root entry, context entry and PASID table > > 2. Seconds level translation under scalable mode > > 3. Queued invalidation (with 256 bits descriptor) > > 4. Pass-through mode > > > > Corresponding intel-iommu driver support will be included in > > kernel 5.0: > > https://www.spinics.net/lists/kernel/msg2985279.html > > > > We will add emulation of full scalable mode capability along with > > guest iommu driver progress later, e.g.: > > 1. First level translation > > 2. Nested translation > > 3. Per-PASID invalidation descriptors > > 4. Page request services for handling recoverable faults > > > > To verify the patches, below cases were tested according to Peter Xu's > > suggestions. > > +---------+----------------------------------------------------------------+----------------------------------------------------------------+ > > | | w/ Device Passthr | w/o Device Passthr | > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > Legacy vfio-pci? > I tested legacy mode with below command. w/o passthr -device intel-iommu \ -netdev tap,id=net1,vhost=off|on \ -device virtio-net-pci,netdev=net1 \ w/ passthr: tried net card, graphic card, and a MDEV (cmd is different) -device intel-iommu,caching-mode=on \ -netdev tap,id=net1,vhost=off|on \ -device virtio-net-pci,netdev=net1 \ -device vfio-pci,host=0000:00:1f.6 \ > I've reviewed the whole series, I would assume that the maintainer > might still test it a bit before a pull but again even before that I > would really like to double confirm this series won't break anything. > > Thanks, > > -- > Peter Xu
On Tue, Mar 05, 2019 at 11:24:24AM +0800, Yi Sun wrote: > On 19-03-05 11:09:34, Peter Xu wrote: > > On Tue, Mar 05, 2019 at 10:34:52AM +0800, Yi Sun wrote: > > > Intel vt-d rev3.0 [1] introduces a new translation mode called > > > 'scalable mode', which enables PASID-granular translations for > > > first level, second level, nested and pass-through modes. The > > > vt-d scalable mode is the key ingredient to enable Scalable I/O > > > Virtualization (Scalable IOV) [2] [3], which allows sharing a > > > device in minimal possible granularity (ADI - Assignable Device > > > Interface). As a result, previous Extended Context (ECS) mode > > > is deprecated (no production ever implements ECS). > > > > > > This patch set emulates a minimal capability set of VT-d scalable > > > mode, equivalent to what is available in VT-d legacy mode today: > > > 1. Scalable mode root entry, context entry and PASID table > > > 2. Seconds level translation under scalable mode > > > 3. Queued invalidation (with 256 bits descriptor) > > > 4. Pass-through mode > > > > > > Corresponding intel-iommu driver support will be included in > > > kernel 5.0: > > > https://www.spinics.net/lists/kernel/msg2985279.html > > > > > > We will add emulation of full scalable mode capability along with > > > guest iommu driver progress later, e.g.: > > > 1. First level translation > > > 2. Nested translation > > > 3. Per-PASID invalidation descriptors > > > 4. Page request services for handling recoverable faults > > > > > > To verify the patches, below cases were tested according to Peter Xu's > > > suggestions. > > > +---------+----------------------------------------------------------------+----------------------------------------------------------------+ > > > | | w/ Device Passthr | w/o Device Passthr | > > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | > > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > Legacy vfio-pci? > > > I tested legacy mode with below command. > > w/o passthr > -device intel-iommu \ > -netdev tap,id=net1,vhost=off|on \ > -device virtio-net-pci,netdev=net1 \ Virtio devices require explicit enabling of DMAR: https://wiki.qemu.org/Features/VT-d#With_Virtio_Devices Otherwise it could probably still be bypassing the translation unit. If you retest any, please feel free to skip some of the tests like kernel build. Quick netperf should work, which can save you time. > > w/ passthr: tried net card, graphic card, and a MDEV (cmd is different) > -device intel-iommu,caching-mode=on \ > -netdev tap,id=net1,vhost=off|on \ > -device virtio-net-pci,netdev=net1 \ > -device vfio-pci,host=0000:00:1f.6 \ This one seems correct to me, and yes covering net/graphic cards should be far enough. Thanks,
On 19-03-05 12:48:36, Peter Xu wrote: > On Tue, Mar 05, 2019 at 11:24:24AM +0800, Yi Sun wrote: > > On 19-03-05 11:09:34, Peter Xu wrote: > > > On Tue, Mar 05, 2019 at 10:34:52AM +0800, Yi Sun wrote: > > > > Intel vt-d rev3.0 [1] introduces a new translation mode called > > > > 'scalable mode', which enables PASID-granular translations for > > > > first level, second level, nested and pass-through modes. The > > > > vt-d scalable mode is the key ingredient to enable Scalable I/O > > > > Virtualization (Scalable IOV) [2] [3], which allows sharing a > > > > device in minimal possible granularity (ADI - Assignable Device > > > > Interface). As a result, previous Extended Context (ECS) mode > > > > is deprecated (no production ever implements ECS). > > > > > > > > This patch set emulates a minimal capability set of VT-d scalable > > > > mode, equivalent to what is available in VT-d legacy mode today: > > > > 1. Scalable mode root entry, context entry and PASID table > > > > 2. Seconds level translation under scalable mode > > > > 3. Queued invalidation (with 256 bits descriptor) > > > > 4. Pass-through mode > > > > > > > > Corresponding intel-iommu driver support will be included in > > > > kernel 5.0: > > > > https://www.spinics.net/lists/kernel/msg2985279.html > > > > > > > > We will add emulation of full scalable mode capability along with > > > > guest iommu driver progress later, e.g.: > > > > 1. First level translation > > > > 2. Nested translation > > > > 3. Per-PASID invalidation descriptors > > > > 4. Page request services for handling recoverable faults > > > > > > > > To verify the patches, below cases were tested according to Peter Xu's > > > > suggestions. > > > > +---------+----------------------------------------------------------------+----------------------------------------------------------------+ > > > > | | w/ Device Passthr | w/o Device Passthr | > > > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | > > > > | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | > > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | > > > > +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ > > > > > > Legacy vfio-pci? > > > > > I tested legacy mode with below command. > > > > w/o passthr > > -device intel-iommu \ > > -netdev tap,id=net1,vhost=off|on \ > > -device virtio-net-pci,netdev=net1 \ > > Virtio devices require explicit enabling of DMAR: > > https://wiki.qemu.org/Features/VT-d#With_Virtio_Devices > > Otherwise it could probably still be bypassing the translation unit. > > If you retest any, please feel free to skip some of the tests like > kernel build. Quick netperf should work, which can save you time. > Thanks! Just made a quick test. It works well. Please check below command. -device intel-iommu,intremap=on,device-iotlb=on \ -netdev tap,id=net1,vhost=off \ -device ioh3420,id=pcie.1,chassis=1 \ -device virtio-net-pci,bus=pcie.1,netdev=net1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on \ > > > > w/ passthr: tried net card, graphic card, and a MDEV (cmd is different) > > -device intel-iommu,caching-mode=on \ > > -netdev tap,id=net1,vhost=off|on \ > > -device virtio-net-pci,netdev=net1 \ > > -device vfio-pci,host=0000:00:1f.6 \ > > This one seems correct to me, and yes covering net/graphic cards > should be far enough. > > Thanks, > > -- > Peter Xu
On Tue, Mar 05, 2019 at 01:15:53PM +0800, Yi Sun wrote: [...] > > > w/o passthr > > > -device intel-iommu \ > > > -netdev tap,id=net1,vhost=off|on \ > > > -device virtio-net-pci,netdev=net1 \ > > > > Virtio devices require explicit enabling of DMAR: > > > > https://wiki.qemu.org/Features/VT-d#With_Virtio_Devices > > > > Otherwise it could probably still be bypassing the translation unit. > > > > If you retest any, please feel free to skip some of the tests like > > kernel build. Quick netperf should work, which can save you time. > > > Thanks! Just made a quick test. It works well. Please check below > command. > > -device intel-iommu,intremap=on,device-iotlb=on \ > -netdev tap,id=net1,vhost=off \ > -device ioh3420,id=pcie.1,chassis=1 \ > -device > virtio-net-pci,bus=pcie.1,netdev=net1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on > \ Please also consider to try vhost=on with/without iommu=pt. Regards,
On 19-03-05 13:36:12, Peter Xu wrote: > On Tue, Mar 05, 2019 at 01:15:53PM +0800, Yi Sun wrote: > > [...] > > > > > w/o passthr > > > > -device intel-iommu \ > > > > -netdev tap,id=net1,vhost=off|on \ > > > > -device virtio-net-pci,netdev=net1 \ > > > > > > Virtio devices require explicit enabling of DMAR: > > > > > > https://wiki.qemu.org/Features/VT-d#With_Virtio_Devices > > > > > > Otherwise it could probably still be bypassing the translation unit. > > > > > > If you retest any, please feel free to skip some of the tests like > > > kernel build. Quick netperf should work, which can save you time. > > > > > Thanks! Just made a quick test. It works well. Please check below > > command. > > > > -device intel-iommu,intremap=on,device-iotlb=on \ > > -netdev tap,id=net1,vhost=off \ > > -device ioh3420,id=pcie.1,chassis=1 \ > > -device > > virtio-net-pci,bus=pcie.1,netdev=net1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on > > \ > > Please also consider to try vhost=on with/without iommu=pt. > Just tested vhost=on/off with/without iommu=pt for legacy mode. All pass. > Regards, > > -- > Peter Xu
On Tue, Mar 05, 2019 at 02:27:48PM +0800, Yi Sun wrote: > On 19-03-05 13:36:12, Peter Xu wrote: > > On Tue, Mar 05, 2019 at 01:15:53PM +0800, Yi Sun wrote: > > > > [...] > > > > > > > w/o passthr > > > > > -device intel-iommu \ > > > > > -netdev tap,id=net1,vhost=off|on \ > > > > > -device virtio-net-pci,netdev=net1 \ > > > > > > > > Virtio devices require explicit enabling of DMAR: > > > > > > > > https://wiki.qemu.org/Features/VT-d#With_Virtio_Devices > > > > > > > > Otherwise it could probably still be bypassing the translation unit. > > > > > > > > If you retest any, please feel free to skip some of the tests like > > > > kernel build. Quick netperf should work, which can save you time. > > > > > > > Thanks! Just made a quick test. It works well. Please check below > > > command. > > > > > > -device intel-iommu,intremap=on,device-iotlb=on \ > > > -netdev tap,id=net1,vhost=off \ > > > -device ioh3420,id=pcie.1,chassis=1 \ > > > -device > > > virtio-net-pci,bus=pcie.1,netdev=net1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on > > > \ > > > > Please also consider to try vhost=on with/without iommu=pt. > > > Just tested vhost=on/off with/without iommu=pt for legacy mode. All pass. Great, thanks.
Intel vt-d rev3.0 [1] introduces a new translation mode called 'scalable mode', which enables PASID-granular translations for first level, second level, nested and pass-through modes. The vt-d scalable mode is the key ingredient to enable Scalable I/O Virtualization (Scalable IOV) [2] [3], which allows sharing a device in minimal possible granularity (ADI - Assignable Device Interface). As a result, previous Extended Context (ECS) mode is deprecated (no production ever implements ECS). This patch set emulates a minimal capability set of VT-d scalable mode, equivalent to what is available in VT-d legacy mode today: 1. Scalable mode root entry, context entry and PASID table 2. Seconds level translation under scalable mode 3. Queued invalidation (with 256 bits descriptor) 4. Pass-through mode Corresponding intel-iommu driver support will be included in kernel 5.0: https://www.spinics.net/lists/kernel/msg2985279.html We will add emulation of full scalable mode capability along with guest iommu driver progress later, e.g.: 1. First level translation 2. Nested translation 3. Per-PASID invalidation descriptors 4. Page request services for handling recoverable faults To verify the patches, below cases were tested according to Peter Xu's suggestions. +---------+----------------------------------------------------------------+----------------------------------------------------------------+ | | w/ Device Passthr | w/o Device Passthr | | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ | | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | virtio-net-pci, vhost=on | virtio-net-pci, vhost=off | | +-------------------------------+--------------------------------+-------------------------------+--------------------------------+ | | netperf | kernel bld | data cp| netperf | kernel bld | data cp | netperf | kernel bld | data cp| netperf | kernel bld | data cp | +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ | Legacy | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ | Scalable| Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | +---------+-------------------------------+--------------------------------+-------------------------------+--------------------------------+ References: [1] https://software.intel.com/en-us/download/intel-virtualization-technology-for-directed-io-architecture-specification [2] https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification [3] https://schd.ws/hosted_files/lc32018/00/LC3-SIOV-final.pdf --- RFC v2->v1 Patch 1: - revert unnecessary rename in 'vtd_ce_present". - use 'ce_size' to judge if it is scalable mode in 'vtd_get_context_entry_from_root'. - remove some 'inline' declarations. - remove unnecessary scalable mode check in 'vtd_ce_type_check' and make corresponding changes in 'vtd_dev_to_context_entry'. - add 'VTD_FR_PASID_TABLE_INV' case in 'vtd_qualified_faults[]'. - create a macro function to handle a repeated pattern about 'is_fpd_set'. - explicitly print 'legacy mode' when calling 'trace_vtd_replay_ce_valid'. Patch 2: - do "(val & VTD_IQT_QT_256_RSV_BIT)" to be clear. Patch 3: - move 's->dma_drain' check from 'vtd_init' to 'vtd_decide_config'. --- Liu, Yi L (2): intel_iommu: scalable mode emulation intel_iommu: add 256 bits qi_desc support Yi Sun (1): intel_iommu: add scalable-mode option to make scalable mode work hw/i386/intel_iommu.c | 559 +++++++++++++++++++++++++++++++++-------- hw/i386/intel_iommu_internal.h | 54 +++- hw/i386/trace-events | 2 +- include/hw/i386/intel_iommu.h | 28 ++- 4 files changed, 533 insertions(+), 110 deletions(-)