| Message ID | 1392757259.28946.46.camel@haakon3.risingtidesystems.com |
|---|---|
| State | New |
| Headers | show |
On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > <SNIP> > > > > > > Hi Yan, > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > capable of (~500K). > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > appears to be the default setting for the driver included in the offical > > > > > virtio-win iso builds, right..? > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > within vioscsi.c code to start..? > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > miniport code..? > > > > > > Hi Nicholas, > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > MSI mode. > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > how things go.. > > > > Just a quick update on progress. > > I've been able to successfully build + load a unsigned vioscsi.sys > driver on Server 2012 with WDK 8.0. > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > performance and efficiency gain, on the order of 100K to 225K IOPs for > 4K block random I/O workload, depending on read/write mix. > One other performance related question.. In vioscsi.c:VioScsiFindAdapter() code, the default setting for adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when indirect mode is enabled in the following bits: if(adaptExt->indirect) { adaptExt->queue_depth = max(2, (pageNum / 4)); } else { adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; } Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default setting for ->queue_depth appears to be 128 (pageNum): #if (INDIRECT_SUPPORTED) if(!adaptExt->dump_mode) { adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC); } if(adaptExt->indirect) { adaptExt->queue_depth = pageNum; } #else adaptExt->indirect = 0; #endif Is there a reason for the lower queue_depth for vioscsi vs. viostor..? How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? Thanks! -nab
On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > <SNIP> > > > > > > > > Hi Yan, > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > capable of (~500K). > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > appears to be the default setting for the driver included in the offical > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > miniport code..? > > > > > > > > Hi Nicholas, > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > MSI mode. > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > how things go.. > > > > > > > Just a quick update on progress. > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > driver on Server 2012 with WDK 8.0. > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > 4K block random I/O workload, depending on read/write mix. > > > > One other performance related question.. > > In vioscsi.c:VioScsiFindAdapter() code, the default setting for > adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when > indirect mode is enabled in the following bits: > > if(adaptExt->indirect) { > adaptExt->queue_depth = max(2, (pageNum / 4)); > } else { > adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; > } > > Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default > setting for ->queue_depth appears to be 128 (pageNum): > > #if (INDIRECT_SUPPORTED) > if(!adaptExt->dump_mode) { > adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC); > } > if(adaptExt->indirect) { > adaptExt->queue_depth = pageNum; > } > #else > adaptExt->indirect = 0; > #endif > > Is there a reason for the lower queue_depth for vioscsi vs. viostor..? It's a horrible work around for the following bug: https://bugzilla.redhat.com/show_bug.cgi?id=1013443 I'm going to remove it as soon as found better solution for it. Best regards, Vadim. > > How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? > > Thanks! > > -nab > >
On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > <SNIP> > > > > > > Hi Yan, > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > capable of (~500K). > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > appears to be the default setting for the driver included in the offical > > > > > virtio-win iso builds, right..? > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > within vioscsi.c code to start..? > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > miniport code..? > > > > > > Hi Nicholas, > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > MSI mode. > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > how things go.. > > > > Just a quick update on progress. > > I've been able to successfully build + load a unsigned vioscsi.sys > driver on Server 2012 with WDK 8.0. > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > performance and efficiency gain, on the order of 100K to 225K IOPs for > 4K block random I/O workload, depending on read/write mix. > > Below is a simple patch to enable MSI operation by default. Any chance > to apply this separate from future mq efforts..? Yes, we differently can enable MSI and rebuild vioscsi. But then we need to re-spin WHQL testing for this particular driver. This process requires a lot of resources, and I doubt that it will be initiated soon, unless we have some significant amount of bug-fixes. Best regards, Vadim. > > Thanks, > > --nab > > From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001 > From: Nicholas Bellinger <nab@linux-iscsi.org> > Date: Fri, 14 Feb 2014 10:26:04 -0800 > Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default > > Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org> > --- > vioscsi/SOURCES | 2 +- > vioscsi/vioscsi.c | 2 -- > vioscsi/vioscsi.inx | 2 +- > vioscsi/vioscsi.vcxproj | 6 +++--- > 4 files changed, 5 insertions(+), 7 deletions(-) > > diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES > index f2083de..f631bd2 100644 > --- a/vioscsi/SOURCES > +++ b/vioscsi/SOURCES > @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES) > C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES) > C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES) > > -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES) > +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES) > C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES) > TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib > > diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c > index 77c0e46..70b9bb4 100644 > --- a/vioscsi/vioscsi.c > +++ b/vioscsi/vioscsi.c > @@ -337,8 +337,6 @@ ENTER_FN(); > adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; > } > > - adaptExt->msix_enabled = FALSE; > - > RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", > ConfigInfo->NumberOfPhysicalBreaks, > adaptExt->queue_depth)); > diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx > index cc94b7c..ec717c6 100644 > --- a/vioscsi/vioscsi.inx > +++ b/vioscsi/vioscsi.inx > @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001 > [pnpsafe_pci_addreg_msix] > HKR, "Interrupt Management",, 0x00000010 > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1 > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > HKR, "Interrupt Management\Affinity Policy",, 0x00000010 > HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5 > diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj > index 889e513..68d4e85 100644 > --- a/vioscsi/vioscsi.vcxproj > +++ b/vioscsi/vioscsi.vcxproj > @@ -66,7 +66,7 @@ > <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'"> > <ClCompile> > <WarningLevel>Level3</WarningLevel> > - <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> > + <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> > </ClCompile> > <Link> > <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> > @@ -95,7 +95,7 @@ > <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'"> > <ClCompile> > <WarningLevel>Level3</WarningLevel> > - <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> > + <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> > </ClCompile> > <Link> > <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> > @@ -140,4 +140,4 @@ > <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> > <ImportGroup Label="ExtensionTargets"> > </ImportGroup> > -</Project> > \ No newline at end of file > +</Project>
On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote: > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > <SNIP> > > > > > > > > Hi Yan, > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > capable of (~500K). > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > appears to be the default setting for the driver included in the offical > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > miniport code..? > > > > > > > > Hi Nicholas, > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > MSI mode. > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > how things go.. > > > > > > > Just a quick update on progress. > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > driver on Server 2012 with WDK 8.0. > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > 4K block random I/O workload, depending on read/write mix. > > > > Below is a simple patch to enable MSI operation by default. Any chance > > to apply this separate from future mq efforts..? > > Yes, we differently can enable MSI and rebuild vioscsi. > But then we need to re-spin WHQL testing for this particular > driver. This process requires a lot of resources, and I doubt that > it will be initiated soon, unless we have some significant amount of > bug-fixes. > Any idea on a rough time frame to expect an official WHQL build with MSI enabled..? Or, would it be possible to generate some -BETA builds that are at least signed and don't require extra hoops to jump through for testing..? Thanks again, --nab
On Wed, 2014-02-19 at 15:25 -0800, Nicholas A. Bellinger wrote: > On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote: > > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > > > <SNIP> > > > > > > > > > > Hi Yan, > > > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > > capable of (~500K). > > > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > > appears to be the default setting for the driver included in the offical > > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > > miniport code..? > > > > > > > > > > Hi Nicholas, > > > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > > MSI mode. > > > > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > > how things go.. > > > > > > > > > > Just a quick update on progress. > > > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > > driver on Server 2012 with WDK 8.0. > > > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > > 4K block random I/O workload, depending on read/write mix. > > > > > > Below is a simple patch to enable MSI operation by default. Any chance > > > to apply this separate from future mq efforts..? > > > > Yes, we differently can enable MSI and rebuild vioscsi. > > But then we need to re-spin WHQL testing for this particular > > driver. This process requires a lot of resources, and I doubt that > > it will be initiated soon, unless we have some significant amount of > > bug-fixes. > > > > Any idea on a rough time frame to expect an official WHQL build with MSI > enabled..? In June for sure :) > Or, would it be possible to generate some -BETA builds that are at least > signed and don't require extra hoops to jump through for testing..? It's doable. I hope we can make a new build next week. Best regards, Vadim. > > Thanks again, > > --nab > >
diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES index f2083de..f631bd2 100644 --- a/vioscsi/SOURCES +++ b/vioscsi/SOURCES @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES) -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES) +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES) C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES) TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c index 77c0e46..70b9bb4 100644 --- a/vioscsi/vioscsi.c +++ b/vioscsi/vioscsi.c @@ -337,8 +337,6 @@ ENTER_FN(); adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; } - adaptExt->msix_enabled = FALSE; - RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", ConfigInfo->NumberOfPhysicalBreaks, adaptExt->queue_depth)); diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx index cc94b7c..ec717c6 100644 --- a/vioscsi/vioscsi.inx +++ b/vioscsi/vioscsi.inx @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001 [pnpsafe_pci_addreg_msix] HKR, "Interrupt Management",, 0x00000010 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 HKR, "Interrupt Management\Affinity Policy",, 0x00000010 HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5 diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj index 889e513..68d4e85 100644 --- a/vioscsi/vioscsi.vcxproj +++ b/vioscsi/vioscsi.vcxproj @@ -66,7 +66,7 @@ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'"> <ClCompile> <WarningLevel>Level3</WarningLevel> - <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> <Link> <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> @@ -95,7 +95,7 @@ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'"> <ClCompile> <WarningLevel>Level3</WarningLevel> - <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> <Link> <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> @@ -140,4 +140,4 @@ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <ImportGroup Label="ExtensionTargets"> </ImportGroup> -</Project> \ No newline at end of file +</Project>