[qemu,v7,4/4] ppc/spapr, vfio: Turn off MSIX emulation for VFIO devices

This adds a possibility for the platform to tell VFIO not to emulate MSIX
so MMIO memory regions do not get split into chunks in flatview and
the entire page can be registered as a KVM memory slot and make direct
MMIO access possible for the guest.

This enables the entire MSIX BAR mapping to the guest for the pseries
platform in order to achieve the maximum MMIO preformance for certain
devices.

Tested on:
LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

Need to split this in two?
---
 hw/ppc/spapr.c |  7 +++++++
 hw/vfio/pci.c  | 13 +++++++++++++
 2 files changed, 20 insertions(+)

On Fri, Feb 09, 2018 at 06:55:03PM +1100, Alexey Kardashevskiy wrote:
> This adds a possibility for the platform to tell VFIO not to emulate MSIX
> so MMIO memory regions do not get split into chunks in flatview and
> the entire page can be registered as a KVM memory slot and make direct
> MMIO access possible for the guest.
> 
> This enables the entire MSIX BAR mapping to the guest for the pseries
> platform in order to achieve the maximum MMIO preformance for certain
> devices.
> 
> Tested on:
> LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

I still think having the property on the machine is an uglier option
than having it on the device, but Alex didn't like that, so I'll roll
with it.

Reivewed-by: David Gibson <david@gibson.dropbear.id.au>

> ---
> 
> Need to split this in two?
> ---
>  hw/ppc/spapr.c |  7 +++++++
>  hw/vfio/pci.c  | 13 +++++++++++++
>  2 files changed, 20 insertions(+)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index eb3e1a7..14d8ecb 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -2855,6 +2855,11 @@ static void spapr_set_modern_hotplug_events(Object *obj, bool value,
>      spapr->use_hotplug_event_source = value;
>  }
>  
> +static bool spapr_get_msix_emulation(Object *obj, Error **errp)
> +{
> +    return true;
> +}
> +
>  static char *spapr_get_resize_hpt(Object *obj, Error **errp)
>  {
>      sPAPRMachineState *spapr = SPAPR_MACHINE(obj);
> @@ -2936,6 +2941,8 @@ static void spapr_instance_init(Object *obj)
>      object_property_set_description(obj, "vsmt",
>                                      "Virtual SMT: KVM behaves as if this were"
>                                      " the host's SMT mode", &error_abort);
> +    object_property_add_bool(obj, "vfio-no-msix-emulation",
> +                             spapr_get_msix_emulation, NULL, NULL);
>  }
>  
>  static void spapr_machine_finalizefn(Object *obj)
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index ae9098d..4a03085 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -1580,6 +1580,19 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)
>       */
>      memory_region_set_enabled(&vdev->pdev.msix_pba_mmio, false);
>  
> +    /*
> +     * The emulated machine may provide a paravirt interface for MSIX setup
> +     * so it is not strictly necessary to emulate MSIX here. This becomes
> +     * helpful when frequently accessed MMIO registers are located in
> +     * subpages adjacent to the MSIX table but the MSIX data containing page
> +     * cannot be mapped because of a host page size bigger than the MSIX table
> +     * alignment.
> +     */
> +    if (object_property_get_bool(OBJECT(qdev_get_machine()),
> +                                 "vfio-no-msix-emulation", NULL)) {
> +        memory_region_set_enabled(&vdev->pdev.msix_table_mmio, false);
> +    }
> +
>      return 0;
>  }
>