From patchwork Wed Jan 16 16:49:20 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Roth X-Patchwork-Id: 212833 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id ABBC22C0086 for ; Thu, 17 Jan 2013 04:03:34 +1100 (EST) Received: from localhost ([::1]:43404 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TvWOi-0005Ci-T9 for incoming@patchwork.ozlabs.org; Wed, 16 Jan 2013 12:03:32 -0500 Received: from eggs.gnu.org ([208.118.235.92]:51506) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TvWOa-0005BK-7q for qemu-devel@nongnu.org; Wed, 16 Jan 2013 12:03:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TvWOW-0000gS-9E for qemu-devel@nongnu.org; Wed, 16 Jan 2013 12:03:24 -0500 Received: from mail-ie0-f177.google.com ([209.85.223.177]:58492) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TvWOW-0000gJ-2G; Wed, 16 Jan 2013 12:03:20 -0500 Received: by mail-ie0-f177.google.com with SMTP id k13so2938910iea.22 for ; Wed, 16 Jan 2013 09:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references; bh=27KAj0jL9vmRoSIlEQCIoBdhd21dBoAe4Fl+zkqvaxE=; b=Ih6yMGaSSwiHqw8azdpia8lxVuTPneaFlDRMJNasbzwq7sdlBjx0JpdxD5dwh1Zuis FPqmsjDjQ2FGJl/JT7jIJeJie5kpoFSITgTYSFJwccPh4laQwKLUc7JPKl9tv2KfQjBC RgJMn3NQtoN62khk88vz8Q/312Qnn77ueK2vMHnzfYrhBb0lXKOJS5/qvmcJiU9X75E3 ZzNuyTgfSxicd27KvKNyrHxXjtoHpi7w2f8I6iP2gdWeobYYQWDxfpbZsWlyPU2oUfXv qRYjC/h3D/aI23/Cj/KV4UymxON8AzJKvgrciW2lOYY8xCUFPgfUADwNVkUCljU6nBzD +TAA== X-Received: by 10.50.57.232 with SMTP id l8mr5245824igq.54.1358355332569; Wed, 16 Jan 2013 08:55:32 -0800 (PST) Received: from localhost ([32.97.110.59]) by mx.google.com with ESMTPS id gs6sm5162244igc.11.2013.01.16.08.55.31 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 16 Jan 2013 08:55:31 -0800 (PST) From: Michael Roth To: qemu-devel@nongnu.org Date: Wed, 16 Jan 2013 10:49:20 -0600 Message-Id: <1358354963-9070-19-git-send-email-mdroth@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1358354963-9070-1-git-send-email-mdroth@linux.vnet.ibm.com> References: <1358354963-9070-1-git-send-email-mdroth@linux.vnet.ibm.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 209.85.223.177 Cc: aliguori@us.ibm.com, qemu-stable@nongnu.org Subject: [Qemu-devel] [PATCH 18/21] vfio-pci: Make host MSI-X enable track guest X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Alex Williamson Guests typically enable MSI-X with all of the vectors in the MSI-X vector table masked. Only when the vector is enabled does the vector get unmasked, resulting in a vector_use callback. These two points, enable and unmask, correspond to pci_enable_msix() and request_irq() for Linux guests. Some drivers rely on VF/PF or PF/fw communication channels that expect the physical state of the device to match the guest visible state of the device. They don't appreciate lazily enabling MSI-X on the physical device. To solve this, enable MSI-X with a single vector when the MSI-X capability is enabled and immediate disable the vector. This leaves the physical device in exactly the same state between host and guest. Furthermore, the brief gap where we enable vector 0, it fires into userspace, not KVM, so the guest doesn't get spurious interrupts. Ideally we could call VFIO_DEVICE_SET_IRQS with the right parameters to enable MSI-X with zero vectors, but this will currently return an error as the Linux MSI-X interfaces do not allow it. Signed-off-by: Alex Williamson Cc: qemu-stable@nongnu.org (cherry picked from commit b0223e29afdc88cc262a764026296414396cd129) Signed-off-by: Michael Roth --- hw/vfio_pci.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/hw/vfio_pci.c b/hw/vfio_pci.c index fbfe670..46dce55 100644 --- a/hw/vfio_pci.c +++ b/hw/vfio_pci.c @@ -562,8 +562,8 @@ static int vfio_enable_vectors(VFIODevice *vdev, bool msix) return ret; } -static int vfio_msix_vector_use(PCIDevice *pdev, - unsigned int nr, MSIMessage msg) +static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, + MSIMessage *msg, IOHandler *handler) { VFIODevice *vdev = DO_UPCAST(VFIODevice, pdev, pdev); VFIOMSIVector *vector; @@ -587,7 +587,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, * Attempt to enable route through KVM irqchip, * default to userspace handling if unavailable. */ - vector->virq = kvm_irqchip_add_msi_route(kvm_state, msg); + vector->virq = msg ? kvm_irqchip_add_msi_route(kvm_state, *msg) : -1; if (vector->virq < 0 || kvm_irqchip_add_irqfd_notifier(kvm_state, &vector->interrupt, vector->virq) < 0) { @@ -596,7 +596,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, vector->virq = -1; } qemu_set_fd_handler(event_notifier_get_fd(&vector->interrupt), - vfio_msi_interrupt, NULL, vector); + handler, NULL, vector); } /* @@ -639,6 +639,12 @@ static int vfio_msix_vector_use(PCIDevice *pdev, return 0; } +static int vfio_msix_vector_use(PCIDevice *pdev, + unsigned int nr, MSIMessage msg) +{ + return vfio_msix_vector_do_use(pdev, nr, &msg, vfio_msi_interrupt); +} + static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { VFIODevice *vdev = DO_UPCAST(VFIODevice, pdev, pdev); @@ -697,6 +703,22 @@ static void vfio_enable_msix(VFIODevice *vdev) vdev->interrupt = VFIO_INT_MSIX; + /* + * Some communication channels between VF & PF or PF & fw rely on the + * physical state of the device and expect that enabling MSI-X from the + * guest enables the same on the host. When our guest is Linux, the + * guest driver call to pci_enable_msix() sets the enabling bit in the + * MSI-X capability, but leaves the vector table masked. We therefore + * can't rely on a vector_use callback (from request_irq() in the guest) + * to switch the physical device into MSI-X mode because that may come a + * long time after pci_enable_msix(). This code enables vector 0 with + * triggering to userspace, then immediately release the vector, leaving + * the physical device with no vectors enabled, but MSI-X enabled, just + * like the guest view. + */ + vfio_msix_vector_do_use(&vdev->pdev, 0, NULL, NULL); + vfio_msix_vector_release(&vdev->pdev, 0); + if (msix_set_vector_notifiers(&vdev->pdev, vfio_msix_vector_use, vfio_msix_vector_release)) { error_report("vfio: msix_set_vector_notifiers failed\n");