From patchwork Wed May 22 09:54:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 1937774 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QuluR2mC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Vkmy12Ngbz1ynR for ; Wed, 22 May 2024 20:00:09 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s9ih2-0003Yu-OD; Wed, 22 May 2024 05:55:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9ign-0003RM-4u for qemu-devel@nongnu.org; Wed, 22 May 2024 05:55:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9igk-0001Q3-N5 for qemu-devel@nongnu.org; Wed, 22 May 2024 05:55:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1716371714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FbYDvgZ3A8fXGUyykwgKl2IroA+m5TOvdcZPY3ucw2s=; b=QuluR2mC4UNjv5dpiha5/bJUjaNuMiNplpNC+tt7ulVSddmsBGMFPG9bXdGAXnRGZ+1DdR 7YsWxdOqyXYcdiWgTLG70EKv5HaaFNANfuu/UNrpily6+GTU+Chk5rRlsvuDxZFHj8QNnH 1nV5TbHrSkUsEoFvKkLUHNtltKyJq60= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-175-V2N0Rg6uP-2KXOGCFsHy_w-1; Wed, 22 May 2024 05:55:10 -0400 X-MC-Unique: V2N0Rg6uP-2KXOGCFsHy_w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 74087380673C; Wed, 22 May 2024 09:55:10 +0000 (UTC) Received: from corto.redhat.com (unknown [10.39.192.133]) by smtp.corp.redhat.com (Postfix) with ESMTP id A79F27414; Wed, 22 May 2024 09:55:08 +0000 (UTC) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Vinayak Kale , "Michael S . Tsirkin" , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 14/47] vfio/pci: migration: Skip config space check for Vendor Specific Information in VSC during restore/load Date: Wed, 22 May 2024 11:54:09 +0200 Message-ID: <20240522095442.195243-15-clg@redhat.com> In-Reply-To: <20240522095442.195243-1-clg@redhat.com> References: <20240522095442.195243-1-clg@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Vinayak Kale In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed. config space check is done in function get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked. Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host. If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical. This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset. VSC check is skipped for machine types >= 9.1. The check would be enforced on older machine types (<= 9.0). Cc: Alex Williamson Cc: Michael S. Tsirkin Cc: Cédric Le Goater Signed-off-by: Vinayak Kale Reviewed-by: Cédric Le Goater Signed-off-by: Cédric Le Goater --- hw/vfio/pci.h | 1 + hw/core/machine.c | 1 + hw/vfio/pci.c | 26 ++++++++++++++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 6e64a2654e690af11b72710530a41135b726e96f..92cd62d1159dbd47d878454f201f9c18112a7692 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -177,6 +177,7 @@ struct VFIOPCIDevice { OnOffAuto ramfb_migrate; bool defer_kvm_irq_routing; bool clear_parent_atomics_on_exit; + bool skip_vsc_check; VFIODisplay *dpy; Notifier irqchip_change_notifier; }; diff --git a/hw/core/machine.c b/hw/core/machine.c index c7ceb1150147eb2bdd5f7ef583d00cee88f306cd..3442f31f9458c4e1bdde366aa40bfe3ad8b321a8 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -35,6 +35,7 @@ GlobalProperty hw_compat_9_0[] = { {"arm-cpu", "backcompat-cntfrq", "true" }, + {"vfio-pci", "skip-vsc-check", "false" }, }; const size_t hw_compat_9_0_len = G_N_ELEMENTS(hw_compat_9_0); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index fc6e54e871508bb0e2a3ac9079a195c086531f21..4789d43c0f9cc7ef94b73adc815377f7222d8c57 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2134,6 +2134,28 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_add_vendor_specific_cap(VFIOPCIDevice *vdev, int pos, + uint8_t size, Error **errp) +{ + PCIDevice *pdev = &vdev->pdev; + + pos = pci_add_capability(pdev, PCI_CAP_ID_VNDR, pos, size, errp); + if (pos < 0) { + return pos; + } + + /* + * Exempt config space check for Vendor Specific Information during + * restore/load. + * Config space check is still enforced for 3 byte VSC header. + */ + if (vdev->skip_vsc_check && size > 3) { + memset(pdev->cmask + pos + 3, 0, size - 3); + } + + return pos; +} + static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp) { ERRP_GUARD(); @@ -2202,6 +2224,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp) vfio_check_af_flr(vdev, pos); ret = pci_add_capability(pdev, cap_id, pos, size, errp); break; + case PCI_CAP_ID_VNDR: + ret = vfio_add_vendor_specific_cap(vdev, pos, size, errp); + break; default: ret = pci_add_capability(pdev, cap_id, pos, size, errp); break; @@ -3391,6 +3416,7 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_LINK("iommufd", VFIOPCIDevice, vbasedev.iommufd, TYPE_IOMMUFD_BACKEND, IOMMUFDBackend *), #endif + DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check, true), DEFINE_PROP_END_OF_LIST(), };