{"id":2235164,"url":"http://patchwork.ozlabs.org/api/1.2/covers/2235164/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/cover/20260508183717.193630-1-tdave@nvidia.com/","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/1.2/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":"","list_archive_url":"","list_archive_url_format":"","commit_url_format":""},"msgid":"<20260508183717.193630-1-tdave@nvidia.com>","list_archive_url":null,"date":"2026-05-08T18:37:09","name":"[RFC,0/8] hw/arm/virt, hw/pci: PCI pre-enumeration and fixed BAR allocation","submitter":{"id":89928,"url":"http://patchwork.ozlabs.org/api/1.2/people/89928/?format=json","name":"Tushar Dave","email":"tdave@nvidia.com"},"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/cover/20260508183717.193630-1-tdave@nvidia.com/mbox/","series":[{"id":503429,"url":"http://patchwork.ozlabs.org/api/1.2/series/503429/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/list/?series=503429","date":"2026-05-08T18:37:09","name":"hw/arm/virt, hw/pci: PCI pre-enumeration and fixed BAR allocation","version":1,"mbox":"http://patchwork.ozlabs.org/series/503429/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/covers/2235164/comments/","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256\n header.s=selector2 header.b=YlIecEpY;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists1p.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)","dkim=none (message not signed)\n header.d=none;dmarc=none action=none header.from=nvidia.com;"],"Received":["from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4gByYk5nMSz1yCg\n\tfor ; Sat, 09 May 2026 04:38:33 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists1p.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1wLQ5D-0006e0-Md; Fri, 08 May 2026 14:37:55 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1wLQ5C-0006co-3A; Fri, 08 May 2026 14:37:54 -0400","from mail-centralusazlp170100005.outbound.protection.outlook.com\n ([2a01:111:f403:c111::5] helo=DM1PR04CU001.outbound.protection.outlook.com)\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1wLQ59-0002Yg-Qt; Fri, 08 May 2026 14:37:53 -0400","from BY5PR12MB4179.namprd12.prod.outlook.com (2603:10b6:a03:211::8)\n by CH3PR12MB8482.namprd12.prod.outlook.com (2603:10b6:610:15b::8)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9891.20; Fri, 8 May\n 2026 18:37:39 +0000","from BY5PR12MB4179.namprd12.prod.outlook.com\n ([fe80::2036:e8b:9b3:f325]) by BY5PR12MB4179.namprd12.prod.outlook.com\n ([fe80::2036:e8b:9b3:f325%4]) with mapi id 15.20.9891.019; Fri, 8 May 2026\n 18:37:39 +0000"],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=JhQrql64GlNjMqFIM+fjr/BrXsgn/GQV6GoLW/ieTnVbk52p/ubjKTftAYoKRJnomOLz3ckDpXyOodktaNeTJRYBoTXHJzS1OQYP24FMsjwvGhTvWBd+JxBk9pVYL0yx3QsMDiDrN/hxcM0wy7q3kMBPZ9zce+LBH91LJnZ3pxXJvb8RzcheJgaU75XYLlzGoczhTVGiuJFCscoiInX25eU0q81WobNErOINpAAK+L5PdV4FdVMPXLJJy0hfDsUL8ZhnKZwcOtqsRPmz6hYvstDPPxTH6tD3oMqma+kzzxFkkGfe/c5sHxc+s3j1ToGdRTNvmYqhJo12V70adjoX1Q==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=aRfNNwwEmFLJUlHdPjGJiCAGsjBF2r2xwf0aAUXmhM8=;\n b=hk63YiaBRsZAmt47Xg8mig4MJkmI1SIP5iK5gZuJBzLmAvuQMGRVpDZM9LOyKnDyVXCAsZ68BS24xcJteX1xvxv7DMoBa3Ppv9oJhgBv11m7YVEjUqhDlvmQGDVi18XTohxKxN7Sy0TsVar2TgDLTYVZ+/BP4/9cW7YDafc7HFwmEJlg/Gv9mV2icQPU9Jl5mBoKoGNvf4kncO2Hj9k9CQLO85rcupsUb0l9tBSR/GjlfAvp4pGJttst+o35/7GLwx9nF8IxjtvNLzpA1uybIMLUTnLqFJpOfGTRHMfri7vwKBxlNhRPNbrM3RYlaS3Yv/ZiQsj0YR7UV2sV8NjEqQ==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;\n dkim=pass header.d=nvidia.com; arc=none","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;\n s=selector2;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=aRfNNwwEmFLJUlHdPjGJiCAGsjBF2r2xwf0aAUXmhM8=;\n b=YlIecEpY69oDM0yc8YchPatLcIKolJMJ6kQaqH0yiEFaHRcXZvXgF83w1SbIknjK1bw/FiNKhf949oVIKdFW+sW3oROhenfweQL26cxULt3CjTXbac0q061szIhJJC0LifoiBJgg1TSYIu2WuQJFmWYBP/xJfWXHB4TROWqx7SAVV1sxxq4AcBLabkS84CvxKtKlzThf/Xr3t0irO33HUlwVoy5d39SuTZKMPHaNyZYJXSdzAjUMP8h3WavWo89orBaf+XQGS76ISbay0stZ5ZRXp9OMlruuCE2qJrDFtRnvggzQWPzRTunyCaSaWlQXyI94Ci64JWAFo58e24x/xg==","From":"Tushar Dave ","To":"qemu-devel@nongnu.org","Cc":"alwilliamson@nvidia.com, jgg@nvidia.com, skolothumtho@nvidia.com,\n qemu-arm@nongnu.org, peter.maydell@linaro.org, mst@redhat.com,\n marcel.apfelbaum@gmail.com, devel@edk2.groups.io","Subject":"[RFC PATCH 0/8] hw/arm/virt,\n hw/pci: PCI pre-enumeration and fixed BAR allocation","Date":"Fri, 8 May 2026 13:37:09 -0500","Message-Id":"<20260508183717.193630-1-tdave@nvidia.com>","X-Mailer":"git-send-email 2.34.1","Content-Transfer-Encoding":"8bit","Content-Type":"text/plain","X-ClientProxiedBy":"BN9PR03CA0959.namprd03.prod.outlook.com\n (2603:10b6:408:108::34) To BY5PR12MB4179.namprd12.prod.outlook.com\n (2603:10b6:a03:211::8)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"BY5PR12MB4179:EE_|CH3PR12MB8482:EE_","X-MS-Office365-Filtering-Correlation-Id":"2ce5c27b-ee78-4e9e-70a9-08dead30e216","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;\n ARA:13230040|366016|376014|1800799024|18002099003|56012099003|3023799003;","X-Microsoft-Antispam-Message-Info":"\n LHOTFabKMP00UYin0pUO2KM5TuYXU+UpjSKwGm8T2nnT5BAiiejNCQSU45m3qy7qMuQ/cNa+gRQlOy+mht3e4wRt+0zIF8FQhISmz14AYAlJ+6cqWpZFNyQLkET7edoBiYCJyvN42j+sot8Luzhrf2QLZAQceaN7tT/agBuhSSgC6OKKf+QMMGD+6VdTmvsZ27K5dHmHOfpseK3PMcKUG5LTyeb92/OO4P1PSWU++nSLzq56c3AGydv0F2mqEBbi9pUwhJI/ZsReWVEQQBvgquPpxe/n8uGthHF5k9Xkgu8KTdb9stSrdPjEr8aIJnHvMmQJ9mlmgSwcCGh/ak+mcnLyswePgZCIcI/+NiWBURjV17sNj0/FXNpMu+GRLF6KQZR5dpVYHb2cLgqYWeQQ49lMWgok/LHYOCMo8ZEXQ2yTA/fdf8WmQEu18ZHfeT8g8C0+TBfJCOvTJwFD7BSYwlMf4Z90okegWkbxw4C5Wzjeky4IV9Zxe9izswny3BooDjvTgZi+EUEaM0iH1okL/kQ3pT8qIDx7lzH75vqZS/TmgAyOxMxq1TdyqMlhVUoAveR7fminEz3zwKjzSnuUETSQeSZX01XEsgtzK9wOgJNabPqfRecOFPbMR4n8cdKpHOCmU9G43EZDJobz349BpXJT6M+MgtMU3v2irQGcw2ICD1VU5XAyd+p+BoPcgAww","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:BY5PR12MB4179.namprd12.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(376014)(1800799024)(18002099003)(56012099003)(3023799003);\n DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"\n HXr+8g7cq3LhsNqix6wP4ThO+j7H7E0HgZKU7ykkNBrYAHR/I1oHTaRUJX8+TrbEv/FDU67pUApBExdQFKFi/iRSp0nn1ndzQDJsRABmsllp3tKHte3YdeZDedK/iElRMqUzdt54tE8k28OLo/G3+aUPBWwu2mWTsrua2ZE25KRaFpP5j3mDABa0Z+tDhTHB0CxTUWqHCqfsRxr9W7q6t+SgCKcyE+EXJhAwmj25YWfH2VaCWGnGOljzRV5IdsmIyykT/0ZXRfOsRRz9xy+n2490Gwosq9L7EiLWl+djDczUN7TTMj6gK5YK8UNURfFa75sVoSPvl7NFeL+j1IcdJJ8bnVnuvuoQgdtxirpcWTD4zsRSTrA6TZCQAhoZm6xQDVswAu3THpQS/mWpUUqSFkFLaN+biTxaLMpb/EB22oTmwzD0ysjqDPvJb+cLu5wr0EqSzWnTWwPBvApScJT4esm2qIyeA1mom8DiOUBu7R9sQ6Xz9/yLRxmfCq9LjxETrYBOM9e8sR8SA2j/BBSEGUtQZq+8hKI6V8mtZECTfKdYnk5A5EDysa8hCh1MWR1zdzPXCh3a89wsn3a2PaLcl+4D+p8GSRGMyCZithDn8LpjI7ReQwKkBfKde5rfHWcm9LJ/UOJaRpLWrnugnFzouDTH+gAhCxr6ornTYg3N58GD13TxN8+yTTKE6ukX0pjxdcDctoS6np6QGOVb25wthv3c9kM/tZiGixII9bOCkFHAR6p6vOnDGTlr0J+8tx678a8eYYGuWQIgZQ9Nh2cg/kZ/Qz3qM0X5ATygcLakfycKOpCXtv7B73uwGPf2FtKv3+vCb/49pGLesWqAPvh11gpZoZBFkf5btqE7m1epS/6s5WTfW1tzSjXuXl7o0IHrl8y07SHpy/WT5SxBphJuNHVxI8eDNrkPHSb77y365sL/xrXeavhcmWjVEhfoUxEb/8m96OtAOS83YQ3Ep3X16MDSnLbKG+N6X6Ru3NvZEOH71mYIkQWX1Pnm7KcOzF0CFhWCGnN+iKqVsGkm3YDN5JtIaGTcuDET2mM245qlfBbyhpHLTEu70nB0I0OgIuOf6FLtP0Frr21hzKHeras6kHEDREuRqZIV85pnWgNGQXmF6bJZu4k8pSm29XKsuFz9/c2Wm8PWJRrmUPq4OXbubRwXoUrgpBDotN4oF0QTE08RiBxgOHKIoh3g1/UfOxfG/fsj5a8gc+MGjgajWSfJsiP8Aomo9XP5uPk+viqbicH62KACSKKlZLxpfeWgGsDYjcZx3BHfCzEHVz/P0k0r+0CCX14BfbMUz0NE++IWo9e8rcCTycJve0f0WuGeypZl9Nuna9qE8O3SzSYMW1hmMYLWXzaI0JHB7+gIRmg2pPH7taL0Cc0N+pFtNOrj8vWinrELY0o3XjcR8YIO9x91Q4OgzfUbJ7syWfdzdmoVVlPGYkrNt2EQ02RYIadLu6FKKOeiLatpTN6uMJxxoq8tXOwhhehdr2B8ufmAG3i33jtnkKhcNiUBvn5zVSgyQH66pEZcVl9kneQ65XrqOcfp2/ZoBguq6kbe9ALTzAtFTsHu6rL08wfz34mpPQyJ184+xwc+aC+SrJOaSHRH4NRkOY8HbgBy8Y81tNnBs4DsWHZLsGcWrPCkxUJEJmo+Hst9Y5JRewgZDAUJMTP5CscyTrlMHD5HjHCUyZRup2/8Zis2OmuQ3rjG2tl60or7ssYbsTbY6Zfxpp4XjKhd0a1qYA==","X-OriginatorOrg":"Nvidia.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 2ce5c27b-ee78-4e9e-70a9-08dead30e216","X-MS-Exchange-CrossTenant-AuthSource":"BY5PR12MB4179.namprd12.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"08 May 2026 18:37:39.7362 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"43083d15-7273-40c1-b7db-39efd9ccc17a","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n u1t59isii7KRrJLCgccMUNqOhPUkgzZwU/gx7YRMIdDsI06wb/rDA2fDxVERu1L9NHEOGuiQgnMysnsyK7qR/A==","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CH3PR12MB8482","Received-SPF":"permerror client-ip=2a01:111:f403:c111::5;\n envelope-from=tdave@nvidia.com;\n helo=DM1PR04CU001.outbound.protection.outlook.com","X-Spam_score_int":"-14","X-Spam_score":"-1.5","X-Spam_bar":"-","X-Spam_report":"(-1.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.44,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n FORGED_SPF_HELO=1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001,\n SPF_NONE=0.001 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"qemu development ","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"},"content":"This RFC introduces a mechanism to specify Guest Physical Addresses\n(GPAs) for PCI BARs, allowing explicit placement of guest MMIO BAR\naddresses to match host physical addresses for assigned devices.\n\nOn some platforms, P2P DMA is performed between devices within the same\nIOMMU group. The PCI fabric ACS is configured to permit direct P2P\nwithout going through the host bridge in order to achieve the required\nperformance.\n\nTo support this multi-device IOMMU group P2P scenario in virtualization,\nthe VM may need to use the same MMIO BAR addresses as the host physical\naddress layout.\n\nThis series implements a per-device PCI property, \"fixed-bars\", which\nallows users to specify fixed BAR addresses. The property is generic\nand is available on any PCI-capable machine. It is a comma-separated\nlist of BAR assignments:\n\n barN@[,barM@]*\n\nThe virt machine builds on this with two additional machine properties:\n\npci-pre-enum\n When enabled, QEMU performs PCI enumeration and resource assignment\n before handing control to firmware (e.g. EDK2). This includes\n programming 64-bit prefetchable BARs according to fixed-bars\n assignments, and programming bridge prefetchable windows.\n A \"pci-enum-done\" device-tree property is set so firmware preserves\n the configuration.\n\npcie-mmio-window\n Defines the MMIO64 window for PCIe devices. When using fixed-bars,\n this allows the aperture to be resized or repositioned so all\n assigned BARs fall within a valid address range.\n\nWhy QEMU programs PCI resources rather than EDK2:\n\nTo support fixed BAR placement, QEMU performs PCI bus enumeration and\nresource assignment prior to firmware execution. EDK2 already provides\na PCD-controlled mechanism (PcdPciDisableBusEnumeration) that allows\nthe platform to skip PCI enumeration and resource allocation. This\nseries leverages that mechanism so that, when enabled, firmware runs in\na discovery-only mode and preserves the configuration established by\nQEMU.\n\nWhen pci-pre-enum is enabled, QEMU runs PCI enumeration and resource\nallocation, prioritizing fixed BARs specified via fixed-bars. If\nallocation fails due to alignment, overlap, or address space constraints,\nQEMU terminates with an error. Otherwise, all BARs and bridge windows are\nfully programmed before firmware execution.\n\nThere is certainly room for improvement, but this RFC aims to gather\nfeedback on the overall approach chosen to address this problem.\n\nWe use the virt machine in this series as the concrete example\nconsuming the fixed-BAR model. Other machines may require their own\nmachine-specific mechanism (such as pcie-mmio-window) if they want to\nadopt the same approach.\n\nExample usage:\n\n -machine virt,...,pcie-mmio-window=0x400000000000:0x400000000000,pci-pre-enum=on \\\n -device vfio-pci,host=0009:06:00.0,id=dev0 \\\n -set device.dev0.fixed-bars=bar2@0x6b8000000000,bar4@0x6c8000000000\n\nTesting:\nThis series was tested on NVIDIA GB300 platforms with a recent Linux\nkernel. GPUDirect P2P between a GPU and a CX8 NIC requires a PCIe\ntopology in the VM that mirrors bare metal (e.g. both devices under the\nsame switch and ACS tuned for the minimal P2P paths needed for GPUDirect\nRDMA).\n\nTODO:\n- The fixed BAR allocator handles 64-bit prefetchable BARs and related\n bridge prefetch windows only. Programming PIO, 32-bit MMIO, and\n 64-bit non-prefetchable BARs, and sizing bridge windows for those\n resource types, is left for follow-up patches.\n- SR-IOV virtual functions are not included when sizing bridge prefetch\n apertures and may require additional work.\n- Add ACPI _DSM so the fixed BARs are preserved.\n\n\nA git branch with this series applied is available at:\nhttps://github.com/tdavenvidia/upstream-qemu/commits/upstream_May_08_26/\n\nThe related EDK2 change is available at:\nhttps://github.com/tdavenvidia/edk2/commits/upstream_May_08_26/\n\nTushar Dave (8):\n hw/pci: add fixed-bars property to allow fixed BAR addresses\n hw/pci: enumerate PCI bus and program bridge bus numbers\n hw/pci: introduce allocator for fixed BAR placement\n hw/pci: pack remaining BARs and update bridge windows\n hw/pci: allocate remaining BARs for buses without fixed BARs\n hw/pci: finalize bridge prefetch windows after BAR allocation\n hw/arm/virt: add pcie-mmio-window machine property\n hw/arm/virt: add pci-pre-enum machine property\n\n hw/arm/virt.c | 157 ++++-\n hw/pci/meson.build | 2 +\n hw/pci/pci-enumerate.c | 144 +++++\n hw/pci/pci-enumerate.h | 15 +\n hw/pci/pci-resource.c | 1099 +++++++++++++++++++++++++++++++++++\n hw/pci/pci-resource.h | 82 +++\n hw/pci/pci.c | 108 ++++\n include/hw/arm/virt.h | 3 +\n include/hw/pci/pci_device.h | 10 +\n 9 files changed, 1615 insertions(+), 5 deletions(-)\n create mode 100644 hw/pci/pci-enumerate.c\n create mode 100644 hw/pci/pci-enumerate.h\n create mode 100644 hw/pci/pci-resource.c\n create mode 100644 hw/pci/pci-resource.h"}