| Message ID | 20260902072804.665639-1-smadhavan@nvidia.com |
|---|---|
| Headers | show |
| Series | PCI/CXL: Add CXL reset support for Type 2 devices | expand |
On Wed, Sep 02, 2026 at 07:27:52AM +0800, Srirangan Madhavan wrote: > Hi folks! > > This series adds CXL Reset support for CXL Type 2 devices through the > existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 > specification [1], including CXL.cache disable, optional cache > writeback, CXL Reset initiation, ResetComplete polling, and ResetError > reporting. > Hi Srirangan, > The userspace ABI is the existing PCI reset interface: > > /sys/bus/pci/devices/.../reset_method > /sys/bus/pci/devices/.../reset > > Userspace can select "cxl_reset" in reset_method and then trigger reset > through the existing reset attribute. > > CXL Reset is exposed conservatively. The target must be a > single-function CXL Type 2 device with CXL.cache and CXL.mem enabled, > and the kernel must have cached HDM state sufficient to validate affected > ranges and restore decoder programming. The reset preserves device > memory; it does not request CXL Reset Memory Clear. > It seems reasonable to me, the whole approach. I wonder if we can document the complete reset-safety contract down ? Especially what guarantees that all CXL.mem users and functions are quiesced ? Besides HDM EP , what are other EP state that should be retained or restored ? I think it will make people understand the mechanism more easily. Best regards, Richard Cheng. > Following Dan's v6 feedback, this uses the existing PCI reset_method > interface instead of adding a memdev-specific sysfs ABI. > > v11 changes from v10 [2]: > - Rebase onto v7.3-rc1, which includes the CXL Type 2 enabling work, and > use the merged public CXL PCI header. > - Rename the built-in support symbol to CONFIG_CXL_RESET and fix > built-in/module linkage for CONFIG_CXL_BUS=m. > - Fix tools/testing/cxl linkage without compiling production helper > objects twice in the mock cxl_core module. > - Simplify decoder snapshot handling and preserve endpoint DPA skip > ownership under the existing DPA lock. > - Keep policy out of shared decoder parsing by accepting spec-legal > zero-size decoder state, while skipping zero-size ranges during reset > validation. > - Tighten early HDM cache setup, capability checks, BAR bounds, MSE > enable/restore handling, and cache publication locking. > - Keep the conservative multifunction-device rejection requested in > earlier review. > - Harden post-reset failure handling so MSE and BME remain disabled when > HDM state cannot be safely restored. > - Serialize CXL bus-reset HDM restore against region updates and flush > posted HDM writes before PCI_COMMAND can restore bus mastering. > - Address the remaining reviewability and naming feedback from v10. > > Motivation: > ----------- > - Type 2 devices need a CXL-specific reset mechanism beyond existing PCI > reset methods. > > - FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is > the architectural reset mechanism for those protocols. > > - The PCI reset_method ABI lets userspace select CXL Reset before > falling back to broader bus reset methods. > > Change Description: > ------------------- > > Patch 1: cxl: Move HDM decoder programming helpers > - Introduce a decoder settings snapshot collected under the DPA lock. > - Move low-level settings-based HDM programming into resource.c. > > Patch 2: cxl: Make HDM commit helpers available to reset code > - Build generic HDM resource helpers under CONFIG_CXL_RESET. > - Export the lock and commit helpers needed when cxl_core is modular. > > Patch 3: cxl: Share HDM decoder decode logic > - Share HDM decoder decode and validation between normal CXL core > enumeration and early PCI HDM cache setup. > - Decode zero-size committed decoder state without imposing caller policy. > > Patch 4: cxl: Cache decoder settings on PCI devices > - Cache CXL core HDM decoder settings in pci_dev->hdm. > - Refresh the cache as decoders are enumerated, committed, or reset. > > Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration > - Snapshot endpoint HDM state during PCI enumeration. > - Cache the HDM register locator as BAR-relative metadata and preserve raw > decoder state for reset restore. > > Patch 6: cxl: Add CXL Device Reset helper > - Add the internal DVSEC reset sequence. > - Disable CXL.cache, perform cache writeback where supported, initiate > CXL Reset, and wait for completion. > > Patch 7: cxl: Validate HDM ranges before CXL reset > - Collect enabled cached HDM ranges, reject busy ranges, and perform CPU > cache invalidation when a runtime backend is available. > > Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices > - Conservatively reject CXL Reset where function scope cannot be assumed. > > Patch 9: cxl: Restore CXL HDM state after PCI reset > - Restore cached global and per-decoder HDM state after reset while the > target IOMMU reset block remains active. > > Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method > - Add "cxl_reset" to the PCI reset-method table ahead of FLR. > > Patch 11: Documentation/ABI: Document CXL Reset PCI reset method > - Document the new reset_method value and reset behavior. > > Patch 12: PCI/CXL: Restore HDM state after CXL bus reset > - Restore cached CXL HDM decoder state after cxl_bus reset. > > The CPU cache invalidation step depends on > cpu_cache_invalidate_memregion() support for the affected address ranges. > If no runtime provider is available, the kernel emits a warning and > continues after the affected HDM ranges have been reserved. > > Example: > > echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method > echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset > > Testing: > - Built every intermediate patch, plus the v7.3-rc1 baseline, with > CONFIG_CXL_BUS=m. All 12 patches passed a full kernel build and > `make M=tools/testing/cxl/` independently. > - Built and booted the series on an arm64 NVIDIA NVL72 system and > successfully exercised CXL Reset through the PCI reset sysfs ABI. > - Earlier revision testing ran 100 iterations of cxl_reset and exercised > cxl_bus separately with HDM state restoration. > > References: > [1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf > [2] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/ > [3] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/ > [4] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/ > [5] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/ > > Srirangan Madhavan (12): > cxl: Move HDM decoder programming helpers > cxl: Make HDM commit helpers available to reset code > cxl: Share HDM decoder decode logic > cxl: Cache decoder settings on PCI devices > cxl: Cache endpoint decoder settings during PCI enumeration > cxl: Add CXL Device Reset helper > cxl: Validate HDM ranges before CXL reset > PCI/CXL: Reject CXL Reset on multifunction devices > cxl: Restore CXL HDM state after PCI reset > PCI/CXL: Expose CXL Reset as a PCI reset method > Documentation/ABI: Document CXL Reset PCI reset method > PCI/CXL: Restore HDM state after CXL bus reset > > Documentation/ABI/testing/sysfs-bus-pci | 15 + > drivers/cxl/Kconfig | 4 + > drivers/cxl/core/Makefile | 2 +- > drivers/cxl/core/core.h | 8 + > drivers/cxl/core/hdm.c | 324 +++--- > drivers/cxl/core/pci.c | 7 +- > drivers/cxl/core/port.c | 19 + > drivers/cxl/core/regs.c | 4 + > drivers/cxl/core/resource.c | 1351 +++++++++++++++++++++++ > drivers/cxl/cxl.h | 41 - > drivers/pci/bus.c | 2 + > drivers/pci/pci.c | 8 + > drivers/pci/probe.c | 2 + > include/cxl/cxl.h | 116 +- > include/linux/pci.h | 8 +- > include/uapi/linux/pci_regs.h | 14 + > tools/testing/cxl/Kbuild | 1 - > 17 files changed, 1715 insertions(+), 211 deletions(-) > create mode 100644 drivers/cxl/core/resource.c > > > base-commit: cee9395acd8043be0644b25c34bfa86623f2b935 > -- > 2.43.0
On Wed, 2 Sep 2026 07:27:52 +0000 Srirangan Madhavan <smadhavan@nvidia.com> wrote: > Hi folks! > > This series adds CXL Reset support for CXL Type 2 devices through the > existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 > specification [1], including CXL.cache disable, optional cache > writeback, CXL Reset initiation, ResetComplete polling, and ResetError > reporting. > > The userspace ABI is the existing PCI reset interface: > > /sys/bus/pci/devices/.../reset_method > /sys/bus/pci/devices/.../reset > > Userspace can select "cxl_reset" in reset_method and then trigger reset > through the existing reset attribute. > > CXL Reset is exposed conservatively. The target must be a > single-function CXL Type 2 device with CXL.cache and CXL.mem enabled, > and the kernel must have cached HDM state sufficient to validate affected > ranges and restore decoder programming. The reset preserves device > memory; it does not request CXL Reset Memory Clear. How does this provide multi-tenancy then? vfio-pci considers the device to be scrubbed of tenant data after a pci_reset_function() and I don't think Manish's series alters that. Thanks, Alex
On 9/4/26 2:15 AM, Richard Cheng wrote: >> and the kernel must have cached HDM state sufficient to validate affected >> ranges and restore decoder programming. The reset preserves device >> memory; it does not request CXL Reset Memory Clear. >> > It seems reasonable to me, the whole approach. > I wonder if we can document the complete reset-safety contract down ? > > Especially what guarantees that all CXL.mem users and functions are quiesced ? > Besides HDM EP , what are other EP state that should be retained or restored ? > > I think it will make people understand the mechanism more easily. > > Best regards, > Richard Cheng. Thank you. I expanded the v12 cover letter and ABI documentation to describe the complete reset-safety contract, including the quiescing assumptions, state restoration, and limitations around memory offlining and mapping revocation.
On 9/9/26 2:13 PM, Alex Williamson wrote: > How does this provide multi-tenancy then? vfio-pci considers the > device to be scrubbed of tenant data after a pci_reset_function() and > I don't think Manish's series alters that. Thanks, > > Alex Hi Alex! I missed considering this. In V12 I'm exposing cxl_reset when the device advertises CXL Reset Memory Clear capability and sets RST_MEM_CLR_EN when initiating reset. I added the gate to prevent exposing a non-scrubbing reset to VFIO.
Hi folks! This series adds CXL Reset support for CXL Type 2 devices through the existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset initiation, ResetComplete polling, and ResetError reporting. The userspace ABI is the existing PCI reset interface: /sys/bus/pci/devices/.../reset_method /sys/bus/pci/devices/.../reset Userspace can select "cxl_reset" in reset_method and then trigger reset through the existing reset attribute. CXL Reset is exposed conservatively. The target must be a single-function CXL Type 2 device with CXL.cache and CXL.mem enabled, and the kernel must have cached HDM state sufficient to validate affected ranges and restore decoder programming. The reset preserves device memory; it does not request CXL Reset Memory Clear. Following Dan's v6 feedback, this uses the existing PCI reset_method interface instead of adding a memdev-specific sysfs ABI. v11 changes from v10 [2]: - Rebase onto v7.3-rc1, which includes the CXL Type 2 enabling work, and use the merged public CXL PCI header. - Rename the built-in support symbol to CONFIG_CXL_RESET and fix built-in/module linkage for CONFIG_CXL_BUS=m. - Fix tools/testing/cxl linkage without compiling production helper objects twice in the mock cxl_core module. - Simplify decoder snapshot handling and preserve endpoint DPA skip ownership under the existing DPA lock. - Keep policy out of shared decoder parsing by accepting spec-legal zero-size decoder state, while skipping zero-size ranges during reset validation. - Tighten early HDM cache setup, capability checks, BAR bounds, MSE enable/restore handling, and cache publication locking. - Keep the conservative multifunction-device rejection requested in earlier review. - Harden post-reset failure handling so MSE and BME remain disabled when HDM state cannot be safely restored. - Serialize CXL bus-reset HDM restore against region updates and flush posted HDM writes before PCI_COMMAND can restore bus mastering. - Address the remaining reviewability and naming feedback from v10. Motivation: ----------- - Type 2 devices need a CXL-specific reset mechanism beyond existing PCI reset methods. - FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is the architectural reset mechanism for those protocols. - The PCI reset_method ABI lets userspace select CXL Reset before falling back to broader bus reset methods. Change Description: ------------------- Patch 1: cxl: Move HDM decoder programming helpers - Introduce a decoder settings snapshot collected under the DPA lock. - Move low-level settings-based HDM programming into resource.c. Patch 2: cxl: Make HDM commit helpers available to reset code - Build generic HDM resource helpers under CONFIG_CXL_RESET. - Export the lock and commit helpers needed when cxl_core is modular. Patch 3: cxl: Share HDM decoder decode logic - Share HDM decoder decode and validation between normal CXL core enumeration and early PCI HDM cache setup. - Decode zero-size committed decoder state without imposing caller policy. Patch 4: cxl: Cache decoder settings on PCI devices - Cache CXL core HDM decoder settings in pci_dev->hdm. - Refresh the cache as decoders are enumerated, committed, or reset. Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration - Snapshot endpoint HDM state during PCI enumeration. - Cache the HDM register locator as BAR-relative metadata and preserve raw decoder state for reset restore. Patch 6: cxl: Add CXL Device Reset helper - Add the internal DVSEC reset sequence. - Disable CXL.cache, perform cache writeback where supported, initiate CXL Reset, and wait for completion. Patch 7: cxl: Validate HDM ranges before CXL reset - Collect enabled cached HDM ranges, reject busy ranges, and perform CPU cache invalidation when a runtime backend is available. Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices - Conservatively reject CXL Reset where function scope cannot be assumed. Patch 9: cxl: Restore CXL HDM state after PCI reset - Restore cached global and per-decoder HDM state after reset while the target IOMMU reset block remains active. Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method - Add "cxl_reset" to the PCI reset-method table ahead of FLR. Patch 11: Documentation/ABI: Document CXL Reset PCI reset method - Document the new reset_method value and reset behavior. Patch 12: PCI/CXL: Restore HDM state after CXL bus reset - Restore cached CXL HDM decoder state after cxl_bus reset. The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no runtime provider is available, the kernel emits a warning and continues after the affected HDM ranges have been reserved. Example: echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset Testing: - Built every intermediate patch, plus the v7.3-rc1 baseline, with CONFIG_CXL_BUS=m. All 12 patches passed a full kernel build and `make M=tools/testing/cxl/` independently. - Built and booted the series on an arm64 NVIDIA NVL72 system and successfully exercised CXL Reset through the PCI reset sysfs ABI. - Earlier revision testing ran 100 iterations of cxl_reset and exercised cxl_bus separately with HDM state restoration. References: [1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf [2] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/ [3] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/ [4] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/ [5] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/ Srirangan Madhavan (12): cxl: Move HDM decoder programming helpers cxl: Make HDM commit helpers available to reset code cxl: Share HDM decoder decode logic cxl: Cache decoder settings on PCI devices cxl: Cache endpoint decoder settings during PCI enumeration cxl: Add CXL Device Reset helper cxl: Validate HDM ranges before CXL reset PCI/CXL: Reject CXL Reset on multifunction devices cxl: Restore CXL HDM state after PCI reset PCI/CXL: Expose CXL Reset as a PCI reset method Documentation/ABI: Document CXL Reset PCI reset method PCI/CXL: Restore HDM state after CXL bus reset Documentation/ABI/testing/sysfs-bus-pci | 15 + drivers/cxl/Kconfig | 4 + drivers/cxl/core/Makefile | 2 +- drivers/cxl/core/core.h | 8 + drivers/cxl/core/hdm.c | 324 +++--- drivers/cxl/core/pci.c | 7 +- drivers/cxl/core/port.c | 19 + drivers/cxl/core/regs.c | 4 + drivers/cxl/core/resource.c | 1351 +++++++++++++++++++++++ drivers/cxl/cxl.h | 41 - drivers/pci/bus.c | 2 + drivers/pci/pci.c | 8 + drivers/pci/probe.c | 2 + include/cxl/cxl.h | 116 +- include/linux/pci.h | 8 +- include/uapi/linux/pci_regs.h | 14 + tools/testing/cxl/Kbuild | 1 - 17 files changed, 1715 insertions(+), 211 deletions(-) create mode 100644 drivers/cxl/core/resource.c base-commit: cee9395acd8043be0644b25c34bfa86623f2b935