| Message ID | 20260923071659.1161191-2-seyeong.kim@canonical.com |
|---|---|
| State | New |
| Headers | show |
| Series | HMC error triggers cascading PF resets and tears down RDMA connections | expand |
On Wed, Sep 23, 2026 at 3:17 PM Seyeong Kim via kernel-team < kernel-team@lists.ubuntu.com> wrote: > BugLink: https://bugs.launchpad.net/bugs/2167995 > > The irdma driver currently issues an unconditional PF reset whenever the > HMC Error interrupt (PFINT_OICR bit 26) fires: > > if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { > ibdev_err(&iwdev->ibdev, "HMC Error\n"); > iwdev->rf->reset = true; > } > > request_reset() issues an IIDC_PFR to ice. In practice a single HMC_ERR > can trigger cascading PF resets, IOMMU faults during teardown, and > teardown of every RDMA connection on the device. > > i40e handles the identically-named interrupt by reading > PFHMC_ERRORINFO and PFHMC_ERRORDATA and logging them without touching > device state; see commit 9c010ee0ea5f ("i40e: Suppress HMC error to > Interrupt message level") which removed the reset as "not necessary". > This patch mirrors that handling on irdma. > > With this change, repeated HMC_ERR no longer produces a reset storm and > RDMA traffic on the device continues uninterrupted. > > Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com> > Link: > https://patch.msgid.link/20260619050044.1807044-1-seyeong.kim@canonical.com > Signed-off-by: Leon Romanovsky <leon@kernel.org> > (backported from commit 4dc9c884c0aafac1e8d37536d88d485135e26883) > missing backport note. > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com> > --- > drivers/infiniband/hw/irdma/i40iw_hw.c | 4 +++- > drivers/infiniband/hw/irdma/icrdma_hw.c | 2 ++ > drivers/infiniband/hw/irdma/icrdma_hw.h | 2 ++ > drivers/infiniband/hw/irdma/irdma.h | 2 ++ > drivers/infiniband/hw/irdma/main.c | 8 ++++++-- > 5 files changed, 15 insertions(+), 3 deletions(-) > > diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.c > b/drivers/infiniband/hw/irdma/i40iw_hw.c > index 50299f58b6b3..e35610f673d7 100644 > --- a/drivers/infiniband/hw/irdma/i40iw_hw.c > +++ b/drivers/infiniband/hw/irdma/i40iw_hw.c > @@ -29,7 +29,9 @@ static u32 i40iw_regs[IRDMA_MAX_REGS] = { > I40E_PFHMC_PDINV, > I40E_GLHMC_VFPDINV(0), > I40E_GLPE_CRITERR, > - 0xffffffff /* PFINT_RATEN not used in FPK */ > + 0xffffffff, /* PFINT_RATEN not used in FPK */ > + 0xffffffff, /* PFHMC_ERRORINFO not used in FPK */ > + 0xffffffff /* PFHMC_ERRORDATA not used in FPK */ > }; > > static u32 i40iw_stat_offsets_32[IRDMA_HW_STAT_INDEX_MAX_32] = { > diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.c > b/drivers/infiniband/hw/irdma/icrdma_hw.c > index 5986fd906308..344d32283878 100644 > --- a/drivers/infiniband/hw/irdma/icrdma_hw.c > +++ b/drivers/infiniband/hw/irdma/icrdma_hw.c > @@ -29,6 +29,8 @@ static u32 icrdma_regs[IRDMA_MAX_REGS] = { > GLHMC_VFPDINV(0), > GLPE_CRITERR, > GLINT_RATE(0), > + PFHMC_ERRORINFO, > + PFHMC_ERRORDATA, > }; > > static u64 icrdma_masks[IRDMA_MAX_MASKS] = { > diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.h > b/drivers/infiniband/hw/irdma/icrdma_hw.h > index b65c463abf0b..86d04f2eb569 100644 > --- a/drivers/infiniband/hw/irdma/icrdma_hw.h > +++ b/drivers/infiniband/hw/irdma/icrdma_hw.h > @@ -40,6 +40,8 @@ > #define GLHMC_VFPDINV(_i) (0x00528300 + ((_i) * 4)) /* _i=0...31 */ > #define GLPE_CRITERR 0x00534000 > #define GLINT_RATE(_INT) (0x0015A000 + ((_INT) * 4)) /* _i=0...2047 > */ /* Reset Source: CORER */ > +#define PFHMC_ERRORINFO 0x00520400 > +#define PFHMC_ERRORDATA 0x00520500 > > #define ICRDMA_DB_ADDR_OFFSET (8 * 1024 * 1024 - 64 * 1024) > > diff --git a/drivers/infiniband/hw/irdma/irdma.h > b/drivers/infiniband/hw/irdma/irdma.h > index 4789e85d717b..ca259ce1d747 100644 > --- a/drivers/infiniband/hw/irdma/irdma.h > +++ b/drivers/infiniband/hw/irdma/irdma.h > @@ -57,6 +57,8 @@ enum irdma_registers { > IRDMA_GLHMC_VFPDINV, > IRDMA_GLPE_CRITERR, > IRDMA_GLINT_RATE, > + IRDMA_PFHMC_ERRORINFO, > + IRDMA_PFHMC_ERRORDATA, > IRDMA_MAX_REGS, /* Must be last entry */ > }; > > diff --git a/drivers/infiniband/hw/irdma/main.c > b/drivers/infiniband/hw/irdma/main.c > index 17ba7bb80e65..47f4a98adc20 100644 > --- a/drivers/infiniband/hw/irdma/main.c > +++ b/drivers/infiniband/hw/irdma/main.c > @@ -128,8 +128,12 @@ static void irdma_iidc_event_handler(struct ice_pf > *pf, struct iidc_event *event > } > } > if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { > - ibdev_err(&iwdev->ibdev, "HMC Error\n"); > - iwdev->rf->reset = true; > + u32 hmc_errinfo = > readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORINFO]); > + u32 hmc_errdata = > readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORDATA]); > + > + /* Log diagnostics; do not reset here. */ > + ibdev_warn(&iwdev->ibdev, "HMC Error: > errinfo=0x%08x errdata=0x%08x\n", > + hmc_errinfo, hmc_errdata); > } > if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) { > ibdev_err(&iwdev->ibdev, "PE Push Error\n"); > -- > 2.43.0 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team >
diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.c b/drivers/infiniband/hw/irdma/i40iw_hw.c index 50299f58b6b3..e35610f673d7 100644 --- a/drivers/infiniband/hw/irdma/i40iw_hw.c +++ b/drivers/infiniband/hw/irdma/i40iw_hw.c @@ -29,7 +29,9 @@ static u32 i40iw_regs[IRDMA_MAX_REGS] = { I40E_PFHMC_PDINV, I40E_GLHMC_VFPDINV(0), I40E_GLPE_CRITERR, - 0xffffffff /* PFINT_RATEN not used in FPK */ + 0xffffffff, /* PFINT_RATEN not used in FPK */ + 0xffffffff, /* PFHMC_ERRORINFO not used in FPK */ + 0xffffffff /* PFHMC_ERRORDATA not used in FPK */ }; static u32 i40iw_stat_offsets_32[IRDMA_HW_STAT_INDEX_MAX_32] = { diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.c b/drivers/infiniband/hw/irdma/icrdma_hw.c index 5986fd906308..344d32283878 100644 --- a/drivers/infiniband/hw/irdma/icrdma_hw.c +++ b/drivers/infiniband/hw/irdma/icrdma_hw.c @@ -29,6 +29,8 @@ static u32 icrdma_regs[IRDMA_MAX_REGS] = { GLHMC_VFPDINV(0), GLPE_CRITERR, GLINT_RATE(0), + PFHMC_ERRORINFO, + PFHMC_ERRORDATA, }; static u64 icrdma_masks[IRDMA_MAX_MASKS] = { diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.h b/drivers/infiniband/hw/irdma/icrdma_hw.h index b65c463abf0b..86d04f2eb569 100644 --- a/drivers/infiniband/hw/irdma/icrdma_hw.h +++ b/drivers/infiniband/hw/irdma/icrdma_hw.h @@ -40,6 +40,8 @@ #define GLHMC_VFPDINV(_i) (0x00528300 + ((_i) * 4)) /* _i=0...31 */ #define GLPE_CRITERR 0x00534000 #define GLINT_RATE(_INT) (0x0015A000 + ((_INT) * 4)) /* _i=0...2047 */ /* Reset Source: CORER */ +#define PFHMC_ERRORINFO 0x00520400 +#define PFHMC_ERRORDATA 0x00520500 #define ICRDMA_DB_ADDR_OFFSET (8 * 1024 * 1024 - 64 * 1024) diff --git a/drivers/infiniband/hw/irdma/irdma.h b/drivers/infiniband/hw/irdma/irdma.h index 4789e85d717b..ca259ce1d747 100644 --- a/drivers/infiniband/hw/irdma/irdma.h +++ b/drivers/infiniband/hw/irdma/irdma.h @@ -57,6 +57,8 @@ enum irdma_registers { IRDMA_GLHMC_VFPDINV, IRDMA_GLPE_CRITERR, IRDMA_GLINT_RATE, + IRDMA_PFHMC_ERRORINFO, + IRDMA_PFHMC_ERRORDATA, IRDMA_MAX_REGS, /* Must be last entry */ }; diff --git a/drivers/infiniband/hw/irdma/main.c b/drivers/infiniband/hw/irdma/main.c index 17ba7bb80e65..47f4a98adc20 100644 --- a/drivers/infiniband/hw/irdma/main.c +++ b/drivers/infiniband/hw/irdma/main.c @@ -128,8 +128,12 @@ static void irdma_iidc_event_handler(struct ice_pf *pf, struct iidc_event *event } } if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { - ibdev_err(&iwdev->ibdev, "HMC Error\n"); - iwdev->rf->reset = true; + u32 hmc_errinfo = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORINFO]); + u32 hmc_errdata = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORDATA]); + + /* Log diagnostics; do not reset here. */ + ibdev_warn(&iwdev->ibdev, "HMC Error: errinfo=0x%08x errdata=0x%08x\n", + hmc_errinfo, hmc_errdata); } if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) { ibdev_err(&iwdev->ibdev, "PE Push Error\n");
BugLink: https://bugs.launchpad.net/bugs/2167995 The irdma driver currently issues an unconditional PF reset whenever the HMC Error interrupt (PFINT_OICR bit 26) fires: if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { ibdev_err(&iwdev->ibdev, "HMC Error\n"); iwdev->rf->reset = true; } request_reset() issues an IIDC_PFR to ice. In practice a single HMC_ERR can trigger cascading PF resets, IOMMU faults during teardown, and teardown of every RDMA connection on the device. i40e handles the identically-named interrupt by reading PFHMC_ERRORINFO and PFHMC_ERRORDATA and logging them without touching device state; see commit 9c010ee0ea5f ("i40e: Suppress HMC error to Interrupt message level") which removed the reset as "not necessary". This patch mirrors that handling on irdma. With this change, repeated HMC_ERR no longer produces a reset storm and RDMA traffic on the device continues uninterrupted. Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com> Link: https://patch.msgid.link/20260619050044.1807044-1-seyeong.kim@canonical.com Signed-off-by: Leon Romanovsky <leon@kernel.org> (backported from commit 4dc9c884c0aafac1e8d37536d88d485135e26883) Signed-off-by: Gerald Yang <gerald.yang@canonical.com> Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com> --- drivers/infiniband/hw/irdma/i40iw_hw.c | 4 +++- drivers/infiniband/hw/irdma/icrdma_hw.c | 2 ++ drivers/infiniband/hw/irdma/icrdma_hw.h | 2 ++ drivers/infiniband/hw/irdma/irdma.h | 2 ++ drivers/infiniband/hw/irdma/main.c | 8 ++++++-- 5 files changed, 15 insertions(+), 3 deletions(-)