diff mbox series

[RFC,v8,5/5] iommu/vt-d: don't loop for timeout device-TLB invalidation request forever

Message ID 20231227025923.536148-6-haifeng.zhao@linux.intel.com
State New
Headers show
Series fix vt-d hard lockup when hotplug ATS capable device | expand

Commit Message

Ethan Zhao Dec. 27, 2023, 2:59 a.m. UTC 
When the device-TLB invalidation (ATS invalidation) timeout happens, the
qi_submit_sync() will restart and loop for the invalidation request
forever till it is done, it will block another invalidation thread such
as the fq_timer to issue invalidation request, cause the system lockup as
following

[exception RIP: native_queued_spin_lock_slowpath+92]

RIP: ffffffffa9d1025c RSP: ffffb202f268cdc8 RFLAGS: 00000002

RAX: 0000000000000101 RBX: ffffffffab36c2a0 RCX: 0000000000000000

RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffab36c2a0

RBP: ffffffffab36c2a0 R8: 0000000000000001 R9: 0000000000000000

R10: 0000000000000010 R11: 0000000000000018 R12: 0000000000000000

R13: 0000000000000004 R14: ffff9e10d71b1c88 R15: ffff9e10d71b1980

ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

--- ---

(the left part of exception see the hotplug case of ATS capable device)

If one endpoint device just no response to the device-TLB invalidation
request, but is not gone, it will bring down the whole system, to avoid
such case, don't try the timeout device-TLB request forever.

and as synchronous program model of current qi_submit_sync() implementation
we couldn't wait for the enough time as PCIe spec said 1min+50%, just break
it in current sync model. (PCIe spec r6.1, sec 10.3.1)

Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com>
---
 drivers/iommu/intel/dmar.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Ethan Zhao Dec. 27, 2023, 3:03 a.m. UTC | #1 
On 12/27/2023 10:59 AM, Ethan Zhao wrote:
> When the device-TLB invalidation (ATS invalidation) timeout happens, the
> qi_submit_sync() will restart and loop for the invalidation request
> forever till it is done, it will block another invalidation thread such
> as the fq_timer to issue invalidation request, cause the system lockup as
> following
>
> [exception RIP: native_queued_spin_lock_slowpath+92]
>
> RIP: ffffffffa9d1025c RSP: ffffb202f268cdc8 RFLAGS: 00000002
>
> RAX: 0000000000000101 RBX: ffffffffab36c2a0 RCX: 0000000000000000
>
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffab36c2a0
>
> RBP: ffffffffab36c2a0 R8: 0000000000000001 R9: 0000000000000000
>
> R10: 0000000000000010 R11: 0000000000000018 R12: 0000000000000000
>
> R13: 0000000000000004 R14: ffff9e10d71b1c88 R15: ffff9e10d71b1980
>
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018


--- ---

#12 [ffffb202f268cdc8] native_queued_spin_lock_slowpath at ffffffffa9d1025c

#13 [ffffb202f268cdc8] do_raw_spin_lock at ffffffffa9d121f1

#14 [ffffb202f268cdd8] _raw_spin_lock_irqsave at ffffffffaa51795b

#15 [ffffb202f268cdf8] iommu_flush_dev_iotlb at ffffffffaa20df48

#16 [ffffb202f268ce28] iommu_flush_iova at ffffffffaa20e182

#17 [ffffb202f268ce60] iova_domain_flush at ffffffffaa220e27

#18 [ffffb202f268ce70] fq_flush_timeout at ffffffffaa221c9d

#19 [ffffb202f268cea8] call_timer_fn at ffffffffa9d46661

#20 [ffffb202f268cf08] run_timer_softirq at ffffffffa9d47933

#21 [ffffb202f268cf98] __softirqentry_text_start at ffffffffaa8000e0

#22 [ffffb202f268cff0] asm_call_sysvec_on_stack at ffffffffaa60114f
> --- ---
>
> (the left part of exception see the hotplug case of ATS capable device)
>
> If one endpoint device just no response to the device-TLB invalidation
> request, but is not gone, it will bring down the whole system, to avoid
> such case, don't try the timeout device-TLB request forever.
>
> and as synchronous program model of current qi_submit_sync() implementation
> we couldn't wait for the enough time as PCIe spec said 1min+50%, just break
> it in current sync model. (PCIe spec r6.1, sec 10.3.1)
>
> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com>
> ---
>   drivers/iommu/intel/dmar.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
> index 76903a8bf963..206ab0b7294f 100644
> --- a/drivers/iommu/intel/dmar.c
> +++ b/drivers/iommu/intel/dmar.c
> @@ -1457,7 +1457,7 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,
>   	reclaim_free_desc(qi);
>   	raw_spin_unlock_irqrestore(&qi->q_lock, flags);
>   
> -	if (rc == -EAGAIN)
> +	if (rc == -EAGAIN && type !=QI_DIOTLB_TYPE && type != QI_DEIOTLB_TYPE)
>   		goto restart;
>   
>   	if (iotlb_start_ktime)
Ethan Zhao Dec. 27, 2023, 3:11 a.m. UTC | #2 
On 12/27/2023 10:59 AM, Ethan Zhao wrote:
> When the device-TLB invalidation (ATS invalidation) timeout happens, the
> qi_submit_sync() will restart and loop for the invalidation request
> forever till it is done, it will block another invalidation thread such
> as the fq_timer to issue invalidation request, cause the system lockup as
> following
>
> [exception RIP: native_queued_spin_lock_slowpath+92]
>
> RIP: ffffffffa9d1025c RSP: ffffb202f268cdc8 RFLAGS: 00000002
>
> RAX: 0000000000000101 RBX: ffffffffab36c2a0 RCX: 0000000000000000
>
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffab36c2a0
>
> RBP: ffffffffab36c2a0 R8: 0000000000000001 R9: 0000000000000000
>
> R10: 0000000000000010 R11: 0000000000000018 R12: 0000000000000000
>
> R13: 0000000000000004 R14: ffff9e10d71b1c88 R15: ffff9e10d71b1980
>
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

Wrong past, please ignore this series.


Thanks,

Ethan


>
> --- ---
>
> (the left part of exception see the hotplug case of ATS capable device)
>
> If one endpoint device just no response to the device-TLB invalidation
> request, but is not gone, it will bring down the whole system, to avoid
> such case, don't try the timeout device-TLB request forever.
>
> and as synchronous program model of current qi_submit_sync() implementation
> we couldn't wait for the enough time as PCIe spec said 1min+50%, just break
> it in current sync model. (PCIe spec r6.1, sec 10.3.1)
>
> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com>
> ---
>   drivers/iommu/intel/dmar.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
> index 76903a8bf963..206ab0b7294f 100644
> --- a/drivers/iommu/intel/dmar.c
> +++ b/drivers/iommu/intel/dmar.c
> @@ -1457,7 +1457,7 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,
>   	reclaim_free_desc(qi);
>   	raw_spin_unlock_irqrestore(&qi->q_lock, flags);
>   
> -	if (rc == -EAGAIN)
> +	if (rc == -EAGAIN && type !=QI_DIOTLB_TYPE && type != QI_DEIOTLB_TYPE)
>   		goto restart;
>   
>   	if (iotlb_start_ktime)
diff mbox series

Patch

diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
index 76903a8bf963..206ab0b7294f 100644
--- a/drivers/iommu/intel/dmar.c
+++ b/drivers/iommu/intel/dmar.c
@@ -1457,7 +1457,7 @@  int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,
 	reclaim_free_desc(qi);
 	raw_spin_unlock_irqrestore(&qi->q_lock, flags);
 
-	if (rc == -EAGAIN)
+	if (rc == -EAGAIN && type !=QI_DIOTLB_TYPE && type != QI_DEIOTLB_TYPE)
 		goto restart;
 
 	if (iotlb_start_ktime)