diff mbox

[2/2] hw/phb3: Ensure PQ bits are cleared in the IVC when masking IRQ

Message ID 1455164733-6560-2-git-send-email-mikey@neuling.org
State Accepted
Headers show

Commit Message

Michael Neuling Feb. 11, 2016, 4:25 a.m. UTC
When we mask an interrupt, we may race with another interrupt coming
in from the hardware.  If this occurs, the P and/or Q bit may end up
being set but we never EOI/clear them.  This could result in a lost
interrupt or the next interrupt that comes in after re-enabling never
being presented.

This patch ensures that when masking an interrupt, any pending P/Q
bits are cleared.

This fixes a bug seen with some CAPI workloads which have lots of
interrupt masking at the same time as high interrupt load.  The fix is
not specific to CAPI though.

Signed-off-by: Michael Neuling <mikey@neuling.org>
---
 hw/phb3.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)

Comments

Vaibhav Jain April 22, 2016, 7:33 a.m. UTC | #1
Tested with genwqe framework to simulate heavy interrupt traffic and cxl
 driver patches at:

 https://github.com/ibm-genwqe/genwqe-user
 http://patchwork.ozlabs.org/patch/613380/
 http://patchwork.ozlabs.org/patch/613381/

 Michael Neuling <mikey@neuling.org> writes:
 > Signed-off-by: Michael Neuling <mikey@neuling.org>
 Tested-by: Vaibhav Jain <vaibhav@linux.vnet.ibm.com>
Andrew Donnellan April 25, 2016, 11:56 p.m. UTC | #2
On 11/02/16 15:25, Michael Neuling wrote:
> When we mask an interrupt, we may race with another interrupt coming
> in from the hardware.  If this occurs, the P and/or Q bit may end up
> being set but we never EOI/clear them.  This could result in a lost
> interrupt or the next interrupt that comes in after re-enabling never
> being presented.
>
> This patch ensures that when masking an interrupt, any pending P/Q
> bits are cleared.
>
> This fixes a bug seen with some CAPI workloads which have lots of
> interrupt masking at the same time as high interrupt load.  The fix is
> not specific to CAPI though.
>
> Signed-off-by: Michael Neuling <mikey@neuling.org>

Tested with an IBM internal CAPI accelerator that hit this issue.

Tested-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
diff mbox

Patch

diff --git a/hw/phb3.c b/hw/phb3.c
index a11f84e..c5c6fe3 100644
--- a/hw/phb3.c
+++ b/hw/phb3.c
@@ -1698,7 +1698,7 @@  static int64_t phb3_msi_set_xive(void *data,
 {
 	struct phb3 *p = data;
 	uint32_t chip, index;
-	uint64_t *cache, ive_num, data64, m_server, m_prio;
+	uint64_t *cache, ive_num, data64, m_server, m_prio, ivc;
 	uint32_t *ive;
 
 	chip = p8_irq_to_chip(isn);
@@ -1743,14 +1743,34 @@  static int64_t phb3_msi_set_xive(void *data,
 	*ive = (m_server << 8) | m_prio;
 	out_be64(p->regs + PHB_IVC_UPDATE, data64);
 
-	/*
-	 * Handle Q bit if we're going to enable the interrupt.
-	 * The OS should make sure the interrupt handler has
-	 * been installed already.
-	 */
 	if (prio != 0xff) {
+		/*
+		 * Handle Q bit if we're going to enable the
+		 * interrupt.  The OS should make sure the interrupt
+		 * handler has been installed already.
+		 */
 		if (phb3_pci_msi_check_q(p, ive_num))
 			phb3_pci_msi_flush_ive(p, ive_num);
+	} else {
+		/* Read from random PHB reg to force flush */
+		in_be64(p->regs + PHB_IVC_UPDATE);
+
+		/* Order with subsequent read of Q */
+		sync();
+
+		/* Clear P, Q and Gen, preserve PE# */
+		ive[1] &= 0x0000ffff;
+
+		/*
+		 * Update the IVC with a match against the old gen
+		 * count. No need to worry about racing with P being
+		 * set in the cache since IRQ is masked at this point.
+		 */
+		ivc = SETFIELD(PHB_IVC_UPDATE_SID, 0ul, ive_num) |
+			PHB_IVC_UPDATE_ENABLE_P |
+			PHB_IVC_UPDATE_ENABLE_Q |
+			PHB_IVC_UPDATE_ENABLE_GEN;
+		out_be64(p->regs + PHB_IVC_UPDATE, ivc);
 	}
 
 	return OPAL_SUCCESS;