[v4,1/3] spapr/rtas: disable the decrementer interrupt when a CPU is unplugged

When a CPU is stopped with the 'stop-self' RTAS call, its state
'halted' is switched to 1 and, in this case, the MSR is not taken into
account anymore in the cpu_has_work() routine. Only the pending
hardware interrupts are checked with their LPCR:PECE* enablement bit.

If the DECR timer fires after 'stop-self' is called and before the CPU
'stop' state is reached, the nearly-dead CPU will have some work to do
and the guest will crash. This case happens very frequently with the
not yet upstream P9 XIVE exploitation mode. In XICS mode, the DECR is
occasionally fired but after 'stop' state, so no work is to be done
and the guest survives.

I suspect there is a race between the QEMU mainloop triggering the
timers and the TCG CPU thread but I could not quite identify the root
cause. To be safe, let's disable in the LPCR all the exceptions which
can cause an exit while the CPU is in power-saving mode and reenable
them when the CPU is started.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---

Changes in v4:

 - used the 'lpcr_pm' field of PowerPCCPUClass

Changes in v3:

 - introduced a cpu_ppc_papr_pece_bits() helper to gather the PECE
   bits depending on the CPU family.   
 - enabled Power-saving mode Exit Cause exceptions only on the boot CPU.
 
Changes in v2:

 - used a new routine ppc_cpu_pvr_match() to discriminate CPU versions
 - removed the LPCR:PECE* enablement bit when the CPU is initialized
   if it is a secondary

 hw/ppc/spapr_rtas.c         | 11 +++++++++++
 target/ppc/translate_init.c |  9 ++++++---
 2 files changed, 17 insertions(+), 3 deletions(-)

On Fri, Nov 24, 2017 at 08:05:48AM +0100, Cédric Le Goater wrote:
> When a CPU is stopped with the 'stop-self' RTAS call, its state
> 'halted' is switched to 1 and, in this case, the MSR is not taken into
> account anymore in the cpu_has_work() routine. Only the pending
> hardware interrupts are checked with their LPCR:PECE* enablement bit.
> 
> If the DECR timer fires after 'stop-self' is called and before the CPU
> 'stop' state is reached, the nearly-dead CPU will have some work to do
> and the guest will crash. This case happens very frequently with the
> not yet upstream P9 XIVE exploitation mode. In XICS mode, the DECR is
> occasionally fired but after 'stop' state, so no work is to be done
> and the guest survives.
> 
> I suspect there is a race between the QEMU mainloop triggering the
> timers and the TCG CPU thread but I could not quite identify the root
> cause. To be safe, let's disable in the LPCR all the exceptions which
> can cause an exit while the CPU is in power-saving mode and reenable
> them when the CPU is started.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>

Applied to ppc-for-2.12.

> ---
> 
> Changes in v4:
> 
>  - used the 'lpcr_pm' field of PowerPCCPUClass
> 
> Changes in v3:
> 
>  - introduced a cpu_ppc_papr_pece_bits() helper to gather the PECE
>    bits depending on the CPU family.   
>  - enabled Power-saving mode Exit Cause exceptions only on the boot CPU.
>  
> Changes in v2:
> 
>  - used a new routine ppc_cpu_pvr_match() to discriminate CPU versions
>  - removed the LPCR:PECE* enablement bit when the CPU is initialized
>    if it is a secondary
> 
>  hw/ppc/spapr_rtas.c         | 11 +++++++++++
>  target/ppc/translate_init.c |  9 ++++++---
>  2 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c
> index cdf0b607a0a0..858adb1bf3a9 100644
> --- a/hw/ppc/spapr_rtas.c
> +++ b/hw/ppc/spapr_rtas.c
> @@ -162,6 +162,7 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
>      if (cpu != NULL) {
>          CPUState *cs = CPU(cpu);
>          CPUPPCState *env = &cpu->env;
> +        PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
>  
>          if (!cs->halted) {
>              rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
> @@ -174,6 +175,10 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
>          kvm_cpu_synchronize_state(cs);
>  
>          env->msr = (1ULL << MSR_SF) | (1ULL << MSR_ME);
> +
> +        /* Enable Power-saving mode Exit Cause exceptions for the new CPU */
> +        env->spr[SPR_LPCR] |= pcc->lpcr_pm;
> +
>          env->nip = start;
>          env->gpr[3] = r3;
>          cs->halted = 0;
> @@ -197,6 +202,7 @@ static void rtas_stop_self(PowerPCCPU *cpu, sPAPRMachineState *spapr,
>  {
>      CPUState *cs = CPU(cpu);
>      CPUPPCState *env = &cpu->env;
> +    PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
>  
>      cs->halted = 1;
>      qemu_cpu_kick(cs);
> @@ -210,6 +216,11 @@ static void rtas_stop_self(PowerPCCPU *cpu, sPAPRMachineState *spapr,
>       * no need to bother with specific bits, we just clear it.
>       */
>      env->msr = 0;
> +
> +    /* Disable Power-saving mode Exit Cause exceptions for the CPU.
> +     * This could deliver an interrupt on a dying CPU and crash the
> +     * guest */
> +    env->spr[SPR_LPCR] &= ~pcc->lpcr_pm;
>  }
>  
>  static inline int sysparm_st(target_ulong addr, target_ulong len,
> diff --git a/target/ppc/translate_init.c b/target/ppc/translate_init.c
> index 828d7e778c3b..78a4a581bab7 100644
> --- a/target/ppc/translate_init.c
> +++ b/target/ppc/translate_init.c
> @@ -8911,6 +8911,7 @@ void cpu_ppc_set_papr(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp)
>      CPUPPCState *env = &cpu->env;
>      ppc_spr_t *lpcr = &env->spr_cb[SPR_LPCR];
>      ppc_spr_t *amor = &env->spr_cb[SPR_AMOR];
> +    CPUState *cs = CPU(cpu);
>  
>      cpu->vhyp = vhyp;
>  
> @@ -8953,10 +8954,12 @@ void cpu_ppc_set_papr(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp)
>          }
>      }
>  
> -    /* Also set the power-saving mode bits which depend on the CPU
> -     * family
> +    /* Only enable Power-saving mode Exit Cause exceptions on the boot
> +     * CPU. The RTAS command start-cpu will enable them on secondaries.
>       */
> -    lpcr->default_value |= pcc->lpcr_pm;
> +    if (cs == first_cpu) {
> +        lpcr->default_value |= pcc->lpcr_pm;
> +    }
>  
>      /* We should be followed by a CPU reset but update the active value
>       * just in case...