| Message ID | 1538390439-17072-1-git-send-email-ego@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | dfd718a2ed1f678e66749ffe41bdeafedf3f4314 |
| Headers | show |
| Series | [v2] powerpc/rtas: Fix a potential race between CPU-Offline & Migration | expand |
On 10/01/2018 05:40 AM, Gautham R. Shenoy wrote: > From: "Gautham R. Shenoy" <ego@linux.vnet.ibm.com> > > Live Partition Migrations require all the present CPUs to execute the > H_JOIN call, and hence rtas_ibm_suspend_me() onlines any offline CPUs > before initiating the migration for this purpose. > > The commit 85a88cabad57 > ("powerpc/pseries: Disable CPU hotplug across migrations") > disables any CPU-hotplug operations once all the offline CPUs are > brought online to prevent any further state change. Once the > CPU-Hotplug operation is disabled, the code assumes that all the CPUs > are online. > > However, there is a minor window in rtas_ibm_suspend_me() between > onlining the offline CPUs and disabling CPU-Hotplug when a concurrent > CPU-offline operations initiated by the userspace can succeed thereby > nullifying the the aformentioned assumption. In this unlikely case > these offlined CPUs will not call H_JOIN, resulting in a system hang. > > Fix this by verifying that all the present CPUs are actually online > after CPU-Hotplug has been disabled, failing which we restore the > state of the offline CPUs in rtas_ibm_suspend_me() and return an > -EBUSY. > > Cc: Nathan Fontenot <nfont@linux.vnet.ibm.com> > Cc: Tyrel Datwyler <tyreld@linux.vnet.ibm.com> > Suggested-by: Michael Ellerman <mpe@ellerman.id.au> > Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com> Reviewed-by: Nathan Fontenot <nfont@linux.vnet.ibm.com> > --- > v2: Restore the state of the offline CPUs if all CPUs aren't onlined. > > arch/powerpc/kernel/rtas.c | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c > index 2c7ed31..d4468cb 100644 > --- a/arch/powerpc/kernel/rtas.c > +++ b/arch/powerpc/kernel/rtas.c > @@ -982,6 +982,15 @@ int rtas_ibm_suspend_me(u64 handle) > } > > cpu_hotplug_disable(); > + > + /* Check if we raced with a CPU-Offline Operation */ > + if (unlikely(!cpumask_equal(cpu_present_mask, cpu_online_mask))) { > + pr_err("%s: Raced against a concurrent CPU-Offline\n", > + __func__); > + atomic_set(&data.error, -EBUSY); > + goto out_hotplug_enable; > + } > + > stop_topology_update(); > > /* Call function on all CPUs. One of us will make the > @@ -996,6 +1005,8 @@ int rtas_ibm_suspend_me(u64 handle) > printk(KERN_ERR "Error doing global join\n"); > > start_topology_update(); > + > +out_hotplug_enable: > cpu_hotplug_enable(); > > /* Take down CPUs not online prior to suspend */ >
On Mon, 2018-10-01 at 10:40:39 UTC, "Gautham R. Shenoy" wrote: > From: "Gautham R. Shenoy" <ego@linux.vnet.ibm.com> > > Live Partition Migrations require all the present CPUs to execute the > H_JOIN call, and hence rtas_ibm_suspend_me() onlines any offline CPUs > before initiating the migration for this purpose. > > The commit 85a88cabad57 > ("powerpc/pseries: Disable CPU hotplug across migrations") > disables any CPU-hotplug operations once all the offline CPUs are > brought online to prevent any further state change. Once the > CPU-Hotplug operation is disabled, the code assumes that all the CPUs > are online. > > However, there is a minor window in rtas_ibm_suspend_me() between > onlining the offline CPUs and disabling CPU-Hotplug when a concurrent > CPU-offline operations initiated by the userspace can succeed thereby > nullifying the the aformentioned assumption. In this unlikely case > these offlined CPUs will not call H_JOIN, resulting in a system hang. > > Fix this by verifying that all the present CPUs are actually online > after CPU-Hotplug has been disabled, failing which we restore the > state of the offline CPUs in rtas_ibm_suspend_me() and return an > -EBUSY. > > Cc: Nathan Fontenot <nfont@linux.vnet.ibm.com> > Cc: Tyrel Datwyler <tyreld@linux.vnet.ibm.com> > Suggested-by: Michael Ellerman <mpe@ellerman.id.au> > Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com> > Reviewed-by: Nathan Fontenot <nfont@linux.vnet.ibm.com> Applied to powerpc next, thanks. https://git.kernel.org/powerpc/c/dfd718a2ed1f678e66749ffe41bdea cheers
diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c index 2c7ed31..d4468cb 100644 --- a/arch/powerpc/kernel/rtas.c +++ b/arch/powerpc/kernel/rtas.c @@ -982,6 +982,15 @@ int rtas_ibm_suspend_me(u64 handle) } cpu_hotplug_disable(); + + /* Check if we raced with a CPU-Offline Operation */ + if (unlikely(!cpumask_equal(cpu_present_mask, cpu_online_mask))) { + pr_err("%s: Raced against a concurrent CPU-Offline\n", + __func__); + atomic_set(&data.error, -EBUSY); + goto out_hotplug_enable; + } + stop_topology_update(); /* Call function on all CPUs. One of us will make the @@ -996,6 +1005,8 @@ int rtas_ibm_suspend_me(u64 handle) printk(KERN_ERR "Error doing global join\n"); start_topology_update(); + +out_hotplug_enable: cpu_hotplug_enable(); /* Take down CPUs not online prior to suspend */