mbox series

[SRU,F,0/5] kvm: properly tear down PV features on hibernate

Message ID 20210520133611.39540-1-andrea.righi@canonical.com
Headers show
Series kvm: properly tear down PV features on hibernate | expand

Message

Andrea Righi May 20, 2021, 1:36 p.m. UTC
[Impact]

In LP: #1918694 we applied a fix and a workaround to solve the
hibernation issues on c5.18xlarge. The workaround was in the form of a
SAUCE patch:

  "UBUNTU: SAUCE: aws: kvm: double the size of hv_clock_boot"

It looks like we can replace this workaround with a proper fix, by
applying this patch:

http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/

This is required because various PV features (Async PF, PV EOI, steal
time) work through memory shared with hypervisor and when we restore
from hibernation we must properly tear down all these features to make
sure hypervisor doesn't write to stale locations after we jump to the
previously hibernated kernel.

For this reason it is safe to apply this patch set also to the all the
generic kernels and not just AWS.

[Test plan]

This can be easily tested on AWS (but it should be reproduced by
hibernating any kvm instance with multiple CPUs). Create a c5.18xlarge
instance, run the memory stress test script (the same test script that
we are using to stress test hibernation), trigger the hibernate event,
trigger the resume event. Repeat a couple of times and the problem is
very likely to happen.

[Fix]

On the AWS kernel replace "UBUNTU: SAUCE: aws: kvm: double the size of
hv_clock_boot" with:

http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/

For the other kernels, simply apply this patch set.

The fix has been tested extensively in the AWS infrastructure with
positive results.

[Regression potential]

This new code introduced by the fix can be executed also when a CPU is
put offline, so we may see potential regressions in the KVM CPU
hot-plugging.

Comments

Andrea Righi May 20, 2021, 1:37 p.m. UTC | #1
BugLink: https://bugs.launchpad.net/bugs/1920944

On Thu, May 20, 2021 at 03:36:06PM +0200, Andrea Righi wrote:
> [Impact]
> 
> In LP: #1918694 we applied a fix and a workaround to solve the
> hibernation issues on c5.18xlarge. The workaround was in the form of a
> SAUCE patch:
> 
>   "UBUNTU: SAUCE: aws: kvm: double the size of hv_clock_boot"
> 
> It looks like we can replace this workaround with a proper fix, by
> applying this patch:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> This is required because various PV features (Async PF, PV EOI, steal
> time) work through memory shared with hypervisor and when we restore
> from hibernation we must properly tear down all these features to make
> sure hypervisor doesn't write to stale locations after we jump to the
> previously hibernated kernel.
> 
> For this reason it is safe to apply this patch set also to the all the
> generic kernels and not just AWS.
> 
> [Test plan]
> 
> This can be easily tested on AWS (but it should be reproduced by
> hibernating any kvm instance with multiple CPUs). Create a c5.18xlarge
> instance, run the memory stress test script (the same test script that
> we are using to stress test hibernation), trigger the hibernate event,
> trigger the resume event. Repeat a couple of times and the problem is
> very likely to happen.
> 
> [Fix]
> 
> On the AWS kernel replace "UBUNTU: SAUCE: aws: kvm: double the size of
> hv_clock_boot" with:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> For the other kernels, simply apply this patch set.
> 
> The fix has been tested extensively in the AWS infrastructure with
> positive results.
> 
> [Regression potential]
> 
> This new code introduced by the fix can be executed also when a CPU is
> put offline, so we may see potential regressions in the KVM CPU
> hot-plugging.
Guilherme G. Piccoli May 20, 2021, 2:51 p.m. UTC | #2
On 20/05/2021 10:36, Andrea Righi wrote:
> [Impact]
> 
> In LP: #1918694 we applied a fix and a workaround to solve the
> hibernation issues on c5.18xlarge. The workaround was in the form of a
> SAUCE patch:
> 
>   "UBUNTU: SAUCE: aws: kvm: double the size of hv_clock_boot"
> 
> It looks like we can replace this workaround with a proper fix, by
> applying this patch:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> This is required because various PV features (Async PF, PV EOI, steal
> time) work through memory shared with hypervisor and when we restore
> from hibernation we must properly tear down all these features to make
> sure hypervisor doesn't write to stale locations after we jump to the
> previously hibernated kernel.
> 
> For this reason it is safe to apply this patch set also to the all the
> generic kernels and not just AWS.
> 
> [Test plan]
> 
> This can be easily tested on AWS (but it should be reproduced by
> hibernating any kvm instance with multiple CPUs). Create a c5.18xlarge
> instance, run the memory stress test script (the same test script that
> we are using to stress test hibernation), trigger the hibernate event,
> trigger the resume event. Repeat a couple of times and the problem is
> very likely to happen.
> 
> [Fix]
> 
> On the AWS kernel replace "UBUNTU: SAUCE: aws: kvm: double the size of
> hv_clock_boot" with:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> For the other kernels, simply apply this patch set.
> 
> The fix has been tested extensively in the AWS infrastructure with
> positive results.
> 
> [Regression potential]
> 
> This new code introduced by the fix can be executed also when a CPU is
> put offline, so we may see potential regressions in the KVM CPU
> hot-plugging.
> 

Thanks Andrea! LGTM:

Acked-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
Tim Gardner May 20, 2021, 3:57 p.m. UTC | #3
Acked-by: Tim Gardner <tim.gardner@canonical.com>

 From background chatter I gather this has been thoroughly tested.

On 5/20/21 7:36 AM, Andrea Righi wrote:
> [Impact]
> 
> In LP: #1918694 we applied a fix and a workaround to solve the
> hibernation issues on c5.18xlarge. The workaround was in the form of a
> SAUCE patch:
> 
>    "UBUNTU: SAUCE: aws: kvm: double the size of hv_clock_boot"
> 
> It looks like we can replace this workaround with a proper fix, by
> applying this patch:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> This is required because various PV features (Async PF, PV EOI, steal
> time) work through memory shared with hypervisor and when we restore
> from hibernation we must properly tear down all these features to make
> sure hypervisor doesn't write to stale locations after we jump to the
> previously hibernated kernel.
> 
> For this reason it is safe to apply this patch set also to the all the
> generic kernels and not just AWS.
> 
> [Test plan]
> 
> This can be easily tested on AWS (but it should be reproduced by
> hibernating any kvm instance with multiple CPUs). Create a c5.18xlarge
> instance, run the memory stress test script (the same test script that
> we are using to stress test hibernation), trigger the hibernate event,
> trigger the resume event. Repeat a couple of times and the problem is
> very likely to happen.
> 
> [Fix]
> 
> On the AWS kernel replace "UBUNTU: SAUCE: aws: kvm: double the size of
> hv_clock_boot" with:
> 
> http://next.patchew.org/Linux/20210414123544.1060604-1-vkuznets@redhat.com/
> 
> For the other kernels, simply apply this patch set.
> 
> The fix has been tested extensively in the AWS infrastructure with
> positive results.
> 
> [Regression potential]
> 
> This new code introduced by the fix can be executed also when a CPU is
> put offline, so we may see potential regressions in the KVM CPU
> hot-plugging.
> 
>