| Message ID | 20210623131352.51873-1-paolo.pisati@canonical.com |
|---|---|
| State | New |
| Headers | show |
| Series | [act] UBUNTU: SAUCE: ubuntu_kernel_selftests: disable memory-hotplug | expand |
On 23/06/2021 15:13, Paolo Pisati wrote: > The memory-hotplug test has been intermittently timing out (or trashing the test > VM, see below) on Impish/Hirsute ppc64el and x86-64 for quite some time now. > > Upon further investigation, we found that memory-hotplug has a tendency to spam > the system logs (kernel.log, syslog and the systemd-journal) with thousands and > thousands (up to several GBs) of dump_page() entries like this: Azure will have it disabled with (already applied): https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1897764 > > ... > [ 898.286185] migrating pfn 11c462 failed ret:1 > [ 898.286186] page:00000000491a3636 refcount:3 mapcount:0 mapping:00000000e646cbed index:0xc00066 pfn:0x11c462 > [ 898.286188] memcg:ffff947290991000 > [ 898.286188] aops:def_blk_aops ino:800002 > [ 898.286191] flags: 0x17ffffc0002022(referenced|active|private|node=0|zone=2|lastcpupid=0x1fffff) > [ 898.286193] raw: 0017ffffc0002022 ffffb3618ba03ba8 ffffb3618ba03ba8 ffff947287522ab0 > [ 898.286195] raw: 0000000000c00066 ffff947281729340 00000003ffffffff ffff947290991000 > [ 898.286196] page dumped because: migration failure > ... > > At this point, two things can happen: > > a) the constant flow of printk() slows down the VM to the point a timeout > triggers (either autotest timeout or kernel selftests timeout, it doesn't > matter), terminates memory-hotplug and the VM resume processing the remaning > ubuntu_kernel_selftests jobs > > or > > b) the filesystem fills up to 100%, memory-hotplug fails, but so does every > remaining test jobs since the VM is in an unusable state at this point Why the filesystem fills up? Journald should have strict limits for its logs... unless we keep logging it to a file and ignore any journald MaxUse/KeepFree? > > Given we already disable memory-hotplug for arm* and cloud kernels, and to avoid > having our tests session be trashed by this single test, i propose to disable it > entirely, or at least until a ratelimit solution is put in place. I would prefer to keep memory hotplug. It's a quite nice test for memory reclaim, migration and compaction. It should really stress the mm. > > If you want to reproduce this issue, just provision an openstack instance > (small, medium or large - size doesn't matter) and you will always endup in > scenario "b". > > Signed-off-by: Paolo Pisati <paolo.pisati@canonical.com> > --- > ubuntu_kernel_selftests/control | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/ubuntu_kernel_selftests/control b/ubuntu_kernel_selftests/control > index e2874196..53c5584a 100644 > --- a/ubuntu_kernel_selftests/control > +++ b/ubuntu_kernel_selftests/control > @@ -12,7 +12,7 @@ DOC = "" > > name = 'ubuntu_kernel_selftests' > > -tests = [ 'setup','breakpoints','cpu-hotplug','efivarfs','memfd','memory-hotplug','mount','net','ptrace','seccomp','timers','powerpc','user','ftrace' ] > +tests = [ 'setup','breakpoints','cpu-hotplug','efivarfs','memfd','mount','net','ptrace','seccomp','timers','powerpc','user','ftrace' ] > > # > # The seccomp tests on 4.19+ on non-x86 are known to be fail and > Best regards, Krzysztof
On Wed, Jun 23, 2021 at 3:24 PM Krzysztof Kozlowski < krzysztof.kozlowski@canonical.com> wrote: > > Why the filesystem fills up? Journald should have strict limits for its > logs... unless we keep logging it to a file and ignore any journald > MaxUse/KeepFree? > /var/log/journald is about 1GB, the rest are kernel.log and syslog - 10G is the default openstack fs size: $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/vda1 9.7G 9.7G 0 100% / $ du -hx --max-depth=1 / 9.6G / 5.3G /var 2.2G /usr 2.1G /home ... -rw-r----- 1 syslog adm 1948643300 Jun 23 09:01 /var/log/kern.log -rw-r----- 1 syslog adm 1949290433 Jun 23 14:12 /var/log/syslog $ du -h --max-depth=1 /var/log/journal 970M journal $ tail -n10 /var/log/syslog Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949015] flags: 0x3ffff80000202a(referenced|dirty|active|private) Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949018] raw: 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068 Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949020] raw: 0000000000008000 c00000001653c5d8 00000003ffffffff c00000000dc43000 Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949022] page dumped because: migration failure Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949023] pages's memcg:c00000000dc43000 Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949057] migrating pfn 179a failed ret:1 Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949059] page:000000000f1e9b6f refcount:3 mapcount:0 mapping:00000000652d2e2f index:0x8000 pfn:0x179a Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949061] aops:def_blk_aops ino:fc00001 Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949063] flags: 0x3ffff80000202a(referenced|dirty|active|private) Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949066] raw: 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068
On 23/06/2021 16:32, Paolo Pisati wrote: > On Wed, Jun 23, 2021 at 3:24 PM Krzysztof Kozlowski > <krzysztof.kozlowski@canonical.com > <mailto:krzysztof.kozlowski@canonical.com>> wrote: > > > Why the filesystem fills up? Journald should have strict limits for its > logs... unless we keep logging it to a file and ignore any journald > MaxUse/KeepFree? > > > /var/log/journald is about 1GB, the rest are kernel.log and syslog - 10G > is the default openstack fs size: > > $ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda1 9.7G 9.7G 0 100% / > > $ du -hx --max-depth=1 / > 9.6G / > 5.3G /var > 2.2G /usr > 2.1G /home > ... > > -rw-r----- 1 syslog adm 1948643300 Jun 23 09:01 /var/log/kern.log > -rw-r----- 1 syslog adm 1949290433 Jun 23 14:12 /var/log/syslog It's the old-syslog logging which can eat entire disk without any self-control. :) I understand the problem now and indeed disabling test makes sense. However how about still keep it for few selected (always passing) systems? This could be specific flavor/cloud with free disk space constraint. > > $ du -h --max-depth=1 /var/log/journal > 970M journal > > $ tail -n10 /var/log/syslog > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949015] flags: > 0x3ffff80000202a(referenced|dirty|active|private) > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949018] raw: > 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068 > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949020] raw: > 0000000000008000 c00000001653c5d8 00000003ffffffff c00000000dc43000 > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949022] page dumped > because: migration failure > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949023] pages's > memcg:c00000000dc43000 > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949057] migrating > pfn 179a failed ret:1 > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949059] > page:000000000f1e9b6f refcount:3 mapcount:0 mapping:00000000652d2e2f > index:0x8000 pfn:0x179a > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949061] > aops:def_blk_aops ino:fc00001 > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949063] flags: > 0x3ffff80000202a(referenced|dirty|active|private) > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949066] raw: > 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068 > Best regards, Krzysztof
On Thu, Jun 24, 2021 at 4:03 PM Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> wrote: > > On 23/06/2021 16:32, Paolo Pisati wrote: > > On Wed, Jun 23, 2021 at 3:24 PM Krzysztof Kozlowski > > <krzysztof.kozlowski@canonical.com > > <mailto:krzysztof.kozlowski@canonical.com>> wrote: > > > > > > Why the filesystem fills up? Journald should have strict limits for its > > logs... unless we keep logging it to a file and ignore any journald > > MaxUse/KeepFree? > > > > > > /var/log/journald is about 1GB, the rest are kernel.log and syslog - 10G > > is the default openstack fs size: > > > > $ df -h / > > Filesystem Size Used Avail Use% Mounted on > > /dev/vda1 9.7G 9.7G 0 100% / > > > > $ du -hx --max-depth=1 / > > 9.6G / > > 5.3G /var > > 2.2G /usr > > 2.1G /home > > ... > > > > -rw-r----- 1 syslog adm 1948643300 Jun 23 09:01 /var/log/kern.log > > -rw-r----- 1 syslog adm 1949290433 Jun 23 14:12 /var/log/syslog > > It's the old-syslog logging which can eat entire disk without any > self-control. :) > > I understand the problem now and indeed disabling test makes sense. > However how about still keep it for few selected (always passing) > systems? This could be specific flavor/cloud with free disk space > constraint. > Hello Paolo, As Krzysztof mentioned this test can be useful for testing memory management,if this is just failing for newer release I think we can disable it on H/I. This can be achieved by either: 1. Have a series check to determine if we need to run this. 2. Or make a copy of the control file, name it as control.ubuntu.bionic / control.ubuntu.focal and etc to host different test plans in the ubuntu_kernel_selftest directory and the mem-hp test can be removed from the control file, this control file will be used by other releases. (off-topic, we might need to review our ubuntu_kernel_selftests test plan as there are more tests have been added to the kselftest directory) Thoughts? Thanks Sam > > > > $ du -h --max-depth=1 /var/log/journal > > 970M journal > > > > $ tail -n10 /var/log/syslog > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949015] flags: > > 0x3ffff80000202a(referenced|dirty|active|private) > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949018] raw: > > 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068 > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949020] raw: > > 0000000000008000 c00000001653c5d8 00000003ffffffff c00000000dc43000 > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949022] page dumped > > because: migration failure > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949023] pages's > > memcg:c00000000dc43000 > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949057] migrating > > pfn 179a failed ret:1 > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949059] > > page:000000000f1e9b6f refcount:3 mapcount:0 mapping:00000000652d2e2f > > index:0x8000 pfn:0x179a > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949061] > > aops:def_blk_aops ino:fc00001 > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949063] flags: > > 0x3ffff80000202a(referenced|dirty|active|private) > > Jun 23 09:00:21 ppisati-ppc64el-large kernel: [ 1293.949066] raw: > > 003ffff80000202a c00000000d657960 c00000000d657960 c00000000bb66068 > > > > > Best regards, > Krzysztof > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
On Wed, Jun 30, 2021 at 02:55:03PM +0800, Po-Hsu Lin wrote: > As Krzysztof mentioned this test can be useful for testing memory > management,if this is just failing for newer release I think we can > disable it on H/I. > This can be achieved by either: > 1. Have a series check to determine if we need to run this. > 2. Or make a copy of the control file, name it as > control.ubuntu.bionic / control.ubuntu.focal and etc to host different > test plans in the ubuntu_kernel_selftest directory and the mem-hp test > can be removed from the control file, this control file will be used > by other releases. > > (off-topic, we might need to review our ubuntu_kernel_selftests test > plan as there are more tests have been added to the kselftest > directory) > Thoughts? I noticed the issue only happens when the test tries to offline all the hotpluggable memory while injecting errors - the same test, without memory injection, is actually ratio limited (usually between 2% to 10% of all hotpluggable memory), so i sent this patch upstream: https://lkml.org/lkml/2021/6/30/646 Would you consider it instead of turning off the test completely?
On 02/07/2021 16:14, Paolo Pisati wrote: > On Wed, Jun 30, 2021 at 02:55:03PM +0800, Po-Hsu Lin wrote: > >> As Krzysztof mentioned this test can be useful for testing memory >> management,if this is just failing for newer release I think we can >> disable it on H/I. >> This can be achieved by either: >> 1. Have a series check to determine if we need to run this. >> 2. Or make a copy of the control file, name it as >> control.ubuntu.bionic / control.ubuntu.focal and etc to host different >> test plans in the ubuntu_kernel_selftest directory and the mem-hp test >> can be removed from the control file, this control file will be used >> by other releases. >> >> (off-topic, we might need to review our ubuntu_kernel_selftests test >> plan as there are more tests have been added to the kselftest >> directory) >> Thoughts? > > I noticed the issue only happens when the test tries to offline all the > hotpluggable memory while injecting errors - the same test, without memory > injection, is actually ratio limited (usually between 2% to 10% of all > hotpluggable memory), so i sent this patch upstream: > > https://lkml.org/lkml/2021/6/30/646 > > Would you consider it instead of turning off the test completely? Yes, I think it is good idea. Best regards, Krzysztof
diff --git a/ubuntu_kernel_selftests/control b/ubuntu_kernel_selftests/control index e2874196..53c5584a 100644 --- a/ubuntu_kernel_selftests/control +++ b/ubuntu_kernel_selftests/control @@ -12,7 +12,7 @@ DOC = "" name = 'ubuntu_kernel_selftests' -tests = [ 'setup','breakpoints','cpu-hotplug','efivarfs','memfd','memory-hotplug','mount','net','ptrace','seccomp','timers','powerpc','user','ftrace' ] +tests = [ 'setup','breakpoints','cpu-hotplug','efivarfs','memfd','mount','net','ptrace','seccomp','timers','powerpc','user','ftrace' ] # # The seccomp tests on 4.19+ on non-x86 are known to be fail and
The memory-hotplug test has been intermittently timing out (or trashing the test VM, see below) on Impish/Hirsute ppc64el and x86-64 for quite some time now. Upon further investigation, we found that memory-hotplug has a tendency to spam the system logs (kernel.log, syslog and the systemd-journal) with thousands and thousands (up to several GBs) of dump_page() entries like this: ... [ 898.286185] migrating pfn 11c462 failed ret:1 [ 898.286186] page:00000000491a3636 refcount:3 mapcount:0 mapping:00000000e646cbed index:0xc00066 pfn:0x11c462 [ 898.286188] memcg:ffff947290991000 [ 898.286188] aops:def_blk_aops ino:800002 [ 898.286191] flags: 0x17ffffc0002022(referenced|active|private|node=0|zone=2|lastcpupid=0x1fffff) [ 898.286193] raw: 0017ffffc0002022 ffffb3618ba03ba8 ffffb3618ba03ba8 ffff947287522ab0 [ 898.286195] raw: 0000000000c00066 ffff947281729340 00000003ffffffff ffff947290991000 [ 898.286196] page dumped because: migration failure ... At this point, two things can happen: a) the constant flow of printk() slows down the VM to the point a timeout triggers (either autotest timeout or kernel selftests timeout, it doesn't matter), terminates memory-hotplug and the VM resume processing the remaning ubuntu_kernel_selftests jobs or b) the filesystem fills up to 100%, memory-hotplug fails, but so does every remaining test jobs since the VM is in an unusable state at this point Given we already disable memory-hotplug for arm* and cloud kernels, and to avoid having our tests session be trashed by this single test, i propose to disable it entirely, or at least until a ratelimit solution is put in place. If you want to reproduce this issue, just provision an openstack instance (small, medium or large - size doesn't matter) and you will always endup in scenario "b". Signed-off-by: Paolo Pisati <paolo.pisati@canonical.com> --- ubuntu_kernel_selftests/control | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)