| Message ID | 201109250056.12545.rjw@sisk.pl |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
Hi. On 25/09/11 08:56, Rafael J. Wysocki wrote: > On Sunday, August 07, 2011, Dave Chinner wrote: >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: >>> From: Rafael J. Wysocki <rjw@sisk.pl> >>> >>> Freeze all filesystems during the freezing of tasks by calling >>> freeze_bdev() for each of them and thaw them during the thawing >>> of tasks with the help of thaw_bdev(). >>> >>> This is needed by hibernation, because some filesystems (e.g. XFS) >>> deadlock with the preallocation of memory used by it if the memory >>> pressure caused by it is too heavy. >>> >>> The additional benefit of this change is that, if something goes >>> wrong after filesystems have been frozen, they will stay in a >>> consistent state and journal replays won't be necessary (e.g. after >>> a failing suspend or resume). In particular, this should help to >>> solve a long-standing issue that in some cases during resume from >>> hibernation the boot loader causes the journal to be replied for the >>> filesystem containing the kernel image and initrd causing it to >>> become inconsistent with the information stored in the hibernation >>> image. >>> >>> This change is based on earlier work by Nigel Cunningham. >>> >>> Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> >>> --- > > Below is an alternative fix, the changelog pretty much explains the idea. > > I've tested it on Toshiba Portege R500, but I don't have an XFS partition > to verify that it really helps, so I'd appreciate it if someone able to > reproduce the original issue could test it and report back. > > Thanks, > Rafael > > --- > From: Rafael J. Wysocki <rjw@sisk.pl> > Subject: PM / Hibernate: Freeze kernel threads after preallocating memory > > There is a problem with the current ordering of hibernate code which > leads to deadlocks in some filesystems' memory shrinkers. Namely, > some filesystems use freezable kernel threads that are inactive when > the hibernate memory preallocation is carried out. Those same > filesystems use memory shrinkers that may be triggered by the > hibernate memory preallocation. If those memory shrinkers wait for > the frozen kernel threads, the hibernate process deadlocks (this > happens with XFS, for one example). > > Apparently, it is not technically viable to redesign the filesystems > in question to avoid the situation described above, so the only > possible solution of this issue is to defer the freezing of kernel > threads until the hibernate memory preallocation is done, which is > implemented by this change. > > Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> TuxOnIce has the following logic at the moment: Freeze all threads. Calculate whether we have enough memory for the image, thaw kernel threads only, free memory and try again if it looks like we don't have enough. I've never heard of a deadlock, though I suppose one would be possible if you had the added complication of userspace drivers/filesystems - it would be good to be able to distinguish and thaw them. It does this prior to the atomic copy, using a user-supplied estimate of the amount of memory drivers will need - the actual amount used is show in debugging info at the end of the cycle. Apart from that, if you have everything else frozen, everything else is pretty deterministic (assuming you don't have any memory leaks in your image-writing code). Regards, Nigel
test results of the patch below: 1. real machine suspends fine but on wakeup, after loading image: hard reset. nvidia gpu => disabled compitz => wakeup worked two times. 2. virtualbox / stress test / xfs and ext4 on 3rd resume, it booted up "normal" like this: [ 3.351813] Freeing unused kernel memory: 568k freed [ 3.460973] Freeing unused kernel memory: 284k freed [ 17.328356] PM: Preparing processes for restore. [ 17.328357] Freezing user space processes ... [ 37.345414] Freezing of tasks failed after 20.01 seconds (1 tasks refusing to freeze, wq_busy=0): [ 37.475244] ffff88001f06fd68 0000000000000086 0000000000000000 0000000000000000 [ 37.526163] ffff88001f06e010 ffff88001f4c4410 0000000000012ec0 ffff88001f06ffd8 [ 37.580110] ffff88001f06ffd8 0000000000012ec0 ffffffff8160d020 ffff88001f4c4410 [ 37.626167] Call Trace: [ 37.626769] [<ffffffff81049944>] schedule+0x55/0x57 [ 37.674925] [<ffffffff81360dbe>] __mutex_lock_common+0x117/0x178 [ 37.792559] [<ffffffff81113ef2>] ? user_path_at+0x61/0x90 [ 37.888501] [<ffffffff81360e35>] __mutex_lock_slowpath+0x16/0x18 [ 37.986966] [<ffffffff81360efb>] mutex_lock+0x1e/0x32 [ 38.086931] [<ffffffffa00a4d43>] show_manufacturer+0x23/0x51 [usbcore] [ 38.212500] [<ffffffff8125cd44>] dev_attr_show+0x22/0x49 [ 38.282319] [<ffffffff810c6f8c>] ? __get_free_pages+0x9/0x38 [ 38.397449] [<ffffffff8115febb>] sysfs_read_file+0xa9/0x12b [ 38.491607] [<ffffffff81107ff6>] vfs_read+0xa6/0x102 [ 38.541994] [<ffffffff81105ebf>] ? do_sys_open+0xee/0x100 [ 38.564907] [<ffffffff8110810b>] sys_read+0x45/0x6c [ 38.578397] [<ffffffff81368412>] system_call_fastpath+0x16/0x1b [ 38.590083] [ 38.598046] Restarting tasks ... done. [ 38.660448] XFS (sda3): Mounting Filesystem restarted the test runs, increased delay between awake and sleep from 20 to 25 sec: 36 time successful hibernate+resume so far. chris On 25.09.2011 00:56, Rafael J. Wysocki wrote: > On Sunday, August 07, 2011, Dave Chinner wrote: >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: >>> From: Rafael J. Wysocki <rjw@sisk.pl> >>> >>> Freeze all filesystems during the freezing of tasks by calling >>> freeze_bdev() for each of them and thaw them during the thawing >>> of tasks with the help of thaw_bdev(). >>> >>> This is needed by hibernation, because some filesystems (e.g. XFS) >>> deadlock with the preallocation of memory used by it if the memory >>> pressure caused by it is too heavy. >>> >>> The additional benefit of this change is that, if something goes >>> wrong after filesystems have been frozen, they will stay in a >>> consistent state and journal replays won't be necessary (e.g. after >>> a failing suspend or resume). In particular, this should help to >>> solve a long-standing issue that in some cases during resume from >>> hibernation the boot loader causes the journal to be replied for the >>> filesystem containing the kernel image and initrd causing it to >>> become inconsistent with the information stored in the hibernation >>> image. >>> >>> This change is based on earlier work by Nigel Cunningham. >>> >>> Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> >>> --- > > Below is an alternative fix, the changelog pretty much explains the idea. > > I've tested it on Toshiba Portege R500, but I don't have an XFS partition > to verify that it really helps, so I'd appreciate it if someone able to > reproduce the original issue could test it and report back. > > Thanks, > Rafael > > --- > From: Rafael J. Wysocki <rjw@sisk.pl> > Subject: PM / Hibernate: Freeze kernel threads after preallocating memory > > There is a problem with the current ordering of hibernate code which > leads to deadlocks in some filesystems' memory shrinkers. Namely, > some filesystems use freezable kernel threads that are inactive when > the hibernate memory preallocation is carried out. Those same > filesystems use memory shrinkers that may be triggered by the > hibernate memory preallocation. If those memory shrinkers wait for > the frozen kernel threads, the hibernate process deadlocks (this > happens with XFS, for one example). > > Apparently, it is not technically viable to redesign the filesystems > in question to avoid the situation described above, so the only > possible solution of this issue is to defer the freezing of kernel > threads until the hibernate memory preallocation is done, which is > implemented by this change. > > Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> > --- > include/linux/freezer.h | 4 +++- > kernel/power/hibernate.c | 12 ++++++++---- > kernel/power/power.h | 3 ++- > kernel/power/process.c | 30 ++++++++++++++++++++---------- > 4 files changed, 33 insertions(+), 16 deletions(-) > > Index: linux/kernel/power/process.c > =================================================================== > --- linux.orig/kernel/power/process.c > +++ linux/kernel/power/process.c > @@ -135,7 +135,7 @@ static int try_to_freeze_tasks(bool sig_ > } > > /** > - * freeze_processes - tell processes to enter the refrigerator > + * freeze_processes - Signal user space processes to enter the refrigerator. > */ > int freeze_processes(void) > { > @@ -143,20 +143,30 @@ int freeze_processes(void) > > printk("Freezing user space processes ... "); > error = try_to_freeze_tasks(true); > - if (error) > - goto Exit; > - printk("done.\n"); > + if (!error) { > + printk("done."); > + oom_killer_disable(); > + } > + printk("\n"); > + BUG_ON(in_atomic()); > + > + return error; > +} > + > +/** > + * freeze_kernel_threads - Make freezable kernel threads go to the refrigerator. > + */ > +int freeze_kernel_threads(void) > +{ > + int error; > > printk("Freezing remaining freezable tasks ... "); > error = try_to_freeze_tasks(false); > - if (error) > - goto Exit; > - printk("done."); > + if (!error) > + printk("done."); > > - oom_killer_disable(); > - Exit: > - BUG_ON(in_atomic()); > printk("\n"); > + BUG_ON(in_atomic()); > > return error; > } > Index: linux/include/linux/freezer.h > =================================================================== > --- linux.orig/include/linux/freezer.h > +++ linux/include/linux/freezer.h > @@ -49,6 +49,7 @@ extern int thaw_process(struct task_stru > > extern void refrigerator(void); > extern int freeze_processes(void); > +extern int freeze_kernel_threads(void); > extern void thaw_processes(void); > > static inline int try_to_freeze(void) > @@ -171,7 +172,8 @@ static inline void clear_freeze_flag(str > static inline int thaw_process(struct task_struct *p) { return 1; } > > static inline void refrigerator(void) {} > -static inline int freeze_processes(void) { BUG(); return 0; } > +static inline int freeze_processes(void) { return -ENOSYS; } > +static inline int freeze_kernel_threads(void) { return -ENOSYS; } > static inline void thaw_processes(void) {} > > static inline int try_to_freeze(void) { return 0; } > Index: linux/kernel/power/power.h > =================================================================== > --- linux.orig/kernel/power/power.h > +++ linux/kernel/power/power.h > @@ -228,7 +228,8 @@ extern int pm_test_level; > #ifdef CONFIG_SUSPEND_FREEZER > static inline int suspend_freeze_processes(void) > { > - return freeze_processes(); > + int error = freeze_processes(); > + return error ? : freeze_kernel_threads(); > } > > static inline void suspend_thaw_processes(void) > Index: linux/kernel/power/hibernate.c > =================================================================== > --- linux.orig/kernel/power/hibernate.c > +++ linux/kernel/power/hibernate.c > @@ -334,13 +334,17 @@ int hibernation_snapshot(int platform_mo > if (error) > goto Close; > > - error = dpm_prepare(PMSG_FREEZE); > - if (error) > - goto Complete_devices; > - > /* Preallocate image memory before shutting down devices. */ > error = hibernate_preallocate_memory(); > if (error) > + goto Close; > + > + error = freeze_kernel_threads(); > + if (error) > + goto Close; > + > + error = dpm_prepare(PMSG_FREEZE); > + if (error) > goto Complete_devices; > > suspend_console();
On Sunday, September 25, 2011, Christoph wrote: > test results of the patch below: > > 1. real machine > > suspends fine but on wakeup, after loading image: hard reset. > nvidia gpu => disabled compitz => wakeup worked two times. Hmm, so there's a separate bug related to NVidia I guess. > 2. virtualbox / stress test / xfs and ext4 > > on 3rd resume, it booted up "normal" like this: > > [ 3.351813] Freeing unused kernel memory: 568k freed > [ 3.460973] Freeing unused kernel memory: 284k freed > > [ 17.328356] PM: Preparing processes for restore. > > [ 17.328357] Freezing user space processes ... > [ 37.345414] Freezing of tasks failed after 20.01 seconds (1 tasks > refusing to freeze, wq_busy=0): > [ 37.475244] ffff88001f06fd68 0000000000000086 0000000000000000 > 0000000000000000 > [ 37.526163] ffff88001f06e010 ffff88001f4c4410 0000000000012ec0 > ffff88001f06ffd8 > [ 37.580110] ffff88001f06ffd8 0000000000012ec0 ffffffff8160d020 > ffff88001f4c4410 > [ 37.626167] Call Trace: > [ 37.626769] [<ffffffff81049944>] schedule+0x55/0x57 > [ 37.674925] [<ffffffff81360dbe>] __mutex_lock_common+0x117/0x178 > [ 37.792559] [<ffffffff81113ef2>] ? user_path_at+0x61/0x90 > [ 37.888501] [<ffffffff81360e35>] __mutex_lock_slowpath+0x16/0x18 > [ 37.986966] [<ffffffff81360efb>] mutex_lock+0x1e/0x32 > [ 38.086931] [<ffffffffa00a4d43>] show_manufacturer+0x23/0x51 [usbcore] > [ 38.212500] [<ffffffff8125cd44>] dev_attr_show+0x22/0x49 > [ 38.282319] [<ffffffff810c6f8c>] ? __get_free_pages+0x9/0x38 > [ 38.397449] [<ffffffff8115febb>] sysfs_read_file+0xa9/0x12b > [ 38.491607] [<ffffffff81107ff6>] vfs_read+0xa6/0x102 > [ 38.541994] [<ffffffff81105ebf>] ? do_sys_open+0xee/0x100 > [ 38.564907] [<ffffffff8110810b>] sys_read+0x45/0x6c > [ 38.578397] [<ffffffff81368412>] system_call_fastpath+0x16/0x1b > [ 38.590083] > [ 38.598046] Restarting tasks ... done. > [ 38.660448] XFS (sda3): Mounting Filesystem > > restarted the test runs, increased delay between awake and sleep from 20 > to 25 sec: > > 36 time successful hibernate+resume so far. OK, cool. Thanks for testing! Rafael > On 25.09.2011 00:56, Rafael J. Wysocki wrote: > > On Sunday, August 07, 2011, Dave Chinner wrote: > >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: > >>> From: Rafael J. Wysocki <rjw@sisk.pl> > >>> > >>> Freeze all filesystems during the freezing of tasks by calling > >>> freeze_bdev() for each of them and thaw them during the thawing > >>> of tasks with the help of thaw_bdev(). > >>> > >>> This is needed by hibernation, because some filesystems (e.g. XFS) > >>> deadlock with the preallocation of memory used by it if the memory > >>> pressure caused by it is too heavy. > >>> > >>> The additional benefit of this change is that, if something goes > >>> wrong after filesystems have been frozen, they will stay in a > >>> consistent state and journal replays won't be necessary (e.g. after > >>> a failing suspend or resume). In particular, this should help to > >>> solve a long-standing issue that in some cases during resume from > >>> hibernation the boot loader causes the journal to be replied for the > >>> filesystem containing the kernel image and initrd causing it to > >>> become inconsistent with the information stored in the hibernation > >>> image. > >>> > >>> This change is based on earlier work by Nigel Cunningham. > >>> > >>> Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> > >>> --- > > > > Below is an alternative fix, the changelog pretty much explains the idea. > > > > I've tested it on Toshiba Portege R500, but I don't have an XFS partition > > to verify that it really helps, so I'd appreciate it if someone able to > > reproduce the original issue could test it and report back. > > > > Thanks, > > Rafael > > > > --- > > From: Rafael J. Wysocki <rjw@sisk.pl> > > Subject: PM / Hibernate: Freeze kernel threads after preallocating memory > > > > There is a problem with the current ordering of hibernate code which > > leads to deadlocks in some filesystems' memory shrinkers. Namely, > > some filesystems use freezable kernel threads that are inactive when > > the hibernate memory preallocation is carried out. Those same > > filesystems use memory shrinkers that may be triggered by the > > hibernate memory preallocation. If those memory shrinkers wait for > > the frozen kernel threads, the hibernate process deadlocks (this > > happens with XFS, for one example). > > > > Apparently, it is not technically viable to redesign the filesystems > > in question to avoid the situation described above, so the only > > possible solution of this issue is to defer the freezing of kernel > > threads until the hibernate memory preallocation is done, which is > > implemented by this change. > > > > Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> > > --- > > include/linux/freezer.h | 4 +++- > > kernel/power/hibernate.c | 12 ++++++++---- > > kernel/power/power.h | 3 ++- > > kernel/power/process.c | 30 ++++++++++++++++++++---------- > > 4 files changed, 33 insertions(+), 16 deletions(-) > > > > Index: linux/kernel/power/process.c > > =================================================================== > > --- linux.orig/kernel/power/process.c > > +++ linux/kernel/power/process.c > > @@ -135,7 +135,7 @@ static int try_to_freeze_tasks(bool sig_ > > } > > > > /** > > - * freeze_processes - tell processes to enter the refrigerator > > + * freeze_processes - Signal user space processes to enter the refrigerator. > > */ > > int freeze_processes(void) > > { > > @@ -143,20 +143,30 @@ int freeze_processes(void) > > > > printk("Freezing user space processes ... "); > > error = try_to_freeze_tasks(true); > > - if (error) > > - goto Exit; > > - printk("done.\n"); > > + if (!error) { > > + printk("done."); > > + oom_killer_disable(); > > + } > > + printk("\n"); > > + BUG_ON(in_atomic()); > > + > > + return error; > > +} > > + > > +/** > > + * freeze_kernel_threads - Make freezable kernel threads go to the refrigerator. > > + */ > > +int freeze_kernel_threads(void) > > +{ > > + int error; > > > > printk("Freezing remaining freezable tasks ... "); > > error = try_to_freeze_tasks(false); > > - if (error) > > - goto Exit; > > - printk("done."); > > + if (!error) > > + printk("done."); > > > > - oom_killer_disable(); > > - Exit: > > - BUG_ON(in_atomic()); > > printk("\n"); > > + BUG_ON(in_atomic()); > > > > return error; > > } > > Index: linux/include/linux/freezer.h > > =================================================================== > > --- linux.orig/include/linux/freezer.h > > +++ linux/include/linux/freezer.h > > @@ -49,6 +49,7 @@ extern int thaw_process(struct task_stru > > > > extern void refrigerator(void); > > extern int freeze_processes(void); > > +extern int freeze_kernel_threads(void); > > extern void thaw_processes(void); > > > > static inline int try_to_freeze(void) > > @@ -171,7 +172,8 @@ static inline void clear_freeze_flag(str > > static inline int thaw_process(struct task_struct *p) { return 1; } > > > > static inline void refrigerator(void) {} > > -static inline int freeze_processes(void) { BUG(); return 0; } > > +static inline int freeze_processes(void) { return -ENOSYS; } > > +static inline int freeze_kernel_threads(void) { return -ENOSYS; } > > static inline void thaw_processes(void) {} > > > > static inline int try_to_freeze(void) { return 0; } > > Index: linux/kernel/power/power.h > > =================================================================== > > --- linux.orig/kernel/power/power.h > > +++ linux/kernel/power/power.h > > @@ -228,7 +228,8 @@ extern int pm_test_level; > > #ifdef CONFIG_SUSPEND_FREEZER > > static inline int suspend_freeze_processes(void) > > { > > - return freeze_processes(); > > + int error = freeze_processes(); > > + return error ? : freeze_kernel_threads(); > > } > > > > static inline void suspend_thaw_processes(void) > > Index: linux/kernel/power/hibernate.c > > =================================================================== > > --- linux.orig/kernel/power/hibernate.c > > +++ linux/kernel/power/hibernate.c > > @@ -334,13 +334,17 @@ int hibernation_snapshot(int platform_mo > > if (error) > > goto Close; > > > > - error = dpm_prepare(PMSG_FREEZE); > > - if (error) > > - goto Complete_devices; > > - > > /* Preallocate image memory before shutting down devices. */ > > error = hibernate_preallocate_memory(); > > if (error) > > + goto Close; > > + > > + error = freeze_kernel_threads(); > > + if (error) > > + goto Close; > > + > > + error = dpm_prepare(PMSG_FREEZE); > > + if (error) > > goto Complete_devices; > > > > suspend_console(); > -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sunday, September 25, 2011, Nigel Cunningham wrote: > Hi. > > On 25/09/11 08:56, Rafael J. Wysocki wrote: > > On Sunday, August 07, 2011, Dave Chinner wrote: > >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: > >>> From: Rafael J. Wysocki <rjw@sisk.pl> > >>> > >>> Freeze all filesystems during the freezing of tasks by calling > >>> freeze_bdev() for each of them and thaw them during the thawing > >>> of tasks with the help of thaw_bdev(). > >>> > >>> This is needed by hibernation, because some filesystems (e.g. XFS) > >>> deadlock with the preallocation of memory used by it if the memory > >>> pressure caused by it is too heavy. > >>> > >>> The additional benefit of this change is that, if something goes > >>> wrong after filesystems have been frozen, they will stay in a > >>> consistent state and journal replays won't be necessary (e.g. after > >>> a failing suspend or resume). In particular, this should help to > >>> solve a long-standing issue that in some cases during resume from > >>> hibernation the boot loader causes the journal to be replied for the > >>> filesystem containing the kernel image and initrd causing it to > >>> become inconsistent with the information stored in the hibernation > >>> image. > >>> > >>> This change is based on earlier work by Nigel Cunningham. > >>> > >>> Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> > >>> --- > > > > Below is an alternative fix, the changelog pretty much explains the idea. > > > > I've tested it on Toshiba Portege R500, but I don't have an XFS partition > > to verify that it really helps, so I'd appreciate it if someone able to > > reproduce the original issue could test it and report back. > > > > Thanks, > > Rafael > > > > --- > > From: Rafael J. Wysocki <rjw@sisk.pl> > > Subject: PM / Hibernate: Freeze kernel threads after preallocating memory > > > > There is a problem with the current ordering of hibernate code which > > leads to deadlocks in some filesystems' memory shrinkers. Namely, > > some filesystems use freezable kernel threads that are inactive when > > the hibernate memory preallocation is carried out. Those same > > filesystems use memory shrinkers that may be triggered by the > > hibernate memory preallocation. If those memory shrinkers wait for > > the frozen kernel threads, the hibernate process deadlocks (this > > happens with XFS, for one example). > > > > Apparently, it is not technically viable to redesign the filesystems > > in question to avoid the situation described above, so the only > > possible solution of this issue is to defer the freezing of kernel > > threads until the hibernate memory preallocation is done, which is > > implemented by this change. > > > > Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl> > > TuxOnIce has the following logic at the moment: Freeze all threads. > Calculate whether we have enough memory for the image, thaw kernel > threads only, free memory and try again if it looks like we don't have > enough. Well, it seems that the freezing of kernel threads in the first step is not necessary. You can do (1) freeze user space, (2) check if there's enough free memory, (3) free memory if necessary, (4) freeze kernel threads instead. Which is what my patch does, actually. :-) > I've never heard of a deadlock, though I suppose one would be > possible if you had the added complication of userspace > drivers/filesystems - it would be good to be able to distinguish and > thaw them. Yes, there is a known problem with FUSE in that area. > It does this prior to the atomic copy, using a user-supplied estimate of > the amount of memory drivers will need - the actual amount used is show > in debugging info at the end of the cycle. Apart from that, if you have > everything else frozen, everything else is pretty deterministic > (assuming you don't have any memory leaks in your image-writing code). Thanks, Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 25.09.2011 15:32, Rafael J. Wysocki wrote: > On Sunday, September 25, 2011, Christoph wrote: >> test results of the patch below: >> >> 1. real machine >> >> suspends fine but on wakeup, after loading image: hard reset. nvidia >> gpu => disabled compitz => wakeup worked two times. > > Hmm, so there's a separate bug related to NVidia I guess. Maybe. Just made another test: the machine (macbook) woke up, loaded image, thaw. It got stuck at vt#1, displaying console with login. Cursor blinking, but no (usb) keyboard or network enabled. Bricked?!! On the other hand I've got another box with nvidia gpu: debian5 32bit 2.6.38.2+ #3 SMP Fri Apr 1 nvidia 260.19.36 It's up since I compiled the kernel: I use it twice a week and I kept it freezed all the time. It was solid rock until today: hard reset on resume. WTF? (I remember this version combo was stable on the macbook but the kernel lacks a solid wireless driver). There are a stupid situation where you can't debug. What else can go wrong? This is off topic but it's a cute kernel crash while I gave the nouveau driver a chance: http://events.ccc.de/camp/2011/wiki/File:Dome22.jpg chris http://events.ccc.de/camp/2011/wiki/DomeTent > >> 2. virtualbox / stress test / xfs and ext4 >> >> on 3rd resume, it booted up "normal" like this: >> >> [ 3.351813] Freeing unused kernel memory: 568k freed [ >> 3.460973] Freeing unused kernel memory: 284k freed >> >> [ 17.328356] PM: Preparing processes for restore. >> ... -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sunday, September 25, 2011, Christoph wrote: > On 25.09.2011 15:32, Rafael J. Wysocki wrote: > > On Sunday, September 25, 2011, Christoph wrote: > >> test results of the patch below: > >> > >> 1. real machine > >> > >> suspends fine but on wakeup, after loading image: hard reset. nvidia > >> gpu => disabled compitz => wakeup worked two times. > > > > Hmm, so there's a separate bug related to NVidia I guess. > > Maybe. > > Just made another test: the machine (macbook) woke up, loaded image, thaw. > It got stuck at vt#1, displaying console with login. Cursor blinking, but > no (usb) keyboard or network enabled. Bricked?!! > > > On the other hand I've got another box with nvidia gpu: > > debian5 32bit > 2.6.38.2+ #3 SMP Fri Apr 1 > nvidia 260.19.36 > > It's up since I compiled the kernel: I use it twice a week and I kept it > freezed all the time. It was solid rock until today: hard reset on resume. > WTF? (I remember this version combo was stable on the macbook but the > kernel lacks a solid wireless driver). If that's an x86_64 system, there is a known bug causing problems like this to happen. There's a patch fixing it, but not conclusive: http://marc.info/?l=linux-kernel&m=131653513414314&w=2 Thanks, Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 26.09.2011 00:10, Rafael J. Wysocki wrote: > On Sunday, September 25, 2011, Christoph wrote: >> On 25.09.2011 15:32, Rafael J. Wysocki wrote: >>> On Sunday, September 25, 2011, Christoph wrote: >>>> test results of the patch below: >>>> >>>> 1. real machine >>>> >>>> suspends fine but on wakeup, after loading image: hard reset. nvidia >>>> gpu => disabled compitz => wakeup worked two times. >>> >>> Hmm, so there's a separate bug related to NVidia I guess. >> >> Maybe. >> >> Just made another test: the machine (macbook) woke up, loaded image, thaw. >> It got stuck at vt#1, displaying console with login. Cursor blinking, but >> no (usb) keyboard or network enabled. Bricked?!! >> >> >> On the other hand I've got another box with nvidia gpu: >> >> debian5 32bit >> 2.6.38.2+ #3 SMP Fri Apr 1 >> nvidia 260.19.36 >> >> It's up since I compiled the kernel: I use it twice a week and I kept it >> freezed all the time. It was solid rock until today: hard reset on resume. >> WTF? (I remember this version combo was stable on the macbook but the >> kernel lacks a solid wireless driver). > > If that's an x86_64 system, there is a known bug causing problems like > this to happen. There's a patch fixing it, but not conclusive: > http://marc.info/?l=linux-kernel&m=131653513414314&w=2 very good. seems to fix resume, at least the 2 times I tested so far. tx! :) chris > > Thanks, > Rafael > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> PM / Freezer: Freeze filesystems while freezing processes (v2) > > On Sunday, August 07, 2011, Dave Chinner wrote: >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: >>> From: Rafael J. Wysocki <rjw@sisk.pl> >>> >>> Freeze all filesystems during the freezing of tasks by calling >>> freeze_bdev() for each of them and thaw them during the thawing of >>> tasks with the help of thaw_bdev(). >>> >>> This is needed by hibernation, because some filesystems (e.g. XFS) >>> deadlock with the preallocation of memory used by it if the memory >>> pressure caused by it is too heavy. >>> ... > > Below is an alternative fix, the changelog pretty much explains the > idea. > > I've tested it on Toshiba Portege R500, but I don't have an XFS > partition to verify that it really helps, so I'd appreciate it if > someone able to reproduce the original issue could test it and report > back. Hi Rafael! Well, the kernel bugtracker is still down and I just like to post my experience with kernel (x64) v3.1-rc8/9 + patches. My machine is a MacBookPro, doomed with 4GB RAM running debian. Bug #1 on the way to hibernate, machine hangs on "PM: Preallocating image memory..." this patch worked for me now for weeks: "[PATCH] PM / Freezer: Freeze filesystems while freezing processes (v2)" https://lkml.org/lkml/2011/9/24/77 I was able to reproduce this bug with virtualbox and tested the patch ~40 cycles. Bug#2 on resume from hibernate, hard reset (x64 only): http://marc.info/?l=linux-kernel&m=131653513414314&w=2 With this patch I haven't got this issue again the last weeks. I wasn't able to reproduce this bug with virtualbox. I only got one pm-hibernate issue. Last line: Disabling non-boot CPUs ... This time I've enabled debug hung task :) schedule_timeout ... workqueue_cpu_callback notifier_call_chain ... __cpu_notify _cpu_down printk disable_nonboot_cpus hibernation_snapshot hibernate ... Any other idea besides the possibility it's caused by evil earth radiation, isn't it? Gruss, chris On 26.09.2011 00:10, Rafael J. Wysocki wrote: > On Sunday, September 25, 2011, Christoph wrote: >> On 25.09.2011 15:32, Rafael J. Wysocki wrote: >>> On Sunday, September 25, 2011, Christoph wrote: >>>> test results of the patch below: >>>> >>>> 1. real machine >>>> >>>> suspends fine but on wakeup, after loading image: hard reset. nvidia >>>> gpu => disabled compitz => wakeup worked two times. >>> >>> Hmm, so there's a separate bug related to NVidia I guess. >> >> Maybe. >> >> Just made another test: the machine (macbook) woke up, loaded image, thaw. >> It got stuck at vt#1, displaying console with login. Cursor blinking, but >> no (usb) keyboard or network enabled. Bricked?!! >> >> >> On the other hand I've got another box with nvidia gpu: >> >> debian5 32bit >> 2.6.38.2+ #3 SMP Fri Apr 1 >> nvidia 260.19.36 >> >> It's up since I compiled the kernel: I use it twice a week and I kept it >> freezed all the time. It was solid rock until today: hard reset on resume. >> WTF? (I remember this version combo was stable on the macbook but the >> kernel lacks a solid wireless driver). > > If that's an x86_64 system, there is a known bug causing problems like > this to happen. There's a patch fixing it, but not conclusive: > http://marc.info/?l=linux-kernel&m=131653513414314&w=2 > > Thanks, > Rafael > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Saturday, October 22, 2011, Christoph wrote: > > PM / Freezer: Freeze filesystems while freezing processes (v2) > > > > On Sunday, August 07, 2011, Dave Chinner wrote: > >> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: > >>> From: Rafael J. Wysocki <rjw@sisk.pl> > >>> > >>> Freeze all filesystems during the freezing of tasks by calling > >>> freeze_bdev() for each of them and thaw them during the thawing of > >>> tasks with the help of thaw_bdev(). > >>> > >>> This is needed by hibernation, because some filesystems (e.g. XFS) > >>> deadlock with the preallocation of memory used by it if the memory > >>> pressure caused by it is too heavy. > >>> > ... > > > > Below is an alternative fix, the changelog pretty much explains the > > idea. > > > > I've tested it on Toshiba Portege R500, but I don't have an XFS > > partition to verify that it really helps, so I'd appreciate it if > > someone able to reproduce the original issue could test it and report > > back. > > Hi Rafael! > > Well, the kernel bugtracker is still down and I just like to post my > experience with kernel (x64) v3.1-rc8/9 + patches. My machine is a > MacBookPro, doomed with 4GB RAM running debian. > > Bug #1 > > on the way to hibernate, machine hangs on > > "PM: Preallocating image memory..." > > this patch worked for me now for weeks: > "[PATCH] PM / Freezer: Freeze filesystems while freezing processes (v2)" > https://lkml.org/lkml/2011/9/24/77 This patch is going to be merged into 3.2. > I was able to reproduce this bug with virtualbox and tested the patch ~40 > cycles. > > Bug#2 > > on resume from hibernate, hard reset (x64 only): > http://marc.info/?l=linux-kernel&m=131653513414314&w=2 > > With this patch I haven't got this issue again the last weeks. Hmm. This issue appears to be still under investigation to me, but perhaps that's taken too much already. Takashi, perhaps you can repost the patch as a proper submission? It would be good to have this regression fixed even if we don't know the real source of it. > I wasn't able to reproduce this bug with virtualbox. > > > > > > I only got one pm-hibernate issue. Last line: > > Disabling non-boot CPUs ... > > This time I've enabled debug hung task :) > > schedule_timeout > ... > workqueue_cpu_callback > notifier_call_chain > ... > __cpu_notify > _cpu_down > printk > disable_nonboot_cpus > hibernation_snapshot > hibernate > ... > > Any other idea besides the possibility it's caused by evil earth > radiation, isn't it? I'm not exactly sure what happened from your description, care to explain? Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
"Rafael J. Wysocki" <rjw@sisk.pl> writes: > On Saturday, October 22, 2011, Christoph wrote: > >>> PM / Freezer: Freeze filesystems while freezing processes (v2) >>> >>> On Sunday, August 07, 2011, Dave Chinner wrote: >>> >>>> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: >>>> >>>>> From: Rafael J. Wysocki <rjw@sisk.pl> >>>>> >>>>> Freeze all filesystems during the freezing of tasks by calling >>>>> freeze_bdev() for each of them and thaw them during the thawing of >>>>> tasks with the help of thaw_bdev(). >>>>> >>>>> This is needed by hibernation, because some filesystems (e.g. XFS) >>>>> deadlock with the preallocation of memory used by it if the memory >>>>> pressure caused by it is too heavy. >>> >>> Below is an alternative fix, the changelog pretty much explains the >>> idea. >>> >>> I've tested it on Toshiba Portege R500, but I don't have an XFS >>> partition to verify that it really helps, so I'd appreciate it if >>> someone able to reproduce the original issue could test it and report >>> back. >> >> Well, the kernel bugtracker is still down and I just like to post my >> experience with kernel (x64) v3.1-rc8/9 + patches. My machine is a >> MacBookPro, doomed with 4GB RAM running debian. >> >> Bug #1 >> >> on the way to hibernate, machine hangs on >> >> "PM: Preallocating image memory..." >> >> this patch worked for me now for weeks: >> "[PATCH] PM / Freezer: Freeze filesystems while freezing processes (v2)" >> https://lkml.org/lkml/2011/9/24/77 > > This patch is going to be merged into 3.2. Hi, I was the original reporter of the Bugzilla issue, just didn't know about this thread until recently. Anyway, I'm running 3.2-rc1 now, which contains the alternative fix, and I can confirm that it indeed works: hibernation does not deadlock on my XFS rooted system anymore during memory preallocation. Thanks everybody for his or her work on the issue! To add something still, preallocation now ends with a couple of seconds of heavy disk activity, but with several seconds of total inactivity beforehand. Is this warranted by some CPU intensive task at that stage?
On Wednesday, November 16, 2011, Ferenc Wagner wrote: > "Rafael J. Wysocki" <rjw@sisk.pl> writes: > > > On Saturday, October 22, 2011, Christoph wrote: > > > >>> PM / Freezer: Freeze filesystems while freezing processes (v2) > >>> > >>> On Sunday, August 07, 2011, Dave Chinner wrote: > >>> > >>>> On Sat, Aug 06, 2011 at 11:17:18PM +0200, Rafael J. Wysocki wrote: > >>>> > >>>>> From: Rafael J. Wysocki <rjw@sisk.pl> > >>>>> > >>>>> Freeze all filesystems during the freezing of tasks by calling > >>>>> freeze_bdev() for each of them and thaw them during the thawing of > >>>>> tasks with the help of thaw_bdev(). > >>>>> > >>>>> This is needed by hibernation, because some filesystems (e.g. XFS) > >>>>> deadlock with the preallocation of memory used by it if the memory > >>>>> pressure caused by it is too heavy. > >>> > >>> Below is an alternative fix, the changelog pretty much explains the > >>> idea. > >>> > >>> I've tested it on Toshiba Portege R500, but I don't have an XFS > >>> partition to verify that it really helps, so I'd appreciate it if > >>> someone able to reproduce the original issue could test it and report > >>> back. > >> > >> Well, the kernel bugtracker is still down and I just like to post my > >> experience with kernel (x64) v3.1-rc8/9 + patches. My machine is a > >> MacBookPro, doomed with 4GB RAM running debian. > >> > >> Bug #1 > >> > >> on the way to hibernate, machine hangs on > >> > >> "PM: Preallocating image memory..." > >> > >> this patch worked for me now for weeks: > >> "[PATCH] PM / Freezer: Freeze filesystems while freezing processes (v2)" > >> https://lkml.org/lkml/2011/9/24/77 > > > > This patch is going to be merged into 3.2. > > Hi, > > I was the original reporter of the Bugzilla issue, just didn't know > about this thread until recently. Anyway, I'm running 3.2-rc1 now, > which contains the alternative fix, and I can confirm that it indeed > works: hibernation does not deadlock on my XFS rooted system anymore > during memory preallocation. Thanks everybody for his or her work on > the issue! > > To add something still, preallocation now ends with a couple of seconds > of heavy disk activity, but with several seconds of total inactivity > beforehand. Is this warranted by some CPU intensive task at that stage? Quilte frankly, I have no idea. Thanks, Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Index: linux/kernel/power/process.c =================================================================== --- linux.orig/kernel/power/process.c +++ linux/kernel/power/process.c @@ -135,7 +135,7 @@ static int try_to_freeze_tasks(bool sig_ } /** - * freeze_processes - tell processes to enter the refrigerator + * freeze_processes - Signal user space processes to enter the refrigerator. */ int freeze_processes(void) { @@ -143,20 +143,30 @@ int freeze_processes(void) printk("Freezing user space processes ... "); error = try_to_freeze_tasks(true); - if (error) - goto Exit; - printk("done.\n"); + if (!error) { + printk("done."); + oom_killer_disable(); + } + printk("\n"); + BUG_ON(in_atomic()); + + return error; +} + +/** + * freeze_kernel_threads - Make freezable kernel threads go to the refrigerator. + */ +int freeze_kernel_threads(void) +{ + int error; printk("Freezing remaining freezable tasks ... "); error = try_to_freeze_tasks(false); - if (error) - goto Exit; - printk("done."); + if (!error) + printk("done."); - oom_killer_disable(); - Exit: - BUG_ON(in_atomic()); printk("\n"); + BUG_ON(in_atomic()); return error; } Index: linux/include/linux/freezer.h =================================================================== --- linux.orig/include/linux/freezer.h +++ linux/include/linux/freezer.h @@ -49,6 +49,7 @@ extern int thaw_process(struct task_stru extern void refrigerator(void); extern int freeze_processes(void); +extern int freeze_kernel_threads(void); extern void thaw_processes(void); static inline int try_to_freeze(void) @@ -171,7 +172,8 @@ static inline void clear_freeze_flag(str static inline int thaw_process(struct task_struct *p) { return 1; } static inline void refrigerator(void) {} -static inline int freeze_processes(void) { BUG(); return 0; } +static inline int freeze_processes(void) { return -ENOSYS; } +static inline int freeze_kernel_threads(void) { return -ENOSYS; } static inline void thaw_processes(void) {} static inline int try_to_freeze(void) { return 0; } Index: linux/kernel/power/power.h =================================================================== --- linux.orig/kernel/power/power.h +++ linux/kernel/power/power.h @@ -228,7 +228,8 @@ extern int pm_test_level; #ifdef CONFIG_SUSPEND_FREEZER static inline int suspend_freeze_processes(void) { - return freeze_processes(); + int error = freeze_processes(); + return error ? : freeze_kernel_threads(); } static inline void suspend_thaw_processes(void) Index: linux/kernel/power/hibernate.c =================================================================== --- linux.orig/kernel/power/hibernate.c +++ linux/kernel/power/hibernate.c @@ -334,13 +334,17 @@ int hibernation_snapshot(int platform_mo if (error) goto Close; - error = dpm_prepare(PMSG_FREEZE); - if (error) - goto Complete_devices; - /* Preallocate image memory before shutting down devices. */ error = hibernate_preallocate_memory(); if (error) + goto Close; + + error = freeze_kernel_threads(); + if (error) + goto Close; + + error = dpm_prepare(PMSG_FREEZE); + if (error) goto Complete_devices; suspend_console();