Patchwork ext4: Rate limit printk in buffer_io_error()

login
register
mail settings
Submitter Anatol Pomazau
Date July 9, 2013, 11:01 p.m.
Message ID <1373410898-26826-1-git-send-email-anatol@google.com>
Download mbox | patch
Permalink /patch/257893/
State Accepted
Headers show

Comments

Anatol Pomazau - July 9, 2013, 11:01 p.m.
From: Anatol Pomozov <anatol.pomozov@gmail.com>

If there are a lot of outstanding buffered IOs when a device is
taken offline (due to hardware errors etc), ext4_end_bio prints
out a message for each failed logical block. While this is desirable,
we see thousands of such lines being printed out before the
serial console gets overwhelmed, causing ext4_end_bio() wait for
the printk to complete.

This in itself isn't a disaster, except for the detail that this
function is being called with the queue lock held.
This causes any other function in the block layer
to spin on its spin_lock_irqsave while the serial console is
draining. If NMI watchdog is enabled on this machine then it
eventually comes along and shoots the machine in the head.

The end result is that losing any one disk causes the machine to
go down. This patch rate limits the printk to bandaid around the
problem.

Tested: xfstests
Change-Id: I8ab5690dcf4f3a67e78be147d45e489fdf4a88d8
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
---
 fs/ext4/page-io.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Theodore Ts'o - July 12, 2013, 2:44 a.m.
On Tue, Jul 09, 2013 at 04:01:38PM -0700, Anatol Pomazau wrote:
> From: Anatol Pomozov <anatol.pomozov@gmail.com>
> 
> If there are a lot of outstanding buffered IOs when a device is
> taken offline (due to hardware errors etc), ext4_end_bio prints
> out a message for each failed logical block. While this is desirable,
> we see thousands of such lines being printed out before the
> serial console gets overwhelmed, causing ext4_end_bio() wait for
> the printk to complete.
> 
> This in itself isn't a disaster, except for the detail that this
> function is being called with the queue lock held.
> This causes any other function in the block layer
> to spin on its spin_lock_irqsave while the serial console is
> draining. If NMI watchdog is enabled on this machine then it
> eventually comes along and shoots the machine in the head.
> 
> The end result is that losing any one disk causes the machine to
> go down. This patch rate limits the printk to bandaid around the
> problem.
> 
> Tested: xfstests
> Change-Id: I8ab5690dcf4f3a67e78be147d45e489fdf4a88d8
> Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>

Thanks, applied.

					- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c
index 4acf1f7..4c9d5e7 100644
--- a/fs/ext4/page-io.c
+++ b/fs/ext4/page-io.c
@@ -25,6 +25,7 @@ 
 #include <linux/kernel.h>
 #include <linux/slab.h>
 #include <linux/mm.h>
+#include <linux/ratelimit.h>
 
 #include "ext4_jbd2.h"
 #include "xattr.h"
@@ -214,7 +215,7 @@  ext4_io_end_t *ext4_init_io_end(struct inode *inode, gfp_t flags)
 static void buffer_io_error(struct buffer_head *bh)
 {
 	char b[BDEVNAME_SIZE];
-	printk(KERN_ERR "Buffer I/O error on device %s, logical block %llu\n",
+	printk_ratelimited(KERN_ERR "Buffer I/O error on device %s, logical block %llu\n",
 			bdevname(bh->b_bdev, b),
 			(unsigned long long)bh->b_blocknr);
 }