Message ID | 20181004104640.28326-1-jack@suse.cz |
---|---|
State | Accepted, archived |
Headers | show |
Series | jbd2: Fix use after free in jbd2_log_do_checkpoint() | expand |
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > The code cleaning transaction's lists of checkpoint buffers has a bug > where it increases bh refcount only after releasing > journal->j_list_lock. Thus the following race is possible: > > CPU0 CPU1 > jbd2_log_do_checkpoint() > jbd2_journal_try_to_free_buffers() > __journal_try_to_free_buffer(bh) > ... > while (transaction->t_checkpoint_io_list) > ... > if (buffer_locked(bh)) { > > <-- IO completes now, buffer gets unlocked --> > > spin_unlock(&journal->j_list_lock); > spin_lock(&journal->j_list_lock); > __jbd2_journal_remove_checkpoint(jh); > spin_unlock(&journal->j_list_lock); > try_to_free_buffers(page); > get_bh(bh) <-- accesses freed bh > > Fix the problem by grabbing bh reference before unlocking > journal->j_list_lock. Hi Jan, nice catch. The patch looks good, you can add Reviewed-by: Lukas Czerner <lczerner@redhat.com> Btw, do you by any chance have a reproducer for this ? -Lukas > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 > Fixes: be1158cc615fd723552f0d9912087423c7cadda5 > Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com > CC: stable@vger.kernel.org > Signed-off-by: Jan Kara <jack@suse.cz> > --- > fs/jbd2/checkpoint.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c > index c125d662777c..26f8d7e46462 100644 > --- a/fs/jbd2/checkpoint.c > +++ b/fs/jbd2/checkpoint.c > @@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) > bh = jh2bh(jh); > > if (buffer_locked(bh)) { > - spin_unlock(&journal->j_list_lock); > get_bh(bh); > + spin_unlock(&journal->j_list_lock); > wait_on_buffer(bh); > /* the journal_head may have gone by now */ > BUFFER_TRACE(bh, "brelse"); > @@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) > jh = transaction->t_checkpoint_io_list; > bh = jh2bh(jh); > if (buffer_locked(bh)) { > - spin_unlock(&journal->j_list_lock); > get_bh(bh); > + spin_unlock(&journal->j_list_lock); > wait_on_buffer(bh); > /* the journal_head may have gone by now */ > BUFFER_TRACE(bh, "brelse"); > -- > 2.16.4 >
On Thu 04-10-18 13:50:12, Lukas Czerner wrote: > On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > > The code cleaning transaction's lists of checkpoint buffers has a bug > > where it increases bh refcount only after releasing > > journal->j_list_lock. Thus the following race is possible: > > > > CPU0 CPU1 > > jbd2_log_do_checkpoint() > > jbd2_journal_try_to_free_buffers() > > __journal_try_to_free_buffer(bh) > > ... > > while (transaction->t_checkpoint_io_list) > > ... > > if (buffer_locked(bh)) { > > > > <-- IO completes now, buffer gets unlocked --> > > > > spin_unlock(&journal->j_list_lock); > > spin_lock(&journal->j_list_lock); > > __jbd2_journal_remove_checkpoint(jh); > > spin_unlock(&journal->j_list_lock); > > try_to_free_buffers(page); > > get_bh(bh) <-- accesses freed bh > > > > Fix the problem by grabbing bh reference before unlocking > > journal->j_list_lock. > > Hi Jan, > > nice catch. The patch looks good, you can add > > Reviewed-by: Lukas Czerner <lczerner@redhat.com> > > Btw, do you by any chance have a reproducer for this ? No, syzbot hit it but the race window is really small so I don't think you can create reasonably reliable reproducer... Honza
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > The code cleaning transaction's lists of checkpoint buffers has a bug > where it increases bh refcount only after releasing > journal->j_list_lock. Thus the following race is possible: > > CPU0 CPU1 > jbd2_log_do_checkpoint() > jbd2_journal_try_to_free_buffers() > __journal_try_to_free_buffer(bh) > ... > while (transaction->t_checkpoint_io_list) > ... > if (buffer_locked(bh)) { > > <-- IO completes now, buffer gets unlocked --> > > spin_unlock(&journal->j_list_lock); > spin_lock(&journal->j_list_lock); > __jbd2_journal_remove_checkpoint(jh); > spin_unlock(&journal->j_list_lock); > try_to_free_buffers(page); > get_bh(bh) <-- accesses freed bh > > Fix the problem by grabbing bh reference before unlocking > journal->j_list_lock. > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 > Fixes: be1158cc615fd723552f0d9912087423c7cadda5 Nit, this normally looks like: Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()") And this is created by: git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n" thanks greg k-h
On Thu 04-10-18 08:44:02, Greg KH wrote: > On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > > The code cleaning transaction's lists of checkpoint buffers has a bug > > where it increases bh refcount only after releasing > > journal->j_list_lock. Thus the following race is possible: > > > > CPU0 CPU1 > > jbd2_log_do_checkpoint() > > jbd2_journal_try_to_free_buffers() > > __journal_try_to_free_buffer(bh) > > ... > > while (transaction->t_checkpoint_io_list) > > ... > > if (buffer_locked(bh)) { > > > > <-- IO completes now, buffer gets unlocked --> > > > > spin_unlock(&journal->j_list_lock); > > spin_lock(&journal->j_list_lock); > > __jbd2_journal_remove_checkpoint(jh); > > spin_unlock(&journal->j_list_lock); > > try_to_free_buffers(page); > > get_bh(bh) <-- accesses freed bh > > > > Fix the problem by grabbing bh reference before unlocking > > journal->j_list_lock. > > > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 > > Fixes: be1158cc615fd723552f0d9912087423c7cadda5 > > Nit, this normally looks like: > > Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()") > > And this is created by: > git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n" OK, ok, will do next time. The full git commit ID is faster to cut-n-paste which shows how lazy I'm ;) Honza
On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote: > On Thu 04-10-18 08:44:02, Greg KH wrote: > > On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > > > The code cleaning transaction's lists of checkpoint buffers has a bug > > > where it increases bh refcount only after releasing > > > journal->j_list_lock. Thus the following race is possible: > > > > > > CPU0 CPU1 > > > jbd2_log_do_checkpoint() > > > jbd2_journal_try_to_free_buffers() > > > __journal_try_to_free_buffer(bh) > > > ... > > > while (transaction->t_checkpoint_io_list) > > > ... > > > if (buffer_locked(bh)) { > > > > > > <-- IO completes now, buffer gets unlocked --> > > > > > > spin_unlock(&journal->j_list_lock); > > > spin_lock(&journal->j_list_lock); > > > __jbd2_journal_remove_checkpoint(jh); > > > spin_unlock(&journal->j_list_lock); > > > try_to_free_buffers(page); > > > get_bh(bh) <-- accesses freed bh > > > > > > Fix the problem by grabbing bh reference before unlocking > > > journal->j_list_lock. > > > > > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 > > > Fixes: be1158cc615fd723552f0d9912087423c7cadda5 > > > > Nit, this normally looks like: > > > > Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()") > > > > And this is created by: > > git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n" > > OK, ok, will do next time. The full git commit ID is faster to cut-n-paste > which shows how lazy I'm ;) I have this in the .gitconfig [pretty] fixes = Fixes: %h (\"%s\") which helps. I think I found it in the docs somewhere. -Lukas > > Honza > -- > Jan Kara <jack@suse.com> > SUSE Labs, CR
On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote: > > OK, ok, will do next time. The full git commit ID is faster to cut-n-paste > which shows how lazy I'm ;) No worires, I'll fix that up for you when I apply it. - Ted
On Thu 04-10-18 17:22:18, Theodore Y. Ts'o wrote: > On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote: > > > > OK, ok, will do next time. The full git commit ID is faster to cut-n-paste > > which shows how lazy I'm ;) > > No worires, I'll fix that up for you when I apply it. Thanks! Honza
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote: > The code cleaning transaction's lists of checkpoint buffers has a bug > where it increases bh refcount only after releasing > journal->j_list_lock. Thus the following race is possible: > > CPU0 CPU1 > jbd2_log_do_checkpoint() > jbd2_journal_try_to_free_buffers() > __journal_try_to_free_buffer(bh) > ... > while (transaction->t_checkpoint_io_list) > ... > if (buffer_locked(bh)) { > > <-- IO completes now, buffer gets unlocked --> > > spin_unlock(&journal->j_list_lock); > spin_lock(&journal->j_list_lock); > __jbd2_journal_remove_checkpoint(jh); > spin_unlock(&journal->j_list_lock); > try_to_free_buffers(page); > get_bh(bh) <-- accesses freed bh > > Fix the problem by grabbing bh reference before unlocking > journal->j_list_lock. > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 > Fixes: be1158cc615fd723552f0d9912087423c7cadda5 > Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com > CC: stable@vger.kernel.org > Signed-off-by: Jan Kara <jack@suse.cz> Applied, thanks (with the Fixes field adjusted). - Ted
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index c125d662777c..26f8d7e46462 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) bh = jh2bh(jh); if (buffer_locked(bh)) { - spin_unlock(&journal->j_list_lock); get_bh(bh); + spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse"); @@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) jh = transaction->t_checkpoint_io_list; bh = jh2bh(jh); if (buffer_locked(bh)) { - spin_unlock(&journal->j_list_lock); get_bh(bh); + spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse");
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible: CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) { <-- IO completes now, buffer gets unlocked --> spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bh Fix the problem by grabbing bh reference before unlocking journal->j_list_lock. Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5 Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com CC: stable@vger.kernel.org Signed-off-by: Jan Kara <jack@suse.cz> --- fs/jbd2/checkpoint.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)