| Message ID | 20190929141157.115845-1-houtao1@huawei.com |
|---|---|
| State | Rejected |
| Delegated to: | Miquel Raynal |
| Headers | show |
| Series | mtd: blkdevs: protect tr->devs list by mtd_table_mutex | expand |
ping ? On 2019/9/29 22:11, Hou Tao wrote: > There may be list corruption if there are concurrent list traversal > and list deletion on tr->devs as showed in the following case: > > CPU 0 CPU 1 > > open /dev/mtdblock1 > > // remove mtd1 > blktrans_notify_remove() > del_mtd_blktrans_dev() > > close /dev/mtdblock1 > blktrans_release > blktrans_dev_put > acquire blktrans_ref_mutex // remove mtd0 > // the final release acquire mtd_table_mutex > blktrans_dev_release() blktrans_notify_remove() > // remove mtdblock1 // next is mtdblock1 > list_del(&dev->list) list_for_each_entry_safe() > > We could fix the problem by acquiring blktrans_ref_mutex during > the traversal of tr->devs, but blktrans_ref_mutex needs to be released > before invoking tr->remote_dev(), so we also need to increase the kref > of current device else the device may be freed and decrease the kref > after the removal. > > Or we could move the list deletion to del_mtd_blktrans_dev(), and protect > the operations on tr->devs by mtd_table_mutex which has already be taken. > > The latter fix is simpler. We also can remove the unnecessary acquisitions > of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on > tr->devs have already been protected by mtd_table_mutex. > > Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") > Signed-off-by: Hou Tao <houtao1@huawei.com> > --- > I found the problem by code review, and could not find a way to > ensure the problem, because the removal of mtd devices almost > comes from the removal of modules, and the open of /dev/mtdblockX > will prevent the module from removing. > --- > drivers/mtd/mtd_blkdevs.c | 9 +++------ > 1 file changed, 3 insertions(+), 6 deletions(-) > > diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c > index 0c05f77f9b21..e04260237a25 100644 > --- a/drivers/mtd/mtd_blkdevs.c > +++ b/drivers/mtd/mtd_blkdevs.c > @@ -35,7 +35,6 @@ static void blktrans_dev_release(struct kref *kref) > blk_mq_free_tag_set(dev->tag_set); > kfree(dev->tag_set); > put_disk(dev->disk); > - list_del(&dev->list); > kfree(dev); > } > > @@ -350,7 +349,6 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) > BUG(); > } > > - mutex_lock(&blktrans_ref_mutex); > list_for_each_entry(d, &tr->devs, list) { > if (new->devnum == -1) { > /* Use first free number */ > @@ -362,7 +360,6 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) > } > } else if (d->devnum == new->devnum) { > /* Required number taken */ > - mutex_unlock(&blktrans_ref_mutex); > return -EBUSY; > } else if (d->devnum > new->devnum) { > /* Required number was free */ > @@ -381,14 +378,12 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) > * with this number. */ > if (new->devnum > (MINORMASK >> tr->part_bits) || > (tr->part_bits && new->devnum >= 27 * 26)) { > - mutex_unlock(&blktrans_ref_mutex); > goto error1; > } > > list_add_tail(&new->list, &tr->devs); > - added: > - mutex_unlock(&blktrans_ref_mutex); > > + added: > mutex_init(&new->lock); > kref_init(&new->ref); > if (!tr->writesect) > @@ -484,6 +479,8 @@ int del_mtd_blktrans_dev(struct mtd_blktrans_dev *old) > BUG(); > } > > + list_del(&old->list); > + > if (old->disk_attributes) > sysfs_remove_group(&disk_to_dev(old->disk)->kobj, > old->disk_attributes); >
On Sun, Sep 29, 2019 at 4:05 PM Hou Tao <houtao1@huawei.com> wrote: > > There may be list corruption if there are concurrent list traversal > and list deletion on tr->devs as showed in the following case: > > CPU 0 CPU 1 > > open /dev/mtdblock1 > > // remove mtd1 > blktrans_notify_remove() > del_mtd_blktrans_dev() > > close /dev/mtdblock1 > blktrans_release > blktrans_dev_put > acquire blktrans_ref_mutex // remove mtd0 > // the final release acquire mtd_table_mutex > blktrans_dev_release() blktrans_notify_remove() > // remove mtdblock1 // next is mtdblock1 > list_del(&dev->list) list_for_each_entry_safe() > > We could fix the problem by acquiring blktrans_ref_mutex during > the traversal of tr->devs, but blktrans_ref_mutex needs to be released > before invoking tr->remote_dev(), so we also need to increase the kref > of current device else the device may be freed and decrease the kref > after the removal. > > Or we could move the list deletion to del_mtd_blktrans_dev(), and protect > the operations on tr->devs by mtd_table_mutex which has already be taken. > > The latter fix is simpler. We also can remove the unnecessary acquisitions > of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on > tr->devs have already been protected by mtd_table_mutex. > > Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") > Signed-off-by: Hou Tao <houtao1@huawei.com> > --- > I found the problem by code review, and could not find a way to > ensure the problem, because the removal of mtd devices almost > comes from the removal of modules, and the open of /dev/mtdblockX > will prevent the module from removing. I'm confused. Can the problem only happen if you remove a mtd while it is open?
Hi, On 2019/10/16 4:10, Richard Weinberger wrote: > On Sun, Sep 29, 2019 at 4:05 PM Hou Tao <houtao1@huawei.com> wrote: >> >> There may be list corruption if there are concurrent list traversal >> and list deletion on tr->devs as showed in the following case: >> >> CPU 0 CPU 1 >> >> open /dev/mtdblock1 >> >> // remove mtd1 >> blktrans_notify_remove() >> del_mtd_blktrans_dev() >> >> close /dev/mtdblock1 >> blktrans_release >> blktrans_dev_put >> acquire blktrans_ref_mutex // remove mtd0 >> // the final release acquire mtd_table_mutex >> blktrans_dev_release() blktrans_notify_remove() >> // remove mtdblock1 // next is mtdblock1 >> list_del(&dev->list) list_for_each_entry_safe() >> >> We could fix the problem by acquiring blktrans_ref_mutex during >> the traversal of tr->devs, but blktrans_ref_mutex needs to be released >> before invoking tr->remote_dev(), so we also need to increase the kref >> of current device else the device may be freed and decrease the kref >> after the removal. >> >> Or we could move the list deletion to del_mtd_blktrans_dev(), and protect >> the operations on tr->devs by mtd_table_mutex which has already be taken. >> >> The latter fix is simpler. We also can remove the unnecessary acquisitions >> of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on >> tr->devs have already been protected by mtd_table_mutex. >> >> Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") >> Signed-off-by: Hou Tao <houtao1@huawei.com> >> --- >> I found the problem by code review, and could not find a way to >> ensure the problem, because the removal of mtd devices almost >> comes from the removal of modules, and the open of /dev/mtdblockX >> will prevent the module from removing. > > I'm confused. Can the problem only happen if you remove a mtd while > it is open? > No. The problem may happen when closing a mtd block device (instead of the mtd char device) for which its mtd device had already been removed. The reason why I can not confirm the problem is that I am trying to confirm the problem by the following steps: (1) insmod block2mtd.ko to create a mtd device (e.g., /dev/mtd0) (2) open /dev/mtdblock0 (3) remove /dev/mtd0 by removing block2mtd.ko step (3) always fails because the opening of /dev/mtdblock0 has already increased the reference count of block2mtd.ko, so /dev/mtd0 can not be removed. If the removal of mtd device doesn't need the removal of the module (for example, the removal comes from unbinding from its driver), the problem may occur. And according the implementation of del_mtd_blktrans_dev(), the scenario is possible as show in the following lines: /* If the device is currently open, tell trans driver to close it, then put mtd device, and don't touch it again */ mutex_lock(&old->lock); if (old->open) { if (old->tr->release) old->tr->release(old); __put_mtd_device(old->mtd); } Regards, Tao > . >
----- Ursprüngliche Mail ----- > Von: "Hou Tao" <houtao1@huawei.com> > An: "Richard Weinberger" <richard.weinberger@gmail.com> > CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "Miquel Raynal" <miquel.raynal@bootlin.com>, "Vignesh Raghavendra" > <vigneshr@ti.com>, "richard" <richard@nod.at>, "Maxim Levitsky" <maximlevitsky@gmail.com>, "Marek Vasut" > <marek.vasut@gmail.com>, "Brian Norris" <computersforpeace@gmail.com>, "David Woodhouse" <dwmw2@infradead.org> > Gesendet: Mittwoch, 16. Oktober 2019 12:55:12 > Betreff: Re: [PATCH] mtd: blkdevs: protect tr->devs list by mtd_table_mutex > Hi, > > On 2019/10/16 4:10, Richard Weinberger wrote: >> On Sun, Sep 29, 2019 at 4:05 PM Hou Tao <houtao1@huawei.com> wrote: >>> >>> There may be list corruption if there are concurrent list traversal >>> and list deletion on tr->devs as showed in the following case: >>> >>> CPU 0 CPU 1 >>> >>> open /dev/mtdblock1 >>> >>> // remove mtd1 >>> blktrans_notify_remove() >>> del_mtd_blktrans_dev() >>> >>> close /dev/mtdblock1 >>> blktrans_release >>> blktrans_dev_put >>> acquire blktrans_ref_mutex // remove mtd0 >>> // the final release acquire mtd_table_mutex >>> blktrans_dev_release() blktrans_notify_remove() >>> // remove mtdblock1 // next is mtdblock1 >>> list_del(&dev->list) list_for_each_entry_safe() >>> >>> We could fix the problem by acquiring blktrans_ref_mutex during >>> the traversal of tr->devs, but blktrans_ref_mutex needs to be released >>> before invoking tr->remote_dev(), so we also need to increase the kref >>> of current device else the device may be freed and decrease the kref >>> after the removal. >>> >>> Or we could move the list deletion to del_mtd_blktrans_dev(), and protect >>> the operations on tr->devs by mtd_table_mutex which has already be taken. >>> >>> The latter fix is simpler. We also can remove the unnecessary acquisitions >>> of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on >>> tr->devs have already been protected by mtd_table_mutex. >>> >>> Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") >>> Signed-off-by: Hou Tao <houtao1@huawei.com> >>> --- >>> I found the problem by code review, and could not find a way to >>> ensure the problem, because the removal of mtd devices almost >>> comes from the removal of modules, and the open of /dev/mtdblockX >>> will prevent the module from removing. >> >> I'm confused. Can the problem only happen if you remove a mtd while >> it is open? >> > No. The problem may happen when closing a mtd block device (instead of > the mtd char device) for which its mtd device had already been removed. > > The reason why I can not confirm the problem is that I am trying to > confirm the problem by the following steps: > (1) insmod block2mtd.ko to create a mtd device (e.g., /dev/mtd0) > (2) open /dev/mtdblock0 > (3) remove /dev/mtd0 by removing block2mtd.ko > > step (3) always fails because the opening of /dev/mtdblock0 has already > increased the reference count of block2mtd.ko, so /dev/mtd0 can not be removed. Ok. But yeah, the problem is real and I'm sure with ubi+gluebi it can be triggered. Thanks, //richard
Hi, On 2019/10/16 19:06, Richard Weinberger wrote: > ----- Ursprüngliche Mail ----- >> Von: "Hou Tao" <houtao1@huawei.com> >> An: "Richard Weinberger" <richard.weinberger@gmail.com> >> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "Miquel Raynal" <miquel.raynal@bootlin.com>, "Vignesh Raghavendra" >> <vigneshr@ti.com>, "richard" <richard@nod.at>, "Maxim Levitsky" <maximlevitsky@gmail.com>, "Marek Vasut" >> <marek.vasut@gmail.com>, "Brian Norris" <computersforpeace@gmail.com>, "David Woodhouse" <dwmw2@infradead.org> >> Gesendet: Mittwoch, 16. Oktober 2019 12:55:12 >> Betreff: Re: [PATCH] mtd: blkdevs: protect tr->devs list by mtd_table_mutex > >> Hi, >> >> On 2019/10/16 4:10, Richard Weinberger wrote: >>> On Sun, Sep 29, 2019 at 4:05 PM Hou Tao <houtao1@huawei.com> wrote: >>>> >>>> There may be list corruption if there are concurrent list traversal >>>> and list deletion on tr->devs as showed in the following case: >>>> >>>> CPU 0 CPU 1 >>>> >>>> open /dev/mtdblock1 >>>> >>>> // remove mtd1 >>>> blktrans_notify_remove() >>>> del_mtd_blktrans_dev() >>>> >>>> close /dev/mtdblock1 >>>> blktrans_release >>>> blktrans_dev_put >>>> acquire blktrans_ref_mutex // remove mtd0 >>>> // the final release acquire mtd_table_mutex >>>> blktrans_dev_release() blktrans_notify_remove() >>>> // remove mtdblock1 // next is mtdblock1 >>>> list_del(&dev->list) list_for_each_entry_safe() >>>> >>>> We could fix the problem by acquiring blktrans_ref_mutex during >>>> the traversal of tr->devs, but blktrans_ref_mutex needs to be released >>>> before invoking tr->remote_dev(), so we also need to increase the kref >>>> of current device else the device may be freed and decrease the kref >>>> after the removal. >>>> >>>> Or we could move the list deletion to del_mtd_blktrans_dev(), and protect >>>> the operations on tr->devs by mtd_table_mutex which has already be taken. >>>> >>>> The latter fix is simpler. We also can remove the unnecessary acquisitions >>>> of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on >>>> tr->devs have already been protected by mtd_table_mutex. >>>> >>>> Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") >>>> Signed-off-by: Hou Tao <houtao1@huawei.com> >>>> --- >>>> I found the problem by code review, and could not find a way to >>>> ensure the problem, because the removal of mtd devices almost >>>> comes from the removal of modules, and the open of /dev/mtdblockX >>>> will prevent the module from removing. >>> >>> I'm confused. Can the problem only happen if you remove a mtd while >>> it is open? >>> >> No. The problem may happen when closing a mtd block device (instead of >> the mtd char device) for which its mtd device had already been removed. >> >> The reason why I can not confirm the problem is that I am trying to >> confirm the problem by the following steps: >> (1) insmod block2mtd.ko to create a mtd device (e.g., /dev/mtd0) >> (2) open /dev/mtdblock0 >> (3) remove /dev/mtd0 by removing block2mtd.ko >> >> step (3) always fails because the opening of /dev/mtdblock0 has already >> increased the reference count of block2mtd.ko, so /dev/mtd0 can not be removed. > > Ok. But yeah, the problem is real and I'm sure with ubi+gluebi it can be triggered. > Thanks for your suggestions. I will try to use ubi+gluebi to confirm the problem. Regards, Tao > Thanks, > //richard > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ >
diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index 0c05f77f9b21..e04260237a25 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -35,7 +35,6 @@ static void blktrans_dev_release(struct kref *kref) blk_mq_free_tag_set(dev->tag_set); kfree(dev->tag_set); put_disk(dev->disk); - list_del(&dev->list); kfree(dev); } @@ -350,7 +349,6 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) BUG(); } - mutex_lock(&blktrans_ref_mutex); list_for_each_entry(d, &tr->devs, list) { if (new->devnum == -1) { /* Use first free number */ @@ -362,7 +360,6 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) } } else if (d->devnum == new->devnum) { /* Required number taken */ - mutex_unlock(&blktrans_ref_mutex); return -EBUSY; } else if (d->devnum > new->devnum) { /* Required number was free */ @@ -381,14 +378,12 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) * with this number. */ if (new->devnum > (MINORMASK >> tr->part_bits) || (tr->part_bits && new->devnum >= 27 * 26)) { - mutex_unlock(&blktrans_ref_mutex); goto error1; } list_add_tail(&new->list, &tr->devs); - added: - mutex_unlock(&blktrans_ref_mutex); + added: mutex_init(&new->lock); kref_init(&new->ref); if (!tr->writesect) @@ -484,6 +479,8 @@ int del_mtd_blktrans_dev(struct mtd_blktrans_dev *old) BUG(); } + list_del(&old->list); + if (old->disk_attributes) sysfs_remove_group(&disk_to_dev(old->disk)->kobj, old->disk_attributes);
There may be list corruption if there are concurrent list traversal and list deletion on tr->devs as showed in the following case: CPU 0 CPU 1 open /dev/mtdblock1 // remove mtd1 blktrans_notify_remove() del_mtd_blktrans_dev() close /dev/mtdblock1 blktrans_release blktrans_dev_put acquire blktrans_ref_mutex // remove mtd0 // the final release acquire mtd_table_mutex blktrans_dev_release() blktrans_notify_remove() // remove mtdblock1 // next is mtdblock1 list_del(&dev->list) list_for_each_entry_safe() We could fix the problem by acquiring blktrans_ref_mutex during the traversal of tr->devs, but blktrans_ref_mutex needs to be released before invoking tr->remote_dev(), so we also need to increase the kref of current device else the device may be freed and decrease the kref after the removal. Or we could move the list deletion to del_mtd_blktrans_dev(), and protect the operations on tr->devs by mtd_table_mutex which has already be taken. The latter fix is simpler. We also can remove the unnecessary acquisitions of blktrans_ref_mutex in add_mtd_blktrans_dev() because operations on tr->devs have already been protected by mtd_table_mutex. Fixes: 048d87199566 ("mtd: blktrans: Hotplug fixes") Signed-off-by: Hou Tao <houtao1@huawei.com> --- I found the problem by code review, and could not find a way to ensure the problem, because the removal of mtd devices almost comes from the removal of modules, and the open of /dev/mtdblockX will prevent the module from removing. --- drivers/mtd/mtd_blkdevs.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)