Patchwork [v2] msi: free msi_desc entry only after we've released the kobject

login
register
mail settings
Submitter Veaceslav Falico
Date Sept. 26, 2013, 9:59 a.m.
Message ID <1380189591-1368-1-git-send-email-vfalico@redhat.com>
Download mbox | patch
Permalink /patch/278150/
State Changes Requested
Headers show

Comments

Veaceslav Falico - Sept. 26, 2013, 9:59 a.m.
Currently, we first do kobject_put(&entry->kobj) and the kfree(entry),
however kobject_put() doesn't guarantee us that it was the last reference
and that the kobj isn't used currently by someone else, so after we
kfree(entry) with the struct kobject - other users will begin using the
freed memory, instead of the actual kobject.

Fix this by using the kobject->release callback, which is called last when
the kobject is indeed not used and is cleaned up - it's msi_kobj_release(),
which can do the kfree(entry) safely (kobject_put/cleanup doesn't use the
kobj itself after ->release() was called, so we're safe).

In case we've failed to create the sysfs directories - just kfree()
it - cause we don't have the kobjects attached.

Also, remove the same functionality from populate_msi_sysfs(), cause on
failure we anyway call free_msi_irqs(), which will take care of all the
kobjects properly.

And add the forgotten pci_dev_put(pdev) in case of failure to register the
kobject in populate_msi_sysfs().

CC: Bjorn Helgaas <bhelgaas@google.com>
CC: Neil Horman <nhorman@tuxdriver.com>
CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
CC: linux-pci@vger.kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Veaceslav Falico <vfalico@redhat.com>
---

Notes:
    v1  -> v2:
    Make it as a standalone patch, which is a bugfix, and add the forgotten
    pci_dev_put() so that it won't break bisecting. The pci_dev_put() will
    go away anyway in the following patchset, which cleans removes
    kobject_del and useless pci_dev_get/put().  Rebased on linux-pci/next.

 drivers/pci/msi.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
Neil Horman - Sept. 26, 2013, 2:42 p.m.
On Thu, Sep 26, 2013 at 11:59:51AM +0200, Veaceslav Falico wrote:
> Currently, we first do kobject_put(&entry->kobj) and the kfree(entry),
> however kobject_put() doesn't guarantee us that it was the last reference
> and that the kobj isn't used currently by someone else, so after we
> kfree(entry) with the struct kobject - other users will begin using the
> freed memory, instead of the actual kobject.
> 
> Fix this by using the kobject->release callback, which is called last when
> the kobject is indeed not used and is cleaned up - it's msi_kobj_release(),
> which can do the kfree(entry) safely (kobject_put/cleanup doesn't use the
> kobj itself after ->release() was called, so we're safe).
> 
> In case we've failed to create the sysfs directories - just kfree()
> it - cause we don't have the kobjects attached.
> 
> Also, remove the same functionality from populate_msi_sysfs(), cause on
> failure we anyway call free_msi_irqs(), which will take care of all the
> kobjects properly.
> 
> And add the forgotten pci_dev_put(pdev) in case of failure to register the
> kobject in populate_msi_sysfs().
> 
> CC: Bjorn Helgaas <bhelgaas@google.com>
> CC: Neil Horman <nhorman@tuxdriver.com>
> CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> CC: linux-pci@vger.kernel.org
> CC: linux-kernel@vger.kernel.org
> Signed-off-by: Veaceslav Falico <vfalico@redhat.com>
> ---
> 
Acked-by: Neil Horman <nhorman@tuxdriver.com>

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Veaceslav Falico - Sept. 28, 2013, 9:37 p.m.
On Thu, Sep 26, 2013 at 11:59:51AM +0200, Veaceslav Falico wrote:
>Currently, we first do kobject_put(&entry->kobj) and the kfree(entry),
>however kobject_put() doesn't guarantee us that it was the last reference
>and that the kobj isn't used currently by someone else, so after we
>kfree(entry) with the struct kobject - other users will begin using the
>freed memory, instead of the actual kobject.

Hi Bjorn,

I've seen that you've dropped this bugfix (and the 3 cleanup patches) with
"Changes Requested", however I don't recall any request to change this.

I'm really sorry for bugging - but I need this fix to get included for my
testing to work :(.

This fix fixes the critical bug when we free the entry with the kobject
.../msi_irqs/IRQ_NR still being used, however the issue with msi_irqs kset
itself still exists, and really is a different issue - and I'm trying to
find a fix for it.

Thank you, and sorry for the mess and the noise.

>
>Fix this by using the kobject->release callback, which is called last when
>the kobject is indeed not used and is cleaned up - it's msi_kobj_release(),
>which can do the kfree(entry) safely (kobject_put/cleanup doesn't use the
>kobj itself after ->release() was called, so we're safe).
>
>In case we've failed to create the sysfs directories - just kfree()
>it - cause we don't have the kobjects attached.
>
>Also, remove the same functionality from populate_msi_sysfs(), cause on
>failure we anyway call free_msi_irqs(), which will take care of all the
>kobjects properly.
>
>And add the forgotten pci_dev_put(pdev) in case of failure to register the
>kobject in populate_msi_sysfs().
>
>CC: Bjorn Helgaas <bhelgaas@google.com>
>CC: Neil Horman <nhorman@tuxdriver.com>
>CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>CC: linux-pci@vger.kernel.org
>CC: linux-kernel@vger.kernel.org
>Signed-off-by: Veaceslav Falico <vfalico@redhat.com>
>---
>
>Notes:
>    v1  -> v2:
>    Make it as a standalone patch, which is a bugfix, and add the forgotten
>    pci_dev_put() so that it won't break bisecting. The pci_dev_put() will
>    go away anyway in the following patchset, which cleans removes
>    kobject_del and useless pci_dev_get/put().  Rebased on linux-pci/next.
>
> drivers/pci/msi.c | 31 ++++++++++++-------------------
> 1 file changed, 12 insertions(+), 19 deletions(-)
>
>diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>index d5f90d6..5d70f49 100644
>--- a/drivers/pci/msi.c
>+++ b/drivers/pci/msi.c
>@@ -374,19 +374,22 @@ static void free_msi_irqs(struct pci_dev *dev)
> 				iounmap(entry->mask_base);
> 		}
>
>+		list_del(&entry->list);
>+
> 		/*
> 		 * Its possible that we get into this path
> 		 * When populate_msi_sysfs fails, which means the entries
> 		 * were not registered with sysfs.  In that case don't
>-		 * unregister them.
>+		 * unregister them, and just free. Otherwise the
>+		 * kobject->release will take care of freeing the entry via
>+		 * msi_kobj_release().
> 		 */
> 		if (entry->kobj.parent) {
> 			kobject_del(&entry->kobj);
> 			kobject_put(&entry->kobj);
>+		} else {
>+			kfree(entry);
> 		}
>-
>-		list_del(&entry->list);
>-		kfree(entry);
> 	}
> }
>
>@@ -509,6 +512,7 @@ static void msi_kobj_release(struct kobject *kobj)
> 	struct msi_desc *entry = to_msi_desc(kobj);
>
> 	pci_dev_put(entry->dev);
>+	kfree(entry);
> }
>
> static struct kobj_type msi_irq_ktype = {
>@@ -522,7 +526,6 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
> 	struct msi_desc *entry;
> 	struct kobject *kobj;
> 	int ret;
>-	int count = 0;
>
> 	pdev->msi_kset = kset_create_and_add("msi_irqs", NULL, &pdev->dev.kobj);
> 	if (!pdev->msi_kset)
>@@ -534,23 +537,13 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
> 		pci_dev_get(pdev);
> 		ret = kobject_init_and_add(kobj, &msi_irq_ktype, NULL,
> 				     "%u", entry->irq);
>-		if (ret)
>-			goto out_unroll;
>-
>-		count++;
>+		if (ret) {
>+			pci_dev_put(pdev);
>+			return ret;
>+		}
> 	}
>
> 	return 0;
>-
>-out_unroll:
>-	list_for_each_entry(entry, &pdev->msi_list, list) {
>-		if (!count)
>-			break;
>-		kobject_del(&entry->kobj);
>-		kobject_put(&entry->kobj);
>-		count--;
>-	}
>-	return ret;
> }
>
> /**
>-- 
>1.8.4
>
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Bjorn Helgaas - Oct. 4, 2013, 4:46 p.m.
On Sat, Sep 28, 2013 at 3:37 PM, Veaceslav Falico <vfalico@redhat.com> wrote:
> On Thu, Sep 26, 2013 at 11:59:51AM +0200, Veaceslav Falico wrote:
>>
>> Currently, we first do kobject_put(&entry->kobj) and the kfree(entry),
>> however kobject_put() doesn't guarantee us that it was the last reference
>> and that the kobj isn't used currently by someone else, so after we
>> kfree(entry) with the struct kobject - other users will begin using the
>> freed memory, instead of the actual kobject.
>
>
> Hi Bjorn,
>
> I've seen that you've dropped this bugfix (and the 3 cleanup patches) with
> "Changes Requested", however I don't recall any request to change this.

I talked to Greg KH about this recently, and he said he might take a
look at doing a more extensive cleanup of populate_msi_sysfs() using
attribute groups, so I don't know if you want to wait and see whether
he does anything, or go ahead on the path you were on.

If you continue, my advice is:

  - Put all these patches in a single series with a version number (I
think the next posting would be v3) to help me keep track of them.

  - In populate_msi_sysfs(), drop the pci_dev_get() (or explain why
it's needed).  My reasoning is that the "msi_irqs" kset should already
hold a reference on the pdev (acquired in kset_create_and_add() ->
kset_register() -> kobject_add_internal()), and each irq entry should
hold a reference on the kset (see kobject_add_internal() again),  so
it is redundant to acquire a reference on the pdev directly.  This
means dropping the pci_dev_put() in msi_kobj_release(), of course.

- Move the kfree(entry) from free_msi_irqs() to msi_kobj_release() (I
think one of your patches already did this).

  - In populate_msi_sysfs(), drop the kobject_del() in the out_unroll
loop.  I think we would only need that if there were a way to create a
new irq entry in "msi_irqs" before the old irq entry was released.
But I don't think that's possible.  We only create irq entries in
populate_msi_sysfs(), which always starts with a fresh, empty
"msi_irqs" kset.

  - In free_msi_irqs(), similarly remove the kobject_del().

  - Add a kobject_del() before each kset_unregister(dev->msi_kset)
call.  This will remove "msi_irqs" from sysfs, so future creates will
succeed even if somebody still has the old "msi_irqs" open.

  - Keep the msi_kset cleanup in populate_msi_sysfs() instead of
relying on free_msi_irqs().  I think it's less error prone to keep the
creation and error path cleanup in the same function.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Veaceslav Falico - Oct. 9, 2013, 11:36 a.m.
On Fri, Oct 04, 2013 at 10:46:31AM -0600, Bjorn Helgaas wrote:
>On Sat, Sep 28, 2013 at 3:37 PM, Veaceslav Falico <vfalico@redhat.com> wrote:
>> On Thu, Sep 26, 2013 at 11:59:51AM +0200, Veaceslav Falico wrote:
>>>
>>> Currently, we first do kobject_put(&entry->kobj) and the kfree(entry),
>>> however kobject_put() doesn't guarantee us that it was the last reference
>>> and that the kobj isn't used currently by someone else, so after we
>>> kfree(entry) with the struct kobject - other users will begin using the
>>> freed memory, instead of the actual kobject.
>>
>>
>> Hi Bjorn,
>>
>> I've seen that you've dropped this bugfix (and the 3 cleanup patches) with
>> "Changes Requested", however I don't recall any request to change this.
>
>I talked to Greg KH about this recently, and he said he might take a
>look at doing a more extensive cleanup of populate_msi_sysfs() using
>attribute groups, so I don't know if you want to wait and see whether
>he does anything, or go ahead on the path you were on.

Sorry for the delay, was sick. I'll continue going ahead, however if
Greg/you don't really need it or are working on it - please say now, so I'll
stop waisting your time.

>
>If you continue, my advice is:
>
>  - Put all these patches in a single series with a version number (I
>think the next posting would be v3) to help me keep track of them.

Will do, if/when there'll be next version. Now they're divided into 1
bugfix and 1 cleanup patchset.

>
>  - In populate_msi_sysfs(), drop the pci_dev_get() (or explain why
>it's needed).  My reasoning is that the "msi_irqs" kset should already
>hold a reference on the pdev (acquired in kset_create_and_add() ->
>kset_register() -> kobject_add_internal()), and each irq entry should
>hold a reference on the kset (see kobject_add_internal() again),  so
>it is redundant to acquire a reference on the pdev directly.  This
>means dropping the pci_dev_put() in msi_kobj_release(), of course.

It's done in my patch

     pci: remove redundant pci_dev_get/put() on kobject (un)register

http://patchwork.ozlabs.org/patch/278201/

>
>- Move the kfree(entry) from free_msi_irqs() to msi_kobj_release() (I
>think one of your patches already did this).

It's done in my patch

	    msi: free msi_desc entry only after we've released the kobject

http://patchwork.ozlabs.org/patch/278150/

>
>  - In populate_msi_sysfs(), drop the kobject_del() in the out_unroll
>loop.  I think we would only need that if there were a way to create a
>new irq entry in "msi_irqs" before the old irq entry was released.
>But I don't think that's possible.  We only create irq entries in
>populate_msi_sysfs(), which always starts with a fresh, empty
>"msi_irqs" kset.

It's done in my patch

     msi: free msi_desc entry only after we've released the kobject

http://patchwork.ozlabs.org/patch/278150/

>
>  - In free_msi_irqs(), similarly remove the kobject_del().

It's done in my patch

	    msi: remove useless kobject_del() in free_msi_irqs()

http://patchwork.ozlabs.org/patch/278202/

>
>  - Add a kobject_del() before each kset_unregister(dev->msi_kset)
>call.  This will remove "msi_irqs" from sysfs, so future creates will
>succeed even if somebody still has the old "msi_irqs" open.

I think it's done in your patch

	kobject: remove kset from sysfs immediately in kset_unregister()

http://patchwork.ozlabs.org/patch/281618/

So it'll collide and use kobject_del() twice.

Or did you actually drop your patch?

>
>  - Keep the msi_kset cleanup in populate_msi_sysfs() instead of
>relying on free_msi_irqs().  I think it's less error prone to keep the
>creation and error path cleanup in the same function.

It is less error prone, however the current design is that "once we fail
something while creating irqs, always call free_msi_irqs()", so, if we add
the msi_kset cleanup to populate_msi_sysfs() (it wasn't there before, so we
can't 'keep' it) - we'll have to verify if it was don in free_msi_irqs(),
cause free_msi_irqs() is being called not only on rollback in
msi_capability_init(), but also in pci_disable_msi() and friends.

>
>Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index d5f90d6..5d70f49 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -374,19 +374,22 @@  static void free_msi_irqs(struct pci_dev *dev)
 				iounmap(entry->mask_base);
 		}
 
+		list_del(&entry->list);
+
 		/*
 		 * Its possible that we get into this path
 		 * When populate_msi_sysfs fails, which means the entries
 		 * were not registered with sysfs.  In that case don't
-		 * unregister them.
+		 * unregister them, and just free. Otherwise the
+		 * kobject->release will take care of freeing the entry via
+		 * msi_kobj_release().
 		 */
 		if (entry->kobj.parent) {
 			kobject_del(&entry->kobj);
 			kobject_put(&entry->kobj);
+		} else {
+			kfree(entry);
 		}
-
-		list_del(&entry->list);
-		kfree(entry);
 	}
 }
 
@@ -509,6 +512,7 @@  static void msi_kobj_release(struct kobject *kobj)
 	struct msi_desc *entry = to_msi_desc(kobj);
 
 	pci_dev_put(entry->dev);
+	kfree(entry);
 }
 
 static struct kobj_type msi_irq_ktype = {
@@ -522,7 +526,6 @@  static int populate_msi_sysfs(struct pci_dev *pdev)
 	struct msi_desc *entry;
 	struct kobject *kobj;
 	int ret;
-	int count = 0;
 
 	pdev->msi_kset = kset_create_and_add("msi_irqs", NULL, &pdev->dev.kobj);
 	if (!pdev->msi_kset)
@@ -534,23 +537,13 @@  static int populate_msi_sysfs(struct pci_dev *pdev)
 		pci_dev_get(pdev);
 		ret = kobject_init_and_add(kobj, &msi_irq_ktype, NULL,
 				     "%u", entry->irq);
-		if (ret)
-			goto out_unroll;
-
-		count++;
+		if (ret) {
+			pci_dev_put(pdev);
+			return ret;
+		}
 	}
 
 	return 0;
-
-out_unroll:
-	list_for_each_entry(entry, &pdev->msi_list, list) {
-		if (!count)
-			break;
-		kobject_del(&entry->kobj);
-		kobject_put(&entry->kobj);
-		count--;
-	}
-	return ret;
 }
 
 /**