| Message ID | 20180817075901.4608-2-david@redhat.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | mm: online/offline_pages called w.o. mem_hotplug_lock | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | next/apply_patch Successfully applied |
| snowpatch_ozlabs/checkpatch | success | Test checkpatch on branch next |
On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: > From: Vitaly Kuznetsov <vkuznets@redhat.com> > > Well require to call add_memory()/add_memory_resource() with > device_hotplug_lock held, to avoid a lock inversion. Allow external modules > (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to > lock device hotplug. > > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> > [modify patch description] > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > drivers/base/core.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/base/core.c b/drivers/base/core.c > index 04bbcd779e11..9010b9e942b5 100644 > --- a/drivers/base/core.c > +++ b/drivers/base/core.c > @@ -700,11 +700,13 @@ void lock_device_hotplug(void) > { > mutex_lock(&device_hotplug_lock); > } > +EXPORT_SYMBOL_GPL(lock_device_hotplug); > > void unlock_device_hotplug(void) > { > mutex_unlock(&device_hotplug_lock); > } > +EXPORT_SYMBOL_GPL(unlock_device_hotplug); If these are going to be "global" symbols, let's properly name them. device_hotplug_lock/unlock would be better. But I am _really_ nervous about letting stuff outside of the driver core mess with this, as people better know what they are doing. Can't we just "lock" the memory stuff instead? Why does the entirety of the driver core need to be messed with here? thanks, greg k-h
On 17.08.2018 10:41, Greg Kroah-Hartman wrote: > On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: >> From: Vitaly Kuznetsov <vkuznets@redhat.com> >> >> Well require to call add_memory()/add_memory_resource() with >> device_hotplug_lock held, to avoid a lock inversion. Allow external modules >> (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to >> lock device hotplug. >> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> >> [modify patch description] >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> drivers/base/core.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/drivers/base/core.c b/drivers/base/core.c >> index 04bbcd779e11..9010b9e942b5 100644 >> --- a/drivers/base/core.c >> +++ b/drivers/base/core.c >> @@ -700,11 +700,13 @@ void lock_device_hotplug(void) >> { >> mutex_lock(&device_hotplug_lock); >> } >> +EXPORT_SYMBOL_GPL(lock_device_hotplug); >> >> void unlock_device_hotplug(void) >> { >> mutex_unlock(&device_hotplug_lock); >> } >> +EXPORT_SYMBOL_GPL(unlock_device_hotplug); > > If these are going to be "global" symbols, let's properly name them. > device_hotplug_lock/unlock would be better. But I am _really_ nervous > about letting stuff outside of the driver core mess with this, as people > better know what they are doing. The only "problem" is that we have kernel modules (for paravirtualized devices) that call add_memory(). This is Hyper-V right now, but we might have other ones in the future. Without them we would not have to export it. We might also get kernel modules that want to call remove_memory() - which will require the device_hotplug_lock as of now. What we could do is a) add_memory() -> _add_memory() and don't export it b) add_memory() takes the device_hotplug_lock and calls _add_memory() . We export that one. c) Use add_memory() in external modules only Similar wrapper would be needed e.g. for remove_memory() later on. > > Can't we just "lock" the memory stuff instead? Why does the entirety of > the driver core need to be messed with here? The main problem is that add_memory() will first take the mem_hotplug_lock, to later on create and attach the memory block devices (to a bus without any drivers) via bus_probe_device(). Here, we will take the device_lock() of these devices. Setting a device online from user space (.online = true) will first take the device_hotplug_lock, then the device_lock(), and _right now_ not the mem_hotplug_lock (see cover letter/patch 2). We have to take mem_hotplug_lock here, but doing it down in e.g. online_pages() will right now create the possibility for a lock inversion. So one alternative is to take the mem_hotplug_lock in core.c before calling device_online()/device_offline(). But this feels very wrong. Thanks! > > thanks, > > greg k-h >
On Fri, Aug 17, 2018 at 10:41 AM Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: > > From: Vitaly Kuznetsov <vkuznets@redhat.com> > > > > Well require to call add_memory()/add_memory_resource() with > > device_hotplug_lock held, to avoid a lock inversion. Allow external modules > > (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to > > lock device hotplug. > > > > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> > > [modify patch description] > > Signed-off-by: David Hildenbrand <david@redhat.com> > > --- > > drivers/base/core.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/drivers/base/core.c b/drivers/base/core.c > > index 04bbcd779e11..9010b9e942b5 100644 > > --- a/drivers/base/core.c > > +++ b/drivers/base/core.c > > @@ -700,11 +700,13 @@ void lock_device_hotplug(void) > > { > > mutex_lock(&device_hotplug_lock); > > } > > +EXPORT_SYMBOL_GPL(lock_device_hotplug); > > > > void unlock_device_hotplug(void) > > { > > mutex_unlock(&device_hotplug_lock); > > } > > +EXPORT_SYMBOL_GPL(unlock_device_hotplug); > > If these are going to be "global" symbols, let's properly name them. > device_hotplug_lock/unlock would be better. Well, device_hotplug_lock is the name of the lock itself. :-) > But I am _really_ nervous about letting stuff outside of the driver core mess > with this, as people better know what they are doing. > > Can't we just "lock" the memory stuff instead? Why does the entirety of > the driver core need to be messed with here? Because, in general, memory hotplug and hotplug of devices are not independent. CPUs and memory may only be possible to take away together and that may be the case for other devices too (say, it wouldn't be a good idea to access a memory block that has just gone away from a device, for DMA and the like). That's why device_hotplug_lock was introduced in the first place. That said I agree that exporting this to drivers doesn't feel particularly safe.
On Fri, Aug 17, 2018 at 10:56 AM David Hildenbrand <david@redhat.com> wrote: > > On 17.08.2018 10:41, Greg Kroah-Hartman wrote: > > On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: > >> From: Vitaly Kuznetsov <vkuznets@redhat.com> > >> > >> Well require to call add_memory()/add_memory_resource() with > >> device_hotplug_lock held, to avoid a lock inversion. Allow external modules > >> (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to > >> lock device hotplug. > >> > >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> > >> [modify patch description] > >> Signed-off-by: David Hildenbrand <david@redhat.com> > >> --- > >> drivers/base/core.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/drivers/base/core.c b/drivers/base/core.c > >> index 04bbcd779e11..9010b9e942b5 100644 > >> --- a/drivers/base/core.c > >> +++ b/drivers/base/core.c > >> @@ -700,11 +700,13 @@ void lock_device_hotplug(void) > >> { > >> mutex_lock(&device_hotplug_lock); > >> } > >> +EXPORT_SYMBOL_GPL(lock_device_hotplug); > >> > >> void unlock_device_hotplug(void) > >> { > >> mutex_unlock(&device_hotplug_lock); > >> } > >> +EXPORT_SYMBOL_GPL(unlock_device_hotplug); > > > > If these are going to be "global" symbols, let's properly name them. > > device_hotplug_lock/unlock would be better. But I am _really_ nervous > > about letting stuff outside of the driver core mess with this, as people > > better know what they are doing. > > The only "problem" is that we have kernel modules (for paravirtualized > devices) that call add_memory(). This is Hyper-V right now, but we might > have other ones in the future. Without them we would not have to export > it. We might also get kernel modules that want to call remove_memory() - > which will require the device_hotplug_lock as of now. > > What we could do is > > a) add_memory() -> _add_memory() and don't export it > b) add_memory() takes the device_hotplug_lock and calls _add_memory() . > We export that one. > c) Use add_memory() in external modules only > > Similar wrapper would be needed e.g. for remove_memory() later on. That would be safer IMO, as it would prevent developers from using add_memory() without the lock, say. If the lock is always going to be required for add_memory(), make it hard (or event impossible) to use the latter without it.
On 17.08.2018 11:03, Rafael J. Wysocki wrote: > On Fri, Aug 17, 2018 at 10:56 AM David Hildenbrand <david@redhat.com> wrote: >> >> On 17.08.2018 10:41, Greg Kroah-Hartman wrote: >>> On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: >>>> From: Vitaly Kuznetsov <vkuznets@redhat.com> >>>> >>>> Well require to call add_memory()/add_memory_resource() with >>>> device_hotplug_lock held, to avoid a lock inversion. Allow external modules >>>> (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to >>>> lock device hotplug. >>>> >>>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> >>>> [modify patch description] >>>> Signed-off-by: David Hildenbrand <david@redhat.com> >>>> --- >>>> drivers/base/core.c | 2 ++ >>>> 1 file changed, 2 insertions(+) >>>> >>>> diff --git a/drivers/base/core.c b/drivers/base/core.c >>>> index 04bbcd779e11..9010b9e942b5 100644 >>>> --- a/drivers/base/core.c >>>> +++ b/drivers/base/core.c >>>> @@ -700,11 +700,13 @@ void lock_device_hotplug(void) >>>> { >>>> mutex_lock(&device_hotplug_lock); >>>> } >>>> +EXPORT_SYMBOL_GPL(lock_device_hotplug); >>>> >>>> void unlock_device_hotplug(void) >>>> { >>>> mutex_unlock(&device_hotplug_lock); >>>> } >>>> +EXPORT_SYMBOL_GPL(unlock_device_hotplug); >>> >>> If these are going to be "global" symbols, let's properly name them. >>> device_hotplug_lock/unlock would be better. But I am _really_ nervous >>> about letting stuff outside of the driver core mess with this, as people >>> better know what they are doing. >> >> The only "problem" is that we have kernel modules (for paravirtualized >> devices) that call add_memory(). This is Hyper-V right now, but we might >> have other ones in the future. Without them we would not have to export >> it. We might also get kernel modules that want to call remove_memory() - >> which will require the device_hotplug_lock as of now. >> >> What we could do is >> >> a) add_memory() -> _add_memory() and don't export it >> b) add_memory() takes the device_hotplug_lock and calls _add_memory() . >> We export that one. >> c) Use add_memory() in external modules only >> >> Similar wrapper would be needed e.g. for remove_memory() later on. > > That would be safer IMO, as it would prevent developers from using > add_memory() without the lock, say. > > If the lock is always going to be required for add_memory(), make it > hard (or event impossible) to use the latter without it. > If there are no objections, I'll go into that direction. But I'll wait for more comments regarding the general concept first. Thanks!
On Fri, Aug 17, 2018 at 11:41:24AM +0200, David Hildenbrand wrote: > On 17.08.2018 11:03, Rafael J. Wysocki wrote: > > On Fri, Aug 17, 2018 at 10:56 AM David Hildenbrand <david@redhat.com> wrote: > >> > >> On 17.08.2018 10:41, Greg Kroah-Hartman wrote: > >>> On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: > >>>> From: Vitaly Kuznetsov <vkuznets@redhat.com> > >>>> > >>>> Well require to call add_memory()/add_memory_resource() with > >>>> device_hotplug_lock held, to avoid a lock inversion. Allow external modules > >>>> (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to > >>>> lock device hotplug. > >>>> > >>>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> > >>>> [modify patch description] > >>>> Signed-off-by: David Hildenbrand <david@redhat.com> > >>>> --- > >>>> drivers/base/core.c | 2 ++ > >>>> 1 file changed, 2 insertions(+) > >>>> > >>>> diff --git a/drivers/base/core.c b/drivers/base/core.c > >>>> index 04bbcd779e11..9010b9e942b5 100644 > >>>> --- a/drivers/base/core.c > >>>> +++ b/drivers/base/core.c > >>>> @@ -700,11 +700,13 @@ void lock_device_hotplug(void) > >>>> { > >>>> mutex_lock(&device_hotplug_lock); > >>>> } > >>>> +EXPORT_SYMBOL_GPL(lock_device_hotplug); > >>>> > >>>> void unlock_device_hotplug(void) > >>>> { > >>>> mutex_unlock(&device_hotplug_lock); > >>>> } > >>>> +EXPORT_SYMBOL_GPL(unlock_device_hotplug); > >>> > >>> If these are going to be "global" symbols, let's properly name them. > >>> device_hotplug_lock/unlock would be better. But I am _really_ nervous > >>> about letting stuff outside of the driver core mess with this, as people > >>> better know what they are doing. > >> > >> The only "problem" is that we have kernel modules (for paravirtualized > >> devices) that call add_memory(). This is Hyper-V right now, but we might > >> have other ones in the future. Without them we would not have to export > >> it. We might also get kernel modules that want to call remove_memory() - > >> which will require the device_hotplug_lock as of now. > >> > >> What we could do is > >> > >> a) add_memory() -> _add_memory() and don't export it > >> b) add_memory() takes the device_hotplug_lock and calls _add_memory() . > >> We export that one. > >> c) Use add_memory() in external modules only > >> > >> Similar wrapper would be needed e.g. for remove_memory() later on. > > > > That would be safer IMO, as it would prevent developers from using > > add_memory() without the lock, say. > > > > If the lock is always going to be required for add_memory(), make it > > hard (or event impossible) to use the latter without it. > > > > If there are no objections, I'll go into that direction. But I'll wait > for more comments regarding the general concept first. It is the middle of the merge window, and maintainers are really busy right now. I doubt you will get many review comments just yet...
On 17.08.2018 12:06, Greg Kroah-Hartman wrote: > On Fri, Aug 17, 2018 at 11:41:24AM +0200, David Hildenbrand wrote: >> On 17.08.2018 11:03, Rafael J. Wysocki wrote: >>> On Fri, Aug 17, 2018 at 10:56 AM David Hildenbrand <david@redhat.com> wrote: >>>> >>>> On 17.08.2018 10:41, Greg Kroah-Hartman wrote: >>>>> On Fri, Aug 17, 2018 at 09:59:00AM +0200, David Hildenbrand wrote: >>>>>> From: Vitaly Kuznetsov <vkuznets@redhat.com> >>>>>> >>>>>> Well require to call add_memory()/add_memory_resource() with >>>>>> device_hotplug_lock held, to avoid a lock inversion. Allow external modules >>>>>> (e.g. hv_balloon) that make use of add_memory()/add_memory_resource() to >>>>>> lock device hotplug. >>>>>> >>>>>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> >>>>>> [modify patch description] >>>>>> Signed-off-by: David Hildenbrand <david@redhat.com> >>>>>> --- >>>>>> drivers/base/core.c | 2 ++ >>>>>> 1 file changed, 2 insertions(+) >>>>>> >>>>>> diff --git a/drivers/base/core.c b/drivers/base/core.c >>>>>> index 04bbcd779e11..9010b9e942b5 100644 >>>>>> --- a/drivers/base/core.c >>>>>> +++ b/drivers/base/core.c >>>>>> @@ -700,11 +700,13 @@ void lock_device_hotplug(void) >>>>>> { >>>>>> mutex_lock(&device_hotplug_lock); >>>>>> } >>>>>> +EXPORT_SYMBOL_GPL(lock_device_hotplug); >>>>>> >>>>>> void unlock_device_hotplug(void) >>>>>> { >>>>>> mutex_unlock(&device_hotplug_lock); >>>>>> } >>>>>> +EXPORT_SYMBOL_GPL(unlock_device_hotplug); >>>>> >>>>> If these are going to be "global" symbols, let's properly name them. >>>>> device_hotplug_lock/unlock would be better. But I am _really_ nervous >>>>> about letting stuff outside of the driver core mess with this, as people >>>>> better know what they are doing. >>>> >>>> The only "problem" is that we have kernel modules (for paravirtualized >>>> devices) that call add_memory(). This is Hyper-V right now, but we might >>>> have other ones in the future. Without them we would not have to export >>>> it. We might also get kernel modules that want to call remove_memory() - >>>> which will require the device_hotplug_lock as of now. >>>> >>>> What we could do is >>>> >>>> a) add_memory() -> _add_memory() and don't export it >>>> b) add_memory() takes the device_hotplug_lock and calls _add_memory() . >>>> We export that one. >>>> c) Use add_memory() in external modules only >>>> >>>> Similar wrapper would be needed e.g. for remove_memory() later on. >>> >>> That would be safer IMO, as it would prevent developers from using >>> add_memory() without the lock, say. >>> >>> If the lock is always going to be required for add_memory(), make it >>> hard (or event impossible) to use the latter without it. >>> >> >> If there are no objections, I'll go into that direction. But I'll wait >> for more comments regarding the general concept first. > > It is the middle of the merge window, and maintainers are really busy > right now. I doubt you will get many review comments just yet... > This has been broken since 2015, so I guess it can wait a bit :)
On Fri, Aug 17, 2018 at 01:04:58PM +0200, David Hildenbrand wrote: > >> If there are no objections, I'll go into that direction. But I'll wait > >> for more comments regarding the general concept first. > > > > It is the middle of the merge window, and maintainers are really busy > > right now. I doubt you will get many review comments just yet... > > > > This has been broken since 2015, so I guess it can wait a bit :) I hope you figured out what needs to be locked why. Your patch description seems to be "only" about locking order ;) I tried to figure out and document that partially with 55adc1d05dca ("mm: add private lock to serialize memory hotplug operations"), and that wasn't easy to figure out. I was especially concerned about sprinkling lock/unlock_device_hotplug() calls, which has the potential to make it the next BKL thing.
On 17.08.2018 13:28, Heiko Carstens wrote: > On Fri, Aug 17, 2018 at 01:04:58PM +0200, David Hildenbrand wrote: >>>> If there are no objections, I'll go into that direction. But I'll wait >>>> for more comments regarding the general concept first. >>> >>> It is the middle of the merge window, and maintainers are really busy >>> right now. I doubt you will get many review comments just yet... >>> >> >> This has been broken since 2015, so I guess it can wait a bit :) > > I hope you figured out what needs to be locked why. Your patch description > seems to be "only" about locking order ;) Well I hope so, too ... but there is a reason for the RFC mark ;) There is definitely a lot of magic in the current code. And that's why it is also not that obvious that locking is wrong. To avoid/fix the locking order problem was the motivation for the original patch that dropped mem_hotplug_lock on one path. So I focused on that in my description. > > I tried to figure out and document that partially with 55adc1d05dca ("mm: > add private lock to serialize memory hotplug operations"), and that wasn't > easy to figure out. I was especially concerned about sprinkling Haven't seen that so far as that was reworked by 3f906ba23689 ("mm/memory-hotplug: switch locking to a percpu rwsem"). Thanks for the pointer. There is a long history to all this. > lock/unlock_device_hotplug() calls, which has the potential to make it the > next BKL thing. Well, the thing with memory hotplug and device_hotplug_lock is that a) ACPI already holds it while adding/removing memory via add_memory() b) we hold it during online/offline of memory (via sysfs calls to device_online()/device_offline()) So it is already pretty much involved in all memory hotplug/unplug activities on x86 (except paravirt). And as far as I understand, there are good reasons to hold the lock in core.c and ACPI. (as mentioned by Rafael) The exceptions are add_memory() called on s390x, hyper-v, xen and ppc (including manual probing). And device_online()/device_offline() called from the kernel. Holding device_hotplug_lock when adding/removing memory from the system doesn't sound too wrong (especially as devices are created/removed). At least that way (documenting and following the rules in the patch description) we might at least get locking right. I am very open to other suggestions (but as noted by Greg, many maintainers might be busy by know). E.g. When adding the memory block devices, we know that there won't be a driver to attach to (as there are no drivers for the "memory" subsystem) - the bus_probe_device() function that takes the device_lock() could pretty much be avoided for that case. But burying such special cases down in core driver code definitely won't make locking related to memory hotplug easier. Thanks for having a look!
On Fri, Aug 17, 2018 at 01:56:35PM +0200, David Hildenbrand wrote: > E.g. When adding the memory block devices, we know that there won't be a > driver to attach to (as there are no drivers for the "memory" subsystem) > - the bus_probe_device() function that takes the device_lock() could > pretty much be avoided for that case. But burying such special cases > down in core driver code definitely won't make locking related to memory > hotplug easier. You don't have to have a driver for a device if you don't want to, or you can just have a default one for all memory devices if you somehow need it. No reason to not do this if it makes things easier for you. thanks, greg k-h
diff --git a/drivers/base/core.c b/drivers/base/core.c index 04bbcd779e11..9010b9e942b5 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -700,11 +700,13 @@ void lock_device_hotplug(void) { mutex_lock(&device_hotplug_lock); } +EXPORT_SYMBOL_GPL(lock_device_hotplug); void unlock_device_hotplug(void) { mutex_unlock(&device_hotplug_lock); } +EXPORT_SYMBOL_GPL(unlock_device_hotplug); int lock_device_hotplug_sysfs(void) {