| Message ID | 20240202222603.141240-3-hamza.mahfooz@amd.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/3] driver core: bus: introduce can_remove() | expand |
[+cc Bartosz] On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: > Removing an amdgpu device that still has user space references allocated > to it causes undefined behaviour. So, implement amdgpu_pci_can_remove() > and disallow devices that still have files allocated to them from being > unbound. Maybe this would help for things that are completely built-in or soldered down, but nothing can prevent a user from physically pulling a card or cable, so I don't think this is a generic solution to the problem of dangling user space references. Maybe Bartosz's recent LPC talk is relevant: https://lpc.events/event/17/contributions/1627/ > Cc: stable@vger.kernel.org > Signed-off-by: Hamza Mahfooz <hamza.mahfooz@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 17 +++++++++++++++++ > 1 file changed, 17 insertions(+) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > index cc69005f5b46..cfa64f3c5be5 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > @@ -2323,6 +2323,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, > return ret; > } > > +static bool amdgpu_pci_can_remove(struct pci_dev *pdev) > +{ > + struct drm_device *dev = pci_get_drvdata(pdev); > + > + mutex_lock(&dev->filelist_mutex); > + > + if (!list_empty(&dev->filelist)) { > + mutex_unlock(&dev->filelist_mutex); > + return false; > + } > + > + mutex_unlock(&dev->filelist_mutex); > + > + return true; > +} > + > static void > amdgpu_pci_remove(struct pci_dev *pdev) > { > @@ -2929,6 +2945,7 @@ static struct pci_driver amdgpu_kms_pci_driver = { > .name = DRIVER_NAME, > .id_table = pciidlist, > .probe = amdgpu_pci_probe, > + .can_remove = amdgpu_pci_can_remove, > .remove = amdgpu_pci_remove, > .shutdown = amdgpu_pci_shutdown, > .driver.pm = &amdgpu_pm_ops, > -- > 2.43.0 >
On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: > Removing an amdgpu device that still has user space references allocated > to it causes undefined behaviour. Then fix that please. There should not be anything special about your hardware that all of the tens of thousands of other devices can't handle today. What happens when I yank your device out of a system with a pci hotplug bus? You can't prevent that either, so this should not be any different at all. sorry, but please, just fix your driver. greg k-h
On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: > Removing an amdgpu device that still has user space references allocated > to it causes undefined behaviour. So, implement amdgpu_pci_can_remove() > and disallow devices that still have files allocated to them from being > unbound. > > Cc: stable@vger.kernel.org > Signed-off-by: Hamza Mahfooz <hamza.mahfooz@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 17 +++++++++++++++++ > 1 file changed, 17 insertions(+) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > index cc69005f5b46..cfa64f3c5be5 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > @@ -2323,6 +2323,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, > return ret; > } > > +static bool amdgpu_pci_can_remove(struct pci_dev *pdev) > +{ > + struct drm_device *dev = pci_get_drvdata(pdev); > + > + mutex_lock(&dev->filelist_mutex); > + > + if (!list_empty(&dev->filelist)) { > + mutex_unlock(&dev->filelist_mutex); > + return false; > + } > + > + mutex_unlock(&dev->filelist_mutex); > + > + return true; Also, to be pedantic, this will not work as right after you returned "true" here, userspace could open a file, causing the same issue you are trying to prevent to have happen, happen. So even if we wanted to do this, which again, we do not, this isn't even a solution for it because it will still cause you problems. greg k-h
On Fri, Feb 02, 2024 at 03:40:03PM -0800, Greg Kroah-Hartman wrote: > On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: > > Removing an amdgpu device that still has user space references allocated > > to it causes undefined behaviour. > > Then fix that please. There should not be anything special about your > hardware that all of the tens of thousands of other devices can't handle > today. > > What happens when I yank your device out of a system with a pci hotplug > bus? You can't prevent that either, so this should not be any different > at all. > > sorry, but please, just fix your driver. fwiw Christian König from amd already rejected this too, I have no idea why this was submitted since the very elaborate plan I developed with a bunch of amd folks was to fix the various lifetime lolz we still have in drm. We unfortunately export the world of internal objects to userspace as uabi objects with dma_buf, dma_fence and everything else, but it's all fixable and we have the plan even documented: https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#device-hot-unplug So yeah anything that isn't that plan of record is very much no-go for drm drivers. Unless we change that plan of course, but that needs a documentation patch first and a big discussion. Aside from an absolute massive pile of kernel-internal refcounting bugs the really big one we agreed on after a lot of discussion is that SIGBUS on dma-buf mmaps is no-go for drm drivers, because it would break way too much userspace in ways which are simply not fixable (since sig handlers are shared in a process, which means the gl/vk driver cannot use it). Otherwise it's bog standard "fix the kernel bugs" work, just a lot of it. Cheers, Sima
Am 06.02.24 um 15:29 schrieb Daniel Vetter: > On Fri, Feb 02, 2024 at 03:40:03PM -0800, Greg Kroah-Hartman wrote: >> On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: >>> Removing an amdgpu device that still has user space references allocated >>> to it causes undefined behaviour. >> Then fix that please. There should not be anything special about your >> hardware that all of the tens of thousands of other devices can't handle >> today. >> >> What happens when I yank your device out of a system with a pci hotplug >> bus? You can't prevent that either, so this should not be any different >> at all. >> >> sorry, but please, just fix your driver. > fwiw Christian König from amd already rejected this too, I have no idea > why this was submitted Well that was my fault. I commented on an internal bug tracker that when sysfs bind/undbind is a different code path from PCI remove/re-scan we could try to reject it. Turned out it isn't a different code path. > since the very elaborate plan I developed with a > bunch of amd folks was to fix the various lifetime lolz we still have in > drm. We unfortunately export the world of internal objects to userspace as > uabi objects with dma_buf, dma_fence and everything else, but it's all > fixable and we have the plan even documented: > > https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#device-hot-unplug > > So yeah anything that isn't that plan of record is very much no-go for drm > drivers. Unless we change that plan of course, but that needs a > documentation patch first and a big discussion. > > Aside from an absolute massive pile of kernel-internal refcounting bugs > the really big one we agreed on after a lot of discussion is that SIGBUS > on dma-buf mmaps is no-go for drm drivers, because it would break way too > much userspace in ways which are simply not fixable (since sig handlers > are shared in a process, which means the gl/vk driver cannot use it). > > Otherwise it's bog standard "fix the kernel bugs" work, just a lot of it. Ignoring a few memory leaks because of messed up refcounting we actually got that working quite nicely. At least hot unplug / hot add seems to be working rather reliable in our internal testing. So it can't be that messed up. Regards, Christian. > > Cheers, Sima
On Tue, Feb 06, 2024 at 07:42:49PM +0100, Christian König wrote: > Am 06.02.24 um 15:29 schrieb Daniel Vetter: > > On Fri, Feb 02, 2024 at 03:40:03PM -0800, Greg Kroah-Hartman wrote: > > > On Fri, Feb 02, 2024 at 05:25:56PM -0500, Hamza Mahfooz wrote: > > > > Removing an amdgpu device that still has user space references allocated > > > > to it causes undefined behaviour. > > > Then fix that please. There should not be anything special about your > > > hardware that all of the tens of thousands of other devices can't handle > > > today. > > > > > > What happens when I yank your device out of a system with a pci hotplug > > > bus? You can't prevent that either, so this should not be any different > > > at all. > > > > > > sorry, but please, just fix your driver. > > fwiw Christian König from amd already rejected this too, I have no idea > > why this was submitted > > Well that was my fault. > > I commented on an internal bug tracker that when sysfs bind/undbind is a > different code path from PCI remove/re-scan we could try to reject it. > > Turned out it isn't a different code path. Yeah it's exactly the same code, and removing the sysfs stuff means we cant test hotunplug without physical hotunplugging stuff anymore. So really not great - if one is buggy so is the other, and sysfs allows us to control the timing a lot better to hit specific issues. -Sima > > since the very elaborate plan I developed with a > > bunch of amd folks was to fix the various lifetime lolz we still have in > > drm. We unfortunately export the world of internal objects to userspace as > > uabi objects with dma_buf, dma_fence and everything else, but it's all > > fixable and we have the plan even documented: > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#device-hot-unplug > > > > So yeah anything that isn't that plan of record is very much no-go for drm > > drivers. Unless we change that plan of course, but that needs a > > documentation patch first and a big discussion. > > > > Aside from an absolute massive pile of kernel-internal refcounting bugs > > the really big one we agreed on after a lot of discussion is that SIGBUS > > on dma-buf mmaps is no-go for drm drivers, because it would break way too > > much userspace in ways which are simply not fixable (since sig handlers > > are shared in a process, which means the gl/vk driver cannot use it). > > > > Otherwise it's bog standard "fix the kernel bugs" work, just a lot of it. > > Ignoring a few memory leaks because of messed up refcounting we actually got > that working quite nicely. > > At least hot unplug / hot add seems to be working rather reliable in our > internal testing. > > So it can't be that messed up. > > Regards, > Christian. > > > > > Cheers, Sima >
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index cc69005f5b46..cfa64f3c5be5 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -2323,6 +2323,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, return ret; } +static bool amdgpu_pci_can_remove(struct pci_dev *pdev) +{ + struct drm_device *dev = pci_get_drvdata(pdev); + + mutex_lock(&dev->filelist_mutex); + + if (!list_empty(&dev->filelist)) { + mutex_unlock(&dev->filelist_mutex); + return false; + } + + mutex_unlock(&dev->filelist_mutex); + + return true; +} + static void amdgpu_pci_remove(struct pci_dev *pdev) { @@ -2929,6 +2945,7 @@ static struct pci_driver amdgpu_kms_pci_driver = { .name = DRIVER_NAME, .id_table = pciidlist, .probe = amdgpu_pci_probe, + .can_remove = amdgpu_pci_can_remove, .remove = amdgpu_pci_remove, .shutdown = amdgpu_pci_shutdown, .driver.pm = &amdgpu_pm_ops,
Removing an amdgpu device that still has user space references allocated to it causes undefined behaviour. So, implement amdgpu_pci_can_remove() and disallow devices that still have files allocated to them from being unbound. Cc: stable@vger.kernel.org Signed-off-by: Hamza Mahfooz <hamza.mahfooz@amd.com> --- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)