| Message ID | 20180830144541.17740-1-vivek.gautam@codeaurora.org |
|---|---|
| Headers | show |
| Series | iommu/arm-smmu: Add runtime pm/sleep support | expand |
Hi Vivek, On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > From: Sricharan R <sricharan@codeaurora.org> > > The smmu device probe/remove and add/remove master device callbacks > gets called when the smmu is not linked to its master, that is without > the context of the master device. So calling runtime apis in those places > separately. > Global locks are also initialized before enabling runtime pm as the > runtime_resume() calls device_reset() which does tlb_sync_global() > that ultimately requires locks to be initialized. > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > [vivek: Cleanup pm runtime calls] > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 81 insertions(+), 8 deletions(-) [snip] > @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > dev_err(&pdev->dev, "removing device with active domains!\n"); > > + arm_smmu_rpm_get(smmu); > /* Turn the thing off */ > writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > + arm_smmu_rpm_put(smmu); > + > + if (pm_runtime_enabled(smmu->dev)) > + pm_runtime_force_suspend(smmu->dev); > + else > + clk_bulk_disable(smmu->num_clks, smmu->clks); > > - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > + clk_bulk_unprepare(smmu->num_clks, smmu->clks); Aren't we missing pm_runtime_disable() here? We'll have the enable count unbalanced if the driver is removed and probed again. Also, if we add pm_runtime_disable(), we can reorder things a bit and simplify into: arm_smmu_rpm_get(smmu); /* Turn the thing off */ writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); if (pm_runtime_enabled()) pm_runtime_disable(); arm_smmu_rpm_put(smmu); clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); Best regards, Tomasz
Hi Tomasz, On 9/7/2018 2:46 PM, Tomasz Figa wrote: > Hi Vivek, > > On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > <vivek.gautam@codeaurora.org> wrote: >> From: Sricharan R <sricharan@codeaurora.org> >> >> The smmu device probe/remove and add/remove master device callbacks >> gets called when the smmu is not linked to its master, that is without >> the context of the master device. So calling runtime apis in those places >> separately. >> Global locks are also initialized before enabling runtime pm as the >> runtime_resume() calls device_reset() which does tlb_sync_global() >> that ultimately requires locks to be initialized. >> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org> >> [vivek: Cleanup pm runtime calls] >> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> >> Reviewed-by: Tomasz Figa <tfiga@chromium.org> >> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> >> --- >> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- >> 1 file changed, 81 insertions(+), 8 deletions(-) > [snip] >> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) >> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) >> dev_err(&pdev->dev, "removing device with active domains!\n"); >> >> + arm_smmu_rpm_get(smmu); >> /* Turn the thing off */ >> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); >> + arm_smmu_rpm_put(smmu); >> + >> + if (pm_runtime_enabled(smmu->dev)) >> + pm_runtime_force_suspend(smmu->dev); >> + else >> + clk_bulk_disable(smmu->num_clks, smmu->clks); >> >> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); >> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > Aren't we missing pm_runtime_disable() here? We'll have the enable > count unbalanced if the driver is removed and probed again. pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not wrong. And, as mentioned in a previous thread [1], we were seeing a warning which we avoided by keeping force_suspend(). [1] https://lkml.org/lkml/2018/7/8/124 Thanks Vivek > > Also, if we add pm_runtime_disable(), we can reorder things a bit and > simplify into: > > arm_smmu_rpm_get(smmu); > > /* Turn the thing off */ > writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > > if (pm_runtime_enabled()) > pm_runtime_disable(); > arm_smmu_rpm_put(smmu); > > clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > > Best regards, > Tomasz
On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > Hi Tomasz, > > > On 9/7/2018 2:46 PM, Tomasz Figa wrote: > > Hi Vivek, > > > > On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > > <vivek.gautam@codeaurora.org> wrote: > >> From: Sricharan R <sricharan@codeaurora.org> > >> > >> The smmu device probe/remove and add/remove master device callbacks > >> gets called when the smmu is not linked to its master, that is without > >> the context of the master device. So calling runtime apis in those places > >> separately. > >> Global locks are also initialized before enabling runtime pm as the > >> runtime_resume() calls device_reset() which does tlb_sync_global() > >> that ultimately requires locks to be initialized. > >> > >> Signed-off-by: Sricharan R <sricharan@codeaurora.org> > >> [vivek: Cleanup pm runtime calls] > >> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > >> Reviewed-by: Tomasz Figa <tfiga@chromium.org> > >> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > >> --- > >> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > >> 1 file changed, 81 insertions(+), 8 deletions(-) > > [snip] > >> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > >> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > >> dev_err(&pdev->dev, "removing device with active domains!\n"); > >> > >> + arm_smmu_rpm_get(smmu); > >> /* Turn the thing off */ > >> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > >> + arm_smmu_rpm_put(smmu); > >> + > >> + if (pm_runtime_enabled(smmu->dev)) > >> + pm_runtime_force_suspend(smmu->dev); > >> + else > >> + clk_bulk_disable(smmu->num_clks, smmu->clks); > >> > >> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > >> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > > Aren't we missing pm_runtime_disable() here? We'll have the enable > > count unbalanced if the driver is removed and probed again. > > pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not > wrong. > And, as mentioned in a previous thread [1], we were seeing a warning > which we avoided > by keeping force_suspend(). > > [1] https://lkml.org/lkml/2018/7/8/124 I see, thanks. I didn't realize that pm_runtime_force_suspend() already disables runtime PM indeed. Sorry for the noise. Best regards, Tomasz
On Fri, Sep 7, 2018 at 3:22 PM Tomasz Figa <tfiga@chromium.org> wrote: > > On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > > > Hi Tomasz, > > > > > > On 9/7/2018 2:46 PM, Tomasz Figa wrote: > > > Hi Vivek, > > > > > > On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > > > <vivek.gautam@codeaurora.org> wrote: > > >> From: Sricharan R <sricharan@codeaurora.org> > > >> > > >> The smmu device probe/remove and add/remove master device callbacks > > >> gets called when the smmu is not linked to its master, that is without > > >> the context of the master device. So calling runtime apis in those places > > >> separately. > > >> Global locks are also initialized before enabling runtime pm as the > > >> runtime_resume() calls device_reset() which does tlb_sync_global() > > >> that ultimately requires locks to be initialized. > > >> > > >> Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > >> [vivek: Cleanup pm runtime calls] > > >> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > >> Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > >> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > >> --- > > >> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > > >> 1 file changed, 81 insertions(+), 8 deletions(-) > > > [snip] > > >> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > > >> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > > >> dev_err(&pdev->dev, "removing device with active domains!\n"); > > >> > > >> + arm_smmu_rpm_get(smmu); > > >> /* Turn the thing off */ > > >> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > > >> + arm_smmu_rpm_put(smmu); > > >> + > > >> + if (pm_runtime_enabled(smmu->dev)) > > >> + pm_runtime_force_suspend(smmu->dev); > > >> + else > > >> + clk_bulk_disable(smmu->num_clks, smmu->clks); > > >> > > >> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > > >> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > > > Aren't we missing pm_runtime_disable() here? We'll have the enable > > > count unbalanced if the driver is removed and probed again. > > > > pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not > > wrong. > > And, as mentioned in a previous thread [1], we were seeing a warning > > which we avoided > > by keeping force_suspend(). > > > > [1] https://lkml.org/lkml/2018/7/8/124 > > I see, thanks. I didn't realize that pm_runtime_force_suspend() > already disables runtime PM indeed. Sorry for the noise. Hi Tomasz, No problem. Thanks for looking back at it. Hi Robin, If you are fine with this series, then can you please consider giving Reviewed-by, so that we are certain that this series will go in the next merge window. Thanks Best regards Vivek
Hi Robin, On Fri, Sep 7, 2018 at 3:52 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > On Fri, Sep 7, 2018 at 3:22 PM Tomasz Figa <tfiga@chromium.org> wrote: > > > > On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > > > > > Hi Tomasz, > > > > > > > > > On 9/7/2018 2:46 PM, Tomasz Figa wrote: > > > > Hi Vivek, > > > > > > > > On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > > > > <vivek.gautam@codeaurora.org> wrote: > > > >> From: Sricharan R <sricharan@codeaurora.org> > > > >> > > > >> The smmu device probe/remove and add/remove master device callbacks > > > >> gets called when the smmu is not linked to its master, that is without > > > >> the context of the master device. So calling runtime apis in those places > > > >> separately. > > > >> Global locks are also initialized before enabling runtime pm as the > > > >> runtime_resume() calls device_reset() which does tlb_sync_global() > > > >> that ultimately requires locks to be initialized. > > > >> > > > >> Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > > >> [vivek: Cleanup pm runtime calls] > > > >> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > > >> Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > > >> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > > >> --- > > > >> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > > > >> 1 file changed, 81 insertions(+), 8 deletions(-) > > > > [snip] > > > >> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > > > >> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > > > >> dev_err(&pdev->dev, "removing device with active domains!\n"); > > > >> > > > >> + arm_smmu_rpm_get(smmu); > > > >> /* Turn the thing off */ > > > >> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > > > >> + arm_smmu_rpm_put(smmu); > > > >> + > > > >> + if (pm_runtime_enabled(smmu->dev)) > > > >> + pm_runtime_force_suspend(smmu->dev); > > > >> + else > > > >> + clk_bulk_disable(smmu->num_clks, smmu->clks); > > > >> > > > >> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > > > >> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > > > > Aren't we missing pm_runtime_disable() here? We'll have the enable > > > > count unbalanced if the driver is removed and probed again. > > > > > > pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not > > > wrong. > > > And, as mentioned in a previous thread [1], we were seeing a warning > > > which we avoided > > > by keeping force_suspend(). > > > > > > [1] https://lkml.org/lkml/2018/7/8/124 > > > > I see, thanks. I didn't realize that pm_runtime_force_suspend() > > already disables runtime PM indeed. Sorry for the noise. > > Hi Tomasz, > No problem. Thanks for looking back at it. > > Hi Robin, > If you are fine with this series, then can you please consider giving > Reviewed-by, so that we are certain that this series will go in the next merge > window. > Thanks Gentle ping. You ack will be very helpful in letting Will pull this series for 4.20. Thanks. Best regards Vivek -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Hi Robin, Will, On Tue, Sep 18, 2018 at 8:41 AM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > Hi Robin, > > On Fri, Sep 7, 2018 at 3:52 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > > > On Fri, Sep 7, 2018 at 3:22 PM Tomasz Figa <tfiga@chromium.org> wrote: > > > > > > On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > > > > > > > Hi Tomasz, > > > > > > > > > > > > On 9/7/2018 2:46 PM, Tomasz Figa wrote: > > > > > Hi Vivek, > > > > > > > > > > On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > > > > > <vivek.gautam@codeaurora.org> wrote: > > > > >> From: Sricharan R <sricharan@codeaurora.org> > > > > >> > > > > >> The smmu device probe/remove and add/remove master device callbacks > > > > >> gets called when the smmu is not linked to its master, that is without > > > > >> the context of the master device. So calling runtime apis in those places > > > > >> separately. > > > > >> Global locks are also initialized before enabling runtime pm as the > > > > >> runtime_resume() calls device_reset() which does tlb_sync_global() > > > > >> that ultimately requires locks to be initialized. > > > > >> > > > > >> Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > > > >> [vivek: Cleanup pm runtime calls] > > > > >> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > > > >> Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > > > >> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > > > >> --- > > > > >> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > > > > >> 1 file changed, 81 insertions(+), 8 deletions(-) > > > > > [snip] > > > > >> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > > > > >> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > > > > >> dev_err(&pdev->dev, "removing device with active domains!\n"); > > > > >> > > > > >> + arm_smmu_rpm_get(smmu); > > > > >> /* Turn the thing off */ > > > > >> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > > > > >> + arm_smmu_rpm_put(smmu); > > > > >> + > > > > >> + if (pm_runtime_enabled(smmu->dev)) > > > > >> + pm_runtime_force_suspend(smmu->dev); > > > > >> + else > > > > >> + clk_bulk_disable(smmu->num_clks, smmu->clks); > > > > >> > > > > >> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > > > > >> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > > > > > Aren't we missing pm_runtime_disable() here? We'll have the enable > > > > > count unbalanced if the driver is removed and probed again. > > > > > > > > pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not > > > > wrong. > > > > And, as mentioned in a previous thread [1], we were seeing a warning > > > > which we avoided > > > > by keeping force_suspend(). > > > > > > > > [1] https://lkml.org/lkml/2018/7/8/124 > > > > > > I see, thanks. I didn't realize that pm_runtime_force_suspend() > > > already disables runtime PM indeed. Sorry for the noise. > > > > Hi Tomasz, > > No problem. Thanks for looking back at it. > > > > Hi Robin, > > If you are fine with this series, then can you please consider giving > > Reviewed-by, so that we are certain that this series will go in the next merge > > window. > > Thanks > > Gentle ping. > You ack will be very helpful in letting Will pull this series for 4.20. > Thanks. I would really appreciate if you could provide your ack for this series. Or if there are any concerns, I am willing to address them. Thanks. Best regards Vivek > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Hi Vivek, On 2018-09-25 6:56 AM, Vivek Gautam wrote: > Hi Robin, Will, > > On Tue, Sep 18, 2018 at 8:41 AM Vivek Gautam > <vivek.gautam@codeaurora.org> wrote: >> >> Hi Robin, >> >> On Fri, Sep 7, 2018 at 3:52 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: >>> >>> On Fri, Sep 7, 2018 at 3:22 PM Tomasz Figa <tfiga@chromium.org> wrote: >>>> >>>> On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: >>>>> >>>>> Hi Tomasz, >>>>> >>>>> >>>>> On 9/7/2018 2:46 PM, Tomasz Figa wrote: >>>>>> Hi Vivek, >>>>>> >>>>>> On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam >>>>>> <vivek.gautam@codeaurora.org> wrote: >>>>>>> From: Sricharan R <sricharan@codeaurora.org> >>>>>>> >>>>>>> The smmu device probe/remove and add/remove master device callbacks >>>>>>> gets called when the smmu is not linked to its master, that is without >>>>>>> the context of the master device. So calling runtime apis in those places >>>>>>> separately. >>>>>>> Global locks are also initialized before enabling runtime pm as the >>>>>>> runtime_resume() calls device_reset() which does tlb_sync_global() >>>>>>> that ultimately requires locks to be initialized. >>>>>>> >>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org> >>>>>>> [vivek: Cleanup pm runtime calls] >>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> >>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org> >>>>>>> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> >>>>>>> --- >>>>>>> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- >>>>>>> 1 file changed, 81 insertions(+), 8 deletions(-) >>>>>> [snip] >>>>>>> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) >>>>>>> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) >>>>>>> dev_err(&pdev->dev, "removing device with active domains!\n"); >>>>>>> >>>>>>> + arm_smmu_rpm_get(smmu); >>>>>>> /* Turn the thing off */ >>>>>>> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); >>>>>>> + arm_smmu_rpm_put(smmu); >>>>>>> + >>>>>>> + if (pm_runtime_enabled(smmu->dev)) >>>>>>> + pm_runtime_force_suspend(smmu->dev); >>>>>>> + else >>>>>>> + clk_bulk_disable(smmu->num_clks, smmu->clks); >>>>>>> >>>>>>> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); >>>>>>> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); >>>>>> Aren't we missing pm_runtime_disable() here? We'll have the enable >>>>>> count unbalanced if the driver is removed and probed again. >>>>> >>>>> pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not >>>>> wrong. >>>>> And, as mentioned in a previous thread [1], we were seeing a warning >>>>> which we avoided >>>>> by keeping force_suspend(). >>>>> >>>>> [1] https://lkml.org/lkml/2018/7/8/124 >>>> >>>> I see, thanks. I didn't realize that pm_runtime_force_suspend() >>>> already disables runtime PM indeed. Sorry for the noise. >>> >>> Hi Tomasz, >>> No problem. Thanks for looking back at it. >>> >>> Hi Robin, >>> If you are fine with this series, then can you please consider giving >>> Reviewed-by, so that we are certain that this series will go in the next merge >>> window. >>> Thanks >> >> Gentle ping. >> You ack will be very helpful in letting Will pull this series for 4.20. >> Thanks. > > I would really appreciate if you could provide your ack for this series. > Or if there are any concerns, I am willing to address them. Apologies, I thought I'd replied to say I'd be getting to this shortly, but apparently not :( FWIW, "shortly" is now tomorrow - I don't *think* there's anything outstanding, but given the number of subtleties we've turned up so far I do just want one last thorough double-check to make sure. Thanks, Robin.
Hi Robin, On Wed, Sep 26, 2018 at 12:25 AM Robin Murphy <robin.murphy@arm.com> wrote: > > Hi Vivek, > > On 2018-09-25 6:56 AM, Vivek Gautam wrote: > > Hi Robin, Will, > > > > On Tue, Sep 18, 2018 at 8:41 AM Vivek Gautam > > <vivek.gautam@codeaurora.org> wrote: > >> > >> Hi Robin, > >> > >> On Fri, Sep 7, 2018 at 3:52 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > >>> > >>> On Fri, Sep 7, 2018 at 3:22 PM Tomasz Figa <tfiga@chromium.org> wrote: > >>>> > >>>> On Fri, Sep 7, 2018 at 6:38 PM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > >>>>> > >>>>> Hi Tomasz, > >>>>> > >>>>> > >>>>> On 9/7/2018 2:46 PM, Tomasz Figa wrote: > >>>>>> Hi Vivek, > >>>>>> > >>>>>> On Thu, Aug 30, 2018 at 11:46 PM Vivek Gautam > >>>>>> <vivek.gautam@codeaurora.org> wrote: > >>>>>>> From: Sricharan R <sricharan@codeaurora.org> > >>>>>>> > >>>>>>> The smmu device probe/remove and add/remove master device callbacks > >>>>>>> gets called when the smmu is not linked to its master, that is without > >>>>>>> the context of the master device. So calling runtime apis in those places > >>>>>>> separately. > >>>>>>> Global locks are also initialized before enabling runtime pm as the > >>>>>>> runtime_resume() calls device_reset() which does tlb_sync_global() > >>>>>>> that ultimately requires locks to be initialized. > >>>>>>> > >>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org> > >>>>>>> [vivek: Cleanup pm runtime calls] > >>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > >>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org> > >>>>>>> Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > >>>>>>> --- > >>>>>>> drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > >>>>>>> 1 file changed, 81 insertions(+), 8 deletions(-) > >>>>>> [snip] > >>>>>>> @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > >>>>>>> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > >>>>>>> dev_err(&pdev->dev, "removing device with active domains!\n"); > >>>>>>> > >>>>>>> + arm_smmu_rpm_get(smmu); > >>>>>>> /* Turn the thing off */ > >>>>>>> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > >>>>>>> + arm_smmu_rpm_put(smmu); > >>>>>>> + > >>>>>>> + if (pm_runtime_enabled(smmu->dev)) > >>>>>>> + pm_runtime_force_suspend(smmu->dev); > >>>>>>> + else > >>>>>>> + clk_bulk_disable(smmu->num_clks, smmu->clks); > >>>>>>> > >>>>>>> - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > >>>>>>> + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > >>>>>> Aren't we missing pm_runtime_disable() here? We'll have the enable > >>>>>> count unbalanced if the driver is removed and probed again. > >>>>> > >>>>> pm_runtime_force_suspend() does a pm_runtime_disable() also if i am not > >>>>> wrong. > >>>>> And, as mentioned in a previous thread [1], we were seeing a warning > >>>>> which we avoided > >>>>> by keeping force_suspend(). > >>>>> > >>>>> [1] https://lkml.org/lkml/2018/7/8/124 > >>>> > >>>> I see, thanks. I didn't realize that pm_runtime_force_suspend() > >>>> already disables runtime PM indeed. Sorry for the noise. > >>> > >>> Hi Tomasz, > >>> No problem. Thanks for looking back at it. > >>> > >>> Hi Robin, > >>> If you are fine with this series, then can you please consider giving > >>> Reviewed-by, so that we are certain that this series will go in the next merge > >>> window. > >>> Thanks > >> > >> Gentle ping. > >> You ack will be very helpful in letting Will pull this series for 4.20. > >> Thanks. > > > > I would really appreciate if you could provide your ack for this series. > > Or if there are any concerns, I am willing to address them. > > Apologies, I thought I'd replied to say I'd be getting to this shortly, > but apparently not :( > > FWIW, "shortly" is now tomorrow - I don't *think* there's anything > outstanding, but given the number of subtleties we've turned up so far I > do just want one last thorough double-check to make sure. Cool. TIA for the review. I hope we have something that we can land :), and then work further to take care of addressing other needs of this driver. Thanks. Best regards Vivek > > Thanks, > Robin. > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On 30/08/18 15:45, Vivek Gautam wrote: > From: Sricharan R <sricharan@codeaurora.org> > > The smmu needs to be functional only when the respective > master's using it are active. The device_link feature > helps to track such functional dependencies, so that the > iommu gets powered when the master device enables itself > using pm_runtime. So by adapting the smmu driver for > runtime pm, above said dependency can be addressed. > > This patch adds the pm runtime/sleep callbacks to the > driver and also the functions to parse the smmu clocks > from DT and enable them in resume/suspend. > > Also, while we enable the runtime pm add a pm sleep suspend > callback that pushes devices to low power state by turning > the clocks off in a system sleep. > Also add corresponding clock enable path in resume callback. > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > Signed-off-by: Archit Taneja <architt@codeaurora.org> > [vivek: rework for clock and pm ops] > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 74 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index fd1b80ef9490..d900e007c3c9 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -48,6 +48,7 @@ > #include <linux/of_iommu.h> > #include <linux/pci.h> > #include <linux/platform_device.h> > +#include <linux/pm_runtime.h> > #include <linux/slab.h> > #include <linux/spinlock.h> > > @@ -205,6 +206,8 @@ struct arm_smmu_device { > u32 num_global_irqs; > u32 num_context_irqs; > unsigned int *irqs; > + struct clk_bulk_data *clks; > + int num_clks; > > u32 cavium_id_base; /* Specific to Cavium */ > > @@ -1896,10 +1899,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu) > struct arm_smmu_match_data { > enum arm_smmu_arch_version version; > enum arm_smmu_implementation model; > + const char * const *clks; > + int num_clks; > }; > > #define ARM_SMMU_MATCH_DATA(name, ver, imp) \ > -static struct arm_smmu_match_data name = { .version = ver, .model = imp } > +static const struct arm_smmu_match_data name = { .version = ver, .model = imp } > > ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU); > ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU); > @@ -1918,6 +1923,23 @@ static const struct of_device_id arm_smmu_of_match[] = { > }; > MODULE_DEVICE_TABLE(of, arm_smmu_of_match); > > +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu, > + const char * const *clks) > +{ > + int i; > + > + if (smmu->num_clks < 1) > + return; > + > + smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks, > + sizeof(*smmu->clks), GFP_KERNEL); > + if (!smmu->clks) > + return; > + > + for (i = 0; i < smmu->num_clks; i++) > + smmu->clks[i].id = clks[i]; > +} > + > #ifdef CONFIG_ACPI > static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu) > { > @@ -2000,6 +2022,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev, > data = of_device_get_match_data(dev); > smmu->version = data->version; > smmu->model = data->model; > + smmu->num_clks = data->num_clks; > + > + arm_smmu_fill_clk_data(smmu, data->clks); > > parse_driver_options(smmu); > > @@ -2098,6 +2123,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > smmu->irqs[i] = irq; > } > > + err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks); > + if (err) > + return err; > + > + err = clk_bulk_prepare_enable(smmu->num_clks, smmu->clks); > + if (err) > + return err; > + Hmm, if we error out beyond here it looks like we should strictly balance that prepare/enable before devres does the clk_bulk_put(), however the probe error path is starting to look like it needs a bit of love in general, so I might just spin a cleanup patch on top (and even then only for the sake of not being a bad example; SMMU probe failure is never a realistic situation for the system to actually recover from). Otherwise, Reviewed-by: Robin Murphy <robin.murphy@arm.com> > err = arm_smmu_device_cfg_probe(smmu); > if (err) > return err; > @@ -2184,6 +2217,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > > /* Turn the thing off */ > writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > + > + clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > + > return 0; > } > > @@ -2192,15 +2228,50 @@ static void arm_smmu_device_shutdown(struct platform_device *pdev) > arm_smmu_device_remove(pdev); > } > > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > { > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > + int ret; > + > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > + if (ret) > + return ret; > > arm_smmu_device_reset(smmu); > + > return 0; > } > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > +{ > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > + > + clk_bulk_disable(smmu->num_clks, smmu->clks); > + > + return 0; > +} > + > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > +{ > + if (pm_runtime_suspended(dev)) > + return 0; > + > + return arm_smmu_runtime_resume(dev); > +} > + > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) > +{ > + if (pm_runtime_suspended(dev)) > + return 0; > + > + return arm_smmu_runtime_suspend(dev); > +} > + > +static const struct dev_pm_ops arm_smmu_pm_ops = { > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) > + SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend, > + arm_smmu_runtime_resume, NULL) > +}; > > static struct platform_driver arm_smmu_driver = { > .driver = { >
On 30/08/18 15:45, Vivek Gautam wrote: > From: Sricharan R <sricharan@codeaurora.org> > > The smmu device probe/remove and add/remove master device callbacks > gets called when the smmu is not linked to its master, that is without > the context of the master device. So calling runtime apis in those places > separately. > Global locks are also initialized before enabling runtime pm as the > runtime_resume() calls device_reset() which does tlb_sync_global() > that ultimately requires locks to be initialized. To the best of my knowledge in this stuff (which is still not quite enough to be *truly* confident...), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > [vivek: Cleanup pm runtime calls] > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 81 insertions(+), 8 deletions(-) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index d900e007c3c9..1bf542010be7 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = { > { 0, NULL}, > }; > > +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu) > +{ > + if (pm_runtime_enabled(smmu->dev)) > + return pm_runtime_get_sync(smmu->dev); > + > + return 0; > +} > + > +static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu) > +{ > + if (pm_runtime_enabled(smmu->dev)) > + pm_runtime_put(smmu->dev); > +} > + > static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) > { > return container_of(dom, struct arm_smmu_domain, domain); > @@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain) > struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); > struct arm_smmu_device *smmu = smmu_domain->smmu; > struct arm_smmu_cfg *cfg = &smmu_domain->cfg; > - int irq; > + int ret, irq; > > if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY) > return; > > + ret = arm_smmu_rpm_get(smmu); > + if (ret < 0) > + return; > + > /* > * Disable the context bank and free the page tables before freeing > * it. > @@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain) > > free_io_pgtable_ops(smmu_domain->pgtbl_ops); > __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx); > + > + arm_smmu_rpm_put(smmu); > } > > static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) > @@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > return -ENODEV; > > smmu = fwspec_smmu(fwspec); > + > + ret = arm_smmu_rpm_get(smmu); > + if (ret < 0) > + return ret; > + > /* Ensure that the domain is finalised */ > ret = arm_smmu_init_domain_context(domain, smmu); > if (ret < 0) > - return ret; > + goto rpm_put; > > /* > * Sanity check the domain. We don't support domains across > @@ -1226,33 +1251,50 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > dev_err(dev, > "cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n", > dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev)); > - return -EINVAL; > + ret = -EINVAL; > + goto rpm_put; > } > > /* Looks ok, so add the device to the domain */ > - return arm_smmu_domain_add_master(smmu_domain, fwspec); > + ret = arm_smmu_domain_add_master(smmu_domain, fwspec); > + > +rpm_put: > + arm_smmu_rpm_put(smmu); > + return ret; > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > + struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > + int ret; > > if (!ops) > return -ENODEV; > > - return ops->map(ops, iova, paddr, size, prot); > + arm_smmu_rpm_get(smmu); > + ret = ops->map(ops, iova, paddr, size, prot); > + arm_smmu_rpm_put(smmu); > + > + return ret; > } > > static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > + struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > + size_t ret; > > if (!ops) > return 0; > > - return ops->unmap(ops, iova, size); > + arm_smmu_rpm_get(smmu); > + ret = ops->unmap(ops, iova, size); > + arm_smmu_rpm_put(smmu); > + > + return ret; > } > > static void arm_smmu_iotlb_sync(struct iommu_domain *domain) > @@ -1407,7 +1449,13 @@ static int arm_smmu_add_device(struct device *dev) > while (i--) > cfg->smendx[i] = INVALID_SMENDX; > > + ret = arm_smmu_rpm_get(smmu); > + if (ret < 0) > + goto out_cfg_free; > + > ret = arm_smmu_master_alloc_smes(dev); > + arm_smmu_rpm_put(smmu); > + > if (ret) > goto out_cfg_free; > > @@ -1427,7 +1475,7 @@ static void arm_smmu_remove_device(struct device *dev) > struct iommu_fwspec *fwspec = dev->iommu_fwspec; > struct arm_smmu_master_cfg *cfg; > struct arm_smmu_device *smmu; > - > + int ret; > > if (!fwspec || fwspec->ops != &arm_smmu_ops) > return; > @@ -1435,8 +1483,15 @@ static void arm_smmu_remove_device(struct device *dev) > cfg = fwspec->iommu_priv; > smmu = cfg->smmu; > > + ret = arm_smmu_rpm_get(smmu); > + if (ret < 0) > + return; > + > iommu_device_unlink(&smmu->iommu, dev); > arm_smmu_master_free_smes(fwspec); > + > + arm_smmu_rpm_put(smmu); > + > iommu_group_remove_device(dev); > kfree(fwspec->iommu_priv); > iommu_fwspec_free(dev); > @@ -2180,6 +2235,17 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > arm_smmu_device_reset(smmu); > arm_smmu_test_smr_masks(smmu); > > + /* > + * We want to avoid touching dev->power.lock in fastpaths unless > + * it's really going to do something useful - pm_runtime_enabled() > + * can serve as an ideal proxy for that decision. So, conditionally > + * enable pm_runtime. > + */ > + if (dev->pm_domain) { > + pm_runtime_set_active(dev); > + pm_runtime_enable(dev); > + } > + > /* > * For ACPI and generic DT bindings, an SMMU will be probed before > * any device which might need it, so we want the bus ops in place > @@ -2215,10 +2281,17 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS)) > dev_err(&pdev->dev, "removing device with active domains!\n"); > > + arm_smmu_rpm_get(smmu); > /* Turn the thing off */ > writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > + arm_smmu_rpm_put(smmu); > + > + if (pm_runtime_enabled(smmu->dev)) > + pm_runtime_force_suspend(smmu->dev); > + else > + clk_bulk_disable(smmu->num_clks, smmu->clks); > > - clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > + clk_bulk_unprepare(smmu->num_clks, smmu->clks); > > return 0; > } >
On 30/08/18 15:45, Vivek Gautam wrote: > From: Sricharan R <sricharan@codeaurora.org> > > Finally add the device link between the master device and > smmu, so that the smmu gets runtime enabled/disabled only when the > master needs it. This is done from add_device callback which gets > called once when the master is added to the smmu. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index 1bf542010be7..166c8c6da24f 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1461,6 +1461,9 @@ static int arm_smmu_add_device(struct device *dev) > > iommu_device_link(&smmu->iommu, dev); > > + device_link_add(dev, smmu->dev, > + DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER); > + > return 0; > > out_cfg_free: >
On 30/08/18 15:45, Vivek Gautam wrote: > qcom,smmu-v2 is an arm,smmu-v2 implementation with specific > clock and power requirements. > On msm8996, multiple cores, viz. mdss, video, etc. use this > smmu. On sdm845, this smmu is used with gpu. > Add bindings for the same. > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Rob Herring <robh@kernel.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index 166c8c6da24f..411e5ac57c64 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -119,6 +119,7 @@ enum arm_smmu_implementation { > GENERIC_SMMU, > ARM_MMU500, > CAVIUM_SMMUV2, > + QCOM_SMMUV2, Hmm, it seems we don't actually need this right now, but maybe that just means there's more imp-def registers and/or errata to come ;) Either way I guess there's no real harm in having it. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > }; > > struct arm_smmu_s2cr { > @@ -1970,6 +1971,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, ARM_SMMU_V1_64K, GENERIC_SMMU); > ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500); > ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2); > > +static const char * const qcom_smmuv2_clks[] = { > + "bus", "iface", > +}; > + > +static const struct arm_smmu_match_data qcom_smmuv2 = { > + .version = ARM_SMMU_V2, > + .model = QCOM_SMMUV2, > + .clks = qcom_smmuv2_clks, > + .num_clks = ARRAY_SIZE(qcom_smmuv2_clks), > +}; > + > static const struct of_device_id arm_smmu_of_match[] = { > { .compatible = "arm,smmu-v1", .data = &smmu_generic_v1 }, > { .compatible = "arm,smmu-v2", .data = &smmu_generic_v2 }, > @@ -1977,6 +1989,7 @@ static const struct of_device_id arm_smmu_of_match[] = { > { .compatible = "arm,mmu-401", .data = &arm_mmu401 }, > { .compatible = "arm,mmu-500", .data = &arm_mmu500 }, > { .compatible = "cavium,smmu-v2", .data = &cavium_smmuv2 }, > + { .compatible = "qcom,smmu-v2", .data = &qcom_smmuv2 }, > { }, > }; > MODULE_DEVICE_TABLE(of, arm_smmu_of_match); >
Hi Robin, On Wed, Sep 26, 2018 at 9:29 PM Robin Murphy <robin.murphy@arm.com> wrote: > > On 30/08/18 15:45, Vivek Gautam wrote: > > qcom,smmu-v2 is an arm,smmu-v2 implementation with specific > > clock and power requirements. > > On msm8996, multiple cores, viz. mdss, video, etc. use this > > smmu. On sdm845, this smmu is used with gpu. > > Add bindings for the same. > > > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > Reviewed-by: Rob Herring <robh@kernel.org> > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > --- > > drivers/iommu/arm-smmu.c | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index 166c8c6da24f..411e5ac57c64 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -119,6 +119,7 @@ enum arm_smmu_implementation { > > GENERIC_SMMU, > > ARM_MMU500, > > CAVIUM_SMMUV2, > > + QCOM_SMMUV2, > > Hmm, it seems we don't actually need this right now, but maybe that just > means there's more imp-def registers and/or errata to come ;) > > Either way I guess there's no real harm in having it. Thanks for the review. Best regards Vivek > > Reviewed-by: Robin Murphy <robin.murphy@arm.com> > > > }; > > > > struct arm_smmu_s2cr { > > @@ -1970,6 +1971,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, ARM_SMMU_V1_64K, GENERIC_SMMU); > > ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500); > > ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2); > > > > +static const char * const qcom_smmuv2_clks[] = { > > + "bus", "iface", > > +}; > > + > > +static const struct arm_smmu_match_data qcom_smmuv2 = { > > + .version = ARM_SMMU_V2, > > + .model = QCOM_SMMUV2, > > + .clks = qcom_smmuv2_clks, > > + .num_clks = ARRAY_SIZE(qcom_smmuv2_clks), > > +}; > > + > > static const struct of_device_id arm_smmu_of_match[] = { > > { .compatible = "arm,smmu-v1", .data = &smmu_generic_v1 }, > > { .compatible = "arm,smmu-v2", .data = &smmu_generic_v2 }, > > @@ -1977,6 +1989,7 @@ static const struct of_device_id arm_smmu_of_match[] = { > > { .compatible = "arm,mmu-401", .data = &arm_mmu401 }, > > { .compatible = "arm,mmu-500", .data = &arm_mmu500 }, > > { .compatible = "cavium,smmu-v2", .data = &cavium_smmuv2 }, > > + { .compatible = "qcom,smmu-v2", .data = &qcom_smmuv2 }, > > { }, > > }; > > MODULE_DEVICE_TABLE(of, arm_smmu_of_match); > > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On Wed, Sep 26, 2018 at 8:57 PM Robin Murphy <robin.murphy@arm.com> wrote: > > On 30/08/18 15:45, Vivek Gautam wrote: > > From: Sricharan R <sricharan@codeaurora.org> > > > > The smmu needs to be functional only when the respective > > master's using it are active. The device_link feature > > helps to track such functional dependencies, so that the > > iommu gets powered when the master device enables itself > > using pm_runtime. So by adapting the smmu driver for > > runtime pm, above said dependency can be addressed. > > > > This patch adds the pm runtime/sleep callbacks to the > > driver and also the functions to parse the smmu clocks > > from DT and enable them in resume/suspend. > > > > Also, while we enable the runtime pm add a pm sleep suspend > > callback that pushes devices to low power state by turning > > the clocks off in a system sleep. > > Also add corresponding clock enable path in resume callback. > > > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > Signed-off-by: Archit Taneja <architt@codeaurora.org> > > [vivek: rework for clock and pm ops] > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > --- > > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > > 1 file changed, 74 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index fd1b80ef9490..d900e007c3c9 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -48,6 +48,7 @@ > > #include <linux/of_iommu.h> > > #include <linux/pci.h> > > #include <linux/platform_device.h> > > +#include <linux/pm_runtime.h> > > #include <linux/slab.h> > > #include <linux/spinlock.h> > > > > @@ -205,6 +206,8 @@ struct arm_smmu_device { > > u32 num_global_irqs; > > u32 num_context_irqs; > > unsigned int *irqs; > > + struct clk_bulk_data *clks; > > + int num_clks; > > > > u32 cavium_id_base; /* Specific to Cavium */ > > > > @@ -1896,10 +1899,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu) > > struct arm_smmu_match_data { > > enum arm_smmu_arch_version version; > > enum arm_smmu_implementation model; > > + const char * const *clks; > > + int num_clks; > > }; > > > > #define ARM_SMMU_MATCH_DATA(name, ver, imp) \ > > -static struct arm_smmu_match_data name = { .version = ver, .model = imp } > > +static const struct arm_smmu_match_data name = { .version = ver, .model = imp } > > > > ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU); > > ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU); > > @@ -1918,6 +1923,23 @@ static const struct of_device_id arm_smmu_of_match[] = { > > }; > > MODULE_DEVICE_TABLE(of, arm_smmu_of_match); > > > > +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu, > > + const char * const *clks) > > +{ > > + int i; > > + > > + if (smmu->num_clks < 1) > > + return; > > + > > + smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks, > > + sizeof(*smmu->clks), GFP_KERNEL); > > + if (!smmu->clks) > > + return; > > + > > + for (i = 0; i < smmu->num_clks; i++) > > + smmu->clks[i].id = clks[i]; > > +} > > + > > #ifdef CONFIG_ACPI > > static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu) > > { > > @@ -2000,6 +2022,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev, > > data = of_device_get_match_data(dev); > > smmu->version = data->version; > > smmu->model = data->model; > > + smmu->num_clks = data->num_clks; > > + > > + arm_smmu_fill_clk_data(smmu, data->clks); > > > > parse_driver_options(smmu); > > > > @@ -2098,6 +2123,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev) > > smmu->irqs[i] = irq; > > } > > > > + err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks); > > + if (err) > > + return err; > > + > > + err = clk_bulk_prepare_enable(smmu->num_clks, smmu->clks); > > + if (err) > > + return err; > > + > > Hmm, if we error out beyond here it looks like we should strictly > balance that prepare/enable before devres does the clk_bulk_put(), > however the probe error path is starting to look like it needs a bit of > love in general, so I might just spin a cleanup patch on top (and even > then only for the sake of not being a bad example; SMMU probe failure is > never a realistic situation for the system to actually recover from). Sure Robin. Thanks for the review on the series. Let me know, I can spin a change for probe failure path cleanup. Best regards Vivek > > Otherwise, > > Reviewed-by: Robin Murphy <robin.murphy@arm.com> > > > err = arm_smmu_device_cfg_probe(smmu); > > if (err) > > return err; > > @@ -2184,6 +2217,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev) > > > > /* Turn the thing off */ > > writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0); > > + > > + clk_bulk_disable_unprepare(smmu->num_clks, smmu->clks); > > + > > return 0; > > } > > > > @@ -2192,15 +2228,50 @@ static void arm_smmu_device_shutdown(struct platform_device *pdev) > > arm_smmu_device_remove(pdev); > > } > > > > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > > { > > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > + int ret; > > + > > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > > + if (ret) > > + return ret; > > > > arm_smmu_device_reset(smmu); > > + > > return 0; > > } > > > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > > +{ > > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > + > > + clk_bulk_disable(smmu->num_clks, smmu->clks); > > + > > + return 0; > > +} > > + > > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > +{ > > + if (pm_runtime_suspended(dev)) > > + return 0; > > + > > + return arm_smmu_runtime_resume(dev); > > +} > > + > > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) > > +{ > > + if (pm_runtime_suspended(dev)) > > + return 0; > > + > > + return arm_smmu_runtime_suspend(dev); > > +} > > + > > +static const struct dev_pm_ops arm_smmu_pm_ops = { > > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) > > + SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend, > > + arm_smmu_runtime_resume, NULL) > > +}; > > > > static struct platform_driver arm_smmu_driver = { > > .driver = { > > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > From: Sricharan R <sricharan@codeaurora.org> > > The smmu needs to be functional only when the respective > master's using it are active. The device_link feature > helps to track such functional dependencies, so that the > iommu gets powered when the master device enables itself > using pm_runtime. So by adapting the smmu driver for > runtime pm, above said dependency can be addressed. > > This patch adds the pm runtime/sleep callbacks to the > driver and also the functions to parse the smmu clocks > from DT and enable them in resume/suspend. > > Also, while we enable the runtime pm add a pm sleep suspend > callback that pushes devices to low power state by turning > the clocks off in a system sleep. > Also add corresponding clock enable path in resume callback. > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > Signed-off-by: Archit Taneja <architt@codeaurora.org> > [vivek: rework for clock and pm ops] > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 74 insertions(+), 3 deletions(-) > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c [...] > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > { > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > + int ret; > + > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > + if (ret) > + return ret; > > arm_smmu_device_reset(smmu); > + > return 0; > } > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > +{ > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > + > + clk_bulk_disable(smmu->num_clks, smmu->clks); > + > + return 0; > +} > + > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > +{ > + if (pm_runtime_suspended(dev)) > + return 0; Looks like you should be able use pm_runtime_force_resume(), instead of using this local trick. Unless I am missing something, of course. In other words, just assign the system sleep callbacks for resume, to pm_runtime_force_resume(). And vice verse for the system suspend callbacks, pm_runtime_force_suspend(), of course. > + > + return arm_smmu_runtime_resume(dev); > +} > + > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) > +{ > + if (pm_runtime_suspended(dev)) > + return 0; > + > + return arm_smmu_runtime_suspend(dev); > +} > + > +static const struct dev_pm_ops arm_smmu_pm_ops = { > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) I am wondering if using the ->suspend|resume() callback is really "late/early" enough in the device suspend phase? Others is using the noirq phase and some is even using the syscore ops. Of course it depends on the behavior of the consumers of iommu device, and I guess not everyone is using device links, which for sure improves things in this regards as well. > + SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend, > + arm_smmu_runtime_resume, NULL) > +}; > > static struct platform_driver arm_smmu_driver = { > .driver = { > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation > BTW, apologize for very late review comments. Besides the above comments, the series looks good to me. Kind regards Uffe
Hi Vivek, On Thu, Aug 30, 2018 at 08:15:36PM +0530, Vivek Gautam wrote: > This series provides the support for turning on the arm-smmu's > clocks/power domains using runtime pm. This is done using > device links between smmu and client devices. The device link > framework keeps the two devices in correct order for power-cycling > across runtime PM or across system-wide PM. > > With addition of a new device link flag DL_FLAG_AUTOREMOVE_SUPPLIER [7], > the device links created between arm-smmu and its clients will be > automatically purged when arm-smmu driver unbinds from its device. > > As not all implementations support clock/power gating, we are checking > for a valid 'smmu->dev's pm_domain' to conditionally enable the runtime > power management for such smmu implementations that can support it. > Otherwise, the clocks are turned to be always on in .probe until .remove. > With conditional runtime pm now, we avoid touching dev->power.lock > in fastpaths for smmu implementations that don't need to do anything > useful with pm_runtime. > This lets us to use the much-argued pm_runtime_get_sync/put_sync() > calls in map/unmap callbacks so that the clients do not have to > worry about handling any of the arm-smmu's power. > > This series also adds support for Qcom's arm-smmu-v2 variant that > has different clocks and power requirements. > > Previous version of this patch series is @ [1]. > > Build tested the series based on 4.19-rc1. I'm going to send my pull request to Joerg early next week (probably Monday), but I'm not keen to include this whilst it has outstanding comments from Ulf. Your errata workaround patch is in a similar situation, with outstanding comments from Robin. Will
HI Ulf, On Fri, Sep 28, 2018 at 5:30 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: > > On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > From: Sricharan R <sricharan@codeaurora.org> > > > > The smmu needs to be functional only when the respective > > master's using it are active. The device_link feature > > helps to track such functional dependencies, so that the > > iommu gets powered when the master device enables itself > > using pm_runtime. So by adapting the smmu driver for > > runtime pm, above said dependency can be addressed. > > > > This patch adds the pm runtime/sleep callbacks to the > > driver and also the functions to parse the smmu clocks > > from DT and enable them in resume/suspend. > > > > Also, while we enable the runtime pm add a pm sleep suspend > > callback that pushes devices to low power state by turning > > the clocks off in a system sleep. > > Also add corresponding clock enable path in resume callback. > > > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > Signed-off-by: Archit Taneja <architt@codeaurora.org> > > [vivek: rework for clock and pm ops] > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > --- > > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > > 1 file changed, 74 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > [...] > > > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > > { > > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > + int ret; > > + > > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > > + if (ret) > > + return ret; > > > > arm_smmu_device_reset(smmu); > > + > > return 0; > > } > > > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > > +{ > > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > + > > + clk_bulk_disable(smmu->num_clks, smmu->clks); > > + > > + return 0; > > +} > > + > > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > +{ > > + if (pm_runtime_suspended(dev)) > > + return 0; > > Looks like you should be able use pm_runtime_force_resume(), instead > of using this local trick. Unless I am missing something, of course. > > In other words, just assign the system sleep callbacks for resume, to > pm_runtime_force_resume(). And vice verse for the system suspend > callbacks, pm_runtime_force_suspend(), of course. Thanks for the review. I will change this as suggested. > > > + > > + return arm_smmu_runtime_resume(dev); > > +} > > + > > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) > > +{ > > + if (pm_runtime_suspended(dev)) > > + return 0; > > + > > + return arm_smmu_runtime_suspend(dev); > > +} > > + > > +static const struct dev_pm_ops arm_smmu_pm_ops = { > > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) > > I am wondering if using the ->suspend|resume() callback is really > "late/early" enough in the device suspend phase? > > Others is using the noirq phase and some is even using the syscore > ops. Of course it depends on the behavior of the consumers of iommu > device, and I guess not everyone is using device links, which for sure > improves things in this regards as well. Well yes, as you said the device links should be able to take care of maintaining the correct suspend/resume order of smmu and its clients, or am I missing your point here? Let me know and I will be happy to incorporate any suggestions. Thanks Regards Vivek > > > + SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend, > > + arm_smmu_runtime_resume, NULL) > > +}; > > > > static struct platform_driver arm_smmu_driver = { > > .driver = { > > -- > > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > > of Code Aurora Forum, hosted by The Linux Foundation > > > > BTW, apologize for very late review comments. > > Besides the above comments, the series looks good to me. > > Kind regards > Uffe > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Will, On Fri, Sep 28, 2018 at 7:27 PM Will Deacon <will.deacon@arm.com> wrote: > > Hi Vivek, > > On Thu, Aug 30, 2018 at 08:15:36PM +0530, Vivek Gautam wrote: > > This series provides the support for turning on the arm-smmu's > > clocks/power domains using runtime pm. This is done using > > device links between smmu and client devices. The device link > > framework keeps the two devices in correct order for power-cycling > > across runtime PM or across system-wide PM. > > > > With addition of a new device link flag DL_FLAG_AUTOREMOVE_SUPPLIER [7], > > the device links created between arm-smmu and its clients will be > > automatically purged when arm-smmu driver unbinds from its device. > > > > As not all implementations support clock/power gating, we are checking > > for a valid 'smmu->dev's pm_domain' to conditionally enable the runtime > > power management for such smmu implementations that can support it. > > Otherwise, the clocks are turned to be always on in .probe until .remove. > > With conditional runtime pm now, we avoid touching dev->power.lock > > in fastpaths for smmu implementations that don't need to do anything > > useful with pm_runtime. > > This lets us to use the much-argued pm_runtime_get_sync/put_sync() > > calls in map/unmap callbacks so that the clients do not have to > > worry about handling any of the arm-smmu's power. > > > > This series also adds support for Qcom's arm-smmu-v2 variant that > > has different clocks and power requirements. > > > > Previous version of this patch series is @ [1]. > > > > Build tested the series based on 4.19-rc1. > > I'm going to send my pull request to Joerg early next week (probably > Monday), but I'm not keen to include this whilst it has outstanding comments > from Ulf. Your errata workaround patch is in a similar situation, with > outstanding comments from Robin. I am going to address Ulf's comments for pm_runtime_force_suspend/resume() calls in system sleep callbacks and respin the series unless he has any more comments regarding the early/late nature of suspend/resume. So will it do if I respin the series today after waiting for Ulf? The workaround series is going for a discussion now, so i think it can wait. Thanks Best regards Vivek > > Will > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
On 1 October 2018 at 07:49, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > HI Ulf, > > On Fri, Sep 28, 2018 at 5:30 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: >> >> On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: >> > From: Sricharan R <sricharan@codeaurora.org> >> > >> > The smmu needs to be functional only when the respective >> > master's using it are active. The device_link feature >> > helps to track such functional dependencies, so that the >> > iommu gets powered when the master device enables itself >> > using pm_runtime. So by adapting the smmu driver for >> > runtime pm, above said dependency can be addressed. >> > >> > This patch adds the pm runtime/sleep callbacks to the >> > driver and also the functions to parse the smmu clocks >> > from DT and enable them in resume/suspend. >> > >> > Also, while we enable the runtime pm add a pm sleep suspend >> > callback that pushes devices to low power state by turning >> > the clocks off in a system sleep. >> > Also add corresponding clock enable path in resume callback. >> > >> > Signed-off-by: Sricharan R <sricharan@codeaurora.org> >> > Signed-off-by: Archit Taneja <architt@codeaurora.org> >> > [vivek: rework for clock and pm ops] >> > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> >> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> >> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> >> > --- >> > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- >> > 1 file changed, 74 insertions(+), 3 deletions(-) >> > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c >> >> [...] >> >> > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) >> > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) >> > { >> > struct arm_smmu_device *smmu = dev_get_drvdata(dev); >> > + int ret; >> > + >> > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); >> > + if (ret) >> > + return ret; >> > >> > arm_smmu_device_reset(smmu); >> > + >> > return 0; >> > } >> > >> > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); >> > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) >> > +{ >> > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); >> > + >> > + clk_bulk_disable(smmu->num_clks, smmu->clks); >> > + >> > + return 0; >> > +} >> > + >> > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) >> > +{ >> > + if (pm_runtime_suspended(dev)) >> > + return 0; >> >> Looks like you should be able use pm_runtime_force_resume(), instead >> of using this local trick. Unless I am missing something, of course. >> >> In other words, just assign the system sleep callbacks for resume, to >> pm_runtime_force_resume(). And vice verse for the system suspend >> callbacks, pm_runtime_force_suspend(), of course. > > Thanks for the review. I will change this as suggested. > >> >> > + >> > + return arm_smmu_runtime_resume(dev); >> > +} >> > + >> > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) >> > +{ >> > + if (pm_runtime_suspended(dev)) >> > + return 0; >> > + >> > + return arm_smmu_runtime_suspend(dev); >> > +} >> > + >> > +static const struct dev_pm_ops arm_smmu_pm_ops = { >> > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) >> >> I am wondering if using the ->suspend|resume() callback is really >> "late/early" enough in the device suspend phase? >> >> Others is using the noirq phase and some is even using the syscore >> ops. Of course it depends on the behavior of the consumers of iommu >> device, and I guess not everyone is using device links, which for sure >> improves things in this regards as well. > > Well yes, as you said the device links should be able to take care of > maintaining the correct suspend/resume order of smmu and its clients, > or am I missing your point here? > Let me know and I will be happy to incorporate any suggestions. > Thanks If it works fine, then you may keep it as is. Just wanted to point out that if any consumers relies on the iommu to operational to say until the suspend-late phase, then this doesn't play. Then you need to move your callbacks to the corresponding same phase. [...] Kind regards Uffe
On Mon, Oct 1, 2018 at 3:09 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: > > On 1 October 2018 at 07:49, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > HI Ulf, > > > > On Fri, Sep 28, 2018 at 5:30 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: > >> > >> On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > >> > From: Sricharan R <sricharan@codeaurora.org> > >> > > >> > The smmu needs to be functional only when the respective > >> > master's using it are active. The device_link feature > >> > helps to track such functional dependencies, so that the > >> > iommu gets powered when the master device enables itself > >> > using pm_runtime. So by adapting the smmu driver for > >> > runtime pm, above said dependency can be addressed. > >> > > >> > This patch adds the pm runtime/sleep callbacks to the > >> > driver and also the functions to parse the smmu clocks > >> > from DT and enable them in resume/suspend. > >> > > >> > Also, while we enable the runtime pm add a pm sleep suspend > >> > callback that pushes devices to low power state by turning > >> > the clocks off in a system sleep. > >> > Also add corresponding clock enable path in resume callback. > >> > > >> > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > >> > Signed-off-by: Archit Taneja <architt@codeaurora.org> > >> > [vivek: rework for clock and pm ops] > >> > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > >> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > >> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > >> > --- > >> > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > >> > 1 file changed, 74 insertions(+), 3 deletions(-) > >> > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > >> > >> [...] > >> > >> > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > >> > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > >> > { > >> > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > >> > + int ret; > >> > + > >> > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > >> > + if (ret) > >> > + return ret; > >> > > >> > arm_smmu_device_reset(smmu); > >> > + > >> > return 0; > >> > } > >> > > >> > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > >> > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > >> > +{ > >> > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > >> > + > >> > + clk_bulk_disable(smmu->num_clks, smmu->clks); > >> > + > >> > + return 0; > >> > +} > >> > + > >> > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > >> > +{ > >> > + if (pm_runtime_suspended(dev)) > >> > + return 0; > >> > >> Looks like you should be able use pm_runtime_force_resume(), instead > >> of using this local trick. Unless I am missing something, of course. > >> > >> In other words, just assign the system sleep callbacks for resume, to > >> pm_runtime_force_resume(). And vice verse for the system suspend > >> callbacks, pm_runtime_force_suspend(), of course. > > > > Thanks for the review. I will change this as suggested. > > > >> > >> > + > >> > + return arm_smmu_runtime_resume(dev); > >> > +} > >> > + > >> > +static int __maybe_unused arm_smmu_pm_suspend(struct device *dev) > >> > +{ > >> > + if (pm_runtime_suspended(dev)) > >> > + return 0; > >> > + > >> > + return arm_smmu_runtime_suspend(dev); > >> > +} > >> > + > >> > +static const struct dev_pm_ops arm_smmu_pm_ops = { > >> > + SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume) > >> > >> I am wondering if using the ->suspend|resume() callback is really > >> "late/early" enough in the device suspend phase? > >> > >> Others is using the noirq phase and some is even using the syscore > >> ops. Of course it depends on the behavior of the consumers of iommu > >> device, and I guess not everyone is using device links, which for sure > >> improves things in this regards as well. > > > > Well yes, as you said the device links should be able to take care of > > maintaining the correct suspend/resume order of smmu and its clients, > > or am I missing your point here? > > Let me know and I will be happy to incorporate any suggestions. > > Thanks > > If it works fine, then you may keep it as is. > > Just wanted to point out that if any consumers relies on the iommu to > operational to say until the suspend-late phase, then this doesn't > play. Then you need to move your callbacks to the corresponding same > phase. Although I have no means to test the suspend-late phase, tests with graphics and display on db820 haven't shown any anomaly. [snip] Best regards Vivek
On Mon, Oct 1, 2018 at 11:19 AM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > HI Ulf, > > On Fri, Sep 28, 2018 at 5:30 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: > > > > On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > > From: Sricharan R <sricharan@codeaurora.org> > > > > > > The smmu needs to be functional only when the respective > > > master's using it are active. The device_link feature > > > helps to track such functional dependencies, so that the > > > iommu gets powered when the master device enables itself > > > using pm_runtime. So by adapting the smmu driver for > > > runtime pm, above said dependency can be addressed. > > > > > > This patch adds the pm runtime/sleep callbacks to the > > > driver and also the functions to parse the smmu clocks > > > from DT and enable them in resume/suspend. > > > > > > Also, while we enable the runtime pm add a pm sleep suspend > > > callback that pushes devices to low power state by turning > > > the clocks off in a system sleep. > > > Also add corresponding clock enable path in resume callback. > > > > > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > > Signed-off-by: Archit Taneja <architt@codeaurora.org> > > > [vivek: rework for clock and pm ops] > > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > > --- > > > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- > > > 1 file changed, 74 insertions(+), 3 deletions(-) > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > > > [...] > > > > > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) > > > { > > > struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > > + int ret; > > > + > > > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); > > > + if (ret) > > > + return ret; > > > > > > arm_smmu_device_reset(smmu); > > > + > > > return 0; > > > } > > > > > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); > > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) > > > +{ > > > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); > > > + > > > + clk_bulk_disable(smmu->num_clks, smmu->clks); > > > + > > > + return 0; > > > +} > > > + > > > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) > > > +{ > > > + if (pm_runtime_suspended(dev)) > > > + return 0; > > > > Looks like you should be able use pm_runtime_force_resume(), instead > > of using this local trick. Unless I am missing something, of course. > > > > In other words, just assign the system sleep callbacks for resume, to > > pm_runtime_force_resume(). And vice verse for the system suspend > > callbacks, pm_runtime_force_suspend(), of course. > > Thanks for the review. I will change this as suggested. Coming back at this - actually Rafael suggested _not_ to use pm_runtime_force_suspend/resume() when Marek had suggested the same [1]. He also mentioned few caveats/limitations of using these APIs for system sleep ops. Let me know your opinion. Thanks. [1] https://lkml.org/lkml/2018/7/11/978 [2] https://lkml.org/lkml/2018/7/23/334 Best regards Vivek
On 1 October 2018 at 12:32, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > On Mon, Oct 1, 2018 at 11:19 AM Vivek Gautam > <vivek.gautam@codeaurora.org> wrote: >> >> HI Ulf, >> >> On Fri, Sep 28, 2018 at 5:30 PM Ulf Hansson <ulf.hansson@linaro.org> wrote: >> > >> > On 30 August 2018 at 16:45, Vivek Gautam <vivek.gautam@codeaurora.org> wrote: >> > > From: Sricharan R <sricharan@codeaurora.org> >> > > >> > > The smmu needs to be functional only when the respective >> > > master's using it are active. The device_link feature >> > > helps to track such functional dependencies, so that the >> > > iommu gets powered when the master device enables itself >> > > using pm_runtime. So by adapting the smmu driver for >> > > runtime pm, above said dependency can be addressed. >> > > >> > > This patch adds the pm runtime/sleep callbacks to the >> > > driver and also the functions to parse the smmu clocks >> > > from DT and enable them in resume/suspend. >> > > >> > > Also, while we enable the runtime pm add a pm sleep suspend >> > > callback that pushes devices to low power state by turning >> > > the clocks off in a system sleep. >> > > Also add corresponding clock enable path in resume callback. >> > > >> > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> >> > > Signed-off-by: Archit Taneja <architt@codeaurora.org> >> > > [vivek: rework for clock and pm ops] >> > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> >> > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> >> > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> >> > > --- >> > > drivers/iommu/arm-smmu.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++-- >> > > 1 file changed, 74 insertions(+), 3 deletions(-) >> > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c >> > >> > [...] >> > >> > > -static int __maybe_unused arm_smmu_pm_resume(struct device *dev) >> > > +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev) >> > > { >> > > struct arm_smmu_device *smmu = dev_get_drvdata(dev); >> > > + int ret; >> > > + >> > > + ret = clk_bulk_enable(smmu->num_clks, smmu->clks); >> > > + if (ret) >> > > + return ret; >> > > >> > > arm_smmu_device_reset(smmu); >> > > + >> > > return 0; >> > > } >> > > >> > > -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume); >> > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev) >> > > +{ >> > > + struct arm_smmu_device *smmu = dev_get_drvdata(dev); >> > > + >> > > + clk_bulk_disable(smmu->num_clks, smmu->clks); >> > > + >> > > + return 0; >> > > +} >> > > + >> > > +static int __maybe_unused arm_smmu_pm_resume(struct device *dev) >> > > +{ >> > > + if (pm_runtime_suspended(dev)) >> > > + return 0; >> > >> > Looks like you should be able use pm_runtime_force_resume(), instead >> > of using this local trick. Unless I am missing something, of course. >> > >> > In other words, just assign the system sleep callbacks for resume, to >> > pm_runtime_force_resume(). And vice verse for the system suspend >> > callbacks, pm_runtime_force_suspend(), of course. >> >> Thanks for the review. I will change this as suggested. > > Coming back at this - actually Rafael suggested _not_ to use > pm_runtime_force_suspend/resume() when Marek had suggested > the same [1]. I see. > He also mentioned few caveats/limitations of using these APIs > for system sleep ops. > Let me know your opinion. Thanks. > > [1] https://lkml.org/lkml/2018/7/11/978 > [2] https://lkml.org/lkml/2018/7/23/334 Me and Rafael have been discussing these topics historically as well. I don't want to get that discussion started again here. If your device is attached to the PCI bus or the ACPI PM domain (and also gets runtime PM enabled), then I suggest you to stick to the currently suggested approach. Otherwise it should be perfectly fine to switch to the *force helpers. Kind regards Uffe
Hi Vivek, On Thu, Aug 30, 2018 at 08:15:38PM +0530, Vivek Gautam wrote: > From: Sricharan R <sricharan@codeaurora.org> > > The smmu device probe/remove and add/remove master device callbacks > gets called when the smmu is not linked to its master, that is without > the context of the master device. So calling runtime apis in those places > separately. > Global locks are also initialized before enabling runtime pm as the > runtime_resume() calls device_reset() which does tlb_sync_global() > that ultimately requires locks to be initialized. > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > [vivek: Cleanup pm runtime calls] > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > --- > drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 81 insertions(+), 8 deletions(-) This doesn't apply on my tree[1], possibly because I've got Robin's non-strict invalidation queued there. However, that got me thinking -- how does this work in conjunction with the timer-based TLB invalidation? Do we need to rpm_{get,put} around flush_iotlb_all()? If so, do we still need the calls in map/unmap when non-strict mode is in use? Will [1] https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=for-joerg/arm-smmu/updates
Hi Will, On Mon, Oct 1, 2018 at 6:29 PM Will Deacon <will.deacon@arm.com> wrote: > > Hi Vivek, > > On Thu, Aug 30, 2018 at 08:15:38PM +0530, Vivek Gautam wrote: > > From: Sricharan R <sricharan@codeaurora.org> > > > > The smmu device probe/remove and add/remove master device callbacks > > gets called when the smmu is not linked to its master, that is without > > the context of the master device. So calling runtime apis in those places > > separately. > > Global locks are also initialized before enabling runtime pm as the > > runtime_resume() calls device_reset() which does tlb_sync_global() > > that ultimately requires locks to be initialized. > > > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > [vivek: Cleanup pm runtime calls] > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > --- > > drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > > 1 file changed, 81 insertions(+), 8 deletions(-) > > This doesn't apply on my tree[1], possibly because I've got Robin's non-strict > invalidation queued there. However, that got me thinking -- how does this > work in conjunction with the timer-based TLB invalidation? Do we need to > rpm_{get,put} around flush_iotlb_all()? If so, do we still need the calls > in map/unmap when non-strict mode is in use? I haven't tested things with flush queues, but from what it looks like both .flush_iotlb_all, and .iotlb_sync callbacks need rpm_get/put(). I will respin the patches. Thanks Vivek > > Will > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=for-joerg/arm-smmu/updates
On Tue, Oct 2, 2018 at 9:44 AM Vivek Gautam <vivek.gautam@codeaurora.org> wrote: > > Hi Will, > > On Mon, Oct 1, 2018 at 6:29 PM Will Deacon <will.deacon@arm.com> wrote: > > > > Hi Vivek, > > > > On Thu, Aug 30, 2018 at 08:15:38PM +0530, Vivek Gautam wrote: > > > From: Sricharan R <sricharan@codeaurora.org> > > > > > > The smmu device probe/remove and add/remove master device callbacks > > > gets called when the smmu is not linked to its master, that is without > > > the context of the master device. So calling runtime apis in those places > > > separately. > > > Global locks are also initialized before enabling runtime pm as the > > > runtime_resume() calls device_reset() which does tlb_sync_global() > > > that ultimately requires locks to be initialized. > > > > > > Signed-off-by: Sricharan R <sricharan@codeaurora.org> > > > [vivek: Cleanup pm runtime calls] > > > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> > > > Reviewed-by: Tomasz Figa <tfiga@chromium.org> > > > Tested-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> > > > --- > > > drivers/iommu/arm-smmu.c | 89 +++++++++++++++++++++++++++++++++++++++++++----- > > > 1 file changed, 81 insertions(+), 8 deletions(-) > > > > This doesn't apply on my tree[1], possibly because I've got Robin's non-strict > > invalidation queued there. However, that got me thinking -- how does this > > work in conjunction with the timer-based TLB invalidation? Do we need to > > rpm_{get,put} around flush_iotlb_all()? If so, do we still need the calls > > in map/unmap when non-strict mode is in use? For map/unmap(), i think there would be no harm in having additional power.usage_count even for the non-strict mode. So, I will just add rpm{get,put} in arm_smmu_flush_iotlb_all(), and arm_smmu_iotlb_sync(). Regards Vivek > > I haven't tested things with flush queues, but from what it looks like > both .flush_iotlb_all, and .iotlb_sync callbacks need rpm_get/put(). > I will respin the patches. > > Thanks > Vivek > > > > Will > > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=for-joerg/arm-smmu/updates > > > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation
This series provides the support for turning on the arm-smmu's clocks/power domains using runtime pm. This is done using device links between smmu and client devices. The device link framework keeps the two devices in correct order for power-cycling across runtime PM or across system-wide PM. With addition of a new device link flag DL_FLAG_AUTOREMOVE_SUPPLIER [7], the device links created between arm-smmu and its clients will be automatically purged when arm-smmu driver unbinds from its device. As not all implementations support clock/power gating, we are checking for a valid 'smmu->dev's pm_domain' to conditionally enable the runtime power management for such smmu implementations that can support it. Otherwise, the clocks are turned to be always on in .probe until .remove. With conditional runtime pm now, we avoid touching dev->power.lock in fastpaths for smmu implementations that don't need to do anything useful with pm_runtime. This lets us to use the much-argued pm_runtime_get_sync/put_sync() calls in map/unmap callbacks so that the clients do not have to worry about handling any of the arm-smmu's power. This series also adds support for Qcom's arm-smmu-v2 variant that has different clocks and power requirements. Previous version of this patch series is @ [1]. Build tested the series based on 4.19-rc1. [v16] * Addressed review comments from Rob about soc specific compatibles. * Removed pm_runtime_get/put calls from arm_smmu_device_probe(), as doing a runtime_get() eventually calls arm_smmu_device_reset(). arm_smmu_device_reset() should be called only after arm_smmu_device_cfg_probe(). Enabling the clocks by default in the probe, and using pm_runtime_set_active() as suggested by Tomasz to disable the clocks when pm_runtime is enabled. [v15] * Added a list of valid values of '<soc>' in "qcom,<soc>-smmu-v2" compatible string as pointed out by Robin, and Rob in the thread [8]: * Added Srini's Tested-by. * Separated out the dt-bindings change from driver change into a new patch as suggested by new checkpatch warning. Rob, I took the liberty of removing your Reviewed-by (for your comment on '<soc>') for the new dt-bindings patch 4/5. Please feel free to review it again. Thanks! [v14] * Moved arm_smmu_device_reset() from arm_smmu_pm_resume() to arm_smmu_runtime_resume() so that the pm_resume callback calls only runtime_resume to resume the device. This should take care of restoring the state of smmu in systems in which smmu lose register state on power-domain collapse. [v13] Addressing Rafael's comments: * Added .suspend pm callback to disable the clocks in system wide suspend. * Added corresponding clock enable in .resume pm callback. * Explicitly enabling/disabling the clocks now when runtime PM is disabled. * device_link_add() doesn't depend on pm_runtime_enabled() as we can use device links across system suspend/resume too. Addressing Robin's comments: * Making device_link_add failures as non-fatal. * Removed IOMMU_OF_DECLARE() declaration as we don't need this after Rob's patch that removed all of these declarations. [v12] * Use new device link's flag introduced in [7] - DL_FLAG_AUTOREMOVE_SUPPLIER. With this devices links are automatically purged when arm-smmu driver unbinds. * Using pm_runtime_force_suspend() instead of pm_runtime_disable() to avoid following warning from arm_smmu_device_remove() [295711.537507] ------------[ cut here ]------------ [295711.544226] Unpreparing enabled smmu_mdp_ahb_clk [295711.549099] WARNING: CPU: 0 PID: 1 at ../drivers/clk/clk.c:697 clk_core_unprepare+0xd8/0xe0 ... [295711.674073] Call trace: [295711.679454] clk_core_unprepare+0xd8/0xe0 [295711.682059] clk_unprepare+0x28/0x40 [295711.685964] clk_bulk_unprepare+0x28/0x40 [295711.689701] arm_smmu_device_remove+0x88/0xd8 [295711.693692] arm_smmu_device_shutdown+0xc/0x18 [295711.698120] platform_drv_shutdown+0x20/0x30 [v11] * Some more cleanups for device link. We don't need an explicit delete for device link from the driver, but just set the flag DL_FLAG_AUTOREMOVE. device_link_add() API description says - "If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically when the consumer device driver unbinds." * Addressed the comments for 'smmu' in arm_smmu_map/unmap(). * Dropped the patch [6] that introduced device_link_del_dev() API. [v10] * Introduce device_link_del_dev() API to delete the link between given consumer and supplier devices. The users of device link do not need to store link pointer to delete the link later. They can straightaway use this API by passing consumer and supplier devices. * Made corresponding changes to arm-smmu driver patch handling the device links. * Dropped the patch [5] that was adding device_link_find() API to device core layer. device_link_del_dev() serves the purpose to directly delete the link between two given devices. [v9] * Removed 'rpm_supported' flag, instead checking on pm_domain to enable runtime pm. * Creating device link only when the runtime pm is enabled, as we don't need a device link besides managing the power dependency between supplier and consumer devices. * Introducing a patch to add device_link_find() API that finds and existing link between supplier and consumer devices. Also, made necessary change to device_link_add() to use this API. * arm_smmu_remove_device() now uses this device_link_find() to find the device link between smmu device and the master device, and then delete this link. * Dropped the destroy_domain_context() fix [4] as it was rather, introducing catastrophically bad problem by destroying 'good dev's domain context. * Added 'Reviwed-by' tag for Tomasz's review. [v8] * Major change - - Added a flag 'rpm_supported' which each platform that supports runtime pm, can enable, and we enable runtime_pm over arm-smmu only when this flag is set. - Adding the conditional pm_runtime_get/put() calls to .map, .unmap and .attach_dev ops. - Dropped the patch [2] that exported pm_runtim_get/put_suupliers(), and also dropped the user driver patch [3] for these APIs. * Clock code further cleanup - doing only clk_bulk_enable() and clk_bulk_disable() in runtime pm callbacks. We shouldn't be taking a slow path (clk_prepare/unprepare()) from these runtime pm callbacks. Thereby, moved clk_bulk_prepare() to arm_smmu_device_probe(), and clk_bulk_unprepare() to arm_smmu_device_remove(). - clk data filling to a common method arm_smmu_fill_clk_data() that fills the clock ids and number of clocks. * Addressed other nits and comments - device_link_add() error path fixed. - Fix for checking negative error value from pm_runtime_get_sync(). - Documentation redo. * Added another patch fixing the error path in arm_smmu_attach_dev() to destroy allocated domain context. ** Change logs for previous versions is available in previous series [9]. [1] https://patchwork.kernel.org/cover/10576921/ [2] https://patchwork.kernel.org/patch/10204945/ [3] https://patchwork.kernel.org/patch/10204925/ [4] https://patchwork.kernel.org/patch/10254105/ [5] https://patchwork.kernel.org/patch/10277975/ [6] https://patchwork.kernel.org/patch/10281613/ [7] https://patchwork.kernel.org/patch/10491481/ [8] https://lore.kernel.org/patchwork/patch/974116/ [9] https://lkml.org/lkml/2018/7/8/124 Sricharan R (3): iommu/arm-smmu: Add pm_runtime/sleep ops iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device iommu/arm-smmu: Add the device_link between masters and smmu Vivek Gautam (2): dt-bindings: arm-smmu: Add bindings for qcom,smmu-v2 iommu/arm-smmu: Add support for qcom,smmu-v2 variant .../devicetree/bindings/iommu/arm,smmu.txt | 39 +++++ drivers/iommu/arm-smmu.c | 180 +++++++++++++++++++-- 2 files changed, 209 insertions(+), 10 deletions(-)