| Message ID | b57c5d00d9ca21c0d00a269010aea7860832e70e.1429097578.git.michal.simek@xilinx.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Albert ARIBAUD |
| Headers | show |
On Wed, Apr 15, 2015 at 12:33:00PM +0100, Michal Simek wrote: > From: Siva Durga Prasad Paladugu <siva.durga.paladugu@xilinx.com> > > Always disable dcache after the flush operation > The following sequence is advisable while disabling d-cache: > 1. disable_dcache() - flushes and disables d-cache > 2. invalidate_dcache_all() - invalid any entry that came to the cache > in the short period after the cache was flushed but before the > cache got disabled For reasons I have described previously (see [1,2,3]), this is unsafe. The first cache flush may achieve nothing. If you need data out at the PoC before disabling the cache, then you should first use maintenance by VA to push that data out. Thanks, Mark. [1] http://lists.denx.de/pipermail/u-boot/2015-February/204403.html [2] http://lists.denx.de/pipermail/u-boot/2015-February/204407.html [3] http://lists.denx.de/pipermail/u-boot/2015-February/204702.html > > Signed-off-by: Siva Durga Prasad Paladugu <sivadur@xilinx.com> > Signed-off-by: Michal Simek <michal.simek@xilinx.com> > --- > > arch/arm/cpu/armv8/cache_v8.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/arm/cpu/armv8/cache_v8.c b/arch/arm/cpu/armv8/cache_v8.c > index c5ec5297cd39..2a0492fbef52 100644 > --- a/arch/arm/cpu/armv8/cache_v8.c > +++ b/arch/arm/cpu/armv8/cache_v8.c > @@ -128,10 +128,10 @@ void dcache_disable(void) > if (!(sctlr & CR_C)) > return; > > - set_sctlr(sctlr & ~(CR_C|CR_M)); > - > flush_dcache_all(); > __asm_invalidate_tlb_all(); > + > + set_sctlr(sctlr & ~(CR_C|CR_M)); > } > > int dcache_status(void) > -- > 2.3.5 > > _______________________________________________ > U-Boot mailing list > U-Boot@lists.denx.de > http://lists.denx.de/mailman/listinfo/u-boot >
Hi Mark. > -----Original Message----- > From: Mark Rutland [mailto:mark.rutland@arm.com] > Sent: Wednesday, April 15, 2015 6:41 PM > To: Michal Simek > Cc: u-boot@lists.denx.de; Tom Rini; Siva Durga Prasad Paladugu; Varun Sethi; > Arnab Basu; York Sun > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache after flush > > On Wed, Apr 15, 2015 at 12:33:00PM +0100, Michal Simek wrote: > > From: Siva Durga Prasad Paladugu <siva.durga.paladugu@xilinx.com> > > > > Always disable dcache after the flush operation The following sequence > > is advisable while disabling d-cache: > > 1. disable_dcache() - flushes and disables d-cache 2. > > invalidate_dcache_all() - invalid any entry that came to the cache > > in the short period after the cache was flushed but before the > > cache got disabled > > For reasons I have described previously (see [1,2,3]), this is unsafe. > The first cache flush may achieve nothing. > > If you need data out at the PoC before disabling the cache, then you should > first use maintenance by VA to push that data out. > > Thanks, > Mark. > > [1] http://lists.denx.de/pipermail/u-boot/2015-February/204403.html > [2] http://lists.denx.de/pipermail/u-boot/2015-February/204407.html > [3] http://lists.denx.de/pipermail/u-boot/2015-February/204702.html > > > > > Signed-off-by: Siva Durga Prasad Paladugu <sivadur@xilinx.com> > > Signed-off-by: Michal Simek <michal.simek@xilinx.com> > > --- > > > > arch/arm/cpu/armv8/cache_v8.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/arch/arm/cpu/armv8/cache_v8.c > > b/arch/arm/cpu/armv8/cache_v8.c index c5ec5297cd39..2a0492fbef52 > > 100644 > > --- a/arch/arm/cpu/armv8/cache_v8.c > > +++ b/arch/arm/cpu/armv8/cache_v8.c > > @@ -128,10 +128,10 @@ void dcache_disable(void) > > if (!(sctlr & CR_C)) > > return; > > > > - set_sctlr(sctlr & ~(CR_C|CR_M)); > > - > > flush_dcache_all(); > > __asm_invalidate_tlb_all(); > > + > > + set_sctlr(sctlr & ~(CR_C|CR_M)); I got your point. But here in this scenario also there is an issue with disable first and then flush_dcache_all(). This is because when we disable the cache and invoke the c routine flush_dcache_all() then the return address of this is stored in a stack(in memory as dcache is disabled). Now in the flush_dcache_all we are invoking the actual asm call to flush dcache which may wipeout the stored return value in stack with cahe contents(main memory). Hence the return from the flush_dcahe_all will fail. To confirm this I modified the dcache_disable in the below way and it worked fine. 1. Disable the dcache. 2. Now I called the __asm_flush_dcache_all(); and then flush_l3_cache(); instead of calling the flush_dcache_all(). Regards, Siva > > } > > > > int dcache_status(void) > > -- > > 2.3.5 > > > > _______________________________________________ > > U-Boot mailing list > > U-Boot@lists.denx.de > > http://lists.denx.de/mailman/listinfo/u-boot > >
On Thu, Apr 16, 2015 at 06:17:59AM +0100, Siva Durga Prasad Paladugu wrote: > Hi Mark. > > > -----Original Message----- > > From: Mark Rutland [mailto:mark.rutland@arm.com] > > Sent: Wednesday, April 15, 2015 6:41 PM > > To: Michal Simek > > Cc: u-boot@lists.denx.de; Tom Rini; Siva Durga Prasad Paladugu; Varun Sethi; > > Arnab Basu; York Sun > > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache after flush > > > > On Wed, Apr 15, 2015 at 12:33:00PM +0100, Michal Simek wrote: > > > From: Siva Durga Prasad Paladugu <siva.durga.paladugu@xilinx.com> > > > > > > Always disable dcache after the flush operation The following sequence > > > is advisable while disabling d-cache: > > > 1. disable_dcache() - flushes and disables d-cache 2. > > > invalidate_dcache_all() - invalid any entry that came to the cache > > > in the short period after the cache was flushed but before the > > > cache got disabled > > > > For reasons I have described previously (see [1,2,3]), this is unsafe. > > The first cache flush may achieve nothing. > > > > If you need data out at the PoC before disabling the cache, then you should > > first use maintenance by VA to push that data out. > > > > Thanks, > > Mark. > > > > [1] http://lists.denx.de/pipermail/u-boot/2015-February/204403.html > > [2] http://lists.denx.de/pipermail/u-boot/2015-February/204407.html > > [3] http://lists.denx.de/pipermail/u-boot/2015-February/204702.html > > > > > > > > Signed-off-by: Siva Durga Prasad Paladugu <sivadur@xilinx.com> > > > Signed-off-by: Michal Simek <michal.simek@xilinx.com> > > > --- > > > > > > arch/arm/cpu/armv8/cache_v8.c | 4 ++-- > > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > > > diff --git a/arch/arm/cpu/armv8/cache_v8.c > > > b/arch/arm/cpu/armv8/cache_v8.c index c5ec5297cd39..2a0492fbef52 > > > 100644 > > > --- a/arch/arm/cpu/armv8/cache_v8.c > > > +++ b/arch/arm/cpu/armv8/cache_v8.c > > > @@ -128,10 +128,10 @@ void dcache_disable(void) > > > if (!(sctlr & CR_C)) > > > return; > > > > > > - set_sctlr(sctlr & ~(CR_C|CR_M)); > > > - > > > flush_dcache_all(); > > > __asm_invalidate_tlb_all(); > > > + > > > + set_sctlr(sctlr & ~(CR_C|CR_M)); > > I got your point. But here in this scenario also there is an issue > with disable first and then flush_dcache_all(). This is because when > we disable the cache and invoke the c routine flush_dcache_all() then > the return address of this is stored in a stack(in memory as dcache is > disabled). Which is why this sequence cannot be written in C, and needs to be performed in assembly, without any memory accesses between the write to the SCTLR and the cache flush. > Now in the flush_dcache_all we are invoking the actual asm call to > flush dcache which may wipeout the stored return value in stack with > cahe contents(main memory). Hence the return from the flush_dcahe_all > will fail. > > To confirm this I modified the dcache_disable in the below way and it worked fine. > 1. Disable the dcache. > 2. Now I called the __asm_flush_dcache_all(); and then flush_l3_cache(); instead of calling the flush_dcache_all(). That also is unsafe; implicit (e.g. stack) accesses at any point after SCTLR.C is cleared and before flush_l3_cache() has completed may see stale data, or get overwritten by stale data. Set/Way ops only flush the CPU-local caches, so you only guarantee that these are clean (and potentially dirty cache lines for the stack could be sat in L3 and written back at any time). So your flush_l3_cache() function might not work. Per ARMv8 the L3 _must_ respect maintenance by VA, so after disabling the MMU you can flush the memory region corresponding to your stack (and any other data you need) by VA to the PoC before executing flush_l3_cache(), in addition to the Set/Way ops used to empty the CPU-local caches. Thanks, Mark.
Hi Mark, > -----Original Message----- > From: Mark Rutland [mailto:mark.rutland@arm.com] > Sent: Thursday, April 16, 2015 3:29 PM > To: Siva Durga Prasad Paladugu > Cc: Michal Simek; u-boot@lists.denx.de; Tom Rini; Varun Sethi; Arnab Basu; > York Sun > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache after flush > > On Thu, Apr 16, 2015 at 06:17:59AM +0100, Siva Durga Prasad Paladugu > wrote: > > Hi Mark. > > > > > -----Original Message----- > > > From: Mark Rutland [mailto:mark.rutland@arm.com] > > > Sent: Wednesday, April 15, 2015 6:41 PM > > > To: Michal Simek > > > Cc: u-boot@lists.denx.de; Tom Rini; Siva Durga Prasad Paladugu; > > > Varun Sethi; Arnab Basu; York Sun > > > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache > > > after flush > > > > > > On Wed, Apr 15, 2015 at 12:33:00PM +0100, Michal Simek wrote: > > > > From: Siva Durga Prasad Paladugu <siva.durga.paladugu@xilinx.com> > > > > > > > > Always disable dcache after the flush operation The following > > > > sequence is advisable while disabling d-cache: > > > > 1. disable_dcache() - flushes and disables d-cache 2. > > > > invalidate_dcache_all() - invalid any entry that came to the cache > > > > in the short period after the cache was flushed but before the > > > > cache got disabled > > > > > > For reasons I have described previously (see [1,2,3]), this is unsafe. > > > The first cache flush may achieve nothing. > > > > > > If you need data out at the PoC before disabling the cache, then you > > > should first use maintenance by VA to push that data out. > > > > > > Thanks, > > > Mark. > > > > > > [1] http://lists.denx.de/pipermail/u-boot/2015-February/204403.html > > > [2] http://lists.denx.de/pipermail/u-boot/2015-February/204407.html > > > [3] http://lists.denx.de/pipermail/u-boot/2015-February/204702.html > > > > > > > > > > > Signed-off-by: Siva Durga Prasad Paladugu <sivadur@xilinx.com> > > > > Signed-off-by: Michal Simek <michal.simek@xilinx.com> > > > > --- > > > > > > > > arch/arm/cpu/armv8/cache_v8.c | 4 ++-- > > > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > > > > > diff --git a/arch/arm/cpu/armv8/cache_v8.c > > > > b/arch/arm/cpu/armv8/cache_v8.c index c5ec5297cd39..2a0492fbef52 > > > > 100644 > > > > --- a/arch/arm/cpu/armv8/cache_v8.c > > > > +++ b/arch/arm/cpu/armv8/cache_v8.c > > > > @@ -128,10 +128,10 @@ void dcache_disable(void) > > > > if (!(sctlr & CR_C)) > > > > return; > > > > > > > > - set_sctlr(sctlr & ~(CR_C|CR_M)); > > > > - > > > > flush_dcache_all(); > > > > __asm_invalidate_tlb_all(); > > > > + > > > > + set_sctlr(sctlr & ~(CR_C|CR_M)); > > > > I got your point. But here in this scenario also there is an issue > > with disable first and then flush_dcache_all(). This is because when > > we disable the cache and invoke the c routine flush_dcache_all() then > > the return address of this is stored in a stack(in memory as dcache is > > disabled). > > Which is why this sequence cannot be written in C, and needs to be > performed in assembly, without any memory accesses between the write to > the SCTLR and the cache flush. > > > Now in the flush_dcache_all we are invoking the actual asm call to > > flush dcache which may wipeout the stored return value in stack with > > cahe contents(main memory). Hence the return from the flush_dcahe_all > > will fail. > > > > To confirm this I modified the dcache_disable in the below way and it > worked fine. > > 1. Disable the dcache. > > 2. Now I called the __asm_flush_dcache_all(); and then flush_l3_cache(); > instead of calling the flush_dcache_all(). > > That also is unsafe; implicit (e.g. stack) accesses at any point after SCTLR.C is > cleared and before flush_l3_cache() has completed may see stale data, or > get overwritten by stale data. > > Set/Way ops only flush the CPU-local caches, so you only guarantee that > these are clean (and potentially dirty cache lines for the stack could be sat in > L3 and written back at any time). So your flush_l3_cache() function might not > work. > > Per ARMv8 the L3 _must_ respect maintenance by VA, so after disabling the > MMU you can flush the memory region corresponding to your stack (and any > other data you need) by VA to the PoC before executing flush_l3_cache(), in > addition to the Set/Way ops used to empty the CPU-local caches. Thanks for explanation. So in that case, the flushing of the required stack or any other data which needs to be flushed should be part of board specific. Am I correct? If yes, then this disable_dcache() should contain a asm call to a routine() (which might be board specific) after disabling the cache to flush the required data and then flush_dcache_all() followed by flush L3 cache.. The current implementation just didn't work. It worked for me with the changes i mentioned in my last mail as I don't have L3 cache in my case. Regards, Siva > > Thanks, > Mark.
> > > Now in the flush_dcache_all we are invoking the actual asm call to > > > flush dcache which may wipeout the stored return value in stack with > > > cahe contents(main memory). Hence the return from the flush_dcahe_all > > > will fail. > > > > > > To confirm this I modified the dcache_disable in the below way and it > > worked fine. > > > 1. Disable the dcache. > > > 2. Now I called the __asm_flush_dcache_all(); and then flush_l3_cache(); > > instead of calling the flush_dcache_all(). > > > > That also is unsafe; implicit (e.g. stack) accesses at any point after SCTLR.C is > > cleared and before flush_l3_cache() has completed may see stale data, or > > get overwritten by stale data. > > > > Set/Way ops only flush the CPU-local caches, so you only guarantee that > > these are clean (and potentially dirty cache lines for the stack could be sat in > > L3 and written back at any time). So your flush_l3_cache() function might not > > work. > > > > Per ARMv8 the L3 _must_ respect maintenance by VA, so after disabling the > > MMU you can flush the memory region corresponding to your stack (and any > > other data you need) by VA to the PoC before executing flush_l3_cache(), in > > addition to the Set/Way ops used to empty the CPU-local caches. > Thanks for explanation. > So in that case, the flushing of the required stack or any other data > which needs to be flushed should be part of board specific. Am I > correct? It could be done in generic code, assuming we know the bounds of memory which will be used, because maintenance by VA should always work. Do we know which memory U-Boot might use (e.g. does it all fall within some static carveout?), or can it dynamically allocate from anywhere in memory? > If yes, then this disable_dcache() should contain a asm call to a > routine() (which might be board specific) after disabling the cache to > flush the required data and then flush_dcache_all() followed by flush > L3 cache.. You could probably get away with: * Load the memory bounds that we need to flush into some registers, or flush some datastructure containing these to memory. * In assembly: - disable the MMU. - flush the PA range(s) we need to use to be able to use C safely. - flush by Set/Way to empry the CPU-local caches * Implementation-specific L3 flushing for anything else. If we only map a small amount of memory, we could simply flush this by VA (knowing that this will drain the CPU and L3 caches, without any special maintenance). Mark.
Hi Mark, > -----Original Message----- > From: Mark Rutland [mailto:mark.rutland@arm.com] > Sent: Friday, April 17, 2015 3:36 PM > To: Siva Durga Prasad Paladugu > Cc: Michal Simek; u-boot@lists.denx.de; Tom Rini; Varun Sethi; Arnab Basu; > York Sun > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache after flush > > > > > Now in the flush_dcache_all we are invoking the actual asm call to > > > > flush dcache which may wipeout the stored return value in stack > > > > with cahe contents(main memory). Hence the return from the > > > > flush_dcahe_all will fail. > > > > > > > > To confirm this I modified the dcache_disable in the below way and > > > > it > > > worked fine. > > > > 1. Disable the dcache. > > > > 2. Now I called the __asm_flush_dcache_all(); and then > > > > flush_l3_cache(); > > > instead of calling the flush_dcache_all(). > > > > > > That also is unsafe; implicit (e.g. stack) accesses at any point > > > after SCTLR.C is cleared and before flush_l3_cache() has completed > > > may see stale data, or get overwritten by stale data. > > > > > > Set/Way ops only flush the CPU-local caches, so you only guarantee > > > that these are clean (and potentially dirty cache lines for the > > > stack could be sat in > > > L3 and written back at any time). So your flush_l3_cache() function > > > might not work. > > > > > > Per ARMv8 the L3 _must_ respect maintenance by VA, so after > > > disabling the MMU you can flush the memory region corresponding to > > > your stack (and any other data you need) by VA to the PoC before > > > executing flush_l3_cache(), in addition to the Set/Way ops used to empty > the CPU-local caches. > > Thanks for explanation. > > So in that case, the flushing of the required stack or any other data > > which needs to be flushed should be part of board specific. Am I > > correct? > > It could be done in generic code, assuming we know the bounds of memory > which will be used, because maintenance by VA should always work. > > Do we know which memory U-Boot might use (e.g. does it all fall within some > static carveout?), or can it dynamically allocate from anywhere in memory? > > > If yes, then this disable_dcache() should contain a asm call to a > > routine() (which might be board specific) after disabling the cache to > > flush the required data and then flush_dcache_all() followed by flush > > L3 cache.. > > You could probably get away with: > > * Load the memory bounds that we need to flush into some registers, or > flush some datastructure containing these to memory. > * In assembly: > - disable the MMU. > - flush the PA range(s) we need to use to be able to use C safely. > - flush by Set/Way to empry the CPU-local caches > * Implementation-specific L3 flushing for anything else. > > If we only map a small amount of memory, we could simply flush this by VA > (knowing that this will drain the CPU and L3 caches, without any special > maintenance). I just looked at one of the old patch from York in the link below. Can you look at this. http://lists.denx.de/pipermail/u-boot/2015-January/200514.html Regards, DP > > Mark. This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.
> > > Thanks for explanation. > > > So in that case, the flushing of the required stack or any other data > > > which needs to be flushed should be part of board specific. Am I > > > correct? > > > > It could be done in generic code, assuming we know the bounds of memory > > which will be used, because maintenance by VA should always work. > > > > Do we know which memory U-Boot might use (e.g. does it all fall within some > > static carveout?), or can it dynamically allocate from anywhere in memory? > > > > > If yes, then this disable_dcache() should contain a asm call to a > > > routine() (which might be board specific) after disabling the cache to > > > flush the required data and then flush_dcache_all() followed by flush > > > L3 cache.. > > > > You could probably get away with: > > > > * Load the memory bounds that we need to flush into some registers, or > > flush some datastructure containing these to memory. > > * In assembly: > > - disable the MMU. > > - flush the PA range(s) we need to use to be able to use C safely. > > - flush by Set/Way to empry the CPU-local caches > > * Implementation-specific L3 flushing for anything else. > > > > If we only map a small amount of memory, we could simply flush this by VA > > (knowing that this will drain the CPU and L3 caches, without any special > > maintenance). > I just looked at one of the old patch from York in the link below. Can you look at this. > http://lists.denx.de/pipermail/u-boot/2015-January/200514.html I'm not sure what you're expecting me to say w.r.t. that. Is there a particular question you have? Thanks, Mark.
Hi Mark, > -----Original Message----- > From: Mark Rutland [mailto:mark.rutland@arm.com] > Sent: Monday, April 20, 2015 3:48 PM > To: Siva Durga Prasad Paladugu > Cc: Michal Simek; u-boot@lists.denx.de; Tom Rini; Varun Sethi; Arnab Basu; > York Sun > Subject: Re: [U-Boot] [PATCH 1/2] armv8: caches: Disable dcache after flush > > > > > Thanks for explanation. > > > > So in that case, the flushing of the required stack or any other > > > > data which needs to be flushed should be part of board specific. > > > > Am I correct? > > > > > > It could be done in generic code, assuming we know the bounds of > > > memory which will be used, because maintenance by VA should always > work. > > > > > > Do we know which memory U-Boot might use (e.g. does it all fall > > > within some static carveout?), or can it dynamically allocate from > anywhere in memory? > > > > > > > If yes, then this disable_dcache() should contain a asm call to a > > > > routine() (which might be board specific) after disabling the > > > > cache to flush the required data and then flush_dcache_all() > > > > followed by flush > > > > L3 cache.. > > > > > > You could probably get away with: > > > > > > * Load the memory bounds that we need to flush into some registers, or > > > flush some datastructure containing these to memory. > > > * In assembly: > > > - disable the MMU. > > > - flush the PA range(s) we need to use to be able to use C safely. > > > - flush by Set/Way to empry the CPU-local caches > > > * Implementation-specific L3 flushing for anything else. > > > > > > If we only map a small amount of memory, we could simply flush this > > > by VA (knowing that this will drain the CPU and L3 caches, without > > > any special maintenance). > > I just looked at one of the old patch from York in the link below. Can you > look at this. > > http://lists.denx.de/pipermail/u-boot/2015-January/200514.html > > I'm not sure what you're expecting me to say w.r.t. that. Is there a particular > question you have? I am mentioning that I looked at the patch in the below link which is same as this scenario.. Here the cleaning of cpu caches and L3 caches were modified for asm. Do you think that we can apply this patch and can live with that? I didn't see any review comments for this patch. http://lists.denx.de/pipermail/u-boot/2015-January/200514.html Regards, Siva > > Thanks, > Mark.
diff --git a/arch/arm/cpu/armv8/cache_v8.c b/arch/arm/cpu/armv8/cache_v8.c index c5ec5297cd39..2a0492fbef52 100644 --- a/arch/arm/cpu/armv8/cache_v8.c +++ b/arch/arm/cpu/armv8/cache_v8.c @@ -128,10 +128,10 @@ void dcache_disable(void) if (!(sctlr & CR_C)) return; - set_sctlr(sctlr & ~(CR_C|CR_M)); - flush_dcache_all(); __asm_invalidate_tlb_all(); + + set_sctlr(sctlr & ~(CR_C|CR_M)); } int dcache_status(void)