| Message ID | 1300391223-11879-2-git-send-email-mspang@csclub.uwaterloo.ca |
|---|---|
| State | Rejected |
| Delegated to: | Albert ARIBAUD |
| Headers | show |
Hi Michael, Curiously, have any idea how to test cache stuff? 2011/3/18 Michael Spang <mspang@csclub.uwaterloo.ca>: > If U-Boot is loaded from RAM and the OS is loaded into an overlapping > region, the instruction cache is not coherent when that OS is started. > We must therefore invalidate the instruction cache in addition to > cleaning the data cache. > > Signed-off-by: Michael Spang <mspang@csclub.uwaterloo.ca> > --- > arch/arm/lib/cache.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c > index 30686fe..047786a 100644 > --- a/arch/arm/lib/cache.c > +++ b/arch/arm/lib/cache.c > @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long dummy2) > asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : "memory"); > /* disable write buffer as well (page 2-22) */ > asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); > + /* invalidate icache for coherence with cleaned dcache */ > + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); > #endif > #ifdef CONFIG_OMAP34XX > void v7_flush_cache_all(void); > -- > 1.7.2.3 > > _______________________________________________ > U-Boot mailing list > U-Boot@lists.denx.de > http://lists.denx.de/mailman/listinfo/u-boot >
On 3/19/11, arden jay <arden.jay@gmail.com> wrote: > Hi Michael, > Curiously, have any idea how to test cache stuff? I don't have any good suggestions for testing cache stuff in general, but this one is pretty easy to test if you have the board in question. Because U-Boot is loaded *as if it were linux* by the manufacturer's bootloader, the initial location of U-Boot is the same as where we ultimately load the Linux kernel. Without this patch the symptom is the failure of Linux to start from U-Boot. Michael
Hi Michael, I still have question. :) When ARM fetch instruction, it firstly try cache. It then should have cache miss, and forces to reload the instruction from memory? Why it will have problem while U-boot & Kernel at the same memory location? 2011/3/20 Michael Spang <mspang@csclub.uwaterloo.ca>: > On 3/19/11, arden jay <arden.jay@gmail.com> wrote: >> Hi Michael, >> Curiously, have any idea how to test cache stuff? > > I don't have any good suggestions for testing cache stuff in general, > but this one is pretty easy to test if you have the board in question. > Because U-Boot is loaded *as if it were linux* by the manufacturer's > bootloader, the initial location of U-Boot is the same as where we > ultimately load the Linux kernel. Without this patch the symptom is > the failure of Linux to start from U-Boot. > > Michael >
Le 03/20/11 06:36, arden jay a écrit : > Hi Michael, > > I still have question. :) > > When ARM fetch instruction, it firstly try cache. > It then should have cache miss, and forces to reload the instruction > from memory? > Why it will have problem while U-boot& Kernel at the same memory location? The error in your assumption is the "it then should have cache miss". A cache won't have a cache miss if it has already been fetched and not yet been invalidated, and the i-cache won't magically invalidate just because you do data writes at the location where the i-cache was fetched from; you end up with an i-cache that pretends to be valid but no longer matches the main memory content. Amicalement,
Hi Albert, I got it, thanks your explaination. 2011/3/20 Albert ARIBAUD <albert.aribaud@free.fr>: > Le 03/20/11 06:36, arden jay a écrit : >> >> Hi Michael, >> >> I still have question. :) >> >> When ARM fetch instruction, it firstly try cache. >> It then should have cache miss, and forces to reload the instruction >> from memory? >> Why it will have problem while U-boot& Kernel at the same memory >> location? > > The error in your assumption is the "it then should have cache miss". A > cache won't have a cache miss if it has already been fetched and not yet > been invalidated, and the i-cache won't magically invalidate just because > you do data writes at the location where the i-cache was fetched from; you > end up with an i-cache that pretends to be valid but no longer matches the > main memory content. > > Amicalement, > -- > Albert. >
On 3/20/11, arden jay <arden.jay@gmail.com> wrote: > Hi Michael, > > I still have question. :) > > When ARM fetch instruction, it firstly try cache. > It then should have cache miss, and forces to reload the instruction > from memory? > > Why it will have problem while U-boot & Kernel at the same memory location? If the instruction fetch misses the cache, then it works fine. That's why invalidating solves the problem. The problem is that it hits, but returns the wrong instructions. The code from U-Boot is loaded into the icache when U-Boot is started (before it is relocated) and some of it is still in the icache when we execute linux. So the CPU ends up fetching the wrong instructions. On ARM if code is stored to RAM, then the modified addresses must be manually invalidated in the instruction cache. The hardware does not do this for us. Michael
Hi Arden, On Sunday 20 March 2011 08:00 AM, arden jay wrote: > Hi Michael, > Curiously, have any idea how to test cache stuff? Recently I did some cache testing. Here is the technique I used for data-cache: To test flush: * Write a known pattern to a region of memory * Flush the region * Invalidate the region * Read back the region and see if you get the original pattern. If the flush was effective you will see the original data. To test Invalidate: * Write a known pattern to a region of memory * Invalidate the region immediately * Read back the region and see if you get the original pattern. You should *not* be seeing the expected pattern for the entire region if the invalidate was successful. Similar tests can be done for the I-cache, but maybe a little more tedious. I agree that these tests may not be 100% fool-proof - for instance what if both invalidate and flush didn't work in the first case. But these should at least catch very obvious errors and can be used in combination with other techniques to make sure that your cache operations are indeed working. Additionally, there was a JTAG debugger technique that I found quite useful. If your processor supports a technique called DAP(Debug Access Port or Dual Access Port, I am not sure) then a debugger like Lauterbach can view memory in two modes, one the normal or CPU mode and the other DAP mode. In normal mode the memory is dumped as if it is viewed from the CPU, so it goes through the cache and you see the cache contents if the area in question is in cache. In DAP mode the real memory contents are dumped. Comparing these two you can see whether cache and memory are coherent for a given area. It worked quite well with Lauterbach and OMAP4. br, Aneesh > > 2011/3/18 Michael Spang<mspang@csclub.uwaterloo.ca>: >> If U-Boot is loaded from RAM and the OS is loaded into an overlapping >> region, the instruction cache is not coherent when that OS is started. >> We must therefore invalidate the instruction cache in addition to >> cleaning the data cache. >> >> Signed-off-by: Michael Spang<mspang@csclub.uwaterloo.ca> >> --- >> arch/arm/lib/cache.c | 2 ++ >> 1 files changed, 2 insertions(+), 0 deletions(-) >> >> diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c >> index 30686fe..047786a 100644 >> --- a/arch/arm/lib/cache.c >> +++ b/arch/arm/lib/cache.c >> @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long dummy2) >> asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : "memory"); >> /* disable write buffer as well (page 2-22) */ >> asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); >> + /* invalidate icache for coherence with cleaned dcache */ >> + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); >> #endif >> #ifdef CONFIG_OMAP34XX >> void v7_flush_cache_all(void); >> -- >> 1.7.2.3 >> >> _______________________________________________ >> U-Boot mailing list >> U-Boot@lists.denx.de >> http://lists.denx.de/mailman/listinfo/u-boot >> > > >
Hi Michael, Thanks, it is the key point: "The problem is that it hits, but returns the wrong instructions." Hi Aneesh, Thanks for your sharing, it makes sense to check that way. 2011/3/21 Aneesh V <aneesh@ti.com>: > Hi Arden, > > On Sunday 20 March 2011 08:00 AM, arden jay wrote: >> >> Hi Michael, >> Curiously, have any idea how to test cache stuff? > > Recently I did some cache testing. Here is the technique I used for > data-cache: > > To test flush: > * Write a known pattern to a region of memory > * Flush the region > * Invalidate the region > * Read back the region and see if you get the original pattern. If the > flush was effective you will see the original data. > > To test Invalidate: > * Write a known pattern to a region of memory > * Invalidate the region immediately > * Read back the region and see if you get the original pattern. You > should *not* be seeing the expected pattern for the entire region if > the invalidate was successful. > > Similar tests can be done for the I-cache, but maybe a little more > tedious. > > I agree that these tests may not be 100% fool-proof - for instance what > if both invalidate and flush didn't work in the first case. But these > should at least catch very obvious errors and can be used in > combination with other techniques to make sure that your cache > operations are indeed working. > > Additionally, there was a JTAG debugger technique that I found quite > useful. If your processor supports a technique called DAP(Debug Access > Port or Dual Access Port, I am not sure) then a debugger like > Lauterbach can view memory in two modes, one the normal or CPU mode and > the other DAP mode. In normal mode the memory is dumped as if it is > viewed from the CPU, so it goes through the cache and you see the cache > contents if the area in question is in cache. In DAP mode the > real memory contents are dumped. Comparing these two you can see > whether cache and memory are coherent for a given area. It worked quite > well with Lauterbach and OMAP4. > > br, > Aneesh > >> >> 2011/3/18 Michael Spang<mspang@csclub.uwaterloo.ca>: >>> >>> If U-Boot is loaded from RAM and the OS is loaded into an overlapping >>> region, the instruction cache is not coherent when that OS is started. >>> We must therefore invalidate the instruction cache in addition to >>> cleaning the data cache. >>> >>> Signed-off-by: Michael Spang<mspang@csclub.uwaterloo.ca> >>> --- >>> arch/arm/lib/cache.c | 2 ++ >>> 1 files changed, 2 insertions(+), 0 deletions(-) >>> >>> diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c >>> index 30686fe..047786a 100644 >>> --- a/arch/arm/lib/cache.c >>> +++ b/arch/arm/lib/cache.c >>> @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long >>> dummy2) >>> asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : >>> "memory"); >>> /* disable write buffer as well (page 2-22) */ >>> asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); >>> + /* invalidate icache for coherence with cleaned dcache */ >>> + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); >>> #endif >>> #ifdef CONFIG_OMAP34XX >>> void v7_flush_cache_all(void); >>> -- >>> 1.7.2.3 >>> >>> _______________________________________________ >>> U-Boot mailing list >>> U-Boot@lists.denx.de >>> http://lists.denx.de/mailman/listinfo/u-boot >>> >> >> >> >
(although this patch is more than two *years* old, it never got properly answered to. I am doing so here to make sure future readers know why it was not applied and won't be.) Hi Michael, On Thu, 17 Mar 2011 15:46:55 -0400, Michael Spang <mspang@csclub.uwaterloo.ca> wrote: > If U-Boot is loaded from RAM and the OS is loaded into an overlapping > region, the instruction cache is not coherent when that OS is started. > We must therefore invalidate the instruction cache in addition to > cleaning the data cache. > > Signed-off-by: Michael Spang <mspang@csclub.uwaterloo.ca> > --- > arch/arm/lib/cache.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c > index 30686fe..047786a 100644 > --- a/arch/arm/lib/cache.c > +++ b/arch/arm/lib/cache.c > @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long dummy2) > asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : "memory"); > /* disable write buffer as well (page 2-22) */ > asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); > + /* invalidate icache for coherence with cleaned dcache */ > + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); > #endif > #ifdef CONFIG_OMAP34XX > void v7_flush_cache_all(void); This patch has obviously not been applied, and won't be, because of two reasons: i) overwriting part of U-Boot when loading the kernel is a bug which should not be papered over, and ii) if we had to do this anyway, the right place to do it would have been where the issue might occur, that is, in the OS boot sequence, not in the cache handling functions. Apologies for not properly answering in due time. Amicalement,
Albert, That's not a correct characterization of the bug. The incoherent cache lines are from before the relocation stage. If U-Boot is relocating from RAM, and later copies the OS there without invalidating those lines, then that's a bug in U-Boot. Michael On Mon, Jul 29, 2013 at 3:19 AM, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote: > (although this patch is more than two *years* old, it never got properly > answered to. I am doing so here to make sure future readers know why it > was not applied and won't be.) > > Hi Michael, > > On Thu, 17 Mar 2011 15:46:55 -0400, Michael Spang > <mspang@csclub.uwaterloo.ca> wrote: > >> If U-Boot is loaded from RAM and the OS is loaded into an overlapping >> region, the instruction cache is not coherent when that OS is started. >> We must therefore invalidate the instruction cache in addition to >> cleaning the data cache. >> >> Signed-off-by: Michael Spang <mspang@csclub.uwaterloo.ca> >> --- >> arch/arm/lib/cache.c | 2 ++ >> 1 files changed, 2 insertions(+), 0 deletions(-) >> >> diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c >> index 30686fe..047786a 100644 >> --- a/arch/arm/lib/cache.c >> +++ b/arch/arm/lib/cache.c >> @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long dummy2) >> asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : "memory"); >> /* disable write buffer as well (page 2-22) */ >> asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); >> + /* invalidate icache for coherence with cleaned dcache */ >> + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); >> #endif >> #ifdef CONFIG_OMAP34XX >> void v7_flush_cache_all(void); > > This patch has obviously not been applied, and won't be, because of > two reasons: i) overwriting part of U-Boot when loading the kernel > is a bug which should not be papered over, and ii) if we had to do this > anyway, the right place to do it would have been where the issue might > occur, that is, in the OS boot sequence, not in the cache handling > functions. > > Apologies for not properly answering in due time. > > Amicalement, > -- > Albert.
Hi Michael, On Mon, 29 Jul 2013 08:57:53 -0400, Michael Spang <mspang@csclub.uwaterloo.ca> wrote: > Albert, > > That's not a correct characterization of the bug. > > The incoherent cache lines are from before the relocation stage. If > U-Boot is relocating from RAM, and later copies the OS there without > invalidating those lines, then that's a bug in U-Boot. Thanks for this pointing out this scenario, which is correct, although it was not raised in the original bug description. I begs however the question whether anything from re-relocation can survive in the icache from between the moment the relocation starts and the moment the OS is given transfer to. IOW, was this issue actually met? Anyway I stand by my statement that even if there is an issue to fix, this patch fixes it in the wrong place. > Michael Amicalement,
On Mon, Jul 29, 2013 at 10:09 AM, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote: > Hi Michael, > > On Mon, 29 Jul 2013 08:57:53 -0400, Michael Spang > <mspang@csclub.uwaterloo.ca> wrote: > >> Albert, >> >> That's not a correct characterization of the bug. >> >> The incoherent cache lines are from before the relocation stage. If >> U-Boot is relocating from RAM, and later copies the OS there without >> invalidating those lines, then that's a bug in U-Boot. > > Thanks for this pointing out this scenario, which is correct, although > it was not raised in the original bug description. > > I begs however the question whether anything from re-relocation can > survive in the icache from between the moment the relocation starts and > the moment the OS is given transfer to. IOW, was this issue actually > met? Yes, I had boot failures without this patch on the device I was porting to (TS-7800). It happened consistently. > > Anyway I stand by my statement that even if there is an issue to fix, > this patch fixes it in the wrong place. Ok, that's fair. I was hoping to repost this series someday and will take that into account. Michael
diff --git a/arch/arm/lib/cache.c b/arch/arm/lib/cache.c index 30686fe..047786a 100644 --- a/arch/arm/lib/cache.c +++ b/arch/arm/lib/cache.c @@ -37,6 +37,8 @@ void flush_cache (unsigned long dummy1, unsigned long dummy2) asm("0: mrc p15, 0, r15, c7, c10, 3\n\t" "bne 0b\n" : : : "memory"); /* disable write buffer as well (page 2-22) */ asm("mcr p15, 0, %0, c7, c10, 4" : : "r" (0)); + /* invalidate icache for coherence with cleaned dcache */ + asm("mcr p15, 0, %0, c7, c5, 0" : : "r" (0)); #endif #ifdef CONFIG_OMAP34XX void v7_flush_cache_all(void);
If U-Boot is loaded from RAM and the OS is loaded into an overlapping region, the instruction cache is not coherent when that OS is started. We must therefore invalidate the instruction cache in addition to cleaning the data cache. Signed-off-by: Michael Spang <mspang@csclub.uwaterloo.ca> --- arch/arm/lib/cache.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)