| Message ID | 1305753895-24845-5-git-send-email-ericvh@gmail.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
On Wed, 2011-05-18 at 16:24 -0500, Eric Van Hensbergen wrote: > For BGP, it is convenient for 'kmalloc' to come back with 32-byte > aligned units for torus DMA > > Signed-off-by: Eric Van Hensbergen <ericvh@gmail.com> > --- > arch/powerpc/include/asm/page_32.h | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/arch/powerpc/include/asm/page_32.h b/arch/powerpc/include/asm/page_32.h > index 68d73b2..fb0a7ae 100644 > --- a/arch/powerpc/include/asm/page_32.h > +++ b/arch/powerpc/include/asm/page_32.h > @@ -9,7 +9,7 @@ > > #define VM_DATA_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS32 > > -#ifdef CONFIG_NOT_COHERENT_CACHE > +#if defined(CONFIG_NOT_COHERENT_CACHE) || defined(CONFIG_BGP) > #define ARCH_DMA_MINALIGN L1_CACHE_BYTES > #endif Is DMA cache coherent on BG/P ? That's odd for a 4xx base :-) Cheers, Ben.
On Thu, May 19, 2011 at 7:36 PM, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > On Wed, 2011-05-18 at 16:24 -0500, Eric Van Hensbergen wrote: >> >> -#ifdef CONFIG_NOT_COHERENT_CACHE >> +#if defined(CONFIG_NOT_COHERENT_CACHE) || defined(CONFIG_BGP) >> #define ARCH_DMA_MINALIGN L1_CACHE_BYTES >> #endif > > Is DMA cache coherent on BG/P ? That's odd for a 4xx base :-) > My understanding of things (which could be totally wrong) is that the DMA we care about on BG/P (namely the Torus and Collective networks) is coherent at the L2. Of course the change in question is talking about L1_CACHE_BYTES, so my reading of this is that its a sleazy way of getting aligned mallocs that make interactions with the tightly coupled networks easier/more-efficient. I'm open to alternative suggestions. -eric
On 05/19/2011 07:36 PM, Benjamin Herrenschmidt wrote: > On Wed, 2011-05-18 at 16:24 -0500, Eric Van Hensbergen wrote: > >> For BGP, it is convenient for 'kmalloc' to come back with 32-byte >> aligned units for torus DMA >> >> Signed-off-by: Eric Van Hensbergen<ericvh@gmail.com> >> --- >> arch/powerpc/include/asm/page_32.h | 2 +- >> 1 files changed, 1 insertions(+), 1 deletions(-) >> >> diff --git a/arch/powerpc/include/asm/page_32.h b/arch/powerpc/include/asm/page_32.h >> index 68d73b2..fb0a7ae 100644 >> --- a/arch/powerpc/include/asm/page_32.h >> +++ b/arch/powerpc/include/asm/page_32.h >> @@ -9,7 +9,7 @@ >> >> #define VM_DATA_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS32 >> >> -#ifdef CONFIG_NOT_COHERENT_CACHE >> +#if defined(CONFIG_NOT_COHERENT_CACHE) || defined(CONFIG_BGP) >> #define ARCH_DMA_MINALIGN L1_CACHE_BYTES >> #endif >> > Is DMA cache coherent on BG/P ? That's odd for a 4xx base :-) > Actually DMA sends invalidate requests to the snoop unit(L2 level). BGP SoC is a bit different from other 4xx base. Some details can be found in www.scc.acad.bg/documentation/team.pdf - kaz > Cheers, > Ben. > > > > _______________________________________________ > bg-linux mailing list > bg-linux@lists.anl-external.org > https://lists.anl-external.org/mailman/listinfo/bg-linux > http://bg-linux.anl-external.org/wiki >
On Thu, 2011-05-19 at 19:47 -0500, Eric Van Hensbergen wrote: > On Thu, May 19, 2011 at 7:36 PM, Benjamin Herrenschmidt > <benh@kernel.crashing.org> wrote: > > On Wed, 2011-05-18 at 16:24 -0500, Eric Van Hensbergen wrote: > >> > >> -#ifdef CONFIG_NOT_COHERENT_CACHE > >> +#if defined(CONFIG_NOT_COHERENT_CACHE) || defined(CONFIG_BGP) > >> #define ARCH_DMA_MINALIGN L1_CACHE_BYTES > >> #endif > > > > Is DMA cache coherent on BG/P ? That's odd for a 4xx base :-) > > > > My understanding of things (which could be totally wrong) is that the > DMA we care about on BG/P (namely the Torus and Collective networks) > is coherent at the L2. Of course the change in question is talking > about L1_CACHE_BYTES, so my reading of this is that its a sleazy way > of getting aligned mallocs that make interactions with the tightly > coupled networks easier/more-efficient. I'm open to alternative > suggestions. But if it's not coherent with L1, then you sould have CONFIG_NOT_COHERENT_CACHE set and not need that patch... or am I missing something ? One thing we should do some day as well is make that whole non-coherent be runtime selected, on the list of things to fix 440+47x in the same kernel. Pfiew.... Cheers, Ben.
On Thu, 2011-05-19 at 20:32 -0500, Kazutomo Yoshii wrote: > > Actually DMA sends invalidate requests to the snoop unit(L2 level). > BGP SoC is a bit different from other 4xx base. Well, some other 44x also have a snooping L2 (more or less), but L1 is usually the problem. Cheers, Ben.
On Fri, 2011-05-20 at 12:08 +1000, Benjamin Herrenschmidt wrote: > On Thu, 2011-05-19 at 20:32 -0500, Kazutomo Yoshii wrote: > > > > Actually DMA sends invalidate requests to the snoop unit(L2 level). > > BGP SoC is a bit different from other 4xx base. > > Well, some other 44x also have a snooping L2 (more or less), but L1 is > usually the problem. Hrm... looking at the doco you pointed me to, it looks like the L1 -is- coherent, it gets snoop kills from the L2. Ok so we do need to make this non-coherent cache stuff a runtime option (well, we need that for 440+476 too anyways so may as well do it now). As for the alignment of kmalloc, it looks like a hack that should be done in the torus code itself. Cheers, Ben.
On 05/19/2011 09:13 PM, Benjamin Herrenschmidt wrote: > On Fri, 2011-05-20 at 12:08 +1000, Benjamin Herrenschmidt wrote: > >> On Thu, 2011-05-19 at 20:32 -0500, Kazutomo Yoshii wrote: >> >>> Actually DMA sends invalidate requests to the snoop unit(L2 level). >>> BGP SoC is a bit different from other 4xx base. >>> >> Well, some other 44x also have a snooping L2 (more or less), but L1 is >> usually the problem. >> > Hrm... looking at the doco you pointed me to, it looks like the L1 -is- > coherent, it gets snoop kills from the L2. > Right, L1D is coherent if write throug is set. L1I(virtually tagged) is not coherent, btw. > Ok so we do need to make this non-coherent cache stuff a runtime option > (well, we need that for 440+476 too anyways so may as well do it now). > What sort of runtime option do you mean? - kaz > As for the alignment of kmalloc, it looks like a hack that should be > done in the torus code itself. > > Cheers, > Ben. > > _______________________________________________ > bg-linux mailing list > bg-linux@lists.anl-external.org > https://lists.anl-external.org/mailman/listinfo/bg-linux > http://bg-linux.anl-external.org/wiki >
On Thu, 2011-05-19 at 22:02 -0500, Kazutomo Yoshii wrote: > On 05/19/2011 09:13 PM, Benjamin Herrenschmidt wrote: > > On Fri, 2011-05-20 at 12:08 +1000, Benjamin Herrenschmidt wrote: > > > >> On Thu, 2011-05-19 at 20:32 -0500, Kazutomo Yoshii wrote: > >> > >>> Actually DMA sends invalidate requests to the snoop unit(L2 level). > >>> BGP SoC is a bit different from other 4xx base. > >>> > >> Well, some other 44x also have a snooping L2 (more or less), but L1 is > >> usually the problem. > >> > > Hrm... looking at the doco you pointed me to, it looks like the L1 -is- > > coherent, it gets snoop kills from the L2. > > > Right, L1D is coherent if write throug is set. > L1I(virtually tagged) is not coherent, btw. Yes, I wrote the horrid hack to ping pong the pages between writable and executable afaik :-) > > Ok so we do need to make this non-coherent cache stuff a runtime option > > (well, we need that for 440+476 too anyways so may as well do it now). > > > What sort of runtime option do you mean? I've asked Tony to give it a go since he's taking over the 476 stuff. The idea is to make it a MMU feature and call into the cache flush conditionally on it rather than under the control of a ifdef. I want it to be possible to build a unified kernel that supports multiple 44x and 47x platforms including BGP if we can make it happen. There's still some roadblocks to support 47x and 44x in the same kernel, most notably the cache line size, but I have hopes I'll get Tony to fix them soon :-) Cheers, Ben. > - kaz > > As for the alignment of kmalloc, it looks like a hack that should be > > done in the torus code itself. > > > > Cheers, > > Ben. > > > > _______________________________________________ > > bg-linux mailing list > > bg-linux@lists.anl-external.org > > https://lists.anl-external.org/mailman/listinfo/bg-linux > > http://bg-linux.anl-external.org/wiki > >
diff --git a/arch/powerpc/include/asm/page_32.h b/arch/powerpc/include/asm/page_32.h index 68d73b2..fb0a7ae 100644 --- a/arch/powerpc/include/asm/page_32.h +++ b/arch/powerpc/include/asm/page_32.h @@ -9,7 +9,7 @@ #define VM_DATA_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS32 -#ifdef CONFIG_NOT_COHERENT_CACHE +#if defined(CONFIG_NOT_COHERENT_CACHE) || defined(CONFIG_BGP) #define ARCH_DMA_MINALIGN L1_CACHE_BYTES #endif
For BGP, it is convenient for 'kmalloc' to come back with 32-byte aligned units for torus DMA Signed-off-by: Eric Van Hensbergen <ericvh@gmail.com> --- arch/powerpc/include/asm/page_32.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)