| Message ID | 20180530103122.27674-1-npiggin@gmail.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 926bc2f100c24d4842b3064b5af44ae964c1d81c |
| Headers | show |
| Series | powerpc/64s: Fix compiler store ordering to SLB shadow area | expand |
Nicholas Piggin <npiggin@gmail.com> writes: > The stores to update the SLB shadow area must be made as they appear > in the C code, so that the hypervisor does not see an entry with > mismatched vsid and esid. Use WRITE_ONCE for this. > > GCC has been observed to elide the first store to esid in the update, > which means that if the hypervisor interrupts the guest after storing > to vsid, it could see an entry with old esid and new vsid, which may > possibly result in memory corruption. > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > --- > arch/powerpc/mm/slb.c | 8 ++++---- > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c > index 66577cc66dc9..2f4b33b24b3b 100644 > --- a/arch/powerpc/mm/slb.c > +++ b/arch/powerpc/mm/slb.c > @@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize, > * updating it. No write barriers are needed here, provided > * we only update the current CPU's SLB shadow buffer. > */ > - p->save_area[index].esid = 0; > - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); > - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); > + WRITE_ONCE(p->save_area[index].esid, 0); > + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); > + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); What's the code-gen for that look like? I suspect it's terrible? Should we just do it in inline-asm I wonder? cheers
On Fri, 01 Jun 2018 00:22:21 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote: > Nicholas Piggin <npiggin@gmail.com> writes: > > > The stores to update the SLB shadow area must be made as they appear > > in the C code, so that the hypervisor does not see an entry with > > mismatched vsid and esid. Use WRITE_ONCE for this. > > > > GCC has been observed to elide the first store to esid in the update, > > which means that if the hypervisor interrupts the guest after storing > > to vsid, it could see an entry with old esid and new vsid, which may > > possibly result in memory corruption. > > > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > > --- > > arch/powerpc/mm/slb.c | 8 ++++---- > > 1 file changed, 4 insertions(+), 4 deletions(-) > > > > diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c > > index 66577cc66dc9..2f4b33b24b3b 100644 > > --- a/arch/powerpc/mm/slb.c > > +++ b/arch/powerpc/mm/slb.c > > @@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize, > > * updating it. No write barriers are needed here, provided > > * we only update the current CPU's SLB shadow buffer. > > */ > > - p->save_area[index].esid = 0; > > - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); > > - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); > > + WRITE_ONCE(p->save_area[index].esid, 0); > > + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); > > + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); > > What's the code-gen for that look like? I suspect it's terrible? Yeah it's not great. > > Should we just do it in inline-asm I wonder? There should be no fundamental correctness reason why we can't store to a volatile with a byteswap store. The other option we could do is add a compiler barrier() between each store. The reason I didn't is that in theory we don't need to invalidate all memory contents here, but in practice probably the end result code generation would be better. Thanks, Nick
Nicholas Piggin <npiggin@gmail.com> writes: > On Fri, 01 Jun 2018 00:22:21 +1000 > Michael Ellerman <mpe@ellerman.id.au> wrote: >> Nicholas Piggin <npiggin@gmail.com> writes: >> > The stores to update the SLB shadow area must be made as they appear >> > in the C code, so that the hypervisor does not see an entry with >> > mismatched vsid and esid. Use WRITE_ONCE for this. >> > >> > GCC has been observed to elide the first store to esid in the update, >> > which means that if the hypervisor interrupts the guest after storing >> > to vsid, it could see an entry with old esid and new vsid, which may >> > possibly result in memory corruption. >> > >> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> >> > --- >> > arch/powerpc/mm/slb.c | 8 ++++---- >> > 1 file changed, 4 insertions(+), 4 deletions(-) >> > >> > diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c >> > index 66577cc66dc9..2f4b33b24b3b 100644 >> > --- a/arch/powerpc/mm/slb.c >> > +++ b/arch/powerpc/mm/slb.c >> > @@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize, >> > * updating it. No write barriers are needed here, provided >> > * we only update the current CPU's SLB shadow buffer. >> > */ >> > - p->save_area[index].esid = 0; >> > - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); >> > - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); >> > + WRITE_ONCE(p->save_area[index].esid, 0); >> > + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); >> > + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); >> >> What's the code-gen for that look like? I suspect it's terrible? > > Yeah it's not great. Actually with GCC 7 the WRITE_ONCE() doesn't make it any worse. Which is a little suspicious. But it is doing the first store: li r10,0 # r10 = 0 ld r29,56(r13) # r29 = paca->slb_shadow_ptr rldicr r8,r31,4,59 # r8 = index rldicr r9,r9,32,31 add r29,r29,r8 # r29 = r29 + index oris r9,r9,65535 std r10,16(r29) # esid = r10 = 0 So I'll just merge this as-is. cheers
On Fri, Jun 01, 2018 at 08:52:27AM +1000, Nicholas Piggin wrote: > On Fri, 01 Jun 2018 00:22:21 +1000 > Michael Ellerman <mpe@ellerman.id.au> wrote: > > Nicholas Piggin <npiggin@gmail.com> writes: > > > - p->save_area[index].esid = 0; > > > - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); > > > - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); > > > + WRITE_ONCE(p->save_area[index].esid, 0); > > > + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); > > > + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); > > > > What's the code-gen for that look like? I suspect it's terrible? > > Yeah it's not great. > > > > > Should we just do it in inline-asm I wonder? That is my recommendation: that will work for all compiler versions. > There should be no fundamental correctness reason why we can't store > to a volatile with a byteswap store. There are may operations that are *not* correct to merge into a volatile memory access, and which are fine is different for every arch. GCC simply disallows combining anything into any volatile memory by default. This is kind of fine because volatile already means "I want this to go slow", in common cases ;-) I'll see what I can do to make the byteswap load/stores work with volatile (for powerpc). > The other option we could do is > add a compiler barrier() between each store. The reason I didn't is > that in theory we don't need to invalidate all memory contents here, > but in practice probably the end result code generation would be > better. Something like p->save_area[index].esid = 0; asm("" : : "m"(p->save_area[index].esid)); p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); asm("" : : "m"(p->save_area[index].vsid)); p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); should do the trick (and once again for the second write to esid, if you want to be sure it is not optimised away). Segher
On Wed, 2018-05-30 at 10:31:22 UTC, Nicholas Piggin wrote: > The stores to update the SLB shadow area must be made as they appear > in the C code, so that the hypervisor does not see an entry with > mismatched vsid and esid. Use WRITE_ONCE for this. > > GCC has been observed to elide the first store to esid in the update, > which means that if the hypervisor interrupts the guest after storing > to vsid, it could see an entry with old esid and new vsid, which may > possibly result in memory corruption. > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Applied to powerpc next, thanks. https://git.kernel.org/powerpc/c/926bc2f100c24d4842b3064b5af44a cheers
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 66577cc66dc9..2f4b33b24b3b 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize, * updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid = 0; - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); + WRITE_ONCE(p->save_area[index].esid, 0); + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); } static inline void slb_shadow_clear(enum slb_index index) { - get_slb_shadow()->save_area[index].esid = 0; + WRITE_ONCE(get_slb_shadow()->save_area[index].esid, 0); } static inline void create_shadowed_slbe(unsigned long ea, int ssize,
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use WRITE_ONCE for this. GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption. Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- arch/powerpc/mm/slb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)