diff mbox series

[RFC] RISC-V: Add proposed Ztso atomic mappings

Message ID 20230505171256.1380528-1-patrick@rivosinc.com
State New
Headers show
Series [RFC] RISC-V: Add proposed Ztso atomic mappings | expand

Commit Message

Patrick O'Neill May 5, 2023, 5:12 p.m. UTC 
The RISC-V Ztso extension currently has no effect on generated code.
With the additional ordering constraints guarenteed by Ztso, we can emit
more optimized atomic mappings than the RVWMO mappings.

This patch implements Andrea Parri's proposed Ztso mappings ("Proposed
Mapping").
  https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst

LLVM has implemented this same mapping (Ztso is still behind a
experimental flag in LLVM, so there is *not* a defined ABI for this yet).
  https://reviews.llvm.org/D143076

2023-05-04 Patrick O'Neill <patrick@rivosinc.com>

gcc/ChangeLog:

	* common/config/riscv/riscv-common.cc: Add Ztso and mark Ztso as
	dependent on 'a' extension.
	* config/riscv/riscv-opts.h (MASK_ZTSO): New mask.
	(TARGET_ZTSO): New target.
	* config/riscv/riscv.cc (riscv_memmodel_needs_amo_acquire): Add
	Ztso case.
	(riscv_memmodel_needs_amo_release): Add Ztso case.
	(riscv_print_operand): Add Ztso case for LR/SC annotations.
	* config/riscv/riscv.md: Import sync-rvwmo.md and sync-ztso.md.
	* config/riscv/riscv.opt: Add Ztso target variable.
	* config/riscv/sync.md (mem_thread_fence_1): Expand to RVWMO or
	Ztso specific insn.
	(atomic_load<mode>): Expand to RVWMO or Ztso specific insn.
	(atomic_store<mode>): Expand to RVWMO or Ztso specific insn.
	* config/riscv/sync-rvwmo.md: New file. Seperate out RVWMO
	specific load/store/fence mappings.
	* config/riscv/sync-ztso.md: New file. Seperate out Ztso
	specific load/store/fence mappings.

gcc/testsuite/ChangeLog:

	* gcc.target/riscv/amo-table-ztso-amo-add-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-amo-add-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-amo-add-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-amo-add-4.c: New test.
	* gcc.target/riscv/amo-table-ztso-amo-add-5.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-4.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-5.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-6.c: New test.
	* gcc.target/riscv/amo-table-ztso-compare-exchange-7.c: New test.
	* gcc.target/riscv/amo-table-ztso-fence-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-fence-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-fence-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-fence-4.c: New test.
	* gcc.target/riscv/amo-table-ztso-fence-5.c: New test.
	* gcc.target/riscv/amo-table-ztso-load-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-load-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-load-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-store-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-store-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-store-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-subword-amo-add-1.c: New test.
	* gcc.target/riscv/amo-table-ztso-subword-amo-add-2.c: New test.
	* gcc.target/riscv/amo-table-ztso-subword-amo-add-3.c: New test.
	* gcc.target/riscv/amo-table-ztso-subword-amo-add-4.c: New test.
	* gcc.target/riscv/amo-table-ztso-subword-amo-add-5.c: New test.

Signed-off-by: Patrick O'Neill <patrick@rivosinc.com>
---
 gcc/common/config/riscv/riscv-common.cc       |   4 +
 gcc/config/riscv/riscv-opts.h                 |   4 +
 gcc/config/riscv/riscv.cc                     |  18 ++-
 gcc/config/riscv/riscv.md                     |   2 +
 gcc/config/riscv/riscv.opt                    |   3 +
 gcc/config/riscv/sync-rvwmo.md                |  96 ++++++++++++++++
 gcc/config/riscv/sync-ztso.md                 |  71 ++++++++++++
 gcc/config/riscv/sync.md                      | 107 ++++++------------
 .../riscv/amo-table-ztso-amo-add-1.c          |  15 +++
 .../riscv/amo-table-ztso-amo-add-2.c          |  15 +++
 .../riscv/amo-table-ztso-amo-add-3.c          |  15 +++
 .../riscv/amo-table-ztso-amo-add-4.c          |  15 +++
 .../riscv/amo-table-ztso-amo-add-5.c          |  15 +++
 .../riscv/amo-table-ztso-compare-exchange-1.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-2.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-3.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-4.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-5.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-6.c |  10 ++
 .../riscv/amo-table-ztso-compare-exchange-7.c |  10 ++
 .../gcc.target/riscv/amo-table-ztso-fence-1.c |  14 +++
 .../gcc.target/riscv/amo-table-ztso-fence-2.c |  14 +++
 .../gcc.target/riscv/amo-table-ztso-fence-3.c |  14 +++
 .../gcc.target/riscv/amo-table-ztso-fence-4.c |  14 +++
 .../gcc.target/riscv/amo-table-ztso-fence-5.c |  15 +++
 .../gcc.target/riscv/amo-table-ztso-load-1.c  |  16 +++
 .../gcc.target/riscv/amo-table-ztso-load-2.c  |  16 +++
 .../gcc.target/riscv/amo-table-ztso-load-3.c  |  17 +++
 .../gcc.target/riscv/amo-table-ztso-store-1.c |  16 +++
 .../gcc.target/riscv/amo-table-ztso-store-2.c |  16 +++
 .../gcc.target/riscv/amo-table-ztso-store-3.c |  16 +++
 .../riscv/amo-table-ztso-subword-amo-add-1.c  |  10 ++
 .../riscv/amo-table-ztso-subword-amo-add-2.c  |  10 ++
 .../riscv/amo-table-ztso-subword-amo-add-3.c  |  10 ++
 .../riscv/amo-table-ztso-subword-amo-add-4.c  |  10 ++
 .../riscv/amo-table-ztso-subword-amo-add-5.c  |  10 ++
 36 files changed, 596 insertions(+), 72 deletions(-)
 create mode 100644 gcc/config/riscv/sync-rvwmo.md
 create mode 100644 gcc/config/riscv/sync-ztso.md
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-4.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-5.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-4.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-5.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-6.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-7.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-4.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-5.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-1.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-2.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-3.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-4.c
 create mode 100644 gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-5.c

Comments

Andrea Parri May 5, 2023, 6:55 p.m. UTC | #1 
On Fri, May 05, 2023 at 10:12:56AM -0700, Patrick O'Neill wrote:
> The RISC-V Ztso extension currently has no effect on generated code.
> With the additional ordering constraints guarenteed by Ztso, we can emit
> more optimized atomic mappings than the RVWMO mappings.
> 
> This patch implements Andrea Parri's proposed Ztso mappings ("Proposed
> Mapping").
>   https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst
> 
> LLVM has implemented this same mapping (Ztso is still behind a
> experimental flag in LLVM, so there is *not* a defined ABI for this yet).
>   https://reviews.llvm.org/D143076

Given the recent patches/discussions, it seems worth pointing out the
the Ztso mappings referred to above was designed to be compatible with
the mappings in Table A.6 and that they are _not_ compatible with the
mappings in Table A.7 or with a "subset" of A.7 (even assuming RVTSO).

  Andrea
Palmer Dabbelt May 5, 2023, 7:18 p.m. UTC | #2 
On Fri, 05 May 2023 11:55:31 PDT (-0700), Andrea Parri wrote:
> On Fri, May 05, 2023 at 10:12:56AM -0700, Patrick O'Neill wrote:
>> The RISC-V Ztso extension currently has no effect on generated code.
>> With the additional ordering constraints guarenteed by Ztso, we can emit
>> more optimized atomic mappings than the RVWMO mappings.
>>
>> This patch implements Andrea Parri's proposed Ztso mappings ("Proposed
>> Mapping").
>>   https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst
>>
>> LLVM has implemented this same mapping (Ztso is still behind a
>> experimental flag in LLVM, so there is *not* a defined ABI for this yet).
>>   https://reviews.llvm.org/D143076
>
> Given the recent patches/discussions, it seems worth pointing out the
> the Ztso mappings referred to above was designed to be compatible with
> the mappings in Table A.6 and that they are _not_ compatible with the
> mappings in Table A.7 or with a "subset" of A.7 (even assuming RVTSO).

I guess that brings up the question of what we should do about WMO/TSO 
compatibility.  IIUC the general plan has been that WMO binaries would 
be compatible with TSO binaries when run on TSO systems, and that TSO 
binaries would require TSO systems.

I suppose it would be possible to have TSO produce binaries that would 
run on WMO systems by just emitting a bunch of extra fences, but I don't 
think anyone wants that?

We've always just assumed that WMO binaries would be compatible with TSO 
binaries, but I don't think it's ever really been concretely discussed.  
Having an ABI break here wouldn't be the craziest idea as it'd let us 
fix some other issues, but that'd certainly need to be pretty widely 
discussed.

Do we have an idea of what A.7-compatible TSO mappings would look like?
Andrea Parri May 5, 2023, 8:10 p.m. UTC | #3 
On Fri, May 05, 2023 at 12:18:12PM -0700, Palmer Dabbelt wrote:
> On Fri, 05 May 2023 11:55:31 PDT (-0700), Andrea Parri wrote:
> > On Fri, May 05, 2023 at 10:12:56AM -0700, Patrick O'Neill wrote:
> > > The RISC-V Ztso extension currently has no effect on generated code.
> > > With the additional ordering constraints guarenteed by Ztso, we can emit
> > > more optimized atomic mappings than the RVWMO mappings.
> > > 
> > > This patch implements Andrea Parri's proposed Ztso mappings ("Proposed
> > > Mapping").
> > >   https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst
> > > 
> > > LLVM has implemented this same mapping (Ztso is still behind a
> > > experimental flag in LLVM, so there is *not* a defined ABI for this yet).
> > >   https://reviews.llvm.org/D143076
> > 
> > Given the recent patches/discussions, it seems worth pointing out the
> > the Ztso mappings referred to above was designed to be compatible with
> > the mappings in Table A.6 and that they are _not_ compatible with the
> > mappings in Table A.7 or with a "subset" of A.7 (even assuming RVTSO).
> 
> I guess that brings up the question of what we should do about WMO/TSO
> compatibility.  IIUC the general plan has been that WMO binaries would be
> compatible with TSO binaries when run on TSO systems, and that TSO binaries
> would require TSO systems.
> 
> I suppose it would be possible to have TSO produce binaries that would run
> on WMO systems by just emitting a bunch of extra fences, but I don't think
> anyone wants that?
> 
> We've always just assumed that WMO binaries would be compatible with TSO
> binaries, but I don't think it's ever really been concretely discussed.
> Having an ABI break here wouldn't be the craziest idea as it'd let us fix
> some other issues, but that'd certainly need to be pretty widely discussed.
> 
> Do we have an idea of what A.7-compatible TSO mappings would look like?

As in riscv-tso-mappings.rst but with

  atomic_store(memory_order_seq_cst)  |  s{b|h|w|d} ; fence rw,rw

would be A.7-compatible: call the resulting mappings "A.6-tso".

A.6-tso is (also) compatible with the following subset of A.7:

C/C++ Construct					| A.7-tso Mapping
------------------------------------------------------------------------------
Non-atomic load					| l{b|h|w|d}
atomic_load(memory_order_relaxed		| l{b|h|w|d}
atomic_load(memory_order_acquire)		| l{b|h|w|d}
atomic_load(memory_order_seq_cst)		| l{b|h|w|d}.aq
------------------------------------------------------------------------------
Non-atomic store				| s{b|h|w|d}
atomic_store(memory_order_relaxed)		| s{b|h|w|d}
atomic_store(memory_order_release)		| s{b|h|w|d}
atomic_store(memory_order_seq_cst)		| s{b|h|w|d}.rl
------------------------------------------------------------------------------
atomic_thread_fence(memory_order_acquire)	| nop
atomic_thread_fence(memory_order_release)	| nop
atomic_thread_fence(memory_order_acq_rel)	| nop
atomic_thread_fence(memory_order_seq_cst)	| fence rw,rw
------------------------------------------------------------------------------
C/C++ Construct					| RVTSO AMO Mapping
atomic_<op>(memory_order_relaxed)		| amo<op>.{w|d}
atomic_<op>(memory_order_acquire)		| amo<op>.{w|d}
atomic_<op>(memory_order_release)		| amo<op>.{w|d}
atomic_<op>(memory_order_acq_rel)		| amo<op>.{w|d}
atomic_<op>(memory_order_seq_cst)		| amo<op>.{w|d}
------------------------------------------------------------------------------
C/C++ Construct					| RVTSO LR/SC Mapping
atomic_<op>(memory_order_relaxed)		| loop: lr.{w|d} ; <op> ;
						|       sc.{w|d} ; bnez loop
atomic_<op>(memory_order_acquire)		| loop: lr.{w|d} ; <op> ;
						|       sc.{w|d} ; bnez loop
atomic_<op>(memory_order_release)		| loop: lr.{w|d} ; <op> ;
						|       sc.{w|d} ; bnez loop
atomic_<op>(memory_order_acq_rel)		| loop: lr.{w|d} ; <op> ;
						|       sc.{w|d} ; bnez loop
atomic_<op>(memory_order_seq_cst)		| loop: lr.{w|d}.aq ; <op> ;
						|       sc.{w|d}.rl ; bnez loop

  Andrea
Hans Boehm May 5, 2023, 9:42 p.m. UTC | #4 
I think A.6-tso also needs to change the last line in the table from
lr.aqrl ... sc to lr.aq ... sc.rl, otherwise I think we have problems with
a subsequent A.7-tso generated l.aq . Otherwise I agree.

I certainly agree that, given the Ztso extension, there should be a
standard compiler-implemented mapping that leverages it. I'm personally
much less enthusiastic about calling it an ABI. I'd like to see clarity
that the RVWMO ABI is the standard we expect portable libraries to be
prepared to use. If they want to test for and use Ztso internally, fine.
But having users deal with two different ABIs seems like a very high cost
for avoiding some (basically no-op?) fences.

Hans



On Fri, May 5, 2023 at 1:11 PM Andrea Parri <andrea@rivosinc.com> wrote:

> On Fri, May 05, 2023 at 12:18:12PM -0700, Palmer Dabbelt wrote:
> > On Fri, 05 May 2023 11:55:31 PDT (-0700), Andrea Parri wrote:
> > > On Fri, May 05, 2023 at 10:12:56AM -0700, Patrick O'Neill wrote:
> > > > The RISC-V Ztso extension currently has no effect on generated code.
> > > > With the additional ordering constraints guarenteed by Ztso, we can
> emit
> > > > more optimized atomic mappings than the RVWMO mappings.
> > > >
> > > > This patch implements Andrea Parri's proposed Ztso mappings
> ("Proposed
> > > > Mapping").
> > > >
> https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst
> > > >
> > > > LLVM has implemented this same mapping (Ztso is still behind a
> > > > experimental flag in LLVM, so there is *not* a defined ABI for this
> yet).
> > > >   https://reviews.llvm.org/D143076
> > >
> > > Given the recent patches/discussions, it seems worth pointing out the
> > > the Ztso mappings referred to above was designed to be compatible with
> > > the mappings in Table A.6 and that they are _not_ compatible with the
> > > mappings in Table A.7 or with a "subset" of A.7 (even assuming RVTSO).
> >
> > I guess that brings up the question of what we should do about WMO/TSO
> > compatibility.  IIUC the general plan has been that WMO binaries would be
> > compatible with TSO binaries when run on TSO systems, and that TSO
> binaries
> > would require TSO systems.
> >
> > I suppose it would be possible to have TSO produce binaries that would
> run
> > on WMO systems by just emitting a bunch of extra fences, but I don't
> think
> > anyone wants that?
> >
> > We've always just assumed that WMO binaries would be compatible with TSO
> > binaries, but I don't think it's ever really been concretely discussed.
> > Having an ABI break here wouldn't be the craziest idea as it'd let us fix
> > some other issues, but that'd certainly need to be pretty widely
> discussed.
> >
> > Do we have an idea of what A.7-compatible TSO mappings would look like?
>
> As in riscv-tso-mappings.rst but with
>
>   atomic_store(memory_order_seq_cst)  |  s{b|h|w|d} ; fence rw,rw
>
> would be A.7-compatible: call the resulting mappings "A.6-tso".
>
> A.6-tso is (also) compatible with the following subset of A.7:
>
> C/C++ Construct                                 | A.7-tso Mapping
>
> ------------------------------------------------------------------------------
> Non-atomic load                                 | l{b|h|w|d}
> atomic_load(memory_order_relaxed                | l{b|h|w|d}
> atomic_load(memory_order_acquire)               | l{b|h|w|d}
> atomic_load(memory_order_seq_cst)               | l{b|h|w|d}.aq
>
> ------------------------------------------------------------------------------
> Non-atomic store                                | s{b|h|w|d}
> atomic_store(memory_order_relaxed)              | s{b|h|w|d}
> atomic_store(memory_order_release)              | s{b|h|w|d}
> atomic_store(memory_order_seq_cst)              | s{b|h|w|d}.rl
>
> ------------------------------------------------------------------------------
> atomic_thread_fence(memory_order_acquire)       | nop
> atomic_thread_fence(memory_order_release)       | nop
> atomic_thread_fence(memory_order_acq_rel)       | nop
> atomic_thread_fence(memory_order_seq_cst)       | fence rw,rw
>
> ------------------------------------------------------------------------------
> C/C++ Construct                                 | RVTSO AMO Mapping
> atomic_<op>(memory_order_relaxed)               | amo<op>.{w|d}
> atomic_<op>(memory_order_acquire)               | amo<op>.{w|d}
> atomic_<op>(memory_order_release)               | amo<op>.{w|d}
> atomic_<op>(memory_order_acq_rel)               | amo<op>.{w|d}
> atomic_<op>(memory_order_seq_cst)               | amo<op>.{w|d}
>
> ------------------------------------------------------------------------------
> C/C++ Construct                                 | RVTSO LR/SC Mapping
> atomic_<op>(memory_order_relaxed)               | loop: lr.{w|d} ; <op> ;
>                                                 |       sc.{w|d} ; bnez
> loop
> atomic_<op>(memory_order_acquire)               | loop: lr.{w|d} ; <op> ;
>                                                 |       sc.{w|d} ; bnez
> loop
> atomic_<op>(memory_order_release)               | loop: lr.{w|d} ; <op> ;
>                                                 |       sc.{w|d} ; bnez
> loop
> atomic_<op>(memory_order_acq_rel)               | loop: lr.{w|d} ; <op> ;
>                                                 |       sc.{w|d} ; bnez
> loop
> atomic_<op>(memory_order_seq_cst)               | loop: lr.{w|d}.aq ; <op>
> ;
>                                                 |       sc.{w|d}.rl ; bnez
> loop
>
>   Andrea
>
Andrew Waterman May 5, 2023, 9:52 p.m. UTC | #5 
On Fri, May 5, 2023 at 2:42 PM Hans Boehm <hboehm@google.com> wrote:
>
> I think A.6-tso also needs to change the last line in the table from lr.aqrl ... sc to lr.aq ... sc.rl, otherwise I think we have problems with a subsequent A.7-tso generated l.aq . Otherwise I agree.
>
> I certainly agree that, given the Ztso extension, there should be a standard compiler-implemented mapping that leverages it. I'm personally much less enthusiastic about calling it an ABI. I'd like to see clarity that the RVWMO ABI is the standard we expect portable libraries to be prepared to use.

There's already a ratified sentiment that effectively implies this.
Ztso is not required by the RVA profiles, and so it follows that any
binary that's compatible across RVA-profile implementations cannot
assume the presence of Ztso.  (I agree the ABI should encode this
property for de jure purposes, too, but it's already a de facto
requirement.)

> If they want to test for and use Ztso internally, fine. But having users deal with two different ABIs seems like a very high cost for avoiding some (basically no-op?) fences.
>
> Hans
>
>
>
> On Fri, May 5, 2023 at 1:11 PM Andrea Parri <andrea@rivosinc.com> wrote:
>>
>> On Fri, May 05, 2023 at 12:18:12PM -0700, Palmer Dabbelt wrote:
>> > On Fri, 05 May 2023 11:55:31 PDT (-0700), Andrea Parri wrote:
>> > > On Fri, May 05, 2023 at 10:12:56AM -0700, Patrick O'Neill wrote:
>> > > > The RISC-V Ztso extension currently has no effect on generated code.
>> > > > With the additional ordering constraints guarenteed by Ztso, we can emit
>> > > > more optimized atomic mappings than the RVWMO mappings.
>> > > >
>> > > > This patch implements Andrea Parri's proposed Ztso mappings ("Proposed
>> > > > Mapping").
>> > > >   https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst
>> > > >
>> > > > LLVM has implemented this same mapping (Ztso is still behind a
>> > > > experimental flag in LLVM, so there is *not* a defined ABI for this yet).
>> > > >   https://reviews.llvm.org/D143076
>> > >
>> > > Given the recent patches/discussions, it seems worth pointing out the
>> > > the Ztso mappings referred to above was designed to be compatible with
>> > > the mappings in Table A.6 and that they are _not_ compatible with the
>> > > mappings in Table A.7 or with a "subset" of A.7 (even assuming RVTSO).
>> >
>> > I guess that brings up the question of what we should do about WMO/TSO
>> > compatibility.  IIUC the general plan has been that WMO binaries would be
>> > compatible with TSO binaries when run on TSO systems, and that TSO binaries
>> > would require TSO systems.
>> >
>> > I suppose it would be possible to have TSO produce binaries that would run
>> > on WMO systems by just emitting a bunch of extra fences, but I don't think
>> > anyone wants that?
>> >
>> > We've always just assumed that WMO binaries would be compatible with TSO
>> > binaries, but I don't think it's ever really been concretely discussed.
>> > Having an ABI break here wouldn't be the craziest idea as it'd let us fix
>> > some other issues, but that'd certainly need to be pretty widely discussed.
>> >
>> > Do we have an idea of what A.7-compatible TSO mappings would look like?
>>
>> As in riscv-tso-mappings.rst but with
>>
>>   atomic_store(memory_order_seq_cst)  |  s{b|h|w|d} ; fence rw,rw
>>
>> would be A.7-compatible: call the resulting mappings "A.6-tso".
>>
>> A.6-tso is (also) compatible with the following subset of A.7:
>>
>> C/C++ Construct                                 | A.7-tso Mapping
>> ------------------------------------------------------------------------------
>> Non-atomic load                                 | l{b|h|w|d}
>> atomic_load(memory_order_relaxed                | l{b|h|w|d}
>> atomic_load(memory_order_acquire)               | l{b|h|w|d}
>> atomic_load(memory_order_seq_cst)               | l{b|h|w|d}.aq
>> ------------------------------------------------------------------------------
>> Non-atomic store                                | s{b|h|w|d}
>> atomic_store(memory_order_relaxed)              | s{b|h|w|d}
>> atomic_store(memory_order_release)              | s{b|h|w|d}
>> atomic_store(memory_order_seq_cst)              | s{b|h|w|d}.rl
>> ------------------------------------------------------------------------------
>> atomic_thread_fence(memory_order_acquire)       | nop
>> atomic_thread_fence(memory_order_release)       | nop
>> atomic_thread_fence(memory_order_acq_rel)       | nop
>> atomic_thread_fence(memory_order_seq_cst)       | fence rw,rw
>> ------------------------------------------------------------------------------
>> C/C++ Construct                                 | RVTSO AMO Mapping
>> atomic_<op>(memory_order_relaxed)               | amo<op>.{w|d}
>> atomic_<op>(memory_order_acquire)               | amo<op>.{w|d}
>> atomic_<op>(memory_order_release)               | amo<op>.{w|d}
>> atomic_<op>(memory_order_acq_rel)               | amo<op>.{w|d}
>> atomic_<op>(memory_order_seq_cst)               | amo<op>.{w|d}
>> ------------------------------------------------------------------------------
>> C/C++ Construct                                 | RVTSO LR/SC Mapping
>> atomic_<op>(memory_order_relaxed)               | loop: lr.{w|d} ; <op> ;
>>                                                 |       sc.{w|d} ; bnez loop
>> atomic_<op>(memory_order_acquire)               | loop: lr.{w|d} ; <op> ;
>>                                                 |       sc.{w|d} ; bnez loop
>> atomic_<op>(memory_order_release)               | loop: lr.{w|d} ; <op> ;
>>                                                 |       sc.{w|d} ; bnez loop
>> atomic_<op>(memory_order_acq_rel)               | loop: lr.{w|d} ; <op> ;
>>                                                 |       sc.{w|d} ; bnez loop
>> atomic_<op>(memory_order_seq_cst)               | loop: lr.{w|d}.aq ; <op> ;
>>                                                 |       sc.{w|d}.rl ; bnez loop
>>
>>   Andrea
Andrea Parri May 5, 2023, 10:01 p.m. UTC | #6 
On Fri, May 05, 2023 at 02:42:38PM -0700, Hans Boehm wrote:
> I think A.6-tso also needs to change the last line in the table from
> lr.aqrl ... sc to lr.aq ... sc.rl, otherwise I think we have problems with
> a subsequent A.7-tso generated l.aq . Otherwise I agree.

Indeed!  Thanks for the correction.

  Andrea
diff mbox series

Patch

diff --git a/gcc/common/config/riscv/riscv-common.cc b/gcc/common/config/riscv/riscv-common.cc
index 309a52def75..6b168d45df1 100644
--- a/gcc/common/config/riscv/riscv-common.cc
+++ b/gcc/common/config/riscv/riscv-common.cc
@@ -108,6 +108,8 @@  static const riscv_implied_info_t riscv_implied_info[] =
   {"zhinx", "zhinxmin"},
   {"zhinxmin", "zfinx"},
 
+  {"ztso", "a"},
+
   {NULL, NULL}
 };
 
@@ -1265,6 +1267,8 @@  static const riscv_ext_flag_table_t riscv_ext_flag_table[] =
   {"svinval", &gcc_options::x_riscv_sv_subext, MASK_SVINVAL},
   {"svnapot", &gcc_options::x_riscv_sv_subext, MASK_SVNAPOT},
 
+  {"ztso", &gcc_options::x_riscv_ztso_subext, MASK_ZTSO},
+
   {"xtheadba",      &gcc_options::x_riscv_xthead_subext, MASK_XTHEADBA},
   {"xtheadbb",      &gcc_options::x_riscv_xthead_subext, MASK_XTHEADBB},
   {"xtheadbs",      &gcc_options::x_riscv_xthead_subext, MASK_XTHEADBS},
diff --git a/gcc/config/riscv/riscv-opts.h b/gcc/config/riscv/riscv-opts.h
index 4207db240ea..e702d77184a 100644
--- a/gcc/config/riscv/riscv-opts.h
+++ b/gcc/config/riscv/riscv-opts.h
@@ -133,6 +133,10 @@  enum riscv_autovec_lmul_enum {
 #define TARGET_ZKSH   ((riscv_zk_subext & MASK_ZKSH) != 0)
 #define TARGET_ZKT    ((riscv_zk_subext & MASK_ZKT) != 0)
 
+#define MASK_ZTSO   (1 << 0)
+
+#define TARGET_ZTSO ((riscv_ztso_subext & MASK_ZTSO) != 0)
+
 #define MASK_VECTOR_ELEN_32    (1 << 0)
 #define MASK_VECTOR_ELEN_64    (1 << 1)
 #define MASK_VECTOR_ELEN_FP_32 (1 << 2)
diff --git a/gcc/config/riscv/riscv.cc b/gcc/config/riscv/riscv.cc
index 45a63cab9c9..8be2254d657 100644
--- a/gcc/config/riscv/riscv.cc
+++ b/gcc/config/riscv/riscv.cc
@@ -4356,6 +4356,10 @@  riscv_union_memmodels (enum memmodel model1, enum memmodel model2)
 static bool
 riscv_memmodel_needs_amo_acquire (enum memmodel model)
 {
+  /* ZTSO amo mappings require no annotations.  */
+  if (TARGET_ZTSO)
+    return false;
+
   switch (model)
     {
       case MEMMODEL_ACQ_REL:
@@ -4379,6 +4383,10 @@  riscv_memmodel_needs_amo_acquire (enum memmodel model)
 static bool
 riscv_memmodel_needs_amo_release (enum memmodel model)
 {
+  /* ZTSO amo mappings require no annotations.  */
+  if (TARGET_ZTSO)
+    return false;
+
   switch (model)
     {
       case MEMMODEL_ACQ_REL:
@@ -4575,14 +4583,20 @@  riscv_print_operand (FILE *file, rtx op, int letter)
       break;
 
     case 'I':
-      if (model == MEMMODEL_SEQ_CST)
+      if (TARGET_ZTSO && model != MEMMODEL_SEQ_CST)
+	/* LR ops only have an annotation for SEQ_CST in the Ztso mapping.  */
+	break;
+      else if (model == MEMMODEL_SEQ_CST)
 	fputs (".aqrl", file);
       else if (riscv_memmodel_needs_amo_acquire (model))
 	fputs (".aq", file);
       break;
 
     case 'J':
-      if (riscv_memmodel_needs_amo_release (model))
+      if (TARGET_ZTSO)
+	/* SC ops need no annotations in the Ztso mapping.  */
+	break;
+      else if (riscv_memmodel_needs_amo_release (model))
 	fputs (".rl", file);
       break;
 
diff --git a/gcc/config/riscv/riscv.md b/gcc/config/riscv/riscv.md
index 7065e68c0b7..2a0687a559a 100644
--- a/gcc/config/riscv/riscv.md
+++ b/gcc/config/riscv/riscv.md
@@ -3199,6 +3199,8 @@ 
 (include "bitmanip.md")
 (include "crypto.md")
 (include "sync.md")
+(include "sync-rvwmo.md")
+(include "sync-ztso.md")
 (include "peephole.md")
 (include "pic.md")
 (include "generic.md")
diff --git a/gcc/config/riscv/riscv.opt b/gcc/config/riscv/riscv.opt
index 63d4710cb15..b1c7f01f4df 100644
--- a/gcc/config/riscv/riscv.opt
+++ b/gcc/config/riscv/riscv.opt
@@ -235,6 +235,9 @@  int riscv_zm_subext
 TargetVariable
 int riscv_sv_subext
 
+TargetVariable
+int riscv_ztso_subext
+
 TargetVariable
 int riscv_xthead_subext
 
diff --git a/gcc/config/riscv/sync-rvwmo.md b/gcc/config/riscv/sync-rvwmo.md
new file mode 100644
index 00000000000..bcc1ec8579d
--- /dev/null
+++ b/gcc/config/riscv/sync-rvwmo.md
@@ -0,0 +1,96 @@ 
+;; Machine description for RISC-V atomic operations.
+;; Copyright (C) 2011-2023 Free Software Foundation, Inc.
+;; Contributed by Andrew Waterman (andrew@sifive.com).
+;; Based on MIPS target for GNU compiler.
+
+;; This file is part of GCC.
+
+;; GCC is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+
+;; GCC is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GCC; see the file COPYING3.  If not see
+;; <http://www.gnu.org/licenses/>.
+
+;; Memory barrier.
+
+(define_insn "mem_thread_fence_rvwmo"
+  [(set (match_operand:BLK 0 "" "")
+	(unspec:BLK [(match_dup 0)] UNSPEC_MEMORY_BARRIER))
+   (match_operand:SI 1 "const_int_operand" "")]  ;; model
+  "!TARGET_ZTSO"
+  {
+    enum memmodel model = (enum memmodel) INTVAL (operands[1]);
+    model = memmodel_base (model);
+
+    if (model == MEMMODEL_SEQ_CST)
+	return "fence\trw,rw";
+    else if (model == MEMMODEL_ACQ_REL)
+	return "fence.tso";
+    else if (model == MEMMODEL_ACQUIRE)
+	return "fence\tr,rw";
+    else if (model == MEMMODEL_RELEASE)
+	return "fence\trw,w";
+    else
+	gcc_unreachable ();
+  }
+  [(set (attr "length") (const_int 4))])
+
+;; Atomic memory operations.
+
+(define_insn "atomic_load_rvwmo<mode>"
+  [(set (match_operand:GPR 0 "register_operand" "=r")
+    (unspec_volatile:GPR
+      [(match_operand:GPR 1 "memory_operand" "A")
+       (match_operand:SI 2 "const_int_operand")]  ;; model
+      UNSPEC_ATOMIC_LOAD))]
+  "TARGET_ATOMIC && !TARGET_ZTSO"
+  {
+    enum memmodel model = (enum memmodel) INTVAL (operands[2]);
+    model = memmodel_base (model);
+
+    if (model == MEMMODEL_SEQ_CST)
+      return "fence\trw,rw\;"
+	     "l<amo>\t%0,%1\;"
+	     "fence\tr,rw";
+    if (model == MEMMODEL_ACQUIRE)
+      return "l<amo>\t%0,%1\;"
+	     "fence\tr,rw";
+    else
+      return "l<amo>\t%0,%1";
+  }
+  [(set_attr "type" "atomic")
+   (set (attr "length") (const_int 12))])
+
+;; Implement atomic stores with conservative fences.
+;; This allows us to be compatible with the ISA manual Table A.6 and Table A.7.
+(define_insn "atomic_store_rvwmo<mode>"
+  [(set (match_operand:GPR 0 "memory_operand" "=A")
+    (unspec_volatile:GPR
+      [(match_operand:GPR 1 "reg_or_0_operand" "rJ")
+       (match_operand:SI 2 "const_int_operand")]  ;; model
+      UNSPEC_ATOMIC_STORE))]
+  "TARGET_ATOMIC && !TARGET_ZTSO"
+  {
+    enum memmodel model = (enum memmodel) INTVAL (operands[2]);
+    model = memmodel_base (model);
+
+    if (model == MEMMODEL_SEQ_CST)
+      return "fence\trw,w\;"
+	     "s<amo>\t%z1,%0\;"
+	     "fence\trw,rw";
+    if (model == MEMMODEL_RELEASE)
+      return "fence\trw,w\;"
+	     "s<amo>\t%z1,%0";
+    else
+      return "s<amo>\t%z1,%0";
+  }
+  [(set_attr "type" "atomic")
+   (set (attr "length") (const_int 12))])
diff --git a/gcc/config/riscv/sync-ztso.md b/gcc/config/riscv/sync-ztso.md
new file mode 100644
index 00000000000..9a0de83861e
--- /dev/null
+++ b/gcc/config/riscv/sync-ztso.md
@@ -0,0 +1,71 @@ 
+;; Machine description for RISC-V atomic operations.
+;; Copyright (C) 2011-2023 Free Software Foundation, Inc.
+;; Contributed by Andrew Waterman (andrew@sifive.com).
+;; Based on MIPS target for GNU compiler.
+
+;; This file is part of GCC.
+
+;; GCC is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+
+;; GCC is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GCC; see the file COPYING3.  If not see
+;; <http://www.gnu.org/licenses/>.
+
+;; Memory barriers.
+
+(define_insn "mem_thread_fence_ztso"
+  [(set (match_operand:BLK 0 "" "")
+	(unspec:BLK [(match_dup 0)] UNSPEC_MEMORY_BARRIER))
+   (match_operand:SI 1 "const_int_operand" "")]  ;; model
+  "TARGET_ZTSO"
+  {
+    enum memmodel model = (enum memmodel) INTVAL (operands[1]);
+    model = memmodel_base (model);
+
+    if (model == MEMMODEL_SEQ_CST)
+	return "fence\trw,rw";
+    else
+	gcc_unreachable ();
+  }
+  [(set (attr "length") (const_int 4))])
+
+;; Atomic memory operations.
+
+(define_insn "atomic_load_ztso<mode>"
+  [(set (match_operand:GPR 0 "register_operand" "=r")
+    (unspec_volatile:GPR
+      [(match_operand:GPR 1 "memory_operand" "A")
+       (match_operand:SI 2 "const_int_operand")]  ;; model
+      UNSPEC_ATOMIC_LOAD))]
+  "TARGET_ATOMIC && TARGET_ZTSO"
+  {
+    enum memmodel model = (enum memmodel) INTVAL (operands[2]);
+    model = memmodel_base (model);
+
+    if (model == MEMMODEL_SEQ_CST)
+      return "fence\trw,rw\;"
+	     "l<amo>\t%0,%1\;";
+    else
+      return "l<amo>\t%0,%1";
+  }
+  [(set_attr "type" "atomic")
+   (set (attr "length") (const_int 12))])
+
+(define_insn "atomic_store_ztso<mode>"
+  [(set (match_operand:GPR 0 "memory_operand" "=A")
+    (unspec_volatile:GPR
+      [(match_operand:GPR 1 "reg_or_0_operand" "rJ")
+       (match_operand:SI 2 "const_int_operand")]  ;; model
+      UNSPEC_ATOMIC_STORE))]
+  "TARGET_ATOMIC && TARGET_ZTSO"
+  "s<amo>\t%z1,%0"
+  [(set_attr "type" "atomic")
+   (set (attr "length") (const_int 12))])
diff --git a/gcc/config/riscv/sync.md b/gcc/config/riscv/sync.md
index 9fc626267de..1de22ce2f71 100644
--- a/gcc/config/riscv/sync.md
+++ b/gcc/config/riscv/sync.md
@@ -37,87 +37,54 @@ 
   [(match_operand:SI 0 "const_int_operand" "")] ;; model
   ""
 {
-  if (INTVAL (operands[0]) != MEMMODEL_RELAXED)
+  enum memmodel model = memmodel_base (INTVAL (operands[0]));
+
+  if (TARGET_ZTSO && model == MEMMODEL_SEQ_CST)
+    {
+      rtx mem = gen_rtx_MEM (BLKmode, gen_rtx_SCRATCH (Pmode));
+      MEM_VOLATILE_P (mem) = 1;
+      emit_insn (gen_mem_thread_fence_ztso (mem, operands[0]));
+    }
+  else if (!TARGET_ZTSO && model != MEMMODEL_RELAXED)
     {
       rtx mem = gen_rtx_MEM (BLKmode, gen_rtx_SCRATCH (Pmode));
       MEM_VOLATILE_P (mem) = 1;
-      emit_insn (gen_mem_thread_fence_1 (mem, operands[0]));
+      emit_insn (gen_mem_thread_fence_rvwmo (mem, operands[0]));
     }
   DONE;
 })
 
-(define_insn "mem_thread_fence_1"
-  [(set (match_operand:BLK 0 "" "")
-	(unspec:BLK [(match_dup 0)] UNSPEC_MEMORY_BARRIER))
-   (match_operand:SI 1 "const_int_operand" "")] ;; model
-  ""
-  {
-    enum memmodel model = (enum memmodel) INTVAL (operands[1]);
-    model = memmodel_base (model);
-    if (model == MEMMODEL_SEQ_CST)
-	return "fence\trw,rw";
-    else if (model == MEMMODEL_ACQ_REL)
-	return "fence.tso";
-    else if (model == MEMMODEL_ACQUIRE)
-	return "fence\tr,rw";
-    else if (model == MEMMODEL_RELEASE)
-	return "fence\trw,w";
-    else
-	gcc_unreachable ();
-  }
-  [(set (attr "length") (const_int 4))])
-
 ;; Atomic memory operations.
 
-(define_insn "atomic_load<mode>"
-  [(set (match_operand:GPR 0 "register_operand" "=r")
-    (unspec_volatile:GPR
-      [(match_operand:GPR 1 "memory_operand" "A")
-       (match_operand:SI 2 "const_int_operand")]      ;; model
-      UNSPEC_ATOMIC_LOAD))]
+(define_expand "atomic_load<mode>"
+  [(match_operand:GPR 0 "register_operand")
+   (match_operand:GPR 1 "memory_operand")
+   (match_operand:SI 2 "const_int_operand")] ;; model
   "TARGET_ATOMIC"
-  {
-    enum memmodel model = (enum memmodel) INTVAL (operands[2]);
-    model = memmodel_base (model);
-
-    if (model == MEMMODEL_SEQ_CST)
-      return "fence\trw,rw\;"
-	     "l<amo>\t%0,%1\;"
-	     "fence\tr,rw";
-    if (model == MEMMODEL_ACQUIRE)
-      return "l<amo>\t%0,%1\;"
-	     "fence\tr,rw";
-    else
-      return "l<amo>\t%0,%1";
-  }
-  [(set_attr "type" "atomic")
-   (set (attr "length") (const_int 12))])
-
-;; Implement atomic stores with conservative fences.
-;; This allows us to be compatible with the ISA manual Table A.6 and Table A.7.
-(define_insn "atomic_store<mode>"
-  [(set (match_operand:GPR 0 "memory_operand" "=A")
-    (unspec_volatile:GPR
-      [(match_operand:GPR 1 "reg_or_0_operand" "rJ")
-       (match_operand:SI 2 "const_int_operand")]      ;; model
-      UNSPEC_ATOMIC_STORE))]
+{
+  if (TARGET_ZTSO)
+    emit_insn (gen_atomic_load_rvwmo<mode> (operands[0], operands[1],
+					    operands[2]));
+  else
+    emit_insn (gen_atomic_load_ztso<mode> (operands[0], operands[1],
+					   operands[2]));
+  DONE;
+})
+
+(define_expand "atomic_store<mode>"
+  [(match_operand:GPR 0 "memory_operand")
+   (match_operand:GPR 1 "reg_or_0_operand")
+   (match_operand:SI 2 "const_int_operand")] ;; model
   "TARGET_ATOMIC"
-  {
-    enum memmodel model = (enum memmodel) INTVAL (operands[2]);
-    model = memmodel_base (model);
-
-    if (model == MEMMODEL_SEQ_CST)
-      return "fence\trw,w\;"
-	     "s<amo>\t%z1,%0\;"
-	     "fence\trw,rw";
-    if (model == MEMMODEL_RELEASE)
-      return "fence\trw,w\;"
-	     "s<amo>\t%z1,%0";
-    else
-      return "s<amo>\t%z1,%0";
-  }
-  [(set_attr "type" "atomic")
-   (set (attr "length") (const_int 12))])
+{
+  if (TARGET_ZTSO)
+    emit_insn (gen_atomic_store_rvwmo<mode> (operands[0], operands[1],
+					     operands[2]));
+  else
+    emit_insn (gen_atomic_store_ztso<mode> (operands[0], operands[1],
+					    operands[2]));
+  DONE;
+})
 
 (define_insn "atomic_<atomic_optab><mode>"
   [(set (match_operand:GPR 0 "memory_operand" "+A")
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-1.c
new file mode 100644
index 00000000000..a7097e9aab9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-1.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	amoadd\.w\tzero,a1,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-2.c
new file mode 100644
index 00000000000..8e993903439
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-2.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that atomic op mappings the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	amoadd\.w\tzero,a1,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-3.c
new file mode 100644
index 00000000000..e1cd209fcb2
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-3.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	amoadd\.w\tzero,a1,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_RELEASE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-4.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-4.c
new file mode 100644
index 00000000000..484d11eb100
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-4.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	amoadd\.w\tzero,a1,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_ACQ_REL);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-5.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-5.c
new file mode 100644
index 00000000000..2117283fd0d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-amo-add-5.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	amoadd\.w\tzero,a1,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_SEQ_CST);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-1.c
new file mode 100644
index 00000000000..03247c632fd
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-1.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-2.c
new file mode 100644
index 00000000000..46356747067
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-2.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_CONSUME, __ATOMIC_CONSUME);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-3.c
new file mode 100644
index 00000000000..cf252dba6ec
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-3.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-4.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-4.c
new file mode 100644
index 00000000000..9a0c86569b5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-4.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-5.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-5.c
new file mode 100644
index 00000000000..2f77ffe8427
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-5.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w.aqrl\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-6.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-6.c
new file mode 100644
index 00000000000..0d0bf26d95a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-6.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_RELEASE, __ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-7.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-7.c
new file mode 100644
index 00000000000..aa2de9393e1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-compare-exchange-7.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that compare exchange mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w.aqrl\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (int bar, int baz, int qux)
+{
+  __atomic_compare_exchange_n(&bar, &baz, qux, 1, __ATOMIC_SEQ_CST, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-1.c
new file mode 100644
index 00000000000..a9ba4a252e7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-1.c
@@ -0,0 +1,14 @@ 
+/* { dg-do compile } */
+/* Verify that fence mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	ret
+*/
+void foo()
+{
+  __atomic_thread_fence(__ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-2.c
new file mode 100644
index 00000000000..9b083431d1a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-2.c
@@ -0,0 +1,14 @@ 
+/* { dg-do compile } */
+/* Verify that fence mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	ret
+*/
+void foo()
+{
+  __atomic_thread_fence(__ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-3.c
new file mode 100644
index 00000000000..db21ee6f031
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-3.c
@@ -0,0 +1,14 @@ 
+/* { dg-do compile } */
+/* Verify that fence mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	ret
+*/
+void foo()
+{
+  __atomic_thread_fence(__ATOMIC_RELEASE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-4.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-4.c
new file mode 100644
index 00000000000..d5bc61b57e8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-4.c
@@ -0,0 +1,14 @@ 
+/* { dg-do compile } */
+/* Verify that fence mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	ret
+*/
+void foo()
+{
+  __atomic_thread_fence(__ATOMIC_ACQ_REL);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-5.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-5.c
new file mode 100644
index 00000000000..e5e5f4e5012
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-fence-5.c
@@ -0,0 +1,15 @@ 
+/* { dg-do compile } */
+/* Verify that fence mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	fence\trw,rw
+**	ret
+*/
+void foo()
+{
+  __atomic_thread_fence(__ATOMIC_SEQ_CST);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-1.c
new file mode 100644
index 00000000000..be5f8d310a4
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-1.c
@@ -0,0 +1,16 @@ 
+/* { dg-do compile } */
+/* Verify that load mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	lw\ta[0-9]+,0\(a0\)
+**	sw\ta[0-9]+,0\(a1\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_load(bar, baz, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-2.c
new file mode 100644
index 00000000000..30583841648
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-2.c
@@ -0,0 +1,16 @@ 
+/* { dg-do compile } */
+/* Verify that load mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	lw\ta[0-9]+,0\(a0\)
+**	sw\ta[0-9]+,0\(a1\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_load(bar, baz, __ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-3.c
new file mode 100644
index 00000000000..c1c3b57bfe5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-load-3.c
@@ -0,0 +1,17 @@ 
+/* { dg-do compile } */
+/* Verify that load mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	fence\trw,rw
+**	lw\ta[0-9]+,0\(a0\)
+**	sw\ta[0-9]+,0\(a1\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_load(bar, baz, __ATOMIC_SEQ_CST);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-1.c
new file mode 100644
index 00000000000..f5519db22f7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-1.c
@@ -0,0 +1,16 @@ 
+/* { dg-do compile } */
+/* Verify that store mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	lw\ta[0-9]+,0\(a1\)
+**	sw\ta[0-9]+,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_store(bar, baz, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-2.c
new file mode 100644
index 00000000000..440f407e66a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-2.c
@@ -0,0 +1,16 @@ 
+/* { dg-do compile } */
+/* Verify that store mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	lw\ta[0-9]+,0\(a1\)
+**	sw\ta[0-9]+,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_store(bar, baz, __ATOMIC_RELEASE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-3.c
new file mode 100644
index 00000000000..b99abe1c4f6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-store-3.c
@@ -0,0 +1,16 @@ 
+/* { dg-do compile } */
+/* Verify that store mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso -O3" } */
+/* { dg-skip-if "" { *-*-* } { "-g" "-flto"} } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+/*
+** foo:
+**	lw\ta[0-9]+,0\(a1\)
+**	sw\ta[0-9]+,0\(a0\)
+**	ret
+*/
+void foo (int* bar, int* baz)
+{
+  __atomic_store(bar, baz, __ATOMIC_SEQ_CST);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-1.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-1.c
new file mode 100644
index 00000000000..e187a6b7335
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-1.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that subword atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (short* bar, short* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_RELAXED);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-2.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-2.c
new file mode 100644
index 00000000000..1e0867624ba
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-2.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that subword atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (short* bar, short* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_ACQUIRE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-3.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-3.c
new file mode 100644
index 00000000000..4b2ef967375
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-3.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that subword atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (short* bar, short* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_RELEASE);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-4.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-4.c
new file mode 100644
index 00000000000..87e5356f0a7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-4.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that subword atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (short* bar, short* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_ACQ_REL);
+}
diff --git a/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-5.c b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-5.c
new file mode 100644
index 00000000000..e8706809fe6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/riscv/amo-table-ztso-subword-amo-add-5.c
@@ -0,0 +1,10 @@ 
+/* { dg-do compile } */
+/* Verify that subword atomic op mappings match the Ztso suggested mapping.  */
+/* { dg-options "-march=rv64id_ztso" } */
+/* { dg-final { scan-assembler-times "lr.w.aqrl\t" 1 } } */
+/* { dg-final { scan-assembler-times "sc.w\t" 1 } } */
+
+void foo (short* bar, short* baz)
+{
+  __atomic_add_fetch(bar, baz, __ATOMIC_SEQ_CST);
+}