From patchwork Thu Sep 9 17:06:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guillermo E. Martinez" X-Patchwork-Id: 1526249 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=BD3jYs3I; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4H555z3Txxz9sR4 for ; Fri, 10 Sep 2021 03:07:18 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id EF83F384B0D2 for ; Thu, 9 Sep 2021 17:07:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EF83F384B0D2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1631207236; bh=ZS+uQf/VSsrmjb7qOCrfI5xOdchfUUmbibOUvV3Ihfg=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=BD3jYs3IQKD+DEdaDYGWjKw5ST3K0utb9Igvc7Ey0CURJ9877RkTCz0QcT0f2x5Mn PXOr5BPx7v0Bnm/KqOFtFu7o8wkQup9hCeb+k3Fi+R5Mujvlly5c0bYk3Yb5tHzw0b 1Qt5eY9xKeKt/KuK0Rm5r8x2aYiuW0rFF7mpl+Ww= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id 9132D3858C2C for ; Thu, 9 Sep 2021 17:06:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 9132D3858C2C Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 189G5Cdj011007 for ; Thu, 9 Sep 2021 17:06:53 GMT Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3ayfrps9uk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 09 Sep 2021 17:06:53 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 189H0vMu176487 for ; Thu, 9 Sep 2021 17:06:51 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2049.outbound.protection.outlook.com [104.47.66.49]) by userp3030.oracle.com with ESMTP id 3axcq39036-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 09 Sep 2021 17:06:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CxM+gcghEC8GfqJm4u5J1qqLDwmUtmSjAgH0/ss/Xr8Pz98aVFhHgg92X1H0Wq7FdijQpniCUDstAKHhEe3jxsIZieJiZEAad1e+MLTbT933vIPRKuEUtkmA1pfQtT+oK6T7cDfMG3sEqSuyFmk+0/H3FEskyQe6wcvFHYBE6h771qBHbqgeJsmB392G8lcrxfJwAbn7Bd13nFymmeyLIv/Pubxb0SbVGnwGCs6y59pZVJSrdjFvDGO5pRQrMawiRLa58X839d7eJ2+Ximm+DvxoS++XgRTVjpGmHhHwSFLRkdHSWFtXglsx8dJbjXffk8vt1QN/h5FiX7P6voFILA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ZS+uQf/VSsrmjb7qOCrfI5xOdchfUUmbibOUvV3Ihfg=; b=Yt4F+/dJOuxRsNYZAM2G2uKNZjP1U3EbLL1BO5yVBY7fDwW9RKAIMXOQntWDmkb7kxlvsFYcErwh7HtX+xe+w6IRJ1l+SH0J8kRomJGfDxygOW0L7sDa1ETaiTmN5e/AXBT3cHzKJCRjovTkivftUYrr3r+LZGYKxu2jEzy6qzi3rDIg8HzVf5IVlVVekZQrC2qtMZwXFo6/ufrFRRWVC2eRvbR1+IwHdGexBpgkNXEF31ESKPRxHapgqDuNr8exJtzB6KhMCPh+UpvDqLdzS6WvEwvRZFznQgI9QSYRDJg+g4iK7ngZ1mFCjU1sqZVQvgmVSVeYavZW+SDDFEiooA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none Received: from BL0PR10MB2852.namprd10.prod.outlook.com (2603:10b6:208:76::22) by BLAPR10MB5041.namprd10.prod.outlook.com (2603:10b6:208:30e::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Thu, 9 Sep 2021 17:06:49 +0000 Received: from BL0PR10MB2852.namprd10.prod.outlook.com ([fe80::3dec:9a58:e63e:c452]) by BL0PR10MB2852.namprd10.prod.outlook.com ([fe80::3dec:9a58:e63e:c452%6]) with mapi id 15.20.4478.026; Thu, 9 Sep 2021 17:06:49 +0000 To: gcc-patches@gcc.gnu.org Subject: [PATCH] bpf: Add support to eBPF atomic operations Date: Thu, 9 Sep 2021 12:06:36 -0500 Message-Id: <20210909170636.365036-1-guillermo.e.martinez@oracle.com> X-Mailer: git-send-email 2.30.2 X-ClientProxiedBy: BYAPR08CA0002.namprd08.prod.outlook.com (2603:10b6:a03:100::15) To BL0PR10MB2852.namprd10.prod.outlook.com (2603:10b6:208:76::22) MIME-Version: 1.0 Received: from sali.us.oracle.com (2606:b400:8004:44::1d) by BYAPR08CA0002.namprd08.prod.outlook.com (2603:10b6:a03:100::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.15 via Frontend Transport; Thu, 9 Sep 2021 17:06:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1bc88d31-a97c-4d01-d534-08d973b4361b X-MS-TrafficTypeDiagnostic: BLAPR10MB5041: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cmZJbOQGB0ProlEW4Pf7YXzHXDcSl0wYZAC5y234f2+gxOjhOBoo261PLOGq2FRTiUzJQ2WXto7laIncc9KD2/+Wp+0gEZpS29BoJVC/UgAcFlFA4m5IDZfJDRtYf+3Qv9NjKPDufmQ0vL78GZnxhYqwqIQmyDLnEWYMova2ldGJqUF/cRi8c2s0MOu6/HDQqhnY628JfArUT44QlXdzaZCzuyY1tGSXw+8TCpfF1oDzGRziCuiGEWq1gJ1KZBPbA6Jk/UFTMI66QlJOy8gYeWUMs09LTJCho+wzu7HDBnxLSagcr8cHika0qBR7okmUdaa63IhwvF2K8AtRoHMNhqkgdIsyGmJ+APHFHlO1QDjkWCTFMie/0FY3bQSSdKabMADbZCMco8STq5h7AdWHEixfrQ3G2N66VNj+pFrCbDkpWY2iiJIAAHFWvUU/Qv5+5EtWBJQzGSD4xk/+NH8lJ8Hdt/b6t9JG8XKdk9vpz5eJYVjJTANG3x5c1jqeB9Xr+qoEJlezMZB3aTkfnzztTHO68fYfpS0dW3JbUf5DASmZnlrWF6/37h3mBVsSAem2hbwGm66lAU3HbS14wTPXBs6pCNEEJD1iEBh+rMqt5MvP5qjUjI9qPjGEQIlmkgHIHFqHid1lIFspIUToHbBd8UBkwqd+MyOhvxeiZmJdSzRpmCEbmBIblZPdTiQaY/4XyiZaGVT7LZ/QNjCOkabnSa3ONi0L9f+9wrHR4UjgjL24IcZbZ4g4riasY6Io+SFj X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR10MB2852.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(83380400001)(316002)(52116002)(36756003)(107886003)(86362001)(103116003)(38100700002)(30864003)(6916009)(8936002)(5660300002)(186003)(966005)(8676002)(66556008)(6666004)(508600001)(66476007)(66946007)(2616005)(7696005)(2906002)(4326008)(1076003)(6486002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ek89gWiZj5NxJ7CdEC8f6JyNZLjQIHmexq1MSXc8UqafPHof2dSwyrf8SLxZ5mlI+Link9aTXW2Pwrdlk7a5sz56Rs+4y/zs7fIfi7d2UwSsvJAPIWkxu5IsrZnn6ZypcPC4q9D5cMSkxeyTolHrtI77uOb54QPcY1ILw47FS70cLG9TMaLpEMV36gmPwQ6iy6MX+LDcPdLFbtXMeUEvNNGMh7K80Kgci4FDoH+vz5u3puUKBz15IUGChWPMqKejo9oPYIpAvrqgJqK8mJbroMBokpjYfbfDfow6ks/hzbxoBnRexHBs7ZE8aJ4nCsrgplsRUo6PEHtgojCzpnvVz3AD6Esegaf4Wl0rqIPsxL1OF36oT5kLSBLYlAZp9lZ/3feqy+RpMCHVndU+nU4iOw7xyJIU3phhU/uhQpWSlzh8I9Gy24RnrJnlgS7UHv37krNKpMki727Ae34Csxgrd+9wJYiM0tv0UCaFBIVBxDMhW6lyi2ETS2s1MtLzLY0tK5ZDfrWx5RDNucW2a/iGMmtph7TfXaV7Ha3In+mOwyd3nqJDOOWxTp/o6vki8SjymL5Weyt3clIIftelpw+xU5PqdZlXV9onMP9zcVyOB6bePm98MbMtNlML7NZmERvvDKyYu3OygPJjqpnkbMoLg/TqukJN3PzOiNI4IjSoNtJJTqy6uhS3IaIGalTnRruB7tZuRxne88yPAcv9i8hSXcyWj0RkyiKBgInQENjkpOzQSBejyGlDq2U1jtdsj1zNPVNF2eLMMabdoW5Z7MlG8i1bRhWQTBSseSUecxKwofDhKal3x2zF6+h1yQ45ClbiUC/UWcol36wx86DSkouTJNSaqY0R+1PnR11BqS9M4+TWYe/BUUQnqbUfoh54p7HLU5whrX7Vs6DVH3IQTFS2txkT73q8RNqaCOL9qYxaSZPiGEGRFI96nJCq7gMkWCy8XIi+2N9Hkf8V1yIRM2S8qo7/AKu6/e/BSCOcGCWqGLeNTN0aGzC7Sj+Y5vLJxkbTdNar1YUuV9Y6lr5gWq3v55QK26qFSoZOTgjIlsxcW6yH+N/1fPLDGmJFOgBc7PiSTUO8eGmcOhrMMs2MSBR6R7OIwHR7NAxoqfRIRgW2D4GOYyimLCF1RAuok+izh2ciqoTPxGSqtYlCofiJk1+iMTV6HprwdZIqkkpyxmHK9JjySXMUp8jYknb7M6HF6FLvy3mLP9gzfXCBXHIh77+s7TpNdkzaeJAg/hhT8LDDyUvkLEfIDusfFCmUzjTTHfZh1busmTf67ZL0PG7wQXob64j4uB40pdkcneJt+ZdOgs8Ws7yGnBwlrh4VQK6VMswf5QCwcQsSSuPGgVZy7WfSSw== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bc88d31-a97c-4d01-d534-08d973b4361b X-MS-Exchange-CrossTenant-AuthSource: BL0PR10MB2852.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2021 17:06:49.0795 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8IEU+BfaB0iAD6eG6YASrf0FFnhmKhVGb4yfZAs/i7yVtiZiQRaNuc+1YE3toAJe6i9NueqMAJTtM/qoSnnVNkvWpK307e3Z7FknxwNiU+0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5041 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10102 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109030001 definitions=main-2109090106 X-Proofpoint-GUID: Pua8w6byo32su5rig1mUbadzCol3GJ7o X-Proofpoint-ORIG-GUID: Pua8w6byo32su5rig1mUbadzCol3GJ7o X-Spam-Status: No, score=-13.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "Guillermo E. Martinez via Gcc-patches" From: "Guillermo E. Martinez" Reply-To: "Guillermo E. Martinez" Cc: "Guillermo E. Martinez" Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" Hello people, This patch add support for atomics operations in eBPF target using the gcc built-in functions: __atomic__fetch __atomic_fetch_ Please if you have comments, don't hesitate to let me know. Kinds Regards, Guillermo eBPF add support for basic atomic operations, the following gcc built-in functions are implemented for bpf target using both: 32 and 64 bits data types: __atomic_fetch_add __atomic_fetch_sub __atomic_fetch_and __atomic_fetch_xor __atomic_fetch_or __atomic_exchange __atomic_compare_exchange_n Also calls to __atomic__fetch are fully supported. New define instructions were added to bpf.md along with its tests for gcc atomic built-in functions. Those operations are fully compliant with: https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html https://www.kernel.org/doc/Documentation/networking/filter.rst This support depends of two previous submissions in CGEN and binutils-gdb projects: https://sourceware.org/pipermail/cgen/2021q3/002774.html https://sourceware.org/pipermail/binutils/2021-August/117798.html gcc/ * config/bpf/bpf.md: Add defines for atomic instructions. gcc/testsuite/ * gcc.target/bpf/atomic-compare-exchange.c: New test. * gcc.target/bpf/atomic-exchange.c: Likewise. * gcc.target/bpf/atomic-fetch-add.c: Likewise. * gcc.target/bpf/atomic-fetch-and.c: Likewise. * gcc.target/bpf/atomic-fetch-or.c: Likewise. * gcc.target/bpf/atomic-fetch-sub.c: Likewise. * gcc.target/bpf/atomic-fetch-xor.c: Likewise. --- gcc/config/bpf/bpf.h | 8 +- gcc/config/bpf/bpf.md | 122 ++++++++++++++++-- gcc/config/bpf/constraints.md | 3 + .../gcc.target/bpf/atomic-compare-exchange.c | 28 ++++ .../gcc.target/bpf/atomic-exchange.c | 19 +++ .../gcc.target/bpf/atomic-fetch-add.c | 26 ++++ .../gcc.target/bpf/atomic-fetch-and.c | 25 ++++ .../gcc.target/bpf/atomic-fetch-or.c | 25 ++++ .../gcc.target/bpf/atomic-fetch-sub.c | 25 ++++ .../gcc.target/bpf/atomic-fetch-xor.c | 25 ++++ 10 files changed, 292 insertions(+), 14 deletions(-) create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-compare-exchange.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-exchange.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-fetch-add.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-fetch-and.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-fetch-or.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-fetch-sub.c create mode 100644 gcc/testsuite/gcc.target/bpf/atomic-fetch-xor.c diff --git a/gcc/config/bpf/bpf.h b/gcc/config/bpf/bpf.h index 82be0c3e190..f8d0f7fb7dd 100644 --- a/gcc/config/bpf/bpf.h +++ b/gcc/config/bpf/bpf.h @@ -176,6 +176,7 @@ enum reg_class { NO_REGS, /* no registers in set. */ + R0, /* register r0. */ ALL_REGS, /* all registers. */ LIM_REG_CLASSES /* max value + 1. */ }; @@ -189,6 +190,7 @@ enum reg_class #define REG_CLASS_NAMES \ { \ "NO_REGS", \ + "R0", \ "ALL_REGS" \ } @@ -202,14 +204,16 @@ enum reg_class #define REG_CLASS_CONTENTS \ { \ 0x00000000, /* NO_REGS */ \ - 0x00000fff, /* ALL_REGS */ \ + 0x00000001, /* R0 */ \ + 0x00000fff, /* ALL_REGS */ \ } /* A C expression whose value is a register class containing hard register REGNO. In general there is more that one such class; choose a class which is "minimal", meaning that no smaller class also contains the register. */ -#define REGNO_REG_CLASS(REGNO) GENERAL_REGS +#define REGNO_REG_CLASS(REGNO) \ + ((REGNO) == 0 ? R0 : GENERAL_REGS) /* A macro whose definition is the name of the class to which a valid base register must belong. A base register is one used in diff --git a/gcc/config/bpf/bpf.md b/gcc/config/bpf/bpf.md index 03830cc250e..f5a4d2d02b4 100644 --- a/gcc/config/bpf/bpf.md +++ b/gcc/config/bpf/bpf.md @@ -25,6 +25,11 @@ (define_c_enum "unspec" [ UNSPEC_LDINDABS UNSPEC_XADD + UNSPEC_XAND + UNSPEC_XOR + UNSPEC_XXOR + UNSPEC_XCHG + UNSPEC_CMPXCHG ]) ;;;; Constants @@ -56,11 +61,10 @@ ;; st generic store instructions for immediates. ;; stx generic store instructions. ;; jmp jump instructions. -;; xadd atomic exchange-and-add instructions. ;; multi multiword sequence (or user asm statements). (define_attr "type" - "unknown,alu,alu32,end,ld,lddw,ldx,st,stx,jmp,xadd,multi" + "unknown,alu,alu32,end,ld,lddw,ldx,st,stx,jmp,multi" (const_string "unknown")) ;; Length of instruction in bytes. @@ -506,17 +510,111 @@ "ldabs\t%0" [(set_attr "type" "ld")]) -;;;; Atomic increments +;;;; Atomic operations (define_mode_iterator AMO [SI DI]) -(define_insn "atomic_add" - [(set (match_operand:AMO 0 "memory_operand" "+m") - (unspec_volatile:AMO - [(plus:AMO (match_dup 0) - (match_operand:AMO 1 "register_operand" "r")) - (match_operand:SI 2 "const_int_operand")] ;; Memory model. - UNSPEC_XADD))] +(define_insn "atomic_fetch_add" + [(set (match_operand:AMO 0 "register_operand" "=r") + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") + (match_operand:AMO 2 "nonmemory_operand" "0") + (match_operand:AMO 3 "const_int_operand")] ;; Memory model + UNSPEC_XADD))] + "" + "xadd\t%1,%0") + +(define_insn "atomic_fetch_and" + [(set (match_operand:AMO 0 "register_operand" "=r") + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") + (match_operand:AMO 2 "nonmemory_operand" "0") + (match_operand:AMO 3 "const_int_operand")] + UNSPEC_XAND))] + "" + "xand\t%1,%0") + +(define_insn "atomic_fetch_or" + [(set (match_operand:AMO 0 "register_operand" "=r") + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") + (match_operand:AMO 2 "nonmemory_operand" "0") + (match_operand:AMO 3 "const_int_operand")] + UNSPEC_XOR))] + "" + "xor\t%1,%0") + +(define_insn "atomic_fetch_xor" + [(set (match_operand:AMO 0 "register_operand" "=r") + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") + (match_operand:AMO 2 "nonmemory_operand" "0") + (match_operand:AMO 3 "const_int_operand")] + UNSPEC_XXOR))] + "" + "xxor\t%1,%0") + +(define_insn "atomic_exchange" + [(set (match_operand:AMO 0 "register_operand" "=r") + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") + (match_operand:AMO 2 "nonmemory_operand" "0") + (match_operand:AMO 3 "const_int_operand")] + UNSPEC_XCHG))] + "" + "xchg\t%1,%0") + +;; eBPF compare and exchange operation atomically compares the +;; value addressed by memory operand(%0) with _R0_(%1), so +;; there is a constraint to load the expected value in mentioned +;; register. If they match it is replaced with desired value(%3). +;; In either case, the value that was there before in %0 is +;; zero-extended and loaded back to R0. + +(define_expand "atomic_compare_and_swap" + [(match_operand:SI 0 "register_operand") ;; bool success + (match_operand:AMO 1 "register_operand") ;; old value + (match_operand:AMO 2 "memory_operand") ;; memory + (match_operand:AMO 3 "register_operand") ;; expected + (match_operand:AMO 4 "register_operand") ;; desired + (match_operand:SI 5 "const_int_operand") ;; is_weak (unused) + (match_operand:SI 6 "const_int_operand") ;; success model (unused) + (match_operand:SI 7 "const_int_operand")] ;; failure model (unused) "" - "xadd\t%0,%1" - [(set_attr "type" "xadd")]) +{ + emit_insn + (gen_atomic_compare_and_swap_1 + (operands[1], operands[2], operands[3], operands[4], operands[6])); + + /* Assume success operation, i.e memory operand + is matched with expected value. + */ + emit_move_insn (operands[0], const1_rtx); + rtx_code_label *success_label = gen_label_rtx (); + + /* At this point eBPF xcmp was executed, now we can ask + * for success exchange, and set suitable value for bool + * operand(%0) + */ + emit_cmp_and_jump_insns (operands[1], operands[3], EQ, 0, + GET_MODE (operands[1]), 1, success_label); + emit_move_insn (operands[0], const0_rtx); + + if (success_label) + { + emit_label (success_label); + LABEL_NUSES (success_label) = 1; + } + DONE; +}) + +(define_insn "atomic_compare_and_swap_1" + [(set (match_operand:AMO 0 "register_operand" "=t") ;; must be r0 + (unspec_volatile:AMO + [(match_operand:AMO 1 "memory_operand" "+o") ;; memory + (match_operand:AMO 2 "register_operand" "0") ;; expected + (match_operand:AMO 3 "register_operand" "r") ;; desired + (match_operand:SI 4 "const_int_operand")] ;; success (unused) + UNSPEC_CMPXCHG))] + "" + "xcmp\t%1,%3") diff --git a/gcc/config/bpf/constraints.md b/gcc/config/bpf/constraints.md index 66b7764d775..9606b6e150e 100644 --- a/gcc/config/bpf/constraints.md +++ b/gcc/config/bpf/constraints.md @@ -29,3 +29,6 @@ (define_constraint "S" "A constant call address." (match_code "const,symbol_ref,label_ref,const_int")) + +(define_register_constraint "t" "R0" + "Register r0") diff --git a/gcc/testsuite/gcc.target/bpf/atomic-compare-exchange.c b/gcc/testsuite/gcc.target/bpf/atomic-compare-exchange.c new file mode 100644 index 00000000000..d522697ec9e --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-compare-exchange.c @@ -0,0 +1,28 @@ +/* { dg-do compile } */ + +long val; +long ptr; +long expected; +long desired; + +void +foo () +{ + int done; + + done = __atomic_compare_exchange_n (&ptr, &expected, desired, + 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); + done = __atomic_compare_exchange_n ((int *)&ptr, (int *)&expected, + (int)desired, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); + + done = __atomic_compare_exchange (&ptr, &expected, &desired, + 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); + done = __atomic_compare_exchange ((int *)&ptr, (int *)&expected, + (int *)&desired, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); + + done = __sync_bool_compare_and_swap (&ptr, expected, desired); + done = __sync_bool_compare_and_swap ((int*)&ptr, expected, desired); +} + +/* { dg-final { scan-assembler "xcmpdw\t.*" } } */ +/* { dg-final { scan-assembler "xcmpw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-exchange.c b/gcc/testsuite/gcc.target/bpf/atomic-exchange.c new file mode 100644 index 00000000000..f4e60568f7f --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-exchange.c @@ -0,0 +1,19 @@ +/* { dg-do compile } */ + +long val; +long ptr; + +void +foo () +{ + long prev; + + __atomic_exchange(&ptr, &val, &prev, __ATOMIC_RELAXED); + prev = __atomic_exchange_n(&ptr, val, __ATOMIC_RELAXED); + + __atomic_exchange((int *)&ptr, (int *)&val, (int *)&prev, __ATOMIC_RELAXED); + prev = __atomic_exchange_n((int *)&ptr, (int)val, __ATOMIC_RELAXED); +} + +/* { dg-final { scan-assembler "xchgdw\t.*" } } */ +/* { dg-final { scan-assembler "xchgw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-fetch-add.c b/gcc/testsuite/gcc.target/bpf/atomic-fetch-add.c new file mode 100644 index 00000000000..2547396d2a1 --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-fetch-add.c @@ -0,0 +1,26 @@ +/* { dg-do compile } */ + +long delta; +long *val; + +void +foo () +{ + long k; + + k = __atomic_fetch_add (val, delta, __ATOMIC_RELAXED); + k = __atomic_fetch_add ((int*)val, delta, __ATOMIC_RELAXED); + + k = __atomic_add_fetch (val, delta, __ATOMIC_RELAXED); + k = __atomic_add_fetch ((int*)val, delta, __ATOMIC_RELAXED); + + k = __sync_fetch_and_add (val, delta, __ATOMIC_RELAXED); + k = __sync_fetch_and_add ((int*)val, delta, __ATOMIC_RELAXED); + + k = __sync_add_and_fetch (val, delta, __ATOMIC_RELAXED); + k = __sync_add_and_fetch ((int*)val, delta, __ATOMIC_RELAXED); + +} + +/* { dg-final { scan-assembler "xadddw\t.*" } } */ +/* { dg-final { scan-assembler "xaddw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-fetch-and.c b/gcc/testsuite/gcc.target/bpf/atomic-fetch-and.c new file mode 100644 index 00000000000..6cc05a824f6 --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-fetch-and.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ + +long mask; +long *val; + +void +foo () +{ + long k; + + k = __atomic_fetch_and (val, mask, __ATOMIC_RELAXED); + k = __atomic_fetch_and ((int*)val, mask, __ATOMIC_RELAXED); + + k = __atomic_and_fetch (val, mask, __ATOMIC_RELAXED); + k = __atomic_and_fetch ((int*)val, mask, __ATOMIC_RELAXED); + + k = __sync_fetch_and_and (val, mask, __ATOMIC_RELAXED); + k = __sync_fetch_and_and ((int*)val, mask, __ATOMIC_RELAXED); + + k = __sync_and_and_fetch (val, mask, __ATOMIC_RELAXED); + k = __sync_and_and_fetch ((int*)val, mask, __ATOMIC_RELAXED); +} + +/* { dg-final { scan-assembler "xanddw\t.*" } } */ +/* { dg-final { scan-assembler "xandw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-fetch-or.c b/gcc/testsuite/gcc.target/bpf/atomic-fetch-or.c new file mode 100644 index 00000000000..af9a999e02a --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-fetch-or.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ + +long bits; +long *val; + +void +foo () +{ + long k; + + k = __atomic_fetch_or (val, bits, __ATOMIC_RELAXED); + k = __atomic_fetch_or ((int *)val, bits, __ATOMIC_RELAXED); + + k = __atomic_or_fetch (val, bits, __ATOMIC_RELAXED); + k = __atomic_or_fetch ((int*)val, bits, __ATOMIC_RELAXED); + + k = __sync_fetch_and_or (val, bits, __ATOMIC_RELAXED); + k = __sync_fetch_and_or ((int*)val, bits, __ATOMIC_RELAXED); + + k = __sync_or_and_fetch (val, bits, __ATOMIC_RELAXED); + k = __sync_or_and_fetch ((int*)val, bits, __ATOMIC_RELAXED); +} + +/* { dg-final { scan-assembler "xordw\t.*" } } */ +/* { dg-final { scan-assembler "xorw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-fetch-sub.c b/gcc/testsuite/gcc.target/bpf/atomic-fetch-sub.c new file mode 100644 index 00000000000..6e6a16caa3c --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-fetch-sub.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ + +long delta; +long *val; + +void +foo () +{ + long k; + + k = __atomic_fetch_sub (val, delta, __ATOMIC_RELAXED); + k = __atomic_fetch_sub ((int*)val, delta, __ATOMIC_RELAXED); + + k = __atomic_sub_fetch (val, delta, __ATOMIC_RELAXED); + k = __atomic_sub_fetch ((int*)val, delta, __ATOMIC_RELAXED); + + k = __sync_fetch_and_sub (val, delta, __ATOMIC_RELAXED); + k = __sync_fetch_and_sub ((int*)val, delta, __ATOMIC_RELAXED); + + k = __sync_sub_and_fetch (val, delta, __ATOMIC_RELAXED); + k = __sync_sub_and_fetch ((int*)val, delta, __ATOMIC_RELAXED); +} + +/* { dg-final { scan-assembler "xadddw\t.*" } } */ +/* { dg-final { scan-assembler "xaddw\t.*" } } */ diff --git a/gcc/testsuite/gcc.target/bpf/atomic-fetch-xor.c b/gcc/testsuite/gcc.target/bpf/atomic-fetch-xor.c new file mode 100644 index 00000000000..433600395a6 --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/atomic-fetch-xor.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ + +long bits; +long *val; + +void +foo () +{ + long k; + + k = __atomic_fetch_xor (val, bits, __ATOMIC_RELAXED); + k = __atomic_fetch_xor ((int *)val, bits, __ATOMIC_RELAXED); + + k = __atomic_xor_fetch (val, bits, __ATOMIC_RELAXED); + k = __atomic_xor_fetch ((int*)val, bits, __ATOMIC_RELAXED); + + k = __sync_fetch_and_xor (val, bits, __ATOMIC_RELAXED); + k = __sync_fetch_and_xor ((int*)val, bits, __ATOMIC_RELAXED); + + k = __sync_xor_and_fetch (val, bits, __ATOMIC_RELAXED); + k = __sync_xor_and_fetch ((int*)val, bits, __ATOMIC_RELAXED); +} + +/* { dg-final { scan-assembler "xxordw\t.*" } } */ +/* { dg-final { scan-assembler "xxorw\t.*" } } */