From patchwork Tue Feb 20 22:56:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Faust X-Patchwork-Id: 1901754 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=DbjanMBl; dkim=pass (1024-bit key; unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com header.b=OeX5aoIr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TfZYN0kMYz20Qg for ; Wed, 21 Feb 2024 09:57:28 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E659C3858D3C for ; Tue, 20 Feb 2024 22:57:25 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id E9BCD3858D20 for ; Tue, 20 Feb 2024 22:56:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E9BCD3858D20 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=oracle.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E9BCD3858D20 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1708469823; cv=pass; b=H/TbHm92cdWI0+6xKUEoZ3zHiJXxayymOchWNDsfWbkAsCCi66NbLriHn3fWQh7rpS/WWIQUSfJFhwUf3/h3v/Mk8JfX8CpidiFx6JYYlgTbja+exMeCzsQTsB8oQ53B1S3Ng9viiadnJjhpXxt8Di72Iq6E+zBYOXZua4NZN+A= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1708469823; c=relaxed/simple; bh=3PAt9QUoyZgXj+5yyZFZtD1Tap4PjnTpfhczXlWHB1g=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=IZuspm1Bw8EbYFeDVcTkK8dG+7i1gyx8rGA8mbkbFJ+jss2Ay7XKPWUnAZ6MUUUlNzMDVHUD/rdx1NSD1fR1SvNL1YPjjnKv1wfME8ubsIqYWaG80baTETnJiDeaRuOmcG/xv2/bJ84mZd6O86IrZkIeIVl0JLFNSy7KlBNdgXY= ARC-Authentication-Results: i=2; server2.sourceware.org Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41KKTY0a003170 for ; Tue, 20 Feb 2024 22:56:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : content-transfer-encoding : content-type : mime-version; s=corp-2023-11-20; bh=A3HfyGYsviceWAVozoGVoFct1j0+9XddgUoUjZLKEm8=; b=DbjanMBlxio5kCbnb/CrpusS+HaM8ZcG+bhMfI3xcOJ89ruINOJjA2yHyztz5yMkSUBj OCXwIQiWLf/6ycMug9csVJqKW/yVywxrW3MyOcDcisVwg1TvzzwSFnC8Mcmb55q7j3lA VjU6Dr7xW+aQIaQlykFBxTxsM7d39ke+JP5gJPoax7eDS+6wT9kk1ObeytVy5ZAcMogV NEDFOw1MxW2H0dty9jHpWP6qSA9+0b2NxvoJGQeuXLbGNLBVmRBN7CA5WtibpR0glJy+ fxNTFjnfvylJs2qadohBCyxFAc4Dxc5Mp6wmqBexK84uJMlA5tEmUuzYE0z3uMPNdKZK cw== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3wanbvg47g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 20 Feb 2024 22:56:58 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41KMiEoD021006 for ; Tue, 20 Feb 2024 22:56:57 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2040.outbound.protection.outlook.com [104.47.66.40]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3wak88anhh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 20 Feb 2024 22:56:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ij9qHZtxCdreWXWT31VfCCwW9Igm2GwRyFXgGjWqXlczvNPSPXkxttE2pFIAJiSlXH532riFhK7GwV8MKEhPc74QV1Qx+aX+vM+tQNXSifVesIwGLFsmAFrTKOnEBbE8Mte8xVYEtlZFgDh7Pm4IMCbkwsNsJbpSRONJPPVNCbdv3g6cexK+vhpth/paCnL0ed/rDwn5gWxK3YMJ6eD1iDdvD7vudRe4NPyyeU6OzQOWo1vailQOKC1ssScJDfWjsOMMETELKJHRIv6B6Shv2Y/Kngxr/pS1IVFCRmD8Z93PVMvJTsbJFc0XUnwGc15BJOAIYVM7zUOd8SLwNsGIlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=A3HfyGYsviceWAVozoGVoFct1j0+9XddgUoUjZLKEm8=; b=eOR1EVdPzlSs9YCtD6AyoSJXfQbQG5gkQScBtN1oRmh3bwMuYC9runxmDpRMDj3OXo5mRpk/lenNksIFJN+1IbWA+llpmZILwhDr4faWGAMqM71JwHR4fUg1fAncG18me7uRS0NTIkeVpA6uIa5EtAbqICHkeLzLiH1hCjgwS8y6DtTzgggp9BUz1q8KR70IN+umzKthRcm3GVOI09JCnrrqLMZx9IgwXYGfbYhw8G1ETGYZepJX9hgy+wGDuQv1KqRrqcLq4l81AETVl0s82OSzfdcriPGKlXAqCSdTw+4LotYot0jqrofxS1Xae01TljEnaAdx/dpywwsk0LHAPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A3HfyGYsviceWAVozoGVoFct1j0+9XddgUoUjZLKEm8=; b=OeX5aoIrrrd+TJvu/H+CebnUZyICIYO34F4GWCIgBjxxb8d6lzauAAfIpJUVTeqPZ1QpGV62KjxIyj3/D66xt1zk61vri6thZpYl8RURwm0yjop4ODHjKXBAJLaVIOHsaKaBFHoG9elyHSknGQ+yShmn7rnqlX+RrDMxz/jldLw= Received: from DS0PR10MB7953.namprd10.prod.outlook.com (2603:10b6:8:1a1::22) by IA0PR10MB7641.namprd10.prod.outlook.com (2603:10b6:208:484::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.39; Tue, 20 Feb 2024 22:56:55 +0000 Received: from DS0PR10MB7953.namprd10.prod.outlook.com ([fe80::9722:fdc5:e685:633]) by DS0PR10MB7953.namprd10.prod.outlook.com ([fe80::9722:fdc5:e685:633%7]) with mapi id 15.20.7292.036; Tue, 20 Feb 2024 22:56:55 +0000 From: David Faust To: gcc-patches@gcc.gnu.org Cc: jose.marchesi@oracle.com Subject: [PATCH v3] bpf: add inline memmove and memcpy expansion Date: Tue, 20 Feb 2024 14:56:51 -0800 Message-ID: <20240220225651.24206-1-david.faust@oracle.com> X-Mailer: git-send-email 2.41.0 X-ClientProxiedBy: PH0PR07CA0097.namprd07.prod.outlook.com (2603:10b6:510:4::12) To DS0PR10MB7953.namprd10.prod.outlook.com (2603:10b6:8:1a1::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7953:EE_|IA0PR10MB7641:EE_ X-MS-Office365-Filtering-Correlation-Id: 434056f1-8b48-4d73-8f8b-08dc32673c41 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vUk4UC6UhRVp9jED9MzoRoWyJyUxyP/1c9WuiNeaKH7YXOjcoYlQEs0FPwMLQkQXsmz+YSYEfRBx8VOXEWPVIQb2jCoRHQrSA2DCs6j+Untm4whUCd1AoFV8PS38NTWEIM2hT6ZQR8bIlr7n+a8B1qIzQlzE9ndXB3EAfUTw5Xrl1nGXPTZElJ1i+oysJn6CdIIQtO4kGmV3mSRnOCkQS5QfjxvWy8RPlxtPKee/s/zGNLYreaGGXpUgB7F8sztD3sDKHPli0j+l6m9ZYy6bvcslgPO7CDugR0ZEu3/rC0QT5LTQL/ni2Ys1I/UiNITkynEU5dGSsZJowIZNUHCQs+760Xxy5MEvBubMIzUlpgqpnh0xLMObI6UrojDJl2BHi0oi2KSzV3kUm/gRTzSSosCL7d1XdlROrQqc/huO7OS2wzs+PHDWaODIxbAh6nHXunRj9BRw7J9hgE0W/1Y13P/hE9D/M9hKxC6q8Kyzeofz3hpDHOXpGVWlhhTTuQQeyePOMp+uKZxPRFWHR2ErxR4eNUqCXPKevmJTWGCUyhFKEakmWhBmF5dYCxLrgDSo X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR10MB7953.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(230273577357003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: G/neWM+nJ3ev9/76O6UcENrmcJ+r5hiuh6YcFjjuQGfvaTNhzxYm+EmTCi/EBkDlWS3+XJIW2vY8B4tx+mwdDDLSUdDEB4S12Js4T+wtW8jxSxLRBH7fstNHe/2N8AEPO/ZQG/9kE7M3M3h3H+sdyzqs8CRHwXQLQIgoUkfNjITRkjLzlPSq3u9CDLFhc0DzgC3OCYJ+qeZiiwEhvEa85U7nC4IB4gNtImv6on6nRhdapkLo9vW7IPTCw+T5cL6ofpsdP9wzd7STLcUXQN2y17ULxdr9LHTvgpUwu1dpQTZwWxiZjLnB9glpPap5tD4Duz5gb+8oi8SMPV1N5UfgOuZDRi0dHmp44fSubFKepIKSLxe4DdAnMNXZGvwoxcnesPGarOm8m51NNZij/hf5wgb+Vl2Dc8/l0RzXzoZDk1/DKamYZSeHEz5cPgoGh0JiQU+xTUZFla/QyDDz5SgihUY2JjHyT1uyP2VtfpwEMLO4q5kMVCxi6oCbO/p8xpKswhkLEBhf/uKGgvnRM4iFl4czEJdcIPq9S4LvcEaUmQXROU6dZveOkkq6yd9iAebmswRK0Th/arMFQhv1GEeYWGcfLVbtp81rI2yEtHxnwzKcRUxfDIYKC2i800xYJXsJhHwrQ/Ur/oh2fjz/bJEl4dxYpZB31DuDbvHei3KxTTIno5AN4gzcna+2YMm7/yvo1DwajpuTM0T3DbMc4M1T6AP13y8galeMXkLDyirL9uj/1mPo8SwKqiO97TzMzFMeZvkdn7eyPJ6I1yoc/MGrUY+6X5ECpUA8snYaCsXoj1uxeP/CR/8KSAKL0U05JdFcDqUmXefzuhu24cdrIlWAZhCxAj3eARtEDzFrX2CMOEbjsOClzlQ+/HG6zMxxEI+ozsZOcE0Rj3NQp+lpPby9mFtNg/cItPanny2jEKhFQSSKUHELrifp9YlQ4LdHXTfAk+DHneUSkvEMCW2gRN9ZkkEJKxvui8JY5W/Di3naHGkI2Z9j2oFB06amj+1FvLrrH5dnxQm8K7EN1Fiq58O80ltJ9WMQHn4KF+zmUvmqD3ZwJDltUsLCMUWxSLNwMU2zqHDuFy/HKCz3ui29XDp8i9EZzPO9nKRdXN/5FVKy0qtYk5147hTuRBK2UTVwQNkgzfQiirwVl56INkw/3wvEAsZNQDW4GARBt2DuWqWvFKDd+pu3JRRRIizUdGJlnf3TPX59ndKkNsOydtKBO2m0hAXuwC6yzq4V8clkwif7jSPzdEE8J1iAJltCQsvjFkCUi5/wvf496yVpTmyCU0hNxaLf8B3jPuRGny598rc1NHSeKJ6yDZQRckyc9/Og/Pkbwe7CWAjh9P5B1CRWxUIPPhU8XrAh8tlXSGLA2norQ4Pl2rh5y16cpESSIn0BxpNg0PeFJ2Cw5hRXeBWbk/U2NaGXaCy4jaViVYfZzLm7hHaSnrYEtWGEdOrLUiYVibvSynLoZZnXe7f6scUELoT/qjoda6OyEwj3U2VXYyct7pKsv3H9+Qc6tlQbw2+H29eFiacBIMsrTl/8uU3U9VRyHyZaQpuNRxzcDwmXqbAZJKogallOE4kjGrXfcphiHmZG X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Ivm7omfqzp0P27wXltNJ6BtSg2XV4wtdIjzqWUIO294IfiliMoH0Toj/kfltNxJpzy9rxAOMJ/m0WNNZNXm16x+J0cRvUhCbzXsrCf/d/twQzxjfQ8H9j8QjTVqQhEjk++v8wKJ4LRuwyEXgcQ4FB+hskbuh3ouJKuKqJAs7KHxe7JOQfKGLPX/FveMI9BsDuvpsolklPMp5845XbzFt6pk5XDFVHANEaVJ4d4ZNzfQ27TPJ/QOk6l9f0DzjtkpatX3jhknbpXxHgH2zQUx3yZ3mH5Zr8En12Mdf1UDxDZo0MZ+DVR4HaeeLMuxv1blC9pi6F4i6uhRY5lixiF+1J/4pv7NQxolMLdxt70gGm9cYH0e+9Bo6c8ywm9EC3uTkwduCSf5jm/aQF+aETg72rOFbAfM/eLIVg+eaBKd51v15+h1ZuCaO2WEhWLiuLTJv9mnLGyeNl2UHiOuj/p8biRWzRxVi+B/GuadkTNar6jtAOnMdi7d2gOCU1oBroU8BUoMRd65mA6x7jRVhmFsQSFC6Uo9tIvfL8Y0LUi+iWLzjhQRb0N348I1g8ShqT8YAQIbfu03xuhIBJiQExUNfUiqqXYvnp5YrbRcnZ6+ecHU= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 434056f1-8b48-4d73-8f8b-08dc32673c41 X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7953.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 22:56:55.4857 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EhYlKg7TeZr74y2qRuPzkgeYyfDweJ25uDURW0wU63BbkCtouLMehnwUSW+sq+UTsA2koi9jq78Ermyq9pdMUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB7641 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-20_06,2024-02-20_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402200164 X-Proofpoint-GUID: hxzL3ia6yrsxppPaK3kBSMAgswzjTqUX X-Proofpoint-ORIG-GUID: hxzL3ia6yrsxppPaK3kBSMAgswzjTqUX X-Spam-Status: No, score=-13.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org [Changes from v2: - Fix incorrectly passing a location instead of OPT_W* for warning (). - Reword warning/error message and test accordingly. ] [Changes from v1: Jose's review comments, all of which I agree with. - Fix 'implments' typo in commit message. - Change check that alignment is CONST_INT to gcc_assert (). - Change default case in alignment switch to gcc_unreachable (). - Reword error message for non-constant size memmove/memcpy, and update test for the error accordingly. - Delete CPYMEM_EXPAND_ERR macro, since it was now only used in one place. ] BPF programs are not typically linked, which means we cannot fall back on library calls to implement __builtin_{memmove,memcpy} and should always expand them inline if possible. GCC already successfully expands these builtins inline in many cases, but failed to do so for a few for simple cases involving overlapping memmove in the kernel BPF selftests and was instead emitting a libcall. This patch implements a simple inline expansion of memcpy and memmove in the BPF backend in a verifier-friendly way, with the caveat that the size must be an integer constant, which is also required by clang. Tested for bpf-unknown-none on x86_64-linux-gnu host. Also tested against the BPF verifier by compiling and loading a test program with overlapping memmove (essentially the memmove-1.c test) which failed before due to a libcall, and now successfully loads and passes the verifier. gcc/ * config/bpf/bpf-protos.h (bpf_expand_cpymem): New. * config/bpf/bpf.cc: (emit_move_loop, bpf_expand_cpymem): New. * config/bpf/bpf.md: (cpymemdi, movmemdi): New define_expands. gcc/testsuite/ * gcc.target/bpf/memcpy-1.c: New test. * gcc.target/bpf/memmove-1.c: New test. * gcc.target/bpf/memmove-2.c: New test. --- gcc/config/bpf/bpf-protos.h | 2 + gcc/config/bpf/bpf.cc | 115 +++++++++++++++++++++++ gcc/config/bpf/bpf.md | 36 +++++++ gcc/testsuite/gcc.target/bpf/memcpy-1.c | 26 +++++ gcc/testsuite/gcc.target/bpf/memmove-1.c | 46 +++++++++ gcc/testsuite/gcc.target/bpf/memmove-2.c | 23 +++++ 6 files changed, 248 insertions(+) create mode 100644 gcc/testsuite/gcc.target/bpf/memcpy-1.c create mode 100644 gcc/testsuite/gcc.target/bpf/memmove-1.c create mode 100644 gcc/testsuite/gcc.target/bpf/memmove-2.c diff --git a/gcc/config/bpf/bpf-protos.h b/gcc/config/bpf/bpf-protos.h index 46d950bd990..366acb87ae4 100644 --- a/gcc/config/bpf/bpf-protos.h +++ b/gcc/config/bpf/bpf-protos.h @@ -35,4 +35,6 @@ const char *bpf_add_core_reloc (rtx *operands, const char *templ); class gimple_opt_pass; gimple_opt_pass *make_pass_lower_bpf_core (gcc::context *ctxt); +bool bpf_expand_cpymem (rtx *, bool); + #endif /* ! GCC_BPF_PROTOS_H */ diff --git a/gcc/config/bpf/bpf.cc b/gcc/config/bpf/bpf.cc index d6ca47eeecb..f9ac263613a 100644 --- a/gcc/config/bpf/bpf.cc +++ b/gcc/config/bpf/bpf.cc @@ -1184,6 +1184,121 @@ bpf_use_by_pieces_infrastructure_p (unsigned HOST_WIDE_INT size, #define TARGET_USE_BY_PIECES_INFRASTRUCTURE_P \ bpf_use_by_pieces_infrastructure_p +/* Helper for bpf_expand_cpymem. Emit an unrolled loop moving the bytes + from SRC to DST. */ + +static void +emit_move_loop (rtx src, rtx dst, machine_mode mode, int offset, int inc, + unsigned iters, unsigned remainder) +{ + rtx reg = gen_reg_rtx (mode); + + /* First copy in chunks as large as alignment permits. */ + for (unsigned int i = 0; i < iters; i++) + { + emit_move_insn (reg, adjust_address (src, mode, offset)); + emit_move_insn (adjust_address (dst, mode, offset), reg); + offset += inc; + } + + /* Handle remaining bytes which might be smaller than the chunks + used above. */ + if (remainder & 4) + { + emit_move_insn (reg, adjust_address (src, SImode, offset)); + emit_move_insn (adjust_address (dst, SImode, offset), reg); + offset += (inc < 0 ? -4 : 4); + remainder -= 4; + } + if (remainder & 2) + { + emit_move_insn (reg, adjust_address (src, HImode, offset)); + emit_move_insn (adjust_address (dst, HImode, offset), reg); + offset += (inc < 0 ? -2 : 2); + remainder -= 2; + } + if (remainder & 1) + { + emit_move_insn (reg, adjust_address (src, QImode, offset)); + emit_move_insn (adjust_address (dst, QImode, offset), reg); + } +} + +/* Expand cpymem/movmem, as from __builtin_memcpy/memmove. + OPERANDS are the same as the cpymem/movmem patterns. + IS_MOVE is true if this is a memmove, false for memcpy. + Return true if we successfully expanded, or false if we cannot + and must punt to a libcall. */ + +bool +bpf_expand_cpymem (rtx *operands, bool is_move) +{ + /* Size must be constant for this expansion to work. */ + if (!CONST_INT_P (operands[2])) + { + const char *name = is_move ? "memmove" : "memcpy"; + if (flag_building_libgcc) + warning (0, "could not inline call to %<__builtin_%s%>: " + "size must be constant", name); + else + error ("could not inline call to %<__builtin_%s%>: " + "size must be constant", name); + return false; + } + + /* Alignment is a CONST_INT. */ + gcc_assert (CONST_INT_P (operands[3])); + + rtx dst = operands[0]; + rtx src = operands[1]; + rtx size = operands[2]; + unsigned HOST_WIDE_INT size_bytes = UINTVAL (size); + unsigned align = UINTVAL (operands[3]); + enum machine_mode mode; + switch (align) + { + case 1: mode = QImode; break; + case 2: mode = HImode; break; + case 4: mode = SImode; break; + case 8: mode = DImode; break; + default: + gcc_unreachable (); + } + + unsigned iters = size_bytes >> ceil_log2 (align); + unsigned remainder = size_bytes & (align - 1); + + int inc = GET_MODE_SIZE (mode); + rtx_code_label *fwd_label, *done_label; + if (is_move) + { + /* For memmove, be careful of overlap. It is not a concern for memcpy. + To handle overlap, we check (at runtime) if SRC < DST, and if so do + the move "backwards" starting from SRC + SIZE. */ + fwd_label = gen_label_rtx (); + done_label = gen_label_rtx (); + + rtx dst_addr = copy_to_mode_reg (Pmode, XEXP (dst, 0)); + rtx src_addr = copy_to_mode_reg (Pmode, XEXP (src, 0)); + emit_cmp_and_jump_insns (src_addr, dst_addr, GEU, NULL_RTX, Pmode, + true, fwd_label, profile_probability::even ()); + + /* Emit the "backwards" unrolled loop. */ + emit_move_loop (src, dst, mode, size_bytes, -inc, iters, remainder); + emit_jump_insn (gen_jump (done_label)); + emit_barrier (); + + emit_label (fwd_label); + } + + emit_move_loop (src, dst, mode, 0, inc, iters, remainder); + + if (is_move) + emit_label (done_label); + + return true; +} + /* Finally, build the GCC target. */ struct gcc_target targetm = TARGET_INITIALIZER; diff --git a/gcc/config/bpf/bpf.md b/gcc/config/bpf/bpf.md index 50df1aaa3e2..ca677bc6b50 100644 --- a/gcc/config/bpf/bpf.md +++ b/gcc/config/bpf/bpf.md @@ -627,4 +627,40 @@ (define_insn "ldabs" "{ldabs\t%0|r0 = *( *) skb[%0]}" [(set_attr "type" "ld")]) +;;; memmove and memcopy + +;; 0 is dst +;; 1 is src +;; 2 is size of copy in bytes +;; 3 is alignment + +(define_expand "cpymemdi" + [(match_operand:BLK 0 "memory_operand") + (match_operand:BLK 1 "memory_operand") + (match_operand:DI 2 "general_operand") + (match_operand:DI 3 "immediate_operand")] + "" +{ + if (bpf_expand_cpymem (operands, false)) + DONE; + FAIL; +}) + +;; 0 is dst +;; 1 is src +;; 2 is size of copy in bytes +;; 3 is alignment + +(define_expand "movmemdi" + [(match_operand:BLK 0 "memory_operand") + (match_operand:BLK 1 "memory_operand") + (match_operand:DI 2 "general_operand") + (match_operand:DI 3 "immediate_operand")] + "" +{ + if (bpf_expand_cpymem (operands, true)) + DONE; + FAIL; +}) + (include "atomic.md") diff --git a/gcc/testsuite/gcc.target/bpf/memcpy-1.c b/gcc/testsuite/gcc.target/bpf/memcpy-1.c new file mode 100644 index 00000000000..6c9707f24e8 --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/memcpy-1.c @@ -0,0 +1,26 @@ +/* Ensure memcpy is expanded inline rather than emitting a libcall. */ + +/* { dg-do compile } */ +/* { dg-options "-O2" } */ + +struct context { + unsigned int data; + unsigned int data_end; + unsigned int data_meta; + unsigned int ingress; + unsigned int queue_index; + unsigned int egress; +}; + +void +cpy_1(struct context *ctx) +{ + void *data = (void *)(long)ctx->data; + char *dest; + dest = data; + dest += 16; + + __builtin_memcpy (dest, data, 8); +} + +/* { dg-final { scan-assembler-times "call" 0 } } */ diff --git a/gcc/testsuite/gcc.target/bpf/memmove-1.c b/gcc/testsuite/gcc.target/bpf/memmove-1.c new file mode 100644 index 00000000000..3b8ba82639e --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/memmove-1.c @@ -0,0 +1,46 @@ +/* Ensure memmove is expanded inline rather than emitting a libcall. */ + +/* { dg-do compile } */ +/* { dg-options "-O2" } */ + +struct context { + unsigned int data; + unsigned int data_end; + unsigned int data_meta; + unsigned int ingress; + unsigned int queue_index; + unsigned int egress; +}; + +void +mov_1_nooverlap (struct context *ctx) +{ + void *data = (void *)(long)ctx->data; + char *dest; + dest = data; + dest += 16; + + __builtin_memmove (dest, data, 12); +} + +void +mov_1_overlap (struct context *ctx) +{ + void *data = (void *)(long)ctx->data; + char *dest; + dest = data; + dest += 4; + + __builtin_memmove (dest, data, 12); +} + +void +mov_1_arbitrary (struct context *ctx_a, struct context *ctx_b) +{ + void *src = (void *)(long)ctx_a->data; + void *dst = (void *)(long)ctx_b->data; + + __builtin_memmove (dst, src, 12); +} + +/* { dg-final { scan-assembler-times "call" 0 } } */ diff --git a/gcc/testsuite/gcc.target/bpf/memmove-2.c b/gcc/testsuite/gcc.target/bpf/memmove-2.c new file mode 100644 index 00000000000..c0f92d40d92 --- /dev/null +++ b/gcc/testsuite/gcc.target/bpf/memmove-2.c @@ -0,0 +1,23 @@ +/* Test that we error if memmove cannot be expanded inline. */ + +/* { dg-do compile } */ +/* { dg-options "-O2" } */ + +typedef unsigned int __u32; + +struct context { + unsigned int data; + unsigned int data_end; + unsigned int data_meta; +}; + +void +mov_2_unsupported (struct context *ctx) +{ + void *data = (void *)(long)ctx->data; + char *dest; + dest = data; + dest += 4; + + __builtin_memmove (dest, data, ctx->data_meta); /* { dg-error "could not inline call" } */ +}