From patchwork Fri Sep 1 23:52:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829017 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=Dk/Ngr2w; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RcvwY1nl5z1ygw for ; Sat, 2 Sep 2023 09:52:45 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 26161385AC26 for ; Fri, 1 Sep 2023 23:52:43 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by sourceware.org (Postfix) with ESMTPS id 4436A3858D20 for ; Fri, 1 Sep 2023 23:52:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4436A3858D20 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-68a4bcf8a97so1909421b3a.1 for ; Fri, 01 Sep 2023 16:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612351; x=1694217151; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bYi/z6gpEpDdyXHw4vWiy63GG5LvJdw0Sd0Jd4Zjqdw=; b=Dk/Ngr2wQDy6XTwTsfO7iGf3RJc6NO8hUgOj+05iYzxx8czJvzmRDx3DRXtQzI+7Jf peZh04xMixtxESTs1kDfioYzro5XNUpdUEDxMqmfQYSOZ1ZeCV8gkqBRqZRcrY/pi4gB Wr0qq5P0uNaMWpFgboNKJCwg6nLI+SndYQhUsptR76qHBTL5NUIpTUcMFUQ2uYkDDXUi PH2vwjyGHUA/jNHfzp5rWYWCUSR9zBPDmZ77JlB9cYaoQ0JczVofmeu+sJj80Vn3Ib87 ZFOElE9n6MDKe/G66RveQqc589prCrDZFS3Px5f1DXvgo7tPoCPZ0TjDueOX6CkfIl27 bhgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612351; x=1694217151; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bYi/z6gpEpDdyXHw4vWiy63GG5LvJdw0Sd0Jd4Zjqdw=; b=KvZCGsLR2+P+OIk1zW5v8g6itO5DRjbQP0XVwvPB0pgf4uRcapgINZDkXjizpoqslY dhd1SfZnBbB2aUP15x9zZ42ntRZKvWA/c9DeKUgPmY06zu63aFhLV6Tg4CXP1/cDTSzz qQB/RLXY/t5MciGfNmKpgShsknV6AQquVMmLNaJleW014aYTMsilhqM9fxkmCVOL5I+K zts2wg2XglkFAkFrueuS/wncgPEhlyXHvfCMsCtKMBQN/7dSTU2wSllCM4uZpFSzxHY5 GGco2k+kkjT8unPm3xCR6mPzM42Y/qdR69VCyj1zAizu33wd1nSG32zO4BZfQ2bKQbGk BRng== X-Gm-Message-State: AOJu0YyoJiITeo5Rnj/u1OOsKTqYq/yD8gvMfhXDig/6EeV7maQXUfLR LywF2KrSwYDfAwCU6Wb1FovYb7lDhzUG7wLIXw4= X-Google-Smtp-Source: AGHT+IF79aNl7eliQUWkEAuKfRdQhw8B9Sfcae2hkGTFL733INZZeuiDUfHfrAVggjZm5Pg5STH8nw== X-Received: by 2002:a05:6a20:748e:b0:13f:8a5f:ad8e with SMTP id p14-20020a056a20748e00b0013f8a5fad8emr4857450pzd.58.1693612350779; Fri, 01 Sep 2023 16:52:30 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:30 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 1/6] riscv: Add Linux hwprobe syscall support Date: Fri, 1 Sep 2023 16:52:19 -0700 Message-Id: <20230901235224.3304592-2-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Add awareness and a thin wrapper function around a new Linux system call that allows callers to get architecture and microarchitecture information about the CPUs from the kernel. This can be used to do things like dynamically choose a memcpy implementation. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v8: - Fix missed 2.39 in abilists (Joseph) - Just return -r (Florian) Changes in v7: - Bumped Versions up to 2.39 (Joseph) - Used INTERNAL_SYSCALL_CALL, and return positive errno to match pthreads API (Florian). - Remove __THROW since it creates a warning in combination with the fortified access decorators. Changes in v6: - Prefixed __riscv_hwprobe() parameters names with __ to avoid user macro namespace pollution (Joseph) Changes in v4: - Remove __USE_GNU (Florian) - __nonnull, __wur, __THROW, and __fortified_attr_access decorations (Florian) - change long to long int (Florian) - Fix comment formatting (Florian) - Update backup kernel header content copy. - Fix function declaration formatting (Florian) - Changed export versions to 2.38 Changes in v3: - Update argument types to match v4 kernel interface Changes in v2: - hwprobe.h: Use __has_include and duplicate Linux content to make compilation work when Linux headers are absent (Adhemerval) - hwprobe.h: Put declaration under __USE_GNU (Adhemerval) - Use INLINE_SYSCALL_CALL (Adhemerval) - Update versions - Update UNALIGNED_MASK to match kernel v3 series. sysdeps/unix/sysv/linux/riscv/Makefile | 4 +- sysdeps/unix/sysv/linux/riscv/Versions | 3 + sysdeps/unix/sysv/linux/riscv/hwprobe.c | 36 ++++++++++ .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 72 +++++++++++++++++++ 6 files changed, 115 insertions(+), 2 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe.c create mode 100644 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 4b6eacb32f..45cc29e40d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,6 @@ ifeq ($(subdir),misc) -sysdep_headers += sys/cachectl.h -sysdep_routines += flush-icache +sysdep_headers += sys/cachectl.h sys/hwprobe.h +sysdep_routines += flush-icache hwprobe endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 5625d2a0b8..8717b62a4a 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -8,4 +8,7 @@ libc { GLIBC_2.27 { __riscv_flush_icache; } + GLIBC_2.39 { + __riscv_hwprobe; + } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c new file mode 100644 index 0000000000..e28194e344 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -0,0 +1,36 @@ +/* RISC-V hardware feature probing support on Linux + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include + +int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) +{ + int r; + + r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + + /* Negate negative errno values to match pthreads API. */ + return -r; +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index b9740a1afc..35cd01cba5 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2436,3 +2436,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.39 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index e3b4656aa2..55b3c53f52 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2636,3 +2636,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.39 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h new file mode 100644 index 0000000000..aa189a4818 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -0,0 +1,72 @@ +/* RISC-V architecture probe interface + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _SYS_HWPROBE_H +#define _SYS_HWPROBE_H 1 + +#include +#include +#ifdef __has_include +# if __has_include () +# include +# endif +#endif + +/* Define a (probably stale) version of the interface if the Linux headers + aren't present. */ +#ifndef RISCV_HWPROBE_KEY_MVENDORID +struct riscv_hwprobe { + signed long long int key; + unsigned long long int value; +}; + +#define RISCV_HWPROBE_KEY_MVENDORID 0 +#define RISCV_HWPROBE_KEY_MARCHID 1 +#define RISCV_HWPROBE_KEY_MIMPID 2 +#define RISCV_HWPROBE_KEY_BASE_BEHAVIOR 3 +#define RISCV_HWPROBE_BASE_BEHAVIOR_IMA (1 << 0) +#define RISCV_HWPROBE_KEY_IMA_EXT_0 4 +#define RISCV_HWPROBE_IMA_FD (1 << 0) +#define RISCV_HWPROBE_IMA_C (1 << 1) +#define RISCV_HWPROBE_IMA_V (1 << 2) +#define RISCV_HWPROBE_EXT_ZBA (1 << 3) +#define RISCV_HWPROBE_EXT_ZBB (1 << 4) +#define RISCV_HWPROBE_EXT_ZBS (1 << 5) +#define RISCV_HWPROBE_KEY_CPUPERF_0 5 +#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) +#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) +#define RISCV_HWPROBE_MISALIGNED_SLOW (2 << 0) +#define RISCV_HWPROBE_MISALIGNED_FAST (3 << 0) +#define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0) +#define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) + +#endif /* RISCV_HWPROBE_KEY_MVENDORID */ + +__BEGIN_DECLS + +extern int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) + __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + +__END_DECLS + +#endif /* sys/hwprobe.h */ From patchwork Fri Sep 1 23:52:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829020 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=Ip4JYdA4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RcvxC0vtWz1ygF for ; Sat, 2 Sep 2023 09:53:19 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 32AA2385084C for ; Fri, 1 Sep 2023 23:53:17 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by sourceware.org (Postfix) with ESMTPS id D3FE83858D37 for ; Fri, 1 Sep 2023 23:52:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D3FE83858D37 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1c1f8aaab9aso20151535ad.1 for ; Fri, 01 Sep 2023 16:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612353; x=1694217153; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aAhWPSgvktYgj2fzRCxEswRwk7DllMq47Tg6ba8f3Mc=; b=Ip4JYdA4cgVy888LWgU1JrSRY3sE7NGsGizI2QntHqwKA2DS6vEu63xVkAk3FySg6/ 7JPccvS4mjwKgVG5DuVdjHNBV/hqGS7nNAJUMNHWUCoZYOBWVZpOCtRHIrtxSMbYS6Ui popTALHrkzYxj7pa1vBjZ97f4SHwZS1+yRM54ik3xRKTgpq2zM0xmlTcTMmjS++4GGg5 QpanvObBChFPsIeDSvGFdAM2DSMFWVbZ4x9V50q63yMHvTzfkjCHX2ZuFaodRbVidpQ4 OA1eBnm/QAPpqNlP1pMCWRM4M3hF8f46HCMIAtPNvRDMFVXoLUL2zODJ+YcyhXHJuy+t 9PJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612353; x=1694217153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aAhWPSgvktYgj2fzRCxEswRwk7DllMq47Tg6ba8f3Mc=; b=DAHESlTob+pNUCE0BaDQKoIilkXFTxYByqzsMwPwMKGxMYQIcDMGSPd5Xndj0g8pub vpwLqArxwxFO5RjtuWb0sBr3RVEgimRynpjGk0pji5ZtxCJzfkP3wKUewsL/O6yBZhtn cYA8R6wghpsh3a3IOQ9veyuY0IXNaz6/nV1Dn2Uz+ouyvjAyhpG5joQ5nk/4E1/vWF0e HQfebZD2f1i3gaal7qKKBkXspQ3CeloH0KOSKvIoR438BayuQ0kzDyfzmvSZyZ2Q/7QT xOIMho5cn3S4w0qKl3JAtGNeCD6AWhCTlNWYscAvLxWMOklfc488JfSdv26D87czsiul xu8A== X-Gm-Message-State: AOJu0YwGQ1toH0AGZDVfppL9RTXsE3RTGybNlc5mXmImD9WcqrSX8ekK Oc5m+9cmzeLc8giK2GVhsB+hibjeeOx3kfy4Ru4= X-Google-Smtp-Source: AGHT+IHaUWpm1HDuE+hZGTtGE5RVoA/t+U6Mc6UQ4xahBZwSIxS/c81NIL6haoKPW8SNZKuidxRMWA== X-Received: by 2002:a17:902:9893:b0:1c0:6dcd:453d with SMTP id s19-20020a170902989300b001c06dcd453dmr4037020plp.20.1693612353427; Fri, 01 Sep 2023 16:52:33 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:33 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 2/6] riscv: Add hwprobe vdso call support Date: Fri, 1 Sep 2023 16:52:20 -0700 Message-Id: <20230901235224.3304592-3-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" The new riscv_hwprobe syscall also comes with a vDSO for faster answers to your most common questions. Call in today to speak with a kernel representative near you! Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- (no changes since v7) Changes in v7: - Use INTERNAL_VSYSCALL_CALL (Florian) Changes in v3: - Add the "return" to the vsyscall - Fix up vdso arg types to match kernel v4 version - Remove ifdef around INLINE_VSYSCALL (Adhemerval) Changes in v2: - Add vDSO interface sysdeps/unix/sysv/linux/dl-vdso-setup.c | 10 ++++++++++ sysdeps/unix/sysv/linux/dl-vdso-setup.h | 3 +++ sysdeps/unix/sysv/linux/riscv/hwprobe.c | 17 ++++++++++++++--- sysdeps/unix/sysv/linux/riscv/sysdep.h | 1 + 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.c b/sysdeps/unix/sysv/linux/dl-vdso-setup.c index 97eaaeac37..ed8b1ef426 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.c +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.c @@ -71,6 +71,16 @@ PROCINFO_CLASS int (*_dl_vdso_clock_getres_time64) (clockid_t, # ifdef HAVE_GET_TBFREQ PROCINFO_CLASS uint64_t (*_dl_vdso_get_tbfreq)(void) RELRO; # endif + +/* RISC-V specific ones. */ +# ifdef HAVE_RISCV_HWPROBE +PROCINFO_CLASS int (*_dl_vdso_riscv_hwprobe)(void *, + size_t, + size_t, + unsigned long *, + unsigned int) RELRO; +# endif + #endif #undef RELRO diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.h b/sysdeps/unix/sysv/linux/dl-vdso-setup.h index 867072b897..39eafd5316 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.h +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.h @@ -47,6 +47,9 @@ setup_vdso_pointers (void) #ifdef HAVE_GET_TBFREQ GLRO(dl_vdso_get_tbfreq) = dl_vdso_vsym (HAVE_GET_TBFREQ); #endif +#ifdef HAVE_RISCV_HWPROBE + GLRO(dl_vdso_riscv_hwprobe) = dl_vdso_vsym (HAVE_RISCV_HWPROBE); +#endif } #endif diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c index e28194e344..6a9a44657f 100644 --- a/sysdeps/unix/sysv/linux/riscv/hwprobe.c +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -27,9 +27,20 @@ int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, unsigned int __flags) { int r; - - r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, - __cpu_count, __cpus, __flags); + __riscv_hwprobe_t vdso_hwprobe = + (__riscv_hwprobe_t)GLRO(dl_vdso_riscv_hwprobe); + + if (vdso_hwprobe != NULL) + { + r = INTERNAL_VSYSCALL_CALL (vdso_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + } + else + { + r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + + } /* Negate negative errno values to match pthreads API. */ return -r; diff --git a/sysdeps/unix/sysv/linux/riscv/sysdep.h b/sysdeps/unix/sysv/linux/riscv/sysdep.h index 5583b96d23..ee015dfeb6 100644 --- a/sysdeps/unix/sysv/linux/riscv/sysdep.h +++ b/sysdeps/unix/sysv/linux/riscv/sysdep.h @@ -156,6 +156,7 @@ /* List of system calls which are supported as vsyscalls (for RV32 and RV64). */ # define HAVE_GETCPU_VSYSCALL "__vdso_getcpu" +# define HAVE_RISCV_HWPROBE "__vdso_riscv_hwprobe" # undef HAVE_INTERNAL_BRK_ADDR_SYMBOL # define HAVE_INTERNAL_BRK_ADDR_SYMBOL 1 From patchwork Fri Sep 1 23:52:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829018 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=KWoDGBVg; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rcvwv70grz1ygF for ; Sat, 2 Sep 2023 09:53:03 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B657C3853D04 for ; Fri, 1 Sep 2023 23:53:01 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by sourceware.org (Postfix) with ESMTPS id 4E903385771E for ; Fri, 1 Sep 2023 23:52:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4E903385771E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1bdca7cc28dso21057025ad.1 for ; Fri, 01 Sep 2023 16:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612356; x=1694217156; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GUwAVLzxmk4oYzLKuvyk7oaHk08bVDBm+qjcAc1NUeM=; b=KWoDGBVgh2ahuP7fU4rbIG6QYkztzd+CDqY8ewdbGdjDSsYYgm+Muj3ZU85mnr8Sgy 811siAlXajH7G2uI8csJM/uaHoatWs1P9+kyL0ao5QihOWagOgQb/eg2O4Cf3qFQGx4v 7KxsYYB2KjGIPAbx01CHq3fL08rkUpA3dVmb7iP5ywwAsotoH3keDfprgueAhPZWlz3h FF9VGk407cX3Nl/JciIXHWh6grxgBE6ucllhOaJSA87kNmIDt4SpoZSxU8AU9y30fEqA wDrc8LqhHeBUMWHlx6iykQbT3WsnhFngh++n3QxkWZlYEt33UdDNAswqug4iVKwhF//p rNvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612356; x=1694217156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GUwAVLzxmk4oYzLKuvyk7oaHk08bVDBm+qjcAc1NUeM=; b=VXcQezsP/RShezoqRpfEYtxe8xqPTrOeWIK4CLN8Z0NXMXJleS+rKKMFqi9Wy0ZPU8 6P6ms3U7WX2cb00Puv/IEt+smtEiBNAivVrrUe2R8e2Eh1cdX77w4/GbKzXMcS8PSCOd bMRz8K+kQEsC9nx6UiA3LxJUule9eP4UoPUAQigm/Q+6uOMl7e5an79rsXv7WQDLNab6 FSUENg8MTXXTderyc+8MD2pglMjgTu2t8xs3hgu/XVg4m2y4lOt0OJUWOQOPE4YlClxQ 47WhGSsWzGeMDag0EZYi3AKZaJiXFdAKSRxonkY9q2ECbPmY4Q07ZoPKCGRNzF8HpH6/ Du6g== X-Gm-Message-State: AOJu0Yx22fo76WHweGDK2hOtUSLhaH0+TNwYGYoMZDrdIcqjRcHKVjuc dy3B9KjftV1jGKytXl8gZBx/aTIgEPF207kVc8Q= X-Google-Smtp-Source: AGHT+IH4aSlbDsl5Xe95QSGb/BUw/lXjLVP9JjwIz5XWZxuzrwyHiF9ftgAdC58jo67c6pR/n7nmlA== X-Received: by 2002:a17:903:32c1:b0:1c2:1443:d62 with SMTP id i1-20020a17090332c100b001c214430d62mr4771731plr.20.1693612355875; Fri, 01 Sep 2023 16:52:35 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:35 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 3/6] riscv: Add __riscv_hwprobe pointer to ifunc calls Date: Fri, 1 Sep 2023 16:52:21 -0700 Message-Id: <20230901235224.3304592-4-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" The new __riscv_hwprobe() function is designed to be used by ifunc selector functions. This presents a challenge for applications and libraries, as ifunc selectors are invoked before all relocations have been performed, so an external call to __riscv_hwprobe() from an ifunc selector won't work. To address this, pass a pointer to the __riscv_hwprobe() vDSO function into ifunc selectors as the second argument (alongside dl_hwcap, which was already being passed). Include a typedef as well for convenience, so that ifunc users don't have to go through contortions to call this routine. Users will need to remember to check the second argument for NULL, both to account for older glibcs that don't pass the function, and older kernels that don't have the vDSO pointer. Signed-off-by: Evan Green --- (no changes since v7) Changes in v7: - Remove __THROW from function pointer type, as it creates warnings together with __fortified_attr_access. sysdeps/riscv/dl-irel.h | 8 ++++---- sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 10 ++++++++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/sysdeps/riscv/dl-irel.h b/sysdeps/riscv/dl-irel.h index eaeec5467c..2147504458 100644 --- a/sysdeps/riscv/dl-irel.h +++ b/sysdeps/riscv/dl-irel.h @@ -31,10 +31,10 @@ static inline ElfW(Addr) __attribute ((always_inline)) elf_ifunc_invoke (ElfW(Addr) addr) { - /* The second argument is a void pointer to preserve the extension - fexibility. */ - return ((ElfW(Addr) (*) (uint64_t, void *)) (addr)) - (GLRO(dl_hwcap), NULL); + /* The third argument is a void pointer to preserve the extension + flexibility. */ + return ((ElfW(Addr) (*) (uint64_t, void *, void *)) (addr)) + (GLRO(dl_hwcap), GLRO(dl_vdso_riscv_hwprobe), NULL); } static inline void diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h index aa189a4818..fd3be5a411 100644 --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -67,6 +67,16 @@ extern int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, __fortified_attr_access (__read_write__, 1, 2) __fortified_attr_access (__read_only__, 4, 3); +/* A pointer to the __riscv_hwprobe vDSO function is passed as the second + argument to ifunc selector routines. Include a function pointer type for + convenience in calling the function in those settings. */ +typedef int (*__riscv_hwprobe_t) (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) + __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + __END_DECLS #endif /* sys/hwprobe.h */ From patchwork Fri Sep 1 23:52:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829019 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=swZHS0he; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rcvx34pVPz1ygF for ; Sat, 2 Sep 2023 09:53:11 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B96D23856092 for ; Fri, 1 Sep 2023 23:53:09 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by sourceware.org (Postfix) with ESMTPS id 4F8603857C44 for ; Fri, 1 Sep 2023 23:52:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4F8603857C44 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1bc0d39b52cso19204625ad.2 for ; Fri, 01 Sep 2023 16:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612358; x=1694217158; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ztinmuxaNjH/z1RZboH05tpq5nAt8cssy5r2/1B6f28=; b=swZHS0hemZoaU99Qs+mqe+nobEAM6UZLtnRzRxb0tlYjeXEtknGZwontvsKPjjcLG3 JXpOxbr1fNh6dVW/X043QaOZJCuanS1a/Jn/di7drEfMymc1nQvDcYxCby0Kw7tiQM+1 x9N7kAgq8cFIMRnrqWiwC+18BZbJniTe7INX6/Pj/AkqvyNRKT6ORmLFNWdExrKn2dc/ 23L9rkiipJBJJC75NGnP3w/+rMYTsFRyMUUBgSKgWyViFaUs5IVT3SrvWyLZupbvRenv p/lnOl1HMh7VlOXKeVYjO9hMpd2K6hSJB+mHyfFS7XW58aXpsNySB/9Ti0ldhNT/hmEQ 7r/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612358; x=1694217158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ztinmuxaNjH/z1RZboH05tpq5nAt8cssy5r2/1B6f28=; b=Pu82UICPbpKJNovlzW/QHIFsV0d+k5t8yJ8LsLQElzP98E1RxavyUjj0fcgbWo+8pD Kcjrr89Rgt+vMqS41LrmXSOW2WJJtYccFcHbCFOiGFETj50TWVKC5asHHVfdvMx5BlNv gTDUElzJKMd2oTT5QtPAiT6Y2K0Cer41kqy40XjgTr74XoGXJKrMgg5UmPoOVhJY+tl8 YN57bsVzBXqkNdvY/xKyxkHkI7/24d+bM57duu/SlR8K3IwAFxaADH/NqJ6Sb/KnVTKa ZHXlkRgRNS5uy6Zx5UUfc1f9iumzYT54sumeeUvfiyaTwK2VMlcgrFEzN5Qym4O4P7LJ 78Rw== X-Gm-Message-State: AOJu0Yy/7YKQcl0tWBC7BgiggIP3JrUsTsG+6XJdi81vHyFokPL6ExeH n1C42qzNwq3YpqLQ1rF4FF9H8ebu7Oll792mFK8= X-Google-Smtp-Source: AGHT+IFsiH4mc3Hnu0KkU3D5Tf5T2TvN6tIDe9HIBpVVvd0nyFGwgI9oH99QBl4iBHx+rdRMAOdDIA== X-Received: by 2002:a17:902:e847:b0:1c2:54c:8bf0 with SMTP id t7-20020a170902e84700b001c2054c8bf0mr4427118plg.54.1693612358033; Fri, 01 Sep 2023 16:52:38 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:37 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 4/6] riscv: Enable multi-arg ifunc resolvers Date: Fri, 1 Sep 2023 16:52:22 -0700 Message-Id: <20230901235224.3304592-5-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" RISC-V is apparently the first architecture to pass more than one argument to ifunc resolvers. The helper macros in libc-symbols.h, __ifunc_resolver(), __ifunc(), and __ifunc_hidden(), are incompatible with this. These macros have an "arg" (non-final) parameter that represents the parameter signature of the ifunc resolver. The result is an inability to pass the required comma through in a single preprocessor argument. Rearrange the __ifunc_resolver() macro to be variadic, and pass the types as those variable parameters. Move the guts of __ifunc() and __ifunc_hidden() into new macros, __ifunc_args(), and __ifunc_args_hidden(), that pass the variable arguments down through to __ifunc_resolver(). Then redefine __ifunc() and __ifunc_hidden(), which are used in a bunch of places, to simply shuffle the arguments down into __ifunc_args[_hidden]. Finally, define a riscv-ifunc.h header, which provides convenience macros to those looking to write ifunc selectors that use both arguments. Signed-off-by: Evan Green --- (no changes since v6) Changes in v6: - Introduced riscv-ifunc.h for multi-arg ifunc selectors. Note: I opted to create another layer of macros (__ifunc_args()) rather than doing the treewide change to rearrange the signature of __ifunc() and __ifunc_hidden(). If folks like the overall approach but would prefer the treewide change, I can do that too. --- include/libc-symbols.h | 28 +++++++++++++++++----------- sysdeps/riscv/riscv-ifunc.h | 27 +++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 11 deletions(-) create mode 100644 sysdeps/riscv/riscv-ifunc.h diff --git a/include/libc-symbols.h b/include/libc-symbols.h index 5794614488..36b92039c5 100644 --- a/include/libc-symbols.h +++ b/include/libc-symbols.h @@ -665,9 +665,9 @@ for linking") #endif /* Helper / base macros for indirect function symbols. */ -#define __ifunc_resolver(type_name, name, expr, arg, init, classifier) \ +#define __ifunc_resolver(type_name, name, expr, init, classifier, ...) \ classifier inhibit_stack_protector \ - __typeof (type_name) *name##_ifunc (arg) \ + __typeof (type_name) *name##_ifunc (__VA_ARGS__) \ { \ init (); \ __typeof (type_name) *res = expr; \ @@ -675,13 +675,13 @@ for linking") } #ifdef HAVE_GCC_IFUNC -# define __ifunc(type_name, name, expr, arg, init) \ +# define __ifunc_args(type_name, name, expr, init, ...) \ extern __typeof (type_name) name __attribute__ \ ((ifunc (#name "_ifunc"))); \ - __ifunc_resolver (type_name, name, expr, arg, init, static) + __ifunc_resolver (type_name, name, expr, init, static, __VA_ARGS__) -# define __ifunc_hidden(type_name, name, expr, arg, init) \ - __ifunc (type_name, name, expr, arg, init) +# define __ifunc_args_hidden(type_name, name, expr, init, ...) \ + __ifunc (type_name, name, expr, init, __VA_ARGS__) #else /* Gcc does not support __attribute__ ((ifunc (...))). Use the old behaviour as fallback. But keep in mind that the debug information for the ifunc @@ -692,18 +692,24 @@ for linking") different signatures. (Gcc support is disabled at least on a ppc64le Ubuntu 14.04 system.) */ -# define __ifunc(type_name, name, expr, arg, init) \ +# define __ifunc_args(type_name, name, expr, init, ...) \ extern __typeof (type_name) name; \ - __typeof (type_name) *name##_ifunc (arg) __asm__ (#name); \ - __ifunc_resolver (type_name, name, expr, arg, init,) \ + __typeof (type_name) *name##_ifunc (__VA_ARGS__) __asm__ (#name); \ + __ifunc_resolver (type_name, name, expr, init, , __VA_ARGS__) \ __asm__ (".type " #name ", %gnu_indirect_function"); -# define __ifunc_hidden(type_name, name, expr, arg, init) \ +# define __ifunc_args_hidden(type_name, name, expr, init, ...) \ extern __typeof (type_name) __libc_##name; \ - __ifunc (type_name, __libc_##name, expr, arg, init) \ + __ifunc (type_name, __libc_##name, expr, __VA_INIT__, init) \ strong_alias (__libc_##name, name); #endif /* !HAVE_GCC_IFUNC */ +#define __ifunc(type_name, name, expr, arg, init) \ + __ifunc_args (type_name, name, expr, init, arg) + +#define __ifunc_hidden(type_name, name, expr, arg, init) \ + __ifunc_args_hidden (type_name, expr, init, arg) + /* The following macros are used for indirect function symbols in libc.so. First of all, you need to have the function prototyped somewhere, say in foo.h: diff --git a/sysdeps/riscv/riscv-ifunc.h b/sysdeps/riscv/riscv-ifunc.h new file mode 100644 index 0000000000..7bff591d1e --- /dev/null +++ b/sysdeps/riscv/riscv-ifunc.h @@ -0,0 +1,27 @@ +/* Common definition for ifunc resolvers. Linux/RISC-V version. + This file is part of the GNU C Library. + Copyright (C) 2023 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +#define INIT_ARCH() + +#define riscv_libc_ifunc(name, expr) \ + __ifunc_args (name, name, expr(hwcap, hwprobe), INIT_ARCH, \ + uint64_t hwcap, __riscv_hwprobe_t hwprobe) From patchwork Fri Sep 1 23:52:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829021 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=TZ0u2yF3; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RcvxZ6RCsz1ygF for ; Sat, 2 Sep 2023 09:53:38 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id F19213858C39 for ; Fri, 1 Sep 2023 23:53:36 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by sourceware.org (Postfix) with ESMTPS id B92583856DDA for ; Fri, 1 Sep 2023 23:52:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B92583856DDA Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1bf078d5f33so20985115ad.3 for ; Fri, 01 Sep 2023 16:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612360; x=1694217160; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4M4T9oKuv8pbgdqOgTCb0F93HmJB1LDIToIgnoaj87w=; b=TZ0u2yF3UFTVuWip8cXKZ0sUa8Po/8hPa6uIhOgjrx+xi7TKWznRzlJUS1fGJ/qAtF 9d0FGF5xX1sWjinSwK1vW13wBE1bCSaLs1XeM49mpfn4FoLCcS5bOUsSOBfGp9ZZ/dpP gcAyniUaykWCM/8AlsYWGK0uk+ZCc1XgWFsA3OPgyFVNIYR3K+VD6SwYdl+nY3w7o3L9 sgL8Gjk01ct4kzYuYkF525scN7XYKSM2VdgkI5gOCrmEqDg0jU6qgWNsbBxeXsetzOmN 424Lahg8TaudYrp1Ohk0YljqQgYkqj8Js8BflmGCozH/tSl/Fdut1zb/yrWj54NcOWYL Levg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612360; x=1694217160; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4M4T9oKuv8pbgdqOgTCb0F93HmJB1LDIToIgnoaj87w=; b=UiFhYiRr2vy/u5rCwPXieJXdJvkka0S12rqtn+sm5mn7l2FfGIE+rzSCMSW9yhFzoz XuPuD5R1IDU+J34E1MTqnLNEq7Lvj9sy0hynj3jyWQabTdSM+nU3udrD8FSRuTdnmDj6 RsNUdnZXRqXXa3s35o9aHrTOgTaTmp7hmLBGlF5OqzavNsVztJVniLgDcLJPnuOk8ZvW a2UNIwxjZbQU1QCjLca0NeCcaITRwqS+vH427O0sbqvSHR69MrWuCGSiP/1SzodqjZku B+cD9HOSO1LlykLdZ34YyY+ugSQtNlNmIckXNAiSlL75iZPdMysE7rhbCUgSFofetJSa WNyA== X-Gm-Message-State: AOJu0YzR494rXdZeOyaGypi0AUVy3A/A/HmTdhuKPKJGoaZdEj8GM4v5 6YyBzm4evVbAcyZ0LbgQ+ERyY2ohN3p7oUBGiCY= X-Google-Smtp-Source: AGHT+IGZ9HpmzXrsHdEQW8FJmJVkNARZt9NRpzlqucGtzsatXRHcXucBqUgIc/NJ8FTq98xZb/tJUA== X-Received: by 2002:a17:902:c412:b0:1c0:77b8:bb10 with SMTP id k18-20020a170902c41200b001c077b8bb10mr6431642plk.10.1693612360475; Fri, 01 Sep 2023 16:52:40 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:40 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 5/6] riscv: Add ifunc helper method to hwprobe.h Date: Fri, 1 Sep 2023 16:52:23 -0700 Message-Id: <20230901235224.3304592-6-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Add a little helper method so it's easier to fetch a single value from the hwprobe function when used within an ifunc selector. Signed-off-by: Evan Green --- (no changes since v7) Changes in v7: - Introduced static inline helper (Richard) sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 24 +++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h index fd3be5a411..7378b1baa2 100644 --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -22,6 +22,7 @@ #include #include +#include #ifdef __has_include # if __has_include () # include @@ -79,4 +80,27 @@ typedef int (*__riscv_hwprobe_t) (struct riscv_hwprobe *__pairs, size_t __pair_c __END_DECLS +/* Helper function usable from ifunc selectors that probes a single key. */ +static inline int __riscv_hwprobe_one(__riscv_hwprobe_t hwprobe_func, + signed long long int key, + unsigned long long int *value) +{ + struct riscv_hwprobe pair; + int rc; + + if (!hwprobe_func) + return ENOSYS; + + pair.key = key; + rc = hwprobe_func(&pair, 1, 0, NULL, 0); + if (rc) + return rc; + + if (pair.key < 0) + return ENOENT; + + *value = pair.value; + return 0; +} + #endif /* sys/hwprobe.h */ From patchwork Fri Sep 1 23:52:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1829022 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=v4viL0XC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rcvy80mpnz1ygF for ; Sat, 2 Sep 2023 09:54:08 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2E5AA3858296 for ; Fri, 1 Sep 2023 23:54:06 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by sourceware.org (Postfix) with ESMTPS id CDE643857706 for ; Fri, 1 Sep 2023 23:52:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CDE643857706 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1bf7423ef3eso19220855ad.3 for ; Fri, 01 Sep 2023 16:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1693612362; x=1694217162; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QkP9/BWYF54S2JFmsoNLjYe3SFILFKlRDe8fZrAVdTk=; b=v4viL0XCS5defBqMfJuFs0m4rURUfmf9KpueIs/PAHsfKDysr+vmPETy2E3R2GkFd9 By63iF/IqBCQsKfCfDbpoTwOqblKzHmkpvElJom/xTkHk3JbJDNXgm+WO46Bht/IxPY7 gtO2h/sy1jlIVniIv/f+Oaoem2EMGdHaVpZ0L5G1RqJkXaRRlAjAqPLFNKhBVnakV3C0 f2tOVZj1c90AFdY/CSyBDmR8W8iP/Eg//jtyzA5PcWi/SkRIAumkAy8v5XfL3OKV4Bdt XDFrbDANjnr4CsBSIO0IIiW8K+Mx3ZR86Xe4HvPHvUueaD5KDyPtDA4N5NSFQbttgXKy EkOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693612362; x=1694217162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QkP9/BWYF54S2JFmsoNLjYe3SFILFKlRDe8fZrAVdTk=; b=HNRSSDuTQxrXWiHzjpoyEgH6C93pLXqJb0NHwhWqa1US3v9t7Q9/x8Eg2a8JtJn0sT NCQ/zso00DTQGcJ9+Eu81d+o1Z1bGRb0J7cxZ4kiseSjlJeOw7ebAKISiR7KanmI7Rtq pMTxt3RiJVu9Vef+Nu9mQvwgtU1Ooos5a275DC9WUW6i+xGGM4VUJb+PU7gt+lmRXrvC S+CCaBdVhKoVEawtZ9ZvJD99fBWt4u3exG5LaGfsSp345b0GAB4OWxqP0U7gadha1UNb FFNrrILiJ8+gXDtH9dRopAHzoCxOC9zWNS3pfZb98VvxGI6pediVlkmvV1Nfyq8Tmqc/ Y+Sg== X-Gm-Message-State: AOJu0YyxrBXaSSUmi7e0XvYlHPPuxQgjgSP6ZVXr2ucwZIDEIO6PvNaa 1ZUWPp3LoqoleIdXdL8Bj5HKUoUn7nLdhaiOOmU= X-Google-Smtp-Source: AGHT+IE9Mw2ptWtkNJWCOT6F2Iyl/Vx+QrmLQojKjgV9BccOJHN8s9D10MBZDdVg/OBu6RFi6PjCpQ== X-Received: by 2002:a17:902:ab1d:b0:1b8:4ec2:5200 with SMTP id ik29-20020a170902ab1d00b001b84ec25200mr3883970plb.2.1693612362464; Fri, 01 Sep 2023 16:52:42 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id c13-20020a170902d48d00b001bee782a1desm3499551plg.181.2023.09.01.16.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Sep 2023 16:52:42 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Subject: [PATCH v8 6/6] riscv: Add and use alignment-ignorant memcpy Date: Fri, 1 Sep 2023 16:52:24 -0700 Message-Id: <20230901235224.3304592-7-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230901235224.3304592-1-evan@rivosinc.com> References: <20230901235224.3304592-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Florian Weimer , Evan Green , vineetg@rivosinc.com Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- (no changes since v7) Changes in v7: - Use new helper function in memcpy ifunc selector (Richard) Changes in v6: - Fix a couple regressions in the assembly from v5 :/ - Use passed hwprobe pointer in memcpy ifunc selector. Changes in v5: - Do unaligned word access for final trailing bytes (Richard) Changes in v4: - Fixed comment style (Florian) Changes in v3: - Word align dest for large memcpy()s. - Add tags - Remove spurious blank line from sysdeps/riscv/memcpy.c Changes in v2: - Used _MASK instead of _FAST value itself. --- sysdeps/riscv/memcopy.h | 26 ++++ sysdeps/riscv/memcpy.c | 63 ++++++++ sysdeps/riscv/memcpy_noalignment.S | 138 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 4 + .../unix/sysv/linux/riscv/memcpy-generic.c | 24 +++ 5 files changed, 255 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..2b685c8aa0 --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,26 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* Redefine the generic memcpy implementation to __memcpy_generic, so + the memcpy ifunc can select between generic and special versions. + In rtld, don't bother with all the ifunciness. */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..285ca85f8e --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,63 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +# include +# include +# include +# include + +# define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (uint64_t dl_hwcap, __riscv_hwprobe_t hwprobe_func) +{ + unsigned long long int value; + + INIT_ARCH (); + + if (__riscv_hwprobe_one(hwprobe_func, RISCV_HWPROBE_KEY_CPUPERF_0, &value)) + return __memcpy_generic; + + if ((value & RISCV_HWPROBE_MISALIGNED_MASK) == RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +riscv_libc_ifunc (__libc_memcpy, select_memcpy_ifunc); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..f3bf8e5867 --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,138 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Bail if 0 */ + beqz a2, 7f + + /* Jump to byte copy if size < SZREG */ + li a4, SZREG + bltu a2, a4, 5f + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 + + /* Copy the first word to get dest word aligned */ + andi a5, t6, SZREG-1 + beqz a5, 1f + REG_L a6, (a1) + REG_S a6, (t6) + + /* Align dst up to a word, move src and size as well. */ + addi t6, t6, SZREG-1 + andi t6, t6, ~(SZREG-1) + sub a5, t6, a0 + add a1, a1, a5 + sub a2, a2, a5 + + /* Recompute page count */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 7f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to last word if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 7f + +4: + /* Copy the last word unaligned */ + add a3, a1, a2 + add a4, t6, a2 + REG_L a5, -SZREG(a3) + REG_S a5, -SZREG(a4) + ret + +5: + /* Copy bytes when the total copy is . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto(__memcpy_generic) + +#include