From patchwork Fri Apr 16 23:59:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467439 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=eZCVj1PJ; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMY9h0X10z9sW1 for ; Sat, 17 Apr 2021 10:00:06 +1000 (AEST) Received: from localhost ([::1]:53386 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYNX-0005CQ-NF for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:00:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41944) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYN5-00059V-GC for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:35 -0400 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]:41737) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN2-0005aQ-LV for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:35 -0400 Received: by mail-pf1-x435.google.com with SMTP id w6so4825520pfc.8 for ; Fri, 16 Apr 2021 16:59:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=unKgkmmxr7goKYi3rp8VTPEkn4g0CMVgXa5W1kzSLnA=; b=eZCVj1PJtk+yFRxOos4AGGE+rvC4ryIdjJ36PaCx2bNm81V7TG7fYy5jB1zVRBVM3L fJKiKpfripfA+BsGUzSAMCij1t3sPJnbK2VkwgEVwycFyUdl0j7ouHfAoUK4ISbbKWDC uJE94Nrq6hvdvFwqsRVb5TYVKNMjWHN0HTa15iUtmsyLfbFrWhTIIrOUZc5UtiTM1T1I TEDkqmyiqjKM8RD9aZI2QsTPWS8wHohfth6M+f5AE+BQjnYrmRCxtMwHSp12jzN5f5a5 s/XJa9ZhhfprvMVOIWwn3vKzKJj+iCIS22FNNuLbppCpJ3/4tvJHuaDYpxVXmYrMdHWZ gazw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=unKgkmmxr7goKYi3rp8VTPEkn4g0CMVgXa5W1kzSLnA=; b=loe4oDtEL6GjzYck1Ov2laGkJ+8lPSnjqM1JS6D1Tb6hLNCE2yK4i++XK+GvoJwXFc mfHZdR2jlEd5JUSFnZqr7KitBpveHsGx1c1/iE+P7+OzujUrzv74mVPTxYjfy624CsUS h+xGrS5+cNBT9mWeDKPbCAAhSltsVE/5ykwBHv9HyfWZ4MoTXy5+yYjjMIbRAA5yrRzn qBA3IRX0CeHSC+l/FmL2qGoExfcE9CCps0BCW9fl39LSHtJEW1c7n09/fu6Q85cB4iDC +TqsJkxNCrR0xz7lLZij8L0l3/pmkJQcfQzEcxDLao5m/FpdF2utLO0qogmMWvibR7Cr dJsg== X-Gm-Message-State: AOAM532nh4gentvRxYLiTIBm0ya16KEXssuABrdohQN1gtM0nnB5yDkP H7KjM8BPfb3u4yvBQrVRiflN37ymZLyKrg== X-Google-Smtp-Source: ABdhPJwGMuYtoaO16hvh+BDAghQGHg4ZDTqMXH4BlauMc8v6/I3sUMvSurQ42tS0cNVUKNIsjvWM4w== X-Received: by 2002:a63:70e:: with SMTP id 14mr1297073pgh.202.1618617571468; Fri, 16 Apr 2021 16:59:31 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:31 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 01/11] target/arm: Add isar_feature_{aa32, aa64, aa64_sve}_bf16 Date: Fri, 16 Apr 2021 16:59:18 -0700 Message-Id: <20210416235928.1631788-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::435; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x435.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Note that the SVE BFLOAT16 support does not require SVE2, it is an independent extension. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/cpu.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 134dc65e34..38db20c721 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -3783,6 +3783,11 @@ static inline bool isar_feature_aa32_predinv(const ARMISARegisters *id) return FIELD_EX32(id->id_isar6, ID_ISAR6, SPECRES) != 0; } +static inline bool isar_feature_aa32_bf16(const ARMISARegisters *id) +{ + return FIELD_EX32(id->id_isar6, ID_ISAR6, BF16) != 0; +} + static inline bool isar_feature_aa32_i8mm(const ARMISARegisters *id) { return FIELD_EX32(id->id_isar6, ID_ISAR6, I8MM) != 0; @@ -4112,6 +4117,11 @@ static inline bool isar_feature_aa64_dcpodp(const ARMISARegisters *id) return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, DPB) >= 2; } +static inline bool isar_feature_aa64_bf16(const ARMISARegisters *id) +{ + return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, BF16) != 0; +} + static inline bool isar_feature_aa64_fp_simd(const ARMISARegisters *id) { /* We always set the AdvSIMD and FP fields identically. */ @@ -4256,6 +4266,11 @@ static inline bool isar_feature_aa64_sve2_bitperm(const ARMISARegisters *id) return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, BITPERM) != 0; } +static inline bool isar_feature_aa64_sve_bf16(const ARMISARegisters *id) +{ + return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, BFLOAT16) != 0; +} + static inline bool isar_feature_aa64_sve2_sha3(const ARMISARegisters *id) { return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, SHA3) != 0; From patchwork Fri Apr 16 23:59:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467442 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=DIlY8w0T; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYC80PhVz9sTD for ; Sat, 17 Apr 2021 10:01:24 +1000 (AEST) Received: from localhost ([::1]:58728 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYOo-0007Tq-1o for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:01:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41994) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYN7-0005CD-5p for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:37 -0400 Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]:56007) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN3-0005b1-Ur for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:36 -0400 Received: by mail-pj1-x1031.google.com with SMTP id s14so9995707pjl.5 for ; Fri, 16 Apr 2021 16:59:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/N6ThJ9lpoodnukC32bBD9hHhS8zuwX7N8aEvzIih5o=; b=DIlY8w0TzDz7sSwpPz0Z10wMUhe4PIfaFC/qPi5eceXtGXYRhLvFYPkJyzTKSFRnct /2rQIZ6S6qev8XKbRX7kcxHPoKzWzoGZTgwHiTLVGmhuCsv/MZKkVy1JCphrFVqz/9Wv oJo48UN0PqhblcO6rVIZlF7FtgCM6D1/KgBH6nQWjTBybefUU0uFQc2KnqbH7SNpvAQA 6d0bl4EgD5dSEBFwnFiKT2ClFlbHwmv8kaVJwZpGh5bJvc3E0mXH0cyYo9NNG1rjXLtE 5TKiyuxq1qqBYD9YZGi7i4moaIPJaZ1u79VIhj+62H6aOHnmq5pzOdWrSBeHpth6tyYy sePw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/N6ThJ9lpoodnukC32bBD9hHhS8zuwX7N8aEvzIih5o=; b=K3WhxCH5EIpG2J/z1FI4izI9l8usJMsaGSGA6sIxiVR0Kf2l1pWoxctdvPzdRodpmQ u3JacVyraL9Y/a9dIW1IA1DSVZbiEa/zyNnngjXNmakQ7+2u+a912lAtwamx1U1JloNz UTxLVHXMYa36qYQF/aagwxU7XmgsGHbLsqsWfm6BNlB4sMBYdHhFhWxjnxpdkZjbzmzI nIulIj3V7nsm48uHm2OCIuBv69IE281AqmlIo/5R+K2f78JPjeZbTVsl9RdW/9a/cak/ NosrJp4mKJuc3KXXFzp7Cux+D8u6ecJkv2cEalzndhBBnbJDItvjullH53ft0L+Q7XfP e7QQ== X-Gm-Message-State: AOAM533ONn8BKqMgCIkgjlv4MxsK+nOpF3x6r1LCMbSAlbz1MjjudZLB 1NRzAq+XlWTzaa2OXOQWpHPO5Ubsbnejrw== X-Google-Smtp-Source: ABdhPJwU5ii5T37hWJnTwZzvk3HK9wNTPfIEy3bdx4Ui55iAr7HYF4eY5NXq7KXaehaR8uCEJN6ITw== X-Received: by 2002:a17:90a:4f41:: with SMTP id w1mr10603836pjl.231.1618617572489; Fri, 16 Apr 2021 16:59:32 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 02/11] target/arm: Unify unallocated path in disas_fp_1src Date: Fri, 16 Apr 2021 16:59:19 -0700 Message-Id: <20210416235928.1631788-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1031; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1031.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/translate-a64.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 668edf3a00..d8ec219bb2 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -6509,8 +6509,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) int rd = extract32(insn, 0, 5); if (mos) { - unallocated_encoding(s); - return; + goto do_unallocated; } switch (opcode) { @@ -6519,8 +6518,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) /* FCVT between half, single and double precision */ int dtype = extract32(opcode, 0, 2); if (type == 2 || dtype == type) { - unallocated_encoding(s); - return; + goto do_unallocated; } if (!fp_access_check(s)) { return; @@ -6532,8 +6530,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) case 0x10 ... 0x13: /* FRINT{32,64}{X,Z} */ if (type > 1 || !dc_isar_feature(aa64_frint, s)) { - unallocated_encoding(s); - return; + goto do_unallocated; } /* fall through */ case 0x0 ... 0x3: @@ -6555,8 +6552,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) break; case 3: if (!dc_isar_feature(aa64_fp16, s)) { - unallocated_encoding(s); - return; + goto do_unallocated; } if (!fp_access_check(s)) { @@ -6565,11 +6561,12 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) handle_fp_1src_half(s, opcode, rd, rn); break; default: - unallocated_encoding(s); + goto do_unallocated; } break; default: + do_unallocated: unallocated_encoding(s); break; } From patchwork Fri Apr 16 23:59:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467443 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=rEqA/G9g; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYDH5Hp8z9sTD for ; Sat, 17 Apr 2021 10:02:23 +1000 (AEST) Received: from localhost ([::1]:34056 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYPl-0000U9-Nq for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:02:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41970) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYN6-0005Az-KD for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:36 -0400 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]:47078) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN4-0005bB-Fq for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:36 -0400 Received: by mail-pf1-x429.google.com with SMTP id d124so19326313pfa.13 for ; Fri, 16 Apr 2021 16:59:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P4NySGYmw96BmCITsEvNWRDoN+LZscH1OmpdX1Sy41g=; b=rEqA/G9gMX0jKakzD36y14c5dEo6QzZcyqVl0AVKo00XUVSrGCGLM+ijH/kacNA+kc sRFfPWxmHpJG++Y6VDwd3NlGfY9WsNnEKyfSAU+KQlMtZBMzXIWMQ5EjU7Rs5zhJvj2W 1Bm0yq759KVfTKts4YSivCJrqewnwaV6c4FwQW/oIBiA/6z2+rkADbpmLaX6uw5FTvJl kKtVx+QtzbbB9LBHt/hNFVoerI1Z44V02SYT0zuKaDyih/WHB7CTG3qrwg0aOyDIq5op 87i3GBdCbHZDYZvMC+Teb+7gkjZhaNAxhhj+oWHTERjd4a8TD3Ly0IxcDR1SB6w+qXak 8LuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P4NySGYmw96BmCITsEvNWRDoN+LZscH1OmpdX1Sy41g=; b=A5yZ5Fn/S6B45r+HS5jx9DEG24D/aZQZbKqKLo7Ot3CYXiqFmyb07xUUutGn3HERsl n+0qn1Xcys1ufzT7ud8uHOIpOreFeM09jM9OAuj3lvMHppBgvZsTlS42tGgyNoF3j4jr cF+M6JWNH3qKO7aen8DphkEEdMqy44FySKQCw1Xuwd4qJBBCKRuADKg9Alh//ar1lKkY DlgJx7JrZxir4hGmiNlOvA0vsuKLF+0fHUwkroDXIXSW1WxGLW8J9On5INJeDG3pTxMW ODz7UQw5EUC2LYs5KLYucfomc1//sIlMn4mayMZXe29y6PlwGPH9FXLY8GsLPAkiUFAc 207Q== X-Gm-Message-State: AOAM532WMF0wQmBEhZycYzvaVaUAaPt/lZlp2iDwYe9I6fBrhAhYB5u9 80V6FIJ7W85vp59KCOtL8w/vJvvxw2T5iQ== X-Google-Smtp-Source: ABdhPJzr38zPH0RFo1SyvfOWiDgJF3N+PEDbUYOpnJb9QVRl1mTxuL/lmKrbqRrcFx3nGV1QhAZFhA== X-Received: by 2002:a63:4513:: with SMTP id s19mr1295269pga.34.1618617573307; Fri, 16 Apr 2021 16:59:33 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 03/11] target/arm: Implement scalar float32 to bfloat16 conversion Date: Fri, 16 Apr 2021 16:59:20 -0700 Message-Id: <20210416235928.1631788-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::429; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is the 64-bit BFCVT and the 32-bit VCVT{B,T}.BF16.F32. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 1 + target/arm/vfp.decode | 2 ++ target/arm/translate-a64.c | 19 +++++++++++++++++++ target/arm/vfp_helper.c | 5 +++++ target/arm/translate-vfp.c.inc | 24 ++++++++++++++++++++++++ 5 files changed, 51 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index 33df62f44d..0892207f80 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -143,6 +143,7 @@ DEF_HELPER_3(vfp_cmped, void, f64, f64, env) DEF_HELPER_2(vfp_fcvtds, f64, f32, env) DEF_HELPER_2(vfp_fcvtsd, f32, f64, env) +DEF_HELPER_FLAGS_2(bfcvt, TCG_CALL_NO_RWG, i32, f32, ptr) DEF_HELPER_2(vfp_uitoh, f16, i32, ptr) DEF_HELPER_2(vfp_uitos, f32, i32, ptr) diff --git a/target/arm/vfp.decode b/target/arm/vfp.decode index 6f7f28f9a4..52535d9b0b 100644 --- a/target/arm/vfp.decode +++ b/target/arm/vfp.decode @@ -205,6 +205,8 @@ VCVT_f64_f16 ---- 1110 1.11 0010 .... 1011 t:1 1.0 .... \ # VCVTB and VCVTT to f16: Vd format is always vd_sp; # Vm format depends on size bit +VCVT_b16_f32 ---- 1110 1.11 0011 .... 1001 t:1 1.0 .... \ + vd=%vd_sp vm=%vm_sp VCVT_f16_f32 ---- 1110 1.11 0011 .... 1010 t:1 1.0 .... \ vd=%vd_sp vm=%vm_sp VCVT_f16_f64 ---- 1110 1.11 0011 .... 1011 t:1 1.0 .... \ diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index d8ec219bb2..d767194cc7 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -6288,6 +6288,9 @@ static void handle_fp_1src_single(DisasContext *s, int opcode, int rd, int rn) case 0x3: /* FSQRT */ gen_helper_vfp_sqrts(tcg_res, tcg_op, cpu_env); goto done; + case 0x6: /* BFCVT */ + gen_fpst = gen_helper_bfcvt; + break; case 0x8: /* FRINTN */ case 0x9: /* FRINTP */ case 0xa: /* FRINTM */ @@ -6565,6 +6568,22 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn) } break; + case 0x6: + switch (type) { + case 1: /* BFCVT */ + if (!dc_isar_feature(aa64_bf16, s)) { + goto do_unallocated; + } + if (!fp_access_check(s)) { + return; + } + handle_fp_1src_single(s, opcode, rd, rn); + break; + default: + goto do_unallocated; + } + break; + default: do_unallocated: unallocated_encoding(s); diff --git a/target/arm/vfp_helper.c b/target/arm/vfp_helper.c index 01b9d8557f..fe7a2a5daa 100644 --- a/target/arm/vfp_helper.c +++ b/target/arm/vfp_helper.c @@ -408,6 +408,11 @@ float32 VFP_HELPER(fcvts, d)(float64 x, CPUARMState *env) return float64_to_float32(x, &env->vfp.fp_status); } +uint32_t HELPER(bfcvt)(float32 x, void *status) +{ + return float32_to_bfloat16(x, status); +} + /* * VFP3 fixed point conversion. The AArch32 versions of fix-to-float * must always round-to-nearest; the AArch64 ones honour the FPSCR diff --git a/target/arm/translate-vfp.c.inc b/target/arm/translate-vfp.c.inc index e20d9c7ba6..709d1fddcf 100644 --- a/target/arm/translate-vfp.c.inc +++ b/target/arm/translate-vfp.c.inc @@ -3003,6 +3003,30 @@ static bool trans_VCVT_f64_f16(DisasContext *s, arg_VCVT_f64_f16 *a) return true; } +static bool trans_VCVT_b16_f32(DisasContext *s, arg_VCVT_b16_f32 *a) +{ + TCGv_ptr fpst; + TCGv_i32 tmp; + + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + + if (!vfp_access_check(s)) { + return true; + } + + fpst = fpstatus_ptr(FPST_FPCR); + tmp = tcg_temp_new_i32(); + + vfp_load_reg32(tmp, a->vm); + gen_helper_bfcvt(tmp, tmp, fpst); + tcg_gen_st16_i32(tmp, cpu_env, vfp_f16_offset(a->vd, a->t)); + tcg_temp_free_ptr(fpst); + tcg_temp_free_i32(tmp); + return true; +} + static bool trans_VCVT_f16_f32(DisasContext *s, arg_VCVT_f16_f32 *a) { TCGv_ptr fpst; From patchwork Fri Apr 16 23:59:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467448 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=ordiiuDT; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYHX2Lxbz9sTD for ; Sat, 17 Apr 2021 10:05:10 +1000 (AEST) Received: from localhost ([::1]:45198 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYSR-0004yn-GV for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:05:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42042) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYN9-0005Hu-BS for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:39 -0400 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:37711) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN5-0005cf-IA for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:39 -0400 Received: by mail-pf1-x42a.google.com with SMTP id o123so19363735pfb.4 for ; Fri, 16 Apr 2021 16:59:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DIH2aCJSKpFkJSYIcY5Sz/FYdp4UfedtSiKpRThzUXc=; b=ordiiuDTo0r+G4KqvQQ+tSQxEsOe73DDWs9TdVCyyhY7pL++mkxu6+/QBsVscmMebo qRJTxTABedrZDWjjb8g3nwY3/faKMrsRMAEw7Cgk2FIO4Y3e4b+Ekc6M4Scg9THo12rj q2Pmt6dENOxzxVlNn81dg46ZMVy38WAZkiPZelPiGWocO5kl4WUrb48zv6RT5jxWo28z 4OLOzu3WIAWM0TO0J4sgpd/tKpyZ/ibxe5V3j0HjJdXSRy59EZVNAz7SlJYSrZRRfmi2 8BjRFM0/2grpMXQ5oUH0KUR9dNoq8CM5qW9IeN+3NO+7o3DY4fkDrtd04UVYhCGx1yPD SNWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DIH2aCJSKpFkJSYIcY5Sz/FYdp4UfedtSiKpRThzUXc=; b=m7FO6OyLPXd3vr6lDJvGqFHDWmzuqCurO1xzfSNK8nG7RxDl9FtGMYdGLRUbWSOsl3 oCPtFnJ8upM5/oUxWVnG9DxhbU6PTSrXuYZzjSQzGEYkriJBf1j5tRKZNeURLWi+Kxxr nOpRtaXkiVviNiQdxspLDvsg9BX+bzGTFmwvZ4hF6pWLgHBLYFMFUkLh/ZxmrfzS4J8k hEOMNNjbor2V2UvNPWKkdE2ljMjBtV6ddxUjhc7T3eoq22bsqc2Fcr52Fa+yV0h0ZNIX Hy0Y4tLLOlrL8B/H9jrixSssD7rVtnoqL/OrsMRN/8hR5wBpbpobkFfqvhPo7xxgljQm b5fA== X-Gm-Message-State: AOAM531/diAwxhxW6koh4dLQo0ghese/yEa89FHE07JXg8kzCnuHteEk WdCF9X3XldsGRVGxTzXCAlvdMRqrCvEvxg== X-Google-Smtp-Source: ABdhPJzm7NoPxtCZGyTUMaKaUkewcUOUUYhBfK6DMeWLcoDmLtuZigokRp6BRp5jhR0J6/gJKOKbHA== X-Received: by 2002:a62:d10c:0:b029:23e:7499:b74b with SMTP id z12-20020a62d10c0000b029023e7499b74bmr10178087pfg.46.1618617574159; Fri, 16 Apr 2021 16:59:34 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:33 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 04/11] target/arm: Implement vector float32 to bfloat16 conversion Date: Fri, 16 Apr 2021 16:59:21 -0700 Message-Id: <20210416235928.1631788-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42a; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFCVT{N,T} for both AArch64 AdvSIMD and SVE, and VCVT.BF16.F32 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper-sve.h | 4 +++ target/arm/helper.h | 1 + target/arm/neon-dp.decode | 1 + target/arm/sve.decode | 2 ++ target/arm/sve_helper.c | 2 ++ target/arm/translate-a64.c | 17 +++++++++++++ target/arm/translate-sve.c | 16 ++++++++++++ target/arm/vfp_helper.c | 7 +++++ target/arm/translate-neon.c.inc | 45 +++++++++++++++++++++++++++++++++ 9 files changed, 95 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index fa7418e706..9287e6f26c 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1197,6 +1197,8 @@ DEF_HELPER_FLAGS_5(sve_fcvt_hd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_fcvt_sd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_bfcvt, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_fcvtzs_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) @@ -2744,6 +2746,8 @@ DEF_HELPER_FLAGS_5(sve2_fcvtnt_sh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve2_fcvtnt_ds, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_bfcvtnt, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve2_fcvtlt_hs, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) diff --git a/target/arm/helper.h b/target/arm/helper.h index 0892207f80..0b52ee6256 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -144,6 +144,7 @@ DEF_HELPER_3(vfp_cmped, void, f64, f64, env) DEF_HELPER_2(vfp_fcvtds, f64, f32, env) DEF_HELPER_2(vfp_fcvtsd, f32, f64, env) DEF_HELPER_FLAGS_2(bfcvt, TCG_CALL_NO_RWG, i32, f32, ptr) +DEF_HELPER_FLAGS_2(bfcvt_pair, TCG_CALL_NO_RWG, i32, i64, ptr) DEF_HELPER_2(vfp_uitoh, f16, i32, ptr) DEF_HELPER_2(vfp_uitos, f32, i32, ptr) diff --git a/target/arm/neon-dp.decode b/target/arm/neon-dp.decode index ec83f10ab3..fd3a01bfa0 100644 --- a/target/arm/neon-dp.decode +++ b/target/arm/neon-dp.decode @@ -521,6 +521,7 @@ Vimm_1r 1111 001 . 1 . 000 ... .... cmode:4 0 . op:1 1 .... @1reg_imm VRINTZ 1111 001 11 . 11 .. 10 .... 0 1011 . . 0 .... @2misc VCVT_F16_F32 1111 001 11 . 11 .. 10 .... 0 1100 0 . 0 .... @2misc_q0 + VCVT_B16_F32 1111 001 11 . 11 .. 10 .... 0 1100 1 . 0 .... @2misc_q0 VRINTM 1111 001 11 . 11 .. 10 .... 0 1101 . . 0 .... @2misc diff --git a/target/arm/sve.decode b/target/arm/sve.decode index 3d7c4fa6e5..bad81580c5 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -987,6 +987,7 @@ FNMLS_zpzzz 01100101 .. 1 ..... 111 ... ..... ..... @rdn_pg_rm_ra # SVE floating-point convert precision FCVT_sh 01100101 10 0010 00 101 ... ..... ..... @rd_pg_rn_e0 FCVT_hs 01100101 10 0010 01 101 ... ..... ..... @rd_pg_rn_e0 +BFCVT 01100101 10 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVT_dh 01100101 11 0010 00 101 ... ..... ..... @rd_pg_rn_e0 FCVT_hd 01100101 11 0010 01 101 ... ..... ..... @rd_pg_rn_e0 FCVT_ds 01100101 11 0010 10 101 ... ..... ..... @rd_pg_rn_e0 @@ -1561,6 +1562,7 @@ RAX1 01000101 00 1 ..... 11110 1 ..... ..... @rd_rn_rm_e0 FCVTXNT_ds 01100100 00 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVTX_ds 01100101 00 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVTNT_sh 01100100 10 0010 00 101 ... ..... ..... @rd_pg_rn_e0 +BFCVTNT 01100100 10 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVTLT_hs 01100100 10 0010 01 101 ... ..... ..... @rd_pg_rn_e0 FCVTNT_ds 01100100 11 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVTLT_sd 01100100 11 0010 11 101 ... ..... ..... @rd_pg_rn_e0 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index c5c3017745..ae3db11c0d 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4570,6 +4570,7 @@ static inline uint64_t vfp_float64_to_uint64_rtz(float64 f, float_status *s) DO_ZPZ_FP(sve_fcvt_sh, uint32_t, H1_4, sve_f32_to_f16) DO_ZPZ_FP(sve_fcvt_hs, uint32_t, H1_4, sve_f16_to_f32) +DO_ZPZ_FP(sve_bfcvt, uint32_t, H1_4, float32_to_bfloat16) DO_ZPZ_FP(sve_fcvt_dh, uint64_t, , sve_f64_to_f16) DO_ZPZ_FP(sve_fcvt_hd, uint64_t, , sve_f16_to_f64) DO_ZPZ_FP(sve_fcvt_ds, uint64_t, , float64_to_float32) @@ -7567,6 +7568,7 @@ void HELPER(NAME)(void *vd, void *vn, void *vg, void *status, uint32_t desc) \ } DO_FCVTNT(sve2_fcvtnt_sh, uint32_t, uint16_t, H1_4, H1_2, sve_f32_to_f16) +DO_FCVTNT(sve_bfcvtnt, uint32_t, uint16_t, H1_4, H1_2, float32_to_bfloat16) DO_FCVTNT(sve2_fcvtnt_ds, uint64_t, uint32_t, H1_4, H1_2, float64_to_float32) #define DO_FCVTLT(NAME, TYPEW, TYPEN, HW, HN, OP) \ diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index d767194cc7..c528fb2cf0 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -10361,6 +10361,13 @@ static void handle_2misc_narrow(DisasContext *s, bool scalar, tcg_temp_free_i32(ahp); } break; + case 0x36: /* BFCVTN, BFCVTN2 */ + { + TCGv_ptr fpst = fpstatus_ptr(FPST_FPCR); + gen_helper_bfcvt_pair(tcg_res[pass], tcg_op, fpst); + tcg_temp_free_ptr(fpst); + } + break; case 0x56: /* FCVTXN, FCVTXN2 */ /* 64 bit to 32 bit float conversion * with von Neumann rounding (round to odd) @@ -12761,6 +12768,16 @@ static void disas_simd_two_reg_misc(DisasContext *s, uint32_t insn) } handle_2misc_narrow(s, false, opcode, 0, is_q, size - 1, rn, rd); return; + case 0x36: /* BFCVTN, BFCVTN2 */ + if (!dc_isar_feature(aa64_bf16, s) || size != 2) { + unallocated_encoding(s); + return; + } + if (!fp_access_check(s)) { + return; + } + handle_2misc_narrow(s, false, opcode, 0, is_q, size - 1, rn, rd); + return; case 0x17: /* FCVTL, FCVTL2 */ if (!fp_access_check(s)) { return; diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index cb0e7a1f68..aacbabd11e 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4715,6 +4715,14 @@ static bool trans_FCVT_hs(DisasContext *s, arg_rpr_esz *a) return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_hs); } +static bool trans_BFCVT(DisasContext *s, arg_rpr_esz *a) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_bfcvt); +} + static bool trans_FCVT_dh(DisasContext *s, arg_rpr_esz *a) { return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_dh); @@ -8405,6 +8413,14 @@ static bool trans_FCVTNT_sh(DisasContext *s, arg_rpr_esz *a) return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve2_fcvtnt_sh); } +static bool trans_BFCVTNT(DisasContext *s, arg_rpr_esz *a) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_bfcvtnt); +} + static bool trans_FCVTNT_ds(DisasContext *s, arg_rpr_esz *a) { if (!dc_isar_feature(aa64_sve2, s)) { diff --git a/target/arm/vfp_helper.c b/target/arm/vfp_helper.c index fe7a2a5daa..3328423cec 100644 --- a/target/arm/vfp_helper.c +++ b/target/arm/vfp_helper.c @@ -413,6 +413,13 @@ uint32_t HELPER(bfcvt)(float32 x, void *status) return float32_to_bfloat16(x, status); } +uint32_t HELPER(bfcvt_pair)(uint64_t pair, void *status) +{ + bfloat16 lo = float32_to_bfloat16(extract64(pair, 0, 32), status); + bfloat16 hi = float32_to_bfloat16(extract64(pair, 32, 32), status); + return deposit32(lo, 16, 16, hi); +} + /* * VFP3 fixed point conversion. The AArch32 versions of fix-to-float * must always round-to-nearest; the AArch64 ones honour the FPSCR diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index f1893b1dc8..8cc53892d6 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -3413,6 +3413,51 @@ static bool trans_VSHLL(DisasContext *s, arg_2misc *a) return true; } +static bool trans_VCVT_B16_F32(DisasContext *s, arg_2misc *a) +{ + TCGv_ptr fpst; + TCGv_i64 tmp; + TCGv_i32 dst0, dst1; + + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + + /* UNDEF accesses to D16-D31 if they don't exist. */ + if (!dc_isar_feature(aa32_simd_r32, s) && + ((a->vd | a->vm) & 0x10)) { + return false; + } + + if ((a->vm & 1) || (a->size != 1)) { + return false; + } + + if (!vfp_access_check(s)) { + return true; + } + + fpst = fpstatus_ptr(FPST_STD); + tmp = tcg_temp_new_i64(); + dst0 = tcg_temp_new_i32(); + dst1 = tcg_temp_new_i32(); + + read_neon_element64(tmp, a->vm, 0, MO_64); + gen_helper_bfcvt_pair(dst0, tmp, fpst); + + read_neon_element64(tmp, a->vm, 1, MO_64); + gen_helper_bfcvt_pair(dst1, tmp, fpst); + + write_neon_element32(dst0, a->vd, 0, MO_32); + write_neon_element32(dst1, a->vd, 1, MO_32); + + tcg_temp_free_i64(tmp); + tcg_temp_free_i32(dst0); + tcg_temp_free_i32(dst1); + tcg_temp_free_ptr(fpst); + return true; +} + static bool trans_VCVT_F16_F32(DisasContext *s, arg_2misc *a) { TCGv_ptr fpst; From patchwork Fri Apr 16 23:59:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467445 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=EZeWxHLv; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYFW1nfqz9sW1 for ; Sat, 17 Apr 2021 10:03:27 +1000 (AEST) Received: from localhost ([::1]:38752 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYQn-0002Kq-9P for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:03:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYN8-0005Fg-8r for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:38 -0400 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]:43621) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN6-0005dU-I9 for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:38 -0400 Received: by mail-pg1-x531.google.com with SMTP id p12so20224138pgj.10 for ; Fri, 16 Apr 2021 16:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=doLgHy4UzU3or1oUi8sNNPd0558SCSQfVTW0p961kLw=; b=EZeWxHLvTsBDaWu7zXTaKI89eMrv/T79BdmHiBX3R5TaCyO5XBy7KJKmLklC9XXEt4 i3JC8BB2eCJU73BQM2RndbYyZPyYCE8yifnNGtVQW2vD5CisAxRYKfJ6/pf11OMKxsxm ki+PA8SsiHU1w5qpDRF5zxJE7AIzgUmsFJ/jESKGaVmMmG3TJnBsaB5CfGvPJ1qU1RgB aJ5JYD+2oKyec4NX48KY/nI/ozWMJ5gMpaerqK45B5gstnm6bOxJPb+qw2+NFHnCScp2 orsWMe4k0LGZiQnZcc94cuOwgkBUXnEe5gX3K+wW7ZrKsBPKzORtALAXHIFk8/3kiYM9 hkgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=doLgHy4UzU3or1oUi8sNNPd0558SCSQfVTW0p961kLw=; b=OUHZjmJ8M2FZMAhJ3N/TbRrPiCCJxFoOZEATn5pp0UkNVJCGtS5hF/KbPv61nr6ipp CPd+YIlxwF9mrNzH7krbZJDhm1B9yaH8bpGAwweSRUe4Wiw82MxMy36opj5qJ6RXBtT3 Ygs2++gGOR7+ESrdWbUHO70ZEO6jGOwrhz9AQCysTBol5ejAYcoBpRFw03JPM0zbxoez L2bAq6pfWCrJUTxyid4Mh4QfbLXZQGHYo3eXiNgkQRoEuoef+t/S7NJDGvI4w02bIPw+ kVs9giZF+V6KxNAqTBMfzHOXaM2pBi8neXyAC6ZPjIJOlxbR2mBqXlArPDjAotMAxkzc NGxA== X-Gm-Message-State: AOAM530lnWtZqvJrra1JIkYfW0Lt47jPQ9j1Uuhx4pIBcNyILSPakOCf LPiJvgOH/jqU7fZUuTA+9uvXEic+bZQdOw== X-Google-Smtp-Source: ABdhPJw5MHZQ9NGLeRxxtdEVy9XgG3INpbvlKnq8ic5rFtHhqp+wozK2rwv3ZbEJVDrjXP3Xs/o+CQ== X-Received: by 2002:a62:52c7:0:b029:255:e78e:5069 with SMTP id g190-20020a6252c70000b0290255e78e5069mr9612330pfb.45.1618617575158; Fri, 16 Apr 2021 16:59:35 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 05/11] fpu: Add float_round_to_odd_inf Date: Fri, 16 Apr 2021 16:59:22 -0700 Message-Id: <20210416235928.1631788-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::531; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x531.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" For Arm BFDOT and BFMMLA, we need a version of round-to-odd that overflows to infinity, instead of the max normal number. Signed-off-by: Richard Henderson --- include/fpu/softfloat-types.h | 4 +++- fpu/softfloat.c | 8 ++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/include/fpu/softfloat-types.h b/include/fpu/softfloat-types.h index 8a3f20fae9..3b757c3d6a 100644 --- a/include/fpu/softfloat-types.h +++ b/include/fpu/softfloat-types.h @@ -134,8 +134,10 @@ typedef enum __attribute__((__packed__)) { float_round_up = 2, float_round_to_zero = 3, float_round_ties_away = 4, - /* Not an IEEE rounding mode: round to the closest odd mantissa value */ + /* Not an IEEE rounding mode: round to closest odd, overflow to max */ float_round_to_odd = 5, + /* Not an IEEE rounding mode: round to closest odd, overflow to inf */ + float_round_to_odd_inf = 6, } FloatRoundMode; /* diff --git a/fpu/softfloat.c b/fpu/softfloat.c index 67cfa0fd82..76097679b0 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -694,13 +694,12 @@ static FloatParts round_canonical(FloatParts p, float_status *s, switch (p.cls) { case float_class_normal: + overflow_norm = false; switch (s->float_rounding_mode) { case float_round_nearest_even: - overflow_norm = false; inc = ((frac & roundeven_mask) != frac_lsbm1 ? frac_lsbm1 : 0); break; case float_round_ties_away: - overflow_norm = false; inc = frac_lsbm1; break; case float_round_to_zero: @@ -717,6 +716,8 @@ static FloatParts round_canonical(FloatParts p, float_status *s, break; case float_round_to_odd: overflow_norm = true; + /* fall through */ + case float_round_to_odd_inf: inc = frac & frac_lsb ? 0 : round_mask; break; default: @@ -771,6 +772,7 @@ static FloatParts round_canonical(FloatParts p, float_status *s, ? frac_lsbm1 : 0); break; case float_round_to_odd: + case float_round_to_odd_inf: inc = frac & frac_lsb ? 0 : round_mask; break; default: @@ -6860,6 +6862,8 @@ float128 float128_round_to_int(float128 a, float_status *status) case float_round_to_zero: break; + default: + g_assert_not_reached(); } return packFloat128( aSign, 0, 0, 0 ); } From patchwork Fri Apr 16 23:59:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467447 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=z8e1r8Z2; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYGf4q4Zz9sTD for ; Sat, 17 Apr 2021 10:04:26 +1000 (AEST) Received: from localhost ([::1]:43298 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYRk-0004A1-MO for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:04:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42140) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYND-0005RK-0l for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:43 -0400 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]:37617) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN8-0005gC-G1 for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:42 -0400 Received: by mail-pj1-x1034.google.com with SMTP id e8-20020a17090a7288b029014e51f5a6baso10231716pjg.2 for ; Fri, 16 Apr 2021 16:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/PkbqLegCo8bMrZ94sGp7SVbjSmm25utWP6wUjeAjjU=; b=z8e1r8Z2IhJqGbH0vHVkeT7GQmTYPO/x7qXn6P1nDOVXkHob/WnS0d7MwWyTAoT/Er PYMGJ8Y58Dv/2ygpJFhGBu/78qqQQ6ShPTL5FdRy2MH46QwYxdEPEcwXeDygHZvl1OSk wSoMAezAh7Mxrz9VgE5aZd8SLwLk2VAgdA1+u8ZAA7Q9XrhObUtEgTZLINJU3JZjOkiF ZJbdvfnWdPZMUxGs92542iLvCK1NjfTmbmX5w99zSe117AxIyHI9zA6/obtoG+p2D84T AtfxvHIQY2qUL8y9bO/zyHQdVUUhKIPu483Y3VtQoEiOSTKEWXqRYpCjU3ybB35IFqwk wLVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/PkbqLegCo8bMrZ94sGp7SVbjSmm25utWP6wUjeAjjU=; b=oGv6sW2zmeqgGmaptI08GT4au8GYe5zogUxphiRqthNdcD2vW/36LcuKeOcyYGeYZB OHrXZx2upA76aR0OfJI/70bB1+Xua3MVwJQ2lkcoiX2EftNkkO2bVZhyPeea9BdX7XFJ GxS9U9lROtKT2tklJ3y6jpH8Owj6VfxFz/lttAwrAOepE/pe5Cq1LZ8L1Bhl29Ox0MbQ c3Y1TQp9EBL0FszlKMaP1x2QT3qc633qkjUlc1dRHrKpG/2SdeJzdT1gmKT6j3RAyT/B fjjanXCWSYUjmnwFYyIF+lJtwsMlk3xI9hotKmf7rpzjw+E5u4Ehwp764o4v2j6jE4RE bpKg== X-Gm-Message-State: AOAM5335cjVwcXgRCXdjGZg+QSPfwMVK0SLNq2j4RzGacaP4KIjvwXk/ /5Wf0B93lS3ID+iCZ5y5Z55g3cRKnnC28A== X-Google-Smtp-Source: ABdhPJyafcEVuKDdy6zsYM1f0WSdb3kJ3pzbJWIJ22RSHWOzTGHBkeXcey/0s8jPc5dKUwsxzTzr0A== X-Received: by 2002:a17:902:122:b029:e8:bde2:7f6c with SMTP id 31-20020a1709020122b02900e8bde27f6cmr12055573plb.29.1618617576175; Fri, 16 Apr 2021 16:59:36 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:35 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 06/11] target/arm: Implement bfloat16 dot product (vector) Date: Fri, 16 Apr 2021 16:59:23 -0700 Message-Id: <20210416235928.1631788-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1034; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1034.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFDOT for both AArch64 AdvSIMD and SVE, and VDOT.BF16 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 3 +++ target/arm/neon-shared.decode | 2 ++ target/arm/sve.decode | 3 +++ target/arm/translate-a64.c | 20 +++++++++++++++++ target/arm/translate-sve.c | 12 ++++++++++ target/arm/vec_helper.c | 40 +++++++++++++++++++++++++++++++++ target/arm/translate-neon.c.inc | 9 ++++++++ 7 files changed, 89 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index 0b52ee6256..eb4cb2b65b 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -1003,6 +1003,9 @@ DEF_HELPER_FLAGS_5(gvec_ummla_b, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_usmmla_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + #ifdef TARGET_AARCH64 #include "helper-a64.h" #include "helper-sve.h" diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode index cc9f4cdd85..31a0839bbb 100644 --- a/target/arm/neon-shared.decode +++ b/target/arm/neon-shared.decode @@ -52,6 +52,8 @@ VUDOT 1111 110 00 . 10 .... .... 1101 . q:1 . 1 .... \ vm=%vm_dp vn=%vn_dp vd=%vd_dp VUSDOT 1111 110 01 . 10 .... .... 1101 . q:1 . 0 .... \ vm=%vm_dp vn=%vn_dp vd=%vd_dp +VDOT_b16 1111 110 00 . 00 .... .... 1101 . q:1 . 0 .... \ + vm=%vm_dp vn=%vn_dp vd=%vd_dp # VFM[AS]L VFML 1111 110 0 s:1 . 10 .... .... 1000 . 0 . 1 .... \ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index bad81580c5..523140ca56 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -1576,6 +1576,9 @@ FMLALT_zzzw 01100100 10 1 ..... 10 0 00 1 ..... ..... @rda_rn_rm_e0 FMLSLB_zzzw 01100100 10 1 ..... 10 1 00 0 ..... ..... @rda_rn_rm_e0 FMLSLT_zzzw 01100100 10 1 ..... 10 1 00 1 ..... ..... @rda_rn_rm_e0 +### SVE2 floating-point bfloat16 dot-product +BFDOT_zzzz 01100100 01 1 ..... 10 0 00 0 ..... ..... @rda_rn_rm_e0 + ### SVE2 floating-point multiply-add long (indexed) FMLALB_zzxw 01100100 10 1 ..... 0100.0 ..... ..... @rrxr_3a esz=2 FMLALT_zzxw 01100100 10 1 ..... 0100.1 ..... ..... @rrxr_3a esz=2 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index c528fb2cf0..fc16e0a126 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -12243,6 +12243,16 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } feature = dc_isar_feature(aa64_fcma, s); break; + case 0x1f: /* BFDOT */ + switch (size) { + case 1: + feature = dc_isar_feature(aa64_bf16, s); + break; + default: + unallocated_encoding(s); + return; + } + break; default: unallocated_encoding(s); return; @@ -12326,6 +12336,16 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } return; + case 0xf: /* BFDOT */ + switch (size) { + case 1: + gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfdot); + break; + default: + g_assert_not_reached(); + } + return; + default: g_assert_not_reached(); } diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index aacbabd11e..3527430c1a 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -8586,3 +8586,15 @@ static bool trans_UMMLA(DisasContext *s, arg_rrrr_esz *a) { return do_i8mm_zzzz_ool(s, a, gen_helper_gvec_ummla_b, 0); } + +static bool trans_BFDOT_zzzz(DisasContext *s, arg_rrrr_esz *a) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + if (sve_access_check(s)) { + gen_gvec_ool_zzzz(s, gen_helper_gvec_bfdot, + a->rd, a->rn, a->rm, a->ra, 0); + } + return true; +} diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index 6c9f1e5146..e227ba6590 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -2655,3 +2655,43 @@ static void do_mmla_b(void *vd, void *vn, void *vm, void *va, uint32_t desc, DO_MMLA_B(gvec_smmla_b, do_smmla_b) DO_MMLA_B(gvec_ummla_b, do_ummla_b) DO_MMLA_B(gvec_usmmla_b, do_usmmla_b) + +/* + * BFloat16 Dot Product + */ + +static float32 bfdotadd(float32 sum, uint32_t e1, uint32_t e2) +{ + /* FPCR is ignored for BFDOT and BFMMLA. */ + float_status bf_status = { + .tininess_before_rounding = float_tininess_before_rounding, + .float_rounding_mode = float_round_to_odd_inf, + .flush_to_zero = true, + .flush_inputs_to_zero = true, + .default_nan_mode = true, + }; + float32 t1, t2; + + /* + * Extract each BFloat16 from the element pair, and shift + * them such that they become float32. + */ + t1 = float32_mul(e1 << 16, e2 << 16, &bf_status); + t2 = float32_mul(e1 & 0xffff0000u, e2 & 0xffff0000u, &bf_status); + t1 = float32_add(t1, t2, &bf_status); + t1 = float32_add(sum, t1, &bf_status); + + return t1; +} + +void HELPER(gvec_bfdot)(void *vd, void *vn, void *vm, void *va, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + float32 *d = vd, *a = va; + uint32_t *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 4; ++i) { + d[i] = bfdotadd(a[i], n[i], m[i]); + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index 8cc53892d6..aed8a565e0 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -287,6 +287,15 @@ static bool trans_VUSDOT(DisasContext *s, arg_VUSDOT *a) gen_helper_gvec_usdot_b); } +static bool trans_VDOT_b16(DisasContext *s, arg_VDOT_b16 *a) +{ + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + return do_neon_ddda(s, a->q * 7, a->vd, a->vn, a->vm, 0, + gen_helper_gvec_bfdot); +} + static bool trans_VFML(DisasContext *s, arg_VFML *a) { int opr_sz; From patchwork Fri Apr 16 23:59:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467450 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=Icahff+1; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYLd73DMz9sTD for ; Sat, 17 Apr 2021 10:07:53 +1000 (AEST) Received: from localhost ([::1]:51688 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYV6-0007t7-1N for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:07:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42158) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYND-0005TF-MW for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:43 -0400 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]:34639) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN8-0005gE-NR for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:43 -0400 Received: by mail-pf1-x42e.google.com with SMTP id 10so10477405pfl.1 for ; Fri, 16 Apr 2021 16:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DIBFt/LCLxrjw+4C/O85MFKcqi0nnU8EEvk4pxFd2ic=; b=Icahff+1pWVKIyt5SA6kD0/DgQqpz3fJ6gL81IjFrr3BNKCGIL/sWcYvCW9rHzbqO7 vV5x13ZYs/yNjrKy8nSnQsGtIQZJM+L92w5sa7crPyGanv/dw/E+9tSWUPoVh6FK0ssd eOHyGTYQsHI3xKntBZ3g4rxXRsSvXKovKSjM6LAL8B95lso/Ws1bZKFic8dzvN0cmgM4 fq3zysKuyeyObV5PHJqBXxsdWnjkaX9f9U9gYh6jDfNi1kw0ayI40pCqgxs6HgwvW36j Vy26cyvo8MxVRmQRe2yhX7ldXSBJgZXZsrKw8XlRbD0upYqkp8TdsyttGL6VmyJrOFzJ Aw0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DIBFt/LCLxrjw+4C/O85MFKcqi0nnU8EEvk4pxFd2ic=; b=ANimyD2UlKlOJGOKMjNfnM0/O0yyBZJGAkZlxGaPu7OgTssho7er7u7Ni3ud3KKtNL J952Gz82c5PQeJ/Yy5OFc+e61U/gNoYO7RFfUZVUMliQKrklRxAj27+dQa49590tWbEh 0+ywZsnpJ61De9mYuhqKstjyGCMLZzw4hkh08XhDkb5Vo0NiwzH6Tz1/7/ojjz4xYkdo D2q7lXSAbLfYDO3Wyu7085jHV2MbDqXZt7XGJBiQZgn2+lK5PA6aRNdcSP4srjsREcti L6uNdlY6xE+l+DYgLtA+7cI94YhVb4gFzUItKePb8zniIu4MFO4CyMrl3n8YnwqtNKHj RpHw== X-Gm-Message-State: AOAM532x27Xfui1pbJ+uOM+ybAGhSOGbAsav1RF3DTiFgfwFCIJ0CoOa d2Akq6xftCbEZkH98DG7/l5zLqP7UPs48w== X-Google-Smtp-Source: ABdhPJy8GPnBQnw5PrXuiQRyhcLkGIjLXt1X09INFAlh+JmySVcTQRe/PYDEjCHDkZalTevPLTKUaA== X-Received: by 2002:aa7:9a92:0:b029:24b:f852:34d8 with SMTP id w18-20020aa79a920000b029024bf85234d8mr10048203pfi.4.1618617577206; Fri, 16 Apr 2021 16:59:37 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 07/11] target/arm: Implement bfloat16 dot product (indexed) Date: Fri, 16 Apr 2021 16:59:24 -0700 Message-Id: <20210416235928.1631788-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42e; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFDOT for both AArch64 AdvSIMD and SVE, and VDOT.BF16 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 2 ++ target/arm/neon-shared.decode | 2 ++ target/arm/sve.decode | 3 +++ target/arm/translate-a64.c | 41 +++++++++++++++++++++++++-------- target/arm/translate-sve.c | 12 ++++++++++ target/arm/vec_helper.c | 20 ++++++++++++++++ target/arm/translate-neon.c.inc | 9 ++++++++ 7 files changed, 80 insertions(+), 9 deletions(-) diff --git a/target/arm/helper.h b/target/arm/helper.h index eb4cb2b65b..af0ee8f693 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -1005,6 +1005,8 @@ DEF_HELPER_FLAGS_5(gvec_usmmla_b, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) #ifdef TARGET_AARCH64 #include "helper-a64.h" diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode index 31a0839bbb..fa3cf14e3a 100644 --- a/target/arm/neon-shared.decode +++ b/target/arm/neon-shared.decode @@ -81,6 +81,8 @@ VUSDOT_scalar 1111 1110 1 . 00 .... .... 1101 . q:1 index:1 0 vm:4 \ vn=%vn_dp vd=%vd_dp VSUDOT_scalar 1111 1110 1 . 00 .... .... 1101 . q:1 index:1 1 vm:4 \ vn=%vn_dp vd=%vd_dp +VDOT_b16_scal 1111 1110 0 . 00 .... .... 1101 . q:1 index:1 0 vm:4 \ + vn=%vn_dp vd=%vd_dp %vfml_scalar_q0_rm 0:3 5:1 %vfml_scalar_q1_index 5:1 3:1 diff --git a/target/arm/sve.decode b/target/arm/sve.decode index 523140ca56..d5e1e5d400 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -1584,3 +1584,6 @@ FMLALB_zzxw 01100100 10 1 ..... 0100.0 ..... ..... @rrxr_3a esz=2 FMLALT_zzxw 01100100 10 1 ..... 0100.1 ..... ..... @rrxr_3a esz=2 FMLSLB_zzxw 01100100 10 1 ..... 0110.0 ..... ..... @rrxr_3a esz=2 FMLSLT_zzxw 01100100 10 1 ..... 0110.1 ..... ..... @rrxr_3a esz=2 + +### SVE2 floating-point bfloat16 dot-product (indexed) +BFDOT_zzxz 01100100 01 1 ..... 010000 ..... ..... @rrxr_2 esz=2 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index fc16e0a126..f60afbbd06 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -13457,8 +13457,22 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) return; } break; - case 0x0f: /* SUDOT, USDOT */ - if (is_scalar || (size & 1) || !dc_isar_feature(aa64_i8mm, s)) { + case 0x0f: + switch (size) { + case 0: /* SUDOT */ + case 2: /* USDOT */ + if (is_scalar || !dc_isar_feature(aa64_i8mm, s)) { + unallocated_encoding(s); + return; + } + break; + case 1: /* BFDOT */ + if (is_scalar || !dc_isar_feature(aa64_bf16, s)) { + unallocated_encoding(s); + return; + } + break; + default: unallocated_encoding(s); return; } @@ -13578,13 +13592,22 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) u ? gen_helper_gvec_udot_idx_b : gen_helper_gvec_sdot_idx_b); return; - case 0x0f: /* SUDOT, USDOT */ - gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index, - extract32(insn, 23, 1) - ? gen_helper_gvec_usdot_idx_b - : gen_helper_gvec_sudot_idx_b); - return; - + case 0x0f: + switch (extract32(insn, 22, 2)) { + case 0: /* SUDOT */ + gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index, + gen_helper_gvec_sudot_idx_b); + return; + case 1: /* BFDOT */ + gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index, + gen_helper_gvec_bfdot_idx); + return; + case 2: /* USDOT */ + gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index, + gen_helper_gvec_usdot_idx_b); + return; + } + g_assert_not_reached(); case 0x11: /* FCMLA #0 */ case 0x13: /* FCMLA #90 */ case 0x15: /* FCMLA #180 */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index 3527430c1a..ef6828c632 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -8598,3 +8598,15 @@ static bool trans_BFDOT_zzzz(DisasContext *s, arg_rrrr_esz *a) } return true; } + +static bool trans_BFDOT_zzxz(DisasContext *s, arg_rrxr_esz *a) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + if (sve_access_check(s)) { + gen_gvec_ool_zzzz(s, gen_helper_gvec_bfdot_idx, + a->rd, a->rn, a->rm, a->ra, a->index); + } + return true; +} diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index e227ba6590..3e26fb0e5f 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -2695,3 +2695,23 @@ void HELPER(gvec_bfdot)(void *vd, void *vn, void *vm, void *va, uint32_t desc) } clear_tail(d, opr_sz, simd_maxsz(desc)); } + +void HELPER(gvec_bfdot_idx)(void *vd, void *vn, void *vm, + void *va, uint32_t desc) +{ + intptr_t i, j, opr_sz = simd_oprsz(desc); + intptr_t index = simd_data(desc); + intptr_t elements = opr_sz / 4; + intptr_t eltspersegment = MIN(16 / 4, elements); + float32 *d = vd, *a = va; + uint32_t *n = vn, *m = vm; + + for (i = 0; i < elements; i += eltspersegment) { + uint32_t m_idx = m[i + H4(index)]; + + for (j = i; j < i + eltspersegment; j++) { + d[j] = bfdotadd(a[j], n[j], m_idx); + } + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index aed8a565e0..bb0adf4756 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -381,6 +381,15 @@ static bool trans_VSUDOT_scalar(DisasContext *s, arg_VSUDOT_scalar *a) gen_helper_gvec_sudot_idx_b); } +static bool trans_VDOT_b16_scal(DisasContext *s, arg_VDOT_b16_scal *a) +{ + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + return do_neon_ddda(s, a->q * 6, a->vd, a->vn, a->vm, a->index, + gen_helper_gvec_bfdot_idx); +} + static bool trans_VFML_scalar(DisasContext *s, arg_VFML_scalar *a) { int opr_sz; From patchwork Fri Apr 16 23:59:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467444 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=e9uMr929; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYF11GKVz9sW1 for ; Sat, 17 Apr 2021 10:03:01 +1000 (AEST) Received: from localhost ([::1]:36510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYQN-0001Rd-6K for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:02:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42108) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYNB-0005Oi-QT for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:41 -0400 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]:47083) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYN9-0005gf-GZ for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:41 -0400 Received: by mail-pf1-x42e.google.com with SMTP id d124so19326398pfa.13 for ; Fri, 16 Apr 2021 16:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fDatyCKaDyEgD6dcplvCszLIvtluHr0ddoiz3pRDaeI=; b=e9uMr929Ijw6nUD/6sXwVV/XkuysLac2/IvDdluzx8zvRmc6neFqdU5pSX3eAHbPf6 6wKN1RJ4Ci52vui8ls+NuIaXKmstCucsr24O2yl7tBzpSPNV8OjYTpG9461F31VmaBu3 tAcKLITSzIvnbjys6/QC/pYsTtgp3fEeBzNxxiY9cszNo8Z6uB9u6bXTw9BdisRQbXii cSL1hagV2bfReq3UGI0SaIvK3kHnUiikladAl44lxrd7IdjPwkziYbROtF+8M8CbX63/ EIkSxzMy1FTwz4PysPtff5Ba1DUp75SoOY7utpIdh35icJ9nGRYkpsETSah3woxdMUL1 CsBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fDatyCKaDyEgD6dcplvCszLIvtluHr0ddoiz3pRDaeI=; b=lXKetHFJeX6seqc7vfRsVCaclLa5oFwUzzDsHnjyEfkccUO/rv6MbJO4KcDF7SgA1r htVWmm0k6kmKW+oWWpZUHO30wUgnVZO5UchfFR5bF/Wh/kXPiZZcwmPNb+Tf3qucqrSv l1nuFA+VuM3azF5w9wCGRDh0kOVveAkKPurw+BjT2pQ+jIeMYNhKejAsXvE1OKWqjJAs VLLkyxA4pH2pnaKeJChjTwXg41+YSB1fH/Wwe9xA37l1CClXpwKG5Eri3PhZZyXnQjwh u9yOlTU29H+v/D8voBBqIxpRvrgAZGB1VBgTMytRhwF9roY1kR9sulTaWBla7w1BUXN+ u2yA== X-Gm-Message-State: AOAM533BEDySwDyGup0ESkd2np+NjnC5tFo1OIG6R6IBj/QSUeNFYi8x A7fgCowNWlR8verEy+PBkhsJB1k+OvlQPg== X-Google-Smtp-Source: ABdhPJzmoZcU6rB5QRle/0B2W00FdYjIUEc9yiGvDIwBsQ6XwPELGp9vufmSz6FTZ7pMtKZd4YJEHw== X-Received: by 2002:a62:824c:0:b029:21b:66f5:c813 with SMTP id w73-20020a62824c0000b029021b66f5c813mr9601350pfd.32.1618617578223; Fri, 16 Apr 2021 16:59:38 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 08/11] target/arm: Implement bfloat16 matrix multiply accumulate Date: Fri, 16 Apr 2021 16:59:25 -0700 Message-Id: <20210416235928.1631788-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42e; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFMMLA for both AArch64 AdvSIMD and SVE, and VMMLA.BF16 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 3 +++ target/arm/neon-shared.decode | 2 ++ target/arm/sve.decode | 6 +++-- target/arm/translate-a64.c | 10 +++++++++ target/arm/translate-sve.c | 12 ++++++++++ target/arm/vec_helper.c | 40 +++++++++++++++++++++++++++++++++ target/arm/translate-neon.c.inc | 9 ++++++++ 7 files changed, 80 insertions(+), 2 deletions(-) diff --git a/target/arm/helper.h b/target/arm/helper.h index af0ee8f693..74f8bc766f 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -1008,6 +1008,9 @@ DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + #ifdef TARGET_AARCH64 #include "helper-a64.h" #include "helper-sve.h" diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode index fa3cf14e3a..4e0a25d27c 100644 --- a/target/arm/neon-shared.decode +++ b/target/arm/neon-shared.decode @@ -67,6 +67,8 @@ VUMMLA 1111 1100 0.10 .... .... 1100 .1.1 .... \ vm=%vm_dp vn=%vn_dp vd=%vd_dp VUSMMLA 1111 1100 1.10 .... .... 1100 .1.0 .... \ vm=%vm_dp vn=%vn_dp vd=%vd_dp +VMMLA_b16 1111 1100 0.00 .... .... 1100 .1.0 .... \ + vm=%vm_dp vn=%vn_dp vd=%vd_dp VCMLA_scalar 1111 1110 0 . rot:2 .... .... 1000 . q:1 index:1 0 vm:4 \ vn=%vn_dp vd=%vd_dp size=1 diff --git a/target/arm/sve.decode b/target/arm/sve.decode index d5e1e5d400..aa8d5e4b8f 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -1519,8 +1519,10 @@ SQRDCMLAH_zzzz 01000100 esz:2 0 rm:5 0011 rot:2 rn:5 rd:5 ra=%reg_movprfx USDOT_zzzz 01000100 .. 0 ..... 011 110 ..... ..... @rda_rn_rm ### SVE2 floating point matrix multiply accumulate - -FMMLA 01100100 .. 1 ..... 111001 ..... ..... @rda_rn_rm +{ + BFMMLA 01100100 01 1 ..... 111 001 ..... ..... @rda_rn_rm_e0 + FMMLA 01100100 .. 1 ..... 111 001 ..... ..... @rda_rn_rm +} ### SVE2 Memory Gather Load Group diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index f60afbbd06..8636eac4a8 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -12243,6 +12243,13 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } feature = dc_isar_feature(aa64_fcma, s); break; + case 0x1d: /* BFMMLA */ + if (size != MO_16 || !is_q) { + unallocated_encoding(s); + return; + } + feature = dc_isar_feature(aa64_bf16, s); + break; case 0x1f: /* BFDOT */ switch (size) { case 1: @@ -12336,6 +12343,9 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } return; + case 0xd: /* BFMMLA */ + gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfmmla); + return; case 0xf: /* BFDOT */ switch (size) { case 1: diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index ef6828c632..9ade521705 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -8610,3 +8610,15 @@ static bool trans_BFDOT_zzxz(DisasContext *s, arg_rrxr_esz *a) } return true; } + +static bool trans_BFMMLA(DisasContext *s, arg_rrrr_esz *a) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + if (sve_access_check(s)) { + gen_gvec_ool_zzzz(s, gen_helper_gvec_bfmmla, + a->rd, a->rn, a->rm, a->ra, 0); + } + return true; +} diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index 3e26fb0e5f..623a0872f3 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -2715,3 +2715,43 @@ void HELPER(gvec_bfdot_idx)(void *vd, void *vn, void *vm, } clear_tail(d, opr_sz, simd_maxsz(desc)); } + +void HELPER(gvec_bfmmla)(void *vd, void *vn, void *vm, void *va, uint32_t desc) +{ + intptr_t s, opr_sz = simd_oprsz(desc); + float32 *d = vd, *a = va; + uint32_t *n = vn, *m = vm; + + for (s = 0; s < opr_sz / 4; s += 4) { + float32 sum00, sum01, sum10, sum11; + + /* + * Process the entire segment at once, writing back the + * results only after we've consumed all of the inputs. + * + * Key to indicies by column: + * i j i k j k + */ + sum00 = a[s + H4(0 + 0)]; + sum00 = bfdotadd(sum00, n[s + H4(0 + 0)], m[s + H4(0 + 0)]); + sum00 = bfdotadd(sum00, n[s + H4(0 + 1)], m[s + H4(0 + 1)]); + + sum01 = a[s + H4(0 + 1)]; + sum01 = bfdotadd(sum01, n[s + H4(0 + 0)], m[s + H4(2 + 0)]); + sum01 = bfdotadd(sum01, n[s + H4(0 + 1)], m[s + H4(2 + 1)]); + + sum10 = a[s + H4(2 + 0)]; + sum10 = bfdotadd(sum10, n[s + H4(2 + 0)], m[s + H4(0 + 0)]); + sum10 = bfdotadd(sum10, n[s + H4(2 + 1)], m[s + H4(0 + 1)]); + + sum11 = a[s + H4(2 + 1)]; + sum11 = bfdotadd(sum11, n[s + H4(2 + 0)], m[s + H4(2 + 0)]); + sum11 = bfdotadd(sum11, n[s + H4(2 + 1)], m[s + H4(2 + 1)]); + + d[s + H4(0 + 0)] = sum00; + d[s + H4(0 + 1)] = sum01; + d[s + H4(2 + 0)] = sum10; + d[s + H4(2 + 1)] = sum11; + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index bb0adf4756..7ce65f691f 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -4117,3 +4117,12 @@ static bool trans_VUSMMLA(DisasContext *s, arg_VUSMMLA *a) return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0, gen_helper_gvec_usmmla_b); } + +static bool trans_VMMLA_b16(DisasContext *s, arg_VMMLA_b16 *a) +{ + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0, + gen_helper_gvec_bfmmla); +} From patchwork Fri Apr 16 23:59:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467449 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=kT0LB1G5; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYKC097Qz9sTD for ; Sat, 17 Apr 2021 10:06:39 +1000 (AEST) Received: from localhost ([::1]:49764 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYTt-00071o-0K for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:06:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYND-0005T0-Iw for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:43 -0400 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]:33552) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYNA-0005hC-FN for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:43 -0400 Received: by mail-pf1-x42b.google.com with SMTP id h11so1712876pfn.0 for ; Fri, 16 Apr 2021 16:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lVASvtsBblbStSZNmfD5xxhqdPNt/cEmCqHJFLMXkbU=; b=kT0LB1G54jAa8Yn10rPBR9qz5Mb6gGVMh08NZg5ypwrvmDZA7WV2SUBkJWc1HaZoU1 WBSuv/AuqgZs75/PldwqNqqGELDI+wSVI87buQsY7W5a35lJgNaO5RJ3S0FY86sgGnS2 koHVDVvgbfnhdaGquecpbfhMu+Wh0Np8pxqnU4UmwzSatK1j0Cx45VA+ClAWP7Mi+bWM tTDK0U7j7m5IQFxLR8NcA4DG6TThF1CAf42dghfV7jh4jCsyTt7p6LrJ3XqU+5OWyh75 SCXy7xcWj5tY8a95GEKPdCIyYjKOcAY6S/OJ623l25h74f6jbBJPdkbSPITQw0cpC1AD PF/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lVASvtsBblbStSZNmfD5xxhqdPNt/cEmCqHJFLMXkbU=; b=r58ws/6qsdDSkL1LQDlaHkK8muDzS+kfDWEIWZ1DU8nyxKlpKKGfa94h+0pEQmxwZ5 qs0lWTRTAuMsrAi+lIj3nWEfv5Ks7cgNl2ApjYWSS7At6Hik9H5vuusLA+psSLJqr3hI sLELjsmMCZxdixsWfSOpOoIa4HZesDWP7tGUSspzzCHWCyA3DfMYI5POH2uuL7a/hkPj Iix75udJPRzICvAiyuy9smO4gtDJ+C3tIcwUxCK5xre0D/eNSF3FmjPoxlVAl/LWT9GF 7U3LPbPHX4+aBGmpVdvy2YNRxNNOmdri7MzX+X/EFsyD9vM6awvBz7oc/dGt0QvYjksc swGw== X-Gm-Message-State: AOAM530DCrdADsriWSLS2Iq0eKI87payjR3KtZgcyNwfUx1GD1Jm38aG ylu/dtZbxczDcECwui3rzPAsZErHi0u0iw== X-Google-Smtp-Source: ABdhPJw5m/6dVrdloBHVCZJfYBv1nPjmsiHgIQ1hEde2vzWLmkdhGv7LSz1NtmHkxYkytJf6n5gkNg== X-Received: by 2002:a63:1347:: with SMTP id 7mr1269321pgt.112.1618617579236; Fri, 16 Apr 2021 16:59:39 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:38 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 09/11] target/arm: Implement bfloat widening fma (vector) Date: Fri, 16 Apr 2021 16:59:26 -0700 Message-Id: <20210416235928.1631788-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42b; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFMLAL{B,T} for both AArch64 AdvSIMD and SVE, and VFMA{B,T}.BF16 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 3 +++ target/arm/neon-shared.decode | 3 +++ target/arm/sve.decode | 3 +++ target/arm/translate-a64.c | 13 +++++++++---- target/arm/translate-sve.c | 30 ++++++++++++++++++++++++++++++ target/arm/vec_helper.c | 16 ++++++++++++++++ target/arm/translate-neon.c.inc | 9 +++++++++ 7 files changed, 73 insertions(+), 4 deletions(-) diff --git a/target/arm/helper.h b/target/arm/helper.h index 74f8bc766f..2c6f0cecfa 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -1011,6 +1011,9 @@ DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(gvec_bfmlal, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + #ifdef TARGET_AARCH64 #include "helper-a64.h" #include "helper-sve.h" diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode index 4e0a25d27c..b61addd98b 100644 --- a/target/arm/neon-shared.decode +++ b/target/arm/neon-shared.decode @@ -70,6 +70,9 @@ VUSMMLA 1111 1100 1.10 .... .... 1100 .1.0 .... \ VMMLA_b16 1111 1100 0.00 .... .... 1100 .1.0 .... \ vm=%vm_dp vn=%vn_dp vd=%vd_dp +VFMA_b16 1111 110 0 0.11 .... .... 1000 . q:1 . 1 .... \ + vm=%vm_dp vn=%vn_dp vd=%vd_dp + VCMLA_scalar 1111 1110 0 . rot:2 .... .... 1000 . q:1 index:1 0 vm:4 \ vn=%vn_dp vd=%vd_dp size=1 VCMLA_scalar 1111 1110 1 . rot:2 .... .... 1000 . q:1 . 0 .... \ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index aa8d5e4b8f..322bef24cf 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -1578,6 +1578,9 @@ FMLALT_zzzw 01100100 10 1 ..... 10 0 00 1 ..... ..... @rda_rn_rm_e0 FMLSLB_zzzw 01100100 10 1 ..... 10 1 00 0 ..... ..... @rda_rn_rm_e0 FMLSLT_zzzw 01100100 10 1 ..... 10 1 00 1 ..... ..... @rda_rn_rm_e0 +BFMLALB_zzzw 01100100 11 1 ..... 10 0 00 0 ..... ..... @rda_rn_rm_e0 +BFMLALT_zzzw 01100100 11 1 ..... 10 0 00 1 ..... ..... @rda_rn_rm_e0 + ### SVE2 floating-point bfloat16 dot-product BFDOT_zzzz 01100100 01 1 ..... 10 0 00 0 ..... ..... @rda_rn_rm_e0 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 8636eac4a8..74794e3da3 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -12250,9 +12250,10 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } feature = dc_isar_feature(aa64_bf16, s); break; - case 0x1f: /* BFDOT */ + case 0x1f: switch (size) { - case 1: + case 1: /* BFDOT */ + case 3: /* BFMLAL{B,T} */ feature = dc_isar_feature(aa64_bf16, s); break; default: @@ -12346,11 +12347,15 @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) case 0xd: /* BFMMLA */ gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfmmla); return; - case 0xf: /* BFDOT */ + case 0xf: switch (size) { - case 1: + case 1: /* BFDOT */ gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfdot); break; + case 3: /* BFMLAL{B,T} */ + gen_gvec_op4_fpst(s, 1, rd, rn, rm, rd, false, is_q, + gen_helper_gvec_bfmlal); + break; default: g_assert_not_reached(); } diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index 9ade521705..3af980caba 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -8622,3 +8622,33 @@ static bool trans_BFMMLA(DisasContext *s, arg_rrrr_esz *a) } return true; } + +static bool do_BFMLAL_zzzw(DisasContext *s, arg_rrrr_esz *a, bool sel) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + if (sve_access_check(s)) { + TCGv_ptr status = fpstatus_ptr(FPST_FPCR); + unsigned vsz = vec_full_reg_size(s); + + tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + vec_full_reg_offset(s, a->ra), + status, vsz, vsz, sel, + gen_helper_gvec_bfmlal); + tcg_temp_free_ptr(status); + } + return true; +} + +static bool trans_BFMLALB_zzzw(DisasContext *s, arg_rrrr_esz *a) +{ + return do_BFMLAL_zzzw(s, a, false); +} + +static bool trans_BFMLALT_zzzw(DisasContext *s, arg_rrrr_esz *a) +{ + return do_BFMLAL_zzzw(s, a, true); +} diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index 623a0872f3..646a364c94 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -2755,3 +2755,19 @@ void HELPER(gvec_bfmmla)(void *vd, void *vn, void *vm, void *va, uint32_t desc) } clear_tail(d, opr_sz, simd_maxsz(desc)); } + +void HELPER(gvec_bfmlal)(void *vd, void *vn, void *vm, void *va, + void *stat, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + intptr_t sel = simd_data(desc); + float32 *d = vd, *a = va; + bfloat16 *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 4; ++i) { + float32 nn = n[H2(i * 2 + sel)] << 16; + float32 mm = m[H2(i * 2 + sel)] << 16; + d[H4(i)] = float32_muladd(nn, mm, a[H4(i)], 0, stat); + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index 7ce65f691f..dd710c8450 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -4126,3 +4126,12 @@ static bool trans_VMMLA_b16(DisasContext *s, arg_VMMLA_b16 *a) return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0, gen_helper_gvec_bfmmla); } + +static bool trans_VFMA_b16(DisasContext *s, arg_VFMA_b16 *a) +{ + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + return do_neon_ddda_fpst(s, 7, a->vd, a->vn, a->vm, a->q, FPST_STD, + gen_helper_gvec_bfmlal); +} From patchwork Fri Apr 16 23:59:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467441 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=nTxDb1C3; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYBX3C8Xz9sTD for ; Sat, 17 Apr 2021 10:00:52 +1000 (AEST) Received: from localhost ([::1]:56954 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYOI-0006gS-Df for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:00:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42186) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYNE-0005VN-I0 for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:44 -0400 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]:34532) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYNB-0005iY-KU for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:44 -0400 Received: by mail-pg1-x530.google.com with SMTP id z16so20219115pga.1 for ; Fri, 16 Apr 2021 16:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gt+d8J8/Jeqw9COlXsH+/eB2+q3ZGydi93ujUMo44n4=; b=nTxDb1C3/Aj0+G0+UZI87ScCA7g9YnYfKaasUIeveuH1+OkDJTFXIrr0VDpmpj879o yh+dYeaEYfoR+ZN9REIrQwmgri43EBPmGGWpJV4vPTYwleIXlfXrAuFT6sAljTCx80tz 3SzsgLHObOAq0wfgUEpO/BOIan3GWVV/XkPg6+YKmn9fyI+BExQzEsQW/zXdc8HZqq1b geJNLtXbScTGJB7X++ox6GMvPuqyiZcYiNCbgI8gmS7iuBI8Hr4KZFsShfGN1dTqSAmA MxZoaaKb3K9+OZxRDmiKBLJMq8n/9ObvaWDjRAbPzOXuh3TQ2X29TrjpFzCxhWOyTWqU qu+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gt+d8J8/Jeqw9COlXsH+/eB2+q3ZGydi93ujUMo44n4=; b=X7n8mG3Cdzqlzfln9uGAlS35SrS/kXaqjeE9GetPMxCx8CecEJ1vTRJ7vyWqFId7vj hL4ZW06UyWZ2Qc1bPIx1/692dS+7PBIwhR5n2kbU6gKHwLqY0Aldb8zoXVu7UgQIcV10 ywss1SQBBLSVYet+SzQOipxv3mbvpOteP9TvIyAzA2hcEZYcyeZPD1jGYhx3efpmwBwv kA+uJXYiJO48fqbYE7mlgOq9DRoTtrHOTrnCKvgTYP/PHD9neZVEEQHFzZ4+ws2CylKK XHdy40kVqQwX4FEqNvfXWvEjDhOeoi/yFxudJqyAo/8PtvYmT1iips7uHootXSEOpTOX BCxQ== X-Gm-Message-State: AOAM5324O30GG7Y4kbo6VIqIZ3n4L5rUDhPF+CjpoOv0BG8isJS5sUS9 8anpyd4pZMotXeVUukrVrqoF9BnGyLBRag== X-Google-Smtp-Source: ABdhPJwyObesdKPkl/njvW5J/SswArEAmN19p18ed1mUh4Z8ibqMh6rx01vP+sR59UkgSuRQR/XQJg== X-Received: by 2002:a65:560d:: with SMTP id l13mr1298302pgs.49.1618617580273; Fri, 16 Apr 2021 16:59:40 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 10/11] target/arm: Implement bfloat widening fma (indexed) Date: Fri, 16 Apr 2021 16:59:27 -0700 Message-Id: <20210416235928.1631788-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::530; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x530.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is BFMLAL{B,T} for both AArch64 AdvSIMD and SVE, and VFMA{B,T}.BF16 for AArch32 NEON. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/helper.h | 2 ++ target/arm/neon-shared.decode | 2 ++ target/arm/sve.decode | 2 ++ target/arm/translate-a64.c | 15 ++++++++++++++- target/arm/translate-sve.c | 30 ++++++++++++++++++++++++++++++ target/arm/vec_helper.c | 22 ++++++++++++++++++++++ target/arm/translate-neon.c.inc | 10 ++++++++++ 7 files changed, 82 insertions(+), 1 deletion(-) diff --git a/target/arm/helper.h b/target/arm/helper.h index 2c6f0cecfa..cbcaab2ce0 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -1013,6 +1013,8 @@ DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_6(gvec_bfmlal, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(gvec_bfmlal_idx, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) #ifdef TARGET_AARCH64 #include "helper-a64.h" diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode index b61addd98b..df80e6ebf6 100644 --- a/target/arm/neon-shared.decode +++ b/target/arm/neon-shared.decode @@ -95,3 +95,5 @@ VFML_scalar 1111 1110 0 . 0 s:1 .... .... 1000 . 0 . 1 index:1 ... \ rm=%vfml_scalar_q0_rm vn=%vn_sp vd=%vd_dp q=0 VFML_scalar 1111 1110 0 . 0 s:1 .... .... 1000 . 1 . 1 . rm:3 \ index=%vfml_scalar_q1_index vn=%vn_dp vd=%vd_dp q=1 +VFMA_b16_scal 1111 1110 0.11 .... .... 1000 . q:1 . 1 . vm:3 \ + index=%vfml_scalar_q1_index vn=%vn_dp vd=%vd_dp diff --git a/target/arm/sve.decode b/target/arm/sve.decode index 322bef24cf..69f979fb47 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -1589,6 +1589,8 @@ FMLALB_zzxw 01100100 10 1 ..... 0100.0 ..... ..... @rrxr_3a esz=2 FMLALT_zzxw 01100100 10 1 ..... 0100.1 ..... ..... @rrxr_3a esz=2 FMLSLB_zzxw 01100100 10 1 ..... 0110.0 ..... ..... @rrxr_3a esz=2 FMLSLT_zzxw 01100100 10 1 ..... 0110.1 ..... ..... @rrxr_3a esz=2 +BFMLALB_zzxw 01100100 11 1 ..... 0100.0 ..... ..... @rrxr_3a esz=2 +BFMLALT_zzxw 01100100 11 1 ..... 0100.1 ..... ..... @rrxr_3a esz=2 ### SVE2 floating-point bfloat16 dot-product (indexed) BFDOT_zzxz 01100100 01 1 ..... 010000 ..... ..... @rrxr_2 esz=2 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 74794e3da3..7842dd51be 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -13480,18 +13480,27 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) unallocated_encoding(s); return; } + size = MO_32; break; case 1: /* BFDOT */ if (is_scalar || !dc_isar_feature(aa64_bf16, s)) { unallocated_encoding(s); return; } + size = MO_32; + break; + case 3: /* BFMLAL{B,T} */ + if (is_scalar || !dc_isar_feature(aa64_bf16, s)) { + unallocated_encoding(s); + return; + } + /* can't set is_fp without other incorrect size checks */ + size = MO_16; break; default: unallocated_encoding(s); return; } - size = MO_32; break; case 0x11: /* FCMLA #0 */ case 0x13: /* FCMLA #90 */ @@ -13621,6 +13630,10 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index, gen_helper_gvec_usdot_idx_b); return; + case 3: /* BFMLAL{B,T} */ + gen_gvec_op4_fpst(s, 1, rd, rn, rm, rd, 0, (index << 1) | is_q, + gen_helper_gvec_bfmlal_idx); + return; } g_assert_not_reached(); case 0x11: /* FCMLA #0 */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index 3af980caba..7f33bc4682 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -8652,3 +8652,33 @@ static bool trans_BFMLALT_zzzw(DisasContext *s, arg_rrrr_esz *a) { return do_BFMLAL_zzzw(s, a, true); } + +static bool do_BFMLAL_zzxw(DisasContext *s, arg_rrxr_esz *a, bool sel) +{ + if (!dc_isar_feature(aa64_sve_bf16, s)) { + return false; + } + if (sve_access_check(s)) { + TCGv_ptr status = fpstatus_ptr(FPST_FPCR); + unsigned vsz = vec_full_reg_size(s); + + tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + vec_full_reg_offset(s, a->ra), + status, vsz, vsz, (a->index << 1) | sel, + gen_helper_gvec_bfmlal_idx); + tcg_temp_free_ptr(status); + } + return true; +} + +static bool trans_BFMLALB_zzxw(DisasContext *s, arg_rrxr_esz *a) +{ + return do_BFMLAL_zzxw(s, a, false); +} + +static bool trans_BFMLALT_zzxw(DisasContext *s, arg_rrxr_esz *a) +{ + return do_BFMLAL_zzxw(s, a, true); +} diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index 646a364c94..0906c5c148 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -2771,3 +2771,25 @@ void HELPER(gvec_bfmlal)(void *vd, void *vn, void *vm, void *va, } clear_tail(d, opr_sz, simd_maxsz(desc)); } + +void HELPER(gvec_bfmlal_idx)(void *vd, void *vn, void *vm, + void *va, void *stat, uint32_t desc) +{ + intptr_t i, j, opr_sz = simd_oprsz(desc); + intptr_t sel = extract32(desc, SIMD_DATA_SHIFT, 1); + intptr_t index = extract32(desc, SIMD_DATA_SHIFT + 1, 3); + intptr_t elements = opr_sz / 4; + intptr_t eltspersegment = MIN(16 / 4, elements); + float32 *d = vd, *a = va; + bfloat16 *n = vn, *m = vm; + + for (i = 0; i < elements; i += eltspersegment) { + float32 m_idx = m[H2(2 * i + index)] << 16; + + for (j = i; j < i + eltspersegment; j++) { + float32 n_j = n[H2(2 * j + sel)] << 16; + d[H4(j)] = float32_muladd(n_j, m_idx, a[H4(j)], 0, stat); + } + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} diff --git a/target/arm/translate-neon.c.inc b/target/arm/translate-neon.c.inc index dd710c8450..bd1068e4d4 100644 --- a/target/arm/translate-neon.c.inc +++ b/target/arm/translate-neon.c.inc @@ -4135,3 +4135,13 @@ static bool trans_VFMA_b16(DisasContext *s, arg_VFMA_b16 *a) return do_neon_ddda_fpst(s, 7, a->vd, a->vn, a->vm, a->q, FPST_STD, gen_helper_gvec_bfmlal); } + +static bool trans_VFMA_b16_scal(DisasContext *s, arg_VFMA_b16_scal *a) +{ + if (!dc_isar_feature(aa32_bf16, s)) { + return false; + } + return do_neon_ddda_fpst(s, 6, a->vd, a->vn, a->vm, + (a->index << 1) | a->q, FPST_STD, + gen_helper_gvec_bfmlal_idx); +} From patchwork Fri Apr 16 23:59:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1467446 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=rGBumnDm; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FMYGR1pymz9sTD for ; Sat, 17 Apr 2021 10:04:15 +1000 (AEST) Received: from localhost ([::1]:42524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXYRZ-0003rN-7h for incoming@patchwork.ozlabs.org; Fri, 16 Apr 2021 20:04:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42192) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXYNF-0005Wy-3Q for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:45 -0400 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]:36736) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXYNC-0005j4-H5 for qemu-devel@nongnu.org; Fri, 16 Apr 2021 19:59:44 -0400 Received: by mail-pg1-x531.google.com with SMTP id j7so10810932pgi.3 for ; Fri, 16 Apr 2021 16:59:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i8izMTvg3e0bJXY5dkxCzGZga98HxaSuFHI21G+m4Dk=; b=rGBumnDmD5WfVGnMfu2kMXWc5jGetCBkH3qLmGghWP/HfDkGqGFhBn9y6ys7NPBix4 2e5s+Kveo9UrJdvhDbyEIrojtMrr+u5ZYmXy/1dSP9+kM1j5mMTAwFPW4uCMD9jChi1K wkwmeaP7x3Nyrg0qGb1kW79wq35A8I3Y2BNa63E6JFbpBP1Fbi0iaQDF0oCLMoD/bWa2 ykzz/klGxUJY3IX+EWezRT5RVAoTVMqTSh/aXWZO8nhbAZo7C5PgCCxsrWhjj3fOE2cb l+GtrzU7uwhtMxkbTHEo681yos7QaW0iAMO1yOF/MaWOr4lVeVIfMidxvBvMbnVBvHm5 aopQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i8izMTvg3e0bJXY5dkxCzGZga98HxaSuFHI21G+m4Dk=; b=aW5gbt24jN0g13GjTIytUTKyWq6GpBsfekudiqghxYGmnJ9dTrc3Dy44RioROVfX/L N1C/xiMDChB6mlKzyVXqmfMKNx/55WqPxHHGImrNxjqMDcxcsFjS20dMc82mAoFLwRNo SQSxzp513pIcQ9cp61iMmg8wEiYIcqM8g+93jYwXRrOwaXBSY9NngziSxiMmsOqiwfS/ wF+y6jr5YneGMHUU50kCivbaFSuw7YrEUa494IOd5eZEatwRQbOrCdae+NpeDLzmW7eh Ah973+OO9Ccx7vV9tPF5hPm986gQQnm+GzEq472691ep/i/FEwNvFLdJWZ4Xq6UusjKf DOcQ== X-Gm-Message-State: AOAM530QvJ9vZXVKY/i4go9VBZdwrPwnm0uKlip7ciI6miIBpp44rGR+ Q2pUna9RUEsoxwuQHPuBdtn2uLIHTJGIZg== X-Google-Smtp-Source: ABdhPJxdbHGh7B5kOjFlN+91F/pNjycaiGKiCeU6sFa9Pw+CcrDiTwjRBbD46/3JSEeAH/zzDORf0w== X-Received: by 2002:a62:2aca:0:b029:23e:1c40:4d1a with SMTP id q193-20020a622aca0000b029023e1c404d1amr9970921pfq.60.1618617581320; Fri, 16 Apr 2021 16:59:41 -0700 (PDT) Received: from localhost.localdomain ([71.212.131.83]) by smtp.gmail.com with ESMTPSA id d71sm560853pfd.83.2021.04.16.16.59.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 16:59:40 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v1 11/11] target/arm: Enable BFloat16 extensions Date: Fri, 16 Apr 2021 16:59:28 -0700 Message-Id: <20210416235928.1631788-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416235928.1631788-1-richard.henderson@linaro.org> References: <20210416235928.1631788-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::531; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x531.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/cpu64.c | 3 +++ target/arm/cpu_tcg.c | 1 + 2 files changed, 4 insertions(+) diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 379f90fab8..db4f48edcf 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -660,6 +660,7 @@ static void aarch64_max_initfn(Object *obj) t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1); t = FIELD_DP64(t, ID_AA64ISAR1, SB, 1); t = FIELD_DP64(t, ID_AA64ISAR1, SPECRES, 1); + t = FIELD_DP64(t, ID_AA64ISAR1, BF16, 1); t = FIELD_DP64(t, ID_AA64ISAR1, FRINTTS, 1); t = FIELD_DP64(t, ID_AA64ISAR1, LRCPC, 2); /* ARMv8.4-RCPC */ t = FIELD_DP64(t, ID_AA64ISAR1, I8MM, 1); @@ -707,6 +708,7 @@ static void aarch64_max_initfn(Object *obj) t = FIELD_DP64(t, ID_AA64ZFR0, SVEVER, 1); t = FIELD_DP64(t, ID_AA64ZFR0, AES, 2); /* PMULL */ t = FIELD_DP64(t, ID_AA64ZFR0, BITPERM, 1); + t = FIELD_DP64(t, ID_AA64ZFR0, BFLOAT16, 1); t = FIELD_DP64(t, ID_AA64ZFR0, SHA3, 1); t = FIELD_DP64(t, ID_AA64ZFR0, SM4, 1); t = FIELD_DP64(t, ID_AA64ZFR0, I8MM, 1); @@ -730,6 +732,7 @@ static void aarch64_max_initfn(Object *obj) u = FIELD_DP32(u, ID_ISAR6, FHM, 1); u = FIELD_DP32(u, ID_ISAR6, SB, 1); u = FIELD_DP32(u, ID_ISAR6, SPECRES, 1); + u = FIELD_DP32(u, ID_ISAR6, BF16, 1); u = FIELD_DP32(u, ID_ISAR6, I8MM, 1); cpu->isar.id_isar6 = u; diff --git a/target/arm/cpu_tcg.c b/target/arm/cpu_tcg.c index 046e476f65..b2463cf109 100644 --- a/target/arm/cpu_tcg.c +++ b/target/arm/cpu_tcg.c @@ -968,6 +968,7 @@ static void arm_max_initfn(Object *obj) t = FIELD_DP32(t, ID_ISAR6, FHM, 1); t = FIELD_DP32(t, ID_ISAR6, SB, 1); t = FIELD_DP32(t, ID_ISAR6, SPECRES, 1); + t = FIELD_DP32(t, ID_ISAR6, BF16, 1); cpu->isar.id_isar6 = t; t = cpu->isar.mvfr1;