{"id":2183189,"url":"http://patchwork.ozlabs.org/api/patches/2183189/?format=json","web_url":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20260112192035.10427-16-ebiggers@kernel.org/","project":{"id":2,"url":"http://patchwork.ozlabs.org/api/projects/2/?format=json","name":"Linux PPC development","link_name":"linuxppc-dev","list_id":"linuxppc-dev.lists.ozlabs.org","list_email":"linuxppc-dev@lists.ozlabs.org","web_url":"https://github.com/linuxppc/wiki/wiki","scm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git","webscm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/","list_archive_url":"https://lore.kernel.org/linuxppc-dev/","list_archive_url_format":"https://lore.kernel.org/linuxppc-dev/{}/","commit_url_format":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id={}"},"msgid":"<20260112192035.10427-16-ebiggers@kernel.org>","list_archive_url":"https://lore.kernel.org/linuxppc-dev/20260112192035.10427-16-ebiggers@kernel.org/","date":"2026-01-12T19:20:13","name":"[v2,15/35] lib/crypto: s390/aes: Migrate optimized code into library","commit_ref":null,"pull_url":null,"state":"handled-elsewhere","archived":false,"hash":"be7f35856860696e21e470f0e0e44d5650a84470","submitter":{"id":74690,"url":"http://patchwork.ozlabs.org/api/people/74690/?format=json","name":"Eric Biggers","email":"ebiggers@kernel.org"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20260112192035.10427-16-ebiggers@kernel.org/mbox/","series":[{"id":488089,"url":"http://patchwork.ozlabs.org/api/series/488089/?format=json","web_url":"http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=488089","date":"2026-01-12T19:19:58","name":"AES library improvements","version":2,"mbox":"http://patchwork.ozlabs.org/series/488089/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/2183189/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/2183189/checks/","tags":{},"related":[],"headers":{"Return-Path":"\n ","X-Original-To":["incoming@patchwork.ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256\n header.s=k20201202 header.b=tRrT//dJ;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org\n (client-ip=112.213.38.117; helo=lists.ozlabs.org;\n envelope-from=linuxppc-dev+bounces-15559-incoming=patchwork.ozlabs.org@lists.ozlabs.org;\n receiver=patchwork.ozlabs.org)","lists.ozlabs.org;\n arc=none smtp.remote-ip=172.234.252.31","lists.ozlabs.org;\n dmarc=pass (p=quarantine dis=none) header.from=kernel.org","lists.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256\n header.s=k20201202 header.b=tRrT//dJ;\n\tdkim-atps=neutral","lists.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org\n (client-ip=172.234.252.31; helo=sea.source.kernel.org;\n envelope-from=ebiggers@kernel.org; receiver=lists.ozlabs.org)"],"Received":["from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4dqj3q4NJRz1xpk\n\tfor ; Tue, 13 Jan 2026 06:24:07 +1100 (AEDT)","from boromir.ozlabs.org (localhost [127.0.0.1])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 4dqj345yxgz2ynC;\n\tTue, 13 Jan 2026 06:23:28 +1100 (AEDT)","from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519)\n\t(No client certificate requested)\n\tby lists.ozlabs.org (Postfix) with ESMTPS id 4dqj332Bg8z2yYs\n\tfor ; Tue, 13 Jan 2026 06:23:27 +1100 (AEDT)","from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])\n\tby sea.source.kernel.org (Postfix) with ESMTP id 5B77344382;\n\tMon, 12 Jan 2026 19:23:26 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id CC557C2BC9E;\n\tMon, 12 Jan 2026 19:23:25 +0000 (UTC)"],"ARC-Seal":"i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1768245808;\n\tcv=none;\n b=QWlHqZCyi2QQPj3MkV4lY6fUJUr+xxD2cvkqdaPeHHZGYupwvFZcTbSBk8YQ3oAtvualfwWPGIXuiWNsiSwXRbtJqnTrYV31d/wYPVin20FUwkD+qji1envvi+J+F2WZtK6yyJPDNFTSycvnZH+UYFYersZhe/TFPiDT/pbMpCp2ldBf8p9L2a8fqWLc1zabpfU7EQtxzovYtMLgiUalEaDaA9F66bRFjplD58mCoEZoueOCelbgorB51qCDwH2/KjoruiO06N2OhPtz3AvgY/B4h92PpBWxOAqxVJmY1t02D9hJyusMOCe3XCJU3+ArBbUemXX5REW3kP7gOEKRhg==","ARC-Message-Signature":"i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707;\n\tt=1768245808; c=relaxed/relaxed;\n\tbh=m7vkOKnDugdkCKq4wtl6aVNFQgURrQRIOwd5WtyOcis=;\n\th=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:\n\t MIME-Version;\n b=OB9uQB+UHIjZUuxmgiw78QSYBStN6eoQj5EGmZj3sV5kWCal2ZqEOU/XzBeEiskvbBBcpR+vrOlV2ZCTvspr9swpqf+bYPnP4zGEk0JXCUZ6NZlf21XGmY5TtNqfWABBKCfMYZNOWPaeahw6bBgRmrIS/PkkXeowrj32c9Tapr7/fmJHeIxfacf3EnDchaUT5IZOUL3fChLje/dgJ/wloekg1ItwXSfElaiArNoRp1hqJ9/E4Pftz3x1ut1amC+SBg1Xso6fnDdBl4vyXPpST5pThbL+ccxZEvhjV/uMQIr81epnt0ENU7zMIemnT/kcM8vDALnE187VuDzAtpzwxg==","ARC-Authentication-Results":"i=1; lists.ozlabs.org;\n dmarc=pass (p=quarantine dis=none) header.from=kernel.org;\n dkim=pass (2048-bit key;\n unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256\n header.s=k20201202 header.b=tRrT//dJ; dkim-atps=neutral;\n spf=pass (client-ip=172.234.252.31; helo=sea.source.kernel.org;\n envelope-from=ebiggers@kernel.org;\n receiver=lists.ozlabs.org) smtp.mailfrom=kernel.org","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1768245806;\n\tbh=KZ6YNHHvsdIxal21I4zQ8OcS7arQu3vmqdqR8uLdXXY=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=tRrT//dJTBOCkmjfH56jwPUVRsaIHuRNm2ahbv5PVt9YrWAROEBi8+/6zhymwFE+Z\n\t J4UgGBMK8cMfSYPN8krzg+rCencfxT8BM/QW2slVnkR5p428tkRskFhl5yoTsHJ4s6\n\t sgmxjp7wFQoOZLmnecSjDJzZ2V9CQpkwYTagF/i2Odd+sYrYyGD4d4ofEi0YqhosbQ\n\t Y1D5Nzrnm993ukQA+aF52HYb9vQxcxt3fWSpD93G+S5yc+B9pWeKCNO4Sw/AdVm92c\n\t 380dtcKmprMIpFdd25S5qpuWgZjN/XDVb9OjUQ8bZIshn/q/QhUX4VANuNidkngMhC\n\t Jknd0YEZiuA2g==","From":"Eric Biggers ","To":"linux-crypto@vger.kernel.org","Cc":"linux-kernel@vger.kernel.org,\n\tArd Biesheuvel ,\n\t\"Jason A . Donenfeld\" ,\n\tHerbert Xu ,\n\tlinux-arm-kernel@lists.infradead.org,\n\tlinuxppc-dev@lists.ozlabs.org,\n\tlinux-riscv@lists.infradead.org,\n\tlinux-s390@vger.kernel.org,\n\tsparclinux@vger.kernel.org,\n\tx86@kernel.org,\n\tHolger Dengler ,\n\tHarald Freudenberger ,\n\tEric Biggers ","Subject":"[PATCH v2 15/35] lib/crypto: s390/aes: Migrate optimized code into\n library","Date":"Mon, 12 Jan 2026 11:20:13 -0800","Message-ID":"<20260112192035.10427-16-ebiggers@kernel.org>","X-Mailer":"git-send-email 2.52.0","In-Reply-To":"<20260112192035.10427-1-ebiggers@kernel.org>","References":"<20260112192035.10427-1-ebiggers@kernel.org>","X-Mailing-List":"linuxppc-dev@lists.ozlabs.org","List-Id":"","List-Help":"","List-Owner":"","List-Post":"","List-Archive":",\n ","List-Subscribe":",\n ,\n ","List-Unsubscribe":"","Precedence":"list","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-Spam-Status":"No, score=-0.2 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,\n\tDKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS\n\tautolearn=disabled version=4.0.1 OzLabs 8","X-Spam-Checker-Version":"SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org"},"content":"Implement aes_preparekey_arch(), aes_encrypt_arch(), and\naes_decrypt_arch() using the CPACF AES instructions.\n\nThen, remove the superseded \"aes-s390\" crypto_cipher.\n\nThe result is that both the AES library and crypto_cipher APIs use the\nCPACF AES instructions, whereas previously only crypto_cipher did (and\nit wasn't enabled by default, which this commit fixes as well).\n\nNote that this preserves the optimization where the AES key is stored in\nraw form rather than expanded form. CPACF just takes the raw key.\n\nAcked-by: Ard Biesheuvel \nSigned-off-by: Eric Biggers \n---\n arch/s390/crypto/Kconfig | 2 -\n arch/s390/crypto/aes_s390.c | 113 ------------------------------------\n include/crypto/aes.h | 3 +\n lib/crypto/Kconfig | 1 +\n lib/crypto/s390/aes.h | 106 +++++++++++++++++++++++++++++++++\n 5 files changed, 110 insertions(+), 115 deletions(-)\n create mode 100644 lib/crypto/s390/aes.h","diff":"diff --git a/arch/s390/crypto/Kconfig b/arch/s390/crypto/Kconfig\nindex f838ca055f6d..79a2d0034258 100644\n--- a/arch/s390/crypto/Kconfig\n+++ b/arch/s390/crypto/Kconfig\n@@ -12,14 +12,12 @@ config CRYPTO_GHASH_S390\n \n \t It is available as of z196.\n \n config CRYPTO_AES_S390\n \ttristate \"Ciphers: AES, modes: ECB, CBC, CTR, XTS, GCM\"\n-\tselect CRYPTO_ALGAPI\n \tselect CRYPTO_SKCIPHER\n \thelp\n-\t Block cipher: AES cipher algorithms (FIPS 197)\n \t AEAD cipher: AES with GCM\n \t Length-preserving ciphers: AES with ECB, CBC, XTS, and CTR modes\n \n \t Architecture: s390\n \ndiff --git a/arch/s390/crypto/aes_s390.c b/arch/s390/crypto/aes_s390.c\nindex d0a295435680..62edc66d5478 100644\n--- a/arch/s390/crypto/aes_s390.c\n+++ b/arch/s390/crypto/aes_s390.c\n@@ -18,11 +18,10 @@\n \n #include \n #include \n #include \n #include \n-#include \n #include \n #include \n #include \n #include \n #include \n@@ -43,11 +42,10 @@ struct s390_aes_ctx {\n \tu8 key[AES_MAX_KEY_SIZE];\n \tint key_len;\n \tunsigned long fc;\n \tunion {\n \t\tstruct crypto_skcipher *skcipher;\n-\t\tstruct crypto_cipher *cip;\n \t} fallback;\n };\n \n struct s390_xts_ctx {\n \tunion {\n@@ -70,113 +68,10 @@ struct gcm_sg_walk {\n \tunsigned int buf_bytes;\n \tu8 *ptr;\n \tunsigned int nbytes;\n };\n \n-static int setkey_fallback_cip(struct crypto_tfm *tfm, const u8 *in_key,\n-\t\tunsigned int key_len)\n-{\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\n-\tsctx->fallback.cip->base.crt_flags &= ~CRYPTO_TFM_REQ_MASK;\n-\tsctx->fallback.cip->base.crt_flags |= (tfm->crt_flags &\n-\t\t\tCRYPTO_TFM_REQ_MASK);\n-\n-\treturn crypto_cipher_setkey(sctx->fallback.cip, in_key, key_len);\n-}\n-\n-static int aes_set_key(struct crypto_tfm *tfm, const u8 *in_key,\n-\t\t unsigned int key_len)\n-{\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\tunsigned long fc;\n-\n-\t/* Pick the correct function code based on the key length */\n-\tfc = (key_len == 16) ? CPACF_KM_AES_128 :\n-\t (key_len == 24) ? CPACF_KM_AES_192 :\n-\t (key_len == 32) ? CPACF_KM_AES_256 : 0;\n-\n-\t/* Check if the function code is available */\n-\tsctx->fc = (fc && cpacf_test_func(&km_functions, fc)) ? fc : 0;\n-\tif (!sctx->fc)\n-\t\treturn setkey_fallback_cip(tfm, in_key, key_len);\n-\n-\tsctx->key_len = key_len;\n-\tmemcpy(sctx->key, in_key, key_len);\n-\treturn 0;\n-}\n-\n-static void crypto_aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)\n-{\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\n-\tif (unlikely(!sctx->fc)) {\n-\t\tcrypto_cipher_encrypt_one(sctx->fallback.cip, out, in);\n-\t\treturn;\n-\t}\n-\tcpacf_km(sctx->fc, &sctx->key, out, in, AES_BLOCK_SIZE);\n-}\n-\n-static void crypto_aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)\n-{\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\n-\tif (unlikely(!sctx->fc)) {\n-\t\tcrypto_cipher_decrypt_one(sctx->fallback.cip, out, in);\n-\t\treturn;\n-\t}\n-\tcpacf_km(sctx->fc | CPACF_DECRYPT,\n-\t\t &sctx->key, out, in, AES_BLOCK_SIZE);\n-}\n-\n-static int fallback_init_cip(struct crypto_tfm *tfm)\n-{\n-\tconst char *name = tfm->__crt_alg->cra_name;\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\n-\tsctx->fallback.cip = crypto_alloc_cipher(name, 0,\n-\t\t\t\t\t\t CRYPTO_ALG_NEED_FALLBACK);\n-\n-\tif (IS_ERR(sctx->fallback.cip)) {\n-\t\tpr_err(\"Allocating AES fallback algorithm %s failed\\n\",\n-\t\t name);\n-\t\treturn PTR_ERR(sctx->fallback.cip);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static void fallback_exit_cip(struct crypto_tfm *tfm)\n-{\n-\tstruct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);\n-\n-\tcrypto_free_cipher(sctx->fallback.cip);\n-\tsctx->fallback.cip = NULL;\n-}\n-\n-static struct crypto_alg aes_alg = {\n-\t.cra_name\t\t=\t\"aes\",\n-\t.cra_driver_name\t=\t\"aes-s390\",\n-\t.cra_priority\t\t=\t300,\n-\t.cra_flags\t\t=\tCRYPTO_ALG_TYPE_CIPHER |\n-\t\t\t\t\tCRYPTO_ALG_NEED_FALLBACK,\n-\t.cra_blocksize\t\t=\tAES_BLOCK_SIZE,\n-\t.cra_ctxsize\t\t=\tsizeof(struct s390_aes_ctx),\n-\t.cra_module\t\t=\tTHIS_MODULE,\n-\t.cra_init = fallback_init_cip,\n-\t.cra_exit = fallback_exit_cip,\n-\t.cra_u\t\t\t=\t{\n-\t\t.cipher = {\n-\t\t\t.cia_min_keysize\t=\tAES_MIN_KEY_SIZE,\n-\t\t\t.cia_max_keysize\t=\tAES_MAX_KEY_SIZE,\n-\t\t\t.cia_setkey\t\t=\taes_set_key,\n-\t\t\t.cia_encrypt\t\t=\tcrypto_aes_encrypt,\n-\t\t\t.cia_decrypt\t\t=\tcrypto_aes_decrypt,\n-\t\t}\n-\t}\n-};\n-\n static int setkey_fallback_skcipher(struct crypto_skcipher *tfm, const u8 *key,\n \t\t\t\t unsigned int len)\n {\n \tstruct s390_aes_ctx *sctx = crypto_skcipher_ctx(tfm);\n \n@@ -1047,11 +942,10 @@ static struct aead_alg gcm_aes_aead = {\n \t\t.cra_driver_name\t= \"gcm-aes-s390\",\n \t\t.cra_module\t\t= THIS_MODULE,\n \t},\n };\n \n-static struct crypto_alg *aes_s390_alg;\n static struct skcipher_alg *aes_s390_skcipher_algs[5];\n static int aes_s390_skciphers_num;\n static struct aead_alg *aes_s390_aead_alg;\n \n static int aes_s390_register_skcipher(struct skcipher_alg *alg)\n@@ -1064,12 +958,10 @@ static int aes_s390_register_skcipher(struct skcipher_alg *alg)\n \treturn ret;\n }\n \n static void aes_s390_fini(void)\n {\n-\tif (aes_s390_alg)\n-\t\tcrypto_unregister_alg(aes_s390_alg);\n \twhile (aes_s390_skciphers_num--)\n \t\tcrypto_unregister_skcipher(aes_s390_skcipher_algs[aes_s390_skciphers_num]);\n \tif (ctrblk)\n \t\tfree_page((unsigned long) ctrblk);\n \n@@ -1088,14 +980,10 @@ static int __init aes_s390_init(void)\n \tcpacf_query(CPACF_KMA, &kma_functions);\n \n \tif (cpacf_test_func(&km_functions, CPACF_KM_AES_128) ||\n \t cpacf_test_func(&km_functions, CPACF_KM_AES_192) ||\n \t cpacf_test_func(&km_functions, CPACF_KM_AES_256)) {\n-\t\tret = crypto_register_alg(&aes_alg);\n-\t\tif (ret)\n-\t\t\tgoto out_err;\n-\t\taes_s390_alg = &aes_alg;\n \t\tret = aes_s390_register_skcipher(&ecb_aes_alg);\n \t\tif (ret)\n \t\t\tgoto out_err;\n \t}\n \n@@ -1154,6 +1042,5 @@ module_exit(aes_s390_fini);\n \n MODULE_ALIAS_CRYPTO(\"aes-all\");\n \n MODULE_DESCRIPTION(\"Rijndael (AES) Cipher Algorithm\");\n MODULE_LICENSE(\"GPL\");\n-MODULE_IMPORT_NS(\"CRYPTO_INTERNAL\");\ndiff --git a/include/crypto/aes.h b/include/crypto/aes.h\nindex bff71cfaedeb..19fd99f383fb 100644\n--- a/include/crypto/aes.h\n+++ b/include/crypto/aes.h\n@@ -44,10 +44,13 @@ union aes_enckey_arch {\n \t * when that code is usable at key preparation time. Otherwise they\n \t * fall back to rndkeys. In the latter case, p8.nrounds (which doesn't\n \t * overlap rndkeys) is set to 0 to differentiate the two formats.\n \t */\n \tstruct p8_aes_key p8;\n+#elif defined(CONFIG_S390)\n+\t/* Used when the CPU supports CPACF AES for this key's length */\n+\tu8 raw_key[AES_MAX_KEY_SIZE];\n #endif\n #endif /* CONFIG_CRYPTO_LIB_AES_ARCH */\n };\n \n union aes_invkey_arch {\ndiff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig\nindex 2690b5ffc5ca..56a9b4f53b0e 100644\n--- a/lib/crypto/Kconfig\n+++ b/lib/crypto/Kconfig\n@@ -17,10 +17,11 @@ config CRYPTO_LIB_AES_ARCH\n \tdefault y if ARM\n \tdefault y if ARM64\n \tdefault y if PPC && (SPE || (PPC64 && VSX))\n \tdefault y if RISCV && 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \\\n \t\t RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS\n+\tdefault y if S390\n \n config CRYPTO_LIB_AESCFB\n \ttristate\n \tselect CRYPTO_LIB_AES\n \tselect CRYPTO_LIB_UTILS\ndiff --git a/lib/crypto/s390/aes.h b/lib/crypto/s390/aes.h\nnew file mode 100644\nindex 000000000000..5466f6ecbce7\n--- /dev/null\n+++ b/lib/crypto/s390/aes.h\n@@ -0,0 +1,106 @@\n+/* SPDX-License-Identifier: GPL-2.0-or-later */\n+/*\n+ * AES optimized using the CP Assist for Cryptographic Functions (CPACF)\n+ *\n+ * Copyright 2026 Google LLC\n+ */\n+#include \n+#include \n+\n+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_cpacf_aes128);\n+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_cpacf_aes192);\n+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_cpacf_aes256);\n+\n+/*\n+ * When the CPU supports CPACF AES for the requested key length, we need only\n+ * save a copy of the raw AES key, as that's what the CPACF instructions need.\n+ *\n+ * When unsupported, fall back to the generic key expansion and en/decryption.\n+ */\n+static void aes_preparekey_arch(union aes_enckey_arch *k,\n+\t\t\t\tunion aes_invkey_arch *inv_k,\n+\t\t\t\tconst u8 *in_key, int key_len, int nrounds)\n+{\n+\tif (key_len == AES_KEYSIZE_128) {\n+\t\tif (static_branch_likely(&have_cpacf_aes128)) {\n+\t\t\tmemcpy(k->raw_key, in_key, AES_KEYSIZE_128);\n+\t\t\treturn;\n+\t\t}\n+\t} else if (key_len == AES_KEYSIZE_192) {\n+\t\tif (static_branch_likely(&have_cpacf_aes192)) {\n+\t\t\tmemcpy(k->raw_key, in_key, AES_KEYSIZE_192);\n+\t\t\treturn;\n+\t\t}\n+\t} else {\n+\t\tif (static_branch_likely(&have_cpacf_aes256)) {\n+\t\t\tmemcpy(k->raw_key, in_key, AES_KEYSIZE_256);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\taes_expandkey_generic(k->rndkeys, inv_k ? inv_k->inv_rndkeys : NULL,\n+\t\t\t in_key, key_len);\n+}\n+\n+static inline bool aes_crypt_s390(const struct aes_enckey *key,\n+\t\t\t\t u8 out[AES_BLOCK_SIZE],\n+\t\t\t\t const u8 in[AES_BLOCK_SIZE], int decrypt)\n+{\n+\tif (key->len == AES_KEYSIZE_128) {\n+\t\tif (static_branch_likely(&have_cpacf_aes128)) {\n+\t\t\tcpacf_km(CPACF_KM_AES_128 | decrypt,\n+\t\t\t\t (void *)key->k.raw_key, out, in,\n+\t\t\t\t AES_BLOCK_SIZE);\n+\t\t\treturn true;\n+\t\t}\n+\t} else if (key->len == AES_KEYSIZE_192) {\n+\t\tif (static_branch_likely(&have_cpacf_aes192)) {\n+\t\t\tcpacf_km(CPACF_KM_AES_192 | decrypt,\n+\t\t\t\t (void *)key->k.raw_key, out, in,\n+\t\t\t\t AES_BLOCK_SIZE);\n+\t\t\treturn true;\n+\t\t}\n+\t} else {\n+\t\tif (static_branch_likely(&have_cpacf_aes256)) {\n+\t\t\tcpacf_km(CPACF_KM_AES_256 | decrypt,\n+\t\t\t\t (void *)key->k.raw_key, out, in,\n+\t\t\t\t AES_BLOCK_SIZE);\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\treturn false;\n+}\n+\n+static void aes_encrypt_arch(const struct aes_enckey *key,\n+\t\t\t u8 out[AES_BLOCK_SIZE],\n+\t\t\t const u8 in[AES_BLOCK_SIZE])\n+{\n+\tif (likely(aes_crypt_s390(key, out, in, 0)))\n+\t\treturn;\n+\taes_encrypt_generic(key->k.rndkeys, key->nrounds, out, in);\n+}\n+\n+static void aes_decrypt_arch(const struct aes_key *key,\n+\t\t\t u8 out[AES_BLOCK_SIZE],\n+\t\t\t const u8 in[AES_BLOCK_SIZE])\n+{\n+\tif (likely(aes_crypt_s390((const struct aes_enckey *)key, out, in,\n+\t\t\t\t CPACF_DECRYPT)))\n+\t\treturn;\n+\taes_decrypt_generic(key->inv_k.inv_rndkeys, key->nrounds, out, in);\n+}\n+\n+#define aes_mod_init_arch aes_mod_init_arch\n+static void aes_mod_init_arch(void)\n+{\n+\tif (cpu_have_feature(S390_CPU_FEATURE_MSA)) {\n+\t\tcpacf_mask_t km_functions;\n+\n+\t\tcpacf_query(CPACF_KM, &km_functions);\n+\t\tif (cpacf_test_func(&km_functions, CPACF_KM_AES_128))\n+\t\t\tstatic_branch_enable(&have_cpacf_aes128);\n+\t\tif (cpacf_test_func(&km_functions, CPACF_KM_AES_192))\n+\t\t\tstatic_branch_enable(&have_cpacf_aes192);\n+\t\tif (cpacf_test_func(&km_functions, CPACF_KM_AES_256))\n+\t\t\tstatic_branch_enable(&have_cpacf_aes256);\n+\t}\n+}\n","prefixes":["v2","15/35"]}