From patchwork Tue Jan 30 02:35:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuxuan Luo X-Patchwork-Id: 1892625 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TP8Rb21rcz1yQ0 for ; Tue, 30 Jan 2024 13:35:54 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rUdyT-0004wk-7A; Tue, 30 Jan 2024 02:35:45 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rUdyA-0004vK-Ln for kernel-team@lists.ubuntu.com; Tue, 30 Jan 2024 02:35:26 +0000 Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 05D783F184 for ; Tue, 30 Jan 2024 02:35:26 +0000 (UTC) Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-a357c92f241so103486966b.0 for ; Mon, 29 Jan 2024 18:35:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706582123; x=1707186923; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zzORRiNejAyvkS/pseKJaeI+zryy40j7Uytz/tlgY4U=; b=qHISYSQWCV9ZlYyUzpwg2KYm2vzPaW0F90L+SFTbprkIjn87uPuBr99w9pAQktk7Oz hz5oD2c0zcSuOfnNu08Fnq7kgO4jvZ9sPgiBbZ4z0JyBMwLxHsYd6r7OgqIVcgEpYMn1 6gljK8AetlvaqKhpQnXnU1vOpZVlh7ulZve7r77RZTn0lv2/vjCMmleO6kODSjW/QK8c Oij+0MP7d1Sb0HDklQwnMq5fRPyIZx29T7myqe8UElPrVQBwwTbUNdofcxcvfJ5VbOOs RgM3y7XyeRhU6C6FLCl+WhW/+VsjPXwKBkprAK8wzncXcfSA74NtCy5wHwj64U4mU9U1 w8Ug== X-Gm-Message-State: AOJu0YxYbsD7rDoPIiaqohhSLzyGqAy0Vz5h/LGAx8easEEs8ehDLUy5 /D4S7iec8JPSjqo8x2TNQOXup4NGjh1yM+nh1H33K/5jXvf7DvrTKtpIf4xDVbYFLuScWmnMmZ/ 3K7JRSbF1fXk9kpaRVoXlcwbqL2ZCZnZEJG59FU2Aq4aqQhYf4NOnGpE4/m2CorQPskp7TO//RC V60Vc1C1AhHq8B X-Received: by 2002:a17:906:e53:b0:a31:1b72:9efd with SMTP id q19-20020a1709060e5300b00a311b729efdmr5491016eji.66.1706582123789; Mon, 29 Jan 2024 18:35:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IFShVC2Vf8qsby1rukTw6sSaAF4a1ztYPvZ1qtV/cMK+Yus9qhx4o0AojByOVDQ/SoqOI9iQg== X-Received: by 2002:a17:906:e53:b0:a31:1b72:9efd with SMTP id q19-20020a1709060e5300b00a311b729efdmr5491009eji.66.1706582123539; Mon, 29 Jan 2024 18:35:23 -0800 (PST) Received: from cache-ubuntu.hsd1.nj.comcast.net ([2001:67c:1562:8007::aac:4795]) by smtp.gmail.com with ESMTPSA id vu2-20020a170907a64200b00a35a9745910sm2017165ejc.137.2024.01.29.18.35.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 18:35:22 -0800 (PST) From: Yuxuan Luo To: kernel-team@lists.ubuntu.com Subject: [SRU][Mantic][PATCH 1/2] io_uring: enable io_mem_alloc/free to be used in other parts Date: Mon, 29 Jan 2024 21:35:18 -0500 Message-Id: <20240130023519.94667-2-yuxuan.luo@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240130023519.94667-1-yuxuan.luo@canonical.com> References: <20240130023519.94667-1-yuxuan.luo@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Jens Axboe In preparation for using these helpers, make them non-static and add them to our internal header. Signed-off-by: Jens Axboe (cherry picked from commit edecf1689768452ba1a64b7aaf3a47a817da651a) CVE-2024-0582 Signed-off-by: Yuxuan Luo --- io_uring/io_uring.c | 4 ++-- io_uring/io_uring.h | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 43d192dcc934d..1b0a27fe41eef 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2659,7 +2659,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, return READ_ONCE(rings->cq.head) == READ_ONCE(rings->cq.tail) ? ret : 0; } -static void io_mem_free(void *ptr) +void io_mem_free(void *ptr) { if (!ptr) return; @@ -2771,7 +2771,7 @@ static void io_rings_free(struct io_ring_ctx *ctx) } } -static void *io_mem_alloc(size_t size) +void *io_mem_alloc(size_t size) { gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN | __GFP_COMP; void *ret; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 0bc145614a6e6..d2bad1df347da 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -86,6 +86,9 @@ bool __io_alloc_req_refill(struct io_ring_ctx *ctx); bool io_match_task_safe(struct io_kiocb *head, struct task_struct *task, bool cancel_all); +void *io_mem_alloc(size_t size); +void io_mem_free(void *ptr); + #if defined(CONFIG_PROVE_LOCKING) static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) { From patchwork Tue Jan 30 02:35:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuxuan Luo X-Patchwork-Id: 1892627 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TP8Rf2Y3Dz1yQ0 for ; Tue, 30 Jan 2024 13:35:58 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rUdyW-0004xi-Un; Tue, 30 Jan 2024 02:35:49 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rUdyB-0004vR-5O for kernel-team@lists.ubuntu.com; Tue, 30 Jan 2024 02:35:27 +0000 Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 5A99A3F336 for ; Tue, 30 Jan 2024 02:35:26 +0000 (UTC) Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-a35247a5d8cso217226166b.2 for ; Mon, 29 Jan 2024 18:35:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706582125; x=1707186925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LL1XPVZ/J7N17CSpZpsMtOesF2BRBcyKpWKpkqPbuOc=; b=llwKnR++T8fWvzv1si5Y1DdMP3aiIKT9fdRurVpv1sGhrxAYs7Lrt16hfSl0snCRUm 00Tjf/WNTZXw1SdTxiq3L2wfWX89VTTPQ6NA6vyMZFJzWdX9Xym8pxgc6NBsTq9IwMLL o/ygIHdsshHLdEnY2piEtqXRdKQkoz1ywqB9abaQbASQeMo/l3jxrHhjKuqBZvyWbDZJ MkuT//maU0iWkOmdrAQ0hHRQU2BnlOz345mAfGAGe7iyqi//aTNLKDBomwxBFSdhx12X 2nxStR4tHRSI6BwuHoz8XOseW02TD8jTv3aVIJl4iBcEHQTeQU923Yk5MyNvEkOkQNqx hTBg== X-Gm-Message-State: AOJu0Ywjsrwg19SxtTC3GCIogHlbcLy/GoFZ81MAwh10gOuGNLxOJQuu IJHgsAL0MSLQYG+ynAb+Aw6I1G09BWzdzWLIndiyC0qDsUYISGQEfqzmxTncjTKVb+IQsFzqopB 9aStDsWuUaXtxE/hL17J49QFImqwu9TohJzUSXMsydJcagMcUTCltQPa/iSiOUsuf4bF3DZJpM3 hiJDii99Iqhxf5 X-Received: by 2002:a17:906:1453:b0:a31:8320:cebc with SMTP id q19-20020a170906145300b00a318320cebcmr6086759ejc.54.1706582125297; Mon, 29 Jan 2024 18:35:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IGTAty3w+KCy6PtblcuDJdz9NnIm6wawkjWudVzbQ1xvh05E2XUp1RX0HoP98UlyJPt2Uz/FA== X-Received: by 2002:a17:906:1453:b0:a31:8320:cebc with SMTP id q19-20020a170906145300b00a318320cebcmr6086757ejc.54.1706582125008; Mon, 29 Jan 2024 18:35:25 -0800 (PST) Received: from cache-ubuntu.hsd1.nj.comcast.net ([2001:67c:1562:8007::aac:4795]) by smtp.gmail.com with ESMTPSA id vu2-20020a170907a64200b00a35a9745910sm2017165ejc.137.2024.01.29.18.35.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 18:35:24 -0800 (PST) From: Yuxuan Luo To: kernel-team@lists.ubuntu.com Subject: [SRU][Mantic][PATCH 2/2] io_uring/kbuf: defer release of mapped buffer rings Date: Mon, 29 Jan 2024 21:35:19 -0500 Message-Id: <20240130023519.94667-3-yuxuan.luo@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240130023519.94667-1-yuxuan.luo@canonical.com> References: <20240130023519.94667-1-yuxuan.luo@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Jens Axboe If a provided buffer ring is setup with IOU_PBUF_RING_MMAP, then the kernel allocates the memory for it and the application is expected to mmap(2) this memory. However, io_uring uses remap_pfn_range() for this operation, so we cannot rely on normal munmap/release on freeing them for us. Stash an io_buf_free entry away for each of these, if any, and provide a helper to free them post ->release(). Cc: stable@vger.kernel.org Fixes: c56e022c0a27 ("io_uring: add support for user mapped provided buffer ring") Reported-by: Jann Horn Signed-off-by: Jens Axboe (cherry picked from commit c392cbecd8eca4c53f2bf508731257d9d0a21c2d) CVE-2024-0582 Signed-off-by: Yuxuan Luo --- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 2 ++ io_uring/kbuf.c | 44 ++++++++++++++++++++++++++++++---- io_uring/kbuf.h | 2 ++ 4 files changed, 46 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 13d19b9be9f4a..5fd664fb71c86 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -327,6 +327,9 @@ struct io_ring_ctx { struct list_head io_buffers_cache; + /* deferred free list, protected by ->uring_lock */ + struct hlist_head io_buf_list; + /* Keep this last, we don't need it for the fast path */ struct wait_queue_head poll_wq; struct io_restriction restrictions; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1b0a27fe41eef..4c21b068b7578 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -323,6 +323,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); + INIT_HLIST_HEAD(&ctx->io_buf_list); io_alloc_cache_init(&ctx->rsrc_node_cache, IO_NODE_ALLOC_CACHE_MAX, sizeof(struct io_rsrc_node)); io_alloc_cache_init(&ctx->apoll_cache, IO_ALLOC_CACHE_MAX, @@ -2942,6 +2943,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) ctx->mm_account = NULL; } io_rings_free(ctx); + io_kbuf_mmap_list_free(ctx); percpu_ref_exit(&ctx->refs); free_uid(ctx->user); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 9123138aa9f48..79cf9131182b4 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -28,6 +28,11 @@ struct io_provide_buf { __u16 bid; }; +struct io_buf_free { + struct hlist_node list; + void *mem; +}; + static inline struct io_buffer_list *io_buffer_get_list(struct io_ring_ctx *ctx, unsigned int bgid) { @@ -218,7 +223,10 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, if (bl->is_mapped) { i = bl->buf_ring->tail - bl->head; if (bl->is_mmap) { - folio_put(virt_to_folio(bl->buf_ring)); + /* + * io_kbuf_list_free() will free the page(s) at + * ->release() time. + */ bl->buf_ring = NULL; bl->is_mmap = 0; } else if (bl->buf_nr_pages) { @@ -523,18 +531,28 @@ static int io_pin_pbuf_ring(struct io_uring_buf_reg *reg, return -EINVAL; } -static int io_alloc_pbuf_ring(struct io_uring_buf_reg *reg, +static int io_alloc_pbuf_ring(struct io_ring_ctx *ctx, + struct io_uring_buf_reg *reg, struct io_buffer_list *bl) { - gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN | __GFP_COMP; + struct io_buf_free *ibf; size_t ring_size; void *ptr; ring_size = reg->ring_entries * sizeof(struct io_uring_buf_ring); - ptr = (void *) __get_free_pages(gfp, get_order(ring_size)); + ptr = io_mem_alloc(ring_size); if (!ptr) return -ENOMEM; + /* Allocate and store deferred free entry */ + ibf = kmalloc(sizeof(*ibf), GFP_KERNEL_ACCOUNT); + if (!ibf) { + io_mem_free(ptr); + return -ENOMEM; + } + ibf->mem = ptr; + hlist_add_head(&ibf->list, &ctx->io_buf_list); + bl->buf_ring = ptr; bl->is_mapped = 1; bl->is_mmap = 1; @@ -591,7 +609,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) if (!(reg.flags & IOU_PBUF_RING_MMAP)) ret = io_pin_pbuf_ring(®, bl); else - ret = io_alloc_pbuf_ring(®, bl); + ret = io_alloc_pbuf_ring(ctx, ®, bl); if (!ret) { bl->nr_entries = reg.ring_entries; @@ -641,3 +659,19 @@ void *io_pbuf_get_address(struct io_ring_ctx *ctx, unsigned long bgid) return bl->buf_ring; } + +/* + * Called at or after ->release(), free the mmap'ed buffers that we used + * for memory mapped provided buffer rings. + */ +void io_kbuf_mmap_list_free(struct io_ring_ctx *ctx) +{ + struct io_buf_free *ibf; + struct hlist_node *tmp; + + hlist_for_each_entry_safe(ibf, tmp, &ctx->io_buf_list, list) { + hlist_del(&ibf->list); + io_mem_free(ibf->mem); + kfree(ibf); + } +} diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index d14345ef61fc8..ecb5f955cd42e 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -51,6 +51,8 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags); int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); +void io_kbuf_mmap_list_free(struct io_ring_ctx *ctx); + unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); void io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags);