From patchwork Mon Apr 15 20:59:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bethany Jamison X-Patchwork-Id: 1923919 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VJKLN56kxz1yZh for ; Tue, 16 Apr 2024 06:59:56 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rwTQX-000685-Cw; Mon, 15 Apr 2024 20:59:45 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rwTQV-00067P-Cg for kernel-team@lists.ubuntu.com; Mon, 15 Apr 2024 20:59:43 +0000 Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 1827A3F67B for ; Mon, 15 Apr 2024 20:59:43 +0000 (UTC) Received: by mail-io1-f70.google.com with SMTP id ca18e2360f4ac-7d63c6a2559so278573339f.1 for ; Mon, 15 Apr 2024 13:59:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713214782; x=1713819582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L/nR8kJh3kWgL4sRCe4U+aKM3SXD951ctr9C60inHFw=; b=muGConCpJm5NhkyhVvxF5AdNdSYVhKfI+lI6lfALAZBeXYwO6qykZ2WgLDMcUfZrUL Ko1W33FY/mawsIQ6TZa3D4Oz9bOS5F9ToNd6UPWVosY//C/bTSb2cdjQj0QzbmyHb/u8 Yve3WhFS2oxLc2Y+7w43aTYRPlb/2JR8xlA4kH3+AJIaaGHRg0yQPwf4aDGh2xoujDdg C05mty5GzVay+x4Ysq1HIgXtrww5JvFgIfIr44WKgEfYfDEKGMKhxfLvjLZpPw5NSXN0 QQ+zhOGnQrCOJwjwK70Nyvd0IYfj9e4W3iHqSRyuLhSbNs9Mf6RKQjChPg486nHLWdEE Nz3g== X-Gm-Message-State: AOJu0YyAmpE8JosURi0/8C/BbdbyfzYQsApZd3cSHWsOSVZ7VM3/OANJ OxE1ESQVvrNCcP9Z9kq319Rk2LUbTDwA2KB0gnCVNPSb8clYeFE1Jn0EMtQXV0RBVsnFw+QPrx3 hcvDe8JkM1QPuOqG7ruCX7SjqhCW0cbcbzL4ykiM6LONtK3wjctEgJJ7a5IDpZ4bOu3UeF9E+Jw AOZuYuqumeyg== X-Received: by 2002:a5e:d904:0:b0:7d3:436b:af49 with SMTP id n4-20020a5ed904000000b007d3436baf49mr599254iop.9.1713214781803; Mon, 15 Apr 2024 13:59:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkVEVHs7nmmc7AnpvMToWSolb/hr1jQGM6rWx/gBKkoZU76MlYdavcpu/bRmedETM3YjEm+A== X-Received: by 2002:a5e:d904:0:b0:7d3:436b:af49 with SMTP id n4-20020a5ed904000000b007d3436baf49mr599249iop.9.1713214781471; Mon, 15 Apr 2024 13:59:41 -0700 (PDT) Received: from smtp.gmail.com (104-218-69-129.dynamic.lnk.ne.allofiber.net. [104.218.69.129]) by smtp.gmail.com with ESMTPSA id c35-20020a029626000000b004828f584db0sm3473435jai.80.2024.04.15.13.59.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 13:59:41 -0700 (PDT) From: Bethany Jamison To: kernel-team@lists.ubuntu.com Subject: [SRU][J][PATCH 1/2] bpf: Defer the free of inner map when necessary Date: Mon, 15 Apr 2024 15:59:38 -0500 Message-Id: <20240415205939.141814-2-bethany.jamison@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415205939.141814-1-bethany.jamison@canonical.com> References: <20240415205939.141814-1-bethany.jamison@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Hou Tao [ Upstream commit 876673364161da50eed6b472d746ef88242b2368 ] When updating or deleting an inner map in map array or map htab, the map may still be accessed by non-sleepable program or sleepable program. However bpf_map_fd_put_ptr() decreases the ref-counter of the inner map directly through bpf_map_put(), if the ref-counter is the last one (which is true for most cases), the inner map will be freed by ops->map_free() in a kworker. But for now, most .map_free() callbacks don't use synchronize_rcu() or its variants to wait for the elapse of a RCU grace period, so after the invocation of ops->map_free completes, the bpf program which is accessing the inner map may incur use-after-free problem. Fix the free of inner map by invoking bpf_map_free_deferred() after both one RCU grace period and one tasks trace RCU grace period if the inner map has been removed from the outer map before. The deferment is accomplished by using call_rcu() or call_rcu_tasks_trace() when releasing the last ref-counter of bpf map. The newly-added rcu_head field in bpf_map shares the same storage space with work field to reduce the size of bpf_map. Fixes: bba1dc0b55ac ("bpf: Remove redundant synchronize_rcu.") Fixes: 638e4b825d52 ("bpf: Allows per-cpu maps and map-in-map in sleepable programs") Signed-off-by: Hou Tao Link: https://lore.kernel.org/r/20231204140425.1480317-5-houtao@huaweicloud.com Signed-off-by: Alexei Starovoitov Signed-off-by: Sasha Levin (cherry picked from commit 62fca83303d608ad4fec3f7428c8685680bb01b0) Signed-off-by: Robert Kolchmeyer Signed-off-by: Sasha Levin (cherry picked from commit 37d98fb9c3144c0fddf7f6e99aece9927ac8dce6 linux-5.15.y) CVE-2023-52447 Signed-off-by: Bethany Jamison --- include/linux/bpf.h | 7 ++++++- kernel/bpf/map_in_map.c | 11 ++++++++--- kernel/bpf/syscall.c | 26 ++++++++++++++++++++++++-- 3 files changed, 38 insertions(+), 6 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 48f3cc3bafea7..5e24027c24714 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -192,9 +192,14 @@ struct bpf_map { */ atomic64_t refcnt ____cacheline_aligned; atomic64_t usercnt; - struct work_struct work; + /* rcu is used before freeing and work is only used during freeing */ + union { + struct work_struct work; + struct rcu_head rcu; + }; struct mutex freeze_mutex; atomic64_t writecnt; + bool free_after_mult_rcu_gp; }; static inline bool map_value_has_spin_lock(const struct bpf_map *map) diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c index af0f15db1bf9a..4cf79f86bf458 100644 --- a/kernel/bpf/map_in_map.c +++ b/kernel/bpf/map_in_map.c @@ -110,10 +110,15 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, void bpf_map_fd_put_ptr(struct bpf_map *map, void *ptr, bool need_defer) { - /* ptr->ops->map_free() has to go through one - * rcu grace period by itself. + struct bpf_map *inner_map = ptr; + + /* The inner map may still be used by both non-sleepable and sleepable + * bpf program, so free it after one RCU grace period and one tasks + * trace RCU grace period. */ - bpf_map_put(ptr); + if (need_defer) + WRITE_ONCE(inner_map->free_after_mult_rcu_gp, true); + bpf_map_put(inner_map); } u32 bpf_map_fd_sys_lookup_elem(void *ptr) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 64206856a05c4..d4b4a47081b51 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -487,6 +487,25 @@ static void bpf_map_put_uref(struct bpf_map *map) } } +static void bpf_map_free_in_work(struct bpf_map *map) +{ + INIT_WORK(&map->work, bpf_map_free_deferred); + schedule_work(&map->work); +} + +static void bpf_map_free_rcu_gp(struct rcu_head *rcu) +{ + bpf_map_free_in_work(container_of(rcu, struct bpf_map, rcu)); +} + +static void bpf_map_free_mult_rcu_gp(struct rcu_head *rcu) +{ + if (rcu_trace_implies_rcu_gp()) + bpf_map_free_rcu_gp(rcu); + else + call_rcu(rcu, bpf_map_free_rcu_gp); +} + /* decrement map refcnt and schedule it for freeing via workqueue * (unrelying map implementation ops->map_free() might sleep) */ @@ -496,8 +515,11 @@ static void __bpf_map_put(struct bpf_map *map, bool do_idr_lock) /* bpf_map_free_id() must be called first */ bpf_map_free_id(map, do_idr_lock); btf_put(map->btf); - INIT_WORK(&map->work, bpf_map_free_deferred); - schedule_work(&map->work); + + if (READ_ONCE(map->free_after_mult_rcu_gp)) + call_rcu_tasks_trace(&map->rcu, bpf_map_free_mult_rcu_gp); + else + bpf_map_free_in_work(map); } } From patchwork Mon Apr 15 20:59:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bethany Jamison X-Patchwork-Id: 1923920 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VJKLN65Zlz1yb4 for ; Tue, 16 Apr 2024 06:59:55 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rwTQY-00069P-Ky; Mon, 15 Apr 2024 20:59:46 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rwTQV-00067Z-U2 for kernel-team@lists.ubuntu.com; Mon, 15 Apr 2024 20:59:43 +0000 Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id A701E3F67B for ; Mon, 15 Apr 2024 20:59:43 +0000 (UTC) Received: by mail-io1-f69.google.com with SMTP id ca18e2360f4ac-7cf265b30e2so432893739f.1 for ; Mon, 15 Apr 2024 13:59:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713214782; x=1713819582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=meR5vdKeQKMycaE3FYTYxYRqLHHafNHdHMQFtYr+3p0=; b=JTzwGXC2Kg0trOSn86CuSMhhlKjSm16RIZkMEoxoTDaSMz8s5D0yQYjIcRRcGPMZfo Od53wghX469gaCjw9HJuu7TfNUb66R+61wW14TnwjURkP+5VxrvBDcqQpvzGf1jRrvkl ipSIFmhBLt9Kta3sMe+BtErzXcYIzvM6xM+FlHD3SFbf8+tI+x/8y4CWMOdPdJsHJq8f 3I7/dKf7Rmi47NcoY4D3w0XsKWNAYXW72G9pKejl4VsGh53+vkVH1TnXo6R1zgD/y2Ei shhDpjLb2IkOw5BKhEuln+MOJQHBYmR8QAJzjMGdK2MS8CEAdlV4tdnaLwzdZFZvBST9 ZeVg== X-Gm-Message-State: AOJu0Yzl5gE38U0n4ngz3hUDJNpfKRlTglbx7JLlmfdJF7McHn5eeX5r 4Dsa0oFeBzXFTGLFQzbyDWYD4lWvXPmaZYlpx3M94t18+JzPswEnzGr444THnLYxfGKgEJN39cg 9la9nn5n5CHhbDk4yWdS9Gnck1VMYX0/L+FOisG2TGc7YH8OpN/R7tL/Zxqd7HsLVwe4xDBYo++ 2IgaIEARkYeg== X-Received: by 2002:a05:6602:3d6:b0:7d5:e175:495d with SMTP id g22-20020a05660203d600b007d5e175495dmr15077113iov.21.1713214782393; Mon, 15 Apr 2024 13:59:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGZygUeJxxDeXqruRR+Uz4PCEmGmzJyZeQItlF7eu+lFhF7NIEhEVhXADTIf9ikR+QmdTSlaQ== X-Received: by 2002:a05:6602:3d6:b0:7d5:e175:495d with SMTP id g22-20020a05660203d600b007d5e175495dmr15077105iov.21.1713214782073; Mon, 15 Apr 2024 13:59:42 -0700 (PDT) Received: from smtp.gmail.com (104-218-69-129.dynamic.lnk.ne.allofiber.net. [104.218.69.129]) by smtp.gmail.com with ESMTPSA id c35-20020a029626000000b004828f584db0sm3473435jai.80.2024.04.15.13.59.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 13:59:41 -0700 (PDT) From: Bethany Jamison To: kernel-team@lists.ubuntu.com Subject: [SRU][J][PATCH 2/2] rcu-tasks: Provide rcu_trace_implies_rcu_gp() Date: Mon, 15 Apr 2024 15:59:39 -0500 Message-Id: <20240415205939.141814-3-bethany.jamison@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415205939.141814-1-bethany.jamison@canonical.com> References: <20240415205939.141814-1-bethany.jamison@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "Paul E. McKenney" As an accident of implementation, an RCU Tasks Trace grace period also acts as an RCU grace period. However, this could change at any time. This commit therefore creates an rcu_trace_implies_rcu_gp() that currently returns true to codify this accident. Code relying on this accident must call this function to verify that this accident is still happening. Reported-by: Hou Tao Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Martin KaFai Lau Link: https://lore.kernel.org/r/20221014113946.965131-2-houtao@huaweicloud.com Signed-off-by: Alexei Starovoitov (cherry picked from commit e6c86c513f440bec5f1046539c7e3c6c653842da) CVE-2023-52447 Signed-off-by: Bethany Jamison --- include/linux/rcupdate.h | 12 ++++++++++++ kernel/rcu/tasks.h | 2 ++ 2 files changed, 14 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 13bddb841ceb1..e3b12de36e924 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -205,6 +205,18 @@ static inline void exit_tasks_rcu_stop(void) { } static inline void exit_tasks_rcu_finish(void) { } #endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ +/** + * rcu_trace_implies_rcu_gp - does an RCU Tasks Trace grace period imply an RCU grace period? + * + * As an accident of implementation, an RCU Tasks Trace grace period also + * acts as an RCU grace period. However, this could change at any time. + * Code relying on this accident must call this function to verify that + * this accident is still happening. + * + * You have been warned! + */ +static inline bool rcu_trace_implies_rcu_gp(void) { return true; } + /** * cond_resched_tasks_rcu_qs - Report potential quiescent states to RCU * diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 96722dae824b5..3ebb0bbe8ede3 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1098,6 +1098,8 @@ static void rcu_tasks_trace_postscan(struct list_head *hop) // Wait for late-stage exiting tasks to finish exiting. // These might have passed the call to exit_tasks_rcu_finish(). + + // If you remove the following line, update rcu_trace_implies_rcu_gp()!!! synchronize_rcu(); // Any tasks that exit after this point will set ->trc_reader_checked. }