From patchwork Fri May 3 18:17:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 1931154 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=GslsdE0N; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VWJwr71pCz1ydX for ; Sat, 4 May 2024 04:19:24 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=GslsdE0N; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VWJwr4HfJz3vZK for ; Sat, 4 May 2024 04:19:24 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=GslsdE0N; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--dmatlack.bounces.google.com (client-ip=2607:f8b0:4864:20::1149; helo=mail-yw1-x1149.google.com; envelope-from=3rco1zggkdkuirfyqfhplttlqj.htrqnszcuuh-ijaqnxyx.teqfgx.twl@flex--dmatlack.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VWJtx6LBWz3dBy for ; Sat, 4 May 2024 04:17:44 +1000 (AEST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-61be23bb01aso110294087b3.2 for ; Fri, 03 May 2024 11:17:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1714760260; x=1715365060; darn=lists.ozlabs.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gxJS1crzpielwDie8rL3hx1PCStQKnU+yTalK3ZSiI0=; b=GslsdE0NFSKGOOns1WJhvMe0BpasHfKKfjMt60Et2JumbWYtzMymolY+4Zhg13blN1 FfgpL5aVGFycz3Myq9tYQ3hSjhvTGqzSlguruYPLgV5kCKCjFu9rDXIhMOwlMk3P9dc9 yw4e1GvvhxhEROGI/pUI7PZf9tcsnNjcsThamp03eOBeluEU3o2DiJIqJaqC3gDzrHw7 hM0YMjC8rEIQ9fqkpRCtq9kRliUHtIWQd8UYGecJVy+M+cyb6+UCeMpC0IXDBwZ5PTxN aBWuZGq/JNKtqcxRGCtamTEf6WkwFpnulXxS9jKhyUvv6olnhtJ9HDJMsJkJpAQaxYD2 H01A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714760260; x=1715365060; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gxJS1crzpielwDie8rL3hx1PCStQKnU+yTalK3ZSiI0=; b=k8UZ+MedAG5p+uMlHtg6xMPej7J+xZFggTi0F36tLc8BVp0TApoqCU+iX6ZeKcnK5U 6GTaOsvNss5OE8K+pwS2LYN1EJeBUnT7lHiB54hpkftTB/bKtHB7Qvzb0t8Yr6lO5H8h iRdRjIIroCZXjD5+26IPjo0JIy8mPwgyyHorTnpMepMSQQGkXGAGLPoy6o3elNZKXvbN D1SftjqUGOAaQcKIYLUAbRFf6ZKsJGmMCDVOIjt5dVFhHD7Bqiy8dyc1tMDRenpT92Wf JG+o8hDJKn81OXaI2wSd3WQyn/WW4YhLleRDLBRCfiJYvdwQmGFh9P92eZvLQ4qYX/SQ 9RxA== X-Forwarded-Encrypted: i=1; AJvYcCVhBC+xW8Gd4ohK4FOl+zpAS6r2WxDkGKRD+LrsVkXgZaG5TubvbfFjRKT1PX/d3Or4qgWmjkfQ1uePgd8mq2Ox9Z8VTwSpBwvWseyKuA== X-Gm-Message-State: AOJu0Yyl5meolFF8Nt8JSRGMfnTgrg1SzA5Fu46jM6WrzvGJnnmwaQBN zYLdlW1pAS+/eIwI9cjyH7lyUOkNwXxQctyDZoJi6EC+OwYVeD9kCfghWfh6VRhbYqUUP/B3+VV +aMNvSeD5iw== X-Google-Smtp-Source: AGHT+IFV8XW0q6Vc+/y+/ULmmUgE0pK+amRiYPmJG0lNXqT19q15J8nIrLHDkebqo9zBkflGqNFe7OG5YJeiaQ== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a05:6902:1023:b0:dd9:1db5:8348 with SMTP id x3-20020a056902102300b00dd91db58348mr1087385ybt.8.1714760260354; Fri, 03 May 2024 11:17:40 -0700 (PDT) Date: Fri, 3 May 2024 11:17:32 -0700 In-Reply-To: <20240503181734.1467938-1-dmatlack@google.com> Mime-Version: 1.0 References: <20240503181734.1467938-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.0.rc1.225.g2a3ae87e7f-goog Message-ID: <20240503181734.1467938-2-dmatlack@google.com> Subject: [PATCH v3 1/3] KVM: Introduce vcpu->wants_to_run From: David Matlack To: Paolo Bonzini X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , Paul Walmsley , David Matlack , linux-riscv@lists.infradead.org, Claudio Imbrenda , Janosch Frank , Marc Zyngier , Huacai Chen , Zenghui Yu , Palmer Dabbelt , Christian Borntraeger , Albert Ou , Suzuki K Poulose , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Atish Patra , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Sean Christopherson , linux-mips@vger.kernel.org, Oliver Upton , James Morse , kvm-riscv@lists.infradead.org, Anup Patel , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Introduce vcpu->wants_to_run to indicate when a vCPU is in its core run loop, i.e. when the vCPU is running the KVM_RUN ioctl and immediate_exit was not set. Replace all references to vcpu->run->immediate_exit with !vcpu->wants_to_run to avoid TOCTOU races with userspace. For example, a malicious userspace could invoked KVM_RUN with immediate_exit=true and then after KVM reads it to set wants_to_run=false, flip it to false. This would result in the vCPU running in KVM_RUN with wants_to_run=false. This wouldn't cause any real bugs today but is a dangerous landmine. Signed-off-by: David Matlack --- arch/arm64/kvm/arm.c | 2 +- arch/loongarch/kvm/vcpu.c | 2 +- arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/powerpc.c | 2 +- arch/riscv/kvm/vcpu.c | 2 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/x86/kvm/x86.c | 4 ++-- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 3 +++ 9 files changed, 12 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index c4a0a35e02c7..c587e5d9396e 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -986,7 +986,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) vcpu_load(vcpu); - if (run->immediate_exit) { + if (!vcpu->wants_to_run) { ret = -EINTR; goto out; } diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c index 3a8779065f73..847ef54f3a84 100644 --- a/arch/loongarch/kvm/vcpu.c +++ b/arch/loongarch/kvm/vcpu.c @@ -1163,7 +1163,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) kvm_complete_iocsr_read(vcpu, run); } - if (run->immediate_exit) + if (!vcpu->wants_to_run) return r; /* Clear exit_reason */ diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 231ac052b506..f1a99962027a 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -436,7 +436,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) vcpu->mmio_needed = 0; } - if (vcpu->run->immediate_exit) + if (!vcpu->wants_to_run) goto out; lose_fpu(1); diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index d32abe7fe6ab..961aadc71de2 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -1852,7 +1852,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) kvm_sigset_activate(vcpu); - if (run->immediate_exit) + if (!vcpu->wants_to_run) r = -EINTR; else r = kvmppc_vcpu_run(vcpu); diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index b5ca9f2e98ac..3d8349470ee6 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -711,7 +711,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) return ret; } - if (run->immediate_exit) { + if (!vcpu->wants_to_run) { kvm_vcpu_srcu_read_unlock(vcpu); return -EINTR; } diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 5147b943a864..b1ea25aacbf9 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -5033,7 +5033,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (vcpu->kvm->arch.pv.dumping) return -EINVAL; - if (kvm_run->immediate_exit) + if (!vcpu->wants_to_run) return -EINTR; if (kvm_run->kvm_valid_regs & ~KVM_SYNC_S390_VALID_FIELDS || diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 2d2619d3eee4..f70ae1558684 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11396,7 +11396,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) kvm_vcpu_srcu_read_lock(vcpu); if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) { - if (kvm_run->immediate_exit) { + if (!vcpu->wants_to_run) { r = -EINTR; goto out; } @@ -11474,7 +11474,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) WARN_ON_ONCE(vcpu->mmio_needed); } - if (kvm_run->immediate_exit) { + if (!vcpu->wants_to_run) { r = -EINTR; goto out; } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index afbc99264ffa..f9b9ce0c3cd9 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -380,6 +380,7 @@ struct kvm_vcpu { bool dy_eligible; } spin_loop; #endif + bool wants_to_run; bool preempted; bool ready; struct kvm_vcpu_arch arch; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 38b498669ef9..bdea5b978f80 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4425,7 +4425,10 @@ static long kvm_vcpu_ioctl(struct file *filp, synchronize_rcu(); put_pid(oldpid); } + vcpu->wants_to_run = !READ_ONCE(vcpu->run->immediate_exit); r = kvm_arch_vcpu_ioctl_run(vcpu); + vcpu->wants_to_run = false; + trace_kvm_userspace_exit(vcpu->run->exit_reason, r); break; } From patchwork Fri May 3 18:17:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 1931152 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=zpvkrnWA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VWJw44v8vz20fb for ; Sat, 4 May 2024 04:18:44 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=zpvkrnWA; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VWJw43b6rz3dXX for ; Sat, 4 May 2024 04:18:44 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=zpvkrnWA; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--dmatlack.bounces.google.com (client-ip=2607:f8b0:4864:20::b4a; helo=mail-yb1-xb4a.google.com; envelope-from=3rio1zggkdkckthashjrnvvnsl.jvtspubewwj-klcspzaz.vgshiz.vyn@flex--dmatlack.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VWJtx6vRGz3dD1 for ; Sat, 4 May 2024 04:17:44 +1000 (AEST) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-de603db5d6aso9890695276.2 for ; Fri, 03 May 2024 11:17:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1714760262; x=1715365062; darn=lists.ozlabs.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QHXu+3wMl1x8qNonydytLIV6nGk0zgXCPmBNIKdSDOU=; b=zpvkrnWAZi1bNpjXMJrc+fB2TFgNySJ/iEEadoBkLZfQldjs6a097CLhdnteH5Uq52 Wk2ZQmmOioaUvrGesPFGrWCCo02XtLRxDZXLRh6VIUjcY0bZ7qMZo4xB4XJVM2dqFRQX Iu4KClkduQkJb38lLiqrv+cEubQfbHtiJwGbcgk3KnGCdbfsJJOSU2fhB1zf+gvYN/u+ FtjW6ZLqlBKOSJC9KzQkOrJxm1bZClNcJvMvuyU/OYGZQELuInkEN3zDsvSEHxXcwboD sA5CqDWXI2c1D3D5hUa2uAMTbV7N6tupWX7TeU80CbqqTeU3Y3YEdsGX9DhXQ2vUTdGa uEzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714760262; x=1715365062; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QHXu+3wMl1x8qNonydytLIV6nGk0zgXCPmBNIKdSDOU=; b=CvQR20AJP3NZPufHlQxOGPuu+LHiyVrCvzRxziYaGLtCGXeqEvXyY2E7K7ul3cx8Di 2wFO2ApHxZj09t8Ff/x5T4OJwUVyoUNB9KGh+elgsZR/QmuHofKFAQdKxbx1YFyBSSNh GR57gBxkKj8UEiLBlraxcm4K20q9525LL8NtkFWfL9VSjKrbdpJ3y6c0QWtRzqGdiugm dZqQmHcdFxtFgJWMmnfeTbENu2y7voFQar1EYxb4ZmkjZsr1R/JLrN7AQqEqrCLchf7R vGmalp9JAA0nm/BCN6VSP6zYgeDrXgWmQau3g2UBH63G+Mw/GExaZdoNb/sZYfmSuejk ogzQ== X-Forwarded-Encrypted: i=1; AJvYcCXvsnAwyVUXO+6HbAorsdbQv96avb7FEPNqepsUczxbjHuN4P9d0CcFkdRqihyk4wKXFJj08eHXAAZEy/v1vAtTrGwI9MhKEGcDCVX5Jg== X-Gm-Message-State: AOJu0YwNrWMnaxON8NJo8TtqG/XpWJtJvihjhrEXZeuA6wCkaxv5RgJU s2f8F1x8kWgNLgMjDoEL1eAP+hi/xDvygj++KdRiC7hvqGbCW3B7nF9WSEaFI7loZOmRXiIR8sx ovyy548wPaA== X-Google-Smtp-Source: AGHT+IH9qLn6o/uHe9WEM8VyAqFxWSMZ3MFnxC1YcYZ+IPtGArmXDrrlLFlgmBP/HenuKUWhjozjdfJl16Pvww== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a25:26cc:0:b0:de4:e042:eee9 with SMTP id m195-20020a2526cc000000b00de4e042eee9mr1003169ybm.6.1714760262024; Fri, 03 May 2024 11:17:42 -0700 (PDT) Date: Fri, 3 May 2024 11:17:33 -0700 In-Reply-To: <20240503181734.1467938-1-dmatlack@google.com> Mime-Version: 1.0 References: <20240503181734.1467938-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.0.rc1.225.g2a3ae87e7f-goog Message-ID: <20240503181734.1467938-3-dmatlack@google.com> Subject: [PATCH v3 2/3] KVM: Ensure new code that references immediate_exit gets extra scrutiny From: David Matlack To: Paolo Bonzini X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , Paul Walmsley , David Matlack , linux-riscv@lists.infradead.org, Claudio Imbrenda , Janosch Frank , Marc Zyngier , Huacai Chen , Zenghui Yu , Palmer Dabbelt , Christian Borntraeger , Albert Ou , Suzuki K Poulose , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Atish Patra , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Sean Christopherson , linux-mips@vger.kernel.org, Oliver Upton , James Morse , kvm-riscv@lists.infradead.org, Anup Patel , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Ensure that any new KVM code that references immediate_exit gets extra scrutiny by renaming it to immediate_exit__unsafe in kernel code. All fields in struct kvm_run are subject to TOCTOU races since they are mapped into userspace, which may be malicious or buggy. To protect KVM, this commit introduces a new macro that appends __unsafe to field names in struct kvm_run, hinting to developers and reviewers that accessing this field must be done carefully. Apply the new macro to immediate_exit, since userspace can make immediate_exit inconsistent with vcpu->wants_to_run, i.e. accessing immediate_exit directly could lead to unexpected bugs in the future. Signed-off-by: David Matlack --- include/uapi/linux/kvm.h | 15 ++++++++++++++- virt/kvm/kvm_main.c | 2 +- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 2190adbe3002..3611ad3b9c2a 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -192,11 +192,24 @@ struct kvm_xen_exit { /* Flags that describe what fields in emulation_failure hold valid data. */ #define KVM_INTERNAL_ERROR_EMULATION_FLAG_INSTRUCTION_BYTES (1ULL << 0) +/* + * struct kvm_run can be modified by userspace at any time, so KVM must be + * careful to avoid TOCTOU bugs. In order to protect KVM, HINT_UNSAFE_IN_KVM() + * renames fields in struct kvm_run from to __unsafe when + * compiled into the kernel, ensuring that any use within KVM is obvious and + * gets extra scrutiny. + */ +#ifdef __KERNEL__ +#define HINT_UNSAFE_IN_KVM(_symbol) _symbol##__unsafe +#else +#define HINT_UNSAFE_IN_KVM(_symbol) _symbol +#endif + /* for KVM_RUN, returned by mmap(vcpu_fd, offset=0) */ struct kvm_run { /* in */ __u8 request_interrupt_window; - __u8 immediate_exit; + __u8 HINT_UNSAFE_IN_KVM(immediate_exit); __u8 padding1[6]; /* out */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index bdea5b978f80..2b29851a90bd 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4425,7 +4425,7 @@ static long kvm_vcpu_ioctl(struct file *filp, synchronize_rcu(); put_pid(oldpid); } - vcpu->wants_to_run = !READ_ONCE(vcpu->run->immediate_exit); + vcpu->wants_to_run = !READ_ONCE(vcpu->run->immediate_exit__unsafe); r = kvm_arch_vcpu_ioctl_run(vcpu); vcpu->wants_to_run = false; From patchwork Fri May 3 18:17:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 1931148 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=OvPGP6ox; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VWJvL2M0rz1ydX for ; Sat, 4 May 2024 04:18:05 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=OvPGP6ox; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VWJvH4mtTz3dFx for ; Sat, 4 May 2024 04:18:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=OvPGP6ox; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--dmatlack.bounces.google.com (client-ip=2607:f8b0:4864:20::1149; helo=mail-yw1-x1149.google.com; envelope-from=3ryo1zggkdkgluibtiksowwotm.kwutqvcfxxk-lmdtqaba.whtija.wzo@flex--dmatlack.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VWJtx6XBYz3dC5 for ; Sat, 4 May 2024 04:17:45 +1000 (AEST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6202fd268c1so8456237b3.2 for ; Fri, 03 May 2024 11:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1714760263; x=1715365063; darn=lists.ozlabs.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XCWz5AboezUKY+fKcSkwtkc56yFvTydQOL6+C1uV1TA=; b=OvPGP6oxxRqqVKeZq5k7FO1qXIyqfbDpO/KPf61DMwOLLRmba4ejgmwF3vI8TM3wm+ I+VsbEvthXrog7g0zcuCCriKwNaUFSjh01wVeaacKYYCVfVvHj+IPmrMe77Cc8td/top cUTF7xvFlnAbBjJLV+dc4bfymlrJA7LFnlz6LOdwDyUkn0j8zi0i0EYpUgb4fizn6yik ekBlSJy6SToAfKSY+tHC6/Vh+3NxlfWJFwE59bYjhMVyy3drRDKkfX9IRqX6r7JCttvj E0uVN4xNaO5nNNjoDfdlRhQPyRNuzCTp/HTuwRmYK79OcI8HlOTQZkue57lS1dgkvQFR IhcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714760263; x=1715365063; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XCWz5AboezUKY+fKcSkwtkc56yFvTydQOL6+C1uV1TA=; b=Zpk4K7HvyYixuXhm6roMbfKkdqr6IT/1i+kNWtOLNyRSZp24wD45Ar/JCjZ6csR1Z/ 4Yu4x4XrqXGRb+GYwoPQyoh8NP1cMYM4SoSf+nNc6MzpdaTFcehPn+s5tcct4kyv5MPa 1iMUDqbRhp5DEIjZ862+jLmBrcWgc1FiHa2lM07Jc1sXdVVSMOIqABtjRPhwubWuRzIA 7wh9j2dYIlLvuNstX89EIwKV8QnZynWbqvKtUjBAF3j8aZaI69GYT7ddPQsQvO622RLA ojv+MRX1VMVYoWdZVI4cOgMTS862fc9btLnaMyQ+PO2WebIIrS+i5ydozZW38iiaila8 XA8A== X-Forwarded-Encrypted: i=1; AJvYcCU2otmtRdugQayU6uPtFhQmgeRl/4JH7m3WZLAsEJcm/h4s8cM3BHfzvwbAqBdZ96H/sGz8hwnZcQ9jw3sbXbGgHWHJJDOGWiJlW2VBjg== X-Gm-Message-State: AOJu0YyQMTWOkfmnqiISWkNZEbagxfoZPkvKn7jxa/FuLZf42dF1Twqm /Tb5/7znoOdboyVQ5bv2OqifLUAhjbUgT5QtzjxsVM6t2nQ5jc2e6Rs1boJjsiILtvITZ20HWGs 1vQcDHFckaA== X-Google-Smtp-Source: AGHT+IG8z91B85kZw12g8ILejBDce9nCeI6uZrPNjYvj7vQXfcVKq57xpnYqIWHHCJ8gvpfpvg4aSbrDSFG7Mg== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a0d:ea05:0:b0:61b:7912:6cad with SMTP id t5-20020a0dea05000000b0061b79126cadmr815080ywe.2.1714760263504; Fri, 03 May 2024 11:17:43 -0700 (PDT) Date: Fri, 3 May 2024 11:17:34 -0700 In-Reply-To: <20240503181734.1467938-1-dmatlack@google.com> Mime-Version: 1.0 References: <20240503181734.1467938-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.0.rc1.225.g2a3ae87e7f-goog Message-ID: <20240503181734.1467938-4-dmatlack@google.com> Subject: [PATCH v3 3/3] KVM: Mark a vCPU as preempted/ready iff it's scheduled out while running From: David Matlack To: Paolo Bonzini X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , Paul Walmsley , David Matlack , linux-riscv@lists.infradead.org, Claudio Imbrenda , Janosch Frank , Marc Zyngier , Huacai Chen , Zenghui Yu , Palmer Dabbelt , Christian Borntraeger , Albert Ou , Suzuki K Poulose , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Atish Patra , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Sean Christopherson , linux-mips@vger.kernel.org, Oliver Upton , James Morse , kvm-riscv@lists.infradead.org, Anup Patel , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Mark a vCPU as preempted/ready if-and-only-if it's scheduled out while running. i.e. Do not mark a vCPU preempted/ready if it's scheduled out during a non-KVM_RUN ioctl() or when userspace is doing KVM_RUN with immediate_exit. Commit 54aa83c90198 ("KVM: x86: do not set st->preempted when going back to user space") stopped marking a vCPU as preempted when returning to userspace, but if userspace then invokes a KVM vCPU ioctl() that gets preempted, the vCPU will be marked preempted/ready. This is arguably incorrect behavior since the vCPU was not actually preempted while the guest was running, it was preempted while doing something on behalf of userspace. This commit also avoids KVM dirtying guest memory after userspace has paused vCPUs, e.g. for Live Migration, which allows userspace to collect the final dirty bitmap before or in parallel with saving vCPU state without having to worry about saving vCPU state triggering writes to guest memory. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 2b29851a90bd..3973e62acc7c 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -6302,7 +6302,7 @@ static void kvm_sched_out(struct preempt_notifier *pn, { struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn); - if (current->on_rq) { + if (current->on_rq && vcpu->wants_to_run) { WRITE_ONCE(vcpu->preempted, true); WRITE_ONCE(vcpu->ready, true); }