From patchwork Mon Apr 11 14:01:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Po-Hsu Lin X-Patchwork-Id: 1615751 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=WQmT9wrz; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KcVtF0Myjz9sBJ for ; Tue, 12 Apr 2022 00:02:44 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1nducm-00026c-7w; Mon, 11 Apr 2022 14:02:36 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1nducj-00025y-Rx for kernel-team@lists.ubuntu.com; Mon, 11 Apr 2022 14:02:33 +0000 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 373D33F80E for ; Mon, 11 Apr 2022 14:02:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1649685753; bh=CfTzDfvZR3UtgToJiKsBz+bVb19Ltt9/fg5C+wJFqsY=; h=From:To:Subject:Date:Message-Id:MIME-Version:Content-Type; b=WQmT9wrzfsCK3s2JnmDM2qfbiAl3wPVY5H2V0ShNJ0AbxV4K9XwFryy/j2Z++bX/k xZ43ebbFL4QHCYez5i69GvezT8xM0PpmtwJH4bQtFc54H343nyR0q2I18/pd1jSxHm l2tc7Izrt1pnUxLSYo2EziGTMloEsO84qMzIu7Mn5n/bT6o9bvN0SD6SztPKs4dVBQ S+ZAOtsJUQ2Mi57DO2AqaVPDLaQR3cBISrzOFNH0VRow/yRVGC0HyEKnbdsy4E1lzy VSBXirZ74JuilILuuO+bru8FkDgsDlxxYzE0AsCRKYdJr9nWNO3LFNSLuNPDBZEiez sGScrskzV+WkQ== Received: by mail-pj1-f72.google.com with SMTP id q10-20020a17090a2dca00b001cb87691fbaso1860129pjm.5 for ; Mon, 11 Apr 2022 07:02:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=CfTzDfvZR3UtgToJiKsBz+bVb19Ltt9/fg5C+wJFqsY=; b=QN56ao7dObljfXwtfmrA4xEYCrVbgBPlyiLpTLGaq+OF7iRh6BIxiKT0VYXMJPdMNQ usD0rlEkrMw8Cx3uAWH0F9YzYhn/pZUFxqkeDjJVvyXctlGp+rGSvgDAKB6SmaA2Ckfh FXiKtgKAa5EtCIr5lai3kxUFC47n6tCil/4Win10H7IT1WbAOQ/YU2tuQv5jZ9GDoUng AJ8OrE9UzPRu6Vr5VZFJUdWfJPr3Q9piaw2SksckTu8fpXwQaSWIwNMQwki3sLzy1BPi +Vn4zDusLj85rMsmBAMklEywA6Z4ora//XzQyfSrvlONYWSE4wIJT/ukrhllysKSX8g1 c1AQ== X-Gm-Message-State: AOAM533+RuJjRnkaElw6lUOCk72fant1Ce9+36YYPILfWezFQu5t3GoW leztD9ek8bsZThqGjAmOXC0MY3Pd49kfB1OD4Pivh9emROMBI1T9V15iVp+8lf9W8JMGthZbS6K IFpYFqvaRvAsabO0PYWm2EjJU1r+cZJfwbpgcQV13 X-Received: by 2002:a17:90a:560a:b0:1bc:72e7:3c13 with SMTP id r10-20020a17090a560a00b001bc72e73c13mr36668157pjf.246.1649685751488; Mon, 11 Apr 2022 07:02:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQIqtHwZuWuPw5bn6djckdGvbFKlLw0jtCSrEMfVIqJUQ/0xT1b3J3DFfGWtwfs5Vm16cdGA== X-Received: by 2002:a17:90a:560a:b0:1bc:72e7:3c13 with SMTP id r10-20020a17090a560a00b001bc72e73c13mr36668123pjf.246.1649685751113; Mon, 11 Apr 2022 07:02:31 -0700 (PDT) Received: from localhost.localdomain (223-136-39-236.emome-ip.hinet.net. [223.136.39.236]) by smtp.gmail.com with ESMTPSA id x6-20020a17090a294600b001cba3274bd0sm2742157pjf.28.2022.04.11.07.02.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Apr 2022 07:02:29 -0700 (PDT) From: Po-Hsu Lin To: kernel-team@lists.ubuntu.com Subject: [Impish][SRU][PATCH 0/3] Fix KVM regression on Impish Date: Mon, 11 Apr 2022 22:01:54 +0800 Message-Id: <20220411140157.97354-1-po-hsu.lin@canonical.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" [Impact] This is caused by commit 08335308 "KVM: x86: check PIR even for vCPUs with disabled APICv", this patch needs 7e1901f6c "KVM: VMX: prepare sync_pir_to_irr for running with APICv disabled" otherwise if APICv is disabled in this vcpu it will trigger warning messages in vmx_sync_pir_to_irr() of vmx.c: WARN_ON(!vcpu->arch.apicv_active); With warnings like: ------------[ cut here ]------------ WARNING: CPU: 13 PID: 6997 at arch/x86/kvm/vmx/vmx.c:6336 vmx_sync_pir_to_irr+0x9e/0xc0 [kvm_intel] ? xfer_to_guest_mode_work+0xe2/0x110 Modules linked in: vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev input_leds ioatdma rapl intel_cstate efi_pstore ipmi_si mei_me mei mac_hid acpi_pad vcpu_run+0x4d/0x220 [kvm] acpi_power_meter sch_fq_codel ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea crc32_pclmul sysfillrect sysimgblt ghash_clmulni_intel fb_sys_fops ixgbe cec aesni_intel rc_core crypto_simd xfrm_algo cryptd drm ahci dca i2c_i801 xhci_pci mdio libahci i2c_smbus lpc_ich xhci_pci_renesas wmi CPU: 13 PID: 6997 Comm: qemu-system-x86 Tainted: G W I 5.13.0-39-generic #44-Ubuntu Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.1008.031920151331 03/19/2015 kvm_arch_vcpu_ioctl_run+0xc5/0x4f0 [kvm] RIP: 0010:vmx_sync_pir_to_irr+0x9e/0xc0 [kvm_intel] Code: e8 47 f5 18 00 8b 93 00 03 00 00 89 45 ec 83 e2 20 85 d2 74 dc 48 8b 55 f0 65 48 2b 14 25 28 00 00 00 75 1d 48 8b 5d f8 c9 c3 <0f> 0b eb 87 f0 80 4b 39 40 8b 93 00 03 00 00 8b 45 ec 83 e2 20 eb RSP: 0018:ffffae4d8d107c98 EFLAGS: 00010046 RAX: 0000000000000000 RBX: ffff99c552942640 RCX: ffff99c5043a72f0 RDX: ffff99c552942640 RSI: 0000000000000001 RDI: ffff99c552942640 RBP: ffffae4d8d107cb0 R08: ffff99c86f6a7140 R09: 0000000000027100 R10: 0000000042280000 R11: 000000000000000a R12: ffff99c552942640 R13: 0000000000000000 R14: ffffae4d8d1a63e0 R15: ffff99c552942640 FS: 00007f6ae9be7640(0000) GS:ffff99c86f680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000010b8a6006 CR4: 00000000001726e0 Call Trace: kvm_vcpu_ioctl+0x243/0x5e0 [kvm] vcpu_enter_guest+0x383/0xf50 [kvm] ? xfer_to_guest_mode_work+0xe2/0x110 ? kvm_vm_ioctl+0x364/0x730 [kvm] ? __fget_files+0x86/0xc0 vcpu_run+0x4d/0x220 [kvm] __x64_sys_ioctl+0x91/0xc0 do_syscall_64+0x61/0xb0 ? fput+0x13/0x20 ? exit_to_user_mode_prepare+0x37/0xb0 ? syscall_exit_to_user_mode+0x27/0x50 ? do_syscall_64+0x6e/0xb0 ? syscall_exit_to_user_mode+0x27/0x50 ? do_syscall_64+0x6e/0xb0 ? do_syscall_64+0x6e/0xb0 ? do_syscall_64+0x6e/0xb0 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f6aebce1a2b Code: ff ff ff 85 c0 79 8b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 f3 0f 00 f7 d8 64 89 01 48 RSP: 002b:00007f6ae8ffe3f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f6aebce1a2b RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000000c RBP: 0000557d3b429b90 R08: 0000557d3a4ebff0 R09: 00000000ffffffff kvm_arch_vcpu_ioctl_run+0xc5/0x4f0 [kvm] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000001 R14: 0000000000003000 R15: 0000000000000000 ---[ end trace 5b722d71a78069b1 ]--- This warning message will be flooding in system log files and eventually eat up all the disk space then crash the server. This issue will gone by either reverting it or adding the fixes below. Reference: https://patchwork.kernel.org/project/kvm/patch/20211118072531.1534938-1-pbonzini@redhat.com/ [Fixes] * 0b8f11737 KVM: Add infrastructure and macro to mark VM as bugged * 673692735 KVM: x86: Use KVM_BUG/KVM_BUG_ON to handle bugs that are fatal to the VM * 7e1901f6c KVM: VMX: prepare sync_pir_to_irr for running with APICv disabled The fix comes in two fold, the first two patches will fix the warning message flooding issue, make it only gets printed once. The third patch will change the prevent this to happen. The first patch needs to be backported as we're missing: 2fdef3a2ae kvm: add PM-notifier fcfe1baedd KVM: stats: Support binary stats retrieval for a VM The second patch needs some context adjustment. And the last one can be cherry-picked. [Test] Test kernels can be found here: https://people.canonical.com/~phlin/kernel/lp-1966499-kvm-warn-flood/ This issue can be verified with LXD: 1. snap install lxd 2. lxc launch images:ubuntu/20.04 --vm vm1 On affected system, the dmesg output will be flooded with this warning message. With patched kernel the VM can be started with clean dmesg. I have this kernel tested on Impish, the F-5.13 has been tested by Daniël Vos (vosdev) on launchpad. Both are working as expected. kvm-unit-tests has also been tested on my Impish instance to ensure there is no other issues. [Where problems could occur] This patchset will change how the KVM bug gets reported in the kernel, if it's incorrect it might affect VMX capability. Paolo Bonzini (1): KVM: VMX: prepare sync_pir_to_irr for running with APICv disabled Sean Christopherson (2): KVM: Add infrastructure and macro to mark VM as bugged KVM: x86: Use KVM_BUG/KVM_BUG_ON to handle bugs that are fatal to the VM arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 60 ++++++++++++++++++++++++++++++------------------ arch/x86/kvm/x86.c | 4 ++++ include/linux/kvm_host.h | 28 +++++++++++++++++++++- virt/kvm/kvm_main.c | 10 ++++---- 5 files changed, 75 insertions(+), 29 deletions(-) Acked-by: Stefan Bader Acked-by: Kleber Sacilotto de Souza