From patchwork Sat Jun 13 21:36:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308761 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krmC26rnz9s1x for ; Sun, 14 Jun 2020 07:47:23 +1000 (AEST) Received: from localhost ([::1]:34826 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkDzl-0000Db-1F for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:47:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59076) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDx5-00058v-LT; Sat, 13 Jun 2020 17:44:35 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:60464 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDx3-00035a-Qu; Sat, 13 Jun 2020 17:44:35 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 2414FF40CEBE9F8A1DDE; Sun, 14 Jun 2020 05:44:25 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:17 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 01/22] arm/cpuhp: Add QMP vcpu params validation support Date: Sat, 13 Jun 2020 22:36:08 +0100 Message-ID: <20200613213629.21984-2-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:26 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" For now, vcpu hotplug is only supported with single socket single thread, single die. NUMA is not supported either and everthing falls into single node. Work to properly support these could be taken later once community agrees with the base framework changes being presented to support ARM vcpu hotplug in QEMU. Hence, these checks. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 37462a6f78..5d1afdd031 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2201,6 +2201,46 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, return NULL; } +static void virt_smp_parse(MachineState *ms, QemuOpts *opts) +{ + if (opts) { + unsigned cpus = qemu_opt_get_number(opts, "cpus", 1); + unsigned sockets = qemu_opt_get_number(opts, "sockets", 1); + unsigned cores = qemu_opt_get_number(opts, "cores", cpus); + unsigned threads = qemu_opt_get_number(opts, "threads", 1); + unsigned int max_cpus; + + if (sockets > 1 || threads > 1) { + error_report("does not support more than one socket or thread"); + exit(1); + } + + if (cores != cpus) { + error_report("cpu topology: " + "sockets (%u) * cores (%u) * threads (%u) < " + "smp_cpus (%u)", + sockets, cores, threads, cpus); + exit(1); + } + + max_cpus = qemu_opt_get_number(opts, "maxcpus", cpus); + if (sockets * cores * threads > max_cpus) { + error_report("cpu topology: " + "sockets (%u) * cores (%u) * threads (%u) > " + "maxcpus (%u)", + sockets, cores, threads, + max_cpus); + exit(1); + } + + ms->smp.max_cpus = max_cpus; + ms->smp.sockets = sockets; + ms->smp.cpus = cpus; + ms->smp.cores = cores; + ms->smp.threads = threads; + } +} + /* * for arm64 kvm_type [7-0] encodes the requested number of bits * in the IPA address space @@ -2266,6 +2306,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->nvdimm_supported = true; mc->auto_enable_numa_with_memhp = true; mc->default_ram_id = "mach-virt.ram"; + mc->smp_parse = virt_smp_parse; object_class_property_add(oc, "acpi", "OnOffAuto", virt_get_acpi, virt_set_acpi, From patchwork Sat Jun 13 21:36:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308757 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krjn1Z4nz9sRW for ; Sun, 14 Jun 2020 07:45:15 +1000 (AEST) Received: from localhost ([::1]:55236 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkDxg-0005Cu-PP for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:45:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59082) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDx6-0005A1-JT; Sat, 13 Jun 2020 17:44:36 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3706 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDx4-00036A-F1; Sat, 13 Jun 2020 17:44:36 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 3046C9153F694E10A53E; Sun, 14 Jun 2020 05:44:30 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:23 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 02/22] arm/cpuhp: Add new ARMCPU core-id property Date: Sat, 13 Jun 2020 22:36:09 +0100 Message-ID: <20200613213629.21984-3-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This shall be used to store user specified core index and shall be directly used as slot-index during hot{plug|unplug} of vcpu. For now, we are not taking into account of other topology info like thread-id, socket-id to derive mp-affinity. Host KVM uses vcpu-id to derive the mpidr for the vcpu of the guest. This is not in exact corroboration with the ARM spec view of the MPIDR. Hence, the concept of threads or SMT bit present as part of the MPIDR_EL1 also gets lost. Also, we need ACPI PPTT Table support in QEMU to be able to export this topology info to the guest VM and the info should be consistent with what host cpu supports if accel=kvm is being used. Perhaps some comments on this will help? @Andrew/drjones@redhat.com Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 5 +++++ target/arm/cpu.c | 5 +++++ target/arm/cpu.h | 1 + 3 files changed, 11 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 5d1afdd031..c4ed955776 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1778,6 +1778,7 @@ static void machvirt_init(MachineState *machine) &error_fatal); aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); + object_property_set_int(cpuobj, n, "core-id", NULL); if (!vms->secure) { object_property_set_bool(cpuobj, false, "has_el3", NULL); @@ -2081,6 +2082,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) { int n; unsigned int max_cpus = ms->smp.max_cpus; + unsigned int smp_threads = ms->smp.threads; VirtMachineState *vms = VIRT_MACHINE(ms); if (ms->possible_cpus) { @@ -2093,8 +2095,11 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) ms->possible_cpus->len = max_cpus; for (n = 0; n < ms->possible_cpus->len; n++) { ms->possible_cpus->cpus[n].type = ms->cpu_type; + ms->possible_cpus->cpus[n].vcpus_count = smp_threads; ms->possible_cpus->cpus[n].arch_id = virt_cpu_mp_affinity(vms, n); + ms->possible_cpus->cpus[n].props.has_core_id = true; + ms->possible_cpus->cpus[n].props.core_id = n; ms->possible_cpus->cpus[n].props.has_thread_id = true; ms->possible_cpus->cpus[n].props.thread_id = n; } diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 32bec156f2..33a58086a9 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -1086,6 +1086,9 @@ static Property arm_cpu_has_dsp_property = static Property arm_cpu_has_mpu_property = DEFINE_PROP_BOOL("has-mpu", ARMCPU, has_mpu, true); +static Property arm_cpu_coreid_property = + DEFINE_PROP_INT32("core-id", ARMCPU, core_id, -1); + /* This is like DEFINE_PROP_UINT32 but it doesn't set the default value, * because the CPU initfn will have already set cpu->pmsav7_dregion to * the right value for that particular CPU type, and we don't want @@ -1168,6 +1171,8 @@ void arm_cpu_post_init(Object *obj) qdev_property_add_static(DEVICE(obj), &arm_cpu_rvbar_property); } + qdev_property_add_static(DEVICE(obj), &arm_cpu_coreid_property); + #ifndef CONFIG_USER_ONLY if (arm_feature(&cpu->env, ARM_FEATURE_EL3)) { /* Add the has_el3 state CPU property only if EL3 is allowed. This will diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 677584e5da..5c4991156e 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -949,6 +949,7 @@ struct ARMCPU { QLIST_HEAD(, ARMELChangeHook) el_change_hooks; int32_t node_id; /* NUMA node this CPU belongs to */ + int32_t core_id; /* core-id of this ARM VCPU */ /* Used to synchronize KVM and QEMU in-kernel device levels */ uint8_t device_irq_level; From patchwork Sat Jun 13 21:36:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308760 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krl332Mkz9s1x for ; Sun, 14 Jun 2020 07:46:23 +1000 (AEST) Received: from localhost ([::1]:59296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkDyn-00073j-34 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:46:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59096) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxA-0005IW-Fe; Sat, 13 Jun 2020 17:44:40 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3774 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDx8-00036Y-LS; Sat, 13 Jun 2020 17:44:40 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 3F683E6CF56D9F5BF53F; Sun, 14 Jun 2020 05:44:35 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:29 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 03/22] arm/cpuhp: Add common cpu utility for possible vcpus Date: Sat, 13 Jun 2020 22:36:10 +0100 Message-ID: <20200613213629.21984-4-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Adds various utility functions which might be required to fetch or check the state of the possible vcpus. This also introduces concept of *disabled* vcpus, which are part of the *possible* vcpus but are not part of the *present* vcpu. This state shall be used during machine init time to check the presence of vcpus. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- cpus-common.c | 20 ++++++++++++++++++++ include/hw/core/cpu.h | 21 +++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/cpus-common.c b/cpus-common.c index 70a9d12981..7cf900289b 100644 --- a/cpus-common.c +++ b/cpus-common.c @@ -23,6 +23,7 @@ #include "hw/core/cpu.h" #include "sysemu/cpus.h" #include "qemu/lockable.h" +#include "hw/boards.h" static QemuMutex qemu_cpu_list_lock; static QemuCond exclusive_cond; @@ -82,6 +83,25 @@ void cpu_list_add(CPUState *cpu) assert(!cpu_index_auto_assigned); } QTAILQ_INSERT_TAIL_RCU(&cpus, cpu, node); + qemu_mutex_unlock(&qemu_cpu_list_lock); +} + +CPUState *qemu_get_possible_cpu(int index) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + const CPUArchIdList *possible_cpus = ms->possible_cpus; + CPUState *cpu; + + assert((index >= 0) && (index < possible_cpus->len)); + + cpu = CPU(possible_cpus->cpus[index].cpu); + + return cpu; +} + +bool qemu_present_cpu(CPUState *cpu) +{ + return (cpu && !cpu->disabled); } void cpu_list_remove(CPUState *cpu) diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 497600c49e..d9cae71ea5 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -419,6 +419,7 @@ struct CPUState { GArray *plugin_mem_cbs; + bool disabled; /* TODO Move common fields from CPUArchState here. */ int cpu_index; int cluster_index; @@ -802,6 +803,26 @@ static inline bool cpu_in_exclusive_context(const CPUState *cpu) */ CPUState *qemu_get_cpu(int index); +/** + * qemu_get_possible_cpu: + * @index: The CPUState@cpu_index value of the CPU to obtain. + * + * Gets a CPU matching @index. + * + * Returns: The possible CPU or %NULL if there is no matching CPU. + */ +CPUState *qemu_get_possible_cpu(int index); + +/** + * qemu_present_cpu: + * @cpu: The vCPU to check + * + * Checks if the vcpu is amongst the present possible vcpus. + * + * Returns: True if it is present possible vcpu else false + */ +bool qemu_present_cpu(CPUState *cpu); + /** * cpu_exists: * @id: Guest-exposed CPU ID to lookup. From patchwork Sat Jun 13 21:36:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308765 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krqv1G28z9s1x for ; Sun, 14 Jun 2020 07:50:33 +1000 (AEST) Received: from localhost ([::1]:45214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE2n-0004vE-IC for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:50:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59216) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxM-0005Vd-J1; Sat, 13 Jun 2020 17:44:53 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:60708 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxH-00037G-SN; Sat, 13 Jun 2020 17:44:51 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 6697AA3006FAB5AE2476; Sun, 14 Jun 2020 05:44:45 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:35 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 04/22] arm/cpuhp: Machine init time change common to vcpu {cold|hot}-plug Date: Sat, 13 Jun 2020 22:36:11 +0100 Message-ID: <20200613213629.21984-5-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:26 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This refactors (+) introduces the common logic required during the initialization of both cold and hot plugged vcpus. This also initializes the *disabled* state of the vcpus which shall be used further during init phases of various other components like GIC, PMU, ACPI etc as part of the virt machine initialization. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 250 +++++++++++++++++++++++++++++++++--------- include/hw/arm/virt.h | 2 + target/arm/cpu.c | 7 ++ target/arm/cpu64.c | 9 ++ 4 files changed, 214 insertions(+), 54 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index c4ed955776..184bed8716 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -203,6 +203,8 @@ static const char *valid_cpus[] = { ARM_CPU_TYPE_NAME("max"), }; +static CPUArchId *virt_find_cpu_slot(MachineState *ms, int vcpuid); + static bool cpu_type_valid(const char *cpu) { int i; @@ -1657,6 +1659,62 @@ static void finalize_gic_version(VirtMachineState *vms) } } +static void virt_cpu_set_properties(Object *cpuobj, const CPUArchId *cpu_slot) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + MemoryRegion *sysmem = get_system_memory(); + VirtMachineState *vms = VIRT_MACHINE(ms); + uint64_t mp_affinity = cpu_slot->arch_id; + CPUState *cs = CPU(cpuobj); + VirtMachineClass *vmc; + + vmc = VIRT_MACHINE_GET_CLASS(ms); + + /* now, set the cpu object property values */ + object_property_set_int(cpuobj, mp_affinity, "mp-affinity", NULL); + + numa_cpu_pre_plug(cpu_slot, DEVICE(cpuobj), &error_fatal); + + if (!vms->secure) { + object_property_set_bool(cpuobj, false, "has_el3", NULL); + } + + if (!vms->virt && object_property_find(cpuobj, "has_el2", NULL)) { + object_property_set_bool(cpuobj, false, "has_el2", NULL); + } + + if (vms->psci_conduit != QEMU_PSCI_CONDUIT_DISABLED) { + object_property_set_int(cpuobj, vms->psci_conduit, "psci-conduit", + NULL); + /* Secondary CPUs start in PSCI powered-down state */ + if (cs->cpu_index > 0) + object_property_set_bool(cpuobj, true, "start-powered-off", + NULL); + } + + if (vmc->kvm_no_adjvtime && + object_property_find(cpuobj, "kvm-no-adjvtime", NULL)) { + object_property_set_bool(cpuobj, true, "kvm-no-adjvtime", NULL); + } + + if (vmc->no_pmu && object_property_find(cpuobj, "pmu", NULL)) { + object_property_set_bool(cpuobj, false, "pmu", NULL); + } + + if (object_property_find(cpuobj, "reset-cbar", NULL)) { + object_property_set_int(cpuobj, vms->memmap[VIRT_CPUPERIPHS].base, + "reset-cbar", &error_abort); + } + + object_property_set_link(cpuobj, OBJECT(sysmem), "memory", + &error_abort); + + if (vms->secure) { + object_property_set_link(cpuobj, OBJECT(vms->secure_sysmem), + "secure-memory", &error_abort); + } +} + static void machvirt_init(MachineState *machine) { VirtMachineState *vms = VIRT_MACHINE(machine); @@ -1706,6 +1764,7 @@ static void machvirt_init(MachineState *machine) memory_region_init(secure_sysmem, OBJECT(machine), "secure-memory", UINT64_MAX); memory_region_add_subregion_overlap(secure_sysmem, 0, sysmem, -1); + vms->secure_sysmem = secure_sysmem; } firmware_loaded = virt_firmware_init(vms, sysmem, @@ -1749,6 +1808,14 @@ static void machvirt_init(MachineState *machine) } vms->smp_cpus = smp_cpus; + vms->max_cpus = max_cpus; + if (vms->gic_version < VIRT_GIC_VERSION_3) { + warn_report("For GICv%d max-cpus must be equal to smp-cpus", + vms->gic_version); + warn_report("Overriding specified max-cpus(%d) with smp-cpus(%d)", + max_cpus, smp_cpus); + vms->max_cpus = smp_cpus; + } if (vms->virt && kvm_enabled()) { error_report("mach-virt: KVM does not support providing " @@ -1761,65 +1828,12 @@ static void machvirt_init(MachineState *machine) possible_cpus = mc->possible_cpu_arch_ids(machine); for (n = 0; n < possible_cpus->len; n++) { Object *cpuobj; - CPUState *cs; - - if (n >= smp_cpus) { - break; - } cpuobj = object_new(possible_cpus->cpus[n].type); - object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id, - "mp-affinity", NULL); - - cs = CPU(cpuobj); - cs->cpu_index = n; - - numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index], DEVICE(cpuobj), - &error_fatal); aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, n, "core-id", NULL); - if (!vms->secure) { - object_property_set_bool(cpuobj, false, "has_el3", NULL); - } - - if (!vms->virt && object_property_find(cpuobj, "has_el2", NULL)) { - object_property_set_bool(cpuobj, false, "has_el2", NULL); - } - - if (vms->psci_conduit != QEMU_PSCI_CONDUIT_DISABLED) { - object_property_set_int(cpuobj, vms->psci_conduit, - "psci-conduit", NULL); - - /* Secondary CPUs start in PSCI powered-down state */ - if (n > 0) { - object_property_set_bool(cpuobj, true, - "start-powered-off", NULL); - } - } - - if (vmc->kvm_no_adjvtime && - object_property_find(cpuobj, "kvm-no-adjvtime", NULL)) { - object_property_set_bool(cpuobj, true, "kvm-no-adjvtime", NULL); - } - - if (vmc->no_pmu && object_property_find(cpuobj, "pmu", NULL)) { - object_property_set_bool(cpuobj, false, "pmu", NULL); - } - - if (object_property_find(cpuobj, "reset-cbar", NULL)) { - object_property_set_int(cpuobj, vms->memmap[VIRT_CPUPERIPHS].base, - "reset-cbar", &error_abort); - } - - object_property_set_link(cpuobj, OBJECT(sysmem), "memory", - &error_abort); - if (vms->secure) { - object_property_set_link(cpuobj, OBJECT(secure_sysmem), - "secure-memory", &error_abort); - } - object_property_set_bool(cpuobj, true, "realized", &error_fatal); object_unref(cpuobj); } @@ -2106,6 +2120,71 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) return ms->possible_cpus; } +static int virt_archid_cmp(const void *a, const void *b) +{ + CPUArchId *archid_a = (CPUArchId *)a; + CPUArchId *archid_b = (CPUArchId *)b; + + return archid_a->arch_id - archid_b->arch_id; +} + +static CPUArchId *virt_find_cpu_slot(MachineState *ms, int vcpuid) +{ + VirtMachineState *vms = VIRT_MACHINE(ms); + CPUArchId arch_id, *found_cpu; + uint64_t mp_affinity; + + mp_affinity = virt_cpu_mp_affinity(vms, vcpuid); + arch_id.arch_id = mp_affinity; + found_cpu = bsearch(&arch_id, ms->possible_cpus->cpus, + ms->possible_cpus->len, + sizeof(*ms->possible_cpus->cpus), virt_archid_cmp); + + assert (found_cpu); + + /* + * RFC: Question: + * For KVM/TCG, MPIDR for vcpu is derived using vcpu-id. + * In fact, as of now there is a linear relation between + * vcpu-id and mpidr(see below fig.) as derived in host + * kvm. Slot-id is the index where vcpu with certain + * arch-id(=mpidr/ap-affinity) is plugged. + * + * Therefore, for now we could use the vcpu-id as slot + * index for getting CPUArchId of the vcpu coresponding + * to this slot(this view is not perfectly consistent + * with the ARM specification view of MPIDR_EL1). + * QEMU/KVM view of cpu topology makes it bit difficult + * to use topo-info(pkg-id, core-id, thread-id) with + * device_add/-device interface which might not match + * with what actual underlying host cpu supports. + * therefore question is do we care about this? and + * is it okay to have view of thread-id inconsistent + * with the host cpu? How should QEMU create PPTT + * for the Guest? + * + * +----+----+----+----+----+----+----+----+ + * MASK | F F | F F | F F | 0 F | + * +----+----+----+----+----+----+----+----+ + * + * | | cluster | cluster | |core| + * |<---------Package-id-------->| |core| + * + * +----+----+----+----+----+----+----+----+ + * MPIDR ||| Res | Aff2 | Aff1 | Aff0 | + * +----+----+----+----+----+----+----+----+ + * \ \ \ | | + * \ 8bit \ 8bit \ |4bit| + * \<------->\<------->\ |<-->| + * \ \ \| | + * +----+----+----+----+----+----+----+----+ + * VCPU-ID | Byte4 | Byte2 | Byte1 | Byte0 | + * +----+----+----+----+----+----+----+----+ + */ + + return found_cpu; +} + static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2151,11 +2230,71 @@ out: error_propagate(errp, local_err); } +static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + MachineState *ms = MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* sanity check the cpu */ + if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { + error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", + ms->cpu_type); + return; + } + + if ((cpu->core_id < 0) || (cpu->core_id >= ms->possible_cpus->len)) { + error_setg(errp, "Invalid core-id %u specified, must be in range 1:%u", + cpu->core_id, ms->possible_cpus->len - 1); + return; + } + + /* + * RFC: Question: + * For now we are not taking into account of other topo info like + * thread-id, socket-id to generate arch-id/mp-affinity. + * The way KVM/Host generates mpidr value and the way ARM spec + * identifies uniquely cpu within the heirarchy is bit inconsistent. + * Perhaps needs more discussion on this? Hence, directly using + * core_id as cpu_index for now. Ideally, slot-index found out using + * the topo info should have been the cpu-index. + */ + cs->cpu_index = cpu->core_id; + + cpu_slot = virt_find_cpu_slot(ms, cpu->core_id); + if (qemu_present_cpu(CPU(cpu_slot->cpu))) { + error_setg(errp, "cpu %d with arch-id %" PRIu64 " exists", + cpu->core_id, cpu_slot->arch_id); + return; + } + virt_cpu_set_properties(OBJECT(cs), cpu_slot); +} + +static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + MachineState *ms = MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* insert the cold/hot-plugged vcpu in the slot */ + cpu_slot = virt_find_cpu_slot(ms, cpu->core_id); + cpu_slot->cpu = OBJECT(dev); + + cs->disabled = false; + return; +} + static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { virt_memory_pre_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_pre_plug(hotplug_dev, dev, errp); } } @@ -2172,6 +2311,8 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, } if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { virt_memory_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_plug(hotplug_dev, dev, errp); } if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { PCIDevice *pdev = PCI_DEVICE(dev); @@ -2193,7 +2334,8 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, DeviceState *dev) { if (object_dynamic_cast(OBJECT(dev), TYPE_SYS_BUS_DEVICE) || - (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM))) { + (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) || + (object_dynamic_cast(OBJECT(dev), TYPE_CPU))){ return HOTPLUG_HANDLER(machine); } if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 31878ddc72..5b8ba64ec2 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -127,6 +127,7 @@ typedef struct { DeviceState *platform_bus_dev; FWCfgState *fw_cfg; PFlashCFI01 *flash[2]; + MemoryRegion *secure_sysmem; bool secure; bool highmem; bool highmem_ecam; @@ -142,6 +143,7 @@ typedef struct { char *pciehb_nodename; const int *irqmap; int smp_cpus; + int max_cpus; void *fdt; int fdt_size; uint32_t clock_phandle; diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 33a58086a9..0c9f5f970e 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -2131,6 +2131,12 @@ static gchar *arm_gdb_arch_name(CPUState *cs) return g_strdup("arm"); } +static int64_t arm_cpu_get_arch_id(CPUState *cs) +{ + ARMCPU *cpu = ARM_CPU(cs); + return cpu->mp_affinity; +} + static void arm_cpu_class_init(ObjectClass *oc, void *data) { ARMCPUClass *acc = ARM_CPU_CLASS(oc); @@ -2147,6 +2153,7 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) cc->has_work = arm_cpu_has_work; cc->cpu_exec_interrupt = arm_cpu_exec_interrupt; cc->dump_state = arm_cpu_dump_state; + cc->get_arch_id = arm_cpu_get_arch_id; cc->set_pc = arm_cpu_set_pc; cc->synchronize_from_tb = arm_cpu_synchronize_from_tb; cc->gdb_read_register = arm_cpu_gdb_read_register; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index cbc5c3868f..41e69cd53f 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -766,11 +766,18 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp) static void aarch64_cpu_initfn(Object *obj) { + CPUState *cs = CPU(obj); + object_property_add_bool(obj, "aarch64", aarch64_cpu_get_aarch64, aarch64_cpu_set_aarch64); object_property_set_description(obj, "aarch64", "Set on/off to enable/disable aarch64 " "execution state "); + /* + * we start every ARM64 vcpu as disabled possible vcpu. It needs to be + * enabled explicitly + */ + cs->disabled = true; } static void aarch64_cpu_finalizefn(Object *obj) @@ -785,7 +792,9 @@ static gchar *aarch64_gdb_arch_name(CPUState *cs) static void aarch64_cpu_class_init(ObjectClass *oc, void *data) { CPUClass *cc = CPU_CLASS(oc); + DeviceClass *dc = DEVICE_CLASS(oc); + dc->user_creatable = true; cc->cpu_exec_interrupt = arm_cpu_exec_interrupt; cc->gdb_read_register = aarch64_cpu_gdb_read_register; cc->gdb_write_register = aarch64_cpu_gdb_write_register; From patchwork Sat Jun 13 21:36:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308762 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krmJ48zdz9sRW for ; Sun, 14 Jun 2020 07:47:28 +1000 (AEST) Received: from localhost ([::1]:34964 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkDzq-0000Gs-81 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:47:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59274) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxP-0005Wc-Pj; Sat, 13 Jun 2020 17:44:55 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:43308 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxM-00037Y-Ti; Sat, 13 Jun 2020 17:44:55 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 73BE68C24A74F8661BE4; Sun, 14 Jun 2020 05:44:50 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:41 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 05/22] arm/cpuhp: Pre-create disabled possible vcpus @machine init Date: Sat, 13 Jun 2020 22:36:12 +0100 Message-ID: <20200613213629.21984-6-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:20 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" In ARMv8 architecture, GIC needs all the vcpus to be created and present when it is initialized. This is because: 1. GICC and MPIDR association must be fixed at the VM initialization time. This is represented by register GIC_TYPER(mp_afffinity, proc_num) 2. GICC(cpu interfaces), GICR(redistributors) etc all must be initialized at the boot time as well. 3. Memory regions associated with GICR etc. cannot be changed(add/del/mod) after VM has inited. This patch adds the support to pre-create all such possible vcpus within the host using the KVM interface as part of the virt machine initialization. These vcpus could later be attached to QOM/ACPI while they are actually hot plugged and made present. NOTE: There is some refactoring related to the kvm_destroy_vcpu/kvm_get_vcpu (to make use of the common code) has been intentionaly left out in RFC version to avoid obscuring the framework change of the cpu hotplug. The existing code being presented in this patch could further be optimized later. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- accel/kvm/kvm-all.c | 31 +++++++++++++++++++++++++++++ hw/arm/virt.c | 46 ++++++++++++++++++++++++++++++++++++++++++-- include/sysemu/kvm.h | 2 ++ target/arm/kvm.c | 32 ++++++++++++++++++++++++++++++ target/arm/kvm_arm.h | 11 +++++++++++ 5 files changed, 120 insertions(+), 2 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index d06cc04079..8e1c7b3d13 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -329,6 +329,37 @@ err: return ret; } +void kvm_park_vcpu(CPUState *cs) +{ + unsigned long vcpu_id = cs->cpu_index; + struct KVMParkedVcpu *vcpu; + + vcpu = g_malloc0(sizeof(*vcpu)); + vcpu->vcpu_id = vcpu_id; + vcpu->kvm_fd = cs->kvm_fd; + QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node); +} + +int kvm_create_vcpu(CPUState *cpu) +{ + unsigned long vcpu_id = cpu->cpu_index; + KVMState *s = kvm_state; + int ret = 0; + + DPRINTF("kvm_create_vcpu\n"); + + ret = kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id); + if (ret < 0) { + goto err; + } + cpu->kvm_fd = ret; + cpu->kvm_state = s; + cpu->vcpu_dirty = true; + +err: + return ret; +} + int kvm_destroy_vcpu(CPUState *cpu) { KVMState *s = kvm_state; diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 184bed8716..8040473d30 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1828,14 +1828,56 @@ static void machvirt_init(MachineState *machine) possible_cpus = mc->possible_cpu_arch_ids(machine); for (n = 0; n < possible_cpus->len; n++) { Object *cpuobj; + CPUState *cs; cpuobj = object_new(possible_cpus->cpus[n].type); + cs = CPU(cpuobj); aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, n, "core-id", NULL); - object_property_set_bool(cpuobj, true, "realized", &error_fatal); - object_unref(cpuobj); + if (n < vms->smp_cpus) { + char *core_id = g_strdup_printf("core%d", n); + qdev_set_id(DEVICE(cpuobj),core_id); + object_property_set_bool(cpuobj, true, "realized", &error_fatal); + g_free(core_id); + object_unref(OBJECT(cs)); + } else { + CPUArchId *cpu_slot; + /* handling for vcpus which are yet to be hot-plugged */ + cs->cpu_index = n; + /* ARM host vcpu features need to be fixed at the boot time */ + virt_cpu_set_properties(cpuobj, &possible_cpus->cpus[n]); + /* + * For KVM, we shall be pre-creating the now disabled/un-plugged + * possbile host vcpus and park them till the time they are + * actually hot plugged. This is required to pre-size the host + * GICC and GICR with the all possible vcpus for this VM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + } + /* + * Add disabled vcpu to cpu slot during the init phase of the virt machine. + * 1. We need this ARMCPU object during the GIC init. This object + * will facilitate in pre-realizing the gic. Any info like + * mp-affinity(required to derive gicr_type) etc could still be + * fetched while preserving QOM abstraction akin to realized + * vcpus. + * 2. Now, after initialization of the virt machine is complete we could use + * two approaches to deal with this ARMCPU object: + * (i) re-use this ARMCPU object during hotplug of this vcpu. + * OR + * (ii) defer release this ARMCPU object after gic has been + * initialized or during pre-plug phase when a vcpu is + * hotplugged. + * + * We will use the (ii) approach and release the ARMCPU objects after GIC + * and machine has been initialized in machine_init_done() phase + */ + cpu_slot = virt_find_cpu_slot(machine, cs->cpu_index); + cpu_slot->cpu = OBJECT(cs); + } } fdt_add_timer_nodes(vms); fdt_add_cpu_nodes(vms); diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index 3b2250471c..ca06bfeb17 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -218,7 +218,9 @@ int kvm_has_intx_set_mask(void); int kvm_init_vcpu(CPUState *cpu); int kvm_cpu_exec(CPUState *cpu); +int kvm_create_vcpu(CPUState *cpu); int kvm_destroy_vcpu(CPUState *cpu); +void kvm_park_vcpu(CPUState *cs); /** * kvm_arm_supports_user_irq diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 4bdbe6dcac..9fd447d111 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -597,6 +597,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + unsigned long vcpu_id = cs->cpu_index; + int ret; + + ret = kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vcpu in the host. This will reset the sys regs + * for this vcpu and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret = kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vcpu. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vcpus + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index 48bf5e16d5..a9e316cfee 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -155,6 +155,17 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called at to pre create all possible kvm vcpus within the the host at the + * virt machine init time. This will also init this pre-created vcpu and + * hence result in vcpu reset at host. These pre created and inited vcpus + * shall be parked for use when ARM vcpus are actually realized. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + /** * kvm_arm_init_serror_injection: * @cs: CPUState From patchwork Sat Jun 13 21:36:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308764 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krng0GFjz9s1x for ; Sun, 14 Jun 2020 07:48:38 +1000 (AEST) Received: from localhost ([::1]:39476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE0y-0002H6-Ll for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:48:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59356) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxV-0005jW-I3; Sat, 13 Jun 2020 17:45:01 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:43436 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxS-000380-Eh; Sat, 13 Jun 2020 17:45:01 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 883C7821393FE162372F; Sun, 14 Jun 2020 05:44:55 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:47 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 06/22] arm/cpuhp: Changes to pre-size GIC with possible vcpus @machine init Date: Sat, 13 Jun 2020 22:36:13 +0100 Message-ID: <20200613213629.21984-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:20 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" GIC needs to be pre-sized with possible vcpus at the initialization time. This is necessary because Memory regions and resources associated with GICC/GICR etc cannot be changed (add/del/modified) after VM has inited. Also, GIC_TYPER needs to be initialized with mp_affinity and cpu interface number association. This cannot be changed after GIC has initialized. Once all the cpu interfaces of the GIC has been inited it needs to be ensured that any updations to the GICC during reset only takes place for the present vcpus and not the disabled ones. Therefore, proper checks are required at various places. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 18 +++++++++--------- hw/intc/arm_gicv3_common.c | 8 ++++++-- hw/intc/arm_gicv3_cpuif.c | 6 ++++++ hw/intc/arm_gicv3_kvm.c | 29 ++++++++++++++++++++++++++--- include/hw/arm/virt.h | 2 +- 5 files changed, 48 insertions(+), 15 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 8040473d30..9e55b20685 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -628,19 +628,19 @@ static void create_v2m(VirtMachineState *vms) static void create_gic(VirtMachineState *vms) { - MachineState *ms = MACHINE(vms); /* We create a standalone GIC */ SysBusDevice *gicbusdev; const char *gictype; int type = vms->gic_version, i; - unsigned int smp_cpus = ms->smp.cpus; + unsigned int max_cpus = vms->max_cpus; + unsigned int smp_cpus = vms->smp_cpus; uint32_t nb_redist_regions = 0; gictype = (type == 3) ? gicv3_class_name() : gic_class_name(); vms->gic = qdev_create(NULL, gictype); qdev_prop_set_uint32(vms->gic, "revision", type); - qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus); + qdev_prop_set_uint32(vms->gic, "num-cpu", max_cpus); /* Note that the num-irq property counts both internal and external * interrupts; there are always 32 of the former (mandated by GIC spec). */ @@ -652,7 +652,7 @@ static void create_gic(VirtMachineState *vms) if (type == 3) { uint32_t redist0_capacity = vms->memmap[VIRT_GIC_REDIST].size / GICV3_REDIST_SIZE; - uint32_t redist0_count = MIN(smp_cpus, redist0_capacity); + uint32_t redist0_count = MIN(max_cpus, redist0_capacity); nb_redist_regions = virt_gicv3_redist_region_count(vms); @@ -665,7 +665,7 @@ static void create_gic(VirtMachineState *vms) vms->memmap[VIRT_HIGH_GIC_REDIST2].size / GICV3_REDIST_SIZE; qdev_prop_set_uint32(vms->gic, "redist-region-count[1]", - MIN(smp_cpus - redist0_count, redist1_capacity)); + MIN(max_cpus - redist0_count, redist1_capacity)); } } else { if (!kvm_irqchip_in_kernel()) { @@ -722,7 +722,7 @@ static void create_gic(VirtMachineState *vms) } else if (vms->virt) { qemu_irq irq = qdev_get_gpio_in(vms->gic, ppibase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, irq); + sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); } qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, @@ -730,11 +730,11 @@ static void create_gic(VirtMachineState *vms) + VIRTUAL_PMU_IRQ)); sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + smp_cpus, + sysbus_connect_irq(gicbusdev, i + max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); } diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index 58ef65f589..bfa514444a 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -348,11 +348,15 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu = g_new0(GICv3CPUState, s->num_cpu); for (i = 0; i < s->num_cpu; i++) { - CPUState *cpu = qemu_get_cpu(i); + CPUState *cpu = qemu_get_possible_cpu(i); uint64_t cpu_affid; int last; - s->cpu[i].cpu = cpu; + if (qemu_present_cpu(cpu)) + s->cpu[i].cpu = cpu; + else + s->cpu[i].cpu = NULL; + s->cpu[i].gic = s; /* Store GICv3CPUState in CPUARMState gicv3state pointer */ gicv3_set_gicv3state(cpu, &s->cpu[i]); diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index 08e000e33c..90d8b0118e 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -779,6 +779,9 @@ void gicv3_cpuif_update(GICv3CPUState *cs) ARMCPU *cpu = ARM_CPU(cs->cpu); CPUARMState *env = &cpu->env; + if (!qemu_present_cpu(cs->cpu)) + return; + g_assert(qemu_mutex_iothread_locked()); trace_gicv3_cpuif_update(gicv3_redist_affid(cs), cs->hppi.irq, @@ -1654,6 +1657,9 @@ static void icc_generate_sgi(CPUARMState *env, GICv3CPUState *cs, for (i = 0; i < s->num_cpu; i++) { GICv3CPUState *ocs = &s->cpu[i]; + if (!qemu_present_cpu(ocs->cpu)) + continue; + if (irm) { /* IRM == 1 : route to all CPUs except self */ if (cs == ocs) { diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index ca43bf87ca..7fe000e53c 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -25,6 +25,7 @@ #include "hw/sysbus.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "sysemu/cpus.h" #include "sysemu/kvm.h" #include "sysemu/runstate.h" #include "kvm_arm.h" @@ -458,6 +459,17 @@ static void kvm_arm_gicv3_put(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* + * To support hotplug of vcpus we need to make sure all gic cpuif/GICC + * are initialized at machvirt init time. Once the init is done we + * release the ARMCPU object for disabled vcpus but this leg could hit + * during reset of GICC later as well i.e. after init has happened and + * all of the cases we want to make sure we dont acess the GICC for + * the disabled VCPUs. + */ + if (!qemu_present_cpu(c->cpu)) + continue; + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, true); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], true); @@ -677,10 +689,21 @@ static void arm_gicv3_icc_reset(CPUARMState *env, const ARMCPRegInfo *ri) return; } + /* + * This shall be called even when vcpu is being hotplugged and other vcpus + * might be running. Host kernel KVM code to handle device access of IOCTLs + * KVM_{GET|SET}_DEVICE_ATTR might fail due to inability to grab vcpu locks + * for all the vcpus. Hence, we need to pause all vcpus to facilitate + * locking within host. + */ + if (!qemu_present_cpu(c->cpu)) + pause_all_vcpus(); /* Initialize to actual HW supported configuration */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(ICC_CTLR_EL1, c->gicr_typer), &c->icc_ctlr_el1[GICV3_NS], false, &error_abort); + if (!qemu_present_cpu(c->cpu)) + resume_all_vcpus(); c->icc_ctlr_el1[GICV3_S] = c->icc_ctlr_el1[GICV3_NS]; } @@ -788,9 +811,9 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) } for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + CPUState *cs = qemu_get_cpu(i); + if (qemu_present_cpu(cs)) + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); } /* Try to create the device via the device control API */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 5b8ba64ec2..38a9cad168 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -178,7 +178,7 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) assert(vms->gic_version == VIRT_GIC_VERSION_3); - return vms->smp_cpus > redist0_capacity ? 2 : 1; + return vms->max_cpus > redist0_capacity ? 2 : 1; } #endif /* QEMU_ARM_VIRT_H */ From patchwork Sat Jun 13 21:36:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308766 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krrQ6D3Zz9s1x for ; Sun, 14 Jun 2020 07:51:02 +1000 (AEST) Received: from localhost ([::1]:47740 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE3I-0005wl-J2 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:51:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59416) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxZ-0005wK-Bt; Sat, 13 Jun 2020 17:45:05 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3775 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxX-00038p-75; Sat, 13 Jun 2020 17:45:05 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 96BE6538ECDCFFAEEE5B; Sun, 14 Jun 2020 05:45:00 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:53 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 07/22] arm/cpuhp: Init PMU at host for all possible vcpus Date: Sat, 13 Jun 2020 22:36:14 +0100 Message-ID: <20200613213629.21984-8-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" PMU for all possible vcpus must be initialized at the virt machine initialization time. This patch refactors existing code to accomodate possible vcpus. This also assumes that all processor being used are identical at least for now but does not affect the normal scanarios where they might not be in future. This assumption only affects the future hotplug scenarios if ever there exists any hetergenous processors. In such a case PMU might not be enabled on some vcpus. Is it acceptable and doable tradeoff for now? This perhaps needs more discussion. please check below link, Link: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00131.html Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++------------- include/hw/arm/virt.h | 1 + 2 files changed, 37 insertions(+), 15 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 9e55b20685..7f938f289b 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -525,23 +525,9 @@ static void fdt_add_gic_node(VirtMachineState *vms) static void fdt_add_pmu_nodes(const VirtMachineState *vms) { - CPUState *cpu; ARMCPU *armcpu; uint32_t irqflags = GIC_FDT_IRQ_FLAGS_LEVEL_HI; - CPU_FOREACH(cpu) { - armcpu = ARM_CPU(cpu); - if (!arm_feature(&armcpu->env, ARM_FEATURE_PMU)) { - return; - } - if (kvm_enabled()) { - if (kvm_irqchip_in_kernel()) { - kvm_arm_pmu_set_irq(cpu, PPI(VIRTUAL_PMU_IRQ)); - } - kvm_arm_pmu_init(cpu); - } - } - if (vms->gic_version == VIRT_GIC_VERSION_2) { irqflags = deposit32(irqflags, GIC_FDT_IRQ_PPI_CPU_START, GIC_FDT_IRQ_PPI_CPU_WIDTH, @@ -1414,6 +1400,38 @@ static void create_secure_ram(VirtMachineState *vms, g_free(nodename); } +static bool virt_pmu_init(VirtMachineState *vms) +{ + CPUArchIdList *possible_cpus = vms->parent.possible_cpus; + ARMCPU *armcpu; + int n; + + /* + * As of now KVM ensures that within the host all the vcpus have same + * features configured. This cannot be changed later and cannot be diferent + * for new vcpus being plugged in. Also, -cpu option/virt machine cpu-type + * ensures all the vcpus are identical. + */ + for (n = 0; n < possible_cpus->len; n++) { + CPUState *cpu = qemu_get_possible_cpu(n); + armcpu = ARM_CPU(cpu); + + if (!arm_feature(&armcpu->env, ARM_FEATURE_PMU)) { + warn_report("Not all vcpus might have PMU initialized"); + return false; + } + + if (kvm_enabled()) { + if (kvm_irqchip_in_kernel()) { + kvm_arm_pmu_set_irq(cpu, PPI(VIRTUAL_PMU_IRQ)); + } + kvm_arm_pmu_init(cpu); + } + } + + return true; +} + static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size) { const VirtMachineState *board = container_of(binfo, VirtMachineState, @@ -1909,7 +1927,10 @@ static void machvirt_init(MachineState *machine) create_gic(vms); - fdt_add_pmu_nodes(vms); + if (!vmc->no_pmu && virt_pmu_init(vms)) { + vms->pmu = true; + fdt_add_pmu_nodes(vms); + } create_uart(vms, VIRT_UART, sysmem, serial_hd(0)); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 38a9cad168..3ffbda6217 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -134,6 +134,7 @@ typedef struct { bool its; bool virt; bool ras; + bool pmu; OnOffAuto acpi; VirtGICType gic_version; VirtIOMMUType iommu; From patchwork Sat Jun 13 21:36:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308769 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krtB24Ydz9s1x for ; Sun, 14 Jun 2020 07:52:34 +1000 (AEST) Received: from localhost ([::1]:55024 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE4l-0000eR-V0 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:52:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59500) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxf-000689-94; Sat, 13 Jun 2020 17:45:11 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:32854 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxd-0003Ba-88; Sat, 13 Jun 2020 17:45:10 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id A998042E24438AB2505D; Sun, 14 Jun 2020 05:45:05 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:59 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 08/22] arm/cpuhp: Enable ACPI support for vcpu hotplug Date: Sat, 13 Jun 2020 22:36:15 +0100 Message-ID: <20200613213629.21984-9-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:26 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" ACPI is required to interface QEMU with the guest. Roughly falls into below cases, 1. Convey the possible vcpus config at the machine init time to the guest using various DSDT tables like MADT etc. 2. Convey vcpu hotplug events to guest(using GED) 3. Assist in evaluation of various ACPI methods(like _EVT, _STA, _OST, _EJ0, _MAT etc.) 4. Provides ACPI cpu hotplug state and 12 Byte memory mapped cpu hotplug control register interface to the OSPM/guest corresponding to each possible vcpu. The register interface consists of various R/W fields and their handling operations. These are called when ever register fields or memory regions are accessed(i.e. read or written) by OSPM when ever it evaluates various ACPI methods. Note: lot of this framework code is inherited from the changes already done for x86 but still some minor changes are required to make it compatible with ARM64.) This patch enables the ACPI support for virtual cpu hotplug in kconfig and during initialization. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 6 +++++- hw/arm/Kconfig | 1 + hw/arm/virt.c | 2 ++ include/hw/acpi/cpu_hotplug.h | 2 ++ include/hw/arm/virt.h | 1 + 5 files changed, 11 insertions(+), 1 deletion(-) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 3d6a500fb7..21fe0463b9 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -218,7 +218,11 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner, state->dev_count = id_list->len; state->devs = g_new0(typeof(*state->devs), state->dev_count); for (i = 0; i < id_list->len; i++) { - state->devs[i].cpu = CPU(id_list->cpus[i].cpu); + struct CPUState *cpu = CPU(id_list->cpus[i].cpu); + if (qemu_present_cpu(cpu)) + state->devs[i].cpu = cpu; + else + state->devs[i].cpu = NULL; state->devs[i].arch_id = id_list->cpus[i].arch_id; } memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state, diff --git a/hw/arm/Kconfig b/hw/arm/Kconfig index 9afa6eee79..cb67fb806b 100644 --- a/hw/arm/Kconfig +++ b/hw/arm/Kconfig @@ -26,6 +26,7 @@ config ARM_VIRT select ACPI_MEMORY_HOTPLUG select ACPI_HW_REDUCED select ACPI_NVDIMM + select ACPI_CPU_HOTPLUG config CHEETAH bool diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 7f938f289b..fe37babe35 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -75,6 +75,7 @@ #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" #include "hw/acpi/generic_event_device.h" +#include "hw/acpi/cpu_hotplug.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" #include "qemu/guest-random.h" @@ -151,6 +152,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCDIMM_ACPI] = { 0x09070000, MEMORY_HOTPLUG_IO_LEN }, [VIRT_ACPI_GED] = { 0x09080000, ACPI_GED_EVT_SEL_LEN }, [VIRT_NVDIMM_ACPI] = { 0x09090000, NVDIMM_ACPI_IO_LEN}, + [VIRT_CPUHP_ACPI] = { 0x090a0000, ACPI_CPU_HOTPLUG_REG_LEN}, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h index 3b932abbbb..48b291e45e 100644 --- a/include/hw/acpi/cpu_hotplug.h +++ b/include/hw/acpi/cpu_hotplug.h @@ -19,6 +19,8 @@ #include "hw/hotplug.h" #include "hw/acpi/cpu.h" +#define ACPI_CPU_HOTPLUG_REG_LEN 12 + typedef struct AcpiCpuHotplug { Object *device; MemoryRegion io; diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 3ffbda6217..e0bd9df69d 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -80,6 +80,7 @@ enum { VIRT_PCDIMM_ACPI, VIRT_ACPI_GED, VIRT_NVDIMM_ACPI, + VIRT_CPUHP_ACPI, VIRT_LOWMEMMAP_LAST, }; From patchwork Sat Jun 13 21:36:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308759 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krkr02Dyz9s1x for ; Sun, 14 Jun 2020 07:46:11 +1000 (AEST) Received: from localhost ([::1]:58116 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkDyb-0006Zh-Kv for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:46:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59680) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxp-0006Vg-87; Sat, 13 Jun 2020 17:45:21 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3707 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxn-0003CN-93; Sat, 13 Jun 2020 17:45:20 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id D13F932AE4BA90251945; Sun, 14 Jun 2020 05:45:15 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:05 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 09/22] arm/cpuhp: Init GED framework with cpu hotplug events Date: Sat, 13 Jun 2020 22:36:16 +0100 Message-ID: <20200613213629.21984-10-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI events when OSPM/guest receives an interrupt listed in the _CRS object of GED. OSPM then maps or demultiplexes the event by evaluating _EVT method. This change adds the support of cpu hotplug event initialization in the existing GED framework. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/generic_event_device.c | 8 ++++++++ hw/arm/virt.c | 3 ++- include/hw/acpi/generic_event_device.h | 5 +++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c index 1cb34111e5..0f2c8a959e 100644 --- a/hw/acpi/generic_event_device.c +++ b/hw/acpi/generic_event_device.c @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = { ACPI_GED_MEM_HOTPLUG_EVT, ACPI_GED_PWR_DOWN_EVT, ACPI_GED_NVDIMM_HOTPLUG_EVT, + ACPI_GED_CPU_HOTPLUG_EVT, }; /* @@ -305,6 +306,13 @@ static void acpi_ged_initfn(Object *obj) sysbus_init_mmio(sbd, &s->container_memhp); acpi_memory_hotplug_init(&s->container_memhp, OBJECT(dev), &s->memhp_state, 0); + + s->cpuhp.device = OBJECT(s); + memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp container", + ACPI_CPU_HOTPLUG_REG_LEN); + sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp); + cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev), + &s->cpuhp_state, 0); } static void acpi_ged_class_init(ObjectClass *class, void *data) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index fe37babe35..e9ead0e2dd 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -552,7 +552,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) DeviceState *dev; MachineState *ms = MACHINE(vms); int irq = vms->irqmap[VIRT_ACPI_GED]; - uint32_t event = ACPI_GED_PWR_DOWN_EVT; + uint32_t event = ACPI_GED_PWR_DOWN_EVT | ACPI_GED_CPU_HOTPLUG_EVT; if (ms->ram_slots) { event |= ACPI_GED_MEM_HOTPLUG_EVT; @@ -567,6 +567,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, vms->memmap[VIRT_ACPI_GED].base); sysbus_mmio_map(SYS_BUS_DEVICE(dev), 1, vms->memmap[VIRT_PCDIMM_ACPI].base); + sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, vms->memmap[VIRT_CPUHP_ACPI].base); sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, qdev_get_gpio_in(vms->gic, irq)); qdev_init_nofail(dev); diff --git a/include/hw/acpi/generic_event_device.h b/include/hw/acpi/generic_event_device.h index 90a9180db5..b04037cf62 100644 --- a/include/hw/acpi/generic_event_device.h +++ b/include/hw/acpi/generic_event_device.h @@ -62,6 +62,7 @@ #include "hw/sysbus.h" #include "hw/acpi/memory_hotplug.h" #include "hw/acpi/ghes.h" +#include "hw/acpi/cpu_hotplug.h" #define ACPI_POWER_BUTTON_DEVICE "PWRB" @@ -84,6 +85,7 @@ #define ACPI_GED_MEM_HOTPLUG_EVT 0x1 #define ACPI_GED_PWR_DOWN_EVT 0x2 #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4 +#define ACPI_GED_CPU_HOTPLUG_EVT 0x8 typedef struct GEDState { MemoryRegion evt; @@ -94,6 +96,9 @@ typedef struct AcpiGedState { SysBusDevice parent_obj; MemHotplugState memhp_state; MemoryRegion container_memhp; + CPUHotplugState cpuhp_state; + MemoryRegion container_cpuhp; + AcpiCpuHotplug cpuhp; GEDState ged_state; uint32_t ged_event_bitmap; qemu_irq irq; From patchwork Sat Jun 13 21:36:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308768 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krrV4dwGz9s1x for ; Sun, 14 Jun 2020 07:51:06 +1000 (AEST) Received: from localhost ([::1]:48166 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE3M-000671-8V for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:51:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxt-0006fk-Mt; Sat, 13 Jun 2020 17:45:25 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3708 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxr-0003Cw-Gr; Sat, 13 Jun 2020 17:45:25 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id DCAD1BD2440337CDA0F7; Sun, 14 Jun 2020 05:45:20 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:11 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 10/22] arm/cpuhp: Update CPUs AML with cpu-(ctrl)dev change Date: Sat, 13 Jun 2020 22:36:17 +0100 Message-ID: <20200613213629.21984-11-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" CPUs Control device(\\_SB.PCI0) register interface for the x86 arch is based on PCI and is IO port based and hence existing cpus AML code assumes _CRS objects would evaluate to a system resource which describes IO Port address. But on ARM arch CPUs control device(\\_SB.PRES) register interface is memory-mapped hence _CRS object should evaluate to system resource which describes memory-mapped base address. This cpus AML code change updates the existing inerface of the build cpus AML function to accept both IO/MEMORY type regions and update the _CRS object correspondingly. NOTE: Beside above CPU scan shall be triggered when OSPM evaluates _EVT method part of the GED framework which is covered in subsequent patch. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 23 ++++++++++++++++------- hw/arm/virt-acpi-build.c | 13 ++++++++++++- hw/i386/acpi-build.c | 2 +- include/hw/acpi/cpu.h | 5 +++-- 4 files changed, 32 insertions(+), 11 deletions(-) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 21fe0463b9..867fdd6993 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -335,9 +335,10 @@ const VMStateDescription vmstate_cpu_hotplug = { #define CPU_EJECT_EVENT "CEJ0" void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, - hwaddr io_base, + hwaddr mmap_io_base, const char *res_root, - const char *event_handler_method) + const char *event_handler_method, + AmlRegionSpace rs) { Aml *ifctx; Aml *field; @@ -365,13 +366,19 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(cpu_ctrl_dev, aml_mutex(CPU_LOCK, 0)); crs = aml_resource_template(); - aml_append(crs, aml_io(AML_DECODE16, io_base, io_base, 1, + if (rs == AML_SYSTEM_IO) { + aml_append(crs, aml_io(AML_DECODE16, mmap_io_base, mmap_io_base, 1, ACPI_CPU_HOTPLUG_REG_LEN)); + } else { + aml_append(crs, aml_memory32_fixed(mmap_io_base, + ACPI_CPU_HOTPLUG_REG_LEN, AML_READ_WRITE)); + } + aml_append(cpu_ctrl_dev, aml_name_decl("_CRS", crs)); /* declare CPU hotplug MMIO region with related access fields */ aml_append(cpu_ctrl_dev, - aml_operation_region("PRST", AML_SYSTEM_IO, aml_int(io_base), + aml_operation_region("PRST", rs, aml_int(mmap_io_base), ACPI_CPU_HOTPLUG_REG_LEN)); field = aml_field("PRST", AML_BYTE_ACC, AML_NOLOCK, @@ -593,9 +600,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(sb_scope, cpus_dev); aml_append(table, sb_scope); - method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED); - aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD)); - aml_append(table, method); + if (event_handler_method) { + method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED); + aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD)); + aml_append(table, method); + } g_free(cphp_res_path); } diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index ca31f70f7f..d40540db61 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -731,7 +731,18 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) * the RTC ACPI device at all when using UEFI. */ scope = aml_scope("\\_SB"); - acpi_dsdt_add_cpus(scope, vms->smp_cpus); + /* if GED is enabled then cpus AML shall be added as part build_cpus_aml */ + if (vms->acpi_dev) { + CPUHotplugFeatures opts = { + .acpi_1_compatible = false, + .has_legacy_cphp = false + }; + + build_cpus_aml(scope, ms, opts, memmap[VIRT_CPUHP_ACPI].base, + "\\_SB", NULL, AML_SYSTEM_MEMORY); + } else { + acpi_dsdt_add_cpus(scope, vms->smp_cpus); + } acpi_dsdt_add_uart(scope, &memmap[VIRT_UART], (irqmap[VIRT_UART] + ARM_SPI_BASE)); acpi_dsdt_add_flash(scope, &memmap[VIRT_FLASH]); diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index 473cbdfffd..4b224d80a5 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -1668,7 +1668,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, .acpi_1_compatible = true, .has_legacy_cphp = true }; build_cpus_aml(dsdt, machine, opts, pm->cpu_hp_io_base, - "\\_SB.PCI0", "\\_GPE._E02"); + "\\_SB.PCI0", "\\_GPE._E02", AML_SYSTEM_IO); } if (pcms->memhp_io_base && nr_mem) { diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index 62f0278ba2..c3a9981dc3 100644 --- a/include/hw/acpi/cpu.h +++ b/include/hw/acpi/cpu.h @@ -53,9 +53,10 @@ typedef struct CPUHotplugFeatures { } CPUHotplugFeatures; void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, - hwaddr io_base, + hwaddr mmap_io_base, const char *res_root, - const char *event_handler_method); + const char *event_handler_method, + AmlRegionSpace rs); void acpi_cpu_ospm_status(CPUHotplugState *cpu_st, ACPIOSTInfoList ***list); From patchwork Sat Jun 13 21:36:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308763 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krnS5jg7z9s1x for ; Sun, 14 Jun 2020 07:48:28 +1000 (AEST) Received: from localhost ([::1]:38470 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE0o-0001ke-Hi for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:48:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59844) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxz-0006tJ-C3; Sat, 13 Jun 2020 17:45:31 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3776 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxx-0003GQ-Dn; Sat, 13 Jun 2020 17:45:31 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id EF046561478D198E2282; Sun, 14 Jun 2020 05:45:25 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:17 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 11/22] arm/cpuhp: Update GED _EVT method AML with cpu scan Date: Sat, 13 Jun 2020 22:36:18 +0100 Message-ID: <20200613213629.21984-12-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" OSPM evaluates _EVT method to map the event. The cpu hotplug event eventually results in start of the cpu scan. Scan figures out the cpu and the kind of event(plug/unplug) and notifies it back to the guest. The change in this patch updates the GED AML _EVT method with the call to \\_SB.CPUS.CSCN which will do above. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/generic_event_device.c | 4 ++++ include/hw/acpi/cpu_hotplug.h | 3 +++ 2 files changed, 7 insertions(+) diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c index 0f2c8a959e..79177deda2 100644 --- a/hw/acpi/generic_event_device.c +++ b/hw/acpi/generic_event_device.c @@ -108,6 +108,10 @@ void build_ged_aml(Aml *table, const char *name, HotplugHandler *hotplug_dev, aml_append(if_ctx, aml_call0(MEMORY_DEVICES_CONTAINER "." MEMORY_SLOT_SCAN_METHOD)); break; + case ACPI_GED_CPU_HOTPLUG_EVT: + aml_append(if_ctx, aml_call0(ACPI_CPU_CONTAINER "." + ACPI_CPU_SCAN_METHOD)); + break; case ACPI_GED_PWR_DOWN_EVT: aml_append(if_ctx, aml_notify(aml_name(ACPI_POWER_BUTTON_DEVICE), diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h index 48b291e45e..f47fc8e79b 100644 --- a/include/hw/acpi/cpu_hotplug.h +++ b/include/hw/acpi/cpu_hotplug.h @@ -20,6 +20,9 @@ #include "hw/acpi/cpu.h" #define ACPI_CPU_HOTPLUG_REG_LEN 12 +#define ACPI_CPU_SCAN_METHOD "CSCN" +#define ACPI_CPU_CONTAINER "\\_SB.CPUS" + typedef struct AcpiCpuHotplug { Object *device; From patchwork Sat Jun 13 21:36:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308767 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krrT3YCbz9s1x for ; Sun, 14 Jun 2020 07:51:05 +1000 (AEST) Received: from localhost ([::1]:47904 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE3K-00060g-5K for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:51:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59976) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDy4-000757-NM; Sat, 13 Jun 2020 17:45:36 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:44024 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDy1-0003KL-LS; Sat, 13 Jun 2020 17:45:36 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 0C2ED2A45448682DC913; Sun, 14 Jun 2020 05:45:31 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:23 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 12/22] arm/cpuhp: MADT Tbl change to size the guest with possible vcpus Date: Sat, 13 Jun 2020 22:36:19 +0100 Message-ID: <20200613213629.21984-13-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:20 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Changes required during building of MADT Table by QEMU to accomodate disabled possible vcpus. This info shall be used by the guest kernel to size up its resources during boot time. This pre-sizing of the guest kernel done on possible vcpus will facilitate hotplug of the disabled vcpus. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index d40540db61..c654e2c9a3 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -603,6 +603,8 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) AcpiMultipleApicTable *madt; AcpiMadtGenericDistributor *gicd; AcpiMadtGenericMsiFrame *gic_msi; + MachineState *ms = &vms->parent; + CPUArchIdList *possible_cpus = ms->possible_cpus; int i; madt = acpi_data_push(table_data, sizeof *madt); @@ -613,11 +615,10 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicd->base_address = cpu_to_le64(memmap[VIRT_GIC_DIST].base); gicd->version = vms->gic_version; - for (i = 0; i < vms->smp_cpus; i++) { + for (i = 0; i < vms->max_cpus; i++) { AcpiMadtGenericCpuInterface *gicc = acpi_data_push(table_data, sizeof(*gicc)); - ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(i)); - + ARMCPU *cpu = ARM_CPU(qemu_get_possible_cpu(i)); gicc->type = ACPI_APIC_GENERIC_CPU_INTERFACE; gicc->length = sizeof(*gicc); if (vms->gic_version == 2) { @@ -626,11 +627,14 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicc->gicv_base_address = cpu_to_le64(memmap[VIRT_GIC_VCPU].base); } gicc->cpu_interface_number = cpu_to_le32(i); - gicc->arm_mpidr = cpu_to_le64(armcpu->mp_affinity); + gicc->arm_mpidr = possible_cpus->cpus[i].arch_id; gicc->uid = cpu_to_le32(i); - gicc->flags = cpu_to_le32(ACPI_MADT_GICC_ENABLED); - - if (arm_feature(&armcpu->env, ARM_FEATURE_PMU)) { + if ( i < vms->smp_cpus ) { + gicc->flags = cpu_to_le32(ACPI_MADT_GICC_ENABLED); + } else { + gicc->flags = cpu_to_le32(0); + } + if ((cpu && arm_feature(&cpu->env, ARM_FEATURE_PMU)) || vms->pmu) { gicc->performance_interrupt = cpu_to_le32(PPI(VIRTUAL_PMU_IRQ)); } if (vms->virt) { From patchwork Sat Jun 13 21:36:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308773 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krwJ2ZnCz9sRW for ; Sun, 14 Jun 2020 07:54:24 +1000 (AEST) Received: from localhost ([::1]:36266 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE6Y-0004ee-3f for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:54:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60088) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyC-0007A9-4b; Sat, 13 Jun 2020 17:45:45 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:33284 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDy7-0003L7-PR; Sat, 13 Jun 2020 17:45:43 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 2238FFF1004DFACDB16F; Sun, 14 Jun 2020 05:45:36 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:29 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 13/22] arm/cpuhp: Add ACPI _MAT entry for Processor object Date: Sat, 13 Jun 2020 22:36:20 +0100 Message-ID: <20200613213629.21984-14-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:26 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Adds a function which builds the ACPI _MAT entry for processor objects. This shall be called from the cpus AML for all possible vcpus. The entry is passed to the guest kernel with ACPI_MADT_GICC_ENABLED flag when it evaluates _MAT object. OSPM evaluates _MAT object in context to the cpu hotplug event. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 5 +++++ hw/arm/virt-acpi-build.c | 20 ++++++++++++++++++++ include/hw/arm/virt.h | 1 + 3 files changed, 26 insertions(+) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 867fdd6993..a79dc65120 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -565,6 +565,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, apic->flags = cpu_to_le32(1); break; } + case ACPI_APIC_GENERIC_CPU_INTERFACE: { + AcpiMadtGenericCpuInterface *gicc = (void *)madt_buf->data; + gicc->flags = cpu_to_le32(1); + break; + } default: assert(0); } diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index c654e2c9a3..354fd775f9 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -593,6 +593,22 @@ build_gtdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } /* MADT */ +static void +build_mat_entry(AcpiDeviceIf *adev, int uid, const CPUArchIdList *arch_ids, + GArray *entry) +{ + AcpiMadtGenericCpuInterface *gicc = acpi_data_push(entry,sizeof(*gicc)); + MachineState *ms = MACHINE(qdev_get_machine()); + CPUArchIdList *possible_cpus = ms->possible_cpus; + + /* fill the relevant fields of _MAT entry for GICC */ + gicc->type = ACPI_APIC_GENERIC_CPU_INTERFACE; + gicc->length = sizeof(*gicc); + gicc->cpu_interface_number = cpu_to_le32(uid); + gicc->arm_mpidr = possible_cpus->cpus[uid].arch_id; + gicc->uid = cpu_to_le32(uid); +} + static void build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { @@ -741,6 +757,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) .acpi_1_compatible = false, .has_legacy_cphp = false }; + AcpiDeviceIfClass *adevc; + /* _MAT entry shall be used within cpus aml */ + adevc = ACPI_DEVICE_IF_CLASS(DEVICE_GET_CLASS(vms->acpi_dev)); + adevc->madt_cpu = build_mat_entry; build_cpus_aml(scope, ms, opts, memmap[VIRT_CPUHP_ACPI].base, "\\_SB", NULL, AML_SYSTEM_MEMORY); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index e0bd9df69d..e8468d8cf6 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -37,6 +37,7 @@ #include "hw/block/flash.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/acpi/acpi_dev_interface.h" #define NUM_GICV2M_SPIS 64 #define NUM_VIRTIO_TRANSPORTS 32 From patchwork Sat Jun 13 21:36:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308775 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49kryT3B2lz9s1x for ; Sun, 14 Jun 2020 07:56:15 +1000 (AEST) Received: from localhost ([::1]:42606 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE8K-0007QL-Bx for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:56:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyF-0007EN-71; Sat, 13 Jun 2020 17:45:47 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3777 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyD-0003Lc-39; Sat, 13 Jun 2020 17:45:46 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 3739E75CD75FBCF2EC6D; Sun, 14 Jun 2020 05:45:41 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:34 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 14/22] arm/cpuhp: Release objects for *disabled* possible vcpus after init Date: Sat, 13 Jun 2020 22:36:21 +0100 Message-ID: <20200613213629.21984-15-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During machvirt_init(), ARMCPU objects are pre-created along with the corresponding KVM vcpus in the host. Disabled possible KVM vcpus are then parked at the per-virt-machine list "kvm_parked_vcpus". Prime purpose to pre-create ARMCPU objects for the disabled vcpus is to facilitate the GIC initialization (pre-sized with possible vcpus). GIC requires all vcpus corresponding to its GICC(GIC CPU Interface) to be initialized and present during its own initialization. After initialization of the machine is complete we release the ARMCPU objects for the disabled vcpus(which shall be re-created at the time when vcpu is hot plugged again. This newly created ARMCPU object is then attached with corresponding parked KVM VCPU). We have few options after the machine init where the disabled ARMCPU object could be released: 1. Release in context to the virt_machine_done() notifier.(This is also our current approach) 2. Defer the release till a new vcpu object is hot plugged. Then release the object in context to the pre_plug() phase. 3. Never release and keep on reusing them and release once at VM exit. This will require some modifications within the interface of qdevice_add() to get old ARMCPU object instead of creating a new one for the hotplug request. Each of the above approaches come with their own pros and cons. This prototype uses the 1st approach.(suggestions are welcome!) Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index e9ead0e2dd..0faf54aa8f 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1403,6 +1403,28 @@ static void create_secure_ram(VirtMachineState *vms, g_free(nodename); } +static void virt_remove_disabled_cpus(VirtMachineState *vms) +{ + MachineState *ms = MACHINE(vms); + int n; + + /* + * RFC: Question: Other approach could have been to keep them forever + * and release it only once when qemu exits as part o finalize or when + * new vcpu is hotplugged. In the later old could be released for the + * newly created object for the same vcpu? + */ + for (n = vms->smp_cpus; n < vms->max_cpus; n++) { + CPUState *cs = qemu_get_possible_cpu(n); + if (!qemu_present_cpu(cs)) { + CPUArchId *cpu_slot; + cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); + cpu_slot->cpu = NULL; + object_unref(OBJECT(cs)); + } + } +} + static bool virt_pmu_init(VirtMachineState *vms) { CPUArchIdList *possible_cpus = vms->parent.possible_cpus; @@ -1500,6 +1522,9 @@ void virt_machine_done(Notifier *notifier, void *data) virt_acpi_setup(vms); virt_build_smbios(vms); + + /* release the disabled ARMCPU objects used during init for pre-sizing */ + virt_remove_disabled_cpus(vms); } static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) From patchwork Sat Jun 13 21:36:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308772 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krv0296Mz9s1x for ; Sun, 14 Jun 2020 07:53:16 +1000 (AEST) Received: from localhost ([::1]:59124 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE5R-0002WC-VQ for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:53:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60290) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyN-0007IP-R0; Sat, 13 Jun 2020 17:45:55 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3778 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyL-0003Ro-UB; Sat, 13 Jun 2020 17:45:55 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 5DD21511CC70ECC0FB03; Sun, 14 Jun 2020 05:45:51 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:40 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 15/22] arm/cpuhp: Update ACPI GED framework to support vcpu hotplug Date: Sat, 13 Jun 2020 22:36:22 +0100 Message-ID: <20200613213629.21984-16-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" ACPI GED shall be used to convey to the guest kernel about any cpu hot-(un)plug events. Therefore, existing ACPI GED framework inside QEMU needs to be enhanced to support CPU hotplug state and events. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/acpi/generic_event_device.c | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c index 79177deda2..df81e9292a 100644 --- a/hw/acpi/generic_event_device.c +++ b/hw/acpi/generic_event_device.c @@ -13,7 +13,9 @@ #include "qapi/error.h" #include "exec/address-spaces.h" #include "hw/acpi/acpi.h" +#include "hw/acpi/cpu.h" #include "hw/acpi/generic_event_device.h" +#include "hw/arm/virt.h" #include "hw/irq.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" @@ -192,12 +194,47 @@ static void acpi_ged_device_plug_cb(HotplugHandler *hotplug_dev, } else { acpi_memory_plug_cb(hotplug_dev, &s->memhp_state, dev, errp); } + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + acpi_cpu_plug_cb(hotplug_dev, &s->cpuhp_state, dev, errp); } else { error_setg(errp, "virt: device plug request for unsupported device" " type: %s", object_get_typename(OBJECT(dev))); } } +static void acpi_ged_device_unplug_request_cb(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + AcpiGedState *s = ACPI_GED(hotplug_dev); + + if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + acpi_cpu_unplug_request_cb(hotplug_dev, &s->cpuhp_state, dev, errp); + } else { + error_setg(errp, "virt: device unplug request for the unsupported device" + " type: %s", object_get_typename(OBJECT(dev))); + } +} + +static void acpi_ged_device_unplug_cb(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + AcpiGedState *s = ACPI_GED(hotplug_dev); + + if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + acpi_cpu_unplug_cb(&s->cpuhp_state, dev, errp); + } else { + error_setg(errp, "virt: device plug request for unsupported device" + " type: %s", object_get_typename(OBJECT(dev))); + } +} + +static void acpi_ged_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list) +{ + AcpiGedState *s = ACPI_GED(adev); + + acpi_cpu_ospm_status(&s->cpuhp_state, list); +} + static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev) { AcpiGedState *s = ACPI_GED(adev); @@ -210,6 +247,8 @@ static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev) sel = ACPI_GED_PWR_DOWN_EVT; } else if (ev & ACPI_NVDIMM_HOTPLUG_STATUS) { sel = ACPI_GED_NVDIMM_HOTPLUG_EVT; + } else if (ev & ACPI_CPU_HOTPLUG_STATUS) { + sel = ACPI_GED_CPU_HOTPLUG_EVT; } else { /* Unknown event. Return without generating interrupt. */ warn_report("GED: Unsupported event %d. No irq injected", ev); @@ -330,8 +369,11 @@ static void acpi_ged_class_init(ObjectClass *class, void *data) dc->vmsd = &vmstate_acpi_ged; hc->plug = acpi_ged_device_plug_cb; + hc->unplug_request = acpi_ged_device_unplug_request_cb; + hc->unplug = acpi_ged_device_unplug_cb; adevc->send_event = acpi_ged_send_event; + adevc->ospm_status = acpi_ged_ospm_status; } static const TypeInfo acpi_ged_info = { From patchwork Sat Jun 13 21:36:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308774 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krx721HBz9s1x for ; Sun, 14 Jun 2020 07:55:07 +1000 (AEST) Received: from localhost ([::1]:39388 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE7F-0005vF-28 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:55:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60394) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyT-0007Nv-SI; Sat, 13 Jun 2020 17:46:02 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3779 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyR-0003S2-KM; Sat, 13 Jun 2020 17:46:01 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 6BC59460F2B0A7FB873E; Sun, 14 Jun 2020 05:45:56 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:46 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 16/22] arm/cpuhp: Add/update basic hot-(un)plug framework Date: Sat, 13 Jun 2020 22:36:23 +0100 Message-ID: <20200613213629.21984-17-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Adds the new cpu hot-unplug hooks and updates the existing hotplug hooks with sanity checks. Note, Functional contents of the hooks(now left with TODO comment) shall be gradually filled in the subsequent patches in an incremental approach to patch and logic building which would be roughly as follows: 1. (Un-)wiring of interrupts between vcpu<->gic 2. Sending events to Guest for hot-(un)plug so that guest can take appropriate actions. 3. Notifying GIC about hot-(un)plug action so that vcpu could be (un-)stitched to the GIC CPU interface. 4. Updating the Guest with Next boot info for this vcpu in the firmware. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 102 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 100 insertions(+), 2 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 0faf54aa8f..ac2941159a 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2324,11 +2324,23 @@ out: static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); CPUArchId *cpu_slot; + if (dev->hotplugged && !vms->acpi_dev) { + error_setg(errp, "GED acpi device does not exists"); + return; + } + + if (dev->hotplugged && (vms->gic_version < VIRT_GIC_VERSION_3)) { + error_setg(errp, "CPU hotplug not supported with GICv%d, use GICv3 or " + "later", vms->gic_version); + return; + } + /* sanity check the cpu */ if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", @@ -2361,6 +2373,10 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, return; } virt_cpu_set_properties(OBJECT(cs), cpu_slot); + + if (dev->hotplugged) { + /* TODO: update GIC about this hotplug change here */ + } } static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, @@ -2375,10 +2391,75 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot = virt_find_cpu_slot(ms, cpu->core_id); cpu_slot->cpu = OBJECT(dev); + if (dev->hotplugged) { + /* TODO: wire the gic-cpu irqs */ + /* TODO: update acpi hotplug state and send cpu hotplug event to guest */ + /* TODO: register this cpu for reset & update F/W info for the next boot */ + } + cs->disabled = false; return; } +static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + CPUState *cs = CPU(dev); + + if (!vms->acpi_dev || !dev->realized) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + if (vms->gic_version < VIRT_GIC_VERSION_3) { + error_setg(errp, "CPU hot-unplug not supported with GICv%d", + vms->gic_version); + return; + } + + if (cs->cpu_index == first_cpu->cpu_index) + { + error_setg(errp, "hot-unplug of ARM boot vcpu %d not supported", + first_cpu->cpu_index); + return; + } + + /* TODO: request cpu hotplug from guest */ + + return; +} + +static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + MachineState *ms = MACHINE(hotplug_dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + if (!vms->acpi_dev || !dev->realized) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + cpu_slot = virt_find_cpu_slot(ms, ARM_CPU(cs)->core_id); + + /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + + /* TODO: unwire the gic-cpu irqs here */ + /* TODO: update the GIC about this hot unplug change */ + + /* TODO: unregister this cpu for reset & update F/W info for the next boot */ + + qemu_opts_del(dev->opts); + dev->opts = NULL; + + cpu_slot->cpu = NULL; + cs->disabled = true; + return; +} + static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2417,8 +2498,23 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, static void virt_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { - error_setg(errp, "device unplug request for unsupported device" - " type: %s", object_get_typename(OBJECT(dev))); + if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug_request(hotplug_dev, dev, errp); + } else { + error_setg(errp, "device unplug request for unsupported type: %s", + object_get_typename(OBJECT(dev))); + } +} + +static void virt_machine_device_unplug_cb(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug(hotplug_dev, dev, errp); + } else { + error_setg(errp, "device unplug for unsupported type: %s", + object_get_typename(OBJECT(dev))); + } } static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, @@ -2535,11 +2631,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15"); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; mc->kvm_type = virt_kvm_type; + mc->has_hotpluggable_cpus = true; assert(!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->pre_plug = virt_machine_device_pre_plug_cb; hc->plug = virt_machine_device_plug_cb; hc->unplug_request = virt_machine_device_unplug_request_cb; + hc->unplug = virt_machine_device_unplug_cb; mc->numa_mem_supported = true; mc->nvdimm_supported = true; mc->auto_enable_numa_with_memhp = true; From patchwork Sat Jun 13 21:36:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308776 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krz90hLnz9s1x for ; Sun, 14 Jun 2020 07:56:53 +1000 (AEST) Received: from localhost ([::1]:44356 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE8w-000084-S0 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:56:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyY-0007XF-IW; Sat, 13 Jun 2020 17:46:06 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3709 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyW-0003SP-1z; Sat, 13 Jun 2020 17:46:06 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7C39C247F741223B5826; Sun, 14 Jun 2020 05:46:01 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:52 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 17/22] arm/cpuhp: Changes to (un)wire GICC<->VCPU IRQs during hot-(un)plug Date: Sat, 13 Jun 2020 22:36:24 +0100 Message-ID: <20200613213629.21984-18-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Refactors the existing gic create code to extract common code to wire the vcpu<->gic interrupts. This function could be used with cold-plug case and also used when vcpu is hot-plugged. It also introduces a new function to unwire the vcpu>->gic interrupts for the vcpu hot-unplug cases. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 144 +++++++++++++++++++++++++++++------------ hw/core/qdev.c | 2 +- include/hw/qdev-core.h | 2 + 3 files changed, 104 insertions(+), 44 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ac2941159a..f0295e940e 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -615,6 +615,103 @@ static void create_v2m(VirtMachineState *vms) fdt_add_v2m_gic_node(vms); } +static void unwire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + unsigned int max_cpus = vms->max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + int irq; + + /* Mapping from the output timer irq lines from the CPU to the + * GIC PPI inputs we use for the virt board. + */ + const int timer_irq[] = { + [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, + [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, + [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, + [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, + }; + + for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { + qdev_disconnect_gpio_out_named(cpudev, NULL, irq); + } + + if (type == 3) { + qdev_disconnect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", 0); + } else if (vms->virt) { + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, cpu + 4 * max_cpus); + } + + /* + * RFC: Question: This currently does not takes care of intimating the devices + * which might be sitting on system bus. Do we need a sysbus_disconnect_irq() + * which also does the job of notification beside disconnection? + */ + qdev_disconnect_gpio_out_named(cpudev, "pmu-interrupt", 0); + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, cpu); + qdev_disconnect_gpio_out_named(gicdev, + SYSBUS_DEVICE_GPIO_IRQ, cpu + max_cpus); + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + 2 * max_cpus); + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + 3 * max_cpus); +} + +static void wire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + unsigned int max_cpus = vms->max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + SysBusDevice *gicbusdev; + int ppibase; + int irq; + + ppibase = NUM_IRQS + cpu * GIC_INTERNAL + GIC_NR_SGIS; + + /* Mapping from the output timer irq lines from the CPU to the + * GIC PPI inputs we use for the virt board. + */ + const int timer_irq[] = { + [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, + [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, + [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, + [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, + }; + + for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { + qdev_connect_gpio_out(cpudev, irq, + qdev_get_gpio_in(gicdev, + ppibase + timer_irq[irq])); + } + + gicbusdev = SYS_BUS_DEVICE(gicdev); + if (type == 3) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + ppibase + ARCH_GIC_MAINT_IRQ); + qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", + 0, irq); + } else if (vms->virt) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + ppibase + ARCH_GIC_MAINT_IRQ); + sysbus_connect_irq(gicbusdev, cpu + 4 * max_cpus, irq); + } + + qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, + qdev_get_gpio_in(gicdev, + ppibase + VIRTUAL_PMU_IRQ)); + sysbus_connect_irq(gicbusdev, cpu, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); + sysbus_connect_irq(gicbusdev, cpu + max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); + sysbus_connect_irq(gicbusdev, cpu + 2 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); + sysbus_connect_irq(gicbusdev, cpu + 3 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); +} + static void create_gic(VirtMachineState *vms) { /* We create a standalone GIC */ @@ -684,47 +781,7 @@ static void create_gic(VirtMachineState *vms) * and the GIC's IRQ/FIQ/VIRQ/VFIQ interrupt outputs to the CPU's inputs. */ for (i = 0; i < smp_cpus; i++) { - DeviceState *cpudev = DEVICE(qemu_get_cpu(i)); - int ppibase = NUM_IRQS + i * GIC_INTERNAL + GIC_NR_SGIS; - int irq; - /* Mapping from the output timer irq lines from the CPU to the - * GIC PPI inputs we use for the virt board. - */ - const int timer_irq[] = { - [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, - [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, - [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, - [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, - }; - - for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { - qdev_connect_gpio_out(cpudev, irq, - qdev_get_gpio_in(vms->gic, - ppibase + timer_irq[irq])); - } - - if (type == 3) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - ppibase + ARCH_GIC_MAINT_IRQ); - qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", - 0, irq); - } else if (vms->virt) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - ppibase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); - } - - qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, - qdev_get_gpio_in(vms->gic, ppibase - + VIRTUAL_PMU_IRQ)); - - sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); + wire_gic_cpu_irqs(vms, qemu_get_cpu(i)); } fdt_add_gic_node(vms); @@ -2382,6 +2439,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); @@ -2392,7 +2450,7 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot->cpu = OBJECT(dev); if (dev->hotplugged) { - /* TODO: wire the gic-cpu irqs */ + wire_gic_cpu_irqs(vms, cs); /* TODO: update acpi hotplug state and send cpu hotplug event to guest */ /* TODO: register this cpu for reset & update F/W info for the next boot */ } @@ -2447,7 +2505,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ - /* TODO: unwire the gic-cpu irqs here */ + unwire_gic_cpu_irqs(vms, cs); /* TODO: update the GIC about this hot unplug change */ /* TODO: unregister this cpu for reset & update F/W info for the next boot */ diff --git a/hw/core/qdev.c b/hw/core/qdev.c index 9e5538aeae..65b3ec7c8f 100644 --- a/hw/core/qdev.c +++ b/hw/core/qdev.c @@ -563,7 +563,7 @@ qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n) /* disconnect a GPIO output, returning the disconnected input (if any) */ -static qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, const char *name, int n) { char *propname = g_strdup_printf("%s[%d]", diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index b870b27966..8434cc5a3e 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -355,6 +355,8 @@ void qdev_connect_gpio_out_named(DeviceState *dev, const char *name, int n, qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n); qemu_irq qdev_intercept_gpio_out(DeviceState *dev, qemu_irq icpt, const char *name, int n); +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, + const char *name, int n); BusState *qdev_get_child_bus(DeviceState *dev, const char *name); From patchwork Sat Jun 13 21:36:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308777 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49ks0h6lPFz9s1x for ; Sun, 14 Jun 2020 07:58:12 +1000 (AEST) Received: from localhost ([::1]:48312 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkEAE-0001pc-Nt for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:58:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60552) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyd-0007iD-Dg; Sat, 13 Jun 2020 17:46:11 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3710 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyb-0003UB-2p; Sat, 13 Jun 2020 17:46:11 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 8C85E43C4B70648A5679; Sun, 14 Jun 2020 05:46:06 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:45:58 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 18/22] arm/cpuhp: Changes to update GIC with vcpu hot-plug notification Date: Sat, 13 Jun 2020 22:36:25 +0100 Message-ID: <20200613213629.21984-19-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Adds the notification support about vcpu hot-(un)plug required to update the GIC so that it can update its vcpu to GIC cpu interface association. NOTE: This is using 'struct VirtMachineState' inside the notifier function. Question: Not sure if it is right to use machine related data structure inside GIC related files? Its design looks to be pretty much abstracted from any machine related stuff. @Peter Maydell Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 12 +++++-- hw/intc/arm_gicv3_common.c | 54 ++++++++++++++++++++++++++++++ hw/intc/arm_gicv3_cpuif.c | 5 +++ hw/intc/gicv3_internal.h | 1 + include/hw/arm/virt.h | 1 + include/hw/intc/arm_gicv3_common.h | 1 + 6 files changed, 72 insertions(+), 2 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index f0295e940e..b4cfd53a59 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1928,6 +1928,8 @@ static void machvirt_init(MachineState *machine) create_fdt(vms); + notifier_list_init(&vms->cpuhp_notifiers); + possible_cpus = mc->possible_cpu_arch_ids(machine); for (n = 0; n < possible_cpus->len; n++) { Object *cpuobj; @@ -2378,6 +2380,12 @@ out: error_propagate(errp, local_err); } +static void virt_update_gic(VirtMachineState *vms, CPUState *cs) +{ + /* notify gic to stitch GICC to this new cpu */ + notifier_list_notify(&vms->cpuhp_notifiers, cs); +} + static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2432,7 +2440,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, virt_cpu_set_properties(OBJECT(cs), cpu_slot); if (dev->hotplugged) { - /* TODO: update GIC about this hotplug change here */ + virt_update_gic(vms, cs); } } @@ -2506,7 +2514,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ unwire_gic_cpu_irqs(vms, cs); - /* TODO: update the GIC about this hot unplug change */ + virt_update_gic(vms, cs); /* TODO: unregister this cpu for reset & update F/W info for the next boot */ diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index bfa514444a..f6b7b359cb 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -31,6 +31,7 @@ #include "gicv3_internal.h" #include "hw/arm/linux-boot-if.h" #include "sysemu/kvm.h" +#include "hw/arm/virt.h" static void gicv3_gicd_no_migration_shift_bug_post_load(GICv3State *cs) @@ -305,8 +306,57 @@ void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, } } +static int arm_gicv3_get_proc_num(GICv3State *s, CPUState *cpu) +{ + uint64_t mp_affinity; + uint64_t gicr_typer; + uint64_t cpu_affid; + int i; + + mp_affinity = object_property_get_uint(OBJECT(cpu), "mp-affinity", NULL); + /* match the cpu mp-affinity to get the gic cpuif number */ + for (i = 0; i < s->num_cpu; i++) { + gicr_typer = s->cpu[i].gicr_typer; + cpu_affid = (gicr_typer >> 32) & 0xFFFFFF; + if (cpu_affid == mp_affinity) { + return i; + } + } + + return -1; +} + +static void arm_gicv3_cpu_update_notifier(Notifier * notifier, void * data) +{ + VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine()); + GICv3State *s = ARM_GICV3_COMMON(vms->gic); + CPUState *cpu = (CPUState *)data; + int gic_cpuif_num; + + /* this shall get us mapped gicv3 cpuif corresponding to mpidr */ + gic_cpuif_num = arm_gicv3_get_proc_num(s, cpu); + if (gic_cpuif_num < 0) { + error_report("Failed to associate cpu %d with any GIC cpuif", + cpu->cpu_index); + abort(); + } + + /* check if update is for vcpu hot-unplug */ + if (qemu_present_cpu(cpu)) { + s->cpu[gic_cpuif_num].cpu = NULL; + return; + } + + /* re-stitch the gic cpuif to this new cpu */ + gicv3_set_gicv3state(cpu, &s->cpu[gic_cpuif_num]); + gicv3_set_cpustate(&s->cpu[gic_cpuif_num], cpu); + + /* TODO: initialize the registers info for this newly added cpu */ +} + static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine()); GICv3State *s = ARM_GICV3_COMMON(dev); int i; @@ -386,12 +436,16 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) (i << 8) | (last << 4); } + + s->cpu_update_notifier.notify = arm_gicv3_cpu_update_notifier; + notifier_list_add(&vms->cpuhp_notifiers, &s->cpu_update_notifier); } static void arm_gicv3_finalize(Object *obj) { GICv3State *s = ARM_GICV3_COMMON(obj); + notifier_remove(&s->cpu_update_notifier); g_free(s->redist_region_count); } diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index 90d8b0118e..b3aa5979ca 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -20,6 +20,11 @@ #include "hw/irq.h" #include "cpu.h" +void gicv3_set_cpustate(GICv3CPUState *s, CPUState *cpu) +{ + s->cpu = cpu; +} + void gicv3_set_gicv3state(CPUState *cpu, GICv3CPUState *s) { ARMCPU *arm_cpu = ARM_CPU(cpu); diff --git a/hw/intc/gicv3_internal.h b/hw/intc/gicv3_internal.h index 05303a55c8..6e14a7a6cd 100644 --- a/hw/intc/gicv3_internal.h +++ b/hw/intc/gicv3_internal.h @@ -409,5 +409,6 @@ static inline void gicv3_cache_all_target_cpustates(GICv3State *s) } void gicv3_set_gicv3state(CPUState *cpu, GICv3CPUState *s); +void gicv3_set_cpustate(GICv3CPUState *s, CPUState *cpu); #endif /* QEMU_ARM_GICV3_INTERNAL_H */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index e8468d8cf6..c287433219 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -158,6 +158,7 @@ typedef struct { DeviceState *gic; DeviceState *acpi_dev; Notifier powerdown_notifier; + NotifierList cpuhp_notifiers; } VirtMachineState; #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM) diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index 31ec9a1ae4..b51f74c728 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -248,6 +248,7 @@ struct GICv3State { GICv3CPUState *gicd_irouter_target[GICV3_MAXIRQ]; uint32_t gicd_nsacr[DIV_ROUND_UP(GICV3_MAXIRQ, 16)]; + Notifier cpu_update_notifier; GICv3CPUState *cpu; }; From patchwork Sat Jun 13 21:36:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308770 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krtn0YCNz9sRW for ; Sun, 14 Jun 2020 07:53:05 +1000 (AEST) Received: from localhost ([::1]:57894 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE5G-000228-Oq for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:53:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60624) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyi-0007v9-Os; Sat, 13 Jun 2020 17:46:16 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:44728 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyg-0003UX-5l; Sat, 13 Jun 2020 17:46:16 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id A0730870F7F55669E4BE; Sun, 14 Jun 2020 05:46:11 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:46:04 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 19/22] arm/cpuhp: Changes required to (re)init the vcpu register info Date: Sat, 13 Jun 2020 22:36:26 +0100 Message-ID: <20200613213629.21984-20-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:20 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" VCPU register info needs to be re-initialized each time vcpu is hot-plugged. This has to be done both for emulation/TCG and KVM case. This is done in context to the GIC update notification for any vcpu hot-(un)plug events. This change adds that support and re-factors existing to maximize the code re-use. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/intc/arm_gicv3.c | 1 + hw/intc/arm_gicv3_common.c | 6 +- hw/intc/arm_gicv3_cpuif.c | 134 ++++++++++++++++------------- hw/intc/arm_gicv3_kvm.c | 7 +- hw/intc/gicv3_internal.h | 1 + include/hw/intc/arm_gicv3_common.h | 1 + 6 files changed, 86 insertions(+), 64 deletions(-) diff --git a/hw/intc/arm_gicv3.c b/hw/intc/arm_gicv3.c index 66eaa97198..86f2f4cca7 100644 --- a/hw/intc/arm_gicv3.c +++ b/hw/intc/arm_gicv3.c @@ -396,6 +396,7 @@ static void arm_gicv3_class_init(ObjectClass *klass, void *data) ARMGICv3Class *agc = ARM_GICV3_CLASS(klass); agcc->post_load = arm_gicv3_post_load; + agcc->init_cpu_reginfo = gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, arm_gic_realize, &agc->parent_realize); } diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index f6b7b359cb..01e4aba4f1 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -330,6 +330,7 @@ static void arm_gicv3_cpu_update_notifier(Notifier * notifier, void * data) { VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine()); GICv3State *s = ARM_GICV3_COMMON(vms->gic); + ARMGICv3CommonClass *c = ARM_GICV3_COMMON_GET_CLASS(s); CPUState *cpu = (CPUState *)data; int gic_cpuif_num; @@ -351,7 +352,10 @@ static void arm_gicv3_cpu_update_notifier(Notifier * notifier, void * data) gicv3_set_gicv3state(cpu, &s->cpu[gic_cpuif_num]); gicv3_set_cpustate(&s->cpu[gic_cpuif_num], cpu); - /* TODO: initialize the registers info for this newly added cpu */ + /* initialize the registers info for this newly added cpu */ + if (c->init_cpu_reginfo) { + c->init_cpu_reginfo(cpu); + } } static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index b3aa5979ca..1126fffa55 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -2605,6 +2605,74 @@ static const ARMCPRegInfo gicv3_cpuif_ich_apxr23_reginfo[] = { REGINFO_SENTINEL }; +void gicv3_init_cpu_reginfo(CPUState *cs) +{ + ARMCPU *cpu = ARM_CPU(cs); + GICv3CPUState *gcs = icc_cs_from_env(&cpu->env); + + /* Note that we can't just use the GICv3CPUState as an opaque pointer + * in define_arm_cp_regs_with_opaque(), because when we're called back + * it might be with code translated by CPU 0 but run by CPU 1, in + * which case we'd get the wrong value. + * So instead we define the regs with no ri->opaque info, and + * get back to the GICv3CPUState from the CPUARMState. + */ + define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + if (arm_feature(&cpu->env, ARM_FEATURE_EL2) + && cpu->gic_num_lrs) { + int j; + + gcs->maintenance_irq = cpu->gicv3_maintenance_interrupt; + + gcs->num_list_regs = cpu->gic_num_lrs; + gcs->vpribits = cpu->gic_vpribits; + gcs->vprebits = cpu->gic_vprebits; + + /* Check against architectural constraints: getting these + * wrong would be a bug in the CPU code defining these, + * and the implementation relies on them holding. + */ + g_assert(gcs->vprebits <= gcs->vpribits); + g_assert(gcs->vprebits >= 5 && gcs->vprebits <= 7); + g_assert(gcs->vpribits >= 5 && gcs->vpribits <= 8); + + define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); + + for (j = 0; j < gcs->num_list_regs; j++) { + /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs + * are split into two cp15 regs, LR (the low part, with the + * same encoding as the AArch64 LR) and LRC (the high part). + */ + ARMCPRegInfo lr_regset[] = { + { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, + .opc0 = 3, .opc1 = 4, .crn = 12, + .crm = 12 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, + .cp = 15, .opc1 = 4, .crn = 12, + .crm = 14 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + REGINFO_SENTINEL + }; + define_arm_cp_regs(cpu, lr_regset); + } + if (gcs->vprebits >= 6) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); + } + if (gcs->vprebits == 7) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); + } + } +} + static void gicv3_cpuif_el_change_hook(ARMCPU *cpu, void *opaque) { GICv3CPUState *cs = opaque; @@ -2621,69 +2689,11 @@ void gicv3_init_cpuif(GICv3State *s) for (i = 0; i < s->num_cpu; i++) { ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - GICv3CPUState *cs = &s->cpu[i]; - - /* Note that we can't just use the GICv3CPUState as an opaque pointer - * in define_arm_cp_regs_with_opaque(), because when we're called back - * it might be with code translated by CPU 0 but run by CPU 1, in - * which case we'd get the wrong value. - * So instead we define the regs with no ri->opaque info, and - * get back to the GICv3CPUState from the CPUARMState. - */ - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); - if (arm_feature(&cpu->env, ARM_FEATURE_EL2) - && cpu->gic_num_lrs) { - int j; - cs->maintenance_irq = cpu->gicv3_maintenance_interrupt; - - cs->num_list_regs = cpu->gic_num_lrs; - cs->vpribits = cpu->gic_vpribits; - cs->vprebits = cpu->gic_vprebits; - - /* Check against architectural constraints: getting these - * wrong would be a bug in the CPU code defining these, - * and the implementation relies on them holding. - */ - g_assert(cs->vprebits <= cs->vpribits); - g_assert(cs->vprebits >= 5 && cs->vprebits <= 7); - g_assert(cs->vpribits >= 5 && cs->vpribits <= 8); - - define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); - - for (j = 0; j < cs->num_list_regs; j++) { - /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs - * are split into two cp15 regs, LR (the low part, with the - * same encoding as the AArch64 LR) and LRC (the high part). - */ - ARMCPRegInfo lr_regset[] = { - { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, - .opc0 = 3, .opc1 = 4, .crn = 12, - .crm = 12 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, - .cp = 15, .opc1 = 4, .crn = 12, - .crm = 14 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - REGINFO_SENTINEL - }; - define_arm_cp_regs(cpu, lr_regset); - } - if (cs->vprebits >= 6) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); - } - if (cs->vprebits == 7) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); - } + if (qemu_present_cpu(CPU(cpu))) { + GICv3CPUState *cs = icc_cs_from_env(&cpu->env); + gicv3_init_cpu_reginfo(CPU(cpu)); + arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs); } - arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs); } } diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 7fe000e53c..8d19af3974 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -781,6 +781,10 @@ static void vm_change_state_handler(void *opaque, int running, } } +static void kvm_gicv3_init_cpu_reginfo(CPUState *cs) +{ + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); +} static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) { @@ -813,7 +817,7 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) for (i = 0; i < s->num_cpu; i++) { CPUState *cs = qemu_get_cpu(i); if (qemu_present_cpu(cs)) - define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); + kvm_gicv3_init_cpu_reginfo(cs); } /* Try to create the device via the device control API */ @@ -902,6 +906,7 @@ static void kvm_arm_gicv3_class_init(ObjectClass *klass, void *data) agcc->pre_save = kvm_arm_gicv3_get; agcc->post_load = kvm_arm_gicv3_put; + agcc->init_cpu_reginfo = kvm_gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, kvm_arm_gicv3_realize, &kgc->parent_realize); device_class_set_parent_reset(dc, kvm_arm_gicv3_reset, &kgc->parent_reset); diff --git a/hw/intc/gicv3_internal.h b/hw/intc/gicv3_internal.h index 6e14a7a6cd..66eec3c3fc 100644 --- a/hw/intc/gicv3_internal.h +++ b/hw/intc/gicv3_internal.h @@ -298,6 +298,7 @@ void gicv3_dist_set_irq(GICv3State *s, int irq, int level); void gicv3_redist_set_irq(GICv3CPUState *cs, int irq, int level); void gicv3_redist_send_sgi(GICv3CPUState *cs, int grp, int irq, bool ns); void gicv3_init_cpuif(GICv3State *s); +void gicv3_init_cpu_reginfo(CPUState *cs); /** * gicv3_cpuif_update: diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index b51f74c728..4d1cbc66db 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -294,6 +294,7 @@ typedef struct ARMGICv3CommonClass { void (*pre_save)(GICv3State *s); void (*post_load)(GICv3State *s); + void (*init_cpu_reginfo)(CPUState *cs); } ARMGICv3CommonClass; void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, From patchwork Sat Jun 13 21:36:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308771 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49krtr6yjhz9s1x for ; Sun, 14 Jun 2020 07:53:08 +1000 (AEST) Received: from localhost ([::1]:58364 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE5K-0002Dg-M6 for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:53:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60688) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyn-00085c-6h; Sat, 13 Jun 2020 17:46:21 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3780 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyl-0003Uq-7x; Sat, 13 Jun 2020 17:46:20 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id AC81130E46CC7F90ACB7; Sun, 14 Jun 2020 05:46:16 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:46:10 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 20/22] arm/cpuhp: Update the guest(via GED) about cpu hot-(un)plug events Date: Sat, 13 Jun 2020 22:36:27 +0100 Message-ID: <20200613213629.21984-21-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During any vcpu hot-(un)plug, running guest VM needs to be intimated about the new vcpu being added or request the deletion of the vcpu which is already part of the guest VM. This is done using the ACPI GED event which eventually gets demultiplexed to a CPU hotplug event and further to specific hot-(un)plug event of a particular vcpu. This change adds the ACPI calls to the existing hot-(un)plug hooks to trigger ACPI GED events from QEMU to guest VM. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index b4cfd53a59..db7eca1b84 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2451,6 +2451,7 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, MachineState *ms = MACHINE(hotplug_dev); ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; /* insert the cold/hot-plugged vcpu in the slot */ @@ -2458,20 +2459,31 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot->cpu = OBJECT(dev); if (dev->hotplugged) { + HotplugHandlerClass *hhc; + wire_gic_cpu_irqs(vms, cs); - /* TODO: update acpi hotplug state and send cpu hotplug event to guest */ + + /* update acpi hotplug state and send cpu hotplug event to guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->plug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) + goto fail; /* TODO: register this cpu for reset & update F/W info for the next boot */ } cs->disabled = false; return; +fail: + error_propagate(errp, local_err); } static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + HotplugHandlerClass *hhc; CPUState *cs = CPU(dev); + Error *local_err = NULL; if (!vms->acpi_dev || !dev->realized) { error_setg(errp, "GED does not exists or device is not realized!"); @@ -2491,9 +2503,15 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, return; } - /* TODO: request cpu hotplug from guest */ + /* request cpu hotplug from guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug_request(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) + goto fail; return; +fail: + error_propagate(errp, local_err); } static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, @@ -2501,7 +2519,9 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, { VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); + HotplugHandlerClass *hhc; CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; if (!vms->acpi_dev || !dev->realized) { @@ -2511,7 +2531,11 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot = virt_find_cpu_slot(ms, ARM_CPU(cs)->core_id); - /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + /* update the acpi cpu hotplug state for cpu hot-unplug */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) + goto fail; unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs); @@ -2524,6 +2548,8 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot->cpu = NULL; cs->disabled = true; return; +fail: + error_propagate(errp, local_err); } static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, From patchwork Sat Jun 13 21:36:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308780 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49ks3P5kVwz9s1x for ; Sun, 14 Jun 2020 08:00:33 +1000 (AEST) Received: from localhost ([::1]:54376 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkECV-0004HG-Ds for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 18:00:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60826) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyx-0008UN-E4; Sat, 13 Jun 2020 17:46:31 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:3781 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDyv-0003XB-BN; Sat, 13 Jun 2020 17:46:31 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id CF2BFCA6A4741F8A654F; Sun, 14 Jun 2020 05:46:26 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:46:16 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 21/22] arm/cpuhp: Changes required for reset and to support next boot Date: Sat, 13 Jun 2020 22:36:28 +0100 Message-ID: <20200613213629.21984-22-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:35 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Updates the firmware config with the next boot cpus information and also registers the reset callback to be called when guest reboots to reset the cpu. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/boot.c | 2 +- hw/arm/virt.c | 18 ++++++++++++++---- include/hw/arm/boot.h | 2 ++ include/hw/arm/virt.h | 1 + 4 files changed, 18 insertions(+), 5 deletions(-) diff --git a/hw/arm/boot.c b/hw/arm/boot.c index fef4072db1..05f329c1e1 100644 --- a/hw/arm/boot.c +++ b/hw/arm/boot.c @@ -675,7 +675,7 @@ fail: return -1; } -static void do_cpu_reset(void *opaque) +void do_cpu_reset(void *opaque) { ARMCPU *cpu = opaque; CPUState *cs = CPU(cpu); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index db7eca1b84..55101c0050 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -47,6 +47,7 @@ #include "sysemu/device_tree.h" #include "sysemu/numa.h" #include "sysemu/runstate.h" +#include "sysemu/reset.h" #include "sysemu/sysemu.h" #include "sysemu/tpm.h" #include "sysemu/kvm.h" @@ -1149,14 +1150,13 @@ static bool virt_firmware_init(VirtMachineState *vms, static FWCfgState *create_fw_cfg(const VirtMachineState *vms, AddressSpace *as) { - MachineState *ms = MACHINE(vms); hwaddr base = vms->memmap[VIRT_FW_CFG].base; hwaddr size = vms->memmap[VIRT_FW_CFG].size; FWCfgState *fw_cfg; char *nodename; fw_cfg = fw_cfg_init_mem_wide(base + 8, base, 8, base + 16, as); - fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, (uint16_t)ms->smp.cpus); + fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); nodename = g_strdup_printf("/fw-cfg@%" PRIx64, base); qemu_fdt_add_subnode(vms->fdt, nodename); @@ -2468,7 +2468,13 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, hhc->plug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); if (local_err) goto fail; - /* TODO: register this cpu for reset & update F/W info for the next boot */ + + /* register this cpu for reset & update F/W info for the next boot */ + qemu_register_reset(do_cpu_reset, ARM_CPU(cs)); + vms->boot_cpus++; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); + } } cs->disabled = false; @@ -2540,7 +2546,11 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs); - /* TODO: unregister this cpu for reset & update F/W info for the next boot */ + qemu_unregister_reset(do_cpu_reset, ARM_CPU(cs)); + vms->boot_cpus--; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); + } qemu_opts_del(dev->opts); dev->opts = NULL; diff --git a/include/hw/arm/boot.h b/include/hw/arm/boot.h index ce2b48b88b..aa156967af 100644 --- a/include/hw/arm/boot.h +++ b/include/hw/arm/boot.h @@ -163,6 +163,8 @@ AddressSpace *arm_boot_address_space(ARMCPU *cpu, int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, hwaddr addr_limit, AddressSpace *as, MachineState *ms); +void do_cpu_reset(void *opaque); + /* Write a secure board setup routine with a dummy handler for SMCs */ void arm_write_secure_board_setup_dummy_smc(ARMCPU *cpu, const struct arm_boot_info *info, diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index c287433219..df785ea6ba 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -147,6 +147,7 @@ typedef struct { const int *irqmap; int smp_cpus; int max_cpus; + uint16_t boot_cpus; void *fdt; int fdt_size; uint32_t clock_phandle; From patchwork Sat Jun 13 21:36:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 1308778 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49ks1X1fWTz9s1x for ; Sun, 14 Jun 2020 07:58:56 +1000 (AEST) Received: from localhost ([::1]:49778 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkEAv-0002R1-VP for incoming@patchwork.ozlabs.org; Sat, 13 Jun 2020 17:58:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDz3-0000IX-H9; Sat, 13 Jun 2020 17:46:37 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:3711 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDz0-0003ZU-Mh; Sat, 13 Jun 2020 17:46:37 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id DF420C054C62182343E8; Sun, 14 Jun 2020 05:46:31 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:46:22 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 22/22] arm/cpuhp: Add support of *unrealize* ARMCPU during vcpu hot-unplug Date: Sat, 13 Jun 2020 22:36:29 +0100 Message-ID: <20200613213629.21984-23-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:30 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During vcpu hot-unplug ARM cpu unrealization shall happen which should do away with all the vcpu thread creations, allocations, registrations which happened as part of the realization process of the ARM cpu. This change introduces the ARM cpu unrealize function taking care of exactly that. Note, initialized vcpus are not destroyed at host KVM but are rather parked in the QEMU/KVM layer. These are later reused once vcpu is hotplugged again. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- exec.c | 24 ++++++++++++ gdbstub.c | 13 +++++++ include/exec/exec-all.h | 8 ++++ include/exec/gdbstub.h | 1 + include/hw/core/cpu.h | 2 + target/arm/cpu-qom.h | 3 ++ target/arm/cpu.c | 86 +++++++++++++++++++++++++++++++++++++++++ target/arm/cpu.h | 13 +++++++ target/arm/helper.c | 31 +++++++++++++++ target/arm/internals.h | 1 + target/arm/kvm64.c | 7 +++- 11 files changed, 188 insertions(+), 1 deletion(-) diff --git a/exec.c b/exec.c index a0bf9d61c8..7e808affdf 100644 --- a/exec.c +++ b/exec.c @@ -869,6 +869,7 @@ void cpu_address_space_init(CPUState *cpu, int asidx, if (!cpu->cpu_ases) { cpu->cpu_ases = g_new0(CPUAddressSpace, cpu->num_ases); + cpu->cpu_ases_ref_count = cpu->num_ases; } newas = &cpu->cpu_ases[asidx]; @@ -881,6 +882,29 @@ void cpu_address_space_init(CPUState *cpu, int asidx, } } +void cpu_address_space_destroy(CPUState *cpu, int asidx) +{ + CPUAddressSpace *cpuas; + + assert(asidx < cpu->num_ases); + assert(asidx == 0 || !kvm_enabled()); + assert(cpu->cpu_ases); + + cpuas = &cpu->cpu_ases[asidx]; + if (tcg_enabled()) { + memory_listener_unregister(&cpuas->tcg_as_listener); + } + + address_space_destroy(cpuas->as); + + if(cpu->cpu_ases_ref_count == 1) { + g_free(cpu->cpu_ases); + cpu->cpu_ases = NULL; + } + + cpu->cpu_ases_ref_count--; +} + AddressSpace *cpu_get_address_space(CPUState *cpu, int asidx) { /* Return the AddressSpace corresponding to the specified index */ diff --git a/gdbstub.c b/gdbstub.c index 6950fd243f..e960268d15 100644 --- a/gdbstub.c +++ b/gdbstub.c @@ -982,6 +982,19 @@ void gdb_register_coprocessor(CPUState *cpu, } } +void gdb_unregister_coprocessor_all(CPUState *cpu) +{ + GDBRegisterState *s, *p; + + p = cpu->gdb_regs; + while (p) { + s = p; + p = p->next; + g_free(s); + } + cpu->gdb_regs = NULL; +} + #ifndef CONFIG_USER_ONLY /* Translate GDB watchpoint type to a flags value for cpu_watchpoint_* */ static inline int xlat_gdb_type(CPUState *cpu, int gdbtype) diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 8792bea07a..44420c144d 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -115,6 +115,14 @@ void cpu_reloading_memory_map(void); */ void cpu_address_space_init(CPUState *cpu, int asidx, const char *prefix, MemoryRegion *mr); +/** + * cpu_address_space_destroy: + * @cpu: CPU for which address space needs to be destroyed + * @asidx: integer index of this address space + * + * Note that with KVM only one address space is supported. + */ +void cpu_address_space_destroy(CPUState *cpu, int asidx); #endif #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_TCG) diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h index 94d8f83e92..db2336d0df 100644 --- a/include/exec/gdbstub.h +++ b/include/exec/gdbstub.h @@ -73,6 +73,7 @@ typedef int (*gdb_set_reg_cb)(CPUArchState *env, uint8_t *buf, int reg); void gdb_register_coprocessor(CPUState *cpu, gdb_get_reg_cb get_reg, gdb_set_reg_cb set_reg, int num_regs, const char *xml, int g_pos); +void gdb_unregister_coprocessor_all(CPUState *cpu); /* * The GDB remote protocol transfers values in target byte order. As diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index d9cae71ea5..851f03dd83 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -379,6 +379,7 @@ struct CPUState { struct qemu_work_item *queued_work_first, *queued_work_last; CPUAddressSpace *cpu_ases; + int cpu_ases_ref_count; int num_ases; AddressSpace *as; MemoryRegion *memory; @@ -410,6 +411,7 @@ struct CPUState { int kvm_fd; struct KVMState *kvm_state; struct kvm_run *kvm_run; + VMChangeStateEntry *vmcse; /* Used for events with 'vcpu' and *without* the 'disabled' properties */ DECLARE_BITMAP(trace_dstate_delayed, CPU_TRACE_DSTATE_MAX_EVENTS); diff --git a/target/arm/cpu-qom.h b/target/arm/cpu-qom.h index 56395b87f6..d943683551 100644 --- a/target/arm/cpu-qom.h +++ b/target/arm/cpu-qom.h @@ -58,6 +58,7 @@ typedef struct ARMCPUClass { const ARMCPUInfo *info; DeviceRealize parent_realize; + DeviceUnrealize parent_unrealize; DeviceReset parent_reset; } ARMCPUClass; @@ -76,7 +77,9 @@ typedef struct AArch64CPUClass { } AArch64CPUClass; void register_cp_regs_for_features(ARMCPU *cpu); +void unregister_cp_regs_for_features(ARMCPU *cpu); void init_cpreg_list(ARMCPU *cpu); +void destroy_cpreg_list(ARMCPU *cpu); /* Callback functions for the generic timer's timers. */ void arm_gt_ptimer_cb(void *opaque); diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 0c9f5f970e..dac42c418d 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -92,6 +92,16 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->pre_el_change_hooks, entry, node); } +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->pre_el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque) { @@ -103,6 +113,16 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node); } +void arm_unregister_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque) { /* Reset a single ARMCPRegInfo register */ @@ -1765,6 +1785,70 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp) acc->parent_realize(dev, errp); } +static void arm_cpu_unrealizefn(DeviceState *dev) +{ + ARMCPUClass *acc = ARM_CPU_GET_CLASS(dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUARMState *env = &cpu->env; + CPUState *cs = CPU(dev); + + /* rock 'n' un-roll, whatever happened in the arm_cpu_realizefn cleanly */ + if (cpu->has_el3 || arm_feature(env, ARM_FEATURE_M_SECURITY)) { + cpu_address_space_destroy(cs, ARMASIdx_S); + } + cpu_address_space_destroy(cs, ARMASIdx_NS); + + destroy_cpreg_list(cpu); + arm_cpu_unregister_gdb_regs(cpu); + unregister_cp_regs_for_features(cpu); + + if (cpu->sau_sregion && arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->sau.rbar); + g_free(env->sau.rlar); + } + + if (arm_feature(env, ARM_FEATURE_PMSA) && + arm_feature(env, ARM_FEATURE_V7) && + cpu->pmsav7_dregion) { + if (arm_feature(env, ARM_FEATURE_V8)) { + g_free(env->pmsav8.rbar[M_REG_NS]); + g_free(env->pmsav8.rlar[M_REG_NS]); + if (arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->pmsav8.rbar[M_REG_S]); + g_free(env->pmsav8.rlar[M_REG_S]); + } + } else { + g_free(env->pmsav7.drbar); + g_free(env->pmsav7.drsr); + g_free(env->pmsav7.dracr); + } + } + + if (arm_feature(env, ARM_FEATURE_PMU)) { + if (!kvm_enabled()) { + arm_unregister_pre_el_change_hooks(cpu); + arm_unregister_el_change_hooks(cpu); + } + +#ifndef CONFIG_USER_ONLY + if (cpu->pmu_timer) { + timer_del(cpu->pmu_timer); + } +#endif + } + + cpu_remove_sync(CPU(dev)); + acc->parent_unrealize(dev); + +#ifndef CONFIG_USER_ONLY + timer_del(cpu->gt_timer[GTIMER_PHYS]); + timer_del(cpu->gt_timer[GTIMER_VIRT]); + timer_del(cpu->gt_timer[GTIMER_HYP]); + timer_del(cpu->gt_timer[GTIMER_SEC]); + timer_del(cpu->gt_timer[GTIMER_HYPVIRT]); +#endif +} + static ObjectClass *arm_cpu_class_by_name(const char *cpu_model) { ObjectClass *oc; @@ -2145,6 +2229,8 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) device_class_set_parent_realize(dc, arm_cpu_realizefn, &acc->parent_realize); + device_class_set_parent_unrealize(dc, arm_cpu_unrealizefn, + &acc->parent_unrealize); device_class_set_props(dc, arm_cpu_properties); device_class_set_parent_reset(dc, arm_cpu_reset, &acc->parent_reset); diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 5c4991156e..a8e7cb9fb1 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -3337,6 +3337,13 @@ static inline AddressSpace *arm_addressspace(CPUState *cs, MemTxAttrs attrs) */ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_pre_el_change_hook: + * unregister all pre EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu); + /** * arm_register_el_change_hook: * Register a hook function which will be called immediately after this @@ -3349,6 +3356,12 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, */ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_el_change_hook: + * unregister all EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_el_change_hooks(ARMCPU *cpu); /** * arm_rebuild_hflags: diff --git a/target/arm/helper.c b/target/arm/helper.c index 972a766730..dc4100ea89 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -502,6 +502,19 @@ void init_cpreg_list(ARMCPU *cpu) g_list_free(keys); } +void destroy_cpreg_list(ARMCPU *cpu) +{ + assert(cpu->cpreg_indexes); + assert(cpu->cpreg_values); + assert(cpu->cpreg_vmstate_indexes); + assert(cpu->cpreg_vmstate_values); + + g_free(cpu->cpreg_indexes); + g_free(cpu->cpreg_values); + g_free(cpu->cpreg_vmstate_indexes); + g_free(cpu->cpreg_vmstate_values); +} + /* * Some registers are not accessible from AArch32 EL3 if SCR.NS == 0. */ @@ -7981,6 +7994,18 @@ void register_cp_regs_for_features(ARMCPU *cpu) #endif } +void unregister_cp_regs_for_features(ARMCPU *cpu) +{ + CPUARMState *env = &cpu->env; + if (arm_feature(env, ARM_FEATURE_M)) { + /* M profile has no coprocessor registers */ + return; + } + + /* empty it all. unregister all the coprocessor registers */ + g_hash_table_remove_all(cpu->cp_regs); +} + void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu) { CPUState *cs = CPU(cpu); @@ -8019,6 +8044,12 @@ void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu) } +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + gdb_unregister_coprocessor_all(cs); +} + /* Sort alphabetically by type name, except for "any". */ static gint arm_cpu_list_compare(gconstpointer a, gconstpointer b) { diff --git a/target/arm/internals.h b/target/arm/internals.h index 4bdbc3a8ac..8ece9e09f5 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -169,6 +169,7 @@ static inline int r14_bank_number(int mode) } void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu); +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu); void arm_translate_init(void); enum arm_fprounding { diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c index f09ed9f4df..b6df17912e 100644 --- a/target/arm/kvm64.c +++ b/target/arm/kvm64.c @@ -775,7 +775,9 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cs); + if (qemu_present_cpu(cs)) + cs->vmcse = qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, + cs); /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); @@ -840,6 +842,9 @@ int kvm_arch_init_vcpu(CPUState *cs) int kvm_arch_destroy_vcpu(CPUState *cs) { + if (qemu_present_cpu(cs)) + qemu_del_vm_change_state_handler(cs->vmcse); + return 0; }