From patchwork Wed Feb 1 13:20:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre Morel X-Patchwork-Id: 1735523 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=AHHhrdXI; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4P6N0C2WtVz23jJ for ; Thu, 2 Feb 2023 00:23:23 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pND3i-00068P-Ak; Wed, 01 Feb 2023 08:21:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pND39-000632-6s; Wed, 01 Feb 2023 08:21:20 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pND34-000434-Nq; Wed, 01 Feb 2023 08:21:18 -0500 Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 311DDZOe026626; Wed, 1 Feb 2023 13:21:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=okONxH0aAPNhNOjJ97fVS/MkzCYU3AKdagPS45hX2Sw=; b=AHHhrdXI7yWHxgetvKML0guSeImgOKCfDMpK4jIeyf1wnJAhSHV9U5qYBucuuWzxik/f SCtrfES3DxWJfROYDn9kbmYkGy4KOOtpy1feEes8/HiMh89BfkKx9p+H4NhX6vcBpVLh 8+no6gO+3BY4WvJZxnVV/XAd+Dvofqb4ZmpH5cI8t11uDVVrcsePNNSUYOWUVpuMhBzd 7pBAm5ZW/vKsv2A5HhwuhQf0I7jCdQimZDS0OeX6Fs3t3DVf2vVeyRx8QU0WwcChV+ln M4VOXDBC/nvHuF00M22pBtviN/wB1EO/W8HV1mKlPFk1B0e4ZZglDQaihLjrJm+sHhG0 zw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3nfrj28q5u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Feb 2023 13:21:03 +0000 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 311Cxhbu012610; Wed, 1 Feb 2023 13:21:02 GMT Received: from ppma06fra.de.ibm.com (48.49.7a9f.ip4.static.sl-reverse.com [159.122.73.72]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3nfrj28q3q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Feb 2023 13:21:02 +0000 Received: from pps.filterd (ppma06fra.de.ibm.com [127.0.0.1]) by ppma06fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 30VGeuwF014228; Wed, 1 Feb 2023 13:20:59 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma06fra.de.ibm.com (PPS) with ESMTPS id 3ncvugkggb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Feb 2023 13:20:59 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 311DKtTE50004226 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 1 Feb 2023 13:20:55 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B518220040; Wed, 1 Feb 2023 13:20:55 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5F78420043; Wed, 1 Feb 2023 13:20:54 +0000 (GMT) Received: from li-c6ac47cc-293c-11b2-a85c-d421c8e4747b.ibm.com.com (unknown [9.179.4.198]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 1 Feb 2023 13:20:54 +0000 (GMT) From: Pierre Morel To: qemu-s390x@nongnu.org Cc: qemu-devel@nongnu.org, borntraeger@de.ibm.com, pasic@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, mst@redhat.com, pbonzini@redhat.com, kvm@vger.kernel.org, ehabkost@redhat.com, marcel.apfelbaum@gmail.com, eblake@redhat.com, armbru@redhat.com, seiden@linux.ibm.com, nrb@linux.ibm.com, nsg@linux.ibm.com, frankja@linux.ibm.com, berrange@redhat.com, clg@kaod.org Subject: [PATCH v15 00/11] s390x: CPU Topology Date: Wed, 1 Feb 2023 14:20:40 +0100 Message-Id: <20230201132051.126868-1-pmorel@linux.ibm.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: A_wJC0wqshsmFYLFQ1Ni1iqpHK8UFc6b X-Proofpoint-ORIG-GUID: 3PMdHPgGzws55iaEqau54oh63mrkQtaH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-02-01_04,2023-01-31_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 suspectscore=0 spamscore=0 malwarescore=0 impostorscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302010112 Received-SPF: pass client-ip=148.163.156.1; envelope-from=pmorel@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Hi, Two important changes in this series: - The ordering of the Topology List Entries, TLE, is done just before building the System information Buffer, SYSIB, during the interception of STSI. The adventage of this is to do the ordering only if it is needed whatever the reason for a change of the topology is, hotplug, qapi, instruction PTF(2) from the guest. - query-topology is no longer needed, it is replaced by an extension of query-cpus-fast. A small change that lead me to remove @Thomas RB: - Add a trampolin function in cpu-topology code for the subsystem reset for extending to PTF reset in the following patch on PTF implementation. @Nina and @Nico RB have been removed from reset patch too in the last series because of small changes too. Implementation discussions ========================== CPU models ---------- Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY. It is the responsibility of the admin to ensure the same CPU model for source and target host in a migration. Migration --------- When the target guest is started, the Multi-processor Topology Change Report (MTCR) bit is set during the creation of the vCPU by KVM. We do not need to migrate its state, in the worst case, the target guest will see the MTCR and actualize its view of the topology without necessity, but this will be done only one time. Reset ----- Reseting the topology is done during subsystem reset, the polarization is reset to horizontal polarization. Topology attributes ------------------- The topology attributes are carried by the CPU object and defined on object creation. In the case the new attributes, socket, book, drawer, dedicated, polarity are not provided QEMU provides defaults values. - Geometry defaults The geometry default are based on the core-id of the core to fill the geometry in a monotone way starting with drawer 0, book 0, and filling socket 0 with the number of cores per socket, then filling socket 1, socket 2 ... etc until the book is complete and all books until the first drawer is complete before starting with the next drawer. This allows to keep existing start scripts and Libvirt existing interface until it is extended. - Modifiers defaults Default polarization is horizontal Default dedication is not dedicated. Dynamic topology modification ----------------------------- QAPI interface is extended with: - a command: 'x-set-cpu-topology' - a query: extension of 'query-cpus-fast' - an event: 'CPU_POLARITY_CHANGE' The admin may use query-cpus-fast to verify the topology provided to the guest and x-set-cpu-topology to modify it. The event CPU_POLARITY_CHANGE is sent when the guest successfuly uses the PTF(2) instruction to request a polarization change. In that case, the admin is supposed to modify the CPU provisioning accordingly. Testing ======= To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches: f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series. Documentation ============= To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series. The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine. Regards, Pierre Pierre Morel (11): s390x/cpu topology: adding s390 specificities to CPU topology s390x/cpu topology: add topology entries on CPU hotplug target/s390x/cpu topology: handle STSI(15) and build the SYSIB s390x/sclp: reporting the maximum nested topology entries s390x/cpu topology: resetting the Topology-Change-Report s390x/cpu topology: interception of PTF instruction target/s390x/cpu topology: activating CPU topology qapi/s390x/cpu topology: x-set-cpu-topology monitor command machine: adding s390 topology to query-cpu-fast qapi/s390x/cpu topology: CPU_POLARITY_CHANGE qapi event docs/s390x/cpu topology: document s390x cpu topology docs/system/s390x/cpu-topology.rst | 294 ++++++++++++++++++ docs/system/target-s390x.rst | 1 + qapi/machine-target.json | 59 ++++ qapi/machine.json | 27 +- include/hw/boards.h | 10 +- include/hw/s390x/cpu-topology.h | 71 +++++ include/hw/s390x/s390-virtio-ccw.h | 6 + include/hw/s390x/sclp.h | 4 +- include/monitor/hmp.h | 1 + target/s390x/cpu.h | 79 +++++ target/s390x/kvm/kvm_s390x.h | 1 + hw/core/machine-qmp-cmds.c | 2 + hw/core/machine-smp.c | 48 ++- hw/core/machine.c | 4 + hw/s390x/cpu-topology.c | 471 +++++++++++++++++++++++++++++ hw/s390x/s390-virtio-ccw.c | 28 +- hw/s390x/sclp.c | 5 + softmmu/vl.c | 6 + target/s390x/cpu-sysemu.c | 27 ++ target/s390x/cpu.c | 7 + target/s390x/cpu_models.c | 1 + target/s390x/kvm/cpu_topology.c | 335 ++++++++++++++++++++ target/s390x/kvm/kvm.c | 45 ++- hmp-commands.hx | 16 + hw/s390x/meson.build | 1 + qemu-options.hx | 7 +- target/s390x/kvm/meson.build | 3 +- 27 files changed, 1539 insertions(+), 20 deletions(-) create mode 100644 docs/system/s390x/cpu-topology.rst create mode 100644 include/hw/s390x/cpu-topology.h create mode 100644 hw/s390x/cpu-topology.c create mode 100644 target/s390x/kvm/cpu_topology.c