{"id":2218901,"url":"http://patchwork.ozlabs.org/api/1.0/patches/2218901/?format=json","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/1.0/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20260402022921.298818-15-kshk@linux.ibm.com>","date":"2026-04-02T02:29:20","name":"[14/15] s390x/pci: Implement migration for emulated devices","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"f7a09cae70f9c27eabe7785526d92388622ec498","submitter":{"id":90249,"url":"http://patchwork.ozlabs.org/api/1.0/people/90249/?format=json","name":"Konstantin Shkolnyy","email":"kshk@linux.ibm.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/patch/20260402022921.298818-15-kshk@linux.ibm.com/mbox/","series":[{"id":498419,"url":"http://patchwork.ozlabs.org/api/1.0/series/498419/?format=json","date":"2026-04-02T02:29:08","name":"s390x/pci: Implement migration for emulated devices","version":1,"mbox":"http://patchwork.ozlabs.org/series/498419/mbox/"}],"check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/2218901/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256\n header.s=pp1 header.b=U9h/5jsV;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4fmQpm0hjYz1yGH\n\tfor ; Thu, 02 Apr 2026 13:31:44 +1100 (AEDT)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1w87pU-0004kQ-E2; Wed, 01 Apr 2026 22:30:44 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1w87ov-0004Eh-6k; Wed, 01 Apr 2026 22:30:10 -0400","from mx0a-001b2d01.pphosted.com ([148.163.156.1])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1w87ot-0001yn-1z; Wed, 01 Apr 2026 22:30:08 -0400","from pps.filterd (m0360083.ppops.net [127.0.0.1])\n by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id\n 631M4hsB560642; Thu, 2 Apr 2026 02:30:03 GMT","from ppma11.dal12v.mail.ibm.com\n (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219])\n by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4d66g231pu-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);\n Thu, 02 Apr 2026 02:30:02 +0000 (GMT)","from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1])\n by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id\n 6321iueg008698;\n Thu, 2 Apr 2026 02:30:02 GMT","from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5])\n by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4d6v11r449-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);\n Thu, 02 Apr 2026 02:30:02 +0000","from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com\n [10.39.53.233])\n by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id\n 6322U03N29491796\n (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Thu, 2 Apr 2026 02:30:01 GMT","from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1])\n by IMSVA (Postfix) with ESMTP id C26C8580F4;\n Thu, 2 Apr 2026 02:30:00 +0000 (GMT)","from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1])\n by IMSVA (Postfix) with ESMTP id 39DB3580F6;\n Thu, 2 Apr 2026 02:30:00 +0000 (GMT)","from WIN-DU0DFC9G5VV.ibm.com (unknown [9.61.247.119])\n by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP;\n Thu, 2 Apr 2026 02:30:00 +0000 (GMT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc\n :content-transfer-encoding:date:from:in-reply-to:message-id\n :mime-version:references:subject:to; s=pp1; bh=t5cVYkxBAOpmPSSXd\n FdsVxwtAJBjRxBXKstytrtL+qc=; b=U9h/5jsV5fZ38DkQM8nGtw9voZEy30wuf\n 0l+2LXyUOLfC7OoA7MIhgM2zZf+CgBRyqfWru2PoItC7qkI+MoY6FRB7zkNyVBIK\n xVSmnP/7N9YxHae6gySPuRhmPsllgxCG1Pq8J1jrU7+nvWu0mBjGyD33xulJ/3Ly\n LqhNfXVJkFyCz/fnUzTaQxfDjRIHWKcsf8OjYZEKnxzjZDiBdCakNdJLknSCxGPe\n 4Id87xLeVdJhOTAuFKjfD8YFvQOMxVYM5RqFL5ZNyCAXoh09tzRU40HyxIpokV+F\n ePuoJ0puOmtIG/Kn/K5auhVOH7ZljY2wD3UBnvqDhfuz6smy3xiZA==","From":"Konstantin Shkolnyy ","To":"mjrosato@linux.ibm.com","Cc":"alifm@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com,\n david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com,\n borntraeger@linux.ibm.com, qemu-s390x@nongnu.org,\n qemu-devel@nongnu.org, Konstantin Shkolnyy ","Subject":"[PATCH 14/15] s390x/pci: Implement migration for emulated devices","Date":"Wed, 1 Apr 2026 21:29:20 -0500","Message-Id":"<20260402022921.298818-15-kshk@linux.ibm.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20260402022921.298818-1-kshk@linux.ibm.com>","References":"<20260402022921.298818-1-kshk@linux.ibm.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-TM-AS-GCONF":"00","X-Authority-Analysis":"v=2.4 cv=Fdo6BZ+6 c=1 sm=1 tr=0 ts=69cdd4aa cx=c_pps\n a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17\n a=A5OVakUREuEA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22\n a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=1WZqjCYO3q-P4G2YYo4A:9","X-Proofpoint-ORIG-GUID":"oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD","X-Proofpoint-Spam-Details-Enc":"AW1haW4tMjYwNDAyMDAxOSBTYWx0ZWRfX/Z0hZxiogMlK\n A07AO9zBdtKJuQnFveUdp1kP61DHQXVuKKNktKgcnvY/TMphbSfrJNa1j42thT6A1xFJ+cAZ1P2\n egPQiQNpwJw+JkjEbO+OFZGztnMme9pfTR3CscTyxIBkz6ylcmQAi/vC1zG4+62SFlBaolL0VS3\n HVkSteYwL9CvIJJvWxL2B5efwizzmveWZDUY6xp/XTvhZjyN2DXaRG4ETAEKmdC65+DKMFh9UxF\n jHV6zik3fGoxkewFMgfiP562g1y06s0TRA87+yj1Y3Jx6MV9gxsDayG7yA4lU4GneHORj6PIpwg\n ObekpDNmtwolZk0lcLUOz97F5ZLOP6vOk4MsRn6b51bWwroK+9Xvr26+Jp3OC6MufqoOZKBZF76\n k0BHOplINqx+neIQgE/9egWJ9JshsvQNg0DXJcNyFFumCb/Fs6iB2xvPQAebnfeej3ptD8fiakj\n nMY2Eh28lzuZVB1U0HA==","X-Proofpoint-GUID":"oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49\n definitions=2026-04-01_05,2026-04-01_02,2025-10-01_01","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n impostorscore=0 clxscore=1015 bulkscore=0 suspectscore=0 priorityscore=1501\n adultscore=0 malwarescore=0 phishscore=0 spamscore=0 lowpriorityscore=0\n classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0\n reason=mlx scancount=1 engine=8.22.0-2603050001 definitions=main-2604020019","Received-SPF":"pass client-ip=148.163.156.1; envelope-from=kshk@linux.ibm.com;\n helo=mx0a-001b2d01.pphosted.com","X-Spam_score_int":"-6","X-Spam_score":"-0.7","X-Spam_bar":"/","X-Spam_report":"(-0.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7,\n RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=1, RCVD_IN_VALIDITY_RPBL_BLOCKED=1,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"qemu development ","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"},"content":"Implement zPCI device state migration, consequently enabling migration\nof VMs that have emulated PCI devices, whether virtio or not.\nMigration is allowed for devices whose function handle has the\nFH_SHM_EMUL bit set. For these devices QEMU will save and restore the\nstate of its zPCI emulator.\n\nPassthrough devices will continue to block migration.\n\nSigned-off-by: Konstantin Shkolnyy \n---\n hw/s390x/s390-pci-bus.c | 112 ++++++++++++++++++++++++++++++--\n hw/s390x/s390-pci-inst.c | 2 +-\n include/hw/s390x/s390-pci-bus.h | 2 +\n 3 files changed, 110 insertions(+), 6 deletions(-)","diff":"diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c\nindex f339a0fd94..b3037b9a8a 100644\n--- a/hw/s390x/s390-pci-bus.c\n+++ b/hw/s390x/s390-pci-bus.c\n@@ -26,6 +26,7 @@\n #include \"hw/pci/pci_bridge.h\"\n #include \"hw/pci/msi.h\"\n #include \"exec/cpu-common.h\"\n+#include \"migration/blocker.h\"\n #include \"qemu/error-report.h\"\n #include \"qemu/module.h\"\n #include \"system/reset.h\"\n@@ -1116,6 +1117,28 @@ static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev)\n return 0;\n }\n \n+static int s390_set_passthrough_migration_blocker(S390PCIBusDevice *pbdev,\n+ Error **errp)\n+{\n+ pbdev->passthrough_migr_blocker = NULL;\n+\n+ if (pbdev->fh & FH_SHM_EMUL) {\n+ return 0;\n+ }\n+ error_setg(&pbdev->passthrough_migr_blocker,\n+ \"Migration blocked by passthrough zPCI device \"\n+ \"fh 0x%x uid %d fid %d\", pbdev->fh, pbdev->uid, pbdev->fid);\n+\n+ return migrate_add_blocker(&pbdev->passthrough_migr_blocker, errp);\n+}\n+\n+static void s390_clear_passthrough_migration_blocker(S390PCIBusDevice *pbdev)\n+{\n+ if (pbdev->passthrough_migr_blocker) {\n+ migrate_del_blocker(&pbdev->passthrough_migr_blocker);\n+ }\n+}\n+\n static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,\n Error **errp)\n {\n@@ -1235,6 +1258,11 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,\n return;\n }\n \n+ if (s390_set_passthrough_migration_blocker(pbdev, errp)) {\n+ s390_pci_msix_free(pbdev);\n+ return;\n+ }\n+\n if (dev->hotplugged) {\n s390_pci_generate_plug_event(HP_EVENT_TO_CONFIGURED ,\n pbdev->fh, pbdev->fid);\n@@ -1271,6 +1299,8 @@ static void s390_pcihost_unplug(HotplugHandler *hotplug_dev, DeviceState *dev,\n return;\n }\n \n+ s390_clear_passthrough_migration_blocker(pbdev);\n+\n s390_pci_generate_plug_event(HP_EVENT_STANDBY_TO_RESERVED,\n pbdev->fh, pbdev->fid);\n bus = pci_get_bus(pci_dev);\n@@ -1600,13 +1630,85 @@ static const Property s390_pci_device_properties[] = {\n true),\n };\n \n-static const VMStateDescription s390_pci_device_vmstate = {\n- .name = TYPE_S390_PCI_DEVICE,\n+static int s390_pci_device_post_load(void *opaque, int version_id)\n+{\n+ S390PCIBusDevice *pbdev = S390_PCI_DEVICE(opaque);\n+\n+ /*\n+ * Now that S390PCIBusDevice fields have been restored, regenerate IOMMU\n+ * state - that includes IOTLB contents and QEMU memory regions.\n+ */\n+ if (pbdev->iommu_enabled) {\n+ assert(pbdev->iommu);\n+ if (s390_pci_is_translation_enabled(pbdev->g_iota)) {\n+ s390_pci_iommu_enable(pbdev);\n+ s390_pci_ioat_replay(pbdev);\n+ } else {\n+ s390_pci_iommu_direct_map_enable(pbdev);\n+ }\n+ }\n+\n /*\n- * TODO: add state handling here, so migration works at least with\n- * emulated pci devices on s390x\n+ * Guest sets fmb_addr by mpcifc.ZPCI_MOD_FC_SET_MEASURE instruction.\n+ * The handler consequently starts fmb_timer. Now that fmb_addr has been\n+ * restored, we may need to restart fmb_timer.\n */\n- .unmigratable = 1,\n+ if (pbdev->fmb_addr) {\n+ assert(!pbdev->fmb_timer);\n+ assert(pbdev->pci_group);\n+ pbdev->fmb_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL,\n+ fmb_update, pbdev);\n+ timer_mod(pbdev->fmb_timer,\n+ qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) +\n+ pbdev->pci_group->zpci_group.mui);\n+ }\n+ return 0;\n+}\n+\n+static const VMStateDescription s390_pci_device_vmstate = {\n+ .name = TYPE_S390_PCI_DEVICE,\n+ .version_id = 1,\n+ .minimum_version_id = 1,\n+ .post_load = s390_pci_device_post_load,\n+ .fields = (const VMStateField[]) {\n+ VMSTATE_UINT32(state, S390PCIBusDevice),\n+ VMSTATE_UINT16(uid, S390PCIBusDevice),\n+ VMSTATE_UINT32(idx, S390PCIBusDevice),\n+ VMSTATE_UINT32(fh, S390PCIBusDevice),\n+ VMSTATE_UINT32(fid, S390PCIBusDevice),\n+ VMSTATE_BOOL(fid_defined, S390PCIBusDevice),\n+ VMSTATE_UINT64(fmb_addr, S390PCIBusDevice),\n+ VMSTATE_UINT32(fmb.format, S390PCIBusDevice),\n+ VMSTATE_UINT32(fmb.sample, S390PCIBusDevice),\n+ VMSTATE_UINT64(fmb.last_update, S390PCIBusDevice),\n+ VMSTATE_UINT64_ARRAY(fmb.counter, S390PCIBusDevice,\n+ ARRAY_SIZE(((S390PCIBusDevice *)0)->fmb.counter)),\n+ VMSTATE_UINT64(fmb.fmt0.dma_rbytes, S390PCIBusDevice),\n+ VMSTATE_UINT64(fmb.fmt0.dma_wbytes, S390PCIBusDevice),\n+ VMSTATE_UINT8(isc, S390PCIBusDevice),\n+ VMSTATE_UINT16(noi, S390PCIBusDevice),\n+ VMSTATE_UINT8(sum, S390PCIBusDevice),\n+ VMSTATE_UINT8(pft, S390PCIBusDevice),\n+ VMSTATE_UINT64(routes.adapter.ind_addr, S390PCIBusDevice),\n+ VMSTATE_UINT64(routes.adapter.summary_addr, S390PCIBusDevice),\n+ VMSTATE_UINT64(routes.adapter.ind_offset, S390PCIBusDevice),\n+ VMSTATE_UINT32(routes.adapter.summary_offset, S390PCIBusDevice),\n+ VMSTATE_UINT32(routes.adapter.adapter_id, S390PCIBusDevice),\n+ VMSTATE_BOOL(iommu_enabled, S390PCIBusDevice),\n+ VMSTATE_UINT64(g_iota, S390PCIBusDevice),\n+ VMSTATE_UINT64(pba, S390PCIBusDevice),\n+ VMSTATE_UINT64(pal, S390PCIBusDevice),\n+ VMSTATE_UINT64(max_dma_limit, S390PCIBusDevice),\n+ VMSTATE_PTR_TO_IND_ADDR(summary_ind, S390PCIBusDevice),\n+ VMSTATE_PTR_TO_IND_ADDR(indicator, S390PCIBusDevice),\n+ VMSTATE_BOOL(pci_unplug_request_processed, S390PCIBusDevice),\n+ VMSTATE_BOOL(unplug_requested, S390PCIBusDevice),\n+ VMSTATE_BOOL(interp, S390PCIBusDevice),\n+ VMSTATE_BOOL(forwarding_assist, S390PCIBusDevice),\n+ VMSTATE_BOOL(aif, S390PCIBusDevice),\n+ VMSTATE_BOOL(rtr_avail, S390PCIBusDevice),\n+ VMSTATE_END_OF_LIST()\n+ }\n };\n \n static void s390_pci_device_class_init(ObjectClass *klass, const void *data)\ndiff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c\nindex 191f549ba7..a9c0a4effb 100644\n--- a/hw/s390x/s390-pci-inst.c\n+++ b/hw/s390x/s390-pci-inst.c\n@@ -1094,7 +1094,7 @@ static int fmb_do_update(S390PCIBusDevice *pbdev, int offset, uint64_t val,\n return ret;\n }\n \n-static void fmb_update(void *opaque)\n+void fmb_update(void *opaque)\n {\n S390PCIBusDevice *pbdev = opaque;\n int64_t t = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL);\ndiff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h\nindex 2fab28e6e0..2edb023112 100644\n--- a/include/hw/s390x/s390-pci-bus.h\n+++ b/include/hw/s390x/s390-pci-bus.h\n@@ -337,6 +337,7 @@ struct S390PCIBusDevice {\n uint16_t uid;\n uint32_t idx;\n uint32_t fh;\n+ Error *passthrough_migr_blocker;\n uint32_t fid;\n bool fid_defined;\n uint64_t fmb_addr;\n@@ -415,5 +416,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_pci(S390pciState *s,\n S390PCIBusDevice *s390_pci_find_next_avail_dev(S390pciState *s,\n S390PCIBusDevice *pbdev);\n void s390_pci_ism_reset(void);\n+void fmb_update(void *opaque);\n \n #endif\n","prefixes":["14/15"]}