Cover Letter Detail
Show a cover letter.
GET /api/1.0/covers/2221183/?format=api
{ "id": 2221183, "url": "http://patchwork.ozlabs.org/api/1.0/covers/2221183/?format=api", "project": { "id": 14, "url": "http://patchwork.ozlabs.org/api/1.0/projects/14/?format=api", "name": "QEMU Development", "link_name": "qemu-devel", "list_id": "qemu-devel.nongnu.org", "list_email": "qemu-devel@nongnu.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20260408165559.157108-1-peterx@redhat.com>", "date": "2026-04-08T16:55:44", "name": "[00/14] migration/vfio: Fix a few issues on API misuse or statistic reports", "submitter": { "id": 67717, "url": "http://patchwork.ozlabs.org/api/1.0/people/67717/?format=api", "name": "Peter Xu", "email": "peterx@redhat.com" }, "series": [ { "id": 499176, "url": "http://patchwork.ozlabs.org/api/1.0/series/499176/?format=api", "date": "2026-04-08T16:55:55", "name": "migration/vfio: Fix a few issues on API misuse or statistic reports", "version": 1, "mbox": "http://patchwork.ozlabs.org/series/499176/mbox/" } ], "headers": { "Return-Path": "<qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>", "X-Original-To": "incoming@patchwork.ozlabs.org", "Delivered-To": "patchwork-incoming@legolas.ozlabs.org", "Authentication-Results": [ "legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=d4Kocig/;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=google header.b=DTWm8Zmy;\n\tdkim-atps=neutral", "legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)" ], "Received": [ "from lists.gnu.org (lists1p.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4frYJk3qXYz1yD3\n\tfor <incoming@patchwork.ozlabs.org>; Thu, 09 Apr 2026 05:38:30 +1000 (AEST)", "from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from <qemu-devel-bounces@nongnu.org>)\n\tid 1wAYT2-0006Uq-Vg; Wed, 08 Apr 2026 15:21:37 -0400", "from eggs.gnu.org ([2001:470:142:3::10])\n by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from <peterx@redhat.com>) id 1wAY48-0006Kz-Hu\n for qemu-devel@nongnu.org; Wed, 08 Apr 2026 14:55:52 -0400", "from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from <peterx@redhat.com>) id 1wAWCF-00026l-Ru\n for qemu-devel@nongnu.org; Wed, 08 Apr 2026 12:56:11 -0400", "from mail-qt1-f197.google.com (mail-qt1-f197.google.com\n [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-602-Tsv1w9nHMv69WVGXS_ix7Q-1; Wed, 08 Apr 2026 12:56:03 -0400", "by mail-qt1-f197.google.com with SMTP id\n d75a77b69052e-50b4b81c632so3267431cf.1\n for <qemu-devel@nongnu.org>; Wed, 08 Apr 2026 09:56:03 -0700 (PDT)", "from x1.com ([142.189.10.167]) by smtp.gmail.com with ESMTPSA id\n d75a77b69052e-50d712c2617sm130491901cf.31.2026.04.08.09.56.00\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Wed, 08 Apr 2026 09:56:00 -0700 (PDT)" ], "DKIM-Signature": [ "v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1775667364;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding;\n bh=vPGwVEPm25tSQpDsHhbt2i6JsuPl+TWsf9MJqPYEbeQ=;\n b=d4Kocig//KHjnCTrQ1m8jBPJEbbk+QIKYEnKE2wVmTsPMit8itmQP3KVEjHr91mZ+U3jhz\n ni0XBSw/A8TaB5Me/2qb09iB9dIZhVd7nKFxeCNmQlRqqXOMjbADGMxFQOgtrFH+0RBN4m\n Vs9KWetm50tzrLP3eiM0bwJ9v5K4SrA=", "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=redhat.com; s=google; t=1775667362; x=1776272162; darn=nongnu.org;\n h=content-transfer-encoding:mime-version:message-id:date:subject:cc\n :to:from:from:to:cc:subject:date:message-id:reply-to;\n bh=vPGwVEPm25tSQpDsHhbt2i6JsuPl+TWsf9MJqPYEbeQ=;\n b=DTWm8ZmyyGgONwr6EOyoJqKWKPAt0hgkgxy/kjToZ4wUnw18T9RELkjVEK79wmC1tK\n qRohfgXmK8Ci3KGQFNFXZ+O6ZUZiEdC/Zto1On9v+3CRare12RsfYxASNd5uukuVaJKX\n 7cPcmc4VIHmrRfMXBzB7BxSfmF6etmZGFinf9U5ekSJa/AckOAYCcdjP9Qgnv3IsVWyo\n FVh7WU4DKelSB0WVaodoc8bJI0WjhqgCUYo5yio9Xr0bXiA9A2BnbM6LHBkYx7PxDZCN\n cpdAOgnpJEG1cW4tCmIEr7cOF0QSqTnTY6FSp65QWXAQSTUQkbSjSWSeQJvUtZhPsII/\n LuYg==" ], "X-MC-Unique": "Tsv1w9nHMv69WVGXS_ix7Q-1", "X-Mimecast-MFC-AGG-ID": "Tsv1w9nHMv69WVGXS_ix7Q_1775667363", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20251104; t=1775667362; x=1776272162;\n h=content-transfer-encoding:mime-version:message-id:date:subject:cc\n :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=vPGwVEPm25tSQpDsHhbt2i6JsuPl+TWsf9MJqPYEbeQ=;\n b=GF3ut/sOsez5uuOtF17HSmzxb2kQkRe/iJCtkTbEA1DoE3ZRbDgSozNxETRictcfto\n IEeiE2K2u9hcrL14K69Zhxm/tEh6ipIE8Oca7Q/LPpxJVyWbJFRbCMSMy160BBV/PtNK\n r766sFVG2vf8lpcsnk5dSWDecBl8dgocl4vEbyQu4rtbJIRFucjF2B3LkN2pw6MXBeIr\n abcgOPNAvPDjslQgWndIFpavPt3RgBmSVXiSG3POlyI/3yDOrl2eUS4OK3zux+WKpSEj\n 08bi1wD0f25g0yf92luHbAWZoP9d6TUK/ERsCMwFRn6vOLJOPXfVcaZzMzIXU3JUyaDS\n vNeA==", "X-Gm-Message-State": "AOJu0YzULKI8cmvXUKHh5W1n5QHlZrFFArEHGxTzi3DrEMuMShgHtX1j\n d1mRcUk1dCnOZgVlkkwyRx39ZPQvQvLNdffR+CimOKecVgjy6mUaEZG6TVf+jmX/+G63OADm6lU\n 72BUjC3/KzM+vilGfgSAj1JU3Ftk55lrHB3uAnB/sPODsIpRkPPHfogNr28Qyka7n55xf54qlRD\n U+6LVRgnmGUqI2MK/MwIvxwyFVRqDBvB4FsWxIkA==", "X-Gm-Gg": "AeBDietGyIbb2c6uuF+N3KWvB15EmwU7oPZqGwFq1YSTtfGPwiVkkMq0kCAOWFEqsHf\n stFHfG0ZAqqfO53Y4tvplzSC7YY8tzfGqWWw0LZCMpehCUD+lQzCeVX4S4o5riUDVALECZc/GfG\n z4sKjA2w8yDgMAXBAlSsdJIHiQsCGQwgwGW8aEuo0wu6+vemw2fnCNmdZa7UaKO2uPbk7WuLcB3\n TdXAM5IBumkjqxn8Rb9rxelFBaXojsAMVuXUEo08e4OPqywzszcSWYLIU55OSRUImQjHmFStDwQ\n QFWg51GjxbMxzE5HZjBSpMRNrb4qYnqVwF+TXMD7Hwap60nuVE6otx5qXRvFPB6BDzJ3CDj7lxZ\n fKZo3m2UU0RRWU5Lsq57bJRBaTrZ9zNMRBeOhYC9KD6I+", "X-Received": [ "by 2002:a05:622a:145:b0:50d:643b:378f with SMTP id\n d75a77b69052e-50d643b421amr330586841cf.57.1775667362229;\n Wed, 08 Apr 2026 09:56:02 -0700 (PDT)", "by 2002:a05:622a:145:b0:50d:643b:378f with SMTP id\n d75a77b69052e-50d643b421amr330586031cf.57.1775667361445;\n Wed, 08 Apr 2026 09:56:01 -0700 (PDT)" ], "From": "Peter Xu <peterx@redhat.com>", "To": "qemu-devel@nongnu.org", "Cc": "\"Maciej S . Szmigiero\" <mail@maciej.szmigiero.name>, =?utf-8?q?Daniel_P_?=\n\t=?utf-8?q?=2E_Berrang=C3=A9?= <berrange@redhat.com>,\n Zhiyi Guo <zhguo@redhat.com>, Juraj Marcin <jmarcin@redhat.com>,\n Peter Xu <peterx@redhat.com>, Prasad Pandit <ppandit@redhat.com>,\n Avihai Horon <avihaih@nvidia.com>, Kirti Wankhede <kwankhede@nvidia.com>,\n\t=?utf-8?q?C=C3=A9dric_Le_Goater?= <clg@redhat.com>,\n Fabiano Rosas <farosas@suse.de>, Joao Martins <joao.m.martins@oracle.com>,\n Markus Armbruster <armbru@redhat.com>, Alex Williamson <alex@shazbot.org>", "Subject": "[PATCH 00/14] migration/vfio: Fix a few issues on API misuse or\n statistic reports", "Date": "Wed, 8 Apr 2026 12:55:44 -0400", "Message-ID": "<20260408165559.157108-1-peterx@redhat.com>", "X-Mailer": "git-send-email 2.53.0", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=\"utf-8\"", "Content-Transfer-Encoding": "8bit", "Received-SPF": "pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com", "X-Spam_score_int": "-25", "X-Spam_score": "-2.6", "X-Spam_bar": "--", "X-Spam_report": "(-2.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.54,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001,\n RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,\n SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no", "X-Spam_action": "no action", "X-BeenThere": "qemu-devel@nongnu.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "qemu development <qemu-devel.nongnu.org>", "List-Unsubscribe": "<https://lists.nongnu.org/mailman/options/qemu-devel>,\n <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>", "List-Archive": "<https://lists.nongnu.org/archive/html/qemu-devel>", "List-Post": "<mailto:qemu-devel@nongnu.org>", "List-Help": "<mailto:qemu-devel-request@nongnu.org?subject=help>", "List-Subscribe": "<https://lists.nongnu.org/mailman/listinfo/qemu-devel>,\n <mailto:qemu-devel-request@nongnu.org?subject=subscribe>", "Errors-To": "qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org", "Sender": "qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org" }, "content": "CI: https://gitlab.com/peterx/qemu/-/pipelines/2437886506\nrfc: https://lore.kernel.org/r/20260319231302.123135-1-peterx@redhat.com\n\nThis is v1 of this series. I dropped RFC because I feel like I collected\nenough feedback on previous version on what is uncertain, meanwhile I also\nmanaged to borrow a system with nVidia RTX6000 2GB vGPU and tested it.\n\nThere're too many trivial things changed since RFC->v1 here, let me only\nmention what majorly has changed:\n\n- This version will assume both VFIO ioctls (reporting either precopy or\n stopcopy size) may report anything (say, garbage), and it shouldn't crash\n QEMU. It will affect what got reported with downtime or remaining data,\n but that's best effort so it's expected. With that in mind, I dropped\n patch 3 as Avihai suggested. IOW, I expect no concern on either overflow\n / underflow or atomicity on reading these values from the VFIO drivers.\n\n- The cached stopcopy_bytes for VFIO reflects always the total size\n (includes precopy sizes).\n\n- Introduced one new patch to report \"system-wide\" remaining data, which\n will start to include VFIO remaining device data. We can't squash that\n directly into \"ram\" section of query-migrate QMP results, so I introduced\n a new \"remaining\" field in query-migrate result for it.\n\n- One more patch \"migration: Make qemu_savevm_query_pending() available\n anytime\" trying to fix a very hard to hit race condition I found when\n testing against virtio-net-failover tests. I can only hit it if I run\n tens of concurrent tests, but it will be needed to fix a crash.\n\nOtherwise the major things should be kept almost as-is. I should also\naddressed all comments I received from rfc version. Please shoot if I\nmissed something.\n\nOverview\n========\n\nVFIO migration was merged quite a while, but we do still see things off\nhere and there. This series tries to address some of them, but only based\non my limited understandings.\n\nTwo major issues I wanted to resolve:\n\n(1) VFIO reports state_pending_{exact|estimate}() differently\n\nIt reports stop-only sizes in exact() only (which includes both precopy and\nstopcopy data), while in estimate() it only reports precopy data. This is\nviolating the API. It was done like it to trigger proper sync on the VFIO\nioctls only but it was only a workaround. This series should fix it by\nintroducing stopcopy size reporting facility for vmstate handlers.\n\n(2) expected_downtime / remaining doesn't take VFIO devices into account\n\nWhen query migration, QEMU reports one field called \"expected-downtime\".\nThe document was phrasing this almost from RAM perspective, but ideally it\nshould be about an estimated blackout window (in milliseconds) if we\nswitchover anytime, based on known information.\n\nThis didn't yet took VFIO into account, especially in the case of VFIO\ndevices that may contain a large amount of device states (like GPUs).\n\nFor problem (2), the use case should be that an mgmt app when migrating a\nVFIO GPU device needs to always adjust downtime for migration to converge,\nbecause when it's involved normal downtime like 300ms will normally not\nsuffice.\n\nNow the issue with that is the mgmt doesn't have a good way to know exactly\nhow well the precopy goes with the whole system and the GPU device.\n\nThe hope is fixed expected_downtime will provide one way for the mgmt app\nto have a reasonable hint for downtime to setup to converge a migration.\n\nMeanwhile, with a system-wise \"remaining\" field introduced, mgmt can query\nthis results at beginning of each iteration to know if a stall is\nhappening, IOW, if it's likely that this migration will not converge at\nall. When detected, mgmt can start to consider the expected_downtime value\nreported above for converging this migration. See more on testing below.\n\nTests\n=====\n\nTested this series with an assigned VFIO device GRID RTX6000-2B, FB memory\n2GB.\n\nThe test covers both correct reporting of system-wise remaining data (which\nused to only cover RAM), and the expected downtime. I verified that using\nthe expected downtime I can converge a VFIO migration immediately according\nto the value reported. Test process as below:\n\nStart the VM and kick off migration until it spins at the end, not\nconverging with default 300ms downtime. It's common for a 2GB vGPU device\ndue to both huge stopsize reported and dramally small mbps reported.\n\nAs a start, update avail-switchover (I chose 1GB over a real 10Gbps port):\n\nThis will stablize bandwidth.\n\nLibvirt's domjobinfo won't be able to see the real remaining data because\nlibvirt still doesn't support the new \"remaining\" field, however we can\nstill see expected_downtime will be reported correctly now (instead of\nreporting zero, before this patch applied):\n\nData remaining: 0.000 B\nMemory remaining: 0.000 B\nExpected downtime: 1910 ms\n\nIf we peek through QEMU monitor, we'll see with the change the system-wise\nremaining data to be 1.9GB (even if RAM keeps reporting 0), and expected\ndowntime keeps the same as what domjobinfo reports as 1.9 seconds:\n\nStatus: active\nTime (ms): total=336919, setup=10, exp_down=1910\nRemaining (bytes): 1.91 GiB\nRAM info:\n Throughput (Mbps): 460.09\n Sizes: pagesize=4 KiB, total=32 GiB\n Transfers: transferred=12.7 GiB, remain=0 B\n Channels: precopy=12.7 GiB, multifd=0 B, postcopy=0 B, vfio=0 B\n Page Types: normal=3306906, zero=7745576\n Page Rates (pps): transfer=14010, dirty=8039\n Others: dirty_syncs=247045\n\nIt means 1.91 seconds are required as lowest downtime per math.\n\nWe can try to set something lower than that, migration will not converge:\n\n...\n...\n\nThen if we update downtime_limit to be slightly larger than expected downtime:\n\nMigration will complete almost immediately.\n\nPeter Xu (14):\n migration: Fix low possibility downtime violation\n migration/qapi: Rename MigrationStats to MigrationRAMStats\n vfio/migration: Cache stop size in VFIOMigration\n migration/treewide: Merge @state_pending_{exact|estimate} APIs\n migration: Use the new save_query_pending() API directly\n migration: Introduce stopcopy_bytes in save_query_pending()\n vfio/migration: Fix incorrect reporting for VFIO pending data\n migration: Make qemu_savevm_query_pending() available anytime\n migration: Move iteration counter out of RAM\n migration: Introduce a helper to return switchover bw estimate\n migration: Calculate expected downtime on demand\n migration: Fix calculation of expected_downtime to take VFIO info\n migration/qapi: Introduce system-wise \"remaining\" reports\n migration/qapi: Update unit for avail-switchover-bandwidth\n\n docs/about/removed-features.rst | 2 +-\n docs/devel/migration/main.rst | 9 +-\n docs/devel/migration/vfio.rst | 9 +-\n qapi/migration.json | 32 +++---\n hw/vfio/vfio-migration-internal.h | 8 ++\n include/migration/register.h | 59 ++++------\n migration/migration-stats.h | 13 ++-\n migration/migration.h | 10 +-\n migration/savevm.h | 9 +-\n hw/s390x/s390-stattrib.c | 9 +-\n hw/vfio/migration.c | 92 +++++++++-------\n migration/block-dirty-bitmap.c | 10 +-\n migration/migration-hmp-cmds.c | 5 +\n migration/migration.c | 172 +++++++++++++++++++++---------\n migration/ram.c | 40 ++-----\n migration/savevm.c | 73 +++++++------\n hw/vfio/trace-events | 3 +-\n migration/trace-events | 3 +-\n 18 files changed, 313 insertions(+), 245 deletions(-)" }