From patchwork Fri Mar 6 09:41:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 1250158 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48YjMS2wstz9s3x for ; Fri, 6 Mar 2020 20:42:26 +1100 (AEDT) Received: from localhost ([::1]:33532 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jA9Us-0006Dg-PC for incoming@patchwork.ozlabs.org; Fri, 06 Mar 2020 04:42:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:54847) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jA9UP-0006DZ-FE for qemu-devel@nongnu.org; Fri, 06 Mar 2020 04:41:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jA9UN-0003y9-AI for qemu-devel@nongnu.org; Fri, 06 Mar 2020 04:41:52 -0500 Received: from mga05.intel.com ([192.55.52.43]:25321) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jA9UN-0003oL-0R for qemu-devel@nongnu.org; Fri, 06 Mar 2020 04:41:51 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2020 01:41:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,521,1574150400"; d="scan'208";a="387776094" Received: from joy-desktop.sh.intel.com ([10.239.13.12]) by orsmga004.jf.intel.com with ESMTP; 06 Mar 2020 01:41:43 -0800 From: Yan Zhao To: qemu-devel@nongnu.org Subject: [PATCH] vfio/migration: fix dirty pages lost bug for ram beyond 3G Date: Fri, 6 Mar 2020 17:41:29 +0800 Message-Id: <1583487689-9813-1-git-send-email-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.7.4 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.43 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Yan Zhao , alex.williamson@redhat.com, intel-gvt-dev@lists.freedesktop.org, kwankhede@nvidia.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" the start address passing to cpu_physical_memory_set_dirty_range() and cpu_physical_memory_set_dirty_lebitmap() is the address within the ram block plus ram block offset. it's safe to set this start address to gpa if total memory is less than 3G, as ram block offset for pc.ram is 0. But if memory size is beyond 3G, gpa is not equal to the offset in its ram block. e.g. for gpa 0x100000000, its offset is actually 0xc0000000. and for gpa 0x42f500000, its offset should be 0x3EF500000. This fix is based on Kirti's VFIO live migration patch set v8. https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg05542.html (for PATCH v8 11/13 https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg05553.html and PATCH v8 12/13 https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg05554.html } The idea behind it should also be applied to other VFIO live migraiton patch series below: https://lists.gnu.org/archive/html/qemu-devel/2019-11/msg01763.html (Kirti v9) https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg04912.html (Yan) https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg01138.html (Yulei RFC v4) https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg05568.html (Yulei RFC) Cc: Kevin Tian Signed-off-by: Yan Zhao --- hw/vfio/common.c | 5 ++++- hw/vfio/migration.c | 8 +++++--- include/hw/vfio/vfio-common.h | 3 ++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 6f36b02..ba1a8ef 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -799,6 +799,7 @@ static void vfio_listerner_log_sync(MemoryListener *listener, MemoryRegionSection *section) { uint64_t start_addr, size, pfn_count; + uint64_t block_start; VFIOGroup *group; VFIODevice *vbasedev; @@ -819,11 +820,13 @@ static void vfio_listerner_log_sync(MemoryListener *listener, start_addr = TARGET_PAGE_ALIGN(section->offset_within_address_space); size = int128_get64(section->size); pfn_count = size >> TARGET_PAGE_BITS; + block_start = TARGET_PAGE_ALIGN(section->offset_within_region + + memory_region_get_ram_addr(section->mr)); QLIST_FOREACH(group, &vfio_group_list, next) { QLIST_FOREACH(vbasedev, &group->device_list, next) { vfio_get_dirty_page_list(vbasedev, start_addr >> TARGET_PAGE_BITS, - pfn_count, TARGET_PAGE_SIZE); + pfn_count, TARGET_PAGE_SIZE, block_start); } } } diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 640bea1..a19b957 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -279,7 +279,8 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) void vfio_get_dirty_page_list(VFIODevice *vbasedev, uint64_t start_pfn, uint64_t pfn_count, - uint64_t page_size) + uint64_t page_size, + uint64_t block_start) { VFIOMigration *migration = vbasedev->migration; VFIORegion *region = &migration->region; @@ -293,6 +294,7 @@ void vfio_get_dirty_page_list(VFIODevice *vbasedev, while (total_pfns > 0) { uint64_t bitmap_size, data_offset = 0; uint64_t start = start_pfn + count; + uint64_t block_start_seg = block_start + count * page_size; void *buf = NULL; bool buffer_mmaped = false; @@ -341,7 +343,7 @@ void vfio_get_dirty_page_list(VFIODevice *vbasedev, break; } else if (copied_pfns == VFIO_DEVICE_DIRTY_PFNS_ALL) { /* Mark all pages dirty for this range */ - cpu_physical_memory_set_dirty_range(start * page_size, + cpu_physical_memory_set_dirty_range(block_start_seg, total_pfns * page_size, DIRTY_MEMORY_MIGRATION); break; @@ -382,7 +384,7 @@ void vfio_get_dirty_page_list(VFIODevice *vbasedev, } cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, - start * page_size, + block_start_seg, copied_pfns); count += copied_pfns; total_pfns -= copied_pfns; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 41ff5eb..6d868fa 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -220,6 +220,7 @@ int vfio_spapr_remove_window(VFIOContainer *container, int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); void vfio_migration_finalize(VFIODevice *vbasedev); void vfio_get_dirty_page_list(VFIODevice *vbasedev, uint64_t start_pfn, - uint64_t pfn_count, uint64_t page_size); + uint64_t pfn_count, uint64_t page_size, + uint64_t block_start); #endif /* HW_VFIO_VFIO_COMMON_H */