From patchwork Thu Jun 20 01:03:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1119084 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45TkgM4GhCz9s5c for ; Thu, 20 Jun 2019 11:26:39 +1000 (AEST) Received: from localhost ([::1]:43000 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdlqX-0004lJ-F3 for incoming@patchwork.ozlabs.org; Wed, 19 Jun 2019 21:26:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40725) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdleI-0002G1-Nt for qemu-devel@nongnu.org; Wed, 19 Jun 2019 21:14:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdlUk-0004kf-FS for qemu-devel@nongnu.org; Wed, 19 Jun 2019 21:04:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59674) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdlUj-0004hf-2m; Wed, 19 Jun 2019 21:04:05 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4292E3162902; Thu, 20 Jun 2019 01:04:03 +0000 (UTC) Received: from probe.bos.redhat.com (dhcp-17-164.bos.redhat.com [10.18.17.164]) by smtp.corp.redhat.com (Postfix) with ESMTP id 650C61001DC3; Thu, 20 Jun 2019 01:03:57 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Wed, 19 Jun 2019 21:03:44 -0400 Message-Id: <20190620010356.19164-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Thu, 20 Jun 2019 01:04:03 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 00/12] bitmaps: introduce 'bitmap' sync mode X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , vsementsov@virtuozzo.com, Wen Congyang , Xie Changlong , Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This series adds a new "BITMAP" sync mode that is meant to replace the existing "INCREMENTAL" sync mode. This mode can be changed by issuing any of three bitmap sync modes; passed as arguments to the job. The three bitmap sync modes are: - CONDITIONAL: This is an alias for the old incremental mode. The bitmap is conditionally synchronized based on the return code of the job upon completion. - NEVER: This is, effectively, the differential backup mode. It never clears the bitmap, as the name suggests. - ALWAYS: Here is the new, exciting thing. The bitmap is always synchronized, even on failure. On success, this is identical to incremental, but on failure it clears only the bits that were copied successfully. This can be used to "resume" incremental backups from later points in times. I wrote this series by accident on my way to implement incremental mode for mirror, but this happened first -- the problem is that Mirror mode uses its existing modes in a very particular way; and this was the best way to add bitmap support into the mirror job properly. In this series: patches 1-2: Adds CONDITIONAL patch 3: Adds NEVER Patches 4-7: Adds ALWAYS. Patches 8-11: Adds iotest 257 Patch 12: Minor permission loosening for RO bitmaps. Next up: - Add these modes to Mirror. (Done*, but need tests.) - Allow the use of bitmaps and bitmap sync modes with non-BITMAP modes; This will allow for resumable/re-tryable full backups. John Snow (12): qapi: add BitmapSyncMode enum block/backup: Add mirror sync mode 'bitmap' block/backup: add 'never' policy to bitmap sync mode hbitmap: Fix merge when b is empty, and result is not an alias of a hbitmap: enable merging across granularities block/dirty-bitmap: add bdrv_dirty_bitmap_claim block/backup: add 'always' bitmap sync policy iotests: add testing shim for script-style python tests iotests: teach run_job to cancel pending jobs iotests: teach FilePath to produce multiple paths iotests: add test 257 for bitmap-mode backups block/backup: loosen restriction on readonly bitmaps qapi/block-core.json | 51 +- include/block/block_int.h | 7 +- include/qemu/hbitmap.h | 8 + block/backup.c | 52 +- block/dirty-bitmap.c | 14 + block/mirror.c | 6 +- block/replication.c | 2 +- blockdev.c | 12 +- util/hbitmap.c | 36 +- tests/qemu-iotests/257 | 412 ++++++ tests/qemu-iotests/257.out | 2199 +++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 83 +- 13 files changed, 2830 insertions(+), 53 deletions(-) create mode 100755 tests/qemu-iotests/257 create mode 100644 tests/qemu-iotests/257.out