Cover Letter Detail
Show a cover letter.
GET /api/covers/813553/?format=api
{ "id": 813553, "url": "http://patchwork.ozlabs.org/api/covers/813553/?format=api", "web_url": "http://patchwork.ozlabs.org/project/qemu-devel/cover/20170913181910.29688-1-mreitz@redhat.com/", "project": { "id": 14, "url": "http://patchwork.ozlabs.org/api/projects/14/?format=api", "name": "QEMU Development", "link_name": "qemu-devel", "list_id": "qemu-devel.nongnu.org", "list_email": "qemu-devel@nongnu.org", "web_url": "", "scm_url": "", "webscm_url": "", "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20170913181910.29688-1-mreitz@redhat.com>", "list_archive_url": null, "date": "2017-09-13T18:18:52", "name": "[00/18] block/mirror: Add active-sync mirroring", "submitter": { "id": 36836, "url": "http://patchwork.ozlabs.org/api/people/36836/?format=api", "name": "Max Reitz", "email": "mreitz@redhat.com" }, "mbox": "http://patchwork.ozlabs.org/project/qemu-devel/cover/20170913181910.29688-1-mreitz@redhat.com/mbox/", "series": [ { "id": 2960, "url": "http://patchwork.ozlabs.org/api/series/2960/?format=api", "web_url": "http://patchwork.ozlabs.org/project/qemu-devel/list/?series=2960", "date": "2017-09-13T18:18:52", "name": "block/mirror: Add active-sync mirroring", "version": 1, "mbox": "http://patchwork.ozlabs.org/series/2960/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/covers/813553/comments/", "headers": { "Return-Path": "<qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>", "X-Original-To": "incoming@patchwork.ozlabs.org", "Delivered-To": "patchwork-incoming@bilbo.ozlabs.org", "Authentication-Results": [ "ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=<UNKNOWN>)", "ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com", "ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=mreitz@redhat.com" ], "Received": [ "from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xsqhf6vWHz9sNV\n\tfor <incoming@patchwork.ozlabs.org>;\n\tThu, 14 Sep 2017 04:20:18 +1000 (AEST)", "from localhost ([::1]:43981 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t<qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)\n\tid 1dsCGm-0007b8-JN\n\tfor incoming@patchwork.ozlabs.org; Wed, 13 Sep 2017 14:20:16 -0400", "from eggs.gnu.org ([2001:4830:134:3::10]:36357)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from <mreitz@redhat.com>) id 1dsCG0-0007Yc-Oq\n\tfor qemu-devel@nongnu.org; Wed, 13 Sep 2017 14:19:30 -0400", "from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from <mreitz@redhat.com>) id 1dsCFz-0004Zh-6v\n\tfor qemu-devel@nongnu.org; Wed, 13 Sep 2017 14:19:28 -0400", "from mx1.redhat.com ([209.132.183.28]:42576)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from <mreitz@redhat.com>)\n\tid 1dsCFs-0004Ts-Dw; Wed, 13 Sep 2017 14:19:20 -0400", "from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id A55F483F3D;\n\tWed, 13 Sep 2017 18:19:18 +0000 (UTC)", "from localhost (ovpn-204-23.brq.redhat.com [10.40.204.23])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 5D8C05D6AE;\n\tWed, 13 Sep 2017 18:19:12 +0000 (UTC)" ], "DMARC-Filter": "OpenDMARC Filter v1.3.2 mx1.redhat.com A55F483F3D", "From": "Max Reitz <mreitz@redhat.com>", "To": "qemu-block@nongnu.org", "Date": "Wed, 13 Sep 2017 20:18:52 +0200", "Message-Id": "<20170913181910.29688-1-mreitz@redhat.com>", "X-Scanned-By": "MIMEDefang 2.79 on 10.5.11.15", "X-Greylist": "Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tWed, 13 Sep 2017 18:19:18 +0000 (UTC)", "X-detected-operating-system": "by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]", "X-Received-From": "209.132.183.28", "Subject": "[Qemu-devel] [PATCH 00/18] block/mirror: Add active-sync mirroring", "X-BeenThere": "qemu-devel@nongnu.org", "X-Mailman-Version": "2.1.21", "Precedence": "list", "List-Id": "<qemu-devel.nongnu.org>", "List-Unsubscribe": "<https://lists.nongnu.org/mailman/options/qemu-devel>,\n\t<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>", "List-Archive": "<http://lists.nongnu.org/archive/html/qemu-devel/>", "List-Post": "<mailto:qemu-devel@nongnu.org>", "List-Help": "<mailto:qemu-devel-request@nongnu.org?subject=help>", "List-Subscribe": "<https://lists.nongnu.org/mailman/listinfo/qemu-devel>,\n\t<mailto:qemu-devel-request@nongnu.org?subject=subscribe>", "Cc": "Kevin Wolf <kwolf@redhat.com>, Fam Zheng <famz@redhat.com>,\n\tqemu-devel@nongnu.org, Max Reitz <mreitz@redhat.com>,\n\tStefan Hajnoczi <stefanha@redhat.com>, John Snow <jsnow@redhat.com>", "Errors-To": "qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org", "Sender": "\"Qemu-devel\"\n\t<qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>" }, "content": "This series implements an active and synchronous mirroring mode.\n\nCurrently, the mirror block job is passive an asynchronous: Depending on\nyour start conditions, some part of the source disk starts as \"dirty\".\nThen, the block job will (as a background operation) continuously copy\ndirty parts to the target disk until all of the source disk is clean.\nIn the meantime, any write to the source disk dirties the affected area.\n\nOne effect of this operational mode is that the job may never converge:\nIf the writes to the source happen faster than the block job copies data\nto the target, the job can never finish.\n\nWhen the active mode implemented in this series is enabled, every write\nrequest to the source will automatically trigger a synchronous write to\nthe target right afterwards. Therefore, the source can never get dirty\nfaster than data is copied to the target. Most importantly, once source\nand target are in sync (BLOCK_JOB_READY is emitted), they will not\ndiverge (unless e.g. an I/O error occurs).\n\nActive mirroring also improves on a second issue of the passive mode: We\ndo not have to read data from the source in order to write it to the\ntarget. When new data is written to the source in active mode, it is\nautomatically mirrored to the target, which saves us the superfluous\nread from the source.\n(Optionally, one can choose to also mirror data read from the source.\nThis does not necessarily help with convergence, but it saves an extra\nread operation (in exchange for slower read access to the source because\nthis mirroring is implemented synchronously).)\n\n\nThere may be a couple of things to do on top of this series:\n- Allow switching between active and passive mode at runtime: This\n should not be too difficult to implement, the main question is how to\n expose it to the user.\n (I seem to recall we wanted some form of block-job-set-option\n command...?)\n\n- Implement an asynchronous active mode: May be detrimental when it\n comes to convergence, but it might be nice to have anyway. May or may\n not be complicated to implement.\n\n- Make the target a BdrvChild of the mirror BDS: This series does some\n work to make the mirror BDS a more integral part of the block job (see\n below for more). One of the things I wanted to do is to make both the\n source and the target plain children of that BDS, and I did have\n patches to do this. However, at some point continuing to do this for\n the target seemed rather difficult, and also a bit pointless, so I\n decided to keep it for later.\n (To be specific, that \"some point\" was exactly when I tried to rebase\n onto 045a2f8254c.)\n\n\n=== Structure of this series ===\n\nThe first half (up until patch 10) restructures parts of the mirror\nblock job:\n\n- Patches 4/5:\n The job is converted to use coroutines instead of AIO.\n (because this is probably where we want to go, and also because active\n mirroring will need to wait on conflicting in-flight operations, and\n I really don't want to wait on in-flight AIO requests)\n\n This is done primarily by patch 5, with patch 4 being necessary\n beforehand.\n\n- Patches 6/7:\n Every in-flight operation gets a CoQueue so it can be waited on\n (because this allows active mirroring operations to wait for\n conflicting writes)\n\n This is started by patch 6, and with patch 7, every bit in the\n in-flight bitmap has at least one corresponding operation in the\n MirrorBlockJob.ops_in_flight list that can be waited on.\n\n- Patches 1/2/3/8/9/10:\n The source node is now no longer used through a BlockBackend (patch 8)\n and also it is now attached to the mirror BDS as the \"file\" child\n instead of the \"backing\" child (patch 10).\n This is mostly because I'd personally like the mirror BDS to be a real\n filter BDS instead of some technicality that needs to be there to\n solve op blocker issues.\n\n Patches 3 and 9 are necessary for patch 10.\n\n Patches 1 and 2 were necessary for this when I decided to include\n another patch to make the target node an immediate child of the mirror\n BDS, too. However, as I wrote above, I later decided to put this idea\n off until later, and as long as the mirror BDS only has a single\n child, those patches are not strictly necessary.\n However, I think that those patches are good to have anyway, so I\n decided to keep them.\n\n\nThe second half (patches 11 to 18) implement active mirroring:\n- Patch 11 is required by patch 12. This in turn is required by the\n active-sync mode when mirroring data read from the source to the\n target, because that functionality needs to be able to find all the\n parts of the data read which are actually dirty so we don't copy clean\n data.\n\n- Patches 13 and 14 prepare for the job for active operations.\n\n- Patch 15 implements active mirroring.\n\n- Patch 16 allows it to be used (by adding a parameter to\n blockdev-mirror and drive-mirror).\n\n- Patch 18 adds an iotest which relies on functionality introduced by\n patch 17.\n\n\n\nMax Reitz (18):\n block: Add BdrvDeletedStatus\n block: BDS deletion during bdrv_drain_recurse\n blockjob: Make drained_{begin,end} public\n block/mirror: Pull out mirror_perform()\n block/mirror: Convert to coroutines\n block/mirror: Use CoQueue to wait on in-flight ops\n block/mirror: Wait for in-flight op conflicts\n block/mirror: Use source as a BdrvChild\n block: Generalize should_update_child() rule\n block/mirror: Make source the file child\n hbitmap: Add @advance param to hbitmap_iter_next()\n block/dirty-bitmap: Add bdrv_dirty_iter_next_area\n block/mirror: Keep write perm for pending writes\n block/mirror: Distinguish active from passive ops\n block/mirror: Add active mirroring\n block/mirror: Add copy mode QAPI interface\n qemu-io: Add background write\n iotests: Add test for active mirroring\n\n qapi/block-core.json | 34 ++-\n include/block/block_int.h | 18 +-\n include/block/blockjob.h | 15 +\n include/block/dirty-bitmap.h | 2 +\n include/qemu/hbitmap.h | 4 +-\n block.c | 50 +++-\n block/dirty-bitmap.c | 54 +++-\n block/io.c | 72 +++--\n block/mirror.c | 633 +++++++++++++++++++++++++++++++++----------\n block/qapi.c | 25 +-\n blockdev.c | 9 +-\n blockjob.c | 20 +-\n qemu-io-cmds.c | 83 +++++-\n tests/test-hbitmap.c | 26 +-\n util/hbitmap.c | 10 +-\n tests/qemu-iotests/141.out | 4 +-\n tests/qemu-iotests/151 | 111 ++++++++\n tests/qemu-iotests/151.out | 5 +\n tests/qemu-iotests/group | 1 +\n 19 files changed, 964 insertions(+), 212 deletions(-)\n create mode 100755 tests/qemu-iotests/151\n create mode 100644 tests/qemu-iotests/151.out" }