From patchwork Thu Aug 9 21:37:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 955854 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41mj1w3zbYz9s3Z for ; Fri, 10 Aug 2018 08:03:32 +1000 (AEST) Received: from localhost ([::1]:53053 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnt1k-00034x-K7 for incoming@patchwork.ozlabs.org; Thu, 09 Aug 2018 18:03:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43650) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnsdI-0004S3-48 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 17:38:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnsdG-00042H-Sm for qemu-devel@nongnu.org; Thu, 09 Aug 2018 17:38:12 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:38094 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnsdA-0003WO-0s; Thu, 09 Aug 2018 17:38:04 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8727F40216F3; Thu, 9 Aug 2018 21:38:03 +0000 (UTC) Received: from localhost (ovpn-204-42.brq.redhat.com [10.40.204.42]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2BA462026D66; Thu, 9 Aug 2018 21:38:03 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Thu, 9 Aug 2018 23:37:50 +0200 Message-Id: <20180809213801.15098-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Thu, 09 Aug 2018 21:38:03 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Thu, 09 Aug 2018 21:38:03 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'mreitz@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 00/11] block: Deal with filters X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Note 1: This series depends on v10 of my “block: Fix some filename generation issues” series. Based-on: <20180809213528.14738-1-mreitz@redhat.com> Note 2: This is technically the first part of my active mirror followup. But just very technically. I noticed that that followup started to consist of two parts, namely (A) fix filtery things in the block layer, and (B) fix active mirror. So I decided to split it. This is part A. Part B comes later. When we introduced filters, we did it a bit casually. Sure, we talked a lot about them before, but that was mostly discussion about where implicit filters should be added to the graph (note that we currently only have two implicit filters, those being mirror and commit). But in the end, we really just designated some drivers filters (Quorum, blkdebug, etc.) and added some specifically (throttle, COR), without really looking through the block layer to see where issues might occur. It turns out vast areas of the block layer just don't know about filters and cannot really handle them. Many cases will work in practice, in others, well, too bad, you cannot use some feature because some part deep inside the block layer looks at your filters and thinks they are format nodes. This series sets out to correct a bit of that. I lost my head many times and I'm sure this series is incomplete in many ways, but it really doesn't do any good if it sits on my disk any longer, it needs to go out now. The most important patches of this series are patches 3 and 4. These introduce functions to encapsulate bs->backing and bs->file accesses. Because sometimes, bs->backing means COW, sometimes it means filtered node. And sometimes, bs->file means metadata storage, and sometimes it means filtered node. With this functions, it's always clear what the caller wants, and it will always get what it wants. Besides that, patch 3 introduces functions to skip filters which may be used by parts of the block layer that just don't care about them. Secondly, the restraints put on mirror's @replaces parameter are revisited and fixed. Thirdly, BDS.backing_file is changed to be constant. I don't quite know why we modify it whenever we change a BDS's backing file, but that's definitely not quite right. This fixes things like being able to perform a commit on a file (using relative filenames) in a directory that's not qemu's CWD. Finally, a number of tests are added. There are probably many things that are worthy of discussion, of which only some come to my head, e.g.: - In which cases do we want to skip filters, in which cases do we want to skip implicit filters? My approach was to basically never skip explicitly added filters, except when it's about finding a file in some tree (e.g. in a backing chain). Maybe there are cases where you think we should skip even explicitly added filters. - I made interesting decisions like “When you mirror from a node, we should indeed mirror from that node, but when replacing it, we should skip leave all implicit filters on top intact.” You may disagree with that. (My reasoning here is that users aren't supposed to know about implicit filters, and therefore, they should not intend to remove them. Also, mirror accepts only root nodes as the source, so you cannot really specify the node below the implicit filters. But you can use @replaces to drop the implicit filters, if you know they are there.) Max Reitz (11): block: Mark commit and mirror as filter drivers blockdev: Check @replaces in blockdev_mirror_common block: Filtered children access functions block: Storage child access function block: Fix check_to_replace_node() iotests: Add tests for mirror @replaces loops block: Leave BDS.backing_file constant iotests: Add filter commit test cases iotests: Add filter mirror test cases iotests: Add test for commit in sub directory iotests: Test committing to overridden backing qapi/block-core.json | 4 + include/block/block.h | 2 + include/block/block_int.h | 53 +++++- block.c | 338 ++++++++++++++++++++++++++++----- block/backup.c | 8 +- block/block-backend.c | 16 +- block/commit.c | 38 ++-- block/io.c | 47 +++-- block/mirror.c | 39 ++-- block/qapi.c | 38 ++-- block/snapshot.c | 40 ++-- block/stream.c | 15 +- blockdev.c | 167 ++++++++++++---- migration/block-dirty-bitmap.c | 4 +- nbd/server.c | 8 +- qemu-img.c | 24 ++- tests/qemu-iotests/020 | 36 ++++ tests/qemu-iotests/020.out | 10 + tests/qemu-iotests/040 | 191 +++++++++++++++++++ tests/qemu-iotests/040.out | 4 +- tests/qemu-iotests/041 | 270 +++++++++++++++++++++++++- tests/qemu-iotests/041.out | 4 +- tests/qemu-iotests/191.out | 1 - 23 files changed, 1133 insertions(+), 224 deletions(-)