From patchwork Mon Apr 29 11:51:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928960 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=HjcOJI6s; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShYp6Pkbz23tW for ; Mon, 29 Apr 2024 21:53:49 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYf-0008Iz-DO; Mon, 29 Apr 2024 07:52:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYX-0008EX-N5; Mon, 29 Apr 2024 07:52:25 -0400 Received: from forwardcorp1b.mail.yandex.net ([2a02:6b8:c02:900:1:45:d181:df01]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYQ-0001nu-9l; Mon, 29 Apr 2024 07:52:25 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTPS id F12D360BB5; Mon, 29 Apr 2024 14:52:11 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-D1OHE8dv; Mon, 29 Apr 2024 14:52:11 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391531; bh=eEOyCVZ3JXSEur7Nadwzc+kvSjKm4xOx7LeBJKl+UEo=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=HjcOJI6sacTnl6OTDlsVzkh2/XbZs2phgcGFcx6rWZncAdsL/jFnDiK/8FG4PLgoQ 5Av06adQDmNUGa5QCXLxKL+6vTyelXuK5YYGRM6mB4h46qUZbbpC1JLLbkp796td38 Patz1IOLC26+uHwxCHKNzgKEZBKD67ulNAw235p4= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Alexander Ivanov , John Snow , Kevin Wolf , Hanna Reitz Subject: [PULL 1/6] blockcommit: Reopen base image as RO after abort Date: Mon, 29 Apr 2024 14:51:52 +0300 Message-Id: <20240429115157.2260885-2-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=2a02:6b8:c02:900:1:45:d181:df01; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Alexander Ivanov If a blockcommit is aborted the base image remains in RW mode, that leads to a fail of subsequent live migration. How to reproduce: $ virsh snapshot-create-as vm snp1 --disk-only *** write something to the disk inside the guest *** $ virsh blockcommit vm vda --active --shallow && virsh blockjob vm vda --abort $ lsof /vzt/vm.qcow2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME qemu-syst 433203 root 45u REG 253,0 1724776448 133 /vzt/vm.qcow2 $ cat /proc/433203/fdinfo/45 pos: 0 flags: 02140002 <==== The last 2 means RW mode If the base image is in RW mode at the end of blockcommit and was in RO mode before blockcommit, reopen the base BDS in RO. Signed-off-by: Alexander Ivanov Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20240404091136.129811-1-alexander.ivanov@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/mirror.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index 1bdce3b657..61f0a717b7 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -93,6 +93,7 @@ typedef struct MirrorBlockJob { int64_t active_write_bytes_in_flight; bool prepared; bool in_drain; + bool base_ro; } MirrorBlockJob; typedef struct MirrorBDSOpaque { @@ -794,6 +795,10 @@ static int mirror_exit_common(Job *job) bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort); bdrv_graph_wrunlock(); + if (abort && s->base_ro && !bdrv_is_read_only(target_bs)) { + bdrv_reopen_set_read_only(target_bs, true, NULL); + } + bdrv_drained_end(target_bs); bdrv_unref(target_bs); @@ -1717,6 +1722,7 @@ static BlockJob *mirror_start_job( bool is_none_mode, BlockDriverState *base, bool auto_complete, const char *filter_node_name, bool is_mirror, MirrorCopyMode copy_mode, + bool base_ro, Error **errp) { MirrorBlockJob *s; @@ -1800,6 +1806,7 @@ static BlockJob *mirror_start_job( bdrv_unref(mirror_top_bs); s->mirror_top_bs = mirror_top_bs; + s->base_ro = base_ro; /* No resize for the target either; while the mirror is still running, a * consistent read isn't necessarily possible. We could possibly allow @@ -2029,7 +2036,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs, speed, granularity, buf_size, backing_mode, zero_target, on_source_error, on_target_error, unmap, NULL, NULL, &mirror_job_driver, is_none_mode, base, false, - filter_node_name, true, copy_mode, errp); + filter_node_name, true, copy_mode, false, errp); } BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs, @@ -2058,7 +2065,7 @@ BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs, on_error, on_error, true, cb, opaque, &commit_active_job_driver, false, base, auto_complete, filter_node_name, false, MIRROR_COPY_MODE_BACKGROUND, - errp); + base_read_only, errp); if (!job) { goto error_restore_flags; } From patchwork Mon Apr 29 11:51:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928961 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=qplkCrgB; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShYp5PV0z1ymX for ; Mon, 29 Apr 2024 21:53:49 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYb-0008HJ-9c; Mon, 29 Apr 2024 07:52:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYX-0008EI-Kh; Mon, 29 Apr 2024 07:52:25 -0400 Received: from forwardcorp1c.mail.yandex.net ([178.154.239.200]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYQ-0001ny-At; Mon, 29 Apr 2024 07:52:24 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1c.mail.yandex.net (Yandex) with ESMTPS id E2F8A60B5D; Mon, 29 Apr 2024 14:52:12 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-AeJvAd0z; Mon, 29 Apr 2024 14:52:12 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391532; bh=I1mOWou9kbYL8/uNkdlX0JTVSXvIsMtggeYmfxIr4Kk=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=qplkCrgBfLpyVoGi9yhMEqMSIrfitGrvGOD6s10m+OkB9KnkYN8JsZ2BCRkoIEqFh 3HcrxNIJJwsbPMTB704i0gQO/eSAwZhbgZQUFmUnI228xjh/kQD/gDMhc7q+tmsplH OKoDJDDvTQk8Kdae+zvzoHuvtlSOC+lo6/ARZiSU= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Fiona Ebner , John Snow , Kevin Wolf , Hanna Reitz Subject: [PULL 2/6] block/copy-before-write: fix permission Date: Mon, 29 Apr 2024 14:51:53 +0300 Message-Id: <20240429115157.2260885-3-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=178.154.239.200; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1c.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org In case when source node does not have any parents, the condition still works as required: backup job do create the parent by block_job_create -> block_job_add_bdrv -> bdrv_root_attach_child Still, in this case checking @perm variable doesn't work, as backup job creates the root blk with empty permissions (as it rely on CBW filter to require correct permissions and don't want to create extra conflicts). So, we should not check @perm. The hack may be dropped entirely when transactional insertion of filter (when we don't try to recalculate permissions in intermediate state, when filter does conflict with original parent of the source node) merged (old big series "[PATCH v5 00/45] Transactional block-graph modifying API"[1] and it's current in-flight part is "[PATCH v8 0/7] blockdev-replace"[2]) [1] https://patchew.org/QEMU/20220330212902.590099-1-vsementsov@openvz.org/ [2] https://patchew.org/QEMU/20231017184444.932733-1-vsementsov@yandex-team.ru/ Signed-off-by: Vladimir Sementsov-Ogievskiy Tested-by: Fiona Ebner Message-Id: <20240313152822.626493-2-vsementsov@yandex-team.ru> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/block/copy-before-write.c b/block/copy-before-write.c index 8aba27a71d..3e3af30c08 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -364,9 +364,13 @@ cbw_child_perm(BlockDriverState *bs, BdrvChild *c, BdrvChildRole role, perm, shared, nperm, nshared); if (!QLIST_EMPTY(&bs->parents)) { - if (perm & BLK_PERM_WRITE) { - *nperm = *nperm | BLK_PERM_CONSISTENT_READ; - } + /* + * Note, that source child may be shared with backup job. Backup job + * does create own blk parent on copy-before-write node, so this + * works even if source node does not have any parents before backup + * start + */ + *nperm = *nperm | BLK_PERM_CONSISTENT_READ; *nshared &= ~(BLK_PERM_WRITE | BLK_PERM_RESIZE); } } From patchwork Mon Apr 29 11:51:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928967 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=e76JOpLT; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShZm4jLMz1ymX for ; Mon, 29 Apr 2024 21:54:40 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYa-0008GU-4x; Mon, 29 Apr 2024 07:52:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYV-0008D9-K5; Mon, 29 Apr 2024 07:52:24 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYQ-0001oB-A9; Mon, 29 Apr 2024 07:52:22 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTPS id 9B14960C24; Mon, 29 Apr 2024 14:52:13 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-87SfCP6k; Mon, 29 Apr 2024 14:52:13 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391533; bh=M9bTcVoVVYx1vTw25y9U4yg7TuT7NkouXUR1hv8vs2M=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=e76JOpLTsC6G/AsTJGKb5fciJVVT8WpQ/bKbepJp6ve0lNFWf3kzT3Kc5R2qg50ZH R+uU5EqtI92aAUQPN3j0nqZio2aCHx1bg08szwfb+1NeFJezh07/pPpmdH+EmZxgaQ v04Fq5mynojGtA5cbMEFWcZDDX/IULJry6TKO8TI= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Fiona Ebner , John Snow , Kevin Wolf , Hanna Reitz Subject: [PULL 3/6] block/copy-before-write: support unligned snapshot-discard Date: Mon, 29 Apr 2024 14:51:54 +0300 Message-Id: <20240429115157.2260885-4-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=178.154.239.136; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org First thing that crashes on unligned access here is bdrv_reset_dirty_bitmap(). Correct way is to align-down the snapshot-discard request. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fiona Ebner Tested-by: Fiona Ebner Message-Id: <20240313152822.626493-3-vsementsov@yandex-team.ru> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/block/copy-before-write.c b/block/copy-before-write.c index 3e3af30c08..6d89af0b29 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -325,14 +325,24 @@ static int coroutine_fn GRAPH_RDLOCK cbw_co_pdiscard_snapshot(BlockDriverState *bs, int64_t offset, int64_t bytes) { BDRVCopyBeforeWriteState *s = bs->opaque; + uint32_t cluster_size = block_copy_cluster_size(s->bcs); + int64_t aligned_offset = QEMU_ALIGN_UP(offset, cluster_size); + int64_t aligned_end = QEMU_ALIGN_DOWN(offset + bytes, cluster_size); + int64_t aligned_bytes; + + if (aligned_end <= aligned_offset) { + return 0; + } + aligned_bytes = aligned_end - aligned_offset; WITH_QEMU_LOCK_GUARD(&s->lock) { - bdrv_reset_dirty_bitmap(s->access_bitmap, offset, bytes); + bdrv_reset_dirty_bitmap(s->access_bitmap, aligned_offset, + aligned_bytes); } - block_copy_reset(s->bcs, offset, bytes); + block_copy_reset(s->bcs, aligned_offset, aligned_bytes); - return bdrv_co_pdiscard(s->target, offset, bytes); + return bdrv_co_pdiscard(s->target, aligned_offset, aligned_bytes); } static void GRAPH_RDLOCK cbw_refresh_filename(BlockDriverState *bs) From patchwork Mon Apr 29 11:51:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928965 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=BNMjTTa4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShZZ6Sqxz23t4 for ; Mon, 29 Apr 2024 21:54:30 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYa-0008Gc-9v; Mon, 29 Apr 2024 07:52:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYX-0008ED-K5; Mon, 29 Apr 2024 07:52:25 -0400 Received: from forwardcorp1c.mail.yandex.net ([178.154.239.200]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYQ-0001oT-7B; Mon, 29 Apr 2024 07:52:24 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1c.mail.yandex.net (Yandex) with ESMTPS id 75C4D609CB; Mon, 29 Apr 2024 14:52:14 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-qgnwwq8a; Mon, 29 Apr 2024 14:52:13 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391533; bh=I+2soJ4/eyyd9cuN7ELJHb5wdeolkF6wmoySJyBdftI=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=BNMjTTa4YVDn4sE9IEzqR0gTCpwUjXkh70B3lmKI/Q4eNcqW0iyHwLOzuTIdDJQNg bfYqMQkUgWTIUeAiTqAL1UJ7xrKVDiZkif3PvGH4zIPgcxMmlrzHoF5Zxv/0/AUvTX uNTy7lR1VNpfxFiSwUitc0mXSJZ+Cx/Yg1QaMN+g= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Fiona Ebner , John Snow , Kevin Wolf , Hanna Reitz Subject: [PULL 4/6] block/copy-before-write: create block_copy bitmap in filter node Date: Mon, 29 Apr 2024 14:51:55 +0300 Message-Id: <20240429115157.2260885-5-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=178.154.239.200; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1c.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Currently block_copy creates copy_bitmap in source node. But that is in bad relation with .independent_close=true of copy-before-write filter: source node may be detached and removed before .bdrv_close() handler called, which should call block_copy_state_free(), which in turn should remove copy_bitmap. That's all not ideal: it would be better if internal bitmap of block-copy object is not attached to any node. But that is not possible now. The simplest solution is just create copy_bitmap in filter node, where anyway two other bitmaps are created. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fiona Ebner Tested-by: Fiona Ebner Message-Id: <20240313152822.626493-4-vsementsov@yandex-team.ru> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/block-copy.c | 3 +- block/copy-before-write.c | 2 +- include/block/block-copy.h | 1 + tests/qemu-iotests/257.out | 112 ++++++++++++++++++------------------- 4 files changed, 60 insertions(+), 58 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 9ee3dd7ef5..8fca2c3698 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -351,6 +351,7 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, } BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, + BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, Error **errp) { @@ -367,7 +368,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, return NULL; } - copy_bitmap = bdrv_create_dirty_bitmap(source->bs, cluster_size, NULL, + copy_bitmap = bdrv_create_dirty_bitmap(copy_bitmap_bs, cluster_size, NULL, errp); if (!copy_bitmap) { return NULL; diff --git a/block/copy-before-write.c b/block/copy-before-write.c index 6d89af0b29..ed2c228da7 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -468,7 +468,7 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags, ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) & bs->file->bs->supported_zero_flags); - s->bcs = block_copy_state_new(bs->file, s->target, bitmap, errp); + s->bcs = block_copy_state_new(bs->file, s->target, bs, bitmap, errp); if (!s->bcs) { error_prepend(errp, "Cannot create block-copy-state: "); return -EINVAL; diff --git a/include/block/block-copy.h b/include/block/block-copy.h index 0700953ab8..8b41643bfa 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -25,6 +25,7 @@ typedef struct BlockCopyState BlockCopyState; typedef struct BlockCopyCallState BlockCopyCallState; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, + BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, Error **errp); diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out index aa76131ca9..c33dd7f3a9 100644 --- a/tests/qemu-iotests/257.out +++ b/tests/qemu-iotests/257.out @@ -120,16 +120,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -596,16 +596,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -865,16 +865,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -1341,16 +1341,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -1610,16 +1610,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -2086,16 +2086,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -2355,16 +2355,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -2831,16 +2831,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -3100,16 +3100,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -3576,16 +3576,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -3845,16 +3845,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -4321,16 +4321,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -4590,16 +4590,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, @@ -5066,16 +5066,16 @@ write -P0x67 0x3fe0000 0x20000 "granularity": 65536, "persistent": false, "recording": false - } - ], - "drive0": [ + }, { "busy": false, "count": 0, "granularity": 65536, "persistent": false, "recording": false - }, + } + ], + "drive0": [ { "busy": false, "count": 458752, From patchwork Mon Apr 29 11:51:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928964 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=mk46DQS0; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShZC4q6Mz1ymX for ; Mon, 29 Apr 2024 21:54:11 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYc-0008IH-Kw; Mon, 29 Apr 2024 07:52:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYZ-0008GM-Nm; Mon, 29 Apr 2024 07:52:27 -0400 Received: from forwardcorp1b.mail.yandex.net ([2a02:6b8:c02:900:1:45:d181:df01]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYQ-0001ol-GK; Mon, 29 Apr 2024 07:52:27 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTPS id 82BC260C36; Mon, 29 Apr 2024 14:52:15 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-VcFi89zK; Mon, 29 Apr 2024 14:52:14 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391534; bh=AdQfubhmAF0PPkZqpSttOuQyyd9tUV5FqMJvZMix/K0=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=mk46DQS0r2Tjj0VZ8lWQ9B+R+/9RioO3T+c3J2tNu0A615ZXxXT8BJGncsG1un6aU yAcQI5Lps4GfkXtMDcMFgrelIAiwxxpGVJd6ue592peaU9xE5fkcIRZpE/GfoZDOWr 3krL54VeSOsSwWjjFvCvUpxCGpI/J8NeoI93bllQ= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Fiona Ebner , Markus Armbruster , John Snow , Kevin Wolf , Hanna Reitz , Wen Congyang , Xie Changlong , Eric Blake Subject: [PULL 5/6] qapi: blockdev-backup: add discard-source parameter Date: Mon, 29 Apr 2024 14:51:56 +0300 Message-Id: <20240429115157.2260885-6-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=2a02:6b8:c02:900:1:45:d181:df01; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Add a parameter that enables discard-after-copy. That is mostly useful in "push backup with fleecing" scheme, when source is snapshot-access format driver node, based on copy-before-write filter snapshot-access API: [guest] [snapshot-access] ~~ blockdev-backup ~~> [backup target] | | | root | file v v [copy-before-write] | | | file | target v v [active disk] [temp.img] In this case discard-after-copy does two things: - discard data in temp.img to save disk space - avoid further copy-before-write operation in discarded area Note that we have to declare WRITE permission on source in copy-before-write filter, for discard to work. Still we can't take it unconditionally, as it will break normal backup from RO source. So, we have to add a parameter and pass it thorough bdrv_open flags. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fiona Ebner Tested-by: Fiona Ebner Acked-by: Markus Armbruster Message-Id: <20240313152822.626493-5-vsementsov@yandex-team.ru> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/backup.c | 5 +++-- block/block-copy.c | 9 +++++++++ block/copy-before-write.c | 15 +++++++++++++-- block/copy-before-write.h | 1 + block/replication.c | 4 ++-- blockdev.c | 2 +- include/block/block-common.h | 2 ++ include/block/block-copy.h | 1 + include/block/block_int-global-state.h | 2 +- qapi/block-core.json | 4 ++++ 10 files changed, 37 insertions(+), 8 deletions(-) diff --git a/block/backup.c b/block/backup.c index ec29d6b810..3dd2e229d2 100644 --- a/block/backup.c +++ b/block/backup.c @@ -356,7 +356,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BlockDriverState *target, int64_t speed, MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap, BitmapSyncMode bitmap_mode, - bool compress, + bool compress, bool discard_source, const char *filter_node_name, BackupPerf *perf, BlockdevOnError on_source_error, @@ -457,7 +457,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, goto error; } - cbw = bdrv_cbw_append(bs, target, filter_node_name, &bcs, errp); + cbw = bdrv_cbw_append(bs, target, filter_node_name, discard_source, + &bcs, errp); if (!cbw) { goto error; } diff --git a/block/block-copy.c b/block/block-copy.c index 8fca2c3698..7e3b378528 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -137,6 +137,7 @@ typedef struct BlockCopyState { CoMutex lock; int64_t in_flight_bytes; BlockCopyMethod method; + bool discard_source; BlockReqList reqs; QLIST_HEAD(, BlockCopyCallState) calls; /* @@ -353,6 +354,7 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, + bool discard_source, Error **errp) { ERRP_GUARD(); @@ -418,6 +420,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, cluster_size), }; + s->discard_source = discard_source; block_copy_set_copy_opts(s, false, false); ratelimit_init(&s->rate_limit); @@ -589,6 +592,12 @@ static coroutine_fn int block_copy_task_entry(AioTask *task) co_put_to_shres(s->mem, t->req.bytes); block_copy_task_end(t, ret); + if (s->discard_source && ret == 0) { + int64_t nbytes = + MIN(t->req.offset + t->req.bytes, s->len) - t->req.offset; + bdrv_co_pdiscard(s->source, t->req.offset, nbytes); + } + return ret; } diff --git a/block/copy-before-write.c b/block/copy-before-write.c index ed2c228da7..cd65524e26 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -44,6 +44,7 @@ typedef struct BDRVCopyBeforeWriteState { BdrvChild *target; OnCbwError on_cbw_error; uint32_t cbw_timeout_ns; + bool discard_source; /* * @lock: protects access to @access_bitmap, @done_bitmap and @@ -357,6 +358,8 @@ cbw_child_perm(BlockDriverState *bs, BdrvChild *c, BdrvChildRole role, uint64_t perm, uint64_t shared, uint64_t *nperm, uint64_t *nshared) { + BDRVCopyBeforeWriteState *s = bs->opaque; + if (!(role & BDRV_CHILD_FILTERED)) { /* * Target child @@ -381,6 +384,10 @@ cbw_child_perm(BlockDriverState *bs, BdrvChild *c, BdrvChildRole role, * start */ *nperm = *nperm | BLK_PERM_CONSISTENT_READ; + if (s->discard_source) { + *nperm = *nperm | BLK_PERM_WRITE; + } + *nshared &= ~(BLK_PERM_WRITE | BLK_PERM_RESIZE); } } @@ -468,7 +475,9 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags, ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) & bs->file->bs->supported_zero_flags); - s->bcs = block_copy_state_new(bs->file, s->target, bs, bitmap, errp); + s->discard_source = flags & BDRV_O_CBW_DISCARD_SOURCE; + s->bcs = block_copy_state_new(bs->file, s->target, bs, bitmap, + flags & BDRV_O_CBW_DISCARD_SOURCE, errp); if (!s->bcs) { error_prepend(errp, "Cannot create block-copy-state: "); return -EINVAL; @@ -535,12 +544,14 @@ static BlockDriver bdrv_cbw_filter = { BlockDriverState *bdrv_cbw_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, + bool discard_source, BlockCopyState **bcs, Error **errp) { BDRVCopyBeforeWriteState *state; BlockDriverState *top; QDict *opts; + int flags = BDRV_O_RDWR | (discard_source ? BDRV_O_CBW_DISCARD_SOURCE : 0); assert(source->total_sectors == target->total_sectors); GLOBAL_STATE_CODE(); @@ -553,7 +564,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, qdict_put_str(opts, "file", bdrv_get_node_name(source)); qdict_put_str(opts, "target", bdrv_get_node_name(target)); - top = bdrv_insert_node(source, opts, BDRV_O_RDWR, errp); + top = bdrv_insert_node(source, opts, flags, errp); if (!top) { return NULL; } diff --git a/block/copy-before-write.h b/block/copy-before-write.h index 6e72bb25e9..01af0cd3c4 100644 --- a/block/copy-before-write.h +++ b/block/copy-before-write.h @@ -39,6 +39,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, + bool discard_source, BlockCopyState **bcs, Error **errp); void bdrv_cbw_drop(BlockDriverState *bs); diff --git a/block/replication.c b/block/replication.c index ca6bd0a720..0415a5e8b7 100644 --- a/block/replication.c +++ b/block/replication.c @@ -582,8 +582,8 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job = backup_job_create( NULL, s->secondary_disk->bs, s->hidden_disk->bs, - 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, NULL, - &perf, + 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, false, + NULL, &perf, BLOCKDEV_ON_ERROR_REPORT, BLOCKDEV_ON_ERROR_REPORT, JOB_INTERNAL, backup_job_completed, bs, NULL, &local_err); diff --git a/blockdev.c b/blockdev.c index 08eccc9052..875a6a4ac1 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2726,7 +2726,7 @@ static BlockJob *do_backup_common(BackupCommon *backup, job = backup_job_create(backup->job_id, bs, target_bs, backup->speed, backup->sync, bmap, backup->bitmap_mode, - backup->compress, + backup->compress, backup->discard_source, backup->filter_node_name, &perf, backup->on_source_error, diff --git a/include/block/block-common.h b/include/block/block-common.h index a846023a09..338fe5ff7a 100644 --- a/include/block/block-common.h +++ b/include/block/block-common.h @@ -243,6 +243,8 @@ typedef enum { read-write fails */ #define BDRV_O_IO_URING 0x40000 /* use io_uring instead of the thread pool */ +#define BDRV_O_CBW_DISCARD_SOURCE 0x80000 /* for copy-before-write filter */ + #define BDRV_O_CACHE_MASK (BDRV_O_NOCACHE | BDRV_O_NO_FLUSH) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index 8b41643bfa..bdc703bacd 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -27,6 +27,7 @@ typedef struct BlockCopyCallState BlockCopyCallState; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, + bool discard_source, Error **errp); /* Function should be called prior any actual copy request */ diff --git a/include/block/block_int-global-state.h b/include/block/block_int-global-state.h index d2201e27f4..eb2d92a226 100644 --- a/include/block/block_int-global-state.h +++ b/include/block/block_int-global-state.h @@ -193,7 +193,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap, BitmapSyncMode bitmap_mode, - bool compress, + bool compress, bool discard_source, const char *filter_node_name, BackupPerf *perf, BlockdevOnError on_source_error, diff --git a/qapi/block-core.json b/qapi/block-core.json index 746d1694c2..df5e07debd 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1610,6 +1610,9 @@ # node specified by @drive. If this option is not given, a node # name is autogenerated. (Since: 4.2) # +# @discard-source: Discard blocks on source which have already been +# copied to the target. (Since 9.1) +# # @x-perf: Performance options. (Since 6.0) # # Features: @@ -1631,6 +1634,7 @@ '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', '*filter-node-name': 'str', + '*discard-source': 'bool', '*x-perf': { 'type': 'BackupPerf', 'features': [ 'unstable' ] } } } From patchwork Mon Apr 29 11:51:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1928959 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=VJRyLPd2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VShYq0QrVz23tX for ; Mon, 29 Apr 2024 21:53:49 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s1PYb-0008Hl-N0; Mon, 29 Apr 2024 07:52:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYY-0008Fo-BU; Mon, 29 Apr 2024 07:52:26 -0400 Received: from forwardcorp1c.mail.yandex.net ([2a02:6b8:c03:500:1:45:d181:df01]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s1PYS-0001or-2l; Mon, 29 Apr 2024 07:52:26 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2a2a:0:640:d546:0]) by forwardcorp1c.mail.yandex.net (Yandex) with ESMTPS id 8F895609DD; Mon, 29 Apr 2024 14:52:16 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b739::1:30]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id 0qOEuD11MuQ0-KVeuepHA; Mon, 29 Apr 2024 14:52:15 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1714391535; bh=7D8uvI1If1J99UfFmUrbXLUBowb77iehvHL1xWcyJAs=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=VJRyLPd2Mu++EkxY2bM8A+BkqTmeZsllp3qG15OQqffZ/5A2v/4fEROuL+NfCxIJD 2agwGpLZnr6cZWW4fvWupXWpgsFBCQa/p6cZUfSEWiSz4+hfFz5SFESBjbM59779Wy 8KQTQpJ1Vn8b/pL+HKGPyopGrCr0PemuP8J/fzXM= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, Fiona Ebner , Kevin Wolf , Hanna Reitz Subject: [PULL 6/6] iotests: add backup-discard-source Date: Mon, 29 Apr 2024 14:51:57 +0300 Message-Id: <20240429115157.2260885-7-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240429115157.2260885-1-vsementsov@yandex-team.ru> References: <20240429115157.2260885-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Received-SPF: pass client-ip=2a02:6b8:c03:500:1:45:d181:df01; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1c.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Add test for a new backup option: discard-source. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fiona Ebner Tested-by: Fiona Ebner Message-Id: <20240313152822.626493-6-vsementsov@yandex-team.ru> Signed-off-by: Vladimir Sementsov-Ogievskiy --- .../qemu-iotests/tests/backup-discard-source | 152 ++++++++++++++++++ .../tests/backup-discard-source.out | 5 + 2 files changed, 157 insertions(+) create mode 100755 tests/qemu-iotests/tests/backup-discard-source create mode 100644 tests/qemu-iotests/tests/backup-discard-source.out diff --git a/tests/qemu-iotests/tests/backup-discard-source b/tests/qemu-iotests/tests/backup-discard-source new file mode 100755 index 0000000000..2391b12acd --- /dev/null +++ b/tests/qemu-iotests/tests/backup-discard-source @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +# +# Test backup discard-source parameter +# +# Copyright (c) Virtuozzo International GmbH. +# Copyright (c) Yandex +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import os + +import iotests +from iotests import qemu_img_create, qemu_img_map, qemu_io + + +temp_img = os.path.join(iotests.test_dir, 'temp') +source_img = os.path.join(iotests.test_dir, 'source') +target_img = os.path.join(iotests.test_dir, 'target') +size = '1M' + + +def get_actual_size(vm, node_name): + nodes = vm.cmd('query-named-block-nodes', flat=True) + node = next(n for n in nodes if n['node-name'] == node_name) + return node['image']['actual-size'] + + +class TestBackup(iotests.QMPTestCase): + def setUp(self): + qemu_img_create('-f', iotests.imgfmt, source_img, size) + qemu_img_create('-f', iotests.imgfmt, temp_img, size) + qemu_img_create('-f', iotests.imgfmt, target_img, size) + qemu_io('-c', 'write 0 1M', source_img) + + self.vm = iotests.VM() + self.vm.launch() + + self.vm.cmd('blockdev-add', { + 'node-name': 'cbw', + 'driver': 'copy-before-write', + 'file': { + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': source_img, + } + }, + 'target': { + 'driver': iotests.imgfmt, + 'discard': 'unmap', + 'node-name': 'temp', + 'file': { + 'driver': 'file', + 'filename': temp_img + } + } + }) + + self.vm.cmd('blockdev-add', { + 'node-name': 'access', + 'discard': 'unmap', + 'driver': 'snapshot-access', + 'file': 'cbw' + }) + + self.vm.cmd('blockdev-add', { + 'driver': iotests.imgfmt, + 'node-name': 'target', + 'file': { + 'driver': 'file', + 'filename': target_img + } + }) + + self.assertLess(get_actual_size(self.vm, 'temp'), 512 * 1024) + + def tearDown(self): + # That should fail, because region is discarded + self.vm.hmp_qemu_io('access', 'read 0 1M') + + self.vm.shutdown() + + self.assertTrue('read failed: Permission denied' in self.vm.get_log()) + + # Final check that temp image is empty + mapping = qemu_img_map(temp_img) + self.assertEqual(len(mapping), 1) + self.assertEqual(mapping[0]['start'], 0) + self.assertEqual(mapping[0]['length'], 1024 * 1024) + self.assertEqual(mapping[0]['data'], False) + + os.remove(temp_img) + os.remove(source_img) + os.remove(target_img) + + def do_backup(self): + self.vm.cmd('blockdev-backup', device='access', + sync='full', target='target', + job_id='backup0', + discard_source=True) + + self.vm.event_wait(name='BLOCK_JOB_COMPLETED') + + def test_discard_written(self): + """ + 1. Guest writes + 2. copy-before-write operation, data is stored to temp + 3. start backup(discard_source=True), check that data is + removed from temp + """ + # Trigger copy-before-write operation + result = self.vm.hmp_qemu_io('cbw', 'write 0 1M') + self.assert_qmp(result, 'return', '') + + # Check that data is written to temporary image + self.assertGreater(get_actual_size(self.vm, 'temp'), 1024 * 1024) + + self.do_backup() + + def test_discard_cbw(self): + """ + 1. do backup(discard_source=True), which should inform + copy-before-write that data is not needed anymore + 2. Guest writes + 3. Check that copy-before-write operation is not done + """ + self.do_backup() + + # Try trigger copy-before-write operation + result = self.vm.hmp_qemu_io('cbw', 'write 0 1M') + self.assert_qmp(result, 'return', '') + + # Check that data is not written to temporary image, as region + # is discarded from copy-before-write process + self.assertLess(get_actual_size(self.vm, 'temp'), 512 * 1024) + + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file']) diff --git a/tests/qemu-iotests/tests/backup-discard-source.out b/tests/qemu-iotests/tests/backup-discard-source.out new file mode 100644 index 0000000000..fbc63e62f8 --- /dev/null +++ b/tests/qemu-iotests/tests/backup-discard-source.out @@ -0,0 +1,5 @@ +.. +---------------------------------------------------------------------- +Ran 2 tests + +OK