From patchwork Tue Oct 20 11:39:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhengui li X-Patchwork-Id: 1384866 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CFsB33tYlz9ryj for ; Tue, 20 Oct 2020 22:40:10 +1100 (AEDT) Received: from localhost ([::1]:48882 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kUpzs-0006m9-QE for incoming@patchwork.ozlabs.org; Tue, 20 Oct 2020 07:40:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56780) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kUpzG-0006Xn-1d; Tue, 20 Oct 2020 07:39:31 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:50000 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kUpzC-0002y2-QX; Tue, 20 Oct 2020 07:39:29 -0400 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 4127E4334A3773969C3A; Tue, 20 Oct 2020 19:39:20 +0800 (CST) Received: from DESKTOP-80C7KIU.china.huawei.com (10.174.187.210) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Tue, 20 Oct 2020 19:39:13 +0800 From: Zhengui li To: , , , Subject: [PATCH v2 1/2] qemu-img: add support for offline rate limit in qemu-img commit Date: Tue, 20 Oct 2020 11:39:05 +0000 Message-ID: <1603193946-30096-1-git-send-email-lizhengui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.187.210] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=lizhengui@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/20 07:04:17 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: xieyingtai@huawei.com, lizhengui@huawei.com, qemu-devel@nongnu.org, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Zhengui Currently, there is no rate limit for qemu-img commit. This may cause the task of qemu-img commit to consume all the bandwidth of the storage. This will affect the IO performance of other processes and virtual machines under shared storage. So we add support for offline rate limit in qemu-img commit to get better quality of sevice. Signed-off-by: Zhengui Reviewed-by: Alberto Garcia --- docs/tools/qemu-img.rst | 4 +++- qemu-img-cmds.hx | 4 ++-- qemu-img.c | 14 ++++++++++++-- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index c35bd64..bcb11b0 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -349,7 +349,7 @@ Command description: state after (the attempt at) repairing it. That is, a successful ``-r all`` will yield the exit code 0, independently of the image state before. -.. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t CACHE] [-b BASE] [-d] [-p] FILENAME +.. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t CACHE] [-b BASE] [-r RATE_LIMIT] [-d] [-p] FILENAME Commit the changes recorded in *FILENAME* in its base image or backing file. If the backing file is smaller than the snapshot, then the backing file will be @@ -371,6 +371,8 @@ Command description: garbage data when read. For this reason, ``-b`` implies ``-d`` (so that the top image stays valid). + The rate limit for the commit process is specified by ``-r``. + .. option:: compare [--object OBJECTDEF] [--image-opts] [-f FMT] [-F FMT] [-T SRC_CACHE] [-p] [-q] [-s] [-U] FILENAME1 FILENAME2 Check if two images have the same content. You can compare images with diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index b89c019..2a31806 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -34,9 +34,9 @@ SRST ERST DEF("commit", img_commit, - "commit [--object objectdef] [--image-opts] [-q] [-f fmt] [-t cache] [-b base] [-d] [-p] filename") + "commit [--object objectdef] [--image-opts] [-q] [-f fmt] [-t cache] [-b base] [-r rate_limit] [-d] [-p] filename") SRST -.. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t CACHE] [-b BASE] [-d] [-p] FILENAME +.. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t CACHE] [-b BASE] [-r RATE_LIMIT] [-d] [-p] FILENAME ERST DEF("compare", img_compare, diff --git a/qemu-img.c b/qemu-img.c index 2103507..ea66139 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -980,6 +980,7 @@ static int img_commit(int argc, char **argv) CommonBlockJobCBInfo cbi; bool image_opts = false; AioContext *aio_context; + int64_t rate_limit = 0; fmt = NULL; cache = BDRV_DEFAULT_CACHE; @@ -991,7 +992,7 @@ static int img_commit(int argc, char **argv) {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {0, 0, 0, 0} }; - c = getopt_long(argc, argv, ":f:ht:b:dpq", + c = getopt_long(argc, argv, ":f:ht:b:dpqr:", long_options, NULL); if (c == -1) { break; @@ -1026,6 +1027,15 @@ static int img_commit(int argc, char **argv) case 'q': quiet = true; break; + case 'r': { + int64_t sval; + + sval = cvtnum("rate limit", optarg); + if (sval < 0) { + return 1; + } + rate_limit = sval; + } break; case OPTION_OBJECT: { QemuOpts *opts; opts = qemu_opts_parse_noisily(&qemu_object_opts, @@ -1099,7 +1109,7 @@ static int img_commit(int argc, char **argv) aio_context = bdrv_get_aio_context(bs); aio_context_acquire(aio_context); - commit_active_start("commit", bs, base_bs, JOB_DEFAULT, 0, + commit_active_start("commit", bs, base_bs, JOB_DEFAULT, rate_limit, BLOCKDEV_ON_ERROR_REPORT, NULL, common_block_job_cb, &cbi, false, &local_err); aio_context_release(aio_context); From patchwork Tue Oct 20 11:39:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhengui li X-Patchwork-Id: 1384867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CFsBk6d6dz9sRK for ; Tue, 20 Oct 2020 22:40:46 +1100 (AEDT) Received: from localhost ([::1]:50056 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kUq0S-0007Fx-UG for incoming@patchwork.ozlabs.org; Tue, 20 Oct 2020 07:40:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56800) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kUpzM-0006Y2-IL; Tue, 20 Oct 2020 07:39:37 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:50088 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kUpzI-0002yD-2l; Tue, 20 Oct 2020 07:39:36 -0400 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 5221D299DA043B1BF8D2; Tue, 20 Oct 2020 19:39:25 +0800 (CST) Received: from DESKTOP-80C7KIU.china.huawei.com (10.174.187.210) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Tue, 20 Oct 2020 19:39:15 +0800 From: Zhengui li To: , , , Subject: [PATCH v2 2/2] qemu-img: add support for rate limit in qemu-img convert Date: Tue, 20 Oct 2020 11:39:06 +0000 Message-ID: <1603193946-30096-2-git-send-email-lizhengui@huawei.com> X-Mailer: git-send-email 2.6.4.windows.1 In-Reply-To: <1603193946-30096-1-git-send-email-lizhengui@huawei.com> References: <1603193946-30096-1-git-send-email-lizhengui@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.210] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=lizhengui@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/20 07:04:17 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: xieyingtai@huawei.com, lizhengui@huawei.com, qemu-devel@nongnu.org, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Zhengui Currently, there is no rate limit for qemu-img convert. This may cause the task of qemu-img convert to consume all the bandwidth of the storage. This will affect the IO performance of other processes and virtual machines under shared storage. So we add support for offline rate limit in qemu-img convert to get better quality of sevice. Signed-off-by: Zhengui Reviewed-by: Alberto Garcia --- docs/tools/qemu-img.rst | 6 +++++- qemu-img-cmds.hx | 4 ++-- qemu-img.c | 30 +++++++++++++++++++++++++++++- 3 files changed, 36 insertions(+), 4 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index bcb11b0..b615aa8 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -188,6 +188,10 @@ Parameters to convert subcommand: allocated target image depending on the host support for getting allocation information. +.. option:: -r + + Rate limit for the convert process + .. option:: --salvage Try to ignore I/O errors when reading. Unless in quiet mode (``-q``), errors @@ -410,7 +414,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index 2a31806..7eb489b 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -46,9 +46,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-r rate_limit] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create, diff --git a/qemu-img.c b/qemu-img.c index ea66139..8dfe1c4 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -50,6 +50,8 @@ #include "block/qapi.h" #include "crypto/init.h" #include "trace/control.h" +#include "qemu/throttle.h" +#include "block/throttle-groups.h" #define QEMU_IMG_VERSION "qemu-img version " QEMU_FULL_VERSION \ "\n" QEMU_COPYRIGHT "\n" @@ -1672,6 +1674,7 @@ enum ImgConvertBlockStatus { }; #define MAX_COROUTINES 16 +#define CONVERT_THROTTLE_GROUP "img_convert" typedef struct ImgConvertState { BlockBackend **src; @@ -2187,6 +2190,17 @@ static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *dst) #define MAX_BUF_SECTORS 32768 +static void set_rate_limit(BlockBackend *blk, int64_t rate_limit) +{ + ThrottleConfig cfg; + + throttle_config_init(&cfg); + cfg.buckets[THROTTLE_BPS_WRITE].avg = rate_limit; + + blk_io_limits_enable(blk, CONVERT_THROTTLE_GROUP); + blk_set_io_limits(blk, &cfg); +} + static int img_convert(int argc, char **argv) { int c, bs_i, flags, src_flags = 0; @@ -2207,6 +2221,7 @@ static int img_convert(int argc, char **argv) bool force_share = false; bool explict_min_sparse = false; bool bitmaps = false; + int64_t rate_limit = 0; ImgConvertState s = (ImgConvertState) { /* Need at least 4k of zeros for sparse detection */ @@ -2229,7 +2244,7 @@ static int img_convert(int argc, char **argv) {"bitmaps", no_argument, 0, OPTION_BITMAPS}, {0, 0, 0, 0} }; - c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU", + c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WUr:", long_options, NULL); if (c == -1) { break; @@ -2326,6 +2341,15 @@ static int img_convert(int argc, char **argv) case 'U': force_share = true; break; + case 'r': { + int64_t sval; + + sval = cvtnum("rate limit", optarg); + if (sval < 0) { + goto fail_getopt; + } + rate_limit = sval; + } break; case OPTION_OBJECT: { QemuOpts *object_opts; object_opts = qemu_opts_parse_noisily(&qemu_object_opts, @@ -2715,6 +2739,10 @@ static int img_convert(int argc, char **argv) s.cluster_sectors = bdi.cluster_size / BDRV_SECTOR_SIZE; } + if (rate_limit) { + set_rate_limit(s.target, rate_limit); + } + ret = convert_do_copy(&s); /* Now copy the bitmaps */