From patchwork Fri Nov 18 02:08:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797678 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFRn38g4z20Wk for ; Wed, 21 Jun 2023 17:26:29 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCa-0007OB-LE; Wed, 21 Jun 2023 03:24:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCP-0007K1-Eu for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:18 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCM-0005H5-Q2 for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:16 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id 2CA6011F0E9; Wed, 21 Jun 2023 07:24:13 +0000 (UTC) From: ~hyman Date: Fri, 18 Nov 2022 10:08:54 +0800 Subject: [PATCH QEMU v6 1/9] softmmu/dirtylimit: Add parameter check for hmp "set_vcpu_dirty_limit" MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-1@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) dirty_rate paraemter of hmp command "set_vcpu_dirty_limit" is invalid if less than 0, so add parameter check for it. Note that this patch also delete the unsolicited help message and clean up the code. Signed-off-by: Hyman Huang(黄勇) Signed-off-by: Markus Armbruster Reviewed-by: Peter Xu Reviewed-by: Juan Quintela --- softmmu/dirtylimit.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 015a9038d1..5c12d26d49 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -515,14 +515,15 @@ void hmp_set_vcpu_dirty_limit(Monitor *mon, const QDict *qdict) int64_t cpu_index = qdict_get_try_int(qdict, "cpu_index", -1); Error *err = NULL; - qmp_set_vcpu_dirty_limit(!!(cpu_index != -1), cpu_index, dirty_rate, &err); - if (err) { - hmp_handle_error(mon, err); - return; + if (dirty_rate < 0) { + error_setg(&err, "invalid dirty page limit %ld", dirty_rate); + goto out; } - monitor_printf(mon, "[Please use 'info vcpu_dirty_limit' to query " - "dirty limit for virtual CPU]\n"); + qmp_set_vcpu_dirty_limit(!!(cpu_index != -1), cpu_index, dirty_rate, &err); + +out: + hmp_handle_error(mon, err); } static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index) From patchwork Wed Jun 7 13:32:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797673 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFQx4fQyz20Wk for ; Wed, 21 Jun 2023 17:25:45 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCX-0007Li-CH; Wed, 21 Jun 2023 03:24:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCP-0007K2-Em for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:18 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCM-0005H6-MX for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:16 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id 6DE5611F0F5; Wed, 21 Jun 2023 07:24:13 +0000 (UTC) From: ~hyman Date: Wed, 07 Jun 2023 21:32:59 +0800 Subject: [PATCH QEMU v6 2/9] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-2@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is in the range of 1 to 1000ms and used to make dirtyrate calculation period configurable. Currently with the "x-vcpu-dirty-limit-period" varies, the total time of live migration changes, test results show the optimal value of "x-vcpu-dirty-limit-period" ranges from 500ms to 1000 ms. "x-vcpu-dirty-limit-period" should be made stable once it proves best value can not be determined with developer's experiments. Signed-off-by: Hyman Huang(黄勇) Signed-off-by: Markus Armbruster Signed-off-by: Juan Quintela --- migration/migration-hmp-cmds.c | 8 ++++++++ migration/options.c | 28 ++++++++++++++++++++++++++++ qapi/migration.json | 34 +++++++++++++++++++++++++++------- 3 files changed, 63 insertions(+), 7 deletions(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 9885d7c9f7..352e9ec716 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -364,6 +364,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) } } } + + monitor_printf(mon, "%s: %" PRIu64 " ms\n", + MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD), + params->x_vcpu_dirty_limit_period); } qapi_free_MigrationParameters(params); @@ -620,6 +624,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) error_setg(&err, "The block-bitmap-mapping parameter can only be set " "through QMP"); break; + case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD: + p->has_x_vcpu_dirty_limit_period = true; + visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); + break; default: assert(0); } diff --git a/migration/options.c b/migration/options.c index b62ab30cd5..9743dea3ab 100644 --- a/migration/options.c +++ b/migration/options.c @@ -80,6 +80,8 @@ #define DEFINE_PROP_MIG_CAP(name, x) \ DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false) +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ + Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, store_global_state, true), @@ -163,6 +165,9 @@ Property migration_properties[] = { DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds), DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_hostname), DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), + DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, + parameters.x_vcpu_dirty_limit_period, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -891,6 +896,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) s->parameters.block_bitmap_mapping); } + params->has_x_vcpu_dirty_limit_period = true; + params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; + return params; } @@ -923,6 +931,7 @@ void migrate_params_init(MigrationParameters *params) params->has_announce_max = true; params->has_announce_rounds = true; params->has_announce_step = true; + params->has_x_vcpu_dirty_limit_period = true; } /* @@ -1083,6 +1092,15 @@ bool migrate_params_check(MigrationParameters *params, Error **errp) } #endif + if (params->has_x_vcpu_dirty_limit_period && + (params->x_vcpu_dirty_limit_period < 1 || + params->x_vcpu_dirty_limit_period > 1000)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "x-vcpu-dirty-limit-period", + "a value between 1 and 1000"); + return false; + } + return true; } @@ -1182,6 +1200,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->has_block_bitmap_mapping = true; dest->block_bitmap_mapping = params->block_bitmap_mapping; } + + if (params->has_x_vcpu_dirty_limit_period) { + dest->x_vcpu_dirty_limit_period = + params->x_vcpu_dirty_limit_period; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -1300,6 +1323,11 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) QAPI_CLONE(BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } + + if (params->has_x_vcpu_dirty_limit_period) { + s->parameters.x_vcpu_dirty_limit_period = + params->x_vcpu_dirty_limit_period; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) diff --git a/qapi/migration.json b/qapi/migration.json index 5bb5ab82a0..67c26d9dea 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -779,9 +779,14 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -799,8 +804,9 @@ 'multifd-channels', 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', - 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'multifd-zlib-level', 'multifd-zstd-level', + 'block-bitmap-mapping', + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } ## # @MigrateSetParameters: @@ -935,9 +941,14 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # TODO: either fuse back into MigrationParameters, or make # MigrationParameters members mandatory @@ -972,7 +983,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @migrate-set-parameters: @@ -1127,9 +1140,14 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -1161,7 +1179,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @query-migrate-parameters: From patchwork Wed Jun 7 14:58:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797677 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFRL4gsxz20Wk for ; Wed, 21 Jun 2023 17:26:05 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCY-0007MA-4P; Wed, 21 Jun 2023 03:24:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCQ-0007K6-83 for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCM-0005H9-TG for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:17 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id AB04A11F0FA; Wed, 21 Jun 2023 07:24:13 +0000 (UTC) From: ~hyman Date: Wed, 07 Jun 2023 22:58:32 +0800 Subject: [PATCH QEMU v6 3/9] qapi/migration: Introduce vcpu-dirty-limit parameters MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-3@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration. "vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters. This two parameters are used to help implement the dirty page rate limit algo of migration. Signed-off-by: Hyman Huang(黄勇) Acked-by: Peter Xu Reviewed-by: Juan Quintela --- migration/migration-hmp-cmds.c | 8 ++++++++ migration/options.c | 21 +++++++++++++++++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 352e9ec716..35e8020bbf 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -368,6 +368,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) monitor_printf(mon, "%s: %" PRIu64 " ms\n", MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD), params->x_vcpu_dirty_limit_period); + + monitor_printf(mon, "%s: %" PRIu64 " MB/s\n", + MigrationParameter_str(MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT), + params->vcpu_dirty_limit); } qapi_free_MigrationParameters(params); @@ -628,6 +632,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_x_vcpu_dirty_limit_period = true; visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); break; + case MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT: + p->has_vcpu_dirty_limit = true; + visit_type_size(v, param, &p->vcpu_dirty_limit, &err); + break; default: assert(0); } diff --git a/migration/options.c b/migration/options.c index 9743dea3ab..8acf5f1d2c 100644 --- a/migration/options.c +++ b/migration/options.c @@ -81,6 +81,7 @@ DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false) #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, @@ -168,6 +169,9 @@ Property migration_properties[] = { DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, parameters.x_vcpu_dirty_limit_period, DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), + DEFINE_PROP_UINT64("vcpu-dirty-limit", MigrationState, + parameters.vcpu_dirty_limit, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -898,6 +902,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_x_vcpu_dirty_limit_period = true; params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; + params->has_vcpu_dirty_limit = true; + params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit; return params; } @@ -932,6 +938,7 @@ void migrate_params_init(MigrationParameters *params) params->has_announce_rounds = true; params->has_announce_step = true; params->has_x_vcpu_dirty_limit_period = true; + params->has_vcpu_dirty_limit = true; } /* @@ -1101,6 +1108,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp) return false; } + if (params->has_vcpu_dirty_limit && + (params->vcpu_dirty_limit < 1)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "vcpu_dirty_limit", + "is invalid, it must greater then 1 MB/s"); + return false; + } + return true; } @@ -1205,6 +1220,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } + if (params->has_vcpu_dirty_limit) { + dest->vcpu_dirty_limit = params->vcpu_dirty_limit; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -1328,6 +1346,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) s->parameters.x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } + if (params->has_vcpu_dirty_limit) { + s->parameters.vcpu_dirty_limit = params->vcpu_dirty_limit; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) diff --git a/qapi/migration.json b/qapi/migration.json index 67c26d9dea..e7243c0c0d 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -783,6 +783,9 @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -806,7 +809,8 @@ 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', 'block-bitmap-mapping', - { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, + 'vcpu-dirty-limit'] } ## # @MigrateSetParameters: @@ -945,6 +949,9 @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -985,7 +992,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @migrate-set-parameters: @@ -1144,6 +1152,9 @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -1181,7 +1192,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @query-migrate-parameters: From patchwork Wed Jun 7 15:30:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797672 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFQr2BcQz20Wk for ; Wed, 21 Jun 2023 17:25:40 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCX-0007Lk-KW; Wed, 21 Jun 2023 03:24:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCP-0007K4-SA for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCM-0005HF-SD for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:17 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id D9CA911F0FC; Wed, 21 Jun 2023 07:24:13 +0000 (UTC) From: ~hyman Date: Wed, 07 Jun 2023 23:30:50 +0800 Subject: [PATCH QEMU v6 4/9] migration: Introduce dirty-limit capability MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-4@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Introduce migration dirty-limit capability, which can be turned on before live migration and limit dirty page rate durty live migration. Introduce migrate_dirty_limit function to help check if dirty-limit capability enabled during live migration. Meanwhile, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcoded. dirty-limit capability is kind of like auto-converge but using dirty limit instead of traditional cpu-throttle to throttle guest down. To enable this feature, turn on the dirty-limit capability before live migration using migrate-set-capabilities, and set the parameters "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably to speed up convergence. Signed-off-by: Hyman Huang(黄勇) Acked-by: Peter Xu Reviewed-by: Juan Quintela --- migration/options.c | 23 +++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 12 +++++++++++- softmmu/dirtylimit.c | 12 +++++++++++- 4 files changed, 46 insertions(+), 2 deletions(-) diff --git a/migration/options.c b/migration/options.c index 8acf5f1d2c..ba1010e08b 100644 --- a/migration/options.c +++ b/migration/options.c @@ -27,6 +27,7 @@ #include "qemu-file.h" #include "ram.h" #include "options.h" +#include "sysemu/kvm.h" /* Maximum migrate downtime set to 2000 seconds */ #define MAX_MIGRATE_DOWNTIME_SECONDS 2000 @@ -194,6 +195,7 @@ Property migration_properties[] = { DEFINE_PROP_MIG_CAP("x-zero-copy-send", MIGRATION_CAPABILITY_ZERO_COPY_SEND), #endif + DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT), DEFINE_PROP_END_OF_LIST(), }; @@ -240,6 +242,13 @@ bool migrate_dirty_bitmaps(void) return s->capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS]; } +bool migrate_dirty_limit(void) +{ + MigrationState *s = migrate_get_current(); + + return s->capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT]; +} + bool migrate_events(void) { MigrationState *s = migrate_get_current(); @@ -556,6 +565,20 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } } + if (new_caps[MIGRATION_CAPABILITY_DIRTY_LIMIT]) { + if (new_caps[MIGRATION_CAPABILITY_AUTO_CONVERGE]) { + error_setg(errp, "dirty-limit conflicts with auto-converge" + " either of then available currently"); + return false; + } + + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty-limit requires KVM with accelerator" + " property 'dirty-ring-size' set"); + return false; + } + } + return true; } diff --git a/migration/options.h b/migration/options.h index 45991af3c2..6f0d837932 100644 --- a/migration/options.h +++ b/migration/options.h @@ -24,6 +24,7 @@ extern Property migration_properties[]; /* capabilities */ bool migrate_auto_converge(void); +bool migrate_dirty_limit(void); bool migrate_background_snapshot(void); bool migrate_block(void); bool migrate_colo(void); diff --git a/qapi/migration.json b/qapi/migration.json index e7243c0c0d..621e6604c6 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -487,6 +487,16 @@ # and should not affect the correctness of postcopy migration. # (since 7.1) # +# @dirty-limit: If enabled, migration will use the dirty-limit algo to +# throttle down guest instead of auto-converge algo. +# Throttle algo only works when vCPU's dirtyrate greater +# than 'vcpu-dirty-limit', read processes in guest os +# aren't penalized any more, so this algo can improve +# performance of vCPU during live migration. This is an +# optional performance feature and should not affect the +# correctness of the existing auto-converge algo. +# (since 8.1) +# # Features: # # @unstable: Members @x-colo and @x-ignore-shared are experimental. @@ -502,7 +512,7 @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt'] } + 'zero-copy-send', 'postcopy-preempt', 'dirty-limit'] } ## # @MigrationCapabilityStatus: diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 5c12d26d49..953ef934bc 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -24,6 +24,9 @@ #include "hw/boards.h" #include "sysemu/kvm.h" #include "trace.h" +#include "migration/misc.h" +#include "migration/migration.h" +#include "migration/options.h" /* * Dirtylimit stop working if dirty page rate error @@ -75,11 +78,18 @@ static bool dirtylimit_quit; static void vcpu_dirty_rate_stat_collect(void) { + MigrationState *s = migrate_get_current(); VcpuStat stat; int i = 0; + int64_t period = DIRTYLIMIT_CALC_TIME_MS; + + if (migrate_dirty_limit() && + migration_is_active(s)) { + period = s->parameters.x_vcpu_dirty_limit_period; + } /* calculate vcpu dirtyrate */ - vcpu_calculate_dirtyrate(DIRTYLIMIT_CALC_TIME_MS, + vcpu_calculate_dirtyrate(period, &stat, GLOBAL_DIRTY_LIMIT, false); From patchwork Wed Jun 7 15:32:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797675 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFR96jzVz20Wk for ; Wed, 21 Jun 2023 17:25:57 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCZ-0007NK-SP; Wed, 21 Jun 2023 03:24:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCS-0007KV-Sx for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:23 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCQ-0005Hi-0R for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:20 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id 1AD3811F0FF; Wed, 21 Jun 2023 07:24:14 +0000 (UTC) From: ~hyman Date: Wed, 07 Jun 2023 23:32:51 +0800 Subject: [PATCH QEMU v6 5/9] migration: Refactor auto-converge capability logic MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-5@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Check if block migration is running before throttling guest down in auto-converge way. Note that this modification is kind of like code clean, because block migration does not depend on auto-converge capability, so the order of checks can be adjusted. Signed-off-by: Hyman Huang(黄勇) Acked-by: Peter Xu Reviewed-by: Juan Quintela --- migration/ram.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 5283a75f02..78746849b5 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -995,7 +995,11 @@ static void migration_trigger_throttle(RAMState *rs) /* During block migration the auto-converge logic incorrectly detects * that ram migration makes no progress. Avoid this by disabling the * throttling logic during the bulk phase of block migration. */ - if (migrate_auto_converge() && !blk_mig_bulk_active()) { + if (blk_mig_bulk_active()) { + return; + } + + if (migrate_auto_converge()) { /* The following detection logic can be refined later. For now: Check to see if the ratio between dirtied bytes and the approx. amount of bytes that just got transferred since the last time From patchwork Thu Jun 15 13:29:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797679 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFRq4GKqz20Wk for ; Wed, 21 Jun 2023 17:26:31 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCY-0007Me-Gd; Wed, 21 Jun 2023 03:24:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCR-0007KB-GW for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCP-0005Hg-VS for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id 49CB311F101; Wed, 21 Jun 2023 07:24:14 +0000 (UTC) From: ~hyman Date: Thu, 15 Jun 2023 21:29:44 +0800 Subject: [PATCH QEMU v6 6/9] migration: Put the detection logic before auto-converge checking MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-6@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) This commit is prepared for the implementation of dirty-limit convergence algo. The detection logic of throttling condition can apply to both auto-converge and dirty-limit algo, putting it's position before the checking logic for auto-converge feature. Signed-off-by: Hyman Huang(黄勇) Reviewed-by: Juan Quintela --- migration/ram.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 78746849b5..b6559f9312 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -999,17 +999,18 @@ static void migration_trigger_throttle(RAMState *rs) return; } - if (migrate_auto_converge()) { - /* The following detection logic can be refined later. For now: - Check to see if the ratio between dirtied bytes and the approx. - amount of bytes that just got transferred since the last time - we were in this routine reaches the threshold. If that happens - twice, start or increase throttling. */ - - if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >= 2)) { + /* + * The following detection logic can be refined later. For now: + * Check to see if the ratio between dirtied bytes and the approx. + * amount of bytes that just got transferred since the last time + * we were in this routine reaches the threshold. If that happens + * twice, start or increase throttling. + */ + if ((bytes_dirty_period > bytes_dirty_threshold) && + (++rs->dirty_rate_high_cnt >= 2)) { + rs->dirty_rate_high_cnt = 0; + if (migrate_auto_converge()) { trace_migration_throttle(); - rs->dirty_rate_high_cnt = 0; mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); } From patchwork Wed Jun 7 16:12:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797676 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFR96r3Sz20Xm for ; Wed, 21 Jun 2023 17:25:57 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCY-0007N2-Vc; Wed, 21 Jun 2023 03:24:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCR-0007KC-PC for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCP-0005Hl-WA for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:19 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id 78CD611F104; Wed, 21 Jun 2023 07:24:14 +0000 (UTC) From: ~hyman Date: Thu, 08 Jun 2023 00:12:40 +0800 Subject: [PATCH QEMU v6 7/9] migration: Implement dirty-limit convergence algo MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-7@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent. Enable dirty page limit if dirty_rate_high_cnt greater than 2 when dirty-limit capability enabled, Disable dirty-limit if migration be canceled. Note that "set_vcpu_dirty_limit", "cancel_vcpu_dirty_limit" commands are not allowed during dirty-limit live migration. Signed-off-by: Hyman Huang(黄勇) Signed-off-by: Markus Armbruster --- migration/migration.c | 3 +++ migration/ram.c | 36 ++++++++++++++++++++++++++++++++++++ migration/trace-events | 1 + softmmu/dirtylimit.c | 29 +++++++++++++++++++++++++++++ 4 files changed, 69 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index dc05c6f6ea..4278b48af0 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -165,6 +165,9 @@ void migration_cancel(const Error *error) if (error) { migrate_set_error(current_migration, error); } + if (migrate_dirty_limit()) { + qmp_cancel_vcpu_dirty_limit(false, -1, NULL); + } migrate_fd_cancel(current_migration); } diff --git a/migration/ram.c b/migration/ram.c index b6559f9312..8a86363216 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -46,6 +46,7 @@ #include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h" @@ -59,6 +60,8 @@ #include "multifd.h" #include "sysemu/runstate.h" #include "options.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */ @@ -984,6 +987,37 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* + * Enable dirty-limit to throttle down the guest + */ +static void migration_dirty_limit_guest(void) +{ + /* + * dirty page rate quota for all vCPUs fetched from + * migration parameter 'vcpu_dirty_limit' + */ + static int64_t quota_dirtyrate; + MigrationState *s = migrate_get_current(); + + /* + * If dirty limit already enabled and migration parameter + * vcpu-dirty-limit untouched. + */ + if (dirtylimit_in_service() && + quota_dirtyrate == s->parameters.vcpu_dirty_limit) { + return; + } + + quota_dirtyrate = s->parameters.vcpu_dirty_limit; + + /* + * Set all vCPU a quota dirtyrate, note that the second + * parameter will be ignored if setting all vCPU for the vm + */ + qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL); + trace_migration_dirty_limit_guest(quota_dirtyrate); +} + static void migration_trigger_throttle(RAMState *rs) { uint64_t threshold = migrate_throttle_trigger_threshold(); @@ -1013,6 +1047,8 @@ static void migration_trigger_throttle(RAMState *rs) trace_migration_throttle(); mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); + } else if (migrate_dirty_limit()) { + migration_dirty_limit_guest(); } } } diff --git a/migration/trace-events b/migration/trace-events index cdaef7a1ea..c5cb280d95 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -91,6 +91,7 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" migration_throttle(void) "" +migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=%d addr=0x%" PRIx64 " flags=0x%x" diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 953ef934bc..5134296667 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -436,6 +436,23 @@ static void dirtylimit_cleanup(void) dirtylimit_state_finalize(); } +/* + * dirty page rate limit is not allowed to set if migration + * is running with dirty-limit capability enabled. + */ +static bool dirtylimit_is_allowed(void) +{ + MigrationState *ms = migrate_get_current(); + + if (migration_is_running(ms->state) && + (!qemu_thread_is_self(&ms->thread)) && + migrate_dirty_limit() && + dirtylimit_in_service()) { + return false; + } + return true; +} + void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, int64_t cpu_index, Error **errp) @@ -449,6 +466,12 @@ void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, return; } + if (!dirtylimit_is_allowed()) { + error_setg(errp, "can't cancel dirty page rate limit while" + " migration is running"); + return; + } + if (!dirtylimit_in_service()) { return; } @@ -499,6 +522,12 @@ void qmp_set_vcpu_dirty_limit(bool has_cpu_index, return; } + if (!dirtylimit_is_allowed()) { + error_setg(errp, "can't set dirty page rate limit while" + " migration is running"); + return; + } + if (!dirty_rate) { qmp_cancel_vcpu_dirty_limit(has_cpu_index, cpu_index, errp); return; From patchwork Wed Jun 7 16:21:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797674 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFR80yt6z20Wk for ; Wed, 21 Jun 2023 17:25:56 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCa-0007Nd-4T; Wed, 21 Jun 2023 03:24:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCS-0007KW-VO for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:23 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCQ-0005Hk-01 for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:20 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id A9C8F11F108; Wed, 21 Jun 2023 07:24:14 +0000 (UTC) From: ~hyman Date: Thu, 08 Jun 2023 00:21:58 +0800 Subject: [PATCH QEMU v6 8/9] migration: Extend query-migrate to provide dirty page limit info MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-8@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Extend query-migrate to provide throttle time and estimated ring full time with dirty-limit capability enabled, through which we can observe if dirty limit take effect during live migration. Signed-off-by: Hyman Huang(黄勇) Signed-off-by: Markus Armbruster Reviewed-by: Juan Quintela Reviewed-by: Juan Quintela --- include/sysemu/dirtylimit.h | 2 ++ migration/migration-hmp-cmds.c | 10 +++++++++ migration/migration.c | 10 +++++++++ qapi/migration.json | 16 +++++++++++++- softmmu/dirtylimit.c | 39 ++++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 1 deletion(-) diff --git a/include/sysemu/dirtylimit.h b/include/sysemu/dirtylimit.h index 8d2c1f3a6b..d11ebbbbdb 100644 --- a/include/sysemu/dirtylimit.h +++ b/include/sysemu/dirtylimit.h @@ -34,4 +34,6 @@ void dirtylimit_set_vcpu(int cpu_index, void dirtylimit_set_all(uint64_t quota, bool enable); void dirtylimit_vcpu_execute(CPUState *cpu); +uint64_t dirtylimit_throttle_time_per_round(void); +uint64_t dirtylimit_ring_full_time(void); #endif diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 35e8020bbf..c115ef2d23 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -190,6 +190,16 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->cpu_throttle_percentage); } + if (info->has_dirty_limit_throttle_time_per_round) { + monitor_printf(mon, "dirty-limit throttle time: %" PRIu64 " us\n", + info->dirty_limit_throttle_time_per_round); + } + + if (info->has_dirty_limit_ring_full_time) { + monitor_printf(mon, "dirty-limit ring full time: %" PRIu64 " us\n", + info->dirty_limit_ring_full_time); + } + if (info->has_postcopy_blocktime) { monitor_printf(mon, "postcopy blocktime: %u\n", info->postcopy_blocktime); diff --git a/migration/migration.c b/migration/migration.c index 4278b48af0..5e1abc9cee 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -64,6 +64,7 @@ #include "yank_functions.h" #include "sysemu/qtest.h" #include "options.h" +#include "sysemu/dirtylimit.h" static NotifierList migration_state_notifiers = NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); @@ -968,6 +969,15 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->dirty_pages_rate = stat64_get(&mig_stats.dirty_pages_rate); } + + if (migrate_dirty_limit() && dirtylimit_in_service()) { + info->has_dirty_limit_throttle_time_per_round = true; + info->dirty_limit_throttle_time_per_round = + dirtylimit_throttle_time_per_round(); + + info->has_dirty_limit_ring_full_time = true; + info->dirty_limit_ring_full_time = dirtylimit_ring_full_time(); + } } static void populate_disk_info(MigrationInfo *info) diff --git a/qapi/migration.json b/qapi/migration.json index 621e6604c6..e9b24fc410 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -250,6 +250,18 @@ # blocked. Present and non-empty when migration is blocked. # (since 6.0) # +# @dirty-limit-throttle-time-per-round: Maximum throttle time (in microseconds) of virtual +# CPUs each dirty ring full round, which shows how +# MigrationCapability dirty-limit affects the guest +# during live migration. (since 8.1) +# +# @dirty-limit-ring-full-time: Estimated average dirty ring full time (in microseconds) +# each dirty ring full round, note that the value equals +# dirty ring memory size divided by average dirty page rate +# of virtual CPU, which can be used to observe the average +# memory load of virtual CPU indirectly. Note that zero +# means guest doesn't dirty memory (since 8.1) +# # Since: 0.14 ## { 'struct': 'MigrationInfo', @@ -267,7 +279,9 @@ '*postcopy-blocktime' : 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], '*compression': 'CompressionStats', - '*socket-address': ['SocketAddress'] } } + '*socket-address': ['SocketAddress'], + '*dirty-limit-throttle-time-per-round': 'uint64', + '*dirty-limit-ring-full-time': 'uint64'} } ## # @query-migrate: diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 5134296667..a0686323e5 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -565,6 +565,45 @@ out: hmp_handle_error(mon, err); } +/* Return the max throttle time of each virtual CPU */ +uint64_t dirtylimit_throttle_time_per_round(void) +{ + CPUState *cpu; + int64_t max = 0; + + CPU_FOREACH(cpu) { + if (cpu->throttle_us_per_full > max) { + max = cpu->throttle_us_per_full; + } + } + + return max; +} + +/* + * Estimate average dirty ring full time of each virtaul CPU. + * Return 0 if guest doesn't dirty memory. + */ +uint64_t dirtylimit_ring_full_time(void) +{ + CPUState *cpu; + uint64_t curr_rate = 0; + int nvcpus = 0; + + CPU_FOREACH(cpu) { + if (cpu->running) { + nvcpus++; + curr_rate += vcpu_dirty_rate_get(cpu->cpu_index); + } + } + + if (!curr_rate || !nvcpus) { + return 0; + } + + return dirtylimit_dirty_ring_full_time(curr_rate / nvcpus); +} + static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index) { DirtyLimitInfo *info = NULL; From patchwork Wed Jun 7 16:46:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ~hyman X-Patchwork-Id: 1797670 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QmFQ35284z20WT for ; Wed, 21 Jun 2023 17:24:59 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBsCc-0007P7-A5; Wed, 21 Jun 2023 03:24:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCT-0007KY-Bz for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:24 -0400 Received: from mail-b.sr.ht ([173.195.146.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBsCQ-0005I7-3X for qemu-devel@nongnu.org; Wed, 21 Jun 2023 03:24:20 -0400 Authentication-Results: mail-b.sr.ht; dkim=none Received: from git.sr.ht (unknown [173.195.146.142]) by mail-b.sr.ht (Postfix) with ESMTPSA id DC4E011F10B; Wed, 21 Jun 2023 07:24:14 +0000 (UTC) From: ~hyman Date: Thu, 08 Jun 2023 00:46:45 +0800 Subject: [PATCH QEMU v6 9/9] tests: Add migration dirty-limit capability test MIME-Version: 1.0 Message-ID: <168733225273.5845.15871826788879741674-9@git.sr.ht> X-Mailer: git.sr.ht In-Reply-To: <168733225273.5845.15871826788879741674-0@git.sr.ht> To: qemu-devel Cc: Juan Quintela , Peter Xu , Paolo Bonzini , "Dr. David Alan Gilbert" , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Richard Henderson , Philippe =?utf-8?q?Mathie?= =?utf-8?q?u-Daud=C3=A9?= , hyman Received-SPF: pass client-ip=173.195.146.151; envelope-from=outgoing@sr.ht; helo=mail-b.sr.ht X-Spam_score_int: 15 X-Spam_score: 1.5 X-Spam_bar: + X-Spam_report: (1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_96_XX=3.405, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ~hyman Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Hyman Huang(黄勇) Add migration dirty-limit capability test if kernel support dirty ring. Migration dirty-limit capability introduce dirty limit capability, two parameters: x-vcpu-dirty-limit-period and vcpu-dirty-limit are introduced to implement the live migration with dirty limit. The test case does the following things: 1. start src, dst vm and enable dirty-limit capability 2. start migrate and set cancel it to check if dirty limit stop working. 3. restart dst vm 4. start migrate and enable dirty-limit capability 5. check if migration satisfy the convergence condition during pre-switchover phase. Signed-off-by: Hyman Huang(黄勇) --- tests/qtest/migration-test.c | 155 +++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index b0c355bbd9..9377560901 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2609,6 +2609,159 @@ static void test_vcpu_dirty_limit(void) dirtylimit_stop_vm(vm); } +static void migrate_dirty_limit_wait_showup(QTestState *from, + const int64_t period, + const int64_t value) +{ + /* Enable dirty limit capability */ + migrate_set_capability(from, "dirty-limit", true); + + /* Set dirty limit parameters */ + migrate_set_parameter_int(from, "x-vcpu-dirty-limit-period", period); + migrate_set_parameter_int(from, "vcpu-dirty-limit", value); + + /* Make sure migrate can't converge */ + migrate_ensure_non_converge(from); + + /* To check limit rate after precopy */ + migrate_set_capability(from, "pause-before-switchover", true); + + /* Wait for the serial output from the source */ + wait_for_serial("src_serial"); +} + +/* + * This test does: + * source target + * migrate_incoming + * migrate + * migrate_cancel + * restart target + * migrate + * + * And see that if dirty limit works correctly + */ +static void test_migrate_dirty_limit(void) +{ + g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); + QTestState *from, *to; + int64_t remaining; + uint64_t throttle_us_per_full; + /* + * We want the test to be stable and as fast as possible. + * E.g., with 1Gb/s bandwith migration may pass without dirty limit, + * so we need to decrease a bandwidth. + */ + const int64_t dirtylimit_period = 1000, dirtylimit_value = 50; + const int64_t max_bandwidth = 400000000; /* ~400Mb/s */ + const int64_t downtime_limit = 250; /* 250ms */ + /* + * We migrate through unix-socket (> 500Mb/s). + * Thus, expected migration speed ~= bandwidth limit (< 500Mb/s). + * So, we can predict expected_threshold + */ + const int64_t expected_threshold = max_bandwidth * downtime_limit / 1000; + int max_try_count = 10; + MigrateCommon args = { + .start = { + .hide_stderr = true, + .use_dirty_ring = true, + }, + .listen_uri = uri, + .connect_uri = uri, + }; + + /* Start src, dst vm */ + if (test_migrate_start(&from, &to, args.listen_uri, &args.start)) { + return; + } + + /* Prepare for dirty limit migration and wait src vm show up */ + migrate_dirty_limit_wait_showup(from, dirtylimit_period, dirtylimit_value); + + /* Start migrate */ + migrate_qmp(from, uri, "{}"); + + /* Wait for dirty limit throttle begin */ + throttle_us_per_full = 0; + while (throttle_us_per_full == 0) { + throttle_us_per_full = + read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_src_stop); + } + + /* Now cancel migrate and wait for dirty limit throttle switch off */ + migrate_cancel(from); + wait_for_migration_status(from, "cancelled", NULL); + + /* Check if dirty limit throttle switched off, set timeout 1ms */ + do { + throttle_us_per_full = + read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_src_stop); + } while (throttle_us_per_full != 0 && --max_try_count); + + /* Assert dirty limit is not in service */ + g_assert_cmpint(throttle_us_per_full, ==, 0); + + args = (MigrateCommon) { + .start = { + .only_target = true, + .use_dirty_ring = true, + }, + .listen_uri = uri, + .connect_uri = uri, + }; + + /* Restart dst vm, src vm already show up so we needn't wait anymore */ + if (test_migrate_start(&from, &to, args.listen_uri, &args.start)) { + return; + } + + /* Start migrate */ + migrate_qmp(from, uri, "{}"); + + /* Wait for dirty limit throttle begin */ + throttle_us_per_full = 0; + while (throttle_us_per_full == 0) { + throttle_us_per_full = + read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_src_stop); + } + + /* + * The dirty limit rate should equals the return value of + * query-vcpu-dirty-limit if dirty limit cap set + */ + g_assert_cmpint(dirtylimit_value, ==, get_limit_rate(from)); + + /* Now, we have tested if dirty limit works, let it converge */ + migrate_set_parameter_int(from, "downtime-limit", downtime_limit); + migrate_set_parameter_int(from, "max-bandwidth", max_bandwidth); + + /* + * Wait for pre-switchover status to check if migration + * satisfy the convergence condition + */ + wait_for_migration_status(from, "pre-switchover", NULL); + + remaining = read_ram_property_int(from, "remaining"); + g_assert_cmpint(remaining, <, + (expected_threshold + expected_threshold / 100)); + + migrate_continue(from, "pre-switchover"); + + qtest_qmp_eventwait(to, "RESUME"); + + wait_for_serial("dest_serial"); + wait_for_migration_complete(from); + + test_migrate_end(from, to, true); +} + static bool kvm_dirty_ring_supported(void) { #if defined(__linux__) && defined(HOST_X86_64) @@ -2816,6 +2969,8 @@ int main(int argc, char **argv) test_precopy_unix_dirty_ring); qtest_add_func("/migration/vcpu_dirty_limit", test_vcpu_dirty_limit); + qtest_add_func("/migration/dirty_limit", + test_migrate_dirty_limit); } ret = g_test_run();