| Message ID | 99ea5e76926164d60a4ee62d0a1831823bc0d7a9.1637403404.git.huangy81@chinatelecom.cn |
|---|---|
| State | New |
| Headers | show |
| Series | [v3,1/3] migration/dirtyrate: implement vCPU dirtyrate calculation periodically | expand |
huangy81@chinatelecom.cn writes: > From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> > > implement dirtyrate calculation periodically basing on > dirty-ring and throttle vCPU until it reachs the quota > dirtyrate given by user. > > introduce qmp commands set-dirty-limit/cancel-dirty-limit to > set/cancel dirty limit on vCPU. Please start sentences with a capital letter. > > Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> [...] > diff --git a/qapi/misc.json b/qapi/misc.json > index 358548a..98e6001 100644 > --- a/qapi/misc.json > +++ b/qapi/misc.json > @@ -527,3 +527,42 @@ > 'data': { '*option': 'str' }, > 'returns': ['CommandLineOptionInfo'], > 'allow-preconfig': true } > + > +## > +# @set-dirty-limit: > +# > +# This command could be used to cap the vCPU memory load, which is also > +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" > +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". > +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used > +# to set the upper limit of dirtyrate for the interested vCPU. "dirtyrate" is not a word. Let's spell it "dirty page rate", for consistency with the documentation in migration.json. Regarding "One should use ...": sounds like you have to run calc-dirty-rate with argument @mode set to @dirty-ring before this command. Correct? What happens when you don't? set-dirty-limit fails? Do you also have to run query-dirty-rate before this command? Speaking of migration.json: should these commands be defined there, next to calc-dirty-rate and query-dirty-rate? > +# > +# @idx: vCPU index to set dirtylimit. > +# > +# @dirtyrate: upper limit of drityrate the specified vCPU could reach (MB/s) Typo "drityrate". Suggest "upper limit for the specified vCPU's dirty page rate (MB/s)". > +# > +# Since: 6.3 > +# > +# Example: > +# {"execute": "set-dirty-limit"} > +# "arguments": { "idx": 0, > +# "dirtyrate": 200 } } > +# > +## > +{ 'command': 'set-dirty-limit', > + 'data': { 'idx': 'int', 'dirtyrate': 'uint64' } } > + > +## > +# @cancel-dirty-limit: > +# > +# @idx: vCPU index to canceled the dirtylimit > +# > +# Since: 6.3 > +# > +# Example: > +# {"execute": "cancel-dirty-limit"} > +# "arguments": { "idx": 0 } } > +# > +## > +{ 'command': 'cancel-dirty-limit', > + 'data': { 'idx': 'int' } } > diff --git a/softmmu/vl.c b/softmmu/vl.c > index 1159a64..170ee23 100644 > --- a/softmmu/vl.c > +++ b/softmmu/vl.c > @@ -3776,5 +3776,6 @@ void qemu_init(int argc, char **argv, char **envp) > qemu_init_displays(); > accel_setup_post(current_machine); > os_setup_post(); > + dirtylimit_setup(current_machine->smp.max_cpus); > resume_mux_open(); > }
在 2021/11/22 15:35, Markus Armbruster 写道: > huangy81@chinatelecom.cn writes: > >> From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> >> >> implement dirtyrate calculation periodically basing on >> dirty-ring and throttle vCPU until it reachs the quota >> dirtyrate given by user. >> >> introduce qmp commands set-dirty-limit/cancel-dirty-limit to >> set/cancel dirty limit on vCPU. > > Please start sentences with a capital letter. > Ok，i'll check the syntax problem next version. >> >> Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> > > [...] > >> diff --git a/qapi/misc.json b/qapi/misc.json >> index 358548a..98e6001 100644 >> --- a/qapi/misc.json >> +++ b/qapi/misc.json >> @@ -527,3 +527,42 @@ >> 'data': { '*option': 'str' }, >> 'returns': ['CommandLineOptionInfo'], >> 'allow-preconfig': true } >> + >> +## >> +# @set-dirty-limit: >> +# >> +# This command could be used to cap the vCPU memory load, which is also >> +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" >> +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". >> +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used >> +# to set the upper limit of dirtyrate for the interested vCPU. > > "dirtyrate" is not a word. Let's spell it "dirty page rate", for > consistency with the documentation in migration.json. Ok, sounds good. > > Regarding "One should use ...": sounds like you have to run > calc-dirty-rate with argument @mode set to @dirty-ring before this > command. Correct? What happens when you don't? set-dirty-limit fails? > > Do you also have to run query-dirty-rate before this command? Actually no, i'll clarify the usage next verison. > > Speaking of migration.json: should these commands be defined there, next > to calc-dirty-rate and query-dirty-rate? I'm struggling too because these commands will be used in migration but it is vCPU they handle. > >> +# >> +# @idx: vCPU index to set dirtylimit. >> +# >> +# @dirtyrate: upper limit of drityrate the specified vCPU could reach (MB/s) > > Typo "drityrate". > > Suggest "upper limit for the specified vCPU's dirty page rate (MB/s)". Ok. > >> +# >> +# Since: 6.3 >> +# >> +# Example: >> +# {"execute": "set-dirty-limit"} >> +# "arguments": { "idx": 0, >> +# "dirtyrate": 200 } } >> +# >> +## >> +{ 'command': 'set-dirty-limit', >> + 'data': { 'idx': 'int', 'dirtyrate': 'uint64' } } >> + >> +## >> +# @cancel-dirty-limit: >> +# >> +# @idx: vCPU index to canceled the dirtylimit >> +# >> +# Since: 6.3 >> +# >> +# Example: >> +# {"execute": "cancel-dirty-limit"} >> +# "arguments": { "idx": 0 } } >> +# >> +## >> +{ 'command': 'cancel-dirty-limit', >> + 'data': { 'idx': 'int' } } >> diff --git a/softmmu/vl.c b/softmmu/vl.c >> index 1159a64..170ee23 100644 >> --- a/softmmu/vl.c >> +++ b/softmmu/vl.c >> @@ -3776,5 +3776,6 @@ void qemu_init(int argc, char **argv, char **envp) >> qemu_init_displays(); >> accel_setup_post(current_machine); >> os_setup_post(); >> + dirtylimit_setup(current_machine->smp.max_cpus); >> resume_mux_open(); >> } >
Hyman Huang <huangy81@chinatelecom.cn> writes: > =E5=9C=A8 2021/11/22 15:35, Markus Armbruster =E5=86=99=E9=81=93: >> huangy81@chinatelecom.cn writes: >> >>> From: Hyman Huang(=E9=BB=84=E5=8B=87) <huangy81@chinatelecom.cn> >>> >>> implement dirtyrate calculation periodically basing on >>> dirty-ring and throttle vCPU until it reachs the quota >>> dirtyrate given by user. >>> >>> introduce qmp commands set-dirty-limit/cancel-dirty-limit to >>> set/cancel dirty limit on vCPU. >> >> Please start sentences with a capital letter. >> > Ok，i'll check the syntax problem next version. >>> >>> Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> >> >> >> [...] >> >>> diff --git a/qapi/misc.json b/qapi/misc.json >>> index 358548a..98e6001 100644 >>> --- a/qapi/misc.json >>> +++ b/qapi/misc.json >>> @@ -527,3 +527,42 @@ >>> 'data': { '*option': 'str' }, >>> 'returns': ['CommandLineOptionInfo'], >>> 'allow-preconfig': true } >>> + >>> +## >>> +# @set-dirty-limit: >>> +# >>> +# This command could be used to cap the vCPU memory load, which is also >>> +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" >>> +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". >>> +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used >>> +# to set the upper limit of dirtyrate for the interested vCPU. >> >> "dirtyrate" is not a word. Let's spell it "dirty page rate", for >> consistency with the documentation in migration.json. > Ok, sounds good. >> >> Regarding "One should use ...": sounds like you have to run >> calc-dirty-rate with argument @mode set to @dirty-ring before this >> command. Correct? What happens when you don't? set-dirty-limit fails? You didn't answer this question. >> Do you also have to run query-dirty-rate before this command? > Actually no, i'll clarify the usage next verison. Regarding "dirty-ring": is this merely a limitation of the implementation, i.e. other modes could be made to work if we cared, or is it more fundamental? >> Speaking of migration.json: should these commands be defined there, next >> to calc-dirty-rate and query-dirty-rate? > I'm struggling too because these commands will be used in migration but > it is vCPU they handle. I think migration.json is more about CPUs than misc.json is. Let's add the new commands to migration.json if migration maintainers are okay with it. [...]
在 2021/11/22 17:10, Markus Armbruster 写道: > Hyman Huang <huangy81@chinatelecom.cn> writes: > >> =E5=9C=A8 2021/11/22 15:35, Markus Armbruster =E5=86=99=E9=81=93: >>> huangy81@chinatelecom.cn writes: >>> >>>> From: Hyman Huang(=E9=BB=84=E5=8B=87) <huangy81@chinatelecom.cn> >>>> >>>> implement dirtyrate calculation periodically basing on >>>> dirty-ring and throttle vCPU until it reachs the quota >>>> dirtyrate given by user. >>>> >>>> introduce qmp commands set-dirty-limit/cancel-dirty-limit to >>>> set/cancel dirty limit on vCPU. >>> >>> Please start sentences with a capital letter. >>> >> Ok，i'll check the syntax problem next version. >>>> >>>> Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> >>> >>> >>> [...] >>> >>>> diff --git a/qapi/misc.json b/qapi/misc.json >>>> index 358548a..98e6001 100644 >>>> --- a/qapi/misc.json >>>> +++ b/qapi/misc.json >>>> @@ -527,3 +527,42 @@ >>>> 'data': { '*option': 'str' }, >>>> 'returns': ['CommandLineOptionInfo'], >>>> 'allow-preconfig': true } >>>> + >>>> +## >>>> +# @set-dirty-limit: >>>> +# >>>> +# This command could be used to cap the vCPU memory load, which is also >>>> +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" >>>> +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". >>>> +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used >>>> +# to set the upper limit of dirtyrate for the interested vCPU. >>> >>> "dirtyrate" is not a word. Let's spell it "dirty page rate", for >>> consistency with the documentation in migration.json. >> Ok, sounds good. >>> >>> Regarding "One should use ...": sounds like you have to run >>> calc-dirty-rate with argument @mode set to @dirty-ring before this >>> command. Correct? What happens when you don't? set-dirty-limit fails? > > You didn't answer this question. set-dirty-limit doesn't do any pre-check about if calc-dirty-rate has executed, so it doesn't fail. Since only executing calc-dirty-rate with dirty-ring mode can we get the vCPU dirty page rate currently(while the dirty-bitmap only get the vm dirty page rate), "One should use ..." maybe misleading, what i actually want to say is "One should use the dirty-ring mode to calculate the vCPU dirty page rate". > >>> Do you also have to run query-dirty-rate before this command? >> Actually no, i'll clarify the usage next verison. > > Regarding "dirty-ring": is this merely a limitation of the > implementation, i.e. other modes could be made to work if we cared, or > is it more fundamental? > >>> Speaking of migration.json: should these commands be defined there, next >>> to calc-dirty-rate and query-dirty-rate? >> I'm struggling too because these commands will be used in migration but >> it is vCPU they handle. > > I think migration.json is more about CPUs than misc.json is. Let's add > the new commands to migration.json if migration maintainers are okay > with it. > > [...] >
Hyman Huang <huangy81@chinatelecom.cn> writes: > 在 2021/11/22 17:10, Markus Armbruster 写道: >> Hyman Huang <huangy81@chinatelecom.cn> writes: >> >>> =E5=9C=A8 2021/11/22 15:35, Markus Armbruster =E5=86=99=E9=81=93: >>>> huangy81@chinatelecom.cn writes: >>>> >>>>> From: Hyman Huang(=E9=BB=84=E5=8B=87) <huangy81@chinatelecom.cn> >>>>> >>>>> implement dirtyrate calculation periodically basing on >>>>> dirty-ring and throttle vCPU until it reachs the quota >>>>> dirtyrate given by user. >>>>> >>>>> introduce qmp commands set-dirty-limit/cancel-dirty-limit to >>>>> set/cancel dirty limit on vCPU. >>>> >>>> Please start sentences with a capital letter. >>>> >>> Ok，i'll check the syntax problem next version. >>>>> >>>>> Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> >>>> >>>> >>>> [...] >>>> >>>>> diff --git a/qapi/misc.json b/qapi/misc.json >>>>> index 358548a..98e6001 100644 >>>>> --- a/qapi/misc.json >>>>> +++ b/qapi/misc.json >>>>> @@ -527,3 +527,42 @@ >>>>> 'data': { '*option': 'str' }, >>>>> 'returns': ['CommandLineOptionInfo'], >>>>> 'allow-preconfig': true } >>>>> + >>>>> +## >>>>> +# @set-dirty-limit: >>>>> +# >>>>> +# This command could be used to cap the vCPU memory load, which is also >>>>> +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" >>>>> +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". >>>>> +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used >>>>> +# to set the upper limit of dirtyrate for the interested vCPU. >>>> >>>> "dirtyrate" is not a word. Let's spell it "dirty page rate", for >>>> consistency with the documentation in migration.json. >>> Ok, sounds good. >>>> >>>> Regarding "One should use ...": sounds like you have to run >>>> calc-dirty-rate with argument @mode set to @dirty-ring before this >>>> command. Correct? What happens when you don't? set-dirty-limit fails? >> You didn't answer this question. > set-dirty-limit doesn't do any pre-check about if calc-dirty-rate has > executed, so it doesn't fail. Peeking at qmp_set_dirty_limit()... it fails when !kvm_dirty_ring_enabled(). kvm_dirty_ring_enabled() returns true when kvm_state->kvm_dirty_ring_size is non-zero. How can it become non-zero? > Since only executing calc-dirty-rate with dirty-ring mode can we get > the vCPU dirty page rate currently(while the dirty-bitmap only get the > vm dirty page rate), "One should use ..." maybe misleading, what i > actually want to say is "One should use the dirty-ring mode to > calculate the vCPU dirty page rate". I'm still confused on what exactly users must do for the page dirty rate limiting to work as intended, and at least as importantly, what happens when they get it wrong. [...]
在 2021/11/22 19:26, Markus Armbruster 写道: > Hyman Huang <huangy81@chinatelecom.cn> writes: > >> 在 2021/11/22 17:10, Markus Armbruster 写道: >>> Hyman Huang <huangy81@chinatelecom.cn> writes: >>> >>>> =E5=9C=A8 2021/11/22 15:35, Markus Armbruster =E5=86=99=E9=81=93: >>>>> huangy81@chinatelecom.cn writes: >>>>> >>>>>> From: Hyman Huang(=E9=BB=84=E5=8B=87) <huangy81@chinatelecom.cn> >>>>>> >>>>>> implement dirtyrate calculation periodically basing on >>>>>> dirty-ring and throttle vCPU until it reachs the quota >>>>>> dirtyrate given by user. >>>>>> >>>>>> introduce qmp commands set-dirty-limit/cancel-dirty-limit to >>>>>> set/cancel dirty limit on vCPU. >>>>> >>>>> Please start sentences with a capital letter. >>>>> >>>> Ok，i'll check the syntax problem next version. >>>>>> >>>>>> Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn> >>>>> >>>>> >>>>> [...] >>>>> >>>>>> diff --git a/qapi/misc.json b/qapi/misc.json >>>>>> index 358548a..98e6001 100644 >>>>>> --- a/qapi/misc.json >>>>>> +++ b/qapi/misc.json >>>>>> @@ -527,3 +527,42 @@ >>>>>> 'data': { '*option': 'str' }, >>>>>> 'returns': ['CommandLineOptionInfo'], >>>>>> 'allow-preconfig': true } >>>>>> + >>>>>> +## >>>>>> +# @set-dirty-limit: >>>>>> +# >>>>>> +# This command could be used to cap the vCPU memory load, which is also >>>>>> +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" >>>>>> +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". >>>>>> +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used >>>>>> +# to set the upper limit of dirtyrate for the interested vCPU. >>>>> >>>>> "dirtyrate" is not a word. Let's spell it "dirty page rate", for >>>>> consistency with the documentation in migration.json. >>>> Ok, sounds good. >>>>> >>>>> Regarding "One should use ...": sounds like you have to run >>>>> calc-dirty-rate with argument @mode set to @dirty-ring before this >>>>> command. Correct? What happens when you don't? set-dirty-limit fails? >>> You didn't answer this question. >> set-dirty-limit doesn't do any pre-check about if calc-dirty-rate has >> executed, so it doesn't fail. > > Peeking at qmp_set_dirty_limit()... it fails when > !kvm_dirty_ring_enabled(). kvm_dirty_ring_enabled() returns true when > kvm_state->kvm_dirty_ring_size is non-zero. How can it become non-zero? If we enable dirty-ring with qemu commandline "-accel kvm,dirty-ring-size=xxx",qemu will parse the dirty-ring-size and set it. So we check if dirty-ring is enabled by the kvm_dirty_ring_size. > >> Since only executing calc-dirty-rate with dirty-ring mode can we get >> the vCPU dirty page rate currently(while the dirty-bitmap only get the >> vm dirty page rate), "One should use ..." maybe misleading, what i >> actually want to say is "One should use the dirty-ring mode to >> calculate the vCPU dirty page rate". > > I'm still confused on what exactly users must do for the page dirty rate > limiting to work as intended, and at least as importantly, what happens > when they get it wrong. User can set-dirty-limit unconditionally and the dirtylimit will work. "One should use ..." just emphasize if users want to know which vCPU is in high memory load and want to limit it's dirty page rate, they can use calc-dirty-rate but it is not prerequisite for set-dirty-limit. Umm, I think "One should use ..." explanation make things complicated. I'll reconsider the comment next version. > > [...] >
diff --git a/cpus-common.c b/cpus-common.c index 6e73d3e..e32612b 100644 --- a/cpus-common.c +++ b/cpus-common.c @@ -23,6 +23,11 @@ #include "hw/core/cpu.h" #include "sysemu/cpus.h" #include "qemu/lockable.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/cpu-throttle.h" +#include "sysemu/kvm.h" +#include "qapi/error.h" +#include "qapi/qapi-commands-misc.h" static QemuMutex qemu_cpu_list_lock; static QemuCond exclusive_cond; @@ -352,3 +357,39 @@ void process_queued_cpu_work(CPUState *cpu) qemu_mutex_unlock(&cpu->work_mutex); qemu_cond_broadcast(&qemu_work_cond); } + +void qmp_set_dirty_limit(int64_t idx, + uint64_t dirtyrate, + Error **errp) +{ + if (!kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty ring not enable, needed by dirty restraint!"); + return; + } + + dirtylimit_calc(); + dirtylimit_vcpu(idx, dirtyrate); +} + +void qmp_cancel_dirty_limit(int64_t idx, + Error **errp) +{ + if (!kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty ring not enable, needed by dirty restraint!"); + return; + } + + if (unlikely(!dirtylimit_cancel_vcpu(idx))) { + dirtylimit_calc_quit(); + } +} + +void dirtylimit_setup(int max_cpus) +{ + if (!kvm_dirty_ring_enabled()) { + return; + } + + dirtylimit_calc_state_init(max_cpus); + dirtylimit_state_init(max_cpus); +} diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index e948e81..11df012 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -881,6 +881,15 @@ void end_exclusive(void); */ void qemu_init_vcpu(CPUState *cpu); +/** + * dirtylimit_setup: + * + * Initializes the global state of dirtylimit calculation and + * dirtylimit itself. This is prepared for vCPU dirtylimit which + * could be triggered during vm lifecycle. + */ +void dirtylimit_setup(int max_cpus); + #define SSTEP_ENABLE 0x1 /* Enable simulated HW single stepping */ #define SSTEP_NOIRQ 0x2 /* Do not use IRQ while single stepping */ #define SSTEP_NOTIMER 0x4 /* Do not Timers while single stepping */ diff --git a/qapi/misc.json b/qapi/misc.json index 358548a..98e6001 100644 --- a/qapi/misc.json +++ b/qapi/misc.json @@ -527,3 +527,42 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true } + +## +# @set-dirty-limit: +# +# This command could be used to cap the vCPU memory load, which is also +# refered as dirtyrate. One should use "calc-dirty-rate" with "dirty-ring" +# and to calculate vCPU dirtyrate and query it with "query-dirty-rate". +# Once getting the vCPU current dirtyrate, "set-dirty-limit" can be used +# to set the upper limit of dirtyrate for the interested vCPU. +# +# @idx: vCPU index to set dirtylimit. +# +# @dirtyrate: upper limit of drityrate the specified vCPU could reach (MB/s) +# +# Since: 6.3 +# +# Example: +# {"execute": "set-dirty-limit"} +# "arguments": { "idx": 0, +# "dirtyrate": 200 } } +# +## +{ 'command': 'set-dirty-limit', + 'data': { 'idx': 'int', 'dirtyrate': 'uint64' } } + +## +# @cancel-dirty-limit: +# +# @idx: vCPU index to canceled the dirtylimit +# +# Since: 6.3 +# +# Example: +# {"execute": "cancel-dirty-limit"} +# "arguments": { "idx": 0 } } +# +## +{ 'command': 'cancel-dirty-limit', + 'data': { 'idx': 'int' } } diff --git a/softmmu/vl.c b/softmmu/vl.c index 1159a64..170ee23 100644 --- a/softmmu/vl.c +++ b/softmmu/vl.c @@ -3776,5 +3776,6 @@ void qemu_init(int argc, char **argv, char **envp) qemu_init_displays(); accel_setup_post(current_machine); os_setup_post(); + dirtylimit_setup(current_machine->smp.max_cpus); resume_mux_open(); }