| Message ID | 1443418711-24106-4-git-send-email-haozhong.zhang@intel.com |
|---|---|
| State | New |
| Headers | show |
On Mon, Sep 28, 2015 at 01:38:31PM +0800, Haozhong Zhang wrote: > When a vcpu is created in KVM, its TSC rate is initially identical to > the host TSC rate. If its state is migrated to a vcpu on another > machine (target machine) which may uses a different host TSC rate, QEMU > on the target machine should notice KVM of the migrated vcpu's TSC > rate. In case that KVM on the target machine supports TSC scaling, guest > programs running on the migrated vcpu will observe the same TSC rate > before and after the migration. > > Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com> > --- > kvm-all.c | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/kvm-all.c b/kvm-all.c > index 0be4615..e8de038 100644 > --- a/kvm-all.c > +++ b/kvm-all.c > @@ -1769,6 +1769,19 @@ void kvm_cpu_synchronize_post_reset(CPUState *cpu) > static void do_kvm_cpu_synchronize_post_init(void *arg) > { > CPUState *cpu = arg; > + CPUX86State *env = &X86_CPU(cpu)->env; > + int r; > + > + /* > + * XXX: KVM_SET_TSC_KHZ must be done before kvm_arch_put_registers(). Could you explain where this requirement comes from? > + */ > + r = kvm_check_extension(cpu->kvm_state, KVM_CAP_TSC_CONTROL); > + if (r && env->tsc_khz) { > + r = kvm_vcpu_ioctl(cpu, KVM_SET_TSC_KHZ, env->tsc_khz); > + if (r < 0) { > + fprintf(stderr, "KVM_SET_TSC_KHZ failed\n"); > + } > + } This is duplicating the existing KVM_SET_TSC_KHZ call at kvm_arch_init_vcpu(). I wonder if there's a way to avoid this duplication. Should we set TSC KHz only at do_kvm_cpu_synchronize_post_init(), and remove the call from kvm_arch_init_vcpu()? Or maybe we shouldn't treat this as VM state, but as configuration, and let management configure the TSC frequency explicitly if the user really needs it to stay the same during migration. (CCing libvir-list to see if they have feedback)
On Mon, Sep 28, 2015 at 01:37:34PM -0300, Eduardo Habkost wrote: > On Mon, Sep 28, 2015 at 01:38:31PM +0800, Haozhong Zhang wrote: > > When a vcpu is created in KVM, its TSC rate is initially identical to > > the host TSC rate. If its state is migrated to a vcpu on another > > machine (target machine) which may uses a different host TSC rate, QEMU > > on the target machine should notice KVM of the migrated vcpu's TSC > > rate. In case that KVM on the target machine supports TSC scaling, guest > > programs running on the migrated vcpu will observe the same TSC rate > > before and after the migration. > > > > Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com> > > --- > > kvm-all.c | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > > > diff --git a/kvm-all.c b/kvm-all.c > > index 0be4615..e8de038 100644 > > --- a/kvm-all.c > > +++ b/kvm-all.c > > @@ -1769,6 +1769,19 @@ void kvm_cpu_synchronize_post_reset(CPUState *cpu) > > static void do_kvm_cpu_synchronize_post_init(void *arg) > > { > > CPUState *cpu = arg; > > + CPUX86State *env = &X86_CPU(cpu)->env; > > + int r; > > + > > + /* > > + * XXX: KVM_SET_TSC_KHZ must be done before kvm_arch_put_registers(). > > Could you explain where this requirement comes from? > kvm_arch_put_registers() below will setup vcpu's MSR_IA32_TSC through KVM ioctl KVM_SET_MSRS. KVM needs to know vcpu's TSC rate so as to correctly scale the TSC value given by QEMU, especially when vcpu's TSC rate is different than the host TSC rate (e.g. it's migrated from another machine w/ different host TSC rate than the current one). > > + */ > > + r = kvm_check_extension(cpu->kvm_state, KVM_CAP_TSC_CONTROL); > > + if (r && env->tsc_khz) { > > + r = kvm_vcpu_ioctl(cpu, KVM_SET_TSC_KHZ, env->tsc_khz); > > + if (r < 0) { > > + fprintf(stderr, "KVM_SET_TSC_KHZ failed\n"); > > + } > > + } > > This is duplicating the existing KVM_SET_TSC_KHZ call at > kvm_arch_init_vcpu(). I wonder if there's a way to avoid this > duplication. Should we set TSC KHz only at > do_kvm_cpu_synchronize_post_init(), and remove the call from > kvm_arch_init_vcpu()? > I'll check if it's safe to remove the call from kvm_arch_init_vcpu(). > Or maybe we shouldn't treat this as VM state, but as configuration, and > let management configure the TSC frequency explicitly if the user really > needs it to stay the same during migration. > > (CCing libvir-list to see if they have feedback) > Thanks for CC. I'll consider to add a command line option to control the configuration of guest TSC fequency. > -- > Eduardo
On Tue, Sep 29, 2015 at 11:43:34AM +0800, Haozhong Zhang wrote: > On Mon, Sep 28, 2015 at 01:37:34PM -0300, Eduardo Habkost wrote: > > On Mon, Sep 28, 2015 at 01:38:31PM +0800, Haozhong Zhang wrote: [...] > > > static void do_kvm_cpu_synchronize_post_init(void *arg) > > > { > > > CPUState *cpu = arg; > > > + CPUX86State *env = &X86_CPU(cpu)->env; > > > + int r; > > > + > > > + /* > > > + * XXX: KVM_SET_TSC_KHZ must be done before kvm_arch_put_registers(). > > > > Could you explain where this requirement comes from? > > > > kvm_arch_put_registers() below will setup vcpu's MSR_IA32_TSC through > KVM ioctl KVM_SET_MSRS. KVM needs to know vcpu's TSC rate so as to > correctly scale the TSC value given by QEMU, especially when vcpu's > TSC rate is different than the host TSC rate (e.g. it's migrated from > another machine w/ different host TSC rate than the current one). Thanks. The comment above could contain a short version of this explanation, e.g.: "KVM needs KVM_SET_TSC_KHZ to be done before KVM_SET_MSRS sets MSR_IA32_TSC (done by kvm_arch_put_registers())". > [...] > > Or maybe we shouldn't treat this as VM state, but as configuration, and > > let management configure the TSC frequency explicitly if the user really > > needs it to stay the same during migration. > > > > (CCing libvir-list to see if they have feedback) > > > > Thanks for CC. I'll consider to add a command line option to control > the configuration of guest TSC fequency. It already exists, -cpu has a "tsc-freq" option.
On Tue, Sep 29, 2015 at 03:02:07PM -0300, Eduardo Habkost wrote: > On Tue, Sep 29, 2015 at 11:43:34AM +0800, Haozhong Zhang wrote: > > On Mon, Sep 28, 2015 at 01:37:34PM -0300, Eduardo Habkost wrote: > > > On Mon, Sep 28, 2015 at 01:38:31PM +0800, Haozhong Zhang wrote: > [...] > > > > static void do_kvm_cpu_synchronize_post_init(void *arg) > > > > { > > > > CPUState *cpu = arg; > > > > + CPUX86State *env = &X86_CPU(cpu)->env; > > > > + int r; > > > > + > > > > + /* > > > > + * XXX: KVM_SET_TSC_KHZ must be done before kvm_arch_put_registers(). > > > > > > Could you explain where this requirement comes from? > > > > > > > kvm_arch_put_registers() below will setup vcpu's MSR_IA32_TSC through > > KVM ioctl KVM_SET_MSRS. KVM needs to know vcpu's TSC rate so as to > > correctly scale the TSC value given by QEMU, especially when vcpu's > > TSC rate is different than the host TSC rate (e.g. it's migrated from > > another machine w/ different host TSC rate than the current one). > > Thanks. The comment above could contain a short version of this > explanation, e.g.: "KVM needs KVM_SET_TSC_KHZ to be done before > KVM_SET_MSRS sets MSR_IA32_TSC (done by kvm_arch_put_registers())". > will include this in the comment > > > [...] > > > Or maybe we shouldn't treat this as VM state, but as configuration, and > > > let management configure the TSC frequency explicitly if the user really > > > needs it to stay the same during migration. > > > > > > (CCing libvir-list to see if they have feedback) > > > > > > > Thanks for CC. I'll consider to add a command line option to control > > the configuration of guest TSC fequency. > > It already exists, -cpu has a "tsc-freq" option. > What I'm considering is to add a "-keep-tsc-freq" option to control whether the TSC frequency should be migrated if "tsc-freq" is not presented. Compared to "tsc-freq", "-keep-tsc-freq" can free users from figuring out the guest TSC frequency manually in the migration. - Haozhong > -- > Eduardo > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 30, 2015 at 08:32:26AM +0800, Haozhong Zhang wrote: > > [...] > > > > Or maybe we shouldn't treat this as VM state, but as configuration, and > > > > let management configure the TSC frequency explicitly if the user really > > > > needs it to stay the same during migration. > > > > > > > > (CCing libvir-list to see if they have feedback) > > > > > > > > > > Thanks for CC. I'll consider to add a command line option to control > > > the configuration of guest TSC fequency. > > > > It already exists, -cpu has a "tsc-freq" option. > > > > What I'm considering is to add a "-keep-tsc-freq" option to control > whether the TSC frequency should be migrated if "tsc-freq" is not > presented. Compared to "tsc-freq", "-keep-tsc-freq" can free users > from figuring out the guest TSC frequency manually in the migration. If you do that, please make it a property of the CPU object, so it will can be set as a "-cpu" option.
On Wed, Sep 30, 2015 at 05:36:11PM -0300, Eduardo Habkost wrote: > On Wed, Sep 30, 2015 at 08:32:26AM +0800, Haozhong Zhang wrote: > > > [...] > > > > > Or maybe we shouldn't treat this as VM state, but as configuration, and > > > > > let management configure the TSC frequency explicitly if the user really > > > > > needs it to stay the same during migration. > > > > > > > > > > (CCing libvir-list to see if they have feedback) > > > > > > > > > > > > > Thanks for CC. I'll consider to add a command line option to control > > > > the configuration of guest TSC fequency. > > > > > > It already exists, -cpu has a "tsc-freq" option. > > > > > > > What I'm considering is to add a "-keep-tsc-freq" option to control > > whether the TSC frequency should be migrated if "tsc-freq" is not > > presented. Compared to "tsc-freq", "-keep-tsc-freq" can free users > > from figuring out the guest TSC frequency manually in the migration. > > If you do that, please make it a property of the CPU object, so it will > can be set as a "-cpu" option. > Yes, I'll do so. - Haozhong > -- > Eduardo > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/kvm-all.c b/kvm-all.c index 0be4615..e8de038 100644 --- a/kvm-all.c +++ b/kvm-all.c @@ -1769,6 +1769,19 @@ void kvm_cpu_synchronize_post_reset(CPUState *cpu) static void do_kvm_cpu_synchronize_post_init(void *arg) { CPUState *cpu = arg; + CPUX86State *env = &X86_CPU(cpu)->env; + int r; + + /* + * XXX: KVM_SET_TSC_KHZ must be done before kvm_arch_put_registers(). + */ + r = kvm_check_extension(cpu->kvm_state, KVM_CAP_TSC_CONTROL); + if (r && env->tsc_khz) { + r = kvm_vcpu_ioctl(cpu, KVM_SET_TSC_KHZ, env->tsc_khz); + if (r < 0) { + fprintf(stderr, "KVM_SET_TSC_KHZ failed\n"); + } + } kvm_arch_put_registers(cpu, KVM_PUT_FULL_STATE); cpu->kvm_vcpu_dirty = false;
When a vcpu is created in KVM, its TSC rate is initially identical to the host TSC rate. If its state is migrated to a vcpu on another machine (target machine) which may uses a different host TSC rate, QEMU on the target machine should notice KVM of the migrated vcpu's TSC rate. In case that KVM on the target machine supports TSC scaling, guest programs running on the migrated vcpu will observe the same TSC rate before and after the migration. Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com> --- kvm-all.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)