[v7,1/5] cpu: Provide vcpu throttling interface

Provide a method to throttle guest cpu execution. CPUState is augmented with
timeout controls and throttle start/stop functions. To throttle the guest cpu
the caller simply has to call the throttle set function and provide a percentage
of throttle time.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
---
 cpus.c            | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/qom/cpu.h | 42 ++++++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)

On 08/09/2015 19:12, Jason J. Herne wrote:
> Provide a method to throttle guest cpu execution. CPUState is augmented with
> timeout controls and throttle start/stop functions. To throttle the guest cpu
> the caller simply has to call the throttle set function and provide a percentage
> of throttle time.
> 
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Juan, please merge through your tree.

Paolo

> ---
>  cpus.c            | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  include/qom/cpu.h | 42 ++++++++++++++++++++++++++++++
>  2 files changed, 120 insertions(+)
> 
> diff --git a/cpus.c b/cpus.c
> index de6469f..b5ff9c9 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -68,6 +68,14 @@ static CPUState *next_cpu;
>  int64_t max_delay;
>  int64_t max_advance;
>  
> +/* vcpu throttling controls */
> +static QEMUTimer *throttle_timer;
> +static unsigned int throttle_percentage;
> +
> +#define CPU_THROTTLE_PCT_MIN 1
> +#define CPU_THROTTLE_PCT_MAX 99
> +#define CPU_THROTTLE_TIMESLICE_NS 10000000
> +
>  bool cpu_is_stopped(CPUState *cpu)
>  {
>      return cpu->stopped || !runstate_is_running();
> @@ -486,10 +494,80 @@ static const VMStateDescription vmstate_timers = {
>      }
>  };
>  
> +static void cpu_throttle_thread(void *opaque)
> +{
> +    CPUState *cpu = opaque;
> +    double pct;
> +    double throttle_ratio;
> +    long sleeptime_ns;
> +
> +    if (!cpu_throttle_get_percentage()) {
> +        return;
> +    }
> +
> +    pct = (double)cpu_throttle_get_percentage()/100;
> +    throttle_ratio = pct / (1 - pct);
> +    sleeptime_ns = (long)(throttle_ratio * CPU_THROTTLE_TIMESLICE_NS);
> +
> +    qemu_mutex_unlock_iothread();
> +    atomic_set(&cpu->throttle_thread_scheduled, 0);
> +    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
> +    qemu_mutex_lock_iothread();
> +}
> +
> +static void cpu_throttle_timer_tick(void *opaque)
> +{
> +    CPUState *cpu;
> +    double pct;
> +
> +    /* Stop the timer if needed */
> +    if (!cpu_throttle_get_percentage()) {
> +        return;
> +    }
> +    CPU_FOREACH(cpu) {
> +        if (!atomic_xchg(&cpu->throttle_thread_scheduled, 1)) {
> +            async_run_on_cpu(cpu, cpu_throttle_thread, cpu);
> +        }
> +    }
> +
> +    pct = (double)cpu_throttle_get_percentage()/100;
> +    timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) +
> +                                   CPU_THROTTLE_TIMESLICE_NS / (1-pct));
> +}
> +
> +void cpu_throttle_set(int new_throttle_pct)
> +{
> +    /* Ensure throttle percentage is within valid range */
> +    new_throttle_pct = MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX);
> +    new_throttle_pct = MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN);
> +
> +    atomic_set(&throttle_percentage, new_throttle_pct);
> +
> +    timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) +
> +                                       CPU_THROTTLE_TIMESLICE_NS);
> +}
> +
> +void cpu_throttle_stop(void)
> +{
> +    atomic_set(&throttle_percentage, 0);
> +}
> +
> +bool cpu_throttle_active(void)
> +{
> +    return (cpu_throttle_get_percentage() != 0);
> +}
> +
> +int cpu_throttle_get_percentage(void)
> +{
> +    return atomic_read(&throttle_percentage);
> +}
> +
>  void cpu_ticks_init(void)
>  {
>      seqlock_init(&timers_state.vm_clock_seqlock, NULL);
>      vmstate_register(NULL, 0, &vmstate_timers, &timers_state);
> +    throttle_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT,
> +                                           cpu_throttle_timer_tick, NULL);
>  }
>  
>  void configure_icount(QemuOpts *opts, Error **errp)
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 39f0f19..db6ec1e 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -310,6 +310,11 @@ struct CPUState {
>      uint32_t can_do_io;
>      int32_t exception_index; /* used by m68k TCG */
>  
> +    /* Used to keep track of an outstanding cpu throttle thread for migration
> +     * autoconverge
> +     */
> +    bool throttle_thread_scheduled;
> +
>      /* Note that this is accessed at the start of every TB via a negative
>         offset from AREG0.  Leave this field at the end so as to make the
>         (absolute value) offset as small as possible.  This reduces code
> @@ -553,6 +558,43 @@ CPUState *qemu_get_cpu(int index);
>   */
>  bool cpu_exists(int64_t id);
>  
> +/**
> + * cpu_throttle_set:
> + * @new_throttle_pct: Percent of sleep time. Valid range is 1 to 99.
> + *
> + * Throttles all vcpus by forcing them to sleep for the given percentage of
> + * time. A throttle_percentage of 25 corresponds to a 75% duty cycle roughly.
> + * (example: 10ms sleep for every 30ms awake).
> + *
> + * cpu_throttle_set can be called as needed to adjust new_throttle_pct.
> + * Once the throttling starts, it will remain in effect until cpu_throttle_stop
> + * is called.
> + */
> +void cpu_throttle_set(int new_throttle_pct);
> +
> +/**
> + * cpu_throttle_stop:
> + *
> + * Stops the vcpu throttling started by cpu_throttle_set.
> + */
> +void cpu_throttle_stop(void);
> +
> +/**
> + * cpu_throttle_active:
> + *
> + * Returns: %true if the vcpus are currently being throttled, %false otherwise.
> + */
> +bool cpu_throttle_active(void);
> +
> +/**
> + * cpu_throttle_get_percentage:
> + *
> + * Returns the vcpu throttle percentage. See cpu_throttle_set for details.
> + *
> + * Returns: The throttle percentage in range 1 to 99.
> + */
> +int cpu_throttle_get_percentage(void);
> +
>  #ifndef CONFIG_USER_ONLY
>  
>  typedef void (*CPUInterruptHandler)(CPUState *, int);
>

"Jason J. Herne" <jjherne@linux.vnet.ibm.com> wrote:
> Provide a method to throttle guest cpu execution. CPUState is augmented with
> timeout controls and throttle start/stop functions. To throttle the guest cpu
> the caller simply has to call the throttle set function and provide a percentage
> of throttle time.
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
>  cpus.c            | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  include/qom/cpu.h | 42 ++++++++++++++++++++++++++++++
>  2 files changed, 120 insertions(+)
>
> diff --git a/cpus.c b/cpus.c
> index de6469f..b5ff9c9 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -68,6 +68,14 @@ static CPUState *next_cpu;
>  int64_t max_delay;
>  int64_t max_advance;
>  
> +/* vcpu throttling controls */
> +static QEMUTimer *throttle_timer;
> +static unsigned int throttle_percentage;
> +
> +#define CPU_THROTTLE_PCT_MIN 1
> +#define CPU_THROTTLE_PCT_MAX 99
> +#define CPU_THROTTLE_TIMESLICE_NS 10000000
> +
>  bool cpu_is_stopped(CPUState *cpu)
>  {
>      return cpu->stopped || !runstate_is_running();
> @@ -486,10 +494,80 @@ static const VMStateDescription vmstate_timers = {
>      }
>  };
>  
> +static void cpu_throttle_thread(void *opaque)
> +{
> +    CPUState *cpu = opaque;
> +    double pct;
> +    double throttle_ratio;
> +    long sleeptime_ns;
> +
> +    if (!cpu_throttle_get_percentage()) {

cpu_throotle_active()?


> +        return;
> +    }
> +
> +    pct = (double)cpu_throttle_get_percentage()/100;
> +    throttle_ratio = pct / (1 - pct);
> +    sleeptime_ns = (long)(throttle_ratio * CPU_THROTTLE_TIMESLICE_NS);
> +
> +    qemu_mutex_unlock_iothread();
> +    atomic_set(&cpu->throttle_thread_scheduled, 0);
> +    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
> +    qemu_mutex_lock_iothread();


Why is this thread safe?

qemu_mutex_lock_iothread() is protecting (at least) cpu_work_first on
each cpu.  How can we be sure that _nothing_ will change that while we
are waiting?   A fast look through the tree don't show anything that
runs here that drops the lock.  I am missing something?

> +}
> +
> +static void cpu_throttle_timer_tick(void *opaque)
> +{
> +    CPUState *cpu;
> +    double pct;
> +
> +    /* Stop the timer if needed */
> +    if (!cpu_throttle_get_percentage()) {

cpu_throotle_active()?

agree with rest of the changes.

Later, Juan.

Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 08/09/2015 19:12, Jason J. Herne wrote:
>> Provide a method to throttle guest cpu execution. CPUState is augmented with
>> timeout controls and throttle start/stop functions. To throttle the guest cpu
>> the caller simply has to call the throttle set function and provide a percentage
>> of throttle time.
>> 
>> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
>> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
>
> Juan, please merge through your tree.

Could you answer to my locking problem on the 1st patch?  I can't see
why we can drop the iothread lock there (for one eternity).  I am
missing something obvious ...

On 09/09/2015 12:41, Juan Quintela wrote:
>> > +    qemu_mutex_unlock_iothread();
>> > +    atomic_set(&cpu->throttle_thread_scheduled, 0);
>> > +    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
>> > +    qemu_mutex_lock_iothread();
> 
> Why is this thread safe?
> 
> qemu_mutex_lock_iothread() is protecting (at least) cpu_work_first on
> each cpu.  How can we be sure that _nothing_ will change that while we
> are waiting?

You only have to be sure that the queued work list remains consistent;
not that nothing changes.

(BTW, there is a queued patch that moves the queued work list to its own
mutex, and indeed it releases that mutex while calling the work function).

> A fast look through the tree don't show anything that
> runs here that drops the lock.

Actually, the existing implementation of throttling does. :)

Paolo

Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 09/09/2015 12:41, Juan Quintela wrote:
>>> > +    qemu_mutex_unlock_iothread();
>>> > +    atomic_set(&cpu->throttle_thread_scheduled, 0);
>>> > +    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
>>> > +    qemu_mutex_lock_iothread();
>> 
>> Why is this thread safe?
>> 
>> qemu_mutex_lock_iothread() is protecting (at least) cpu_work_first on
>> each cpu.  How can we be sure that _nothing_ will change that while we
>> are waiting?
>
> You only have to be sure that the queued work list remains consistent;
> not that nothing changes.


But nothing else is protected by the iothread?  That is the part that I
can't see.

> (BTW, there is a queued patch that moves the queued work list to its own
> mutex, and indeed it releases that mutex while calling the work function).


>> A fast look through the tree don't show anything that
>> runs here that drops the lock.
>
> Actually, the existing implementation of throttling does. :)

See, that happens when you search in a modified tree O:-)

Thanks for the fast answer.

Later, Juan.

On 09/09/2015 13:01, Juan Quintela wrote:
> Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 09/09/2015 12:41, Juan Quintela wrote:
>>>>> +    qemu_mutex_unlock_iothread();
>>>>> +    atomic_set(&cpu->throttle_thread_scheduled, 0);
>>>>> +    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
>>>>> +    qemu_mutex_lock_iothread();
>>>
>>> Why is this thread safe?
>>>
>>> qemu_mutex_lock_iothread() is protecting (at least) cpu_work_first on
>>> each cpu.  How can we be sure that _nothing_ will change that while we
>>> are waiting?
>>
>> You only have to be sure that the queued work list remains consistent;
>> not that nothing changes.
> 
> 
> But nothing else is protected by the iothread?

Not at this point.  Notice how qemu_kvm_wait_io_event calls
qemu_cond_wait just before qemu_wait_io_event_common (which in turn is
what calls flush_queued_work).

So you can be quite sure that qemu_wait_io_event_common runs at a point
where there's nothing hidden that relies on the iothread mutex.

Paolo