[v3] docs: add cpu-hotplug.txt

This document describes how to use cpu hotplug in QEMU.

Signed-off-by: Dou Liyang <douly.fnst@cn.fujitsu.com>
---
Change log v2 -> v3:
  From drew's advice:
    1. modify the examples.
    2. Fix some syntax.

Change log v1 -> v2:
  From Fam's advice:
    1. Fix some comment.

Change log v1:
  From Igor's advice:
    1. Remove any mentioning of apic-id from the document.
    2. Remove the "device_del qom_path" from the CPU hot-unplug.
    3. Fix some comment.

 docs/cpu-hotplug.txt | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)
 create mode 100644 docs/cpu-hotplug.txt

On Mon, Aug 15, 2016 at 01:49:48PM +0800, Dou Liyang wrote:
> This document describes how to use cpu hotplug in QEMU.
> 
> Signed-off-by: Dou Liyang <douly.fnst@cn.fujitsu.com>
> ---
> Change log v2 -> v3:
>   From drew's advice:
>     1. modify the examples.
>     2. Fix some syntax.
> 
> Change log v1 -> v2:
>   From Fam's advice:
>     1. Fix some comment.
> 
> Change log v1:
>   From Igor's advice:
>     1. Remove any mentioning of apic-id from the document.
>     2. Remove the "device_del qom_path" from the CPU hot-unplug.
>     3. Fix some comment.
> 
>  docs/cpu-hotplug.txt | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 127 insertions(+)
>  create mode 100644 docs/cpu-hotplug.txt
> 
> diff --git a/docs/cpu-hotplug.txt b/docs/cpu-hotplug.txt
> new file mode 100644
> index 0000000..892505f
> --- /dev/null
> +++ b/docs/cpu-hotplug.txt
> @@ -0,0 +1,127 @@
> +QEMU CPU hotplug
> +================
> +
> +This document explains how to use the CPU hotplug feature in QEMU,
> +which regards the CPU as a device, using -device/device_add and
> +device_del.
> +
> +QEMU support was merged for 2.7.
> +
> +Guest support is required for CPU hotplug to work.
> +
> +CPU hot-plug
> +------------
> +
> +In order to be able to hotplug CPUs, QEMU has to be told the maximum
> +number of CPUs which the guest can have. This is done at startup time
> +by means of the -smp command-line option, which has the following
> +format:
> +
> + -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads]
> +	[,sockets=sockets]
> +
> +Where,
> +
> + - "cpus"    sets the number of CPUs to 'n' [default=1].
> + - "maxcpus" sets the maximum number of CPUs, including offline VCPUs
> +   for hotplug, etc.

Not sure what the 'etc.' is for. Drop it if there's nothing else,
or state explicitly what the other reason(s) are.

> + - "cores"   sets the number of CPU cores on one socket.
> + - "threads" sets the number of threads on one CPU core.
> + - "sockets" sets the number of discrete sockets in the system. On
> +   sPAPR, sockets have no real meaning, And it has no real effect.

Please correct this list to be in either ascending or descending
hierarchical order; i.e. sockets,cores,threads or threads,cores,sockets

I think the sPAPR note should be in ()

> +
> +For example, the following command-line:
> +
> + qemu [...] -smp 3,maxcpus=8,sockets=2,cores=2,threads=2
> +
> +Creates a guest with 3 VCPUs and it supports up to 8 VCPUs. The

lowercase c on Creates, as we're continuing the above sentence.

/it supports/supports/

> +CPU topology is sockets (2) * cores (2) * threads (2) and can't be
> +greater than maxcpus. When the guest is booted, the guest will see

The CPU topology, sockets (2) * cores (2) * threads (2), allows for
8 CPU slots, which exactly matches maxcpus.

/booted/boot/


> +3 VCPUs. More on this below.
> +
> +Query possible available CPU objects

/possible//

> +------------------------------------
> +
> +The VCPUs should be hotplugged by socket/core/thread-id parameters

, after parameters

> +describing the available CPU objects.
> +
> +Before adding the VCPUs, we should know those topology parameters,
> +so that we can find the available location (socket,core,thread) for
> +a new VCPU.
> +
> +There are two ways to obtain them:
> +
> +1. Using the properties advertised by QEMU via the QMP command
> +"query-hotpluggable-cpus".

For enumerated lists please space additional lines under the start
of the text, e.g.

1. This line is too long and therefore needs to be continued on the
   next line.

> +2. Using the corresponding HMP command "info hotpluggable-cpus".
> +
> +For example, a monitor command can be used to list the possible CPU
> +objects:
> +
> +  (qemu) info hotpluggable-cpus
> +
> +Select the hotpluggable CPUs including "CPUInstance Properties" for
> +hotpluging. Such as this:

hotplugging 

> +
> +  ...
> +  type: "qemu64-x86_64-cpu"
> +  vcpus_count: "1"
> +  CPUInstance Properties:
> +    socket-id: "0"
> +    core-id: "1"
> +    thread-id: "0"
> +  ...
> +
> +Hotplug CPUs
> +------------
> +
> +A monitor command can be used to hotplug CPUs:
> +
> + - "device_add": creates a VCPU device and inserts it into the
> +	specific place as a device
> +
> +For example, the following command adds a VCPU which has the id cpu1
> +in the specific position of the guest's CPU sockets which was discussed
> +earlier by using the socket/core/thread-id:

No need to refer to the previous example. It makes no difference in this
example. I suggest just

 For example, the following command adds a VCPU, which has the id cpu1,
 to a specific location in the topology (socket=0,core=1,thread=1):

> +
> +  (qemu) device_add qemu64-x86_64-cpu,id=cpu1,socket-id=0,core-id=1,thread-id=1
> +
> +Where,
> +
> + - "qemu64-x86_64-cpu" is the CPU model.
> + - "id" is the unique identifier in the device set. It is required.
> + - "socket-id/core-id/thread-id" represent the designated position

/position/location,/

> +   which is obtained form the above possible list of CPUs.
> +
> +It's also possible to start a guest with a cpu cold-plugged into a
> +specific location (socket,core,thread).
> +
> +In the following command line example, a guest which has 3 VCPUs is
> +created:
> +
> + qemu  [...] -smp 1,maxcpus=8,sockets=2,cores=2,threads=2 \
> +	-device qemu64-x86_64-cpu,id=cpu1,socket-id=1, \
> +	core-id=1,thread-id=0 \
> +	-device qemu64-x86_64-cpu,id=cpu2,socket-id=1, \
> +	core-id=1,thread-id=1 \
> +
> +Two VCPUs are cold-plugged by "-device" parameter, which are in the
> +same socket and core but with different thread-ids. After that, the

, after core

> +guest has an additional five VCPUs to be hot-plugged when needed.
> +
> +CPU hot-unplug
> +--------------
> +
> +In order to be able to hot unplug a CPU device, QEMU removes CPU
> +devices by using the ids which were assigned when hotplugging the
> +CPU device.
> +
> +A monitor command can be used to hot unplug CPUs:
> +
> + - "device_del": deletes a CPU device
> +
> +For example, assuming that the CPU device with id "cpu1" exists and has
> +been offline, the following command tries to remove it.

 For example, assuming that a CPU device with an id "cpu1" exists, and has
 been offlined, then the following command tries to remove it:

> +
> +  (qemu) device_del cpu1
> +
> -- 
> 2.5.5

Thanks,
drew

On Mon, Aug 15, 2016 at 01:49:48PM +0800, Dou Liyang wrote:
> This document describes how to use cpu hotplug in QEMU.
> 
> Signed-off-by: Dou Liyang <douly.fnst@cn.fujitsu.com>
> ---
> Change log v2 -> v3:
>   From drew's advice:
>     1. modify the examples.
>     2. Fix some syntax.
> 
> Change log v1 -> v2:
>   From Fam's advice:
>     1. Fix some comment.
> 
> Change log v1:
>   From Igor's advice:
>     1. Remove any mentioning of apic-id from the document.
>     2. Remove the "device_del qom_path" from the CPU hot-unplug.
>     3. Fix some comment.
> 
>  docs/cpu-hotplug.txt | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 127 insertions(+)
>  create mode 100644 docs/cpu-hotplug.txt
> 
> diff --git a/docs/cpu-hotplug.txt b/docs/cpu-hotplug.txt
> new file mode 100644
> index 0000000..892505f
> --- /dev/null
> +++ b/docs/cpu-hotplug.txt
> @@ -0,0 +1,127 @@
> +QEMU CPU hotplug
> +================
> +
> +This document explains how to use the CPU hotplug feature in QEMU,
> +which regards the CPU as a device, using -device/device_add and
> +device_del.
> +
> +QEMU support was merged for 2.7.
> +
> +Guest support is required for CPU hotplug to work.
> +
> +CPU hot-plug
> +------------
> +
> +In order to be able to hotplug CPUs, QEMU has to be told the maximum
> +number of CPUs which the guest can have. This is done at startup time
> +by means of the -smp command-line option, which has the following
> +format:
> +
> + -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads]
> +	[,sockets=sockets]
> +
> +Where,
> +
> + - "cpus"    sets the number of CPUs to 'n' [default=1].
> + - "maxcpus" sets the maximum number of CPUs, including offline VCPUs
> +   for hotplug, etc.
> + - "cores"   sets the number of CPU cores on one socket.
> + - "threads" sets the number of threads on one CPU core.
> + - "sockets" sets the number of discrete sockets in the system. On
> +   sPAPR, sockets have no real meaning, And it has no real effect.

While the comment about sockets on sPAPR is correct, it's not
particularly important, and I think just confuses the issue this early
in the document.  At the qemu level setting sockets= does have an
effect in terms of how various things are presented, it just doesn't
have a meaningful effect on the guest.

> +For example, the following command-line:
> +
> + qemu [...] -smp 3,maxcpus=8,sockets=2,cores=2,threads=2
> +
> +Creates a guest with 3 VCPUs and it supports up to 8 VCPUs. The
> +CPU topology is sockets (2) * cores (2) * threads (2) and can't be
> +greater than maxcpus. When the guest is booted, the guest will see
> +3 VCPUs. More on this below.
> +
> +Query possible available CPU objects
> +------------------------------------
> +
> +The VCPUs should be hotplugged by socket/core/thread-id parameters
> +describing the available CPU objects.

This isn't very clear to me.  Perhaps:

    To add a VCPU, it must be identified by socket-id, core-id and/or
    thread-id parameters.

> +Before adding the VCPUs, we should know those topology parameters,
> +so that we can find the available location (socket,core,thread) for
> +a new VCPU.
> +
> +There are two ways to obtain them:
> +
> +1. Using the properties advertised by QEMU via the QMP command
> +"query-hotpluggable-cpus".

For starters, we really want to discourage guessing CPU parameters
without query-hotpluggable-cpus.  But worse, even now this method
won't work:
   - You have to know details of the machine type to know which of the
     soket/core/thread parmaeters are necessary and/or available
   - socket-id and core-id, even thread-id aren't necessarily
     contiguous and can vary from system to system.  For example on a
     2 core x 2 thread system on x86 I believe there will be:
		core 0, thread 0
     		core 0, thread 1
		core 1, thread 0
		core 1, thread 1
     But on sPAPR it will be:
     		core 0, thread 0
		core 0, thread 1
		core 2, thread 0
		core 2, thread 1
    (because we always match the core-id to the cpu_index of that
    core's thread 0).

In short, only method (2) should every be suggested.

> +2. Using the corresponding HMP command "info hotpluggable-cpus".
> +
> +For example, a monitor command can be used to list the possible CPU
> +objects:
> +
> +  (qemu) info hotpluggable-cpus
> +
> +Select the hotpluggable CPUs including "CPUInstance Properties" for
> +hotpluging. Such as this:
> +
> +  ...
> +  type: "qemu64-x86_64-cpu"
> +  vcpus_count: "1"
> +  CPUInstance Properties:
> +    socket-id: "0"
> +    core-id: "1"
> +    thread-id: "0"
> +  ...

It would be good to included spapr examples here as well, to show how
the output can vary between platforms (spapr will only list core-id,
not thread or socket ids).

> +Hotplug CPUs
> +------------
> +
> +A monitor command can be used to hotplug CPUs:
> +
> + - "device_add": creates a VCPU device and inserts it into the
> +	specific place as a device

I'd avoid the term "VCPU device", because in a number of contexts
"VCPU" refers specifically to a single VCPU thread.  The device that's
added here may be a core or other composite device containing several
VCPU threads.

That phrasing also reads very oddly to me, but I'm not quite sure what
to replace it with.

> +
> +For example, the following command adds a VCPU which has the id cpu1
> +in the specific position of the guest's CPU sockets which was discussed
> +earlier by using the socket/core/thread-id:1
> +
> +  (qemu) device_add qemu64-x86_64-cpu,id=cpu1,socket-id=0,core-id=1,thread-id=1
> +
> +Where,
> +
> + - "qemu64-x86_64-cpu" is the CPU model.
> + - "id" is the unique identifier in the device set. It is required.

This is not actually required, although it's probably a good idea in
most cases.

> + - "socket-id/core-id/thread-id" represent the designated position
> +   which is obtained form the above possible list of CPUs.
> +
> +It's also possible to start a guest with a cpu cold-plugged into a
> +specific location (socket,core,thread).
> +
> +In the following command line example, a guest which has 3 VCPUs is
> +created:
> +
> + qemu  [...] -smp 1,maxcpus=8,sockets=2,cores=2,threads=2 \
> +	-device qemu64-x86_64-cpu,id=cpu1,socket-id=1, \
> +	core-id=1,thread-id=0 \
> +	-device qemu64-x86_64-cpu,id=cpu2,socket-id=1, \
> +	core-id=1,thread-id=1 \
> +
> +Two VCPUs are cold-plugged by "-device" parameter, which are in the
> +same socket and core but with different thread-ids. After that, the
> +guest has an additional five VCPUs to be hot-plugged when needed.
> +
> +CPU hot-unplug
> +--------------
> +
> +In order to be able to hot unplug a CPU device, QEMU removes CPU
> +devices by using the ids which were assigned when hotplugging the
> +CPU device.
> +
> +A monitor command can be used to hot unplug CPUs:
> +
> + - "device_del": deletes a CPU device
> +
> +For example, assuming that the CPU device with id "cpu1" exists and has
> +been offline, the following command tries to remove it.

The cpu doesn't necessarily need to be offlined already.  I'm not sure
about x86, but on spapr, gibing this command will automatically
co-ordinate with the guest to offline the cpu before removing it.

> +  (qemu) device_del cpu1
> +