Patchwork [22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

login
register
mail settings
Submitter Igor Mammedov
Date April 5, 2013, 2:37 p.m.
Message ID <1365172636-28628-25-git-send-email-imammedo@redhat.com>
Download mbox | patch
Permalink /patch/234184/
State New
Headers show

Comments

Igor Mammedov - April 5, 2013, 2:37 p.m.
... via current_machine->cpu_hot_add() hook called by cpu-set QMP command,
for x86 target.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v4:
  * merge "qmp: add cpu-add qmp command" & "target-i386: implement CPU hot-add" patches
  * move notifier call to CPUCLass.realize()
  * add hook cpu_hot_add to QEMUMachine
  * make QEMUMachineInitArgs global and keep default cpu_model there

v3:
  * it appears that 'online/offline' in cpu-set are confusing people
    with what command actually does and users might have to distinguish
    if 'offline' is not implemented by parsing error message. To simplify
    things replace cpu-set with cpu-add command to show more clear what
    command does and just add cpu-del when CPU remove is implemented.

v2:
  * s/cpu_set/cpu-set/
  * qmp doc style fix
  * use bool type instead of opencodding online/offline string
     suggested-by: Eric Blake <eblake@redhat.com>
---
 hw/boards.h            |  3 +++
 hw/i386/pc.c           | 20 ++++++++++++++++++++
 qapi-schema.json       | 11 +++++++++++
 qmp-commands.hx        | 23 +++++++++++++++++++++++
 qmp.c                  | 10 ++++++++++
 stubs/do_cpu_hot_add.c |  7 +++++++
 vl.c                   |  6 +++++-
 7 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 stubs/do_cpu_hot_add.c
Eduardo Habkost - April 5, 2013, 5:10 p.m.
On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
[...]
> diff --git a/qapi-schema.json b/qapi-schema.json
> index db542f6..a760ed5 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -1387,6 +1387,17 @@
>  { 'command': 'cpu', 'data': {'index': 'int'} }
>  
>  ##
> +# @cpu-add
> +#
> +# Adds CPU with specified id
> +#
> +# @id: cpu id of CPU to be created

Can we have the semantics/constraints of "id" documented here? Is it an
arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
it have to be the index of the CPU in the CPU list? How the IDs of
existing CPUs set using "-smp" are allocated?

I am looking at the code right now to understand how this implementation
works, but the documentation could contain or point to documentation on
how the "id" parameter is used and interpreted.

> +#
> +# Returns: Nothing on success
> +##
> +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> +
> +##
Eduardo Habkost - April 5, 2013, 5:24 p.m.
On Fri, Apr 05, 2013 at 02:10:54PM -0300, Eduardo Habkost wrote:
> On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index db542f6..a760ed5 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -1387,6 +1387,17 @@
> >  { 'command': 'cpu', 'data': {'index': 'int'} }
> >  
> >  ##
> > +# @cpu-add
> > +#
> > +# Adds CPU with specified id
> > +#
> > +# @id: cpu id of CPU to be created
> 
> Can we have the semantics/constraints of "id" documented here? Is it an
> arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> it have to be the index of the CPU in the CPU list? How the IDs of
> existing CPUs set using "-smp" are allocated?
> 
> I am looking at the code right now to understand how this implementation
> works, but the documentation could contain or point to documentation on
> how the "id" parameter is used and interpreted.

So, the answer to my own question seems to be on patch 21/22. Let me
check if I understand this correctly:

 @id: cpu id of the CPU to be created. It must be one of the the
      available IDs listed at the "/machine/icc-bridge/cpu[0..N]" links.

Is the above correct?

Now, my question is: suppose a caller starts QEMU as:

 $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
        -numa node,cpus=0-8   -numa node,cpus=9-17 \
        -numa node,cpus=18-26 -numa node,cpus=27-35

and the caller wants to hot-add all VCPUs in the last CPU socket (that
means: all the VCPUs in the last NUMA node, that means: CPU indexes
27-35). What should the caller do to find out which of the
/machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
VCPUs?


> 
> > +#
> > +# Returns: Nothing on success
> > +##
> > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > +
> > +##
> -- 
> Eduardo
>
Igor Mammedov - April 9, 2013, 8:19 p.m.
On Fri, 5 Apr 2013 14:10:54 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index db542f6..a760ed5 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -1387,6 +1387,17 @@
> >  { 'command': 'cpu', 'data': {'index': 'int'} }
> >  
> >  ##
> > +# @cpu-add
> > +#
> > +# Adds CPU with specified id
> > +#
> > +# @id: cpu id of CPU to be created
> 
> Can we have the semantics/constraints of "id" documented here? Is it an
> arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
it's generic function so documenting it as APIC ID is not appropriate.

I for sure should document it on cpu-hotplug wiki page though, for x86 use
case for starters. i.e. how to use QMP to get a list of available/free IDs.
and in which order to use them.

> it have to be the index of the CPU in the CPU list? How the IDs of
> existing CPUs set using "-smp" are allocated?
With current -smp implementation the same way as it was before,
and for migration to work hot-plugged CPU has to be the next unused APIC
ID in their sequence, so that target qemu could be started with "-smp n+1".

But -smp along with -numa should be reworked to allow specifying guest visible
CPU IDs for arbitrary CPU hotplug to work.

when we done with QOMifying CPUs it might be possible to use -device for them
and keeping -smp for compat/shorcut purposes.

> 
> I am looking at the code right now to understand how this implementation
> works, but the documentation could contain or point to documentation on
> how the "id" parameter is used and interpreted.
I'll add pointer to wiki and describe there target-i386 use-case.

> 
> > +#
> > +# Returns: Nothing on success
> > +##
> > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > +
> > +##
> -- 
> Eduardo
Eduardo Habkost - April 9, 2013, 8:46 p.m.
On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> On Fri, 5 Apr 2013 14:10:54 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > [...]
> > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > index db542f6..a760ed5 100644
> > > --- a/qapi-schema.json
> > > +++ b/qapi-schema.json
> > > @@ -1387,6 +1387,17 @@
> > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > >  
> > >  ##
> > > +# @cpu-add
> > > +#
> > > +# Adds CPU with specified id
> > > +#
> > > +# @id: cpu id of CPU to be created
> > 
> > Can we have the semantics/constraints of "id" documented here? Is it an
> > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> it's generic function so documenting it as APIC ID is not appropriate.
> 
> I for sure should document it on cpu-hotplug wiki page though, for x86 use
> case for starters. i.e. how to use QMP to get a list of available/free IDs.
> and in which order to use them.
> 
> > it have to be the index of the CPU in the CPU list? How the IDs of
> > existing CPUs set using "-smp" are allocated?
> With current -smp implementation the same way as it was before,
> and for migration to work hot-plugged CPU has to be the next unused APIC
> ID in their sequence, so that target qemu could be started with "-smp n+1".

The problem is that it's hard to find out what's the APIC ID for each
CPU mentioned in the command-line.

For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
be:

online on startup:
package 0, core 0: 0 1 2
package 0, core 1: 4 5 6
package 0, core 2: 8 9 10
package 1, core 0: 16 17 18
package 1, core 1: 20 21 22
package 1, core 2: 24 25 26

offline on startup:
package 2, core 0: 32 33 34
package 2, core 1: 36 37 38
package 2, core 2: 40 41 42
package 3, core 0: 48 49 50
package 3, core 1: 52 53 54
package 3, core 2: 56 57 58


What should the caller do to find out the correct ID for each of the 36
VCPUs? This should be clearly documented.


> 
> But -smp along with -numa should be reworked to allow specifying guest visible
> CPU IDs for arbitrary CPU hotplug to work.

I'm curious how you plan to make this work while keeping command-line
compatibility. See the question I sent on my other message, about how to
map the IDs used on -numa (that are "CPU indexes") to the IDs required
by cpu-add.


> 
> when we done with QOMifying CPUs it might be possible to use -device for them
> and keeping -smp for compat/shorcut purposes.
> 
> > 
> > I am looking at the code right now to understand how this implementation
> > works, but the documentation could contain or point to documentation on
> > how the "id" parameter is used and interpreted.
> I'll add pointer to wiki and describe there target-i386 use-case.

Thanks! Could you try to document it succintly inside qapi-schema.json
as well? Maybe just a pointer to other documents would be useful.
Igor Mammedov - April 9, 2013, 9:05 p.m.
On Tue, 9 Apr 2013 17:46:21 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > On Fri, 5 Apr 2013 14:10:54 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > [...]
> > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > index db542f6..a760ed5 100644
> > > > --- a/qapi-schema.json
> > > > +++ b/qapi-schema.json
> > > > @@ -1387,6 +1387,17 @@
> > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > >  
> > > >  ##
> > > > +# @cpu-add
> > > > +#
> > > > +# Adds CPU with specified id
> > > > +#
> > > > +# @id: cpu id of CPU to be created
> > > 
> > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > it's generic function so documenting it as APIC ID is not appropriate.
> > 
> > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > and in which order to use them.
> > 
> > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > existing CPUs set using "-smp" are allocated?
> > With current -smp implementation the same way as it was before,
> > and for migration to work hot-plugged CPU has to be the next unused APIC
> > ID in their sequence, so that target qemu could be started with "-smp n+1".
> 
> The problem is that it's hard to find out what's the APIC ID for each
> CPU mentioned in the command-line.
> 
> For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> be:
> 
> online on startup:
> package 0, core 0: 0 1 2
> package 0, core 1: 4 5 6
> package 0, core 2: 8 9 10
> package 1, core 0: 16 17 18
> package 1, core 1: 20 21 22
> package 1, core 2: 24 25 26
> 
> offline on startup:
> package 2, core 0: 32 33 34
> package 2, core 1: 36 37 38
> package 2, core 2: 40 41 42
> package 3, core 0: 48 49 50
> package 3, core 1: 52 53 54
> package 3, core 2: 56 57 58
> 
> 
> What should the caller do to find out the correct ID for each of the 36
> VCPUs? This should be clearly documented.
Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
as /machine/icc-bridge/cpu[0..n]
And since in above sequence IDs are monotonously increasing it's enough to use
the next unused APIC ID to make -smp n+1 work. 
> 
> 
> > 
> > But -smp along with -numa should be reworked to allow specifying guest visible
> > CPU IDs for arbitrary CPU hotplug to work.
> 
> I'm curious how you plan to make this work while keeping command-line
> compatibility. See the question I sent on my other message, about how to
> map the IDs used on -numa (that are "CPU indexes") to the IDs required
> by cpu-add.
It doesn't mean that we should stick to bad/insufficient interface forever. 
We could add new one that does it right and keep old one for a time being for
compatibility.

> 
> > 
> > when we done with QOMifying CPUs it might be possible to use -device for them
> > and keeping -smp for compat/shorcut purposes.
> > 
> > > 
> > > I am looking at the code right now to understand how this implementation
> > > works, but the documentation could contain or point to documentation on
> > > how the "id" parameter is used and interpreted.
> > I'll add pointer to wiki and describe there target-i386 use-case.
> 
> Thanks! Could you try to document it succintly inside qapi-schema.json
> as well? Maybe just a pointer to other documents would be useful.
It's very target specific, so separate document probably would make more sense.

> -- 
> Eduardo
>
Eduardo Habkost - April 11, 2013, 3:12 p.m.
On Tue, Apr 09, 2013 at 11:05:26PM +0200, Igor Mammedov wrote:
> On Tue, 9 Apr 2013 17:46:21 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > > On Fri, 5 Apr 2013 14:10:54 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > > [...]
> > > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > > index db542f6..a760ed5 100644
> > > > > --- a/qapi-schema.json
> > > > > +++ b/qapi-schema.json
> > > > > @@ -1387,6 +1387,17 @@
> > > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > > >  
> > > > >  ##
> > > > > +# @cpu-add
> > > > > +#
> > > > > +# Adds CPU with specified id
> > > > > +#
> > > > > +# @id: cpu id of CPU to be created
> > > > 
> > > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > > it's generic function so documenting it as APIC ID is not appropriate.
> > > 
> > > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > > and in which order to use them.
> > > 
> > > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > > existing CPUs set using "-smp" are allocated?
> > > With current -smp implementation the same way as it was before,
> > > and for migration to work hot-plugged CPU has to be the next unused APIC
> > > ID in their sequence, so that target qemu could be started with "-smp n+1".
> > 
> > The problem is that it's hard to find out what's the APIC ID for each
> > CPU mentioned in the command-line.
> > 
> > For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> > will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> > be:
> > 
> > online on startup:
> > package 0, core 0: 0 1 2
> > package 0, core 1: 4 5 6
> > package 0, core 2: 8 9 10
> > package 1, core 0: 16 17 18
> > package 1, core 1: 20 21 22
> > package 1, core 2: 24 25 26
> > 
> > offline on startup:
> > package 2, core 0: 32 33 34
> > package 2, core 1: 36 37 38
> > package 2, core 2: 40 41 42
> > package 3, core 0: 48 49 50
> > package 3, core 1: 52 53 54
> > package 3, core 2: 56 57 58
> > 
> > 
> > What should the caller do to find out the correct ID for each of the 36
> > VCPUs? This should be clearly documented.
> Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
> as /machine/icc-bridge/cpu[0..n]
> And since in above sequence IDs are monotonously increasing it's enough to use
> the next unused APIC ID to make -smp n+1 work. 

This can be one method to map CPU indexes to APIC IDs, yes. I find it
hard to explain and hard to use, and it imposes constraints on the way
the target-specific IDs are calculated (requiring them to be
monotonically increasing). But it may be a reasonable solution by now.

I have one additional question about the icc-bridge paths: are the link
paths on icc-bridge explicitly documented/required to be APIC IDs, or
the caller should assume they are arbitrary IDs with no specific
meaning?


> > 
> > 
> > > 
> > > But -smp along with -numa should be reworked to allow specifying guest visible
> > > CPU IDs for arbitrary CPU hotplug to work.
> > 
> > I'm curious how you plan to make this work while keeping command-line
> > compatibility. See the question I sent on my other message, about how to
> > map the IDs used on -numa (that are "CPU indexes") to the IDs required
> > by cpu-add.
> It doesn't mean that we should stick to bad/insufficient interface forever. 
> We could add new one that does it right and keep old one for a time being for
> compatibility.

The problem is that I only see three possible kinds of CPU identifiers
that could work in the command-line:

 * Arbitrary user-defined IDs
 * CPU indexes (the current interface)
 * Topology-based identifiers/paths (e.g.
   "/machine/numa_node[0]/cpu_socket[1]/core[2]/thread[1]")

I don't think we can asily use APIC IDs on the command-line because the
caller simply doesn't know what will be the APIC ID for each VCPU.

But this is a problem we can discuss and solve later. By now, we are
stuck with the legacy CPU-index-based interfaces.

> 
> > 
> > > 
> > > when we done with QOMifying CPUs it might be possible to use -device for them
> > > and keeping -smp for compat/shorcut purposes.
> > > 
> > > > 
> > > > I am looking at the code right now to understand how this implementation
> > > > works, but the documentation could contain or point to documentation on
> > > > how the "id" parameter is used and interpreted.
> > > I'll add pointer to wiki and describe there target-i386 use-case.
> > 
> > Thanks! Could you try to document it succintly inside qapi-schema.json
> > as well? Maybe just a pointer to other documents would be useful.
> It's very target specific, so separate document probably would make more sense.

The IDs are target-specific, but do we really need to make the interface
specification/usage to be target-specific? We could have a
target-independent interface to find out what are the available/valid
IDs to use on cpu-add.
Igor Mammedov - April 11, 2013, 3:17 p.m.
On Fri, 5 Apr 2013 14:24:35 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 02:10:54PM -0300, Eduardo Habkost wrote:
> > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > [...]
> > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > index db542f6..a760ed5 100644
> > > --- a/qapi-schema.json
> > > +++ b/qapi-schema.json
> > > @@ -1387,6 +1387,17 @@
> > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > >  
> > >  ##
> > > +# @cpu-add
> > > +#
> > > +# Adds CPU with specified id
> > > +#
> > > +# @id: cpu id of CPU to be created
> > 
> > Can we have the semantics/constraints of "id" documented here? Is it an
> > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > it have to be the index of the CPU in the CPU list? How the IDs of
> > existing CPUs set using "-smp" are allocated?
> > 
> > I am looking at the code right now to understand how this implementation
> > works, but the documentation could contain or point to documentation on
> > how the "id" parameter is used and interpreted.
> 
> So, the answer to my own question seems to be on patch 21/22. Let me
> check if I understand this correctly:
> 
>  @id: cpu id of the CPU to be created. It must be one of the the
>       available IDs listed at the "/machine/icc-bridge/cpu[0..N]" links.
> 
> Is the above correct?
yes.

> 
> Now, my question is: suppose a caller starts QEMU as:
> 
>  $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
>         -numa node,cpus=0-8   -numa node,cpus=9-17 \
>         -numa node,cpus=18-26 -numa node,cpus=27-35
> 
> and the caller wants to hot-add all VCPUs in the last CPU socket (that
> means: all the VCPUs in the last NUMA node, that means: CPU indexes
> 27-35). What should the caller do to find out which of the
> /machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
> VCPUs?
If one wants plug in a specific CPU, It won't work with -numa yet.

to make it work we need to specify sockets on -numa cmd. line and then
construct synthetic QOM containers tree that could looklike:

/machine/numa_nodes/[0..N]/sockets/[0..M]/cpus/[0..X]

then command line could look like:
$ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
       -numa node,sockets=0-1 -numa node,sockets=2-3
> 
> 
> > 
> > > +#
> > > +# Returns: Nothing on success
> > > +##
> > > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > > +
> > > +##
> > -- 
> > Eduardo
> > 
> 
> -- 
> Eduardo
>
Igor Mammedov - April 11, 2013, 3:37 p.m.
On Thu, 11 Apr 2013 12:12:37 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Tue, Apr 09, 2013 at 11:05:26PM +0200, Igor Mammedov wrote:
> > On Tue, 9 Apr 2013 17:46:21 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > > > On Fri, 5 Apr 2013 14:10:54 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > > > [...]
> > > > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > > > index db542f6..a760ed5 100644
> > > > > > --- a/qapi-schema.json
> > > > > > +++ b/qapi-schema.json
> > > > > > @@ -1387,6 +1387,17 @@
> > > > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > > > >  
> > > > > >  ##
> > > > > > +# @cpu-add
> > > > > > +#
> > > > > > +# Adds CPU with specified id
> > > > > > +#
> > > > > > +# @id: cpu id of CPU to be created
> > > > > 
> > > > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > > > it's generic function so documenting it as APIC ID is not appropriate.
> > > > 
> > > > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > > > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > > > and in which order to use them.
> > > > 
> > > > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > > > existing CPUs set using "-smp" are allocated?
> > > > With current -smp implementation the same way as it was before,
> > > > and for migration to work hot-plugged CPU has to be the next unused APIC
> > > > ID in their sequence, so that target qemu could be started with "-smp n+1".
> > > 
> > > The problem is that it's hard to find out what's the APIC ID for each
> > > CPU mentioned in the command-line.
> > > 
> > > For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> > > will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> > > be:
> > > 
> > > online on startup:
> > > package 0, core 0: 0 1 2
> > > package 0, core 1: 4 5 6
> > > package 0, core 2: 8 9 10
> > > package 1, core 0: 16 17 18
> > > package 1, core 1: 20 21 22
> > > package 1, core 2: 24 25 26
> > > 
> > > offline on startup:
> > > package 2, core 0: 32 33 34
> > > package 2, core 1: 36 37 38
> > > package 2, core 2: 40 41 42
> > > package 3, core 0: 48 49 50
> > > package 3, core 1: 52 53 54
> > > package 3, core 2: 56 57 58
> > > 
> > > 
> > > What should the caller do to find out the correct ID for each of the 36
> > > VCPUs? This should be clearly documented.
> > Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
> > as /machine/icc-bridge/cpu[0..n]
> > And since in above sequence IDs are monotonously increasing it's enough to use
> > the next unused APIC ID to make -smp n+1 work. 
> 
> This can be one method to map CPU indexes to APIC IDs, yes. I find it
> hard to explain and hard to use, and it imposes constraints on the way
> the target-specific IDs are calculated (requiring them to be
> monotonically increasing). But it may be a reasonable solution by now.
Arbitrary CPU hotplug + migration, require ability to specify which CPU
to create on target. It's probably post CPU-unplug topic or might be
fixed with it.

> 
> I have one additional question about the icc-bridge paths: are the link
> paths on icc-bridge explicitly documented/required to be APIC IDs, or
> the caller should assume they are arbitrary IDs with no specific
> meaning?
since it's on icc-bridge, they are APIC IDs, but from user's POV I wouldn't
care and treat it as opaque.

If we do it in a generic way, then I'll say they are arbitrary IDs.

> 
> > > 
> > > 
> > > > 
> > > > But -smp along with -numa should be reworked to allow specifying guest visible
> > > > CPU IDs for arbitrary CPU hotplug to work.
> > > 
> > > I'm curious how you plan to make this work while keeping command-line
> > > compatibility. See the question I sent on my other message, about how to
> > > map the IDs used on -numa (that are "CPU indexes") to the IDs required
> > > by cpu-add.
> > It doesn't mean that we should stick to bad/insufficient interface forever. 
> > We could add new one that does it right and keep old one for a time being for
> > compatibility.
> 
> The problem is that I only see three possible kinds of CPU identifiers
> that could work in the command-line:
> 
>  * Arbitrary user-defined IDs
>  * CPU indexes (the current interface)
>  * Topology-based identifiers/paths (e.g.
>    "/machine/numa_node[0]/cpu_socket[1]/core[2]/thread[1]")
+1 to topology, others will allow to specify bad configuration

> 
> I don't think we can asily use APIC IDs on the command-line because the
> caller simply doesn't know what will be the APIC ID for each VCPU.
Agree, especially regarding -numa option, it should consume sockets instead.

user might still potentially use opaque ID in cmd line if he will add CPUs
with -device apic_id=xxx, but first he should probe qemu with the same topology
and read complete topology with IDs qemu provides.

> 
> But this is a problem we can discuss and solve later. By now, we are
> stuck with the legacy CPU-index-based interfaces.
> 
> > 
> > > 
> > > > 
> > > > when we done with QOMifying CPUs it might be possible to use -device for them
> > > > and keeping -smp for compat/shorcut purposes.
> > > > 
> > > > > 
> > > > > I am looking at the code right now to understand how this implementation
> > > > > works, but the documentation could contain or point to documentation on
> > > > > how the "id" parameter is used and interpreted.
> > > > I'll add pointer to wiki and describe there target-i386 use-case.
> > > 
> > > Thanks! Could you try to document it succintly inside qapi-schema.json
> > > as well? Maybe just a pointer to other documents would be useful.
> > It's very target specific, so separate document probably would make more sense.
> 
> The IDs are target-specific, but do we really need to make the interface
> specification/usage to be target-specific? We could have a
> target-independent interface to find out what are the available/valid
> IDs to use on cpu-add.
Answered to this in another email about -numa and showed how ID discovery
could be made in unified cross target way.

> 
> -- 
> Eduardo
Eduardo Habkost - April 11, 2013, 3:49 p.m.
On Thu, Apr 11, 2013 at 05:17:29PM +0200, Igor Mammedov wrote:
[...]
> > 
> > Now, my question is: suppose a caller starts QEMU as:
> > 
> >  $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
> >         -numa node,cpus=0-8   -numa node,cpus=9-17 \
> >         -numa node,cpus=18-26 -numa node,cpus=27-35
> > 
> > and the caller wants to hot-add all VCPUs in the last CPU socket (that
> > means: all the VCPUs in the last NUMA node, that means: CPU indexes
> > 27-35). What should the caller do to find out which of the
> > /machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
> > VCPUs?
> If one wants plug in a specific CPU, It won't work with -numa yet.

OK, documenting it as unsupported by now would be enough to me. :-)

It looks like -numa is the only interface that requires CPUs to be
individually identified in the command-line, right?

> 
> to make it work we need to specify sockets on -numa cmd. line and then
> construct synthetic QOM containers tree that could looklike:
> 
> /machine/numa_nodes/[0..N]/sockets/[0..M]/cpus/[0..X]
> 
> then command line could look like:
> $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
>        -numa node,sockets=0-1 -numa node,sockets=2-3

Makes sense to me. Using socket numbers is one way of making the CPU
identifiers be topology-based (as mentioned on a previous message), and
also make it stricter to allow only topologies that actually make sense
for a "/machine/numa_nodes/.../sockets/.../threads/..." tree.

> [...]

Patch

diff --git a/hw/boards.h b/hw/boards.h
index 425bdc7..de8f92a 100644
--- a/hw/boards.h
+++ b/hw/boards.h
@@ -18,6 +18,8 @@  typedef struct QEMUMachineInitArgs {
     const char *cpu_model;
 } QEMUMachineInitArgs;
 
+extern QEMUMachineInitArgs *machine_args;
+
 typedef void QEMUMachineInitFunc(QEMUMachineInitArgs *args);
 
 typedef void QEMUMachineResetFunc(void);
@@ -43,6 +45,7 @@  typedef struct QEMUMachine {
     GlobalProperty *compat_props;
     struct QEMUMachine *next;
     const char *hw_version;
+    void (*hot_add_cpu)(const int64_t id, Error **errp);
 } QEMUMachine;
 
 int qemu_register_machine(QEMUMachine *m);
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 86e5365..1f94134 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -53,6 +53,7 @@ 
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
 #include "hw/icc_bus.h"
+#include "hw/boards.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -896,6 +897,23 @@  static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
     return cpu;
 }
 
+static void do_cpu_hot_add(const int64_t id, Error **errp)
+{
+    if (cpu_exists(id)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", it already exists", id);
+        return;
+    }
+
+    if (id >= pc_apic_id_limit(max_cpus)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", max allowed: 0x%x", id, pc_apic_id_limit(max_cpus) - 1);
+        return;
+    }
+
+    pc_new_cpu(machine_args->cpu_model, id, NULL, errp);
+}
+
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
@@ -910,7 +928,9 @@  void pc_cpus_init(const char *cpu_model)
 #else
         cpu_model = "qemu32";
 #endif
+        machine_args->cpu_model = cpu_model;
     }
+    current_machine->hot_add_cpu = do_cpu_hot_add;
 
     ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
                                                  TYPE_ICC_BRIDGE, NULL));
diff --git a/qapi-schema.json b/qapi-schema.json
index db542f6..a760ed5 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1387,6 +1387,17 @@ 
 { 'command': 'cpu', 'data': {'index': 'int'} }
 
 ##
+# @cpu-add
+#
+# Adds CPU with specified id
+#
+# @id: cpu id of CPU to be created
+#
+# Returns: Nothing on success
+##
+{ 'command': 'cpu-add', 'data': {'id': 'int'} }
+
+##
 # @memsave:
 #
 # Save a portion of guest memory to a file.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 1e0e11e..79b2ba7 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -407,6 +407,29 @@  Example:
 EQMP
 
     {
+        .name       = "cpu-add",
+        .args_type  = "id:i",
+        .mhandler.cmd_new = qmp_marshal_input_cpu_add,
+    },
+
+SQMP
+cpu-add
+-------
+
+Adds virtual cpu
+
+Arguments:
+
+- "id": cpu id (json-int)
+
+Example:
+
+-> { "execute": "cpu-add", "arguments": { "id": 2 } }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "memsave",
         .args_type  = "val:l,size:i,filename:s,cpu:i?",
         .mhandler.cmd_new = qmp_marshal_input_memsave,
diff --git a/qmp.c b/qmp.c
index 55b056b..8a9fd9e 100644
--- a/qmp.c
+++ b/qmp.c
@@ -24,6 +24,7 @@ 
 #include "hw/qdev.h"
 #include "sysemu/blockdev.h"
 #include "qom/qom-qobject.h"
+#include "hw/boards.h"
 
 NameInfo *qmp_query_name(Error **errp)
 {
@@ -108,6 +109,15 @@  void qmp_cpu(int64_t index, Error **errp)
     /* Just do nothing */
 }
 
+void qmp_cpu_add(int64_t id, Error **errp)
+{
+    if (current_machine->hot_add_cpu) {
+        current_machine->hot_add_cpu(id, errp);
+    } else {
+        error_setg(errp, "Not supported");
+    }
+}
+
 #ifndef CONFIG_VNC
 /* If VNC support is enabled, the "true" query-vnc command is
    defined in the VNC subsystem */
diff --git a/stubs/do_cpu_hot_add.c b/stubs/do_cpu_hot_add.c
new file mode 100644
index 0000000..1f6d7a6
--- /dev/null
+++ b/stubs/do_cpu_hot_add.c
@@ -0,0 +1,7 @@ 
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+
+void do_cpu_hot_add(const int64_t id, Error **errp)
+{
+    error_setg(errp, "Not implemented");
+}
diff --git a/vl.c b/vl.c
index 97f0349..b2ea58c 100644
--- a/vl.c
+++ b/vl.c
@@ -179,6 +179,8 @@  int main(int argc, char **argv)
 #define MAX_VIRTIO_CONSOLES 1
 #define MAX_SCLP_CONSOLES 1
 
+QEMUMachineInitArgs *machine_args;
+
 static const char *data_dir[16];
 static int data_dir_idx;
 const char *bios_name = NULL;
@@ -4310,13 +4312,15 @@  int main(int argc, char **argv, char **envp)
                                  .kernel_cmdline = kernel_cmdline,
                                  .initrd_filename = initrd_filename,
                                  .cpu_model = cpu_model };
+    machine_args = &args;
+    current_machine = machine;
+
     machine->init(&args);
 
     cpu_synchronize_all_post_init();
 
     set_numa_modes();
 
-    current_machine = machine;
 
     /* init USB devices */
     if (usb_enabled(false)) {