| Message ID | 20230222142105.84700-12-pmorel@linux.ibm.com |
|---|---|
| State | New |
| Headers | show |
| Series | s390x: CPU Topology | expand |
On 22/02/2023 15.21, Pierre Morel wrote: > Add some basic examples for the definition of cpu topology > in s390x. > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> > --- > docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ > docs/system/target-s390x.rst | 1 + > 2 files changed, 379 insertions(+) > create mode 100644 docs/system/s390x/cpu-topology.rst > > diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst > new file mode 100644 > index 0000000000..d470e28b97 > --- /dev/null > +++ b/docs/system/s390x/cpu-topology.rst > @@ -0,0 +1,378 @@ > +CPU topology on s390x > +===================== > + > +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of > +topology containers: drawers, books, sockets, defining a tree shaped > +hierarchy. > + > +The socket container contains one or more CPU entries consisting > +of a bitmap of three dentical CPU attributes: What do you mean by "dentical" here? > +- CPU type > +- polarization entitlement > +- dedication > + > +Note also that since 7.2 threads are no longer supported in the topology > +and the ``-smp`` command line argument accepts only ``threads=1``. > + > +Prerequisites > +------------- > + > +To use CPU topology a Linux QEMU/KVM machine providing the CPU topology facility > +(STFLE bit 11) is required. > + > +However, since this facility has been enabled by default in an early version > +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM > +that QEMU supports CPU topology. This still sounds like a mix of documentation for the user and documentation for developers. The normal user won't know what a KVM capability is and does not need to know that QEMU does this automatically under the hood. If you still want to mention it, I'd maybe rather phrase it like this: "To use the CPU topology, you need to run with KVM on a s390x host that uses the Linux kernel v6.0 or newer (which provide the so-called ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the CPU topology facility via the so-called STFLE bit 11 to the VM)." ... or something similar? [...] > +When a CPU is defined by the ``-device`` command argument, the > +tree topology attributes must be all defined or all not defined. > + > +.. code-block:: bash > + > + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 > + > +or > + > +.. code-block:: bash > + > + -device gen16b-s390x-cpu,core-id=1,dedication=true > + > +If none of the tree attributes (drawer, book, sockets), are specified > +for the ``-device`` argument, as for all CPUs defined on the ``-smp`` s/defined on/defined with/ ? > +command argument the topology tree attributes will be set by simply > +adding the CPUs to the topology based on the core-id starting with > +core-0 at position 0 of socket-0, book-0, drawer-0. > + > +QEMU will not try to solve collisions and will report an error if the > +CPU topology, explicitely or implicitely defined on a ``-device`` explicitly or implicitly > +argument collides with the definition of a CPU implicitely defined implicitly > +on the ``-smp`` argument. > + > +When the topology modifier attributes are not defined for the > +``-device`` command argument they takes following default values: > + > +- dedication: ``false`` > +- entitlement: ``medium`` > + > + > +Hot plug > +++++++++ > + > +New CPUs can be plugged using the device_add hmp command as in: > + > +.. code-block:: bash > + > + (qemu) device_add gen16b-s390x-cpu,core-id=9 > + > +The same placement of the CPU is derived from the core-id as described above. > + > +The topology can of course be fully defined: > + > +.. code-block:: bash > + > + (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 > + > + > +Examples > +++++++++ > + > +In the following machine we define 8 sockets with 4 cores each. > + > +.. code-block:: bash > + > + $ qemu-system-s390x -m 2G \ > + -cpu gen16b,ctop=on \ > + -smp cpus=5,sockets=8,cores=4,maxcpus=32 \ > + -device host-s390x-cpu,core-id=14 \ > + > +A new CPUs can be plugged using the device_add hmp command as before: > + > +.. code-block:: bash > + > + (qemu) device_add gen16b-s390x-cpu,core-id=9 > + > +The core-id defines the placement of the core in the topology by > +starting with core 0 in socket 0 up to maxcpus. > + > +In the example above: > + > +* There are 5 CPUs provided to the guest with the ``-smp`` command line > + They will take the core-ids 0,1,2,3,4 > + As we have 4 cores in a socket, we have 4 CPUs provided > + to the guest in socket 0, with core-ids 0,1,2,3. > + The last cpu, with core-id 4, will be on socket 1. > + > +* the core with ID 14 provided by the ``-device`` command line will > + be placed in socket 3, with core-id 14 > + > +* the core with ID 9 provided by the ``device_add`` qmp command will > + be placed in socket 2, with core-id 9 > + > + > +Polarization, entitlement and dedication > +---------------------------------------- > + > +Polarization > +++++++++++++ > + > +The polarization is an indication given by the ``guest`` to the host > +that it is able to make use of CPU provisioning information. > +The guest indicates the polarization by using the PTF instruction. > + > +Polarization is define two models of CPU provisioning: horizontal > +and vertical. > + > +The horizontal polarization is the default model on boot and after > +subsystem reset in which the guest considers all vCPUs being having > +an equal provisioning of CPUs by the host. > + > +In the vertical polarization model the guest can make use of the > +vCPU entitlement information provided by the host to optimize > +kernel thread scheduling. > + > +A subsystem reset puts all vCPU of the configuration into the > +horizontal polarization. > + > +Entitlement > ++++++++++++ > + > +The vertical polarization specifies that guest's vCPU can get "that *the* guest's vCPU" ? > +different real CPU provisions: > + > +- a vCPU with vertical high entitlement specifies that this > + vCPU gets 100% of the real CPU provisioning. > + > +- a vCPU with vertical medium entitlement specifies that this > + vCPU shares the real CPU with other vCPUs. > + > +- a vCPU with vertical low entitlement specifies that this > + vCPU only gets real CPU provisioning when no other vCPUs needs it. > + > +In the case a vCPU with vertical high entitlement does not use > +the real CPU, the unused "slack" can be dispatched to other vCPU > +with medium or low entitlement. > + > +The admin specifies a vCPU as ``dedicated`` when the vCPU is fully dedicated > +to a single real CPU. > + > +The dedicated bit is an indication of affinity of a vCPU for a real CPU > +while the entitlement indicates the sharing or exclusivity of use. > + > +Defining the topology on command line > +------------------------------------- > + > +The topology can entirely be defined using -device cpu statements, > +with the exception of CPU 0 which must be defined with the -smp > +argument. > + > +For example, here we set the position of the cores 1,2,3 to > +drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, > +book 0, socket 0 with all horizontal polarization and not dedicated. > +The core 4, will be set on its default position on socket 1 > +(since we have 4 core per socket) and we define it with dedication and > +vertical high entitlement. > + > +.. code-block:: bash > + > + $ qemu-system-s390x -m 2G \ > + -cpu gen16b,ctop=on \ > + -smp cpus=1,sockets=8,cores=4,maxcpus=32 \ > + \ > + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \ > + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \ > + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \ > + \ > + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \ > + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \ > + \ > + -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \ > + > +QAPI interface for topology > +--------------------------- > + > +Let's start QEMU with the following command: > + > +.. code-block:: bash > + > + qemu-system-s390x \ > + -enable-kvm \ > + -cpu z14,ctop=on \ > + -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \ > + \ > + -device z14-s390x-cpu,core-id=19,polarization=3 \ > + -device z14-s390x-cpu,core-id=11,polarization=1 \ > + -device z14-s390x-cpu,core-id=112,polarization=3 \ > + ... > + > +and see the result when using the QAPI interface. > + > +addons to query-cpus-fast > ++++++++++++++++++++++++++ > + > +The command query-cpus-fast allows the admin to query the topology I'd just say "allows to query", without "the admin". > +tree and modifiers for all configured vCPUs. > + > +.. code-block:: QMP > + > + { "execute": "query-cpus-fast" } > + { > + "return": [ > + { > + "dedicated": false, > + "thread-id": 536993, > + "props": { > + "core-id": 0, > + "socket-id": 0, > + "drawer-id": 0, > + "book-id": 0 > + }, > + "cpu-state": "operating", > + "entitlement": "medium", > + "qom-path": "/machine/unattached/device[0]", > + "cpu-index": 0, > + "target": "s390x" > + }, > + { > + "dedicated": false, > + "thread-id": 537003, > + "props": { > + "core-id": 19, > + "socket-id": 1, > + "drawer-id": 0, > + "book-id": 2 > + }, > + "cpu-state": "operating", > + "entitlement": "high", > + "qom-path": "/machine/peripheral-anon/device[0]", > + "cpu-index": 19, > + "target": "s390x" > + }, > + { > + "dedicated": false, > + "thread-id": 537004, > + "props": { > + "core-id": 11, > + "socket-id": 1, > + "drawer-id": 0, > + "book-id": 1 > + }, > + "cpu-state": "operating", > + "entitlement": "low", > + "qom-path": "/machine/peripheral-anon/device[1]", > + "cpu-index": 11, > + "target": "s390x" > + }, > + { > + "dedicated": true, > + "thread-id": 537005, > + "props": { > + "core-id": 112, > + "socket-id": 0, > + "drawer-id": 3, > + "book-id": 2 > + }, > + "cpu-state": "operating", > + "entitlement": "high", > + "qom-path": "/machine/peripheral-anon/device[2]", > + "cpu-index": 112, > + "target": "s390x" > + } > + ] > + } > + > + > +set-cpu-topology > +++++++++++++++++ > + > +The command set-cpu-topology allows the admin to modify the topology "allows to modify" > +tree or the topology modifiers of a vCPU in the configuration. > + > +.. code-block:: QMP > + > + -> { "execute": "set-cpu-topology", > + "arguments": { > + "core-id": 11, > + "socket-id": 0, > + "book-id": 0, > + "drawer-id": 0, > + "entitlement": low, > + "dedicated": false > + } > + } > + <- {"return": {}} > + > +The core-id parameter is the only non optional parameter and every > +unspecified parameter keeps its previous value. > + > +event CPU_POLARIZATION_CHANGE > ++++++++++++++++++++++++++++++ > + > +When a guest is requests a modification of the polarization, > +QEMU sends a CPU_POLARIZATION_CHANGE event. > + > +When requesting the change, the guest only specifies horizontal or > +vertical polarization. > +It is the job of the admin to set the dedication and fine grained vertical entitlement > +in response to this event. > + > +Note that a vertical polarized dedicated vCPU can only have a high > +entitlement, this gives 6 possibilities for vCPU polarization: > + > +- Horizontal > +- Horizontal dedicated > +- Vertical low > +- Vertical medium > +- Vertical high > +- Vertical high dedicated > + > +Example of the event received when the guest issues the CPU instruction > +Perform Topology Function PTF(0) to request an horizontal polarization: > + > +.. code-block:: QMP > + > + <- { "event": "CPU_POLARIZATION_CHANGE", > + "data": { "polarization": 0 }, > + "timestamp": { "seconds": 1401385907, "microseconds": 422329 } } Thomas
On 2/27/23 14:58, Thomas Huth wrote: > On 22/02/2023 15.21, Pierre Morel wrote: >> Add some basic examples for the definition of cpu topology >> in s390x. >> >> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> >> --- >> docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ >> docs/system/target-s390x.rst | 1 + >> 2 files changed, 379 insertions(+) >> create mode 100644 docs/system/s390x/cpu-topology.rst >> >> diff --git a/docs/system/s390x/cpu-topology.rst >> b/docs/system/s390x/cpu-topology.rst >> new file mode 100644 >> index 0000000000..d470e28b97 >> --- /dev/null >> +++ b/docs/system/s390x/cpu-topology.rst >> @@ -0,0 +1,378 @@ >> +CPU topology on s390x >> +===================== >> + >> +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of >> +topology containers: drawers, books, sockets, defining a tree shaped >> +hierarchy. >> + >> +The socket container contains one or more CPU entries consisting >> +of a bitmap of three dentical CPU attributes: > > What do you mean by "dentical" here? :D i.. dentical I change it to identical > >> +- CPU type >> +- polarization entitlement >> +- dedication >> + >> +Note also that since 7.2 threads are no longer supported in the >> topology >> +and the ``-smp`` command line argument accepts only ``threads=1``. >> + >> +Prerequisites >> +------------- >> + >> +To use CPU topology a Linux QEMU/KVM machine providing the CPU >> topology facility >> +(STFLE bit 11) is required. >> + >> +However, since this facility has been enabled by default in an early >> version >> +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to >> notify KVM >> +that QEMU supports CPU topology. > > This still sounds like a mix of documentation for the user and > documentation for developers. The normal user won't know what a KVM > capability is and does not need to know that QEMU does this > automatically under the hood. > > If you still want to mention it, I'd maybe rather phrase it like this: > > "To use the CPU topology, you need to run with KVM on a s390x host > that uses the Linux kernel v6.0 or newer (which provide the so-called > ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal > the CPU topology facility via the so-called STFLE bit 11 to the VM)." > > ... or something similar? Yes, OK. > > [...] >> +When a CPU is defined by the ``-device`` command argument, the >> +tree topology attributes must be all defined or all not defined. >> + >> +.. code-block:: bash >> + >> + -device >> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 >> + >> +or >> + >> +.. code-block:: bash >> + >> + -device gen16b-s390x-cpu,core-id=1,dedication=true >> + >> +If none of the tree attributes (drawer, book, sockets), are specified >> +for the ``-device`` argument, as for all CPUs defined on the ``-smp`` > > s/defined on/defined with/ ? OK > >> +command argument the topology tree attributes will be set by simply >> +adding the CPUs to the topology based on the core-id starting with >> +core-0 at position 0 of socket-0, book-0, drawer-0. >> + >> +QEMU will not try to solve collisions and will report an error if the >> +CPU topology, explicitely or implicitely defined on a ``-device`` > > explicitly or implicitly OK > >> +argument collides with the definition of a CPU implicitely defined > > implicitly Yes > >> +on the ``-smp`` argument. >> + >> +When the topology modifier attributes are not defined for the >> +``-device`` command argument they takes following default values: >> + >> +- dedication: ``false`` >> +- entitlement: ``medium`` >> + >> + >> +Hot plug >> +++++++++ >> + >> +New CPUs can be plugged using the device_add hmp command as in: >> + >> +.. code-block:: bash >> + >> + (qemu) device_add gen16b-s390x-cpu,core-id=9 >> + >> +The same placement of the CPU is derived from the core-id as >> described above. >> + >> +The topology can of course be fully defined: >> + >> +.. code-block:: bash >> + >> + (qemu) device_add >> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 >> + >> + >> +Examples >> +++++++++ >> + >> +In the following machine we define 8 sockets with 4 cores each. >> + >> +.. code-block:: bash >> + >> + $ qemu-system-s390x -m 2G \ >> + -cpu gen16b,ctop=on \ >> + -smp cpus=5,sockets=8,cores=4,maxcpus=32 \ >> + -device host-s390x-cpu,core-id=14 \ >> + >> +A new CPUs can be plugged using the device_add hmp command as before: >> + >> +.. code-block:: bash >> + >> + (qemu) device_add gen16b-s390x-cpu,core-id=9 >> + >> +The core-id defines the placement of the core in the topology by >> +starting with core 0 in socket 0 up to maxcpus. >> + >> +In the example above: >> + >> +* There are 5 CPUs provided to the guest with the ``-smp`` command line >> + They will take the core-ids 0,1,2,3,4 >> + As we have 4 cores in a socket, we have 4 CPUs provided >> + to the guest in socket 0, with core-ids 0,1,2,3. >> + The last cpu, with core-id 4, will be on socket 1. >> + >> +* the core with ID 14 provided by the ``-device`` command line will >> + be placed in socket 3, with core-id 14 >> + >> +* the core with ID 9 provided by the ``device_add`` qmp command will >> + be placed in socket 2, with core-id 9 >> + >> + >> +Polarization, entitlement and dedication >> +---------------------------------------- >> + >> +Polarization >> +++++++++++++ >> + >> +The polarization is an indication given by the ``guest`` to the host >> +that it is able to make use of CPU provisioning information. >> +The guest indicates the polarization by using the PTF instruction. >> + >> +Polarization is define two models of CPU provisioning: horizontal >> +and vertical. >> + >> +The horizontal polarization is the default model on boot and after >> +subsystem reset in which the guest considers all vCPUs being having >> +an equal provisioning of CPUs by the host. >> + >> +In the vertical polarization model the guest can make use of the >> +vCPU entitlement information provided by the host to optimize >> +kernel thread scheduling. >> + >> +A subsystem reset puts all vCPU of the configuration into the >> +horizontal polarization. >> + >> +Entitlement >> ++++++++++++ >> + >> +The vertical polarization specifies that guest's vCPU can get > > "that *the* guest's vCPU" ? Yes > >> +different real CPU provisions: >> + >> +- a vCPU with vertical high entitlement specifies that this >> + vCPU gets 100% of the real CPU provisioning. >> + >> +- a vCPU with vertical medium entitlement specifies that this >> + vCPU shares the real CPU with other vCPUs. >> + >> +- a vCPU with vertical low entitlement specifies that this >> + vCPU only gets real CPU provisioning when no other vCPUs needs it. >> + >> +In the case a vCPU with vertical high entitlement does not use >> +the real CPU, the unused "slack" can be dispatched to other vCPU >> +with medium or low entitlement. >> + >> +The admin specifies a vCPU as ``dedicated`` when the vCPU is fully >> dedicated >> +to a single real CPU. >> + >> +The dedicated bit is an indication of affinity of a vCPU for a real CPU >> +while the entitlement indicates the sharing or exclusivity of use. >> + >> +Defining the topology on command line >> +------------------------------------- >> + >> +The topology can entirely be defined using -device cpu statements, >> +with the exception of CPU 0 which must be defined with the -smp >> +argument. >> + >> +For example, here we set the position of the cores 1,2,3 to >> +drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, >> +book 0, socket 0 with all horizontal polarization and not dedicated. >> +The core 4, will be set on its default position on socket 1 >> +(since we have 4 core per socket) and we define it with dedication and >> +vertical high entitlement. >> + >> +.. code-block:: bash >> + >> + $ qemu-system-s390x -m 2G \ >> + -cpu gen16b,ctop=on \ >> + -smp cpus=1,sockets=8,cores=4,maxcpus=32 \ >> + \ >> + -device >> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \ >> + -device >> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \ >> + -device >> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \ >> + \ >> + -device >> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \ >> + -device >> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \ >> + \ >> + -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \ >> + >> +QAPI interface for topology >> +--------------------------- >> + >> +Let's start QEMU with the following command: >> + >> +.. code-block:: bash >> + >> + qemu-system-s390x \ >> + -enable-kvm \ >> + -cpu z14,ctop=on \ >> + -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \ >> + \ >> + -device z14-s390x-cpu,core-id=19,polarization=3 \ >> + -device z14-s390x-cpu,core-id=11,polarization=1 \ >> + -device z14-s390x-cpu,core-id=112,polarization=3 \ >> + ... >> + >> +and see the result when using the QAPI interface. >> + >> +addons to query-cpus-fast >> ++++++++++++++++++++++++++ >> + >> +The command query-cpus-fast allows the admin to query the topology > > I'd just say "allows to query", without "the admin". OK > >> +tree and modifiers for all configured vCPUs. >> + >> +.. code-block:: QMP >> + >> + { "execute": "query-cpus-fast" } >> + { >> + "return": [ >> + { >> + "dedicated": false, >> + "thread-id": 536993, >> + "props": { >> + "core-id": 0, >> + "socket-id": 0, >> + "drawer-id": 0, >> + "book-id": 0 >> + }, >> + "cpu-state": "operating", >> + "entitlement": "medium", >> + "qom-path": "/machine/unattached/device[0]", >> + "cpu-index": 0, >> + "target": "s390x" >> + }, >> + { >> + "dedicated": false, >> + "thread-id": 537003, >> + "props": { >> + "core-id": 19, >> + "socket-id": 1, >> + "drawer-id": 0, >> + "book-id": 2 >> + }, >> + "cpu-state": "operating", >> + "entitlement": "high", >> + "qom-path": "/machine/peripheral-anon/device[0]", >> + "cpu-index": 19, >> + "target": "s390x" >> + }, >> + { >> + "dedicated": false, >> + "thread-id": 537004, >> + "props": { >> + "core-id": 11, >> + "socket-id": 1, >> + "drawer-id": 0, >> + "book-id": 1 >> + }, >> + "cpu-state": "operating", >> + "entitlement": "low", >> + "qom-path": "/machine/peripheral-anon/device[1]", >> + "cpu-index": 11, >> + "target": "s390x" >> + }, >> + { >> + "dedicated": true, >> + "thread-id": 537005, >> + "props": { >> + "core-id": 112, >> + "socket-id": 0, >> + "drawer-id": 3, >> + "book-id": 2 >> + }, >> + "cpu-state": "operating", >> + "entitlement": "high", >> + "qom-path": "/machine/peripheral-anon/device[2]", >> + "cpu-index": 112, >> + "target": "s390x" >> + } >> + ] >> + } >> + >> + >> +set-cpu-topology >> +++++++++++++++++ >> + >> +The command set-cpu-topology allows the admin to modify the topology > > "allows to modify" OK > >> +tree or the topology modifiers of a vCPU in the configuration. >> + >> +.. code-block:: QMP >> + >> + -> { "execute": "set-cpu-topology", >> + "arguments": { >> + "core-id": 11, >> + "socket-id": 0, >> + "book-id": 0, >> + "drawer-id": 0, >> + "entitlement": low, >> + "dedicated": false >> + } >> + } >> + <- {"return": {}} >> + >> +The core-id parameter is the only non optional parameter and every >> +unspecified parameter keeps its previous value. >> + >> +event CPU_POLARIZATION_CHANGE >> ++++++++++++++++++++++++++++++ >> + >> +When a guest is requests a modification of the polarization, >> +QEMU sends a CPU_POLARIZATION_CHANGE event. >> + >> +When requesting the change, the guest only specifies horizontal or >> +vertical polarization. >> +It is the job of the admin to set the dedication and fine grained >> vertical entitlement >> +in response to this event. >> + >> +Note that a vertical polarized dedicated vCPU can only have a high >> +entitlement, this gives 6 possibilities for vCPU polarization: >> + >> +- Horizontal >> +- Horizontal dedicated >> +- Vertical low >> +- Vertical medium >> +- Vertical high >> +- Vertical high dedicated >> + >> +Example of the event received when the guest issues the CPU instruction >> +Perform Topology Function PTF(0) to request an horizontal polarization: >> + >> +.. code-block:: QMP >> + >> + <- { "event": "CPU_POLARIZATION_CHANGE", >> + "data": { "polarization": 0 }, >> + "timestamp": { "seconds": 1401385907, "microseconds": 422329 } } > > Thomas > > Thanks! Regards, Pierre
On 27/02/2023 15.17, Pierre Morel wrote: > > On 2/27/23 14:58, Thomas Huth wrote: >> On 22/02/2023 15.21, Pierre Morel wrote: >>> Add some basic examples for the definition of cpu topology >>> in s390x. >>> >>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> >>> --- >>> docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ >>> docs/system/target-s390x.rst | 1 + >>> 2 files changed, 379 insertions(+) >>> create mode 100644 docs/system/s390x/cpu-topology.rst >>> >>> diff --git a/docs/system/s390x/cpu-topology.rst >>> b/docs/system/s390x/cpu-topology.rst >>> new file mode 100644 >>> index 0000000000..d470e28b97 >>> --- /dev/null >>> +++ b/docs/system/s390x/cpu-topology.rst >>> @@ -0,0 +1,378 @@ >>> +CPU topology on s390x >>> +===================== >>> + >>> +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of >>> +topology containers: drawers, books, sockets, defining a tree shaped >>> +hierarchy. >>> + >>> +The socket container contains one or more CPU entries consisting >>> +of a bitmap of three dentical CPU attributes: >> >> What do you mean by "dentical" here? > > :D i.. dentical > > I change it to identical Ok, but even with "i" at the beginning, it does not make too much sense here to me - I'd interpret "identical" as "same", but these attributes have clearly different meanings, haven't they? Thomas
On 2/27/23 15:27, Thomas Huth wrote: > On 27/02/2023 15.17, Pierre Morel wrote: >> >> On 2/27/23 14:58, Thomas Huth wrote: >>> On 22/02/2023 15.21, Pierre Morel wrote: >>>> Add some basic examples for the definition of cpu topology >>>> in s390x. >>>> >>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> >>>> --- >>>> docs/system/s390x/cpu-topology.rst | 378 >>>> +++++++++++++++++++++++++++++ >>>> docs/system/target-s390x.rst | 1 + >>>> 2 files changed, 379 insertions(+) >>>> create mode 100644 docs/system/s390x/cpu-topology.rst >>>> >>>> diff --git a/docs/system/s390x/cpu-topology.rst >>>> b/docs/system/s390x/cpu-topology.rst >>>> new file mode 100644 >>>> index 0000000000..d470e28b97 >>>> --- /dev/null >>>> +++ b/docs/system/s390x/cpu-topology.rst >>>> @@ -0,0 +1,378 @@ >>>> +CPU topology on s390x >>>> +===================== >>>> + >>>> +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of >>>> +topology containers: drawers, books, sockets, defining a tree shaped >>>> +hierarchy. >>>> + >>>> +The socket container contains one or more CPU entries consisting >>>> +of a bitmap of three dentical CPU attributes: >>> >>> What do you mean by "dentical" here? >> >> :D i.. dentical >> >> I change it to identical > > Ok, but even with "i" at the beginning, it does not make too much > sense here to me - I'd interpret "identical" as "same", but these > attributes have clearly different meanings, haven't they? > > Thomas > > Ah OK I understand what is unclear. What I mean is that in each socket we have several CPU TLE entries each entry has different attributes values and contains CPU bit in the mask for CPU with identical attributes. For example, in the case of horizontal polarization, we have one CPU TLE entry for low entitlement, one CPU TLE for medium entitlement, one for high entitlement and one for high entitlement with dedicated CPU in the case of horizontal polarization we have one CPU TLE for non dedicated CPU and one for dedicated CPU. Only CPU TLE with at least one bit (CPU) set in the mask is written inside the SYSIB. Regards, Pierre
On Wed, 2023-02-22 at 15:21 +0100, Pierre Morel wrote: > Add some basic examples for the definition of cpu topology > in s390x. > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> > --- > docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ > docs/system/target-s390x.rst | 1 + > 2 files changed, 379 insertions(+) > create mode 100644 docs/system/s390x/cpu-topology.rst > > diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst > new file mode 100644 > index 0000000000..d470e28b97 > --- /dev/null > +++ b/docs/system/s390x/cpu-topology.rst > @@ -0,0 +1,378 @@ > [...] > + > +set-cpu-topology > +++++++++++++++++ > + > +The command set-cpu-topology allows the admin to modify the topology > +tree or the topology modifiers of a vCPU in the configuration. > + > +.. code-block:: QMP > + > + -> { "execute": "set-cpu-topology", > + "arguments": { > + "core-id": 11, > + "socket-id": 0, > + "book-id": 0, > + "drawer-id": 0, > + "entitlement": low, > + "dedicated": false > + } > + } > + <- {"return": {}} This fails when building the documenation. You need to get rid of the arrows and need "" around the low. [...]
On 3/1/23 16:52, Nina Schoetterl-Glausch wrote: > On Wed, 2023-02-22 at 15:21 +0100, Pierre Morel wrote: >> Add some basic examples for the definition of cpu topology >> in s390x. >> >> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> >> --- >> docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ >> docs/system/target-s390x.rst | 1 + >> 2 files changed, 379 insertions(+) >> create mode 100644 docs/system/s390x/cpu-topology.rst >> >> diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst >> new file mode 100644 >> index 0000000000..d470e28b97 >> --- /dev/null >> +++ b/docs/system/s390x/cpu-topology.rst >> @@ -0,0 +1,378 @@ >> > [...] >> + >> +set-cpu-topology >> +++++++++++++++++ >> + >> +The command set-cpu-topology allows the admin to modify the topology >> +tree or the topology modifiers of a vCPU in the configuration. >> + >> +.. code-block:: QMP >> + >> + -> { "execute": "set-cpu-topology", >> + "arguments": { >> + "core-id": 11, >> + "socket-id": 0, >> + "book-id": 0, >> + "drawer-id": 0, >> + "entitlement": low, >> + "dedicated": false >> + } >> + } >> + <- {"return": {}} > This fails when building the documenation. > You need to get rid of the arrows and need "" around the low. > > > [...] right, thanks Pierre
diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst new file mode 100644 index 0000000000..d470e28b97 --- /dev/null +++ b/docs/system/s390x/cpu-topology.rst @@ -0,0 +1,378 @@ +CPU topology on s390x +===================== + +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of +topology containers: drawers, books, sockets, defining a tree shaped +hierarchy. + +The socket container contains one or more CPU entries consisting +of a bitmap of three dentical CPU attributes: + +- CPU type +- polarization entitlement +- dedication + +Note also that since 7.2 threads are no longer supported in the topology +and the ``-smp`` command line argument accepts only ``threads=1``. + +Prerequisites +------------- + +To use CPU topology a Linux QEMU/KVM machine providing the CPU topology facility +(STFLE bit 11) is required. + +However, since this facility has been enabled by default in an early version +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM +that QEMU supports CPU topology. + +Enabling CPU topology +--------------------- + +Currently, CPU topology is only enabled in the host model by default. + +Enabling CPU topology in a CPU model is done by setting the CPU flag +``ctop`` to ``on`` like in: + +.. code-block:: bash + + -cpu gen16b,ctop=on + +Having the topology disabled by default allows migration between +old and new QEMU without adding new flags. + +Default topology usage +---------------------- + +The CPU topology can be specified on the QEMU command line +with the ``-smp`` or the ``-device`` QEMU command arguments. + +If none of the containers attributes (drawers, books, sockets) are +specified for the ``-smp`` flag, the number of these containers +is ``1`` . + +.. code-block:: bash + + -smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32 + +or + +.. code-block:: bash + + -smp cpus=5,sockets=8,cores=4,maxcpus=32 + +When a CPU is defined by the ``-smp`` command argument, its position +inside the topology is calculated by adding the CPUs to the topology +based on the core-id starting with core-0 at position 0 of socket-0, +book-0, drawer-0 and filling all CPUs of socket-0 before to fill socket-1 +of book-0 and so on up to the last socket of the last book of the last +drawer. + +When a CPU is defined by the ``-device`` command argument, the +tree topology attributes must be all defined or all not defined. + +.. code-block:: bash + + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 + +or + +.. code-block:: bash + + -device gen16b-s390x-cpu,core-id=1,dedication=true + +If none of the tree attributes (drawer, book, sockets), are specified +for the ``-device`` argument, as for all CPUs defined on the ``-smp`` +command argument the topology tree attributes will be set by simply +adding the CPUs to the topology based on the core-id starting with +core-0 at position 0 of socket-0, book-0, drawer-0. + +QEMU will not try to solve collisions and will report an error if the +CPU topology, explicitely or implicitely defined on a ``-device`` +argument collides with the definition of a CPU implicitely defined +on the ``-smp`` argument. + +When the topology modifier attributes are not defined for the +``-device`` command argument they takes following default values: + +- dedication: ``false`` +- entitlement: ``medium`` + + +Hot plug +++++++++ + +New CPUs can be plugged using the device_add hmp command as in: + +.. code-block:: bash + + (qemu) device_add gen16b-s390x-cpu,core-id=9 + +The same placement of the CPU is derived from the core-id as described above. + +The topology can of course be fully defined: + +.. code-block:: bash + + (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 + + +Examples +++++++++ + +In the following machine we define 8 sockets with 4 cores each. + +.. code-block:: bash + + $ qemu-system-s390x -m 2G \ + -cpu gen16b,ctop=on \ + -smp cpus=5,sockets=8,cores=4,maxcpus=32 \ + -device host-s390x-cpu,core-id=14 \ + +A new CPUs can be plugged using the device_add hmp command as before: + +.. code-block:: bash + + (qemu) device_add gen16b-s390x-cpu,core-id=9 + +The core-id defines the placement of the core in the topology by +starting with core 0 in socket 0 up to maxcpus. + +In the example above: + +* There are 5 CPUs provided to the guest with the ``-smp`` command line + They will take the core-ids 0,1,2,3,4 + As we have 4 cores in a socket, we have 4 CPUs provided + to the guest in socket 0, with core-ids 0,1,2,3. + The last cpu, with core-id 4, will be on socket 1. + +* the core with ID 14 provided by the ``-device`` command line will + be placed in socket 3, with core-id 14 + +* the core with ID 9 provided by the ``device_add`` qmp command will + be placed in socket 2, with core-id 9 + + +Polarization, entitlement and dedication +---------------------------------------- + +Polarization +++++++++++++ + +The polarization is an indication given by the ``guest`` to the host +that it is able to make use of CPU provisioning information. +The guest indicates the polarization by using the PTF instruction. + +Polarization is define two models of CPU provisioning: horizontal +and vertical. + +The horizontal polarization is the default model on boot and after +subsystem reset in which the guest considers all vCPUs being having +an equal provisioning of CPUs by the host. + +In the vertical polarization model the guest can make use of the +vCPU entitlement information provided by the host to optimize +kernel thread scheduling. + +A subsystem reset puts all vCPU of the configuration into the +horizontal polarization. + +Entitlement ++++++++++++ + +The vertical polarization specifies that guest's vCPU can get +different real CPU provisions: + +- a vCPU with vertical high entitlement specifies that this + vCPU gets 100% of the real CPU provisioning. + +- a vCPU with vertical medium entitlement specifies that this + vCPU shares the real CPU with other vCPUs. + +- a vCPU with vertical low entitlement specifies that this + vCPU only gets real CPU provisioning when no other vCPUs needs it. + +In the case a vCPU with vertical high entitlement does not use +the real CPU, the unused "slack" can be dispatched to other vCPU +with medium or low entitlement. + +The admin specifies a vCPU as ``dedicated`` when the vCPU is fully dedicated +to a single real CPU. + +The dedicated bit is an indication of affinity of a vCPU for a real CPU +while the entitlement indicates the sharing or exclusivity of use. + +Defining the topology on command line +------------------------------------- + +The topology can entirely be defined using -device cpu statements, +with the exception of CPU 0 which must be defined with the -smp +argument. + +For example, here we set the position of the cores 1,2,3 to +drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, +book 0, socket 0 with all horizontal polarization and not dedicated. +The core 4, will be set on its default position on socket 1 +(since we have 4 core per socket) and we define it with dedication and +vertical high entitlement. + +.. code-block:: bash + + $ qemu-system-s390x -m 2G \ + -cpu gen16b,ctop=on \ + -smp cpus=1,sockets=8,cores=4,maxcpus=32 \ + \ + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \ + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \ + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \ + \ + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \ + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \ + \ + -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \ + +QAPI interface for topology +--------------------------- + +Let's start QEMU with the following command: + +.. code-block:: bash + + qemu-system-s390x \ + -enable-kvm \ + -cpu z14,ctop=on \ + -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \ + \ + -device z14-s390x-cpu,core-id=19,polarization=3 \ + -device z14-s390x-cpu,core-id=11,polarization=1 \ + -device z14-s390x-cpu,core-id=112,polarization=3 \ + ... + +and see the result when using the QAPI interface. + +addons to query-cpus-fast ++++++++++++++++++++++++++ + +The command query-cpus-fast allows the admin to query the topology +tree and modifiers for all configured vCPUs. + +.. code-block:: QMP + + { "execute": "query-cpus-fast" } + { + "return": [ + { + "dedicated": false, + "thread-id": 536993, + "props": { + "core-id": 0, + "socket-id": 0, + "drawer-id": 0, + "book-id": 0 + }, + "cpu-state": "operating", + "entitlement": "medium", + "qom-path": "/machine/unattached/device[0]", + "cpu-index": 0, + "target": "s390x" + }, + { + "dedicated": false, + "thread-id": 537003, + "props": { + "core-id": 19, + "socket-id": 1, + "drawer-id": 0, + "book-id": 2 + }, + "cpu-state": "operating", + "entitlement": "high", + "qom-path": "/machine/peripheral-anon/device[0]", + "cpu-index": 19, + "target": "s390x" + }, + { + "dedicated": false, + "thread-id": 537004, + "props": { + "core-id": 11, + "socket-id": 1, + "drawer-id": 0, + "book-id": 1 + }, + "cpu-state": "operating", + "entitlement": "low", + "qom-path": "/machine/peripheral-anon/device[1]", + "cpu-index": 11, + "target": "s390x" + }, + { + "dedicated": true, + "thread-id": 537005, + "props": { + "core-id": 112, + "socket-id": 0, + "drawer-id": 3, + "book-id": 2 + }, + "cpu-state": "operating", + "entitlement": "high", + "qom-path": "/machine/peripheral-anon/device[2]", + "cpu-index": 112, + "target": "s390x" + } + ] + } + + +set-cpu-topology +++++++++++++++++ + +The command set-cpu-topology allows the admin to modify the topology +tree or the topology modifiers of a vCPU in the configuration. + +.. code-block:: QMP + + -> { "execute": "set-cpu-topology", + "arguments": { + "core-id": 11, + "socket-id": 0, + "book-id": 0, + "drawer-id": 0, + "entitlement": low, + "dedicated": false + } + } + <- {"return": {}} + +The core-id parameter is the only non optional parameter and every +unspecified parameter keeps its previous value. + +event CPU_POLARIZATION_CHANGE ++++++++++++++++++++++++++++++ + +When a guest is requests a modification of the polarization, +QEMU sends a CPU_POLARIZATION_CHANGE event. + +When requesting the change, the guest only specifies horizontal or +vertical polarization. +It is the job of the admin to set the dedication and fine grained vertical entitlement +in response to this event. + +Note that a vertical polarized dedicated vCPU can only have a high +entitlement, this gives 6 possibilities for vCPU polarization: + +- Horizontal +- Horizontal dedicated +- Vertical low +- Vertical medium +- Vertical high +- Vertical high dedicated + +Example of the event received when the guest issues the CPU instruction +Perform Topology Function PTF(0) to request an horizontal polarization: + +.. code-block:: QMP + + <- { "event": "CPU_POLARIZATION_CHANGE", + "data": { "polarization": 0 }, + "timestamp": { "seconds": 1401385907, "microseconds": 422329 } } diff --git a/docs/system/target-s390x.rst b/docs/system/target-s390x.rst index f6f11433c7..94c981e732 100644 --- a/docs/system/target-s390x.rst +++ b/docs/system/target-s390x.rst @@ -34,3 +34,4 @@ Architectural features .. toctree:: s390x/bootdevices s390x/protvirt + s390x/cpu-topology
Add some basic examples for the definition of cpu topology in s390x. Signed-off-by: Pierre Morel <pmorel@linux.ibm.com> --- docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ docs/system/target-s390x.rst | 1 + 2 files changed, 379 insertions(+) create mode 100644 docs/system/s390x/cpu-topology.rst