[v3,11/16] hw/arm/raspi: Use -smp cores=<N> option to restrict enabled cores
diff mbox series

Message ID 20191019234715.25750-12-f4bug@amsat.org
State New
Headers show
Series
  • hw/arm/raspi: Add thermal/timer, improve address space, run U-boot
Related show

Commit Message

Philippe Mathieu-Daudé Oct. 19, 2019, 11:47 p.m. UTC
The abstract TYPE_BCM283X device provides a 'enabled-cpus' property
to restrict the number of cores powered on reset. This because on
real hardware the GPU is responsible of starting the cores and keep
them spinning until the Linux kernel is ready to use them.
When using the -kernel paramenter, QEMU does this by installing the
'raspi_smpboot' code when arm_boot_info::write_board_setup() is
called. This is a special feature to help the Linux kernel, and can
only be used with a Linux kernel.

Even if loaded with the -kernel option, U-boot is not Linux, thus
is not recognized as it and the raspi_smpboot code is not installed.

Upon introduction of this machine in commit 1df7d1f9303, the -smp <N>
option allowd to limit the number of cores powered on reset.
Unfortunately later commit 72649619341 added a check which made this
feature unusable:

  $ qemu-system-aarch64 -M raspi3 -smp 1
  qemu-system-aarch64: Invalid SMP CPUs 1. The min CPUs supported by machine 'raspi3' is 4

Fortunately, the -smp option allow various kind of CPU topology:

  -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]
           set the number of CPUs to 'n' [default=1]
           maxcpus= maximum number of total cpus, including
           offline CPUs for hotplug, etc
           cores= number of CPU cores on one socket (for PC, it's on one die)
           threads= number of threads on one CPU core
           dies= number of CPU dies on one socket (for PC only)
           sockets= number of discrete sockets in the system

Let's use the 'cores' argument to specify the number of cores powered
at reset to restore this feature, and allow to boot U-boot.

We can now run U-boot using:

  $ qemu-system-aarch64 -M raspi3 -smp 4,cores=1 ...

Reported-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 hw/arm/raspi.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Peter Maydell Oct. 24, 2019, 1:23 p.m. UTC | #1
On Sun, 20 Oct 2019 at 00:47, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> The abstract TYPE_BCM283X device provides a 'enabled-cpus' property
> to restrict the number of cores powered on reset. This because on
> real hardware the GPU is responsible of starting the cores and keep
> them spinning until the Linux kernel is ready to use them.
> When using the -kernel paramenter, QEMU does this by installing the
> 'raspi_smpboot' code when arm_boot_info::write_board_setup() is
> called. This is a special feature to help the Linux kernel, and can
> only be used with a Linux kernel.
>
> Even if loaded with the -kernel option, U-boot is not Linux, thus
> is not recognized as it and the raspi_smpboot code is not installed.
>
> Upon introduction of this machine in commit 1df7d1f9303, the -smp <N>
> option allowd to limit the number of cores powered on reset.
> Unfortunately later commit 72649619341 added a check which made this
> feature unusable:
>
>   $ qemu-system-aarch64 -M raspi3 -smp 1
>   qemu-system-aarch64: Invalid SMP CPUs 1. The min CPUs supported by machine 'raspi3' is 4
>
> Fortunately, the -smp option allow various kind of CPU topology:
>
>   -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]
>            set the number of CPUs to 'n' [default=1]
>            maxcpus= maximum number of total cpus, including
>            offline CPUs for hotplug, etc
>            cores= number of CPU cores on one socket (for PC, it's on one die)
>            threads= number of threads on one CPU core
>            dies= number of CPU dies on one socket (for PC only)
>            sockets= number of discrete sockets in the system
>
> Let's use the 'cores' argument to specify the number of cores powered
> at reset to restore this feature, and allow to boot U-boot.

From the help message quoted, 'cores' is about NUMA topology,
it's nothing to do with "how many cores do we have powerd on
on reset". We should definitely not be borrowing it for
an unrelated purpose.

> We can now run U-boot using:
>
>   $ qemu-system-aarch64 -M raspi3 -smp 4,cores=1 ...

Either:
 (a) we need to support "act like the firmware/GPU" for
booting guest binaries on this board, in the same way that our PSCI
support is effectively emulating firmware-level facilities
 (b) the u-boot binary should be provided to QEMU in the
right image format to use the kernel boot ABI
 (c) u-boot could be wrapped in a small guest binary that
deals with handling all the secondary cores

I'm not really a huge fan of (a) because we already have a
huge pile of random weird ways of loading guest images
that are undocumented, behave differently from board to
board or architecture to architecture, etc. I'm not too
keen on making that swamp worse, even though there's an
argument that the right UI for a raspi board model is to
provide the same "load an ELF file" handling that the real
hardware does for ELF files via the GPU/firmware combo.

thanks
-- PMM
Philippe Mathieu-Daudé Oct. 24, 2019, 1:37 p.m. UTC | #2
On 10/24/19 3:23 PM, Peter Maydell wrote:
> On Sun, 20 Oct 2019 at 00:47, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>>
>> The abstract TYPE_BCM283X device provides a 'enabled-cpus' property
>> to restrict the number of cores powered on reset. This because on
>> real hardware the GPU is responsible of starting the cores and keep
>> them spinning until the Linux kernel is ready to use them.
>> When using the -kernel paramenter, QEMU does this by installing the
>> 'raspi_smpboot' code when arm_boot_info::write_board_setup() is
>> called. This is a special feature to help the Linux kernel, and can
>> only be used with a Linux kernel.
>>
>> Even if loaded with the -kernel option, U-boot is not Linux, thus
>> is not recognized as it and the raspi_smpboot code is not installed.
>>
>> Upon introduction of this machine in commit 1df7d1f9303, the -smp <N>
>> option allowd to limit the number of cores powered on reset.
>> Unfortunately later commit 72649619341 added a check which made this
>> feature unusable:
>>
>>    $ qemu-system-aarch64 -M raspi3 -smp 1
>>    qemu-system-aarch64: Invalid SMP CPUs 1. The min CPUs supported by machine 'raspi3' is 4
>>
>> Fortunately, the -smp option allow various kind of CPU topology:
>>
>>    -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]
>>             set the number of CPUs to 'n' [default=1]
>>             maxcpus= maximum number of total cpus, including
>>             offline CPUs for hotplug, etc
>>             cores= number of CPU cores on one socket (for PC, it's on one die)
>>             threads= number of threads on one CPU core
>>             dies= number of CPU dies on one socket (for PC only)
>>             sockets= number of discrete sockets in the system
>>
>> Let's use the 'cores' argument to specify the number of cores powered
>> at reset to restore this feature, and allow to boot U-boot.
> 
>  From the help message quoted, 'cores' is about NUMA topology,
> it's nothing to do with "how many cores do we have powerd on
> on reset". We should definitely not be borrowing it for
> an unrelated purpose.

Yes, I was not sure. Is -smp only about NUMA topology?

Should I try adding another argument, such 'online_cpus_mask'?

>> We can now run U-boot using:
>>
>>    $ qemu-system-aarch64 -M raspi3 -smp 4,cores=1 ...
> 
> Either:
>   (a) we need to support "act like the firmware/GPU" for
> booting guest binaries on this board, in the same way that our PSCI
> support is effectively emulating firmware-level facilities
>   (b) the u-boot binary should be provided to QEMU in the
> right image format to use the kernel boot ABI

Well, u-boot is in the same format than the /boot/bootcode.bin 
closed-source firmware provided by the Raspberry Pi foundation. This is 
the format the GPU firmware understand when loading it on the 1st ARM core.

I'll see what I can do (ab)using the -bios parameter.

>   (c) u-boot could be wrapped in a small guest binary that
> deals with handling all the secondary cores
> 
> I'm not really a huge fan of (a) because we already have a
> huge pile of random weird ways of loading guest images
> that are undocumented, behave differently from board to
> board or architecture to architecture, etc. I'm not too
> keen on making that swamp worse, even though there's an
> argument that the right UI for a raspi board model is to
> provide the same "load an ELF file" handling that the real
> hardware does for ELF files via the GPU/firmware combo.

This patch doesn't modify hw/arm/boot.c and reuse -smp features,
it didn't seemed too invasive. Anyway I'll look for another path.

Thanks,

Phil.
Bonnans, Laurent Nov. 7, 2019, 10:41 a.m. UTC | #3
On 10/20/19 1:47 AM, Philippe Mathieu-Daudé wrote:
> The abstract TYPE_BCM283X device provides a 'enabled-cpus' property
> to restrict the number of cores powered on reset. This because on
> real hardware the GPU is responsible of starting the cores and keep
> them spinning until the Linux kernel is ready to use them.
> When using the -kernel paramenter, QEMU does this by installing the
> 'raspi_smpboot' code when arm_boot_info::write_board_setup() is
> called. This is a special feature to help the Linux kernel, and can
> only be used with a Linux kernel.
>
> Even if loaded with the -kernel option, U-boot is not Linux, thus
> is not recognized as it and the raspi_smpboot code is not installed.
>
> Upon introduction of this machine in commit 1df7d1f9303, the -smp <N>
> option allowd to limit the number of cores powered on reset.
> Unfortunately later commit 72649619341 added a check which made this
> feature unusable:
>
>    $ qemu-system-aarch64 -M raspi3 -smp 1
>    qemu-system-aarch64: Invalid SMP CPUs 1. The min CPUs supported by machine 'raspi3' is 4
>
> Fortunately, the -smp option allow various kind of CPU topology:
>
>    -smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]
>             set the number of CPUs to 'n' [default=1]
>             maxcpus= maximum number of total cpus, including
>             offline CPUs for hotplug, etc
>             cores= number of CPU cores on one socket (for PC, it's on one die)
>             threads= number of threads on one CPU core
>             dies= number of CPU dies on one socket (for PC only)
>             sockets= number of discrete sockets in the system
>
> Let's use the 'cores' argument to specify the number of cores powered
> at reset to restore this feature, and allow to boot U-boot.
>
> We can now run U-boot using:
>
>    $ qemu-system-aarch64 -M raspi3 -smp 4,cores=1 ...
>
> Reported-by: Laurent Bonnans <laurent.bonnans@here.com>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   hw/arm/raspi.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/hw/arm/raspi.c b/hw/arm/raspi.c
> index 569d85c11a..45d3f91f95 100644
> --- a/hw/arm/raspi.c
> +++ b/hw/arm/raspi.c
> @@ -190,8 +190,8 @@ static void raspi_init(MachineState *machine, int version)
>       /* Setup the SOC */
>       object_property_add_const_link(OBJECT(&s->soc), "ram", OBJECT(&s->ram),
>                                      &error_abort);
> -    object_property_set_int(OBJECT(&s->soc), machine->smp.cpus, "enabled-cpus",
> -                            &error_abort);
> +    object_property_set_int(OBJECT(&s->soc), machine->smp.cores,
> +                            "enabled-cpus", &error_abort);
>       int board_rev = version == 3 ? 0xa02082 : 0xa21041;
>       object_property_set_int(OBJECT(&s->soc), board_rev, "board-rev",
>                               &error_abort);

Hi Phil,

Thanks for the patch, I finally got the chance to make some tests (sorry 
about the delay).

Using the proposed -smp options indeed helps to run u-boot elfs 
directly. However, the cores fail to start when switching to linux 
(tested on raspi2):

[ 0.071030] smp: Bringing up secondary CPUs ...
[ 1.157876] CPU1: failed to come online
[ 2.219899] CPU2: failed to come online
[ 3.285412] CPU3: failed to come online
[ 3.286137] smp: Brought up 1 node, 1 CPU
[ 3.286766] SMP: Total of 1 processors activated (125.00 BogoMIPS).
[ 3.287442] CPU: All CPU(s) started in SVC mode.

The behavior persist even without using the option on the command line.
The normal behavior is restored if I use "-smp 4, cores=4"

Greetings,

Laurent

Patch
diff mbox series

diff --git a/hw/arm/raspi.c b/hw/arm/raspi.c
index 569d85c11a..45d3f91f95 100644
--- a/hw/arm/raspi.c
+++ b/hw/arm/raspi.c
@@ -190,8 +190,8 @@  static void raspi_init(MachineState *machine, int version)
     /* Setup the SOC */
     object_property_add_const_link(OBJECT(&s->soc), "ram", OBJECT(&s->ram),
                                    &error_abort);
-    object_property_set_int(OBJECT(&s->soc), machine->smp.cpus, "enabled-cpus",
-                            &error_abort);
+    object_property_set_int(OBJECT(&s->soc), machine->smp.cores,
+                            "enabled-cpus", &error_abort);
     int board_rev = version == 3 ? 0xa02082 : 0xa21041;
     object_property_set_int(OBJECT(&s->soc), board_rev, "board-rev",
                             &error_abort);