[U-Boot,v2,23/23] sunxi: A64: add 32-bit SPL support

When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
use the more compact Thumb2 encoding, which only exists for AArch32
code. This makes the SPL rather big, up to a point where any code
additions or even a different compiler may easily exceed the 32KB limit
that the Allwinner BROM imposes.
Introduce a separate, mostly generic sun50i-a64 configuration, which
defines the CPU_V7 symbol and thus will create a 32-bit binary using
the memory-saving Thumb2 encoding.
This should only be used for the SPL, the U-Boot proper should still be
using the existing 64-bit configuration. The SPL code can switch to
AArch64 if needed, so a 32-bit SPL can be combined with a 64-bit U-Boot
proper to eventually launch arm64 kernels.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 board/sunxi/Kconfig            | 14 ++++++++++++--
 configs/pine64_plus_defconfig  |  2 +-
 configs/sun50i_spl32_defconfig | 10 ++++++++++
 include/configs/sunxi-common.h |  2 +-
 4 files changed, 24 insertions(+), 4 deletions(-)
 create mode 100644 configs/sun50i_spl32_defconfig

Hi Andre,

On 4 December 2016 at 18:52, Andre Przywara <andre.przywara@arm.com> wrote:
> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
> use the more compact Thumb2 encoding, which only exists for AArch32
> code. This makes the SPL rather big, up to a point where any code
> additions or even a different compiler may easily exceed the 32KB limit
> that the Allwinner BROM imposes.
> Introduce a separate, mostly generic sun50i-a64 configuration, which
> defines the CPU_V7 symbol and thus will create a 32-bit binary using
> the memory-saving Thumb2 encoding.
> This should only be used for the SPL, the U-Boot proper should still be
> using the existing 64-bit configuration. The SPL code can switch to
> AArch64 if needed, so a 32-bit SPL can be combined with a 64-bit U-Boot
> proper to eventually launch arm64 kernels.

So if I understand correctly, you want SPL to be 32-bit and U-Boot
proper to be 64-bit? And you are adding a new board config for that?

Instead, can you do something similar to tegra, which uses ARMv4t for
SPL and ARMv7 for U-Boot proper?

>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  board/sunxi/Kconfig            | 14 ++++++++++++--
>  configs/pine64_plus_defconfig  |  2 +-
>  configs/sun50i_spl32_defconfig | 10 ++++++++++
>  include/configs/sunxi-common.h |  2 +-
>  4 files changed, 24 insertions(+), 4 deletions(-)
>  create mode 100644 configs/sun50i_spl32_defconfig

Regards,
Simon

On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
> use the more compact Thumb2 encoding, which only exists for AArch32
> code. This makes the SPL rather big, up to a point where any code
> additions or even a different compiler may easily exceed the 32KB limit
> that the Allwinner BROM imposes.
> Introduce a separate, mostly generic sun50i-a64 configuration, which
> defines the CPU_V7 symbol and thus will create a 32-bit binary using
> the memory-saving Thumb2 encoding.

"mostly generic". Where do you draw the line? How do you deal with a
board that would use a different UART? a different MMC? different
memory configuration.?

This looks like it's not generic at all, it's just a configuration for
the Pine64.

Maxime

Hi,

On 06/12/16 11:28, Maxime Ripard wrote:
> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>> use the more compact Thumb2 encoding, which only exists for AArch32
>> code. This makes the SPL rather big, up to a point where any code
>> additions or even a different compiler may easily exceed the 32KB limit
>> that the Allwinner BROM imposes.
>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>> the memory-saving Thumb2 encoding.
> 
> "mostly generic". Where do you draw the line? How do you deal with a
> board that would use a different UART? a different MMC? different
> memory configuration.?

My impression was that it's rather pointless to provide another set of
32-bit SPL defconfigs for each board again, especially given that for
the SPL's needs the boards so far seem to be very similar.
For the loading part we will probably go with what the BROM already
started: load more data from one of the BROM boot sources, which is
fixed in the SoC and can't be really changed by a board vendor anyway.
Which really leaves the DRAM setup and the UART.
I can't predict the future, but so far those A64 boards look fairly
similar in this respect. So I just avoid having another SPL defconfig
for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
because this doesn't hurt on the Pine64, so less churn here.

So if you know of any board which breaks this assumption, I am happy to
hear about it and see if it can be integrated.

Actually I had the idea of eventually going the other way around:
Providing one U-Boot proper A64 defconfig and let the DT (provided by
the SPL) sort out the differences. Then we might be able to live with
separate SPL defconfigs. But that's another patchset and probably quite
some work.

> This looks like it's not generic at all, it's just a configuration for
> the Pine64.

And the BananaPi M64. And the upcoming Olimex board (trusting their
latest published schematics).
If in need, we can always provide separate defconfigs for "odd" boards.

So this is the best solution I came up with, if you have a better one: I
am all ears.
Would it make sense to rename this file to not claim universal coverage?

Cheers,
Andre.

On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
> Hi,
> 
> On 06/12/16 11:28, Maxime Ripard wrote:
> > On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
> >> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
> >> use the more compact Thumb2 encoding, which only exists for AArch32
> >> code. This makes the SPL rather big, up to a point where any code
> >> additions or even a different compiler may easily exceed the 32KB limit
> >> that the Allwinner BROM imposes.
> >> Introduce a separate, mostly generic sun50i-a64 configuration, which
> >> defines the CPU_V7 symbol and thus will create a 32-bit binary using
> >> the memory-saving Thumb2 encoding.
> > 
> > "mostly generic". Where do you draw the line? How do you deal with a
> > board that would use a different UART? a different MMC? different
> > memory configuration.?
> 
> My impression was that it's rather pointless to provide another set of
> 32-bit SPL defconfigs for each board again, especially given that for
> the SPL's needs the boards so far seem to be very similar.
> For the loading part we will probably go with what the BROM already
> started: load more data from one of the BROM boot sources, which is
> fixed in the SoC and can't be really changed by a board vendor anyway.
> Which really leaves the DRAM setup and the UART.

So you plan on enabling all BROM boot sources as well (NAND, SPI) ?

> I can't predict the future, but so far those A64 boards look fairly
> similar in this respect. So I just avoid having another SPL defconfig
> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
> because this doesn't hurt on the Pine64, so less churn here.
> 
> So if you know of any board which breaks this assumption, I am happy to
> hear about it and see if it can be integrated.

I know at least of one board that uses the UART3 on A33, instead of
UART0. The trend is very clear on the A64 and the previous SoCs, but
we also had some variations, so we need to take that into
account. Which brings me back to my original question, where do you
draw the line ? :)

And that would prevent us from doing any kind of DRAM settings
enhancements in the future, every one using the best common
denominator, which seems inefficient.

> Actually I had the idea of eventually going the other way around:
> Providing one U-Boot proper A64 defconfig and let the DT (provided by
> the SPL) sort out the differences. Then we might be able to live with
> separate SPL defconfigs. But that's another patchset and probably quite
> some work.

That would work for MMC and UART (in u-boot, not in the SPL), but not
for the RAM setup.

Maxime

Hi,

On 12/12/16 15:13, Maxime Ripard wrote:
> On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
>> Hi,
>>
>> On 06/12/16 11:28, Maxime Ripard wrote:
>>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
>>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>>>> use the more compact Thumb2 encoding, which only exists for AArch32
>>>> code. This makes the SPL rather big, up to a point where any code
>>>> additions or even a different compiler may easily exceed the 32KB limit
>>>> that the Allwinner BROM imposes.
>>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>>>> the memory-saving Thumb2 encoding.
>>>
>>> "mostly generic". Where do you draw the line? How do you deal with a
>>> board that would use a different UART? a different MMC? different
>>> memory configuration.?
>>
>> My impression was that it's rather pointless to provide another set of
>> 32-bit SPL defconfigs for each board again, especially given that for
>> the SPL's needs the boards so far seem to be very similar.
>> For the loading part we will probably go with what the BROM already
>> started: load more data from one of the BROM boot sources, which is
>> fixed in the SoC and can't be really changed by a board vendor anyway.
>> Which really leaves the DRAM setup and the UART.
> 
> So you plan on enabling all BROM boot sources as well (NAND, SPI) ?

In fact SPI works already (with little to no changes).
And I don't care about NAND, really ;-)
Is anyone aware of an A64 board using this?

>> I can't predict the future, but so far those A64 boards look fairly
>> similar in this respect. So I just avoid having another SPL defconfig
>> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
>> because this doesn't hurt on the Pine64, so less churn here.
>>
>> So if you know of any board which breaks this assumption, I am happy to
>> hear about it and see if it can be integrated.
> 
> I know at least of one board that uses the UART3 on A33, instead of
> UART0. The trend is very clear on the A64 and the previous SoCs, but
> we also had some variations, so we need to take that into
> account. Which brings me back to my original question, where do you
> draw the line ? :)

I don't know, and to make this clear: I see the point in having separate
configs for the SPL, but due to the 32-bit/64-bit split we probably need
_two_ sets of defconfigs, which gets pretty messy very quickly.
Especially given that they are very similar.

So how do we avoid this? Can we somehow share a defconfig between armv8
and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
the same file.

> And that would prevent us from doing any kind of DRAM settings
> enhancements in the future, every one using the best common
> denominator, which seems inefficient.

Yeah, I think this is a much better point. I was a lot in DRAM land in
the past few days, and I think we _have_ different DRAMs already:
1) The Pine64 uses DDR3-1333 DRAM chips @ 672 MHz
2) The SoPine and the Pinebook use LPDDR3 DRAM, which requires a
slightly different setup, also the stock frequency in that one boot0.bin
floating around is much lower (524 MHz, IIRC).
3) The BananaPi M64 and Theobroma's board use SKhynix DDR3 DRAM chips
which are from a DDR3-1600 bin. I think Philipp reported to have it
running at 800 MHz with some sane JEDEC based timing happily, so it's
worth to enable this there.

Which brings us to the following complications:
a) We have similar, but still different DRAM _controllers_ (H3, A64,
H5). Those have a slightly different register set, though it seems to be
mostly missing/added features, so nothing really conflicting.
b) We have different DRAM _chips_ being used on top of possibly any of
those controllers.

So this brings us from "one type of DRAM chip on top of one DRAM
controller" today for the H3 driver to "multiple DRAM chips on top of
slightly different controllers", which is a totally different story.
I was looking into significantly reworking the DRAM driver to address
that, but was hoping to defer this to a later stage.

While the DRAM controller part can probably be solved by #ifdef'ing or
static parameters, I wonder if we should really explore how to address
different DRAM chips properly:
I) We create a DT binding loosely based on the "jedec,lpddr2-timings"
one in the kernel and use of-platdata as Simon suggested.
II) We create some fixed tables of standard (JEDEC) timings in the
driver and let one of those tables be selected at runtime based on some
parameters, for instance in the SPL header. This could as easy as
"DDR3-1333" vs. "LPDDR3-1066". This would allow us to adopt an existing
SPL to multiple chips/boards without needing to rebuild it, possibly
even open the door to auto-detection. For instance I found the LPDDR3
boot0 bailing out on the Pine64, so we might at least detect the
difference there.
III) Something in between.

It's just a pity that this could hold off upstreaming the Pine64 SPL
support.

Cheers,
Andre.

>> Actually I had the idea of eventually going the other way around:
>> Providing one U-Boot proper A64 defconfig and let the DT (provided by
>> the SPL) sort out the differences. Then we might be able to live with
>> separate SPL defconfigs. But that's another patchset and probably quite
>> some work.
> 
> That would work for MMC and UART (in u-boot, not in the SPL), but not
> for the RAM setup.

On Tue, Dec 13, 2016 at 12:04 AM, Andre Przywara <andre.przywara@arm.com> wrote:
> Hi,
>
> On 12/12/16 15:13, Maxime Ripard wrote:
>> On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
>>> Hi,
>>>
>>> On 06/12/16 11:28, Maxime Ripard wrote:
>>>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
>>>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>>>>> use the more compact Thumb2 encoding, which only exists for AArch32
>>>>> code. This makes the SPL rather big, up to a point where any code
>>>>> additions or even a different compiler may easily exceed the 32KB limit
>>>>> that the Allwinner BROM imposes.
>>>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>>>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>>>>> the memory-saving Thumb2 encoding.
>>>>
>>>> "mostly generic". Where do you draw the line? How do you deal with a
>>>> board that would use a different UART? a different MMC? different
>>>> memory configuration.?
>>>
>>> My impression was that it's rather pointless to provide another set of
>>> 32-bit SPL defconfigs for each board again, especially given that for
>>> the SPL's needs the boards so far seem to be very similar.
>>> For the loading part we will probably go with what the BROM already
>>> started: load more data from one of the BROM boot sources, which is
>>> fixed in the SoC and can't be really changed by a board vendor anyway.
>>> Which really leaves the DRAM setup and the UART.
>>
>> So you plan on enabling all BROM boot sources as well (NAND, SPI) ?
>
> In fact SPI works already (with little to no changes).
> And I don't care about NAND, really ;-)
> Is anyone aware of an A64 board using this?
>
>>> I can't predict the future, but so far those A64 boards look fairly
>>> similar in this respect. So I just avoid having another SPL defconfig
>>> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
>>> because this doesn't hurt on the Pine64, so less churn here.
>>>
>>> So if you know of any board which breaks this assumption, I am happy to
>>> hear about it and see if it can be integrated.
>>
>> I know at least of one board that uses the UART3 on A33, instead of
>> UART0. The trend is very clear on the A64 and the previous SoCs, but
>> we also had some variations, so we need to take that into
>> account. Which brings me back to my original question, where do you
>> draw the line ? :)
>
> I don't know, and to make this clear: I see the point in having separate
> configs for the SPL, but due to the 32-bit/64-bit split we probably need
> _two_ sets of defconfigs, which gets pretty messy very quickly.
> Especially given that they are very similar.
>
> So how do we avoid this? Can we somehow share a defconfig between armv8
> and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
> the same file.
>
>> And that would prevent us from doing any kind of DRAM settings
>> enhancements in the future, every one using the best common
>> denominator, which seems inefficient.
>
> Yeah, I think this is a much better point. I was a lot in DRAM land in
> the past few days, and I think we _have_ different DRAMs already:
> 1) The Pine64 uses DDR3-1333 DRAM chips @ 672 MHz
> 2) The SoPine and the Pinebook use LPDDR3 DRAM, which requires a
> slightly different setup, also the stock frequency in that one boot0.bin
> floating around is much lower (524 MHz, IIRC).
> 3) The BananaPi M64 and Theobroma's board use SKhynix DDR3 DRAM chips
> which are from a DDR3-1600 bin. I think Philipp reported to have it
> running at 800 MHz with some sane JEDEC based timing happily, so it's
> worth to enable this there.
>
> Which brings us to the following complications:
> a) We have similar, but still different DRAM _controllers_ (H3, A64,
> H5). Those have a slightly different register set, though it seems to be
> mostly missing/added features, so nothing really conflicting.
> b) We have different DRAM _chips_ being used on top of possibly any of
> those controllers.
>
> So this brings us from "one type of DRAM chip on top of one DRAM
> controller" today for the H3 driver to "multiple DRAM chips on top of
> slightly different controllers", which is a totally different story.
> I was looking into significantly reworking the DRAM driver to address
> that, but was hoping to defer this to a later stage.
>
> While the DRAM controller part can probably be solved by #ifdef'ing or
> static parameters, I wonder if we should really explore how to address
> different DRAM chips properly:
> I) We create a DT binding loosely based on the "jedec,lpddr2-timings"
> one in the kernel and use of-platdata as Simon suggested.
> II) We create some fixed tables of standard (JEDEC) timings in the
> driver and let one of those tables be selected at runtime based on some
> parameters, for instance in the SPL header. This could as easy as
> "DDR3-1333" vs. "LPDDR3-1066". This would allow us to adopt an existing
> SPL to multiple chips/boards without needing to rebuild it, possibly
> even open the door to auto-detection. For instance I found the LPDDR3
> boot0 bailing out on the Pine64, so we might at least detect the
> difference there.

Bailing out should be expected. LPDDR3 runs at a lower voltage, so the
standard DDR3 chip w/ LPDDR3 boot0 would not get sufficient power to
work. Also, it seems LPDDR3 uses a different protocol. I'm not an expert
on this though. See

https://blogs.synopsys.com/committedtomemory/2014/01/10/when-is-lpddr3-not-lpddr3-when-its-ddr3l/
https://blogs.synopsys.com/committedtomemory/files/2014/01/DDR3-DDR3L-LPDDR3-Comparison.jpg

As I see it, there would need to be 2 settings:

  a) RAM type. We already have a Kconfig option to support LPDDR2 for
     the A83T.

  b) DRAM speed bin. This would probably select one of the standard
     JEDEC timings for the selected DRAM type.

Note that Allwinner's boot0 supposedly just has all the DRAM parameters
in its header, including options to use standard or custom timings.

Regards
ChenYu

> III) Something in between.
>
> It's just a pity that this could hold off upstreaming the Pine64 SPL
> support.
>
> Cheers,
> Andre.
>
>>> Actually I had the idea of eventually going the other way around:
>>> Providing one U-Boot proper A64 defconfig and let the DT (provided by
>>> the SPL) sort out the differences. Then we might be able to live with
>>> separate SPL defconfigs. But that's another patchset and probably quite
>>> some work.
>>
>> That would work for MMC and UART (in u-boot, not in the SPL), but not
>> for the RAM setup.

Hi,

On 12/12/16 16:18, Chen-Yu Tsai wrote:
> On Tue, Dec 13, 2016 at 12:04 AM, Andre Przywara <andre.przywara@arm.com> wrote:
>> Hi,
>>
>> On 12/12/16 15:13, Maxime Ripard wrote:
>>> On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
>>>> Hi,
>>>>
>>>> On 06/12/16 11:28, Maxime Ripard wrote:
>>>>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
>>>>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>>>>>> use the more compact Thumb2 encoding, which only exists for AArch32
>>>>>> code. This makes the SPL rather big, up to a point where any code
>>>>>> additions or even a different compiler may easily exceed the 32KB limit
>>>>>> that the Allwinner BROM imposes.
>>>>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>>>>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>>>>>> the memory-saving Thumb2 encoding.
>>>>>
>>>>> "mostly generic". Where do you draw the line? How do you deal with a
>>>>> board that would use a different UART? a different MMC? different
>>>>> memory configuration.?
>>>>
>>>> My impression was that it's rather pointless to provide another set of
>>>> 32-bit SPL defconfigs for each board again, especially given that for
>>>> the SPL's needs the boards so far seem to be very similar.
>>>> For the loading part we will probably go with what the BROM already
>>>> started: load more data from one of the BROM boot sources, which is
>>>> fixed in the SoC and can't be really changed by a board vendor anyway.
>>>> Which really leaves the DRAM setup and the UART.
>>>
>>> So you plan on enabling all BROM boot sources as well (NAND, SPI) ?
>>
>> In fact SPI works already (with little to no changes).
>> And I don't care about NAND, really ;-)
>> Is anyone aware of an A64 board using this?
>>
>>>> I can't predict the future, but so far those A64 boards look fairly
>>>> similar in this respect. So I just avoid having another SPL defconfig
>>>> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
>>>> because this doesn't hurt on the Pine64, so less churn here.
>>>>
>>>> So if you know of any board which breaks this assumption, I am happy to
>>>> hear about it and see if it can be integrated.
>>>
>>> I know at least of one board that uses the UART3 on A33, instead of
>>> UART0. The trend is very clear on the A64 and the previous SoCs, but
>>> we also had some variations, so we need to take that into
>>> account. Which brings me back to my original question, where do you
>>> draw the line ? :)
>>
>> I don't know, and to make this clear: I see the point in having separate
>> configs for the SPL, but due to the 32-bit/64-bit split we probably need
>> _two_ sets of defconfigs, which gets pretty messy very quickly.
>> Especially given that they are very similar.
>>
>> So how do we avoid this? Can we somehow share a defconfig between armv8
>> and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
>> the same file.
>>
>>> And that would prevent us from doing any kind of DRAM settings
>>> enhancements in the future, every one using the best common
>>> denominator, which seems inefficient.
>>
>> Yeah, I think this is a much better point. I was a lot in DRAM land in
>> the past few days, and I think we _have_ different DRAMs already:
>> 1) The Pine64 uses DDR3-1333 DRAM chips @ 672 MHz
>> 2) The SoPine and the Pinebook use LPDDR3 DRAM, which requires a
>> slightly different setup, also the stock frequency in that one boot0.bin
>> floating around is much lower (524 MHz, IIRC).
>> 3) The BananaPi M64 and Theobroma's board use SKhynix DDR3 DRAM chips
>> which are from a DDR3-1600 bin. I think Philipp reported to have it
>> running at 800 MHz with some sane JEDEC based timing happily, so it's
>> worth to enable this there.
>>
>> Which brings us to the following complications:
>> a) We have similar, but still different DRAM _controllers_ (H3, A64,
>> H5). Those have a slightly different register set, though it seems to be
>> mostly missing/added features, so nothing really conflicting.
>> b) We have different DRAM _chips_ being used on top of possibly any of
>> those controllers.
>>
>> So this brings us from "one type of DRAM chip on top of one DRAM
>> controller" today for the H3 driver to "multiple DRAM chips on top of
>> slightly different controllers", which is a totally different story.
>> I was looking into significantly reworking the DRAM driver to address
>> that, but was hoping to defer this to a later stage.
>>
>> While the DRAM controller part can probably be solved by #ifdef'ing or
>> static parameters, I wonder if we should really explore how to address
>> different DRAM chips properly:
>> I) We create a DT binding loosely based on the "jedec,lpddr2-timings"
>> one in the kernel and use of-platdata as Simon suggested.
>> II) We create some fixed tables of standard (JEDEC) timings in the
>> driver and let one of those tables be selected at runtime based on some
>> parameters, for instance in the SPL header. This could as easy as
>> "DDR3-1333" vs. "LPDDR3-1066". This would allow us to adopt an existing
>> SPL to multiple chips/boards without needing to rebuild it, possibly
>> even open the door to auto-detection. For instance I found the LPDDR3
>> boot0 bailing out on the Pine64, so we might at least detect the
>> difference there.
> 
> Bailing out should be expected. LPDDR3 runs at a lower voltage, so the
> standard DDR3 chip w/ LPDDR3 boot0 would not get sufficient power to
> work.

I don't think this is the reason. The AXP803 has a pin to externally
select a reset voltage for DCDC5, which drives the DRAM chips. So a
board vendor would connect this pin appropriately to match the
_soldered_ DRAM chips.

> Also, it seems LPDDR3 uses a different protocol.

Yes, that's what I found also, and I hope that this is sufficient enough
to reliably tell the two apart. Somehow the algorithm found out that
something is wrong and quit. I was wondering if this could be used in a
reliable way.

> I'm not an expert
> on this though. See
> 
> https://blogs.synopsys.com/committedtomemory/2014/01/10/when-is-lpddr3-not-lpddr3-when-its-ddr3l/
> https://blogs.synopsys.com/committedtomemory/files/2014/01/DDR3-DDR3L-LPDDR3-Comparison.jpg

Thanks for those links, I will add these to my constantly piling up heap
of "how DRAM works" documents that populate my desktop lately, leading
to the OOM killer going 'round yesterday on my machine ;-)

> As I see it, there would need to be 2 settings:
> 
>   a) RAM type. We already have a Kconfig option to support LPDDR2 for
>      the A83T.

Yes, though it's just DDR3 vs LPDDR3 in our case.

>   b) DRAM speed bin. This would probably select one of the standard
>      JEDEC timings for the selected DRAM type.

Yes.

> 
> Note that Allwinner's boot0 supposedly just has all the DRAM parameters
> in its header, including options to use standard or custom timings.

By staring at the boot0 disassembly lately I believe the existing
boot0's ignore all those numerous parameters unless a bit in this tpr13
parameter is set (which is clear on the A64 one).
Instead they just derive the timing parameters from the given frequency,
though in a slightly non-standard (wrong?) way.

So yes: I hope that just "memory type" and "frequency/JEDEC speed bin"
should be sufficient to get the right DRAM setup, which could then
happily live as two Kconfig parameters.

Cheers,
Andre.


> 
>> III) Something in between.
>>
>> It's just a pity that this could hold off upstreaming the Pine64 SPL
>> support.
>>
>> Cheers,
>> Andre.
>>
>>>> Actually I had the idea of eventually going the other way around:
>>>> Providing one U-Boot proper A64 defconfig and let the DT (provided by
>>>> the SPL) sort out the differences. Then we might be able to live with
>>>> separate SPL defconfigs. But that's another patchset and probably quite
>>>> some work.
>>>
>>> That would work for MMC and UART (in u-boot, not in the SPL), but not
>>> for the RAM setup.

On Mon, Dec 12, 2016 at 04:04:23PM +0000, Andre Przywara wrote:
> Hi,
> 
> On 12/12/16 15:13, Maxime Ripard wrote:
> > On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
> >> Hi,
> >>
> >> On 06/12/16 11:28, Maxime Ripard wrote:
> >>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
> >>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
> >>>> use the more compact Thumb2 encoding, which only exists for AArch32
> >>>> code. This makes the SPL rather big, up to a point where any code
> >>>> additions or even a different compiler may easily exceed the 32KB limit
> >>>> that the Allwinner BROM imposes.
> >>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
> >>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
> >>>> the memory-saving Thumb2 encoding.
> >>>
> >>> "mostly generic". Where do you draw the line? How do you deal with a
> >>> board that would use a different UART? a different MMC? different
> >>> memory configuration.?
> >>
> >> My impression was that it's rather pointless to provide another set of
> >> 32-bit SPL defconfigs for each board again, especially given that for
> >> the SPL's needs the boards so far seem to be very similar.
> >> For the loading part we will probably go with what the BROM already
> >> started: load more data from one of the BROM boot sources, which is
> >> fixed in the SoC and can't be really changed by a board vendor anyway.
> >> Which really leaves the DRAM setup and the UART.
> > 
> > So you plan on enabling all BROM boot sources as well (NAND, SPI) ?
> 
> In fact SPI works already (with little to no changes).

It's the little changes that I'm interested in to be honest :)

> And I don't care about NAND, really ;-)
> Is anyone aware of an A64 board using this?

Well, it's one of the possible boot source, so we have to consider it
and not ignore it entirely hoping that no one will use it, ever.

SPI booting was not used for 5-6 years, until someone started that
trend and now we have a significant number of boards implementing it.

> >> I can't predict the future, but so far those A64 boards look fairly
> >> similar in this respect. So I just avoid having another SPL defconfig
> >> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
> >> because this doesn't hurt on the Pine64, so less churn here.
> >>
> >> So if you know of any board which breaks this assumption, I am happy to
> >> hear about it and see if it can be integrated.
> > 
> > I know at least of one board that uses the UART3 on A33, instead of
> > UART0. The trend is very clear on the A64 and the previous SoCs, but
> > we also had some variations, so we need to take that into
> > account. Which brings me back to my original question, where do you
> > draw the line ? :)
> 
> I don't know, and to make this clear: I see the point in having separate
> configs for the SPL, but due to the 32-bit/64-bit split we probably need
> _two_ sets of defconfigs, which gets pretty messy very quickly.
> Especially given that they are very similar.
> 
> So how do we avoid this? Can we somehow share a defconfig between armv8
> and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
> the same file.

I guess the easiest and most robust solution to do this would be to
just generate it from the defconfig. It really feels from your patches
that it's just a matter of sed -i 's/CONFIG_MACH_SUN50I/CONFIG_MACH_SUN50I_32'
on the "real" defconfig.

And we don't have to matter about keeping it in sync, or the board
specific init that might need to be done.

Maxime

Hi,

On 16/12/16 14:52, Maxime Ripard wrote:
> On Mon, Dec 12, 2016 at 04:04:23PM +0000, Andre Przywara wrote:
>> Hi,
>>
>> On 12/12/16 15:13, Maxime Ripard wrote:
>>> On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
>>>> Hi,
>>>>
>>>> On 06/12/16 11:28, Maxime Ripard wrote:
>>>>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
>>>>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>>>>>> use the more compact Thumb2 encoding, which only exists for AArch32
>>>>>> code. This makes the SPL rather big, up to a point where any code
>>>>>> additions or even a different compiler may easily exceed the 32KB limit
>>>>>> that the Allwinner BROM imposes.
>>>>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>>>>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>>>>>> the memory-saving Thumb2 encoding.
>>>>>
>>>>> "mostly generic". Where do you draw the line? How do you deal with a
>>>>> board that would use a different UART? a different MMC? different
>>>>> memory configuration.?
>>>>
>>>> My impression was that it's rather pointless to provide another set of
>>>> 32-bit SPL defconfigs for each board again, especially given that for
>>>> the SPL's needs the boards so far seem to be very similar.
>>>> For the loading part we will probably go with what the BROM already
>>>> started: load more data from one of the BROM boot sources, which is
>>>> fixed in the SoC and can't be really changed by a board vendor anyway.
>>>> Which really leaves the DRAM setup and the UART.
>>>
>>> So you plan on enabling all BROM boot sources as well (NAND, SPI) ?
>>
>> In fact SPI works already (with little to no changes).
> 
> It's the little changes that I'm interested in to be honest :)

Well, thinking about it again I think normal SPI boot (legacy U-Boot
image as the payload) requires no changes: just enable it in the
defconfig. The patches I was talking about were about enabling FIT
support on top of it.

The reason why it works is due to Siarhei's SPL SPI code and due to the
fact it is kind of "supported" on these boards, which is not true for
NAND on most boards, AFAIK.
But if we get support for that, it would just work the same way, due to
the boot source detection. So I think handling multiple boot sources
within one SPL binary is a general sunxi SPL feature already implemented
today. I definitely use the same thing for FEL, SPI and SD and eMMC.

>> And I don't care about NAND, really ;-)
>> Is anyone aware of an A64 board using this?
> 
> Well, it's one of the possible boot source, so we have to consider it
> and not ignore it entirely hoping that no one will use it, ever.

But there isn't any support for it so far, is there? Even if we wanted
to compile an SPL just for NAND.

> SPI booting was not used for 5-6 years, until someone started that
> trend and now we have a significant number of boards implementing it.
> 
>>>> I can't predict the future, but so far those A64 boards look fairly
>>>> similar in this respect. So I just avoid having another SPL defconfig
>>>> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
>>>> because this doesn't hurt on the Pine64, so less churn here.
>>>>
>>>> So if you know of any board which breaks this assumption, I am happy to
>>>> hear about it and see if it can be integrated.
>>>
>>> I know at least of one board that uses the UART3 on A33, instead of
>>> UART0. The trend is very clear on the A64 and the previous SoCs, but
>>> we also had some variations, so we need to take that into
>>> account. Which brings me back to my original question, where do you
>>> draw the line ? :)
>>
>> I don't know, and to make this clear: I see the point in having separate
>> configs for the SPL, but due to the 32-bit/64-bit split we probably need
>> _two_ sets of defconfigs, which gets pretty messy very quickly.
>> Especially given that they are very similar.
>>
>> So how do we avoid this? Can we somehow share a defconfig between armv8
>> and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
>> the same file.
> 
> I guess the easiest and most robust solution to do this would be to
> just generate it from the defconfig. It really feels from your patches
> that it's just a matter of sed -i 's/CONFIG_MACH_SUN50I/CONFIG_MACH_SUN50I_32'
> on the "real" defconfig.

Yes, the difference between the defconfigs is really minimal. They look
more different at the moment because we don't need Ethernet and USB in
the SPL, for instance, and the SPI support is SPL only atm.
But yeah, we could unify them, no question.

BUT: What do you mean exactly with: "just generate it from the defconfig"?
Some Makefile hack? Like detecting 32 vs 64 with the help of the ARCH
environment variable or ${CROSS_COMPILE}gcc -dumpmachine output?

> And we don't have to matter about keeping it in sync, or the board
> specific init that might need to be done.

Agreed.

Cheers,
Andre.

On 05/12/16 06:26, Simon Glass wrote:
> Hi Andre,
> 
> On 4 December 2016 at 18:52, Andre Przywara <andre.przywara@arm.com> wrote:
>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
>> use the more compact Thumb2 encoding, which only exists for AArch32
>> code. This makes the SPL rather big, up to a point where any code
>> additions or even a different compiler may easily exceed the 32KB limit
>> that the Allwinner BROM imposes.
>> Introduce a separate, mostly generic sun50i-a64 configuration, which
>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
>> the memory-saving Thumb2 encoding.
>> This should only be used for the SPL, the U-Boot proper should still be
>> using the existing 64-bit configuration. The SPL code can switch to
>> AArch64 if needed, so a 32-bit SPL can be combined with a 64-bit U-Boot
>> proper to eventually launch arm64 kernels.
> 
> So if I understand correctly, you want SPL to be 32-bit and U-Boot
> proper to be 64-bit?

Yes, that is _one_ possible option, mostly driven by size constraints
and due to AArch64 code being much bigger than Thumb2.

> And you are adding a new board config for that?

Yes, for now this one (separate) defconfig aims to cover all A64 boards.
This is not ideal (as Maxime pointed out already), so if you know a nice
way of using a single defconfig for one board and configuring it once
with CPU_V7 and then again with ARM64 set, I am all ears.

Also I haven't found a make target to just build the SPL (possibly
another one for just the U-Boot proper). Maybe this would help things?

> Instead, can you do something similar to tegra, which uses ARMv4t for
> SPL and ARMv7 for U-Boot proper?

You will need two different (cross-)compilers, so just setting some
compiler options will not help.
Besides compiling the SPL as 32-bit is only an option, the SPL also
works as a pure 64-bit binary. And people expressed the wish of having
the option of using both ways - at least for the time being.

Cheers,
Andre.

>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  board/sunxi/Kconfig            | 14 ++++++++++++--
>>  configs/pine64_plus_defconfig  |  2 +-
>>  configs/sun50i_spl32_defconfig | 10 ++++++++++
>>  include/configs/sunxi-common.h |  2 +-
>>  4 files changed, 24 insertions(+), 4 deletions(-)
>>  create mode 100644 configs/sun50i_spl32_defconfig
> 
> Regards,
> Simon
>

On Fri, Dec 16, 2016 at 03:39:06PM +0000, Andre Przywara wrote:
> On 16/12/16 14:52, Maxime Ripard wrote:
> > On Mon, Dec 12, 2016 at 04:04:23PM +0000, Andre Przywara wrote:
> >> Hi,
> >>
> >> On 12/12/16 15:13, Maxime Ripard wrote:
> >>> On Tue, Dec 06, 2016 at 12:22:59PM +0000, Andre Przywara wrote:
> >>>> Hi,
> >>>>
> >>>> On 06/12/16 11:28, Maxime Ripard wrote:
> >>>>> On Mon, Dec 05, 2016 at 01:52:30AM +0000, Andre Przywara wrote:
> >>>>>> When compiling the SPL for the Allwinner A64 in AArch64 mode, we can't
> >>>>>> use the more compact Thumb2 encoding, which only exists for AArch32
> >>>>>> code. This makes the SPL rather big, up to a point where any code
> >>>>>> additions or even a different compiler may easily exceed the 32KB limit
> >>>>>> that the Allwinner BROM imposes.
> >>>>>> Introduce a separate, mostly generic sun50i-a64 configuration, which
> >>>>>> defines the CPU_V7 symbol and thus will create a 32-bit binary using
> >>>>>> the memory-saving Thumb2 encoding.
> >>>>>
> >>>>> "mostly generic". Where do you draw the line? How do you deal with a
> >>>>> board that would use a different UART? a different MMC? different
> >>>>> memory configuration.?
> >>>>
> >>>> My impression was that it's rather pointless to provide another set of
> >>>> 32-bit SPL defconfigs for each board again, especially given that for
> >>>> the SPL's needs the boards so far seem to be very similar.
> >>>> For the loading part we will probably go with what the BROM already
> >>>> started: load more data from one of the BROM boot sources, which is
> >>>> fixed in the SoC and can't be really changed by a board vendor anyway.
> >>>> Which really leaves the DRAM setup and the UART.
> >>>
> >>> So you plan on enabling all BROM boot sources as well (NAND, SPI) ?
> >>
> >> In fact SPI works already (with little to no changes).
> > 
> > It's the little changes that I'm interested in to be honest :)
> 
> Well, thinking about it again I think normal SPI boot (legacy U-Boot
> image as the payload) requires no changes: just enable it in the
> defconfig. The patches I was talking about were about enabling FIT
> support on top of it.
> 
> The reason why it works is due to Siarhei's SPL SPI code and due to the
> fact it is kind of "supported" on these boards, which is not true for
> NAND on most boards, AFAIK.
> But if we get support for that, it would just work the same way, due to
> the boot source detection. So I think handling multiple boot sources
> within one SPL binary is a general sunxi SPL feature already implemented
> today. I definitely use the same thing for FEL, SPI and SD and eMMC.

At what offset for the SPL? U-Boot binary? with or without redundacy
for U-Boot? for the environment? with an environment in the first
place?

All those things are user configurable, and will also depend on some
board features (staring with the size of the EEPROM embedded on that
board). We should treat them as user configurable, and not just as
"meh, don't care, works for my setup".

> >> And I don't care about NAND, really ;-)
> >> Is anyone aware of an A64 board using this?
> > 
> > Well, it's one of the possible boot source, so we have to consider it
> > and not ignore it entirely hoping that no one will use it, ever.
> 
> But there isn't any support for it so far, is there? Even if we wanted
> to compile an SPL just for NAND.

The thing is once you started telling people to use something,
removing that something somewhere down the road is *not* nice. So I'd
really like to have it taken into account.

> > SPI booting was not used for 5-6 years, until someone started that
> > trend and now we have a significant number of boards implementing it.
> > 
> >>>> I can't predict the future, but so far those A64 boards look fairly
> >>>> similar in this respect. So I just avoid having another SPL defconfig
> >>>> for the BananaPi M64, for instance. I just added MMC_SUNXI_SLOT_EXTRA
> >>>> because this doesn't hurt on the Pine64, so less churn here.
> >>>>
> >>>> So if you know of any board which breaks this assumption, I am happy to
> >>>> hear about it and see if it can be integrated.
> >>>
> >>> I know at least of one board that uses the UART3 on A33, instead of
> >>> UART0. The trend is very clear on the A64 and the previous SoCs, but
> >>> we also had some variations, so we need to take that into
> >>> account. Which brings me back to my original question, where do you
> >>> draw the line ? :)
> >>
> >> I don't know, and to make this clear: I see the point in having separate
> >> configs for the SPL, but due to the 32-bit/64-bit split we probably need
> >> _two_ sets of defconfigs, which gets pretty messy very quickly.
> >> Especially given that they are very similar.
> >>
> >> So how do we avoid this? Can we somehow share a defconfig between armv8
> >> and armv7? In the moment "CONFIG_CPU_V7" and "CONFIG_ARM64" conflict in
> >> the same file.
> > 
> > I guess the easiest and most robust solution to do this would be to
> > just generate it from the defconfig. It really feels from your patches
> > that it's just a matter of sed -i 's/CONFIG_MACH_SUN50I/CONFIG_MACH_SUN50I_32'
> > on the "real" defconfig.
> 
> Yes, the difference between the defconfigs is really minimal. They look
> more different at the moment because we don't need Ethernet and USB in
> the SPL, for instance, and the SPI support is SPL only atm.
> But yeah, we could unify them, no question.
> 
> BUT: What do you mean exactly with: "just generate it from the defconfig"?
> Some Makefile hack? Like detecting 32 vs 64 with the help of the ARCH
> environment variable or ${CROSS_COMPILE}gcc -dumpmachine output?

Plugging into a makefile target seems for reasonable.

In particular, I guess that would imply:
  - Generating an spl/include/generated/autoconf.h
  - Since that file is included through include/linux/kconfig.h,
    create / copy that one over
  - Change UBOOTINCLUDE to have a different one for the SPL that would
    automatically pick the right kconfig.h

I guess some of that can also be eased through the include order in
the C flags.

Once that is done, you can easily mangle your configuration in the
Makefile rule only for the SPL.

Maxime

On Sat, Dec 17, 2016 at 02:44:46PM +0000, André Przywara wrote:
> > Instead, can you do something similar to tegra, which uses ARMv4t for
> > SPL and ARMv7 for U-Boot proper?
> 
> You will need two different (cross-)compilers, so just setting some
> compiler options will not help.
> Besides compiling the SPL as 32-bit is only an option, the SPL also
> works as a pure 64-bit binary. And people expressed the wish of having
> the option of using both ways - at least for the time being.

Honestly, I have the feeling that this is the only thing that holds
this serie back. Maybe you can send it without it first, we merge it,
and then we work on the 32 bits SPL. That way, we don't delay
everything else but the very last patch.

Thanks!
Maxime

Salut Maxime,

On 19/12/16 08:20, Maxime Ripard wrote:
> On Sat, Dec 17, 2016 at 02:44:46PM +0000, André Przywara wrote:
>>> Instead, can you do something similar to tegra, which uses ARMv4t for
>>> SPL and ARMv7 for U-Boot proper?
>>
>> You will need two different (cross-)compilers, so just setting some
>> compiler options will not help.
>> Besides compiling the SPL as 32-bit is only an option, the SPL also
>> works as a pure 64-bit binary. And people expressed the wish of having
>> the option of using both ways - at least for the time being.
> 
> Honestly, I have the feeling that this is the only thing that holds
> this serie back. Maybe you can send it without it first, we merge it,
> and then we work on the 32 bits SPL. That way, we don't delay
> everything else but the very last patch.

Feel free to do so already.
I deliberately made the series this way, so you can merge anything up to
and including "[PATCH v3 21/26] sunxi: A64: enable SPL" and get the
64-bit SPL functionality. If you like, you can also take the others but
the last patch as well.

Actually I would be very glad to get that two-digit patch count off my
back, as this makes handling much easier.
Also I have the H5 support ready on top of that ...

Thanks!

Cheers,
Andre.