| Message ID | 20230125222741.303259-1-jagan@edgeble.ai |
|---|---|
| Headers | show |
| Series | arm: Add Rockchip RK3588 support | expand |
Hi Jagan, On 2023-01-25 23:27, Jagan Teki wrote: > This series support Rockchip RK3588. All the device tree files are > synced from linux-next with the proper SHA1 mentioned in the commit > messages. > > Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > it is failing to load ATF entry from SPL and hang. > > Verified below BL31 versions, > bl31-v1.15 > bl31-v1.21 > bl31-v1.22 > bl31-v1.23 > bl31-v1.24 > bl31-v1.25 > bl31-v1.26 > > Rever-engineered with respect to rockchip u-boot by using the same > FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > mainline showing the same issue. > > Log: > > LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > change to F1: 528MHz > change to F2: 1068MHz > change to F3: 1560MHz > change to F0: 2112MHz > out > > U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > Trying to boot from MMC1 > bl31_entry: atf_entry start > << hang >> > > Any information on BL31 for RK3588 please share. I had a similar strange booing issue with RK3568 and mainline U-Boot, turned out to be related to all parts of ATF not being properly loaded into PMU SRAM. Using my series at [1] I managed to get ATF to be fully loaded into PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that the segment being loaded ended up corrupted. The use of 512 bytes alignment of the FIT helped mitigate that issue. Vendor U-Boot use a bounce buffer for all parts that is written into SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). You can also find newer bl31 at [2], up to version v1.32. [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35 Regards, Jonas > > Any inputs? > Jagan. > > Jagan Teki (16): > rockchip: mkimage: Add rk3588 support > arm: rockchip: Add cru header for rk3588 > arm: rockchip: Add grf header for rk3588 > dt-bindings: clk: Add dt-binding header for RK3588 > clk: rockchip: Add rk3588 clk support > clk: rockchip: pll: Add pll_rk3588 type for rk3588 > ram: rockchip: Add rk3588 ddr driver support > dt-bindings: power: Add power-domain header for rk3588 > dt-bindings: reset: add rk3588 reset definitions > arm: rockchip: Add ioc header for rk3588 > arm64: dts: rockchip: Add base DT for rk3588 SoC > arm64: dts: rockchip: rk3588: Add Edgeble Neu6 Model A SoM > arm64: dts: rockchip: rk3588: Add Edgeble Neu6 Model A IO > arm: rockchip: Add RK3588 arch core support > ARM: dts: rockchip: Add rk3588-u-boot.dtsi > board: rockchip: Add Edgeble Neural Compute Module 6 > > arch/arm/dts/Makefile | 3 + > .../dts/rk3588-edgeble-neu6a-io-u-boot.dtsi | 23 + > arch/arm/dts/rk3588-edgeble-neu6a-io.dts | 27 + > arch/arm/dts/rk3588-edgeble-neu6a.dtsi | 32 + > arch/arm/dts/rk3588-pinctrl.dtsi | 516 +++ > arch/arm/dts/rk3588-u-boot.dtsi | 101 + > arch/arm/dts/rk3588.dtsi | 58 + > arch/arm/dts/rk3588s-pinctrl.dtsi | 3403 +++++++++++++++++ > arch/arm/dts/rk3588s.dtsi | 1703 +++++++++ > arch/arm/include/asm/arch-rk3588/boot0.h | 11 + > arch/arm/include/asm/arch-rk3588/gpio.h | 11 + > arch/arm/include/asm/arch-rockchip/clock.h | 24 + > .../include/asm/arch-rockchip/cru_rk3588.h | 451 +++ > .../include/asm/arch-rockchip/grf_rk3588.h | 35 + > .../include/asm/arch-rockchip/ioc_rk3588.h | 102 + > arch/arm/mach-rockchip/Kconfig | 20 + > arch/arm/mach-rockchip/Makefile | 1 + > arch/arm/mach-rockchip/rk3588/Kconfig | 30 + > arch/arm/mach-rockchip/rk3588/Makefile | 9 + > arch/arm/mach-rockchip/rk3588/clk_rk3588.c | 33 + > arch/arm/mach-rockchip/rk3588/rk3588.c | 162 + > arch/arm/mach-rockchip/rk3588/syscon_rk3588.c | 32 + > board/edgeble/neural-compute-module-6/Kconfig | 15 + > .../neural-compute-module-6/MAINTAINERS | 6 + > .../edgeble/neural-compute-module-6/Makefile | 7 + > board/edgeble/neural-compute-module-6/neu6.c | 4 + > configs/neu6a-io-rk3588_defconfig | 68 + > doc/board/rockchip/rockchip.rst | 2 + > drivers/clk/rockchip/Makefile | 1 + > drivers/clk/rockchip/clk_pll.c | 267 +- > drivers/clk/rockchip/clk_rk3588.c | 2019 ++++++++++ > drivers/ram/rockchip/Makefile | 1 + > drivers/ram/rockchip/sdram_rk3588.c | 56 + > include/configs/neural-compute-module-6.h | 15 + > include/configs/rk3588_common.h | 32 + > .../dt-bindings/clock/rockchip,rk3588-cru.h | 766 ++++ > include/dt-bindings/power/rk3588-power.h | 69 + > .../dt-bindings/reset/rockchip,rk3588-cru.h | 754 ++++ > tools/rkcommon.c | 1 + > 39 files changed, 10867 insertions(+), 3 deletions(-) > create mode 100644 arch/arm/dts/rk3588-edgeble-neu6a-io-u-boot.dtsi > create mode 100644 arch/arm/dts/rk3588-edgeble-neu6a-io.dts > create mode 100644 arch/arm/dts/rk3588-edgeble-neu6a.dtsi > create mode 100644 arch/arm/dts/rk3588-pinctrl.dtsi > create mode 100644 arch/arm/dts/rk3588-u-boot.dtsi > create mode 100644 arch/arm/dts/rk3588.dtsi > create mode 100644 arch/arm/dts/rk3588s-pinctrl.dtsi > create mode 100644 arch/arm/dts/rk3588s.dtsi > create mode 100644 arch/arm/include/asm/arch-rk3588/boot0.h > create mode 100644 arch/arm/include/asm/arch-rk3588/gpio.h > create mode 100644 arch/arm/include/asm/arch-rockchip/cru_rk3588.h > create mode 100644 arch/arm/include/asm/arch-rockchip/grf_rk3588.h > create mode 100644 arch/arm/include/asm/arch-rockchip/ioc_rk3588.h > create mode 100644 arch/arm/mach-rockchip/rk3588/Kconfig > create mode 100644 arch/arm/mach-rockchip/rk3588/Makefile > create mode 100644 arch/arm/mach-rockchip/rk3588/clk_rk3588.c > create mode 100644 arch/arm/mach-rockchip/rk3588/rk3588.c > create mode 100644 arch/arm/mach-rockchip/rk3588/syscon_rk3588.c > create mode 100644 board/edgeble/neural-compute-module-6/Kconfig > create mode 100644 board/edgeble/neural-compute-module-6/MAINTAINERS > create mode 100644 board/edgeble/neural-compute-module-6/Makefile > create mode 100644 board/edgeble/neural-compute-module-6/neu6.c > create mode 100644 configs/neu6a-io-rk3588_defconfig > create mode 100644 drivers/clk/rockchip/clk_rk3588.c > create mode 100644 drivers/ram/rockchip/sdram_rk3588.c > create mode 100644 include/configs/neural-compute-module-6.h > create mode 100644 include/configs/rk3588_common.h > create mode 100644 include/dt-bindings/clock/rockchip,rk3588-cru.h > create mode 100644 include/dt-bindings/power/rk3588-power.h > create mode 100644 include/dt-bindings/reset/rockchip,rk3588-cru.h >
Hi Jonas, On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > Hi Jagan, > > On 2023-01-25 23:27, Jagan Teki wrote: > > This series support Rockchip RK3588. All the device tree files are > > synced from linux-next with the proper SHA1 mentioned in the commit > > messages. > > > > Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > it is failing to load ATF entry from SPL and hang. > > > > Verified below BL31 versions, > > bl31-v1.15 > > bl31-v1.21 > > bl31-v1.22 > > bl31-v1.23 > > bl31-v1.24 > > bl31-v1.25 > > bl31-v1.26 > > > > Rever-engineered with respect to rockchip u-boot by using the same > > FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > mainline showing the same issue. > > > > Log: > > > > LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > change to F1: 528MHz > > change to F2: 1068MHz > > change to F3: 1560MHz > > change to F0: 2112MHz > > out > > > > U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > Trying to boot from MMC1 > > bl31_entry: atf_entry start > > << hang >> > > > > Any information on BL31 for RK3588 please share. > > I had a similar strange booing issue with RK3568 and mainline U-Boot, > turned out to be related to all parts of ATF not being properly loaded > into PMU SRAM. > > Using my series at [1] I managed to get ATF to be fully loaded into > PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > the segment being loaded ended up corrupted. > > The use of 512 bytes alignment of the FIT helped mitigate that issue. > Vendor U-Boot use a bounce buffer for all parts that is written into > SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > You can also find newer bl31 at [2], up to version v1.32. > > [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 > [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35 Thanks for the details. I did apply this set on the master. No change in the behavior, used BL31 and ddr from [2] as well as in rkbin/master. Jagan.
Hi Jagan, On 2023-01-26 17:51, Jagan Teki wrote: > Hi Jonas, > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >> >> Hi Jagan, >> >> On 2023-01-25 23:27, Jagan Teki wrote: >>> This series support Rockchip RK3588. All the device tree files are >>> synced from linux-next with the proper SHA1 mentioned in the commit >>> messages. >>> >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>> it is failing to load ATF entry from SPL and hang. >>> >>> Verified below BL31 versions, >>> bl31-v1.15 >>> bl31-v1.21 >>> bl31-v1.22 >>> bl31-v1.23 >>> bl31-v1.24 >>> bl31-v1.25 >>> bl31-v1.26 >>> >>> Rever-engineered with respect to rockchip u-boot by using the same >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>> mainline showing the same issue. >>> >>> Log: >>> >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> change to F1: 528MHz >>> change to F2: 1068MHz >>> change to F3: 1560MHz >>> change to F0: 2112MHz >>> out >>> >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>> Trying to boot from MMC1 >>> bl31_entry: atf_entry start >>> << hang >> >>> >>> Any information on BL31 for RK3588 please share. >> >> I had a similar strange booing issue with RK3568 and mainline U-Boot, >> turned out to be related to all parts of ATF not being properly loaded >> into PMU SRAM. >> >> Using my series at [1] I managed to get ATF to be fully loaded into >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that >> the segment being loaded ended up corrupted. >> >> The use of 512 bytes alignment of the FIT helped mitigate that issue. >> Vendor U-Boot use a bounce buffer for all parts that is written into >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). >> >> You can also find newer bl31 at [2], up to version v1.32. >> >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > Thanks for the details. I did apply this set on the master. No change > in the behavior, used BL31 and ddr from [2] as well as in > rkbin/master. I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y and it looked like it failed to read data into memory, see below. It also looks like the sdhci compatible is not supported by the driver. Something that may need to be added to driver to properly read data? DDR V1.09 a930779e06 typ 22/11/21-17:50:56 LPDDR4X, 2112MHz channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB Manufacturer ID:0x6 CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% change to F1: 528MHz change to F2: 1068MHz change to F3: 1560MHz change to F0: 2112MHz out U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) Trying to boot from MMC1 ## Checking hash(es) for config config_1 ... OK ## Checking hash(es) for Image atf_1 ... sha256 error! Bad hash value for 'hash' hash node in 'atf_1' image node mmc_load_image_raw_sector: mmc block read error Trying to boot from MMC1 ## Checking hash(es) for config config_1 ... OK ## Checking hash(es) for Image atf_1 ... sha256 error! Bad hash value for 'hash' hash node in 'atf_1' image node mmc_load_image_raw_sector: mmc block read error SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### Regards, Jonas > > Jagan.
On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > Hi Jagan, > On 2023-01-26 17:51, Jagan Teki wrote: > > Hi Jonas, > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > >> > >> Hi Jagan, > >> > >> On 2023-01-25 23:27, Jagan Teki wrote: > >>> This series support Rockchip RK3588. All the device tree files are > >>> synced from linux-next with the proper SHA1 mentioned in the commit > >>> messages. > >>> > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>> it is failing to load ATF entry from SPL and hang. > >>> > >>> Verified below BL31 versions, > >>> bl31-v1.15 > >>> bl31-v1.21 > >>> bl31-v1.22 > >>> bl31-v1.23 > >>> bl31-v1.24 > >>> bl31-v1.25 > >>> bl31-v1.26 > >>> > >>> Rever-engineered with respect to rockchip u-boot by using the same > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > >>> mainline showing the same issue. > >>> > >>> Log: > >>> > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>> change to F1: 528MHz > >>> change to F2: 1068MHz > >>> change to F3: 1560MHz > >>> change to F0: 2112MHz > >>> out > >>> > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > >>> Trying to boot from MMC1 > >>> bl31_entry: atf_entry start > >>> << hang >> > >>> > >>> Any information on BL31 for RK3588 please share. > >> > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > >> turned out to be related to all parts of ATF not being properly loaded > >> into PMU SRAM. > >> > >> Using my series at [1] I managed to get ATF to be fully loaded into > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > >> the segment being loaded ended up corrupted. > >> > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > >> Vendor U-Boot use a bounce buffer for all parts that is written into > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > >> > >> You can also find newer bl31 at [2], up to version v1.32. > >> > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > Thanks for the details. I did apply this set on the master. No change > > in the behavior, used BL31 and ddr from [2] as well as in > > rkbin/master. > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > and it looked like it failed to read data into memory, see below. > > It also looks like the sdhci compatible is not supported by the driver. > Something that may need to be added to driver to properly read data? > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > LPDDR4X, 2112MHz > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > Manufacturer ID:0x6 > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > change to F1: 528MHz > change to F2: 1068MHz > change to F3: 1560MHz > change to F0: 2112MHz > out > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > Trying to boot from MMC1 > ## Checking hash(es) for config config_1 ... OK > ## Checking hash(es) for Image atf_1 ... sha256 error! > Bad hash value for 'hash' hash node in 'atf_1' image node > mmc_load_image_raw_sector: mmc block read error > Trying to boot from MMC1 > ## Checking hash(es) for config config_1 ... OK > ## Checking hash(es) for Image atf_1 ... sha256 error! > Bad hash value for 'hash' hash node in 'atf_1' image node Look like this is something wrong with your patch series or master changes on binman, not with the driver. I have observed the same if I enable CONFIG_SPL_FIT_SIGNATURE. Jagan.
On Thu, 26 Jan 2023 at 23:12, Jagan Teki <jagan@edgeble.ai> wrote: > > On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > Hi Jagan, > > On 2023-01-26 17:51, Jagan Teki wrote: > > > Hi Jonas, > > > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > >> > > >> Hi Jagan, > > >> > > >> On 2023-01-25 23:27, Jagan Teki wrote: > > >>> This series support Rockchip RK3588. All the device tree files are > > >>> synced from linux-next with the proper SHA1 mentioned in the commit > > >>> messages. > > >>> > > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > >>> it is failing to load ATF entry from SPL and hang. > > >>> > > >>> Verified below BL31 versions, > > >>> bl31-v1.15 > > >>> bl31-v1.21 > > >>> bl31-v1.22 > > >>> bl31-v1.23 > > >>> bl31-v1.24 > > >>> bl31-v1.25 > > >>> bl31-v1.26 > > >>> > > >>> Rever-engineered with respect to rockchip u-boot by using the same > > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > >>> mainline showing the same issue. > > >>> > > >>> Log: > > >>> > > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> change to F1: 528MHz > > >>> change to F2: 1068MHz > > >>> change to F3: 1560MHz > > >>> change to F0: 2112MHz > > >>> out > > >>> > > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > >>> Trying to boot from MMC1 > > >>> bl31_entry: atf_entry start > > >>> << hang >> > > >>> > > >>> Any information on BL31 for RK3588 please share. > > >> > > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > > >> turned out to be related to all parts of ATF not being properly loaded > > >> into PMU SRAM. > > >> > > >> Using my series at [1] I managed to get ATF to be fully loaded into > > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > > >> the segment being loaded ended up corrupted. > > >> > > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > > >> Vendor U-Boot use a bounce buffer for all parts that is written into > > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > >> > > >> You can also find newer bl31 at [2], up to version v1.32. > > >> > > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > > Thanks for the details. I did apply this set on the master. No change > > > in the behavior, used BL31 and ddr from [2] as well as in > > > rkbin/master. > > > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > > and it looked like it failed to read data into memory, see below. > > > > It also looks like the sdhci compatible is not supported by the driver. > > Something that may need to be added to driver to properly read data? > > > > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > > LPDDR4X, 2112MHz > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > Manufacturer ID:0x6 > > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > > change to F1: 528MHz > > change to F2: 1068MHz > > change to F3: 1560MHz > > change to F0: 2112MHz > > out > > > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > > Trying to boot from MMC1 > > ## Checking hash(es) for config config_1 ... OK > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf_1' image node > > mmc_load_image_raw_sector: mmc block read error > > Trying to boot from MMC1 > > ## Checking hash(es) for config config_1 ... OK > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf_1' image node > > Look like this is something wrong with your patch series or master > changes on binman, not with the driver. I have observed the same if I > enable CONFIG_SPL_FIT_SIGNATURE. Do you have a repo where it worked before with CONFIG_SPL_FIT_SIGNATURE? Jagan.
Hi Jagan, On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > > On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > Hi Jagan, > > On 2023-01-26 17:51, Jagan Teki wrote: > > > Hi Jonas, > > > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > >> > > >> Hi Jagan, > > >> > > >> On 2023-01-25 23:27, Jagan Teki wrote: > > >>> This series support Rockchip RK3588. All the device tree files are > > >>> synced from linux-next with the proper SHA1 mentioned in the commit > > >>> messages. > > >>> > > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > >>> it is failing to load ATF entry from SPL and hang. > > >>> > > >>> Verified below BL31 versions, > > >>> bl31-v1.15 > > >>> bl31-v1.21 > > >>> bl31-v1.22 > > >>> bl31-v1.23 > > >>> bl31-v1.24 > > >>> bl31-v1.25 > > >>> bl31-v1.26 > > >>> > > >>> Rever-engineered with respect to rockchip u-boot by using the same > > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > >>> mainline showing the same issue. > > >>> > > >>> Log: > > >>> > > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > >>> change to F1: 528MHz > > >>> change to F2: 1068MHz > > >>> change to F3: 1560MHz > > >>> change to F0: 2112MHz > > >>> out > > >>> > > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > >>> Trying to boot from MMC1 > > >>> bl31_entry: atf_entry start > > >>> << hang >> > > >>> > > >>> Any information on BL31 for RK3588 please share. > > >> > > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > > >> turned out to be related to all parts of ATF not being properly loaded > > >> into PMU SRAM. > > >> > > >> Using my series at [1] I managed to get ATF to be fully loaded into > > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > > >> the segment being loaded ended up corrupted. > > >> > > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > > >> Vendor U-Boot use a bounce buffer for all parts that is written into > > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > >> > > >> You can also find newer bl31 at [2], up to version v1.32. > > >> > > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > > Thanks for the details. I did apply this set on the master. No change > > > in the behavior, used BL31 and ddr from [2] as well as in > > > rkbin/master. > > > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > > and it looked like it failed to read data into memory, see below. > > > > It also looks like the sdhci compatible is not supported by the driver. > > Something that may need to be added to driver to properly read data? > > > > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > > LPDDR4X, 2112MHz > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > Manufacturer ID:0x6 > > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > > change to F1: 528MHz > > change to F2: 1068MHz > > change to F3: 1560MHz > > change to F0: 2112MHz > > out > > > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > > Trying to boot from MMC1 > > ## Checking hash(es) for config config_1 ... OK > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf_1' image node > > mmc_load_image_raw_sector: mmc block read error > > Trying to boot from MMC1 > > ## Checking hash(es) for config config_1 ... OK > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf_1' image node > > Look like this is something wrong with your patch series or master > changes on binman, not with the driver. I have observed the same if I > enable CONFIG_SPL_FIT_SIGNATURE. There are some more changes in dm/master that I'm about to pull in. One of them from Jonas Karlman adds hash nodes so may be involved. Regards, Simon
Hi Simon, On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > > Hi Jagan, > > On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > > > > On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > > > Hi Jagan, > > > On 2023-01-26 17:51, Jagan Teki wrote: > > > > Hi Jonas, > > > > > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > > >> > > > >> Hi Jagan, > > > >> > > > >> On 2023-01-25 23:27, Jagan Teki wrote: > > > >>> This series support Rockchip RK3588. All the device tree files are > > > >>> synced from linux-next with the proper SHA1 mentioned in the commit > > > >>> messages. > > > >>> > > > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > > >>> it is failing to load ATF entry from SPL and hang. > > > >>> > > > >>> Verified below BL31 versions, > > > >>> bl31-v1.15 > > > >>> bl31-v1.21 > > > >>> bl31-v1.22 > > > >>> bl31-v1.23 > > > >>> bl31-v1.24 > > > >>> bl31-v1.25 > > > >>> bl31-v1.26 > > > >>> > > > >>> Rever-engineered with respect to rockchip u-boot by using the same > > > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > > >>> mainline showing the same issue. > > > >>> > > > >>> Log: > > > >>> > > > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > >>> change to F1: 528MHz > > > >>> change to F2: 1068MHz > > > >>> change to F3: 1560MHz > > > >>> change to F0: 2112MHz > > > >>> out > > > >>> > > > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > > >>> Trying to boot from MMC1 > > > >>> bl31_entry: atf_entry start > > > >>> << hang >> > > > >>> > > > >>> Any information on BL31 for RK3588 please share. > > > >> > > > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > > > >> turned out to be related to all parts of ATF not being properly loaded > > > >> into PMU SRAM. > > > >> > > > >> Using my series at [1] I managed to get ATF to be fully loaded into > > > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > > > >> the segment being loaded ended up corrupted. > > > >> > > > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > > > >> Vendor U-Boot use a bounce buffer for all parts that is written into > > > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > > >> > > > >> You can also find newer bl31 at [2], up to version v1.32. > > > >> > > > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > > > Thanks for the details. I did apply this set on the master. No change > > > > in the behavior, used BL31 and ddr from [2] as well as in > > > > rkbin/master. > > > > > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > > > and it looked like it failed to read data into memory, see below. > > > > > > It also looks like the sdhci compatible is not supported by the driver. > > > Something that may need to be added to driver to properly read data? > > > > > > > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > > > LPDDR4X, 2112MHz > > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > Manufacturer ID:0x6 > > > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > > > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > > > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > > > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > > > change to F1: 528MHz > > > change to F2: 1068MHz > > > change to F3: 1560MHz > > > change to F0: 2112MHz > > > out > > > > > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > > > Trying to boot from MMC1 > > > ## Checking hash(es) for config config_1 ... OK > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > mmc_load_image_raw_sector: mmc block read error > > > Trying to boot from MMC1 > > > ## Checking hash(es) for config config_1 ... OK > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > > Look like this is something wrong with your patch series or master > > changes on binman, not with the driver. I have observed the same if I > > enable CONFIG_SPL_FIT_SIGNATURE. > > There are some more changes in dm/master that I'm about to pull in. > One of them from Jonas Karlman adds hash nodes so may be involved. I found the same issue on the dm/master U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) Trying to boot from MMC1 ## Checking hash(es) for config config-1 ... OK ## Checking hash(es) for Image atf-1 ... sha256 error! Bad hash value for 'hash' hash node in 'atf-1' image node mmc_load_image_raw_sector: mmc block read error SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### Jagan.
On 2023-01-26 19:26, Jagan Teki wrote: > Hi Simon, > > On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >> >> Hi Jagan, >> >> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>> >>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>> >>>> Hi Jagan, >>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>> Hi Jonas, >>>>> >>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>> >>>>>> Hi Jagan, >>>>>> >>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>> messages. >>>>>>> >>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>> >>>>>>> Verified below BL31 versions, >>>>>>> bl31-v1.15 >>>>>>> bl31-v1.21 >>>>>>> bl31-v1.22 >>>>>>> bl31-v1.23 >>>>>>> bl31-v1.24 >>>>>>> bl31-v1.25 >>>>>>> bl31-v1.26 >>>>>>> >>>>>>> Rever-engineered with respect to rockchip u-boot by using the same >>>>>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>>>>>> mainline showing the same issue. >>>>>>> >>>>>>> Log: >>>>>>> >>>>>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>> change to F1: 528MHz >>>>>>> change to F2: 1068MHz >>>>>>> change to F3: 1560MHz >>>>>>> change to F0: 2112MHz >>>>>>> out >>>>>>> >>>>>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>>>>>> Trying to boot from MMC1 >>>>>>> bl31_entry: atf_entry start >>>>>>> << hang >> >>>>>>> >>>>>>> Any information on BL31 for RK3588 please share. >>>>>> >>>>>> I had a similar strange booing issue with RK3568 and mainline U-Boot, >>>>>> turned out to be related to all parts of ATF not being properly loaded >>>>>> into PMU SRAM. >>>>>> >>>>>> Using my series at [1] I managed to get ATF to be fully loaded into >>>>>> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that >>>>>> the segment being loaded ended up corrupted. >>>>>> >>>>>> The use of 512 bytes alignment of the FIT helped mitigate that issue. >>>>>> Vendor U-Boot use a bounce buffer for all parts that is written into >>>>>> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). >>>>>> >>>>>> You can also find newer bl31 at [2], up to version v1.32. >>>>>> >>>>>> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>>>>>> Thanks for the details. I did apply this set on the master. No change >>>>> in the behavior, used BL31 and ddr from [2] as well as in >>>>> rkbin/master. >>>> >>>> I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y >>>> and it looked like it failed to read data into memory, see below. >>>> >>>> It also looks like the sdhci compatible is not supported by the driver. >>>> Something that may need to be added to driver to properly read data? >>>> >>>> >>>> DDR V1.09 a930779e06 typ 22/11/21-17:50:56 >>>> LPDDR4X, 2112MHz >>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>> Manufacturer ID:0x6 >>>> CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% >>>> CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% >>>> CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% >>>> CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% >>>> change to F1: 528MHz >>>> change to F2: 1068MHz >>>> change to F3: 1560MHz >>>> change to F0: 2112MHz >>>> out >>>> >>>> U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) >>>> Trying to boot from MMC1 >>>> ## Checking hash(es) for config config_1 ... OK >>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>>> mmc_load_image_raw_sector: mmc block read error >>>> Trying to boot from MMC1 >>>> ## Checking hash(es) for config config_1 ... OK >>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>> >>> Look like this is something wrong with your patch series or master >>> changes on binman, not with the driver. I have observed the same if I >>> enable CONFIG_SPL_FIT_SIGNATURE. >> >> There are some more changes in dm/master that I'm about to pull in. >> One of them from Jonas Karlman adds hash nodes so may be involved. > > I found the same issue on the dm/master > > U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) > Trying to boot from MMC1 > ## Checking hash(es) for config config-1 ... OK > ## Checking hash(es) for Image atf-1 ... sha256 error! > Bad hash value for 'hash' hash node in 'atf-1' image node > mmc_load_image_raw_sector: mmc block read error > SPL: failed to boot from all boot devices > ### ERROR ### Please RESET the board ### On my RK3568 ROCK 3A board this is working correctly, see below. This was using u-boot master 17e8e58fe62c019b2cc26af221b6defc3368229f with a few patches on top, see [1]. ------------- U-Boot SPL 2023.01 (Jan 26 2023 - 18:29:56 +0000) Trying to boot from MMC1 ## Checking hash(es) for config config-1 ... OK ## Checking hash(es) for Image atf-1 ... sha256+ OK ## Checking hash(es) for Image u-boot ... sha256+ OK ## Checking hash(es) for Image fdt-1 ... sha256+ OK ## Checking hash(es) for Image atf-2 ... sha256+ OK ## Checking hash(es) for Image atf-3 ... sha256+ OK ## Checking hash(es) for Image atf-4 ... sha256+ OK ## Checking hash(es) for Image atf-5 ... sha256+ OK ## Checking hash(es) for Image atf-6 ... sha256+ OK INFO: Preloader serial: 2 NOTICE: BL31: v2.3():v2.3-460-g2c8be93f9:huan.he NOTICE: BL31: Built : 19:45:17, Nov 8 2022 INFO: GICv3 without legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: pmu v1 is valid 220114 INFO: dfs DDR fsp_param[0].freq_mhz= 1560MHz INFO: dfs DDR fsp_param[1].freq_mhz= 324MHz INFO: dfs DDR fsp_param[2].freq_mhz= 528MHz INFO: dfs DDR fsp_param[3].freq_mhz= 780MHz INFO: Using opteed sec cpu_context! INFO: boot cpu mask: 0 INFO: BL31: Initializing runtime services WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK ERROR: Error initializing runtime service opteed_fast INFO: BL31: Preparing for EL3 exit to normal world INFO: Entry point address = 0xa00000 INFO: SPSR = 0x3c9 U-Boot 2023.01 (Jan 26 2023 - 18:29:56 +0000) Model: Rockchip RK3568 EVB1 DDR4 V10 Board DRAM: 8 GiB (effective 7.7 GiB) Core: 58 devices, 17 uclasses, devicetree: separate MMC: mmc@fe2b0000: 1, mmc@fe310000: 0 ... ------------- However, on my RK3588 ROCK 5B device I get a similar error you get. Seems to be the mmc reading that times out all of sudden. It can read the FIP and config, and then some mmc command/transfer times out. See below for details with LOG_DEBUG defined at top of include/log.h This was trying to boot from a SD-card, trying to use eMMC fails earlier. Because there is no driver for emmc compatible, SPL only tries to boot from SD-card. I believe this is related to your << hang >>, reading atf from mmc fails in the background and there is no error message. That is why I suggested trying with CONFIG_SPL_FIT_SIGNATURE=y, to see if there is any hidden issue trying to load ATF. ------------- Trying to boot from MMC1 ## Checking hash(es) for config config-1 ... OK mmc_load_image_raw_sector: mmc block read error Trying to boot from MMC1 mmc_load_image_raw_sector: mmc block read error SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### ------------- Trying to boot from MMC1 0 - 0 'mmc@fe2c0000' - found blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 clock is disabled (0Hz) Buswidth = 0, clock: 0 Buswidth = 1, clock: 0 clock is enabled (400000Hz) Buswidth = 1, clock: 400000 Sending CMD0 Sending CMD8 Sending CMD55 Sending CMD41 Sending CMD55 Sending CMD41 Sending CMD2 Sending CMD3 Sending CMD9 Sending CMD7 Sending CMD55 Sending CMD51 Sending CMD6 Sending CMD55 Sending CMD6 Buswidth = 4, clock: 400000 Sending CMD6 clock is enabled (50000000Hz) Buswidth = 4, clock: 50000000 clk_set_rate(clk=500000, rate=50000000) rockchip_dwmmc_get_mmc_clk: err=-2 spl: mmc boot mode: raw blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD17 hdr read sector 4000, count=1 Found FIT size=a00, ptr=ac0, limit=100000: aligned to 5000c0 blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD18 Sending CMD12 fit read sector 4000, sectors=5, dst=5000c0, count=5, size=0xa00 Selecting default config 'rk3588-evb.dtb' ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND OK firmware: 'atf-1' blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD18 dwmci_data_transfer: Timeout waiting for data! mmc_load_image_raw_sector: mmc block read error spl: mmc boot mode: fs Trying to boot from MMC1 0 - 0 'mmc@fe2c0000' - found blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 spl: mmc boot mode: raw blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 dwmci_send_cmd: Response Timeout. hdr read sector 4000, count=0 mmc_load_image_raw_sector: mmc block read error spl: mmc boot mode: fs SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### ------------- [1] https://github.com/Kwiboo/u-boot-rockchip/compare/17e8e58fe62c019b2cc26af221b6defc3368229f...123bee8cb2229ebae35810371827d2b73b71248b Regards, Jonas > > Jagan.
Hi Jagan, On Thu, 26 Jan 2023 at 11:27, Jagan Teki <jagan@edgeble.ai> wrote: > > Hi Simon, > > On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > > > > Hi Jagan, > > > > On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > > > > > > On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > > > > > Hi Jagan, > > > > On 2023-01-26 17:51, Jagan Teki wrote: > > > > > Hi Jonas, > > > > > > > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > >> > > > > >> Hi Jagan, > > > > >> > > > > >> On 2023-01-25 23:27, Jagan Teki wrote: > > > > >>> This series support Rockchip RK3588. All the device tree files are > > > > >>> synced from linux-next with the proper SHA1 mentioned in the commit > > > > >>> messages. > > > > >>> > > > > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > > > >>> it is failing to load ATF entry from SPL and hang. > > > > >>> > > > > >>> Verified below BL31 versions, > > > > >>> bl31-v1.15 > > > > >>> bl31-v1.21 > > > > >>> bl31-v1.22 > > > > >>> bl31-v1.23 > > > > >>> bl31-v1.24 > > > > >>> bl31-v1.25 > > > > >>> bl31-v1.26 > > > > >>> > > > > >>> Rever-engineered with respect to rockchip u-boot by using the same > > > > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > > > >>> mainline showing the same issue. > > > > >>> > > > > >>> Log: > > > > >>> > > > > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > > > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > >>> change to F1: 528MHz > > > > >>> change to F2: 1068MHz > > > > >>> change to F3: 1560MHz > > > > >>> change to F0: 2112MHz > > > > >>> out > > > > >>> > > > > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > > > >>> Trying to boot from MMC1 > > > > >>> bl31_entry: atf_entry start > > > > >>> << hang >> > > > > >>> > > > > >>> Any information on BL31 for RK3588 please share. > > > > >> > > > > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > > > > >> turned out to be related to all parts of ATF not being properly loaded > > > > >> into PMU SRAM. > > > > >> > > > > >> Using my series at [1] I managed to get ATF to be fully loaded into > > > > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > > > > >> the segment being loaded ended up corrupted. > > > > >> > > > > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > > > > >> Vendor U-Boot use a bounce buffer for all parts that is written into > > > > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > > > >> > > > > >> You can also find newer bl31 at [2], up to version v1.32. > > > > >> > > > > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > > > > Thanks for the details. I did apply this set on the master. No change > > > > > in the behavior, used BL31 and ddr from [2] as well as in > > > > > rkbin/master. > > > > > > > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > > > > and it looked like it failed to read data into memory, see below. > > > > > > > > It also looks like the sdhci compatible is not supported by the driver. > > > > Something that may need to be added to driver to properly read data? > > > > > > > > > > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > > > > LPDDR4X, 2112MHz > > > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > Manufacturer ID:0x6 > > > > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > > > > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > > > > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > > > > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > > > > change to F1: 528MHz > > > > change to F2: 1068MHz > > > > change to F3: 1560MHz > > > > change to F0: 2112MHz > > > > out > > > > > > > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > > > > Trying to boot from MMC1 > > > > ## Checking hash(es) for config config_1 ... OK > > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > > mmc_load_image_raw_sector: mmc block read error > > > > Trying to boot from MMC1 > > > > ## Checking hash(es) for config config_1 ... OK > > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > > > > Look like this is something wrong with your patch series or master > > > changes on binman, not with the driver. I have observed the same if I > > > enable CONFIG_SPL_FIT_SIGNATURE. > > > > There are some more changes in dm/master that I'm about to pull in. > > One of them from Jonas Karlman adds hash nodes so may be involved. > > I found the same issue on the dm/master > > U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) > Trying to boot from MMC1 > ## Checking hash(es) for config config-1 ... OK > ## Checking hash(es) for Image atf-1 ... sha256 error! > Bad hash value for 'hash' hash node in 'atf-1' image node > mmc_load_image_raw_sector: mmc block read error > SPL: failed to boot from all boot devices > ### ERROR ### Please RESET the board ### Is the FIT image broken? You can use check_sign or dump_image to see. Regards, Simon
On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: > > On 2023-01-26 19:26, Jagan Teki wrote: > > Hi Simon, > > > > On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > >> > >> Hi Jagan, > >> > >> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > >>> > >>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>> > >>>> Hi Jagan, > >>>> On 2023-01-26 17:51, Jagan Teki wrote: > >>>>> Hi Jonas, > >>>>> > >>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>> > >>>>>> Hi Jagan, > >>>>>> > >>>>>> On 2023-01-25 23:27, Jagan Teki wrote: > >>>>>>> This series support Rockchip RK3588. All the device tree files are > >>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit > >>>>>>> messages. > >>>>>>> > >>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>>>>>> it is failing to load ATF entry from SPL and hang. > >>>>>>> > >>>>>>> Verified below BL31 versions, > >>>>>>> bl31-v1.15 > >>>>>>> bl31-v1.21 > >>>>>>> bl31-v1.22 > >>>>>>> bl31-v1.23 > >>>>>>> bl31-v1.24 > >>>>>>> bl31-v1.25 > >>>>>>> bl31-v1.26 > >>>>>>> > >>>>>>> Rever-engineered with respect to rockchip u-boot by using the same > >>>>>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > >>>>>>> mainline showing the same issue. > >>>>>>> > >>>>>>> Log: > >>>>>>> > >>>>>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > >>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>>>>> change to F1: 528MHz > >>>>>>> change to F2: 1068MHz > >>>>>>> change to F3: 1560MHz > >>>>>>> change to F0: 2112MHz > >>>>>>> out > >>>>>>> > >>>>>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > >>>>>>> Trying to boot from MMC1 > >>>>>>> bl31_entry: atf_entry start > >>>>>>> << hang >> > >>>>>>> > >>>>>>> Any information on BL31 for RK3588 please share. > >>>>>> > >>>>>> I had a similar strange booing issue with RK3568 and mainline U-Boot, > >>>>>> turned out to be related to all parts of ATF not being properly loaded > >>>>>> into PMU SRAM. > >>>>>> > >>>>>> Using my series at [1] I managed to get ATF to be fully loaded into > >>>>>> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > >>>>>> the segment being loaded ended up corrupted. > >>>>>> > >>>>>> The use of 512 bytes alignment of the FIT helped mitigate that issue. > >>>>>> Vendor U-Boot use a bounce buffer for all parts that is written into > >>>>>> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > >>>>>> > >>>>>> You can also find newer bl31 at [2], up to version v1.32. > >>>>>> > >>>>>> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>>>>>> Thanks for the details. I did apply this set on the master. No change > >>>>> in the behavior, used BL31 and ddr from [2] as well as in > >>>>> rkbin/master. > >>>> > >>>> I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > >>>> and it looked like it failed to read data into memory, see below. > >>>> > >>>> It also looks like the sdhci compatible is not supported by the driver. > >>>> Something that may need to be added to driver to properly read data? > >>>> > >>>> > >>>> DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > >>>> LPDDR4X, 2112MHz > >>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > >>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > >>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > >>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > >>>> Manufacturer ID:0x6 > >>>> CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > >>>> CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > >>>> CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > >>>> CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > >>>> change to F1: 528MHz > >>>> change to F2: 1068MHz > >>>> change to F3: 1560MHz > >>>> change to F0: 2112MHz > >>>> out > >>>> > >>>> U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > >>>> Trying to boot from MMC1 > >>>> ## Checking hash(es) for config config_1 ... OK > >>>> ## Checking hash(es) for Image atf_1 ... sha256 error! > >>>> Bad hash value for 'hash' hash node in 'atf_1' image node > >>>> mmc_load_image_raw_sector: mmc block read error > >>>> Trying to boot from MMC1 > >>>> ## Checking hash(es) for config config_1 ... OK > >>>> ## Checking hash(es) for Image atf_1 ... sha256 error! > >>>> Bad hash value for 'hash' hash node in 'atf_1' image node > >>> > >>> Look like this is something wrong with your patch series or master > >>> changes on binman, not with the driver. I have observed the same if I > >>> enable CONFIG_SPL_FIT_SIGNATURE. > >> > >> There are some more changes in dm/master that I'm about to pull in. > >> One of them from Jonas Karlman adds hash nodes so may be involved. > > > > I found the same issue on the dm/master > > > > U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) > > Trying to boot from MMC1 > > ## Checking hash(es) for config config-1 ... OK > > ## Checking hash(es) for Image atf-1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf-1' image node > > mmc_load_image_raw_sector: mmc block read error > > SPL: failed to boot from all boot devices > > ### ERROR ### Please RESET the board ### > > On my RK3568 ROCK 3A board this is working correctly, see below. > > This was using u-boot master 17e8e58fe62c019b2cc26af221b6defc3368229f > with a few patches on top, see [1]. > > ------------- > U-Boot SPL 2023.01 (Jan 26 2023 - 18:29:56 +0000) > Trying to boot from MMC1 > ## Checking hash(es) for config config-1 ... OK > ## Checking hash(es) for Image atf-1 ... sha256+ OK > ## Checking hash(es) for Image u-boot ... sha256+ OK > ## Checking hash(es) for Image fdt-1 ... sha256+ OK > ## Checking hash(es) for Image atf-2 ... sha256+ OK > ## Checking hash(es) for Image atf-3 ... sha256+ OK > ## Checking hash(es) for Image atf-4 ... sha256+ OK > ## Checking hash(es) for Image atf-5 ... sha256+ OK > ## Checking hash(es) for Image atf-6 ... sha256+ OK > INFO: Preloader serial: 2 > NOTICE: BL31: v2.3():v2.3-460-g2c8be93f9:huan.he > NOTICE: BL31: Built : 19:45:17, Nov 8 2022 > INFO: GICv3 without legacy support detected. > INFO: ARM GICv3 driver initialized in EL3 > INFO: pmu v1 is valid 220114 > INFO: dfs DDR fsp_param[0].freq_mhz= 1560MHz > INFO: dfs DDR fsp_param[1].freq_mhz= 324MHz > INFO: dfs DDR fsp_param[2].freq_mhz= 528MHz > INFO: dfs DDR fsp_param[3].freq_mhz= 780MHz > INFO: Using opteed sec cpu_context! > INFO: boot cpu mask: 0 > INFO: BL31: Initializing runtime services > WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK > ERROR: Error initializing runtime service opteed_fast > INFO: BL31: Preparing for EL3 exit to normal world > INFO: Entry point address = 0xa00000 > INFO: SPSR = 0x3c9 > > > U-Boot 2023.01 (Jan 26 2023 - 18:29:56 +0000) > > Model: Rockchip RK3568 EVB1 DDR4 V10 Board > DRAM: 8 GiB (effective 7.7 GiB) > Core: 58 devices, 17 uclasses, devicetree: separate > MMC: mmc@fe2b0000: 1, mmc@fe310000: 0 > ... > ------------- > > However, on my RK3588 ROCK 5B device I get a similar error you get. > Seems to be the mmc reading that times out all of sudden. > > It can read the FIP and config, and then some mmc command/transfer times out. > See below for details with LOG_DEBUG defined at top of include/log.h > > This was trying to boot from a SD-card, trying to use eMMC fails earlier. > Because there is no driver for emmc compatible, SPL only tries to boot from SD-card. > > I believe this is related to your << hang >>, reading atf > from mmc fails in the background and there is no error message. > > That is why I suggested trying with CONFIG_SPL_FIT_SIGNATURE=y, > to see if there is any hidden issue trying to load ATF. > > ------------- > Trying to boot from MMC1 > ## Checking hash(es) for config config-1 ... OK > mmc_load_image_raw_sector: mmc block read error > Trying to boot from MMC1 > mmc_load_image_raw_sector: mmc block read error > SPL: failed to boot from all boot devices > ### ERROR ### Please RESET the board ### > ------------- > Trying to boot from MMC1 > 0 > - 0 'mmc@fe2c0000' > - found > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > clock is disabled (0Hz) > Buswidth = 0, clock: 0 > Buswidth = 1, clock: 0 > clock is enabled (400000Hz) > Buswidth = 1, clock: 400000 > Sending CMD0 > Sending CMD8 > Sending CMD55 > Sending CMD41 > Sending CMD55 > Sending CMD41 > Sending CMD2 > Sending CMD3 > Sending CMD9 > Sending CMD7 > Sending CMD55 > Sending CMD51 > Sending CMD6 > Sending CMD55 > Sending CMD6 > Buswidth = 4, clock: 400000 > Sending CMD6 > clock is enabled (50000000Hz) > Buswidth = 4, clock: 50000000 > clk_set_rate(clk=500000, rate=50000000) > rockchip_dwmmc_get_mmc_clk: err=-2 > spl: mmc boot mode: raw > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD17 > hdr read sector 4000, count=1 > Found FIT > size=a00, ptr=ac0, limit=100000: aligned to 5000c0 > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD18 > Sending CMD12 > fit read sector 4000, sectors=5, dst=5000c0, count=5, size=0xa00 > Selecting default config 'rk3588-evb.dtb' > ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND > OK > firmware: 'atf-1' > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD18 > dwmci_data_transfer: Timeout waiting for data! > mmc_load_image_raw_sector: mmc block read error > spl: mmc boot mode: fs > Trying to boot from MMC1 > 0 > - 0 'mmc@fe2c0000' > - found > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > spl: mmc boot mode: raw > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > dwmci_send_cmd: Response Timeout. > hdr read sector 4000, count=0 > mmc_load_image_raw_sector: mmc block read error > spl: mmc boot mode: fs > SPL: failed to boot from all boot devices > ### ERROR ### Please RESET the board ### I'm thinking this is fit/binman issue rather than mmc. U-Boot SPL 2023.01-00176-gb21fb7a9c0-dirty (Jan 27 2023 - 00:43:52 +0530) SPL malloc() before relocation used 0xbe0 bytes (2 KB) >>SPL: board_init_r() spl_init size=a0, ptr=a0, limit=100000: 500000 size=8, ptr=a8, limit=100000: 5000a0 fdtdec_get_addr_size_auto_parent: na=2, ns=2, fdtdec_get_addr_size_fixed: reg: addr=00000000x ofnode_read_u32_index: bus-width: x (4) ofnode_read_bool: non-removable: false ofnode_read_u32_index: fifo-depth: x (256) ofnode_read_bool: fifo-mode: false ofnode_read_bool: u-boot,spl-fifo-mode: false ofnode_read_u32_array: clock-freq-min-max: fdtdec_get_int_array: clock-freq-min-max get_prop_check_min_len: clock-freq-min-max ofnode_read_u32_index: max-frequency: x (200000000) clk_set_defaults(mmc@fe2c0000) clk_set_default_parents: could not read assigned-clock-parents for 3f60700 ofnode_read_prop: assigned-clock-rates: <not found> fdtdec_get_int: #clock-cells: x (1) fdtdec_get_int: #clock-cells: x (1) Looking for clock-controller@fd7c0000 Looking for clock-controller@fd7c0000 - checking clock-1 - checking clock-controller@fd7c0000 - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) clk_of_xlate_default(clk=500000) clk_request(dev=3f60320, clk=500000) clk_set_rate(clk=500000, rate=400000) rockchip_dwmmc_get_mmc_clk: err=-2 Trying to boot from MMC1 0 - 0 'mmc@fe2c0000' - found blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 clock is disabled (0Hz) Buswidth = 0, clock: 0 Buswidth = 1, clock: 0 clock is enabled (400000Hz) Buswidth = 1, clock: 400000 Sending CMD0 Sending CMD8 Sending CMD55 Sending CMD41 Sending CMD55 Sending CMD41 Sending CMD55 Sending CMD41 Sending CMD2 Sending CMD3 Sending CMD9 Sending CMD7 Sending CMD55 Unaligned buffer length u size=40, ptr=100, limit=100000: aligned to 5000c0 Sending CMD51 Sending CMD6 Sending CMD55 Sending CMD6 Buswidth = 4, clock: 400000 Sending CMD6 clock is enabled (50000000Hz) Buswidth = 4, clock: 50000000 clk_set_rate(clk=500000, rate=50000000) rockchip_dwmmc_get_mmc_clk: err=-2 spl: mmc boot mode: raw blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD17 hdr read sector 4000, count=1 Found FIT size=a00, ptr=b00, limit=100000: aligned to 500100 blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD18 Sending CMD12 fit read sector 4000, sectors=5, dst=500100, count=5, size=0xa00 Selecting default config 'rk3588-edgeble-neu6a-io.dtb' ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND OK firmware: 'atf-1' blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 Sending CMD16 Sending CMD18 dwmci_data_transfer: Timeout waiting for data! Sending CMD12 External data: dst=40000, offset=ad200, size=2e940 ## Checking hash(es) for Image atf-1 ... fit_image_verify_required_sigs: No signature node found: FDT_ERR_NOTFOUND sha256 error! Bad hash value for 'hash' hash node in 'atf-1' image node mmc_load_image_raw_sector: mmc block read error spl: mmc boot mode: fs SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### Please check this repo, it is on top of the dm/master https://github.com/edgeble/u-boot/tree/neu6-v1 Jagan.
Hi Simon, On Fri, 27 Jan 2023 at 00:45, Simon Glass <sjg@chromium.org> wrote: > > Hi Jagan, > > On Thu, 26 Jan 2023 at 11:27, Jagan Teki <jagan@edgeble.ai> wrote: > > > > Hi Simon, > > > > On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi Jagan, > > > > > > On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > > > > > > > > On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > > > > > > > Hi Jagan, > > > > > On 2023-01-26 17:51, Jagan Teki wrote: > > > > > > Hi Jonas, > > > > > > > > > > > > On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > > > > > >> > > > > > >> Hi Jagan, > > > > > >> > > > > > >> On 2023-01-25 23:27, Jagan Teki wrote: > > > > > >>> This series support Rockchip RK3588. All the device tree files are > > > > > >>> synced from linux-next with the proper SHA1 mentioned in the commit > > > > > >>> messages. > > > > > >>> > > > > > >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > > > > > >>> it is failing to load ATF entry from SPL and hang. > > > > > >>> > > > > > >>> Verified below BL31 versions, > > > > > >>> bl31-v1.15 > > > > > >>> bl31-v1.21 > > > > > >>> bl31-v1.22 > > > > > >>> bl31-v1.23 > > > > > >>> bl31-v1.24 > > > > > >>> bl31-v1.25 > > > > > >>> bl31-v1.26 > > > > > >>> > > > > > >>> Rever-engineered with respect to rockchip u-boot by using the same > > > > > >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > > > > > >>> mainline showing the same issue. > > > > > >>> > > > > > >>> Log: > > > > > >>> > > > > > >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > > > > > >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > > >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > > >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > > >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > > > > > >>> change to F1: 528MHz > > > > > >>> change to F2: 1068MHz > > > > > >>> change to F3: 1560MHz > > > > > >>> change to F0: 2112MHz > > > > > >>> out > > > > > >>> > > > > > >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > > > > > >>> Trying to boot from MMC1 > > > > > >>> bl31_entry: atf_entry start > > > > > >>> << hang >> > > > > > >>> > > > > > >>> Any information on BL31 for RK3588 please share. > > > > > >> > > > > > >> I had a similar strange booing issue with RK3568 and mainline U-Boot, > > > > > >> turned out to be related to all parts of ATF not being properly loaded > > > > > >> into PMU SRAM. > > > > > >> > > > > > >> Using my series at [1] I managed to get ATF to be fully loaded into > > > > > >> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that > > > > > >> the segment being loaded ended up corrupted. > > > > > >> > > > > > >> The use of 512 bytes alignment of the FIT helped mitigate that issue. > > > > > >> Vendor U-Boot use a bounce buffer for all parts that is written into > > > > > >> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). > > > > > >> > > > > > >> You can also find newer bl31 at [2], up to version v1.32. > > > > > >> > > > > > >> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891>>> [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35>> > > > > > > Thanks for the details. I did apply this set on the master. No change > > > > > > in the behavior, used BL31 and ddr from [2] as well as in > > > > > > rkbin/master. > > > > > > > > > > I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y > > > > > and it looked like it failed to read data into memory, see below. > > > > > > > > > > It also looks like the sdhci compatible is not supported by the driver. > > > > > Something that may need to be added to driver to properly read data? > > > > > > > > > > > > > > > DDR V1.09 a930779e06 typ 22/11/21-17:50:56 > > > > > LPDDR4X, 2112MHz > > > > > channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > > channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > > channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > > channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB > > > > > Manufacturer ID:0x6 > > > > > CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% > > > > > CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% > > > > > CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% > > > > > CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% > > > > > change to F1: 528MHz > > > > > change to F2: 1068MHz > > > > > change to F3: 1560MHz > > > > > change to F0: 2112MHz > > > > > out > > > > > > > > > > U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) > > > > > Trying to boot from MMC1 > > > > > ## Checking hash(es) for config config_1 ... OK > > > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > > > mmc_load_image_raw_sector: mmc block read error > > > > > Trying to boot from MMC1 > > > > > ## Checking hash(es) for config config_1 ... OK > > > > > ## Checking hash(es) for Image atf_1 ... sha256 error! > > > > > Bad hash value for 'hash' hash node in 'atf_1' image node > > > > > > > > Look like this is something wrong with your patch series or master > > > > changes on binman, not with the driver. I have observed the same if I > > > > enable CONFIG_SPL_FIT_SIGNATURE. > > > > > > There are some more changes in dm/master that I'm about to pull in. > > > One of them from Jonas Karlman adds hash nodes so may be involved. > > > > I found the same issue on the dm/master > > > > U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) > > Trying to boot from MMC1 > > ## Checking hash(es) for config config-1 ... OK > > ## Checking hash(es) for Image atf-1 ... sha256 error! > > Bad hash value for 'hash' hash node in 'atf-1' image node > > mmc_load_image_raw_sector: mmc block read error > > SPL: failed to boot from all boot devices > > ### ERROR ### Please RESET the board ### > > Is the FIT image broken? You can use check_sign or dump_image to see. This seems okay, let me know if you see any issue. > ./tools/fit_check_sign -f u-boot.itb -k arch/arm/dts/rk3588-edgeble-neu6a-io.dtb Verifying Hash Integrity for node 'config-1'... Verified OK, loading images ## Loading kernel from FIT Image at 7f98532c7000 ... Using 'config-1' configuration Verifying Hash Integrity ... OK Could not find subimage node type 'kernel' ## Loading fdt from FIT Image at 7f98532c7000 ... Using 'config-1' configuration Verifying Hash Integrity ... OK Trying 'fdt-1' fdt subimage Description: fdt-rk3588-edgeble-neu6a-io Created: Fri Jan 27 01:01:31 2023 Type: Flat Device Tree Compression: uncompressed Data Size: 41408 Bytes = 40.44 KiB = 0.04 MiB Architecture: Unknown Architecture Hash algo: sha256 Hash value: a5de5505624412d216c6e985fb3669e84785b3675765bc85f08d8b0dbeffdbd7 Verifying Hash Integrity ... sha256+ OK Decrypting Data ... OK Loading Flat Device Tree ## Loading ramdisk from FIT Image at 7f98532c7000 ... Using 'config-1' configuration Verifying Hash Integrity ... OK Could not find subimage node type 'ramdisk' Signature check OK Thanks, Jagan.
Hi Jagan, On 2023-01-26 20:17, Jagan Teki wrote: > On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >> >> On 2023-01-26 19:26, Jagan Teki wrote: >>> Hi Simon, >>> >>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>> >>>> Hi Jagan, >>>> >>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>> >>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>> >>>>>> Hi Jagan, >>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>> Hi Jonas, >>>>>>> >>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>> >>>>>>>> Hi Jagan, >>>>>>>> >>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>> messages. >>>>>>>>> >>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>> >>>>>>>>> Verified below BL31 versions, >>>>>>>>> bl31-v1.15 >>>>>>>>> bl31-v1.21 >>>>>>>>> bl31-v1.22 >>>>>>>>> bl31-v1.23 >>>>>>>>> bl31-v1.24 >>>>>>>>> bl31-v1.25 >>>>>>>>> bl31-v1.26 >>>>>>>>> >>>>>>>>> Rever-engineered with respect to rockchip u-boot by using the same >>>>>>>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>>>>>>>> mainline showing the same issue. >>>>>>>>> >>>>>>>>> Log: >>>>>>>>> >>>>>>>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>>>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>> change to F1: 528MHz >>>>>>>>> change to F2: 1068MHz >>>>>>>>> change to F3: 1560MHz >>>>>>>>> change to F0: 2112MHz >>>>>>>>> out >>>>>>>>> >>>>>>>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>>>>>>>> Trying to boot from MMC1 >>>>>>>>> bl31_entry: atf_entry start >>>>>>>>> << hang >> >>>>>>>>> >>>>>>>>> Any information on BL31 for RK3588 please share. >>>>>>>> >>>>>>>> I had a similar strange booing issue with RK3568 and mainline U-Boot, >>>>>>>> turned out to be related to all parts of ATF not being properly loaded >>>>>>>> into PMU SRAM. >>>>>>>> >>>>>>>> Using my series at [1] I managed to get ATF to be fully loaded into >>>>>>>> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that >>>>>>>> the segment being loaded ended up corrupted. >>>>>>>> >>>>>>>> The use of 512 bytes alignment of the FIT helped mitigate that issue. >>>>>>>> Vendor U-Boot use a bounce buffer for all parts that is written into >>>>>>>> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). >>>>>>>> >>>>>>>> You can also find newer bl31 at [2], up to version v1.32. >>>>>>>> >>>>>>>> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35 Thanks for the details. I did apply this set on the master. No change >>>>>>> in the behavior, used BL31 and ddr from [2] as well as in >>>>>>> rkbin/master. >>>>>> >>>>>> I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y >>>>>> and it looked like it failed to read data into memory, see below. >>>>>> >>>>>> It also looks like the sdhci compatible is not supported by the driver. >>>>>> Something that may need to be added to driver to properly read data? >>>>>> >>>>>> >>>>>> DDR V1.09 a930779e06 typ 22/11/21-17:50:56 >>>>>> LPDDR4X, 2112MHz >>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>> Manufacturer ID:0x6 >>>>>> CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% >>>>>> CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% >>>>>> CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% >>>>>> CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% >>>>>> change to F1: 528MHz >>>>>> change to F2: 1068MHz >>>>>> change to F3: 1560MHz >>>>>> change to F0: 2112MHz >>>>>> out >>>>>> >>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) >>>>>> Trying to boot from MMC1 >>>>>> ## Checking hash(es) for config config_1 ... OK >>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>>>>> mmc_load_image_raw_sector: mmc block read error >>>>>> Trying to boot from MMC1 >>>>>> ## Checking hash(es) for config config_1 ... OK >>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>>>> >>>>> Look like this is something wrong with your patch series or master >>>>> changes on binman, not with the driver. I have observed the same if I >>>>> enable CONFIG_SPL_FIT_SIGNATURE. >>>> >>>> There are some more changes in dm/master that I'm about to pull in. >>>> One of them from Jonas Karlman adds hash nodes so may be involved. >>> >>> I found the same issue on the dm/master >>> >>> U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) >>> Trying to boot from MMC1 >>> ## Checking hash(es) for config config-1 ... OK >>> ## Checking hash(es) for Image atf-1 ... sha256 error! >>> Bad hash value for 'hash' hash node in 'atf-1' image node >>> mmc_load_image_raw_sector: mmc block read error >>> SPL: failed to boot from all boot devices >>> ### ERROR ### Please RESET the board ### >> >> On my RK3568 ROCK 3A board this is working correctly, see below. >> >> This was using u-boot master 17e8e58fe62c019b2cc26af221b6defc3368229f >> with a few patches on top, see [1]. >> >> ------------- >> U-Boot SPL 2023.01 (Jan 26 2023 - 18:29:56 +0000) >> Trying to boot from MMC1 >> ## Checking hash(es) for config config-1 ... OK >> ## Checking hash(es) for Image atf-1 ... sha256+ OK >> ## Checking hash(es) for Image u-boot ... sha256+ OK >> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >> ## Checking hash(es) for Image atf-2 ... sha256+ OK >> ## Checking hash(es) for Image atf-3 ... sha256+ OK >> ## Checking hash(es) for Image atf-4 ... sha256+ OK >> ## Checking hash(es) for Image atf-5 ... sha256+ OK >> ## Checking hash(es) for Image atf-6 ... sha256+ OK >> INFO: Preloader serial: 2 >> NOTICE: BL31: v2.3():v2.3-460-g2c8be93f9:huan.he >> NOTICE: BL31: Built : 19:45:17, Nov 8 2022 >> INFO: GICv3 without legacy support detected. >> INFO: ARM GICv3 driver initialized in EL3 >> INFO: pmu v1 is valid 220114 >> INFO: dfs DDR fsp_param[0].freq_mhz= 1560MHz >> INFO: dfs DDR fsp_param[1].freq_mhz= 324MHz >> INFO: dfs DDR fsp_param[2].freq_mhz= 528MHz >> INFO: dfs DDR fsp_param[3].freq_mhz= 780MHz >> INFO: Using opteed sec cpu_context! >> INFO: boot cpu mask: 0 >> INFO: BL31: Initializing runtime services >> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >> ERROR: Error initializing runtime service opteed_fast >> INFO: BL31: Preparing for EL3 exit to normal world >> INFO: Entry point address = 0xa00000 >> INFO: SPSR = 0x3c9 >> >> >> U-Boot 2023.01 (Jan 26 2023 - 18:29:56 +0000) >> >> Model: Rockchip RK3568 EVB1 DDR4 V10 Board >> DRAM: 8 GiB (effective 7.7 GiB) >> Core: 58 devices, 17 uclasses, devicetree: separate >> MMC: mmc@fe2b0000: 1, mmc@fe310000: 0 >> ... >> ------------- >> >> However, on my RK3588 ROCK 5B device I get a similar error you get. >> Seems to be the mmc reading that times out all of sudden. >> >> It can read the FIP and config, and then some mmc command/transfer times out. >> See below for details with LOG_DEBUG defined at top of include/log.h >> >> This was trying to boot from a SD-card, trying to use eMMC fails earlier. >> Because there is no driver for emmc compatible, SPL only tries to boot from SD-card. >> >> I believe this is related to your << hang >>, reading atf >> from mmc fails in the background and there is no error message. >> >> That is why I suggested trying with CONFIG_SPL_FIT_SIGNATURE=y, >> to see if there is any hidden issue trying to load ATF. >> >> ------------- >> Trying to boot from MMC1 >> ## Checking hash(es) for config config-1 ... OK >> mmc_load_image_raw_sector: mmc block read error >> Trying to boot from MMC1 >> mmc_load_image_raw_sector: mmc block read error >> SPL: failed to boot from all boot devices >> ### ERROR ### Please RESET the board ### >> ------------- >> Trying to boot from MMC1 >> 0 >> - 0 'mmc@fe2c0000' >> - found >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> clock is disabled (0Hz) >> Buswidth = 0, clock: 0 >> Buswidth = 1, clock: 0 >> clock is enabled (400000Hz) >> Buswidth = 1, clock: 400000 >> Sending CMD0 >> Sending CMD8 >> Sending CMD55 >> Sending CMD41 >> Sending CMD55 >> Sending CMD41 >> Sending CMD2 >> Sending CMD3 >> Sending CMD9 >> Sending CMD7 >> Sending CMD55 >> Sending CMD51 >> Sending CMD6 >> Sending CMD55 >> Sending CMD6 >> Buswidth = 4, clock: 400000 >> Sending CMD6 >> clock is enabled (50000000Hz) >> Buswidth = 4, clock: 50000000 >> clk_set_rate(clk=500000, rate=50000000) >> rockchip_dwmmc_get_mmc_clk: err=-2 >> spl: mmc boot mode: raw >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD17 >> hdr read sector 4000, count=1 >> Found FIT >> size=a00, ptr=ac0, limit=100000: aligned to 5000c0 >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD18 >> Sending CMD12 >> fit read sector 4000, sectors=5, dst=5000c0, count=5, size=0xa00 >> Selecting default config 'rk3588-evb.dtb' >> ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND >> OK >> firmware: 'atf-1' >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD18 >> dwmci_data_transfer: Timeout waiting for data! >> mmc_load_image_raw_sector: mmc block read error >> spl: mmc boot mode: fs >> Trying to boot from MMC1 >> 0 >> - 0 'mmc@fe2c0000' >> - found >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> spl: mmc boot mode: raw >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> dwmci_send_cmd: Response Timeout. >> hdr read sector 4000, count=0 >> mmc_load_image_raw_sector: mmc block read error >> spl: mmc boot mode: fs >> SPL: failed to boot from all boot devices >> ### ERROR ### Please RESET the board ### > > I'm thinking this is fit/binman issue rather than mmc. > > U-Boot SPL 2023.01-00176-gb21fb7a9c0-dirty (Jan 27 2023 - 00:43:52 +0530) > SPL malloc() before relocation used 0xbe0 bytes (2 KB) >>> SPL: board_init_r() > spl_init > size=a0, ptr=a0, limit=100000: 500000 > size=8, ptr=a8, limit=100000: 5000a0 > fdtdec_get_addr_size_auto_parent: na=2, ns=2, > fdtdec_get_addr_size_fixed: reg: addr=00000000x > ofnode_read_u32_index: bus-width: x (4) > ofnode_read_bool: non-removable: false > ofnode_read_u32_index: fifo-depth: x (256) > ofnode_read_bool: fifo-mode: false > ofnode_read_bool: u-boot,spl-fifo-mode: false > ofnode_read_u32_array: clock-freq-min-max: fdtdec_get_int_array: > clock-freq-min-max > get_prop_check_min_len: clock-freq-min-max > ofnode_read_u32_index: max-frequency: x (200000000) > clk_set_defaults(mmc@fe2c0000) > clk_set_default_parents: could not read assigned-clock-parents for 3f60700 > ofnode_read_prop: assigned-clock-rates: <not found> > fdtdec_get_int: #clock-cells: x (1) > fdtdec_get_int: #clock-cells: x (1) > Looking for clock-controller@fd7c0000 > Looking for clock-controller@fd7c0000 > - checking clock-1 > - checking clock-controller@fd7c0000 > - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) > - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) > clk_of_xlate_default(clk=500000) > clk_request(dev=3f60320, clk=500000) > clk_set_rate(clk=500000, rate=400000) > rockchip_dwmmc_get_mmc_clk: err=-2 > Trying to boot from MMC1 > 0 > - 0 'mmc@fe2c0000' > - found > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > clock is disabled (0Hz) > Buswidth = 0, clock: 0 > Buswidth = 1, clock: 0 > clock is enabled (400000Hz) > Buswidth = 1, clock: 400000 > Sending CMD0 > Sending CMD8 > Sending CMD55 > Sending CMD41 > Sending CMD55 > Sending CMD41 > Sending CMD55 > Sending CMD41 > Sending CMD2 > Sending CMD3 > Sending CMD9 > Sending CMD7 > Sending CMD55 > Unaligned buffer length u > size=40, ptr=100, limit=100000: aligned to 5000c0 > Sending CMD51 > Sending CMD6 > Sending CMD55 > Sending CMD6 > Buswidth = 4, clock: 400000 > Sending CMD6 > clock is enabled (50000000Hz) > Buswidth = 4, clock: 50000000 > clk_set_rate(clk=500000, rate=50000000) > rockchip_dwmmc_get_mmc_clk: err=-2 > spl: mmc boot mode: raw > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD17 > hdr read sector 4000, count=1 > Found FIT > size=a00, ptr=b00, limit=100000: aligned to 500100 > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD18 > Sending CMD12 > fit read sector 4000, sectors=5, dst=500100, count=5, size=0xa00 > Selecting default config 'rk3588-edgeble-neu6a-io.dtb' > ## Checking hash(es) for config config-1 ... > fit_config_verify_required_keys: No signature node found: > FDT_ERR_NOTFOUND > OK > firmware: 'atf-1' > blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 > Sending CMD16 > Sending CMD18 > dwmci_data_transfer: Timeout waiting for data! > Sending CMD12 > External data: dst=40000, offset=ad200, size=2e940 > ## Checking hash(es) for Image atf-1 ... > fit_image_verify_required_sigs: No signature node found: > FDT_ERR_NOTFOUND > sha256 error! > Bad hash value for 'hash' hash node in 'atf-1' image node > mmc_load_image_raw_sector: mmc block read error > spl: mmc boot mode: fs > SPL: failed to boot from all boot devices > ### ERROR ### Please RESET the board ### As you can see in the logs above there is timeout waiting for data. I managed to find the issue and have a workaround that gets me longer in the boot process, there still seem to be other issue with the rk3588 startup. -------- U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) Trying to boot from MMC1 ## Checking hash(es) for config config-1 ... OK ## Checking hash(es) for Image atf-1 ... sha256+ OK ## Checking hash(es) for Image u-boot ... sha256+ OK ## Checking hash(es) for Image fdt-1 ... sha256+ OK ## Checking hash(es) for Image atf-2 ... sha256+ OK ## Checking hash(es) for Image atf-3 ... sha256+ OK INFO: Preloader serial: 2 NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang NOTICE: BL31: Built : 09:59:49, Nov 21 2022 INFO: spec: 0x1 INFO: ext 32k is not valid INFO: ddr: stride-en 4CH INFO: GICv3 without legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 INFO: system boots from cpu-hwid-0 INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz INFO: BL31: Initialising Exception Handling Framework INFO: BL31: Initializing runtime services WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK ERROR: Error initializing runtime service opteed_fast INFO: BL31: Preparing for EL3 exit to normal world INFO: Entry point address = 0xa00000 INFO: SPSR = 0x3c9 "Synchronous Abort" handler, esr 0x96000000 elr: 0000000000a23650 lr : 0000000000a24d9c x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 x2 : 0000000000000000 x3 : 0000000000b80ff0 x4 : 0000000000b80ff0 x5 : 0000000000b80e88 x6 : 0000000000000054 x7 : 0000000000000044 x8 : 000000000000000a x9 : 0000000000000000 x10: 0000000000000034 x11: 0000000000000002 x12: 0000000000001988 x13: 0000000000b7fadc x14: 0000000000a7e808 x15: 0000000000a7e808 x16: 0000000000000000 x17: 0000000000000000 x18: 0000000000b7fe50 x19: 0000000000b7fbe8 x20: 000000003c14dc00 x21: 000000003c14dc00 x22: 0000000000a7e808 x23: 0000000000000000 x24: 0000000000000000 x25: 0000000000000000 x26: 0000000000000000 x27: 0000000000000000 x28: 0000000000000000 x29: 0000000000b7fb80 Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) Resetting CPU ... -------- This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e with some rk3568 patches and this series, see [1]. The last 3 commits contains workaround to issue with sdmmc clock. dwmmc driver set sclk to (uint)-2, my workaround just adds a fallback to default 400khz clock. Next issue is the sync abort, looks it happens when u-boot tries to set clock rates based on devicetree. this is the last debug line before the crash. clk_set_rate(clk=0000000000b7fba8, rate=1008000000) [1] https://github.com/Kwiboo/u-boot-rockchip/compare/060a65e899859dcbf42049a18be20ce7118e7c0e...be21c34067d6e3a2036ce70d571700170c59d270 Regards, Jonas > > Please check this repo, it is on top of the dm/master > https://github.com/edgeble/u-boot/tree/neu6-v1 > > Jagan.
On 2023-01-26 23:16, Jonas Karlman wrote: > Hi Jagan, > On 2023-01-26 20:17, Jagan Teki wrote: >> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>> >>> On 2023-01-26 19:26, Jagan Teki wrote: >>>> Hi Simon, >>>> >>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>> >>>>> Hi Jagan, >>>>> >>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>> >>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>> >>>>>>> Hi Jagan, >>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>> Hi Jonas, >>>>>>>> >>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>> >>>>>>>>> Hi Jagan, >>>>>>>>> >>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>> messages. >>>>>>>>>> >>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>> >>>>>>>>>> Verified below BL31 versions, >>>>>>>>>> bl31-v1.15 >>>>>>>>>> bl31-v1.21 >>>>>>>>>> bl31-v1.22 >>>>>>>>>> bl31-v1.23 >>>>>>>>>> bl31-v1.24 >>>>>>>>>> bl31-v1.25 >>>>>>>>>> bl31-v1.26 >>>>>>>>>> >>>>>>>>>> Rever-engineered with respect to rockchip u-boot by using the same >>>>>>>>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>>>>>>>>> mainline showing the same issue. >>>>>>>>>> >>>>>>>>>> Log: >>>>>>>>>> >>>>>>>>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>>>>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>>>>>>>> change to F1: 528MHz >>>>>>>>>> change to F2: 1068MHz >>>>>>>>>> change to F3: 1560MHz >>>>>>>>>> change to F0: 2112MHz >>>>>>>>>> out >>>>>>>>>> >>>>>>>>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>>>>>>>>> Trying to boot from MMC1 >>>>>>>>>> bl31_entry: atf_entry start >>>>>>>>>> << hang >> >>>>>>>>>> >>>>>>>>>> Any information on BL31 for RK3588 please share. >>>>>>>>> >>>>>>>>> I had a similar strange booing issue with RK3568 and mainline U-Boot, >>>>>>>>> turned out to be related to all parts of ATF not being properly loaded >>>>>>>>> into PMU SRAM. >>>>>>>>> >>>>>>>>> Using my series at [1] I managed to get ATF to be fully loaded into >>>>>>>>> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that >>>>>>>>> the segment being loaded ended up corrupted. >>>>>>>>> >>>>>>>>> The use of 512 bytes alignment of the FIT helped mitigate that issue. >>>>>>>>> Vendor U-Boot use a bounce buffer for all parts that is written into >>>>>>>>> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range). >>>>>>>>> >>>>>>>>> You can also find newer bl31 at [2], up to version v1.32. >>>>>>>>> >>>>>>>>> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35 Thanks for the details. I did apply this set on the master. No change >>>>>>>> in the behavior, used BL31 and ddr from [2] as well as in >>>>>>>> rkbin/master. >>>>>>> >>>>>>> I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y >>>>>>> and it looked like it failed to read data into memory, see below. >>>>>>> >>>>>>> It also looks like the sdhci compatible is not supported by the driver. >>>>>>> Something that may need to be added to driver to properly read data? >>>>>>> >>>>>>> >>>>>>> DDR V1.09 a930779e06 typ 22/11/21-17:50:56 >>>>>>> LPDDR4X, 2112MHz >>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB >>>>>>> Manufacturer ID:0x6 >>>>>>> CH0 RX Vref:31.7%, TX Vref:21.8%,20.8% >>>>>>> CH1 RX Vref:31.7%, TX Vref:21.8%,21.8% >>>>>>> CH2 RX Vref:32.7%, TX Vref:22.8%,21.8% >>>>>>> CH3 RX Vref:32.7%, TX Vref:21.8%,20.8% >>>>>>> change to F1: 528MHz >>>>>>> change to F2: 1068MHz >>>>>>> change to F3: 1560MHz >>>>>>> change to F0: 2112MHz >>>>>>> out >>>>>>> >>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000) >>>>>>> Trying to boot from MMC1 >>>>>>> ## Checking hash(es) for config config_1 ... OK >>>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>>>>>> mmc_load_image_raw_sector: mmc block read error >>>>>>> Trying to boot from MMC1 >>>>>>> ## Checking hash(es) for config config_1 ... OK >>>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error! >>>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node >>>>>> >>>>>> Look like this is something wrong with your patch series or master >>>>>> changes on binman, not with the driver. I have observed the same if I >>>>>> enable CONFIG_SPL_FIT_SIGNATURE. >>>>> >>>>> There are some more changes in dm/master that I'm about to pull in. >>>>> One of them from Jonas Karlman adds hash nodes so may be involved. >>>> >>>> I found the same issue on the dm/master >>>> >>>> U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530) >>>> Trying to boot from MMC1 >>>> ## Checking hash(es) for config config-1 ... OK >>>> ## Checking hash(es) for Image atf-1 ... sha256 error! >>>> Bad hash value for 'hash' hash node in 'atf-1' image node >>>> mmc_load_image_raw_sector: mmc block read error >>>> SPL: failed to boot from all boot devices >>>> ### ERROR ### Please RESET the board ### >>> >>> On my RK3568 ROCK 3A board this is working correctly, see below. >>> >>> This was using u-boot master 17e8e58fe62c019b2cc26af221b6defc3368229f >>> with a few patches on top, see [1]. >>> >>> ------------- >>> U-Boot SPL 2023.01 (Jan 26 2023 - 18:29:56 +0000) >>> Trying to boot from MMC1 >>> ## Checking hash(es) for config config-1 ... OK >>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>> ## Checking hash(es) for Image atf-4 ... sha256+ OK >>> ## Checking hash(es) for Image atf-5 ... sha256+ OK >>> ## Checking hash(es) for Image atf-6 ... sha256+ OK >>> INFO: Preloader serial: 2 >>> NOTICE: BL31: v2.3():v2.3-460-g2c8be93f9:huan.he >>> NOTICE: BL31: Built : 19:45:17, Nov 8 2022 >>> INFO: GICv3 without legacy support detected. >>> INFO: ARM GICv3 driver initialized in EL3 >>> INFO: pmu v1 is valid 220114 >>> INFO: dfs DDR fsp_param[0].freq_mhz= 1560MHz >>> INFO: dfs DDR fsp_param[1].freq_mhz= 324MHz >>> INFO: dfs DDR fsp_param[2].freq_mhz= 528MHz >>> INFO: dfs DDR fsp_param[3].freq_mhz= 780MHz >>> INFO: Using opteed sec cpu_context! >>> INFO: boot cpu mask: 0 >>> INFO: BL31: Initializing runtime services >>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>> ERROR: Error initializing runtime service opteed_fast >>> INFO: BL31: Preparing for EL3 exit to normal world >>> INFO: Entry point address = 0xa00000 >>> INFO: SPSR = 0x3c9 >>> >>> >>> U-Boot 2023.01 (Jan 26 2023 - 18:29:56 +0000) >>> >>> Model: Rockchip RK3568 EVB1 DDR4 V10 Board >>> DRAM: 8 GiB (effective 7.7 GiB) >>> Core: 58 devices, 17 uclasses, devicetree: separate >>> MMC: mmc@fe2b0000: 1, mmc@fe310000: 0 >>> ... >>> ------------- >>> >>> However, on my RK3588 ROCK 5B device I get a similar error you get. >>> Seems to be the mmc reading that times out all of sudden. >>> >>> It can read the FIP and config, and then some mmc command/transfer times out. >>> See below for details with LOG_DEBUG defined at top of include/log.h >>> >>> This was trying to boot from a SD-card, trying to use eMMC fails earlier. >>> Because there is no driver for emmc compatible, SPL only tries to boot from SD-card. >>> >>> I believe this is related to your << hang >>, reading atf >>> from mmc fails in the background and there is no error message. >>> >>> That is why I suggested trying with CONFIG_SPL_FIT_SIGNATURE=y, >>> to see if there is any hidden issue trying to load ATF. >>> >>> ------------- >>> Trying to boot from MMC1 >>> ## Checking hash(es) for config config-1 ... OK >>> mmc_load_image_raw_sector: mmc block read error >>> Trying to boot from MMC1 >>> mmc_load_image_raw_sector: mmc block read error >>> SPL: failed to boot from all boot devices >>> ### ERROR ### Please RESET the board ### >>> ------------- >>> Trying to boot from MMC1 >>> 0 >>> - 0 'mmc@fe2c0000' >>> - found >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> clock is disabled (0Hz) >>> Buswidth = 0, clock: 0 >>> Buswidth = 1, clock: 0 >>> clock is enabled (400000Hz) >>> Buswidth = 1, clock: 400000 >>> Sending CMD0 >>> Sending CMD8 >>> Sending CMD55 >>> Sending CMD41 >>> Sending CMD55 >>> Sending CMD41 >>> Sending CMD2 >>> Sending CMD3 >>> Sending CMD9 >>> Sending CMD7 >>> Sending CMD55 >>> Sending CMD51 >>> Sending CMD6 >>> Sending CMD55 >>> Sending CMD6 >>> Buswidth = 4, clock: 400000 >>> Sending CMD6 >>> clock is enabled (50000000Hz) >>> Buswidth = 4, clock: 50000000 >>> clk_set_rate(clk=500000, rate=50000000) >>> rockchip_dwmmc_get_mmc_clk: err=-2 >>> spl: mmc boot mode: raw >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> Sending CMD16 >>> Sending CMD17 >>> hdr read sector 4000, count=1 >>> Found FIT >>> size=a00, ptr=ac0, limit=100000: aligned to 5000c0 >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> Sending CMD16 >>> Sending CMD18 >>> Sending CMD12 >>> fit read sector 4000, sectors=5, dst=5000c0, count=5, size=0xa00 >>> Selecting default config 'rk3588-evb.dtb' >>> ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND >>> OK >>> firmware: 'atf-1' >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> Sending CMD16 >>> Sending CMD18 >>> dwmci_data_transfer: Timeout waiting for data! >>> mmc_load_image_raw_sector: mmc block read error >>> spl: mmc boot mode: fs >>> Trying to boot from MMC1 >>> 0 >>> - 0 'mmc@fe2c0000' >>> - found >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> spl: mmc boot mode: raw >>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >>> Sending CMD16 >>> dwmci_send_cmd: Response Timeout. >>> hdr read sector 4000, count=0 >>> mmc_load_image_raw_sector: mmc block read error >>> spl: mmc boot mode: fs >>> SPL: failed to boot from all boot devices >>> ### ERROR ### Please RESET the board ### >> >> I'm thinking this is fit/binman issue rather than mmc. >> >> U-Boot SPL 2023.01-00176-gb21fb7a9c0-dirty (Jan 27 2023 - 00:43:52 +0530) >> SPL malloc() before relocation used 0xbe0 bytes (2 KB) >>>> SPL: board_init_r() >> spl_init >> size=a0, ptr=a0, limit=100000: 500000 >> size=8, ptr=a8, limit=100000: 5000a0 >> fdtdec_get_addr_size_auto_parent: na=2, ns=2, >> fdtdec_get_addr_size_fixed: reg: addr=00000000x >> ofnode_read_u32_index: bus-width: x (4) >> ofnode_read_bool: non-removable: false >> ofnode_read_u32_index: fifo-depth: x (256) >> ofnode_read_bool: fifo-mode: false >> ofnode_read_bool: u-boot,spl-fifo-mode: false >> ofnode_read_u32_array: clock-freq-min-max: fdtdec_get_int_array: >> clock-freq-min-max >> get_prop_check_min_len: clock-freq-min-max >> ofnode_read_u32_index: max-frequency: x (200000000) >> clk_set_defaults(mmc@fe2c0000) >> clk_set_default_parents: could not read assigned-clock-parents for 3f60700 >> ofnode_read_prop: assigned-clock-rates: <not found> >> fdtdec_get_int: #clock-cells: x (1) >> fdtdec_get_int: #clock-cells: x (1) >> Looking for clock-controller@fd7c0000 >> Looking for clock-controller@fd7c0000 >> - checking clock-1 >> - checking clock-controller@fd7c0000 >> - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) >> - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0) >> clk_of_xlate_default(clk=500000) >> clk_request(dev=3f60320, clk=500000) >> clk_set_rate(clk=500000, rate=400000) >> rockchip_dwmmc_get_mmc_clk: err=-2 >> Trying to boot from MMC1 >> 0 >> - 0 'mmc@fe2c0000' >> - found >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> clock is disabled (0Hz) >> Buswidth = 0, clock: 0 >> Buswidth = 1, clock: 0 >> clock is enabled (400000Hz) >> Buswidth = 1, clock: 400000 >> Sending CMD0 >> Sending CMD8 >> Sending CMD55 >> Sending CMD41 >> Sending CMD55 >> Sending CMD41 >> Sending CMD55 >> Sending CMD41 >> Sending CMD2 >> Sending CMD3 >> Sending CMD9 >> Sending CMD7 >> Sending CMD55 >> Unaligned buffer length u >> size=40, ptr=100, limit=100000: aligned to 5000c0 >> Sending CMD51 >> Sending CMD6 >> Sending CMD55 >> Sending CMD6 >> Buswidth = 4, clock: 400000 >> Sending CMD6 >> clock is enabled (50000000Hz) >> Buswidth = 4, clock: 50000000 >> clk_set_rate(clk=500000, rate=50000000) >> rockchip_dwmmc_get_mmc_clk: err=-2 >> spl: mmc boot mode: raw >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD17 >> hdr read sector 4000, count=1 >> Found FIT >> size=a00, ptr=b00, limit=100000: aligned to 500100 >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD18 >> Sending CMD12 >> fit read sector 4000, sectors=5, dst=500100, count=5, size=0xa00 >> Selecting default config 'rk3588-edgeble-neu6a-io.dtb' >> ## Checking hash(es) for config config-1 ... >> fit_config_verify_required_keys: No signature node found: >> FDT_ERR_NOTFOUND >> OK >> firmware: 'atf-1' >> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0 >> Sending CMD16 >> Sending CMD18 >> dwmci_data_transfer: Timeout waiting for data! >> Sending CMD12 >> External data: dst=40000, offset=ad200, size=2e940 >> ## Checking hash(es) for Image atf-1 ... >> fit_image_verify_required_sigs: No signature node found: >> FDT_ERR_NOTFOUND >> sha256 error! >> Bad hash value for 'hash' hash node in 'atf-1' image node >> mmc_load_image_raw_sector: mmc block read error >> spl: mmc boot mode: fs >> SPL: failed to boot from all boot devices >> ### ERROR ### Please RESET the board ### > > As you can see in the logs above there is timeout waiting for data. > > I managed to find the issue and have a workaround that gets me longer > in the boot process, there still seem to be other issue with the rk3588 > startup. > > -------- > U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) > Trying to boot from MMC1 > ## Checking hash(es) for config config-1 ... OK > ## Checking hash(es) for Image atf-1 ... sha256+ OK > ## Checking hash(es) for Image u-boot ... sha256+ OK > ## Checking hash(es) for Image fdt-1 ... sha256+ OK > ## Checking hash(es) for Image atf-2 ... sha256+ OK > ## Checking hash(es) for Image atf-3 ... sha256+ OK > INFO: Preloader serial: 2 > NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang > NOTICE: BL31: Built : 09:59:49, Nov 21 2022 > INFO: spec: 0x1 > INFO: ext 32k is not valid > INFO: ddr: stride-en 4CH > INFO: GICv3 without legacy support detected. > INFO: ARM GICv3 driver initialized in EL3 > INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 > INFO: system boots from cpu-hwid-0 > INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 > INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz > INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz > INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz > INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz > INFO: BL31: Initialising Exception Handling Framework > INFO: BL31: Initializing runtime services > WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK > ERROR: Error initializing runtime service opteed_fast > INFO: BL31: Preparing for EL3 exit to normal world > INFO: Entry point address = 0xa00000 > INFO: SPSR = 0x3c9 > "Synchronous Abort" handler, esr 0x96000000 > elr: 0000000000a23650 lr : 0000000000a24d9c > x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 > x2 : 0000000000000000 x3 : 0000000000b80ff0 > x4 : 0000000000b80ff0 x5 : 0000000000b80e88 > x6 : 0000000000000054 x7 : 0000000000000044 > x8 : 000000000000000a x9 : 0000000000000000 > x10: 0000000000000034 x11: 0000000000000002 > x12: 0000000000001988 x13: 0000000000b7fadc > x14: 0000000000a7e808 x15: 0000000000a7e808 > x16: 0000000000000000 x17: 0000000000000000 > x18: 0000000000b7fe50 x19: 0000000000b7fbe8 > x20: 000000003c14dc00 x21: 000000003c14dc00 > x22: 0000000000a7e808 x23: 0000000000000000 > x24: 0000000000000000 x25: 0000000000000000 > x26: 0000000000000000 x27: 0000000000000000 > x28: 0000000000000000 x29: 0000000000b7fb80 > > Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) > Resetting CPU ... > -------- > > This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e > with some rk3568 patches and this series, see [1]. > > The last 3 commits contains workaround to issue with sdmmc clock. > dwmmc driver set sclk to (uint)-2, my workaround just adds a > fallback to default 400khz clock. > > Next issue is the sync abort, looks it happens when u-boot > tries to set clock rates based on devicetree. this is the > last debug line before the crash. > > clk_set_rate(clk=0000000000b7fba8, rate=1008000000) With the commit at [2] I can now successfully run U-Boot proper. Source of the two main issues to get this series to run have been the scmi clocks. Vendor u-boot first load its scmi driver before trying to set the cpu clocks. We can just remove it and leave setting a faster cpu rate to linux. I also noticed that my sdram size series only detect the first two channels of memory, will respin a v2 of that series to add detection of all 4 channels of memory. [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 Regards, Jonas > > [1] https://github.com/Kwiboo/u-boot-rockchip/compare/060a65e899859dcbf42049a18be20ce7118e7c0e...be21c34067d6e3a2036ce70d571700170c59d270 > > Regards, > Jonas > >> >> Please check this repo, it is on top of the dm/master >> https://github.com/edgeble/u-boot/tree/neu6-v1 >> >> Jagan. >
On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: > > On 2023-01-26 23:16, Jonas Karlman wrote: > > Hi Jagan, > > On 2023-01-26 20:17, Jagan Teki wrote: > >> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: > >>> > >>> On 2023-01-26 19:26, Jagan Teki wrote: > >>>> Hi Simon, > >>>> > >>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > >>>>> > >>>>> Hi Jagan, > >>>>> > >>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > >>>>>> > >>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>> > >>>>>>> Hi Jagan, > >>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: > >>>>>>>> Hi Jonas, > >>>>>>>> > >>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>> > >>>>>>>>> Hi Jagan, > >>>>>>>>> > >>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: > >>>>>>>>>> This series support Rockchip RK3588. All the device tree files are > >>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit > >>>>>>>>>> messages. > >>>>>>>>>> > >>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>>>>>>>>> it is failing to load ATF entry from SPL and hang. > >>>>>>>>>> > >>>>>>>>>> Verified below BL31 versions, > >>>>>>>>>> bl31-v1.15 > >>>>>>>>>> bl31-v1.21 > >>>>>>>>>> bl31-v1.22 > >>>>>>>>>> bl31-v1.23 > >>>>>>>>>> bl31-v1.24 > >>>>>>>>>> bl31-v1.25 > >>>>>>>>>> bl31-v1.26 > >>>>>>>>>> < snip > > > > > As you can see in the logs above there is timeout waiting for data. > > > > I managed to find the issue and have a workaround that gets me longer > > in the boot process, there still seem to be other issue with the rk3588 > > startup. > > > > -------- > > U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) > > Trying to boot from MMC1 > > ## Checking hash(es) for config config-1 ... OK > > ## Checking hash(es) for Image atf-1 ... sha256+ OK > > ## Checking hash(es) for Image u-boot ... sha256+ OK > > ## Checking hash(es) for Image fdt-1 ... sha256+ OK > > ## Checking hash(es) for Image atf-2 ... sha256+ OK > > ## Checking hash(es) for Image atf-3 ... sha256+ OK > > INFO: Preloader serial: 2 > > NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang > > NOTICE: BL31: Built : 09:59:49, Nov 21 2022 > > INFO: spec: 0x1 > > INFO: ext 32k is not valid > > INFO: ddr: stride-en 4CH > > INFO: GICv3 without legacy support detected. > > INFO: ARM GICv3 driver initialized in EL3 > > INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 > > INFO: system boots from cpu-hwid-0 > > INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 > > INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz > > INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz > > INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz > > INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz > > INFO: BL31: Initialising Exception Handling Framework > > INFO: BL31: Initializing runtime services > > WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK > > ERROR: Error initializing runtime service opteed_fast > > INFO: BL31: Preparing for EL3 exit to normal world > > INFO: Entry point address = 0xa00000 > > INFO: SPSR = 0x3c9 > > "Synchronous Abort" handler, esr 0x96000000 > > elr: 0000000000a23650 lr : 0000000000a24d9c > > x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 > > x2 : 0000000000000000 x3 : 0000000000b80ff0 > > x4 : 0000000000b80ff0 x5 : 0000000000b80e88 > > x6 : 0000000000000054 x7 : 0000000000000044 > > x8 : 000000000000000a x9 : 0000000000000000 > > x10: 0000000000000034 x11: 0000000000000002 > > x12: 0000000000001988 x13: 0000000000b7fadc > > x14: 0000000000a7e808 x15: 0000000000a7e808 > > x16: 0000000000000000 x17: 0000000000000000 > > x18: 0000000000b7fe50 x19: 0000000000b7fbe8 > > x20: 000000003c14dc00 x21: 000000003c14dc00 > > x22: 0000000000a7e808 x23: 0000000000000000 > > x24: 0000000000000000 x25: 0000000000000000 > > x26: 0000000000000000 x27: 0000000000000000 > > x28: 0000000000000000 x29: 0000000000b7fb80 > > > > Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) > > Resetting CPU ... > > -------- > > > > This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e > > with some rk3568 patches and this series, see [1]. > > > > The last 3 commits contains workaround to issue with sdmmc clock. > > dwmmc driver set sclk to (uint)-2, my workaround just adds a > > fallback to default 400khz clock. > > > > Next issue is the sync abort, looks it happens when u-boot > > tries to set clock rates based on devicetree. this is the > > last debug line before the crash. > > > > clk_set_rate(clk=0000000000b7fba8, rate=1008000000) > > With the commit at [2] I can now successfully run U-Boot proper. > > Source of the two main issues to get this series to run have been the scmi clocks. > Vendor u-boot first load its scmi driver before trying to set the cpu clocks. > We can just remove it and leave setting a faster cpu rate to linux. > > I also noticed that my sdram size series only detect the first two channels of memory, > will respin a v2 of that series to add detection of all 4 channels of memory. > > [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 Okay. We need to find a way to handle the clock value 400Khz generically via the CLK framework or eMMC can be worth checking as it doesn't involve SCMI and have a working patch set before MW. I did that and was able to detect eMMC in U-Boot proper but got some issues while booting from eMMC [3] [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f Jagan.
On 2023-01-27 14:21, Jagan Teki wrote: > On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >> >> On 2023-01-26 23:16, Jonas Karlman wrote: >>> Hi Jagan, >>> On 2023-01-26 20:17, Jagan Teki wrote: >>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>> >>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>> Hi Simon, >>>>>> >>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>> >>>>>>> Hi Jagan, >>>>>>> >>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>> >>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>> >>>>>>>>> Hi Jagan, >>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>> Hi Jonas, >>>>>>>>>> >>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Jagan, >>>>>>>>>>> >>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>> messages. >>>>>>>>>>>> >>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>> >>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>> > > < snip > > >>> >>> As you can see in the logs above there is timeout waiting for data. >>> >>> I managed to find the issue and have a workaround that gets me longer >>> in the boot process, there still seem to be other issue with the rk3588 >>> startup. >>> >>> -------- >>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>> Trying to boot from MMC1 >>> ## Checking hash(es) for config config-1 ... OK >>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>> INFO: Preloader serial: 2 >>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>> INFO: spec: 0x1 >>> INFO: ext 32k is not valid >>> INFO: ddr: stride-en 4CH >>> INFO: GICv3 without legacy support detected. >>> INFO: ARM GICv3 driver initialized in EL3 >>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>> INFO: system boots from cpu-hwid-0 >>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>> INFO: BL31: Initialising Exception Handling Framework >>> INFO: BL31: Initializing runtime services >>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>> ERROR: Error initializing runtime service opteed_fast >>> INFO: BL31: Preparing for EL3 exit to normal world >>> INFO: Entry point address = 0xa00000 >>> INFO: SPSR = 0x3c9 >>> "Synchronous Abort" handler, esr 0x96000000 >>> elr: 0000000000a23650 lr : 0000000000a24d9c >>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>> x6 : 0000000000000054 x7 : 0000000000000044 >>> x8 : 000000000000000a x9 : 0000000000000000 >>> x10: 0000000000000034 x11: 0000000000000002 >>> x12: 0000000000001988 x13: 0000000000b7fadc >>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>> x16: 0000000000000000 x17: 0000000000000000 >>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>> x22: 0000000000a7e808 x23: 0000000000000000 >>> x24: 0000000000000000 x25: 0000000000000000 >>> x26: 0000000000000000 x27: 0000000000000000 >>> x28: 0000000000000000 x29: 0000000000b7fb80 >>> >>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>> Resetting CPU ... >>> -------- >>> >>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>> with some rk3568 patches and this series, see [1]. >>> >>> The last 3 commits contains workaround to issue with sdmmc clock. >>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>> fallback to default 400khz clock. >>> >>> Next issue is the sync abort, looks it happens when u-boot >>> tries to set clock rates based on devicetree. this is the >>> last debug line before the crash. >>> >>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >> >> With the commit at [2] I can now successfully run U-Boot proper. >> >> Source of the two main issues to get this series to run have been the scmi clocks. >> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >> We can just remove it and leave setting a faster cpu rate to linux. >> >> I also noticed that my sdram size series only detect the first two channels of memory, >> will respin a v2 of that series to add detection of all 4 channels of memory. >> >> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 > > Okay. We need to find a way to handle the clock value 400Khz > generically via the CLK framework or eMMC can be worth checking as it > doesn't involve SCMI and have a working patch set before MW. I did > that and was able to detect eMMC in U-Boot proper but got some issues > while booting from eMMC [3] I have an updated branch at [4] that should support booting from sdmmc and sdhci. Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc and I have included some rk3568 patches and your rk3588 rfc series. I added a few fixup on top of that and a few additional patches, please see commit message for a very brief note on why the change was needed. Feel free to squash fixups and pick commits up to and possible including "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" for a v2 of this series. The remaining sdhci patches needs a little bit more work, I can send a separate series with emmc patches once they are fully ready. The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. I think this is a more appropriate way to work around this issue, instead of patching u-boot spl_fit or sdhci core to use bounce buffers in a very special case. [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 Regards, Jonas > > [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f > > Jagan.
Hi Jonas, Jagan, On 2023/1/26 06:47, Jonas Karlman wrote: > Hi Jagan, > > On 2023-01-25 23:27, Jagan Teki wrote: >> This series support Rockchip RK3588. All the device tree files are >> synced from linux-next with the proper SHA1 mentioned in the commit >> messages. >> >> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >> it is failing to load ATF entry from SPL and hang. >> >> Verified below BL31 versions, >> bl31-v1.15 >> bl31-v1.21 >> bl31-v1.22 >> bl31-v1.23 >> bl31-v1.24 >> bl31-v1.25 >> bl31-v1.26 >> >> Rever-engineered with respect to rockchip u-boot by using the same >> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >> mainline showing the same issue. >> >> Log: >> >> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >> change to F1: 528MHz >> change to F2: 1068MHz >> change to F3: 1560MHz >> change to F0: 2112MHz >> out >> >> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >> Trying to boot from MMC1 >> bl31_entry: atf_entry start >> << hang >> >> >> Any information on BL31 for RK3588 please share. > I had a similar strange booing issue with RK3568 and mainline U-Boot, > turned out to be related to all parts of ATF not being properly loaded > into PMU SRAM. For this issue, could you try to add below property for mmc dts node? "u-boot,spl-fifo-mode" The emmc/sdmmc controller do not have a direct path to the SRAM, so we can't use its internal DMA to do the data transfer. The "fifo-mode" will use CPU to do the data copy instead of the internal DMA. Thanks, - Kever
Hi Kever, On 2023-01-29 10:47, Kever Yang wrote: > Hi Jonas, Jagan, > > On 2023/1/26 06:47, Jonas Karlman wrote: >> Hi Jagan, >> >> On 2023-01-25 23:27, Jagan Teki wrote: >>> This series support Rockchip RK3588. All the device tree files are >>> synced from linux-next with the proper SHA1 mentioned in the commit >>> messages. >>> >>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>> it is failing to load ATF entry from SPL and hang. >>> >>> Verified below BL31 versions, >>> bl31-v1.15 >>> bl31-v1.21 >>> bl31-v1.22 >>> bl31-v1.23 >>> bl31-v1.24 >>> bl31-v1.25 >>> bl31-v1.26 >>> >>> Rever-engineered with respect to rockchip u-boot by using the same >>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>> mainline showing the same issue. >>> >>> Log: >>> >>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>> change to F1: 528MHz >>> change to F2: 1068MHz >>> change to F3: 1560MHz >>> change to F0: 2112MHz >>> out >>> >>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>> Trying to boot from MMC1 >>> bl31_entry: atf_entry start >>> << hang >> >>> >>> Any information on BL31 for RK3588 please share. >> I had a similar strange booing issue with RK3568 and mainline U-Boot, >> turned out to be related to all parts of ATF not being properly loaded >> into PMU SRAM. > > For this issue, could you try to add below property for mmc dts node? > > "u-boot,spl-fifo-mode" > > The emmc/sdmmc controller do not have a direct path to the SRAM, so we > can't use > > its internal DMA to do the data transfer. The "fifo-mode" will use CPU > to do the data > > copy instead of the internal DMA. For sdmmc this worked, but for emmc it did not, trying to use the emmc without SDMA seemed to cause issues reading data in general, did not fully investigate why. I am thinking we could use some sort of mechanism to signal mkimage that we want to keep the parts that should be loaded into SRAM as embedded data instead of external data. That way the FIT can be loaded using DMA into DRAM, and the embedded data will then be memcpy into SRAM using CPU. I quickly tested [0] and this seem to work and we do not need to use the fifo-mode to work around this DMA to SRAM issue. Will work on a proper RFC for such solution. [0] https://github.com/Kwiboo/u-boot-rockchip/commit/551b02a5cd7d28244f44b2e7d7a29196305c26f6 Regards, Jonas > > > Thanks, > > - Kever >
Hi Jonas, On 2023/1/29 17:58, Jonas Karlman wrote: > Hi Kever, > On 2023-01-29 10:47, Kever Yang wrote: >> Hi Jonas, Jagan, >> >> On 2023/1/26 06:47, Jonas Karlman wrote: >>> Hi Jagan, >>> >>> On 2023-01-25 23:27, Jagan Teki wrote: >>>> This series support Rockchip RK3588. All the device tree files are >>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>> messages. >>>> >>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>> it is failing to load ATF entry from SPL and hang. >>>> >>>> Verified below BL31 versions, >>>> bl31-v1.15 >>>> bl31-v1.21 >>>> bl31-v1.22 >>>> bl31-v1.23 >>>> bl31-v1.24 >>>> bl31-v1.25 >>>> bl31-v1.26 >>>> >>>> Rever-engineered with respect to rockchip u-boot by using the same >>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but >>>> mainline showing the same issue. >>>> >>>> Log: >>>> >>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) >>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB >>>> change to F1: 528MHz >>>> change to F2: 1068MHz >>>> change to F3: 1560MHz >>>> change to F0: 2112MHz >>>> out >>>> >>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) >>>> Trying to boot from MMC1 >>>> bl31_entry: atf_entry start >>>> << hang >> >>>> >>>> Any information on BL31 for RK3588 please share. >>> I had a similar strange booing issue with RK3568 and mainline U-Boot, >>> turned out to be related to all parts of ATF not being properly loaded >>> into PMU SRAM. >> For this issue, could you try to add below property for mmc dts node? >> >> "u-boot,spl-fifo-mode" >> >> The emmc/sdmmc controller do not have a direct path to the SRAM, so we >> can't use >> >> its internal DMA to do the data transfer. The "fifo-mode" will use CPU >> to do the data >> >> copy instead of the internal DMA. > For sdmmc this worked, but for emmc it did not, trying to use the emmc without > SDMA seemed to cause issues reading data in general, did not fully investigate why. The sdmmc is using driver rockchip_dw_mmc.c while the emmc is using the driver rockchip_sdhci.c, I think this is the root cause for the "spl-fifo-mode" only only works on sdmmc. Thanks, - Kever > I am thinking we could use some sort of mechanism to signal mkimage that we want to > keep the parts that should be loaded into SRAM as embedded data instead of external data. > That way the FIT can be loaded using DMA into DRAM, and the embedded data will then > be memcpy into SRAM using CPU. > > I quickly tested [0] and this seem to work and we do not need to use the fifo-mode > to work around this DMA to SRAM issue. Will work on a proper RFC for such solution. > > [0] https://github.com/Kwiboo/u-boot-rockchip/commit/551b02a5cd7d28244f44b2e7d7a29196305c26f6 > > Regards, > Jonas > >> >> Thanks, >> >> - Kever >>
Hi Kever, On Mon, 30 Jan 2023 at 06:25, Kever Yang <kever.yang@rock-chips.com> wrote: > > Hi Jonas, > > > On 2023/1/29 17:58, Jonas Karlman wrote: > > Hi Kever, > > On 2023-01-29 10:47, Kever Yang wrote: > >> Hi Jonas, Jagan, > >> > >> On 2023/1/26 06:47, Jonas Karlman wrote: > >>> Hi Jagan, > >>> > >>> On 2023-01-25 23:27, Jagan Teki wrote: > >>>> This series support Rockchip RK3588. All the device tree files are > >>>> synced from linux-next with the proper SHA1 mentioned in the commit > >>>> messages. > >>>> > >>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>>> it is failing to load ATF entry from SPL and hang. > >>>> > >>>> Verified below BL31 versions, > >>>> bl31-v1.15 > >>>> bl31-v1.21 > >>>> bl31-v1.22 > >>>> bl31-v1.23 > >>>> bl31-v1.24 > >>>> bl31-v1.25 > >>>> bl31-v1.26 > >>>> > >>>> Rever-engineered with respect to rockchip u-boot by using the same > >>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but > >>>> mainline showing the same issue. > >>>> > >>>> Log: > >>>> > >>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530) > >>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB > >>>> change to F1: 528MHz > >>>> change to F2: 1068MHz > >>>> change to F3: 1560MHz > >>>> change to F0: 2112MHz > >>>> out > >>>> > >>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530) > >>>> Trying to boot from MMC1 > >>>> bl31_entry: atf_entry start > >>>> << hang >> > >>>> > >>>> Any information on BL31 for RK3588 please share. > >>> I had a similar strange booing issue with RK3568 and mainline U-Boot, > >>> turned out to be related to all parts of ATF not being properly loaded > >>> into PMU SRAM. > >> For this issue, could you try to add below property for mmc dts node? > >> > >> "u-boot,spl-fifo-mode" > >> > >> The emmc/sdmmc controller do not have a direct path to the SRAM, so we > >> can't use > >> > >> its internal DMA to do the data transfer. The "fifo-mode" will use CPU > >> to do the data > >> > >> copy instead of the internal DMA. > > For sdmmc this worked, but for emmc it did not, trying to use the emmc without > > SDMA seemed to cause issues reading data in general, did not fully investigate why. > > > The sdmmc is using driver rockchip_dw_mmc.c while the emmc is using the > driver rockchip_sdhci.c, > > I think this is the root cause for the "spl-fifo-mode" only only works > on sdmmc. Apart from this, the 400KHZ clock workaround is required to boot the SDMMC. I feel this hack of 400KHZ cannot be proper, we need to handle that via CLK drivers to get the rate. Jagan.
On 1/29/23 11:04, Jonas Karlman wrote: > On 2023-01-27 14:21, Jagan Teki wrote: >> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>> >>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>> Hi Jagan, >>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>> >>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>> Hi Simon, >>>>>>> >>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>> >>>>>>>> Hi Jagan, >>>>>>>> >>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>> >>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>> >>>>>>>>>> Hi Jagan, >>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>> Hi Jonas, >>>>>>>>>>> >>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>> >>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>> messages. >>>>>>>>>>>>> >>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>> >>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>> >> >> < snip > >> >>>> >>>> As you can see in the logs above there is timeout waiting for data. >>>> >>>> I managed to find the issue and have a workaround that gets me longer >>>> in the boot process, there still seem to be other issue with the rk3588 >>>> startup. >>>> >>>> -------- >>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>> Trying to boot from MMC1 >>>> ## Checking hash(es) for config config-1 ... OK >>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>> INFO: Preloader serial: 2 >>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>> INFO: spec: 0x1 >>>> INFO: ext 32k is not valid >>>> INFO: ddr: stride-en 4CH >>>> INFO: GICv3 without legacy support detected. >>>> INFO: ARM GICv3 driver initialized in EL3 >>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>> INFO: system boots from cpu-hwid-0 >>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>> INFO: BL31: Initialising Exception Handling Framework >>>> INFO: BL31: Initializing runtime services >>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>> ERROR: Error initializing runtime service opteed_fast >>>> INFO: BL31: Preparing for EL3 exit to normal world >>>> INFO: Entry point address = 0xa00000 >>>> INFO: SPSR = 0x3c9 >>>> "Synchronous Abort" handler, esr 0x96000000 >>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>> x8 : 000000000000000a x9 : 0000000000000000 >>>> x10: 0000000000000034 x11: 0000000000000002 >>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>> x16: 0000000000000000 x17: 0000000000000000 >>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>> x24: 0000000000000000 x25: 0000000000000000 >>>> x26: 0000000000000000 x27: 0000000000000000 >>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>> >>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>> Resetting CPU ... >>>> -------- >>>> >>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>> with some rk3568 patches and this series, see [1]. >>>> >>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>> fallback to default 400khz clock. >>>> >>>> Next issue is the sync abort, looks it happens when u-boot >>>> tries to set clock rates based on devicetree. this is the >>>> last debug line before the crash. >>>> >>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>> >>> With the commit at [2] I can now successfully run U-Boot proper. >>> >>> Source of the two main issues to get this series to run have been the scmi clocks. >>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>> We can just remove it and leave setting a faster cpu rate to linux. >>> >>> I also noticed that my sdram size series only detect the first two channels of memory, >>> will respin a v2 of that series to add detection of all 4 channels of memory. >>> >>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >> >> Okay. We need to find a way to handle the clock value 400Khz >> generically via the CLK framework or eMMC can be worth checking as it >> doesn't involve SCMI and have a working patch set before MW. I did >> that and was able to detect eMMC in U-Boot proper but got some issues >> while booting from eMMC [3] > > I have an updated branch at [4] that should support booting from sdmmc and sdhci. > Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. > > This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc > and I have included some rk3568 patches and your rk3588 rfc series. > I added a few fixup on top of that and a few additional patches, please see commit message > for a very brief note on why the change was needed. > Feel free to squash fixups and pick commits up to and possible including > "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" > for a v2 of this series. > > The remaining sdhci patches needs a little bit more work, > I can send a separate series with emmc patches once they are fully ready. > > The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. > This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. > I think this is a more appropriate way to work around this issue, instead of patching > u-boot spl_fit or sdhci core to use bounce buffers in a very special case. > > [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 Hello Jagan, Jonas, I wanted to chip into this discussion, to ask whether you did anything more on the SD clock matter ? I am currently using your workaround that creates dummy clocks in the DT and then the SD node just uses those, and it works, in the SPL, if and only if the bootrom also loads the SPL from SD. I tried to have the SPL inside the SPI flash, or load it to DDR using the rockusb protocol, and then, initializing the SD-Card from SPL fails utterly, namely, it cannot communicate with the card. I did some changes to have the pinctrl working at SPL level, which appears to be fine, but I would like to see what can be done about the clocks. Having the cru and all the required drivers at SPL level is the way to go ? The SPL should run before SCMI is required so it should be able to change all the clocks at the clock controller level ? Thanks, Eugen > > Regards, > Jonas > >> >> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >> >> Jagan. >
Hi Eugen, On 2023-03-08 09:57, Eugen Hristev wrote: > On 1/29/23 11:04, Jonas Karlman wrote: >> On 2023-01-27 14:21, Jagan Teki wrote: >>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>> >>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>> Hi Jagan, >>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>> >>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>> Hi Simon, >>>>>>>> >>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>> >>>>>>>>> Hi Jagan, >>>>>>>>> >>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>> >>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Jagan, >>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>> >>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>> messages. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>> >>> >>> < snip > >>> >>>>> >>>>> As you can see in the logs above there is timeout waiting for data. >>>>> >>>>> I managed to find the issue and have a workaround that gets me longer >>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>> startup. >>>>> >>>>> -------- >>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>> Trying to boot from MMC1 >>>>> ## Checking hash(es) for config config-1 ... OK >>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>> INFO: Preloader serial: 2 >>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>> INFO: spec: 0x1 >>>>> INFO: ext 32k is not valid >>>>> INFO: ddr: stride-en 4CH >>>>> INFO: GICv3 without legacy support detected. >>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>> INFO: system boots from cpu-hwid-0 >>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>> INFO: BL31: Initialising Exception Handling Framework >>>>> INFO: BL31: Initializing runtime services >>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>> ERROR: Error initializing runtime service opteed_fast >>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>> INFO: Entry point address = 0xa00000 >>>>> INFO: SPSR = 0x3c9 >>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>> >>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>> Resetting CPU ... >>>>> -------- >>>>> >>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>> with some rk3568 patches and this series, see [1]. >>>>> >>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>> fallback to default 400khz clock. >>>>> >>>>> Next issue is the sync abort, looks it happens when u-boot >>>>> tries to set clock rates based on devicetree. this is the >>>>> last debug line before the crash. >>>>> >>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>> >>>> With the commit at [2] I can now successfully run U-Boot proper. >>>> >>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>> We can just remove it and leave setting a faster cpu rate to linux. >>>> >>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>> >>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>> >>> Okay. We need to find a way to handle the clock value 400Khz >>> generically via the CLK framework or eMMC can be worth checking as it >>> doesn't involve SCMI and have a working patch set before MW. I did >>> that and was able to detect eMMC in U-Boot proper but got some issues >>> while booting from eMMC [3] >> >> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >> >> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >> and I have included some rk3568 patches and your rk3588 rfc series. >> I added a few fixup on top of that and a few additional patches, please see commit message >> for a very brief note on why the change was needed. >> Feel free to squash fixups and pick commits up to and possible including >> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >> for a v2 of this series. >> >> The remaining sdhci patches needs a little bit more work, >> I can send a separate series with emmc patches once they are fully ready. >> >> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >> I think this is a more appropriate way to work around this issue, instead of patching >> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >> >> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >> > Hello Jagan, Jonas, > > I wanted to chip into this discussion, to ask whether you did anything > more on the SD clock matter ? I have been busy this past week but have now had time to take a new look at the sdmmc issue, along with completing some other fixes. > > I am currently using your workaround that creates dummy clocks in the DT > and then the SD node just uses those, and it works, in the SPL, if and > only if the bootrom also loads the SPL from SD. > I tried to have the SPL inside the SPI flash, or load it to DDR using > the rockusb protocol, and then, initializing the SD-Card from SPL fails > utterly, namely, it cannot communicate with the card. > I did some changes to have the pinctrl working at SPL level, which > appears to be fine, but I would like to see what can be done about the > clocks. > Having the cru and all the required drivers at SPL level is the way to > go ? The SPL should run before SCMI is required so it should be able to > change all the clocks at the clock controller level ? I fully agree that we should have some sort of scmi clk driver so that we can control the sdmmc clk in both SPL and U-Boot proper. As you have noticed the current workaround only works because bootrom leave the clocks in a working state after it has loaded TPL/SPL from the sdmmc device. When TPL/SPL is loaded from any other source it is not be possible to read from the sdmmc device in u-boot. After having played around with the scmi agent driver and being inspired by the dummy scmi clk driver in vendor u-boot I have managed to create something that could work. See top three commits from [5] for a working proof-of-concept. What I did was to enable the scmi agent driver for use in u-boot proper and keeping it disabled in SPL build. Then added a new scmi clk driver that is enabled in SPL build, the rk3588 clk driver hooks the new clk driver to the scmi_clk ofnode. I only implemented the get/set rate ops for the two sdmmc clocks. With this both SPL and U-Boot proper should be able to configure the sdmmc clocks. The initial fixes commits in that branch should hit the list soon. Will send the sdmmc related commits once I have had some time to do more testing. [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes Regards, Jonas > > Thanks, > Eugen > >> >> Regards, >> Jonas >> >>> >>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>> >>> Jagan. >> >
On 3/13/23 00:34, Jonas Karlman wrote: > Hi Eugen, > > On 2023-03-08 09:57, Eugen Hristev wrote: >> On 1/29/23 11:04, Jonas Karlman wrote: >>> On 2023-01-27 14:21, Jagan Teki wrote: >>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>> >>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>> Hi Jagan, >>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>> >>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>> Hi Simon, >>>>>>>>> >>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>> >>>>>>>>>> Hi Jagan, >>>>>>>>>> >>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>> >>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>> >>>> >>>> < snip > >>>> >>>>>> >>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>> >>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>> startup. >>>>>> >>>>>> -------- >>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>> Trying to boot from MMC1 >>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>> INFO: Preloader serial: 2 >>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>> INFO: spec: 0x1 >>>>>> INFO: ext 32k is not valid >>>>>> INFO: ddr: stride-en 4CH >>>>>> INFO: GICv3 without legacy support detected. >>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>> INFO: system boots from cpu-hwid-0 >>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>> INFO: BL31: Initializing runtime services >>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>> INFO: Entry point address = 0xa00000 >>>>>> INFO: SPSR = 0x3c9 >>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>> >>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>> Resetting CPU ... >>>>>> -------- >>>>>> >>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>> with some rk3568 patches and this series, see [1]. >>>>>> >>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>> fallback to default 400khz clock. >>>>>> >>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>> tries to set clock rates based on devicetree. this is the >>>>>> last debug line before the crash. >>>>>> >>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>> >>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>> >>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>> >>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>> >>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>> >>>> Okay. We need to find a way to handle the clock value 400Khz >>>> generically via the CLK framework or eMMC can be worth checking as it >>>> doesn't involve SCMI and have a working patch set before MW. I did >>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>> while booting from eMMC [3] >>> >>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>> >>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>> and I have included some rk3568 patches and your rk3588 rfc series. >>> I added a few fixup on top of that and a few additional patches, please see commit message >>> for a very brief note on why the change was needed. >>> Feel free to squash fixups and pick commits up to and possible including >>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>> for a v2 of this series. >>> >>> The remaining sdhci patches needs a little bit more work, >>> I can send a separate series with emmc patches once they are fully ready. >>> >>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>> I think this is a more appropriate way to work around this issue, instead of patching >>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>> >>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>> >> Hello Jagan, Jonas, >> >> I wanted to chip into this discussion, to ask whether you did anything >> more on the SD clock matter ? > > I have been busy this past week but have now had time to take a new look > at the sdmmc issue, along with completing some other fixes. > >> >> I am currently using your workaround that creates dummy clocks in the DT >> and then the SD node just uses those, and it works, in the SPL, if and >> only if the bootrom also loads the SPL from SD. >> I tried to have the SPL inside the SPI flash, or load it to DDR using >> the rockusb protocol, and then, initializing the SD-Card from SPL fails >> utterly, namely, it cannot communicate with the card. >> I did some changes to have the pinctrl working at SPL level, which >> appears to be fine, but I would like to see what can be done about the >> clocks. >> Having the cru and all the required drivers at SPL level is the way to >> go ? The SPL should run before SCMI is required so it should be able to >> change all the clocks at the clock controller level ? > > I fully agree that we should have some sort of scmi clk driver so that > we can control the sdmmc clk in both SPL and U-Boot proper. > > As you have noticed the current workaround only works because bootrom > leave the clocks in a working state after it has loaded TPL/SPL from > the sdmmc device. When TPL/SPL is loaded from any other source it is > not be possible to read from the sdmmc device in u-boot. > > After having played around with the scmi agent driver and being inspired > by the dummy scmi clk driver in vendor u-boot I have managed to create > something that could work. See top three commits from [5] for a working > proof-of-concept. > > What I did was to enable the scmi agent driver for use in u-boot proper > and keeping it disabled in SPL build. Then added a new scmi clk driver > that is enabled in SPL build, the rk3588 clk driver hooks the new clk > driver to the scmi_clk ofnode. I only implemented the get/set rate ops > for the two sdmmc clocks. With this both SPL and U-Boot proper should > be able to configure the sdmmc clocks. > > The initial fixes commits in that branch should hit the list soon. > Will send the sdmmc related commits once I have had some time to do > more testing. > > [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes Hi Jonas, I managed to get it working in SPL as well, with a similar code inspired from the vendor tree. About the solution, I am not sure whether it should be called a scmi clock, and whether it should be part of clk_rk3588 Honestly by looking at the DT node, I find the description of the clock tree wrong to begin with. The SD should not have the 2 clocks tied to the scmi agent, because this is true only if it's running in normal world with a secure world agent that will act as a middle man. So if we are running at a higher EL, as in the SPL case, the clocks should be tied directly to the CRU. So the node itself is described bad IMO. Anyway, if the node is set in stone from Linux, there isn;t much we can do. I am also thinking whether we should have the SCMI enabled in SPL as well, but the agent/clock should just access the CRU directly, meaning to do something at the agent level : #ifdef CONFIG_SPL_BUILD -> access CRU #else -> send SCMI message #endif Does this make more sense ? Or have some kind of wrapper driver that would act as a dispatcher depending on the SPL/proper build ? Meanwhile, I will test your patches to see how they work on my setup, I also have some things in progress including pinctrl in SPL for the rock5b. Thanks for your detailed description, Eugen > > Regards, > Jonas > >> >> Thanks, >> Eugen >> >>> >>> Regards, >>> Jonas >>> >>>> >>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>> >>>> Jagan. >>> >> >
On 2023-03-13 09:42, Eugen Hristev wrote: > On 3/13/23 00:34, Jonas Karlman wrote: >> Hi Eugen, >> >> On 2023-03-08 09:57, Eugen Hristev wrote: >>> On 1/29/23 11:04, Jonas Karlman wrote: >>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>> >>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>> Hi Jagan, >>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>> >>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>> Hi Simon, >>>>>>>>>> >>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Jagan, >>>>>>>>>>> >>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>> >>>>> >>>>> < snip > >>>>> >>>>>>> >>>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>>> >>>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>>> startup. >>>>>>> >>>>>>> -------- >>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>> Trying to boot from MMC1 >>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>> INFO: Preloader serial: 2 >>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>> INFO: spec: 0x1 >>>>>>> INFO: ext 32k is not valid >>>>>>> INFO: ddr: stride-en 4CH >>>>>>> INFO: GICv3 without legacy support detected. >>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>> INFO: BL31: Initializing runtime services >>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>> INFO: Entry point address = 0xa00000 >>>>>>> INFO: SPSR = 0x3c9 >>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>> >>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>> Resetting CPU ... >>>>>>> -------- >>>>>>> >>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>> >>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>> fallback to default 400khz clock. >>>>>>> >>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>> last debug line before the crash. >>>>>>> >>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>> >>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>> >>>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>> >>>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>>> >>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>> >>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>> generically via the CLK framework or eMMC can be worth checking as it >>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>>> while booting from eMMC [3] >>>> >>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>> >>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>> I added a few fixup on top of that and a few additional patches, please see commit message >>>> for a very brief note on why the change was needed. >>>> Feel free to squash fixups and pick commits up to and possible including >>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>>> for a v2 of this series. >>>> >>>> The remaining sdhci patches needs a little bit more work, >>>> I can send a separate series with emmc patches once they are fully ready. >>>> >>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>>> I think this is a more appropriate way to work around this issue, instead of patching >>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>>> >>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>> >>> Hello Jagan, Jonas, >>> >>> I wanted to chip into this discussion, to ask whether you did anything >>> more on the SD clock matter ? >> >> I have been busy this past week but have now had time to take a new look >> at the sdmmc issue, along with completing some other fixes. >> >>> >>> I am currently using your workaround that creates dummy clocks in the DT >>> and then the SD node just uses those, and it works, in the SPL, if and >>> only if the bootrom also loads the SPL from SD. >>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>> the rockusb protocol, and then, initializing the SD-Card from SPL fails >>> utterly, namely, it cannot communicate with the card. >>> I did some changes to have the pinctrl working at SPL level, which >>> appears to be fine, but I would like to see what can be done about the >>> clocks. >>> Having the cru and all the required drivers at SPL level is the way to >>> go ? The SPL should run before SCMI is required so it should be able to >>> change all the clocks at the clock controller level ? >> >> I fully agree that we should have some sort of scmi clk driver so that >> we can control the sdmmc clk in both SPL and U-Boot proper. >> >> As you have noticed the current workaround only works because bootrom >> leave the clocks in a working state after it has loaded TPL/SPL from >> the sdmmc device. When TPL/SPL is loaded from any other source it is >> not be possible to read from the sdmmc device in u-boot. >> >> After having played around with the scmi agent driver and being inspired >> by the dummy scmi clk driver in vendor u-boot I have managed to create >> something that could work. See top three commits from [5] for a working >> proof-of-concept. >> >> What I did was to enable the scmi agent driver for use in u-boot proper >> and keeping it disabled in SPL build. Then added a new scmi clk driver >> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >> for the two sdmmc clocks. With this both SPL and U-Boot proper should >> be able to configure the sdmmc clocks. >> >> The initial fixes commits in that branch should hit the list soon. >> Will send the sdmmc related commits once I have had some time to do >> more testing. >> >> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >> > Hi Jonas, > > I managed to get it working in SPL as well, with a similar code inspired > from the vendor tree. > About the solution, I am not sure whether it should be called a scmi > clock, and whether it should be part of clk_rk3588 > Honestly by looking at the DT node, I find the description of the clock > tree wrong to begin with. > The SD should not have the 2 clocks tied to the scmi agent, because this > is true only if it's running in normal world with a secure world agent > that will act as a middle man. So if we are running at a higher EL, as > in the SPL case, the clocks should be tied directly to the CRU. > So the node itself is described bad IMO. > > Anyway, if the node is set in stone from Linux, there isn;t much we can do. > I am also thinking whether we should have the SCMI enabled in SPL as > well, but the agent/clock should just access the CRU directly, meaning > to do something at the agent level : > #ifdef CONFIG_SPL_BUILD > -> access CRU > #else > -> send SCMI message > #endif > Does this make more sense ? > > Or have some kind of wrapper driver that would act as a dispatcher > depending on the SPL/proper build ? Looks like the sdmmc clock is controlled by the securecru registers and I expect that this can only be configured in secure world, not sure how this could have been modeled differently. With the approach that I took I think the normal clk framework will behave as such dispatcher and clock gets tied to cru in SPL. #ifdef CONFIG_SPL_BUILD -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs #else -> bind protocol@14 to scmi clk driver -> send SCMI message #endif Naming and placement of the SPL securecru driver could be improved. Not sure if any other soc beside rk3588 will need this at this moment. > > > Meanwhile, I will test your patches to see how they work on my setup, I > also have some things in progress including pinctrl in SPL for the rock5b. > Thanks for your detailed description, Thanks for the hint at pinctrl, I made some updates to [5] and could verify that sdmmc works when booting from emmc thanks to your pinctrl commit. Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe and build a spi firmware image. There is an issue with non-DMA access in sdhci that requires the HACK commit for proper loading of atf, was hoping to disable SDMA in SPL like what is done using fifo-mode for sdmmc. But only first sector is read successfully then it fails, see below. Will take a closer look before I post eMMC series. Trying to boot from MMC2 1 - 0 'mmc@fe2c0000' - 1 'mmc@fe2e0000' - found blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 clock is disabled (0Hz) clock is enabled (400000Hz) clk_set_rate(clk=5000d8, rate=400000) size=200, ptr=3b8, limit=100000: 5001b8 clock is enabled (25000000Hz) clk_set_rate(clk=5000d8, rate=25000000) clk_set_rate(clk=5000d8, rate=25000000) clock is enabled (52000000Hz) clk_set_rate(clk=5000d8, rate=52000000) spl: mmc boot mode: raw blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 hdr read sector 4000, count=1 Found FIT size=a00, ptr=dc0, limit=100000: aligned to 5003c0 blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 mmc_load_image_raw_sector: mmc block read error spl: mmc boot mode: fs Trying to boot from MMC1 [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc Regards, Jonas > Eugen > >> >> Regards, >> Jonas >> >>> >>> Thanks, >>> Eugen >>> >>>> >>>> Regards, >>>> Jonas >>>> >>>>> >>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>> >>>>> Jagan. >>>> >>> >> >
On 3/13/23 12:00, Jonas Karlman wrote: > On 2023-03-13 09:42, Eugen Hristev wrote: >> On 3/13/23 00:34, Jonas Karlman wrote: >>> Hi Eugen, >>> >>> On 2023-03-08 09:57, Eugen Hristev wrote: >>>> On 1/29/23 11:04, Jonas Karlman wrote: >>>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>> >>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>>> Hi Jagan, >>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>> >>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>>> Hi Simon, >>>>>>>>>>> >>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>>> >>>>>> >>>>>> < snip > >>>>>> >>>>>>>> >>>>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>>>> >>>>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>>>> startup. >>>>>>>> >>>>>>>> -------- >>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>>> Trying to boot from MMC1 >>>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>>> INFO: Preloader serial: 2 >>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>>> INFO: spec: 0x1 >>>>>>>> INFO: ext 32k is not valid >>>>>>>> INFO: ddr: stride-en 4CH >>>>>>>> INFO: GICv3 without legacy support detected. >>>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>>> INFO: BL31: Initializing runtime services >>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>>> INFO: Entry point address = 0xa00000 >>>>>>>> INFO: SPSR = 0x3c9 >>>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>>> >>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>>> Resetting CPU ... >>>>>>>> -------- >>>>>>>> >>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>>> >>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>>> fallback to default 400khz clock. >>>>>>>> >>>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>>> last debug line before the crash. >>>>>>>> >>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>>> >>>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>>> >>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>>> >>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>>>> >>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>>> >>>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>>> generically via the CLK framework or eMMC can be worth checking as it >>>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>>>> while booting from eMMC [3] >>>>> >>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>>> >>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>>> I added a few fixup on top of that and a few additional patches, please see commit message >>>>> for a very brief note on why the change was needed. >>>>> Feel free to squash fixups and pick commits up to and possible including >>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>>>> for a v2 of this series. >>>>> >>>>> The remaining sdhci patches needs a little bit more work, >>>>> I can send a separate series with emmc patches once they are fully ready. >>>>> >>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>>>> I think this is a more appropriate way to work around this issue, instead of patching >>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>>>> >>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>>> >>>> Hello Jagan, Jonas, >>>> >>>> I wanted to chip into this discussion, to ask whether you did anything >>>> more on the SD clock matter ? >>> >>> I have been busy this past week but have now had time to take a new look >>> at the sdmmc issue, along with completing some other fixes. >>> >>>> >>>> I am currently using your workaround that creates dummy clocks in the DT >>>> and then the SD node just uses those, and it works, in the SPL, if and >>>> only if the bootrom also loads the SPL from SD. >>>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails >>>> utterly, namely, it cannot communicate with the card. >>>> I did some changes to have the pinctrl working at SPL level, which >>>> appears to be fine, but I would like to see what can be done about the >>>> clocks. >>>> Having the cru and all the required drivers at SPL level is the way to >>>> go ? The SPL should run before SCMI is required so it should be able to >>>> change all the clocks at the clock controller level ? >>> >>> I fully agree that we should have some sort of scmi clk driver so that >>> we can control the sdmmc clk in both SPL and U-Boot proper. >>> >>> As you have noticed the current workaround only works because bootrom >>> leave the clocks in a working state after it has loaded TPL/SPL from >>> the sdmmc device. When TPL/SPL is loaded from any other source it is >>> not be possible to read from the sdmmc device in u-boot. >>> >>> After having played around with the scmi agent driver and being inspired >>> by the dummy scmi clk driver in vendor u-boot I have managed to create >>> something that could work. See top three commits from [5] for a working >>> proof-of-concept. >>> >>> What I did was to enable the scmi agent driver for use in u-boot proper >>> and keeping it disabled in SPL build. Then added a new scmi clk driver >>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >>> for the two sdmmc clocks. With this both SPL and U-Boot proper should >>> be able to configure the sdmmc clocks. >>> >>> The initial fixes commits in that branch should hit the list soon. >>> Will send the sdmmc related commits once I have had some time to do >>> more testing. >>> >>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >>> >> Hi Jonas, >> >> I managed to get it working in SPL as well, with a similar code inspired >> from the vendor tree. >> About the solution, I am not sure whether it should be called a scmi >> clock, and whether it should be part of clk_rk3588 >> Honestly by looking at the DT node, I find the description of the clock >> tree wrong to begin with. >> The SD should not have the 2 clocks tied to the scmi agent, because this >> is true only if it's running in normal world with a secure world agent >> that will act as a middle man. So if we are running at a higher EL, as >> in the SPL case, the clocks should be tied directly to the CRU. >> So the node itself is described bad IMO. >> >> Anyway, if the node is set in stone from Linux, there isn;t much we can do. >> I am also thinking whether we should have the SCMI enabled in SPL as >> well, but the agent/clock should just access the CRU directly, meaning >> to do something at the agent level : >> #ifdef CONFIG_SPL_BUILD >> -> access CRU >> #else >> -> send SCMI message >> #endif >> Does this make more sense ? >> >> Or have some kind of wrapper driver that would act as a dispatcher >> depending on the SPL/proper build ? > > Looks like the sdmmc clock is controlled by the securecru registers and I > expect that this can only be configured in secure world, not sure how this > could have been modeled differently. Since it can work with a clock given by either scmi or securecru, I would expect all clocks to be in the list, and the driver could start the needed clock as per the EL level it's running into :) e.g. clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi HCLK_SD> , <&cru basic other clocks..> Just my vision of how it could be modeled > > With the approach that I took I think the normal clk framework will behave > as such dispatcher and clock gets tied to cru in SPL. > > #ifdef CONFIG_SPL_BUILD > -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs > #else > -> bind protocol@14 to scmi clk driver -> send SCMI message > #endif This looks right, just that we would have to bring a lot of bloat to SPL , firmware subsystem and things we may not want. > > Naming and placement of the SPL securecru driver could be improved. > Not sure if any other soc beside rk3588 will need this at this moment. securecru sounds better for me, I don't think we can have a scmi clock fake driver like in vendor uboot, it has nothing 'scmi' about it. > >> >> >> Meanwhile, I will test your patches to see how they work on my setup, I >> also have some things in progress including pinctrl in SPL for the rock5b. >> Thanks for your detailed description, > > Thanks for the hint at pinctrl, I made some updates to [5] and could verify > that sdmmc works when booting from emmc thanks to your pinctrl commit. > Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe > and build a spi firmware image. > > There is an issue with non-DMA access in sdhci that requires the HACK commit > for proper loading of atf, was hoping to disable SDMA in SPL like what is done > using fifo-mode for sdmmc. But only first sector is read successfully then it > fails, see below. Will take a closer look before I post eMMC series. In your patches, I see this : https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 Perhaps the fifo-mode is intended for the sdhci(emmc) and not for sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, but the sdmmc can ? one possible workaround is to have DMA to DRAM and then relocate it to SRAM using the CPU. Having DMA disabled for the whole IP may have downsides, but this is U-boot, we don't expect to have anything else to do with the CPU while the DMA master works its magic, and the CPU should be faster. I tested your patches together with the series I sent now ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to boot correctly from SD-Card Once you send them to the ML I can retest them. Thanks ! > > Trying to boot from MMC2 > 1 > - 0 'mmc@fe2c0000' > - 1 'mmc@fe2e0000' > - found > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > clock is disabled (0Hz) > clock is enabled (400000Hz) > clk_set_rate(clk=5000d8, rate=400000) > size=200, ptr=3b8, limit=100000: 5001b8 > clock is enabled (25000000Hz) > clk_set_rate(clk=5000d8, rate=25000000) > clk_set_rate(clk=5000d8, rate=25000000) > clock is enabled (52000000Hz) > clk_set_rate(clk=5000d8, rate=52000000) > spl: mmc boot mode: raw > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > hdr read sector 4000, count=1 > Found FIT > size=a00, ptr=dc0, limit=100000: aligned to 5003c0 > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 > mmc_load_image_raw_sector: mmc block read error > spl: mmc boot mode: fs > Trying to boot from MMC1 > > [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc > > Regards, > Jonas > > >> Eugen >> >>> >>> Regards, >>> Jonas >>> >>>> >>>> Thanks, >>>> Eugen >>>> >>>>> >>>>> Regards, >>>>> Jonas >>>>> >>>>>> >>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>>> >>>>>> Jagan. >>>>> >>>> >>> >> >
On 3/13/23 16:21, Eugen Hristev wrote: > On 3/13/23 12:00, Jonas Karlman wrote: >> On 2023-03-13 09:42, Eugen Hristev wrote: >>> On 3/13/23 00:34, Jonas Karlman wrote: >>>> Hi Eugen, >>>> >>>> On 2023-03-08 09:57, Eugen Hristev wrote: >>>>> On 1/29/23 11:04, Jonas Karlman wrote: >>>>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>> >>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>>>> Hi Jagan, >>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>>>> Hi Simon, >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman >>>>>>>>>>>>>> <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman >>>>>>>>>>>>>>>> <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device >>>>>>>>>>>>>>>>>> tree files are >>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned >>>>>>>>>>>>>>>>>> in the commit >>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible >>>>>>>>>>>>>>>>>> with U-Boot so >>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>> < snip > >>>>>>> >>>>>>>>> >>>>>>>>> As you can see in the logs above there is timeout waiting for >>>>>>>>> data. >>>>>>>>> >>>>>>>>> I managed to find the issue and have a workaround that gets me >>>>>>>>> longer >>>>>>>>> in the boot process, there still seem to be other issue with >>>>>>>>> the rk3588 >>>>>>>>> startup. >>>>>>>>> >>>>>>>>> -------- >>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>>>> Trying to boot from MMC1 >>>>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>>>> INFO: Preloader serial: 2 >>>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>>>> INFO: spec: 0x1 >>>>>>>>> INFO: ext 32k is not valid >>>>>>>>> INFO: ddr: stride-en 4CH >>>>>>>>> INFO: GICv3 without legacy support detected. >>>>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>>>> INFO: BL31: Initializing runtime services >>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device >>>>>>>>> without OPTEE initialization. SMC`s destined for OPTEE will >>>>>>>>> return SMC_UNK >>>>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>>>> INFO: Entry point address = 0xa00000 >>>>>>>>> INFO: SPSR = 0x3c9 >>>>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>>>> >>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>>>> Resetting CPU ... >>>>>>>>> -------- >>>>>>>>> >>>>>>>>> This was running on top of u-boot-dm/master >>>>>>>>> 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>>>> >>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>>>> fallback to default 400khz clock. >>>>>>>>> >>>>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>>>> last debug line before the crash. >>>>>>>>> >>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>>>> >>>>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>>>> >>>>>>>> Source of the two main issues to get this series to run have >>>>>>>> been the scmi clocks. >>>>>>>> Vendor u-boot first load its scmi driver before trying to set >>>>>>>> the cpu clocks. >>>>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>>>> >>>>>>>> I also noticed that my sdram size series only detect the first >>>>>>>> two channels of memory, >>>>>>>> will respin a v2 of that series to add detection of all 4 >>>>>>>> channels of memory. >>>>>>>> >>>>>>>> [2] >>>>>>>> https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>>>> >>>>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>>>> generically via the CLK framework or eMMC can be worth checking >>>>>>> as it >>>>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>>>> that and was able to detect eMMC in U-Boot proper but got some >>>>>>> issues >>>>>>> while booting from eMMC [3] >>>>>> >>>>>> I have an updated branch at [4] that should support booting from >>>>>> sdmmc and sdhci. >>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>>>> >>>>>> This branch is based on u-boot/master >>>>>> f147aa80f52989c7455022ca1ab959e8545feccc >>>>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>>>> I added a few fixup on top of that and a few additional patches, >>>>>> please see commit message >>>>>> for a very brief note on why the change was needed. >>>>>> Feel free to squash fixups and pick commits up to and possible >>>>>> including >>>>>> "board: rockchip: Sync evb-rk3568_defconfig with >>>>>> evb-rk3588_defconfig" >>>>>> for a v2 of this series. >>>>>> >>>>>> The remaining sdhci patches needs a little bit more work, >>>>>> I can send a separate series with emmc patches once they are fully >>>>>> ready. >>>>>> >>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 >>>>>> embedded in the FIT. >>>>>> This ensures we do not run into problems trying to use dma from >>>>>> emmc into secure pmu sram. >>>>>> I think this is a more appropriate way to work around this issue, >>>>>> instead of patching >>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very >>>>>> special case. >>>>>> >>>>>> [4] >>>>>> https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>>>> >>>>> Hello Jagan, Jonas, >>>>> >>>>> I wanted to chip into this discussion, to ask whether you did anything >>>>> more on the SD clock matter ? >>>> >>>> I have been busy this past week but have now had time to take a new >>>> look >>>> at the sdmmc issue, along with completing some other fixes. >>>> >>>>> >>>>> I am currently using your workaround that creates dummy clocks in >>>>> the DT >>>>> and then the SD node just uses those, and it works, in the SPL, if and >>>>> only if the bootrom also loads the SPL from SD. >>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>>>> the rockusb protocol, and then, initializing the SD-Card from SPL >>>>> fails >>>>> utterly, namely, it cannot communicate with the card. >>>>> I did some changes to have the pinctrl working at SPL level, which >>>>> appears to be fine, but I would like to see what can be done about the >>>>> clocks. >>>>> Having the cru and all the required drivers at SPL level is the way to >>>>> go ? The SPL should run before SCMI is required so it should be >>>>> able to >>>>> change all the clocks at the clock controller level ? >>>> >>>> I fully agree that we should have some sort of scmi clk driver so that >>>> we can control the sdmmc clk in both SPL and U-Boot proper. >>>> >>>> As you have noticed the current workaround only works because bootrom >>>> leave the clocks in a working state after it has loaded TPL/SPL from >>>> the sdmmc device. When TPL/SPL is loaded from any other source it is >>>> not be possible to read from the sdmmc device in u-boot. >>>> >>>> After having played around with the scmi agent driver and being >>>> inspired >>>> by the dummy scmi clk driver in vendor u-boot I have managed to create >>>> something that could work. See top three commits from [5] for a working >>>> proof-of-concept. >>>> >>>> What I did was to enable the scmi agent driver for use in u-boot proper >>>> and keeping it disabled in SPL build. Then added a new scmi clk driver >>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should >>>> be able to configure the sdmmc clocks. >>>> >>>> The initial fixes commits in that branch should hit the list soon. >>>> Will send the sdmmc related commits once I have had some time to do >>>> more testing. >>>> >>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >>>> >>> Hi Jonas, >>> >>> I managed to get it working in SPL as well, with a similar code inspired >>> from the vendor tree. >>> About the solution, I am not sure whether it should be called a scmi >>> clock, and whether it should be part of clk_rk3588 >>> Honestly by looking at the DT node, I find the description of the clock >>> tree wrong to begin with. >>> The SD should not have the 2 clocks tied to the scmi agent, because this >>> is true only if it's running in normal world with a secure world agent >>> that will act as a middle man. So if we are running at a higher EL, as >>> in the SPL case, the clocks should be tied directly to the CRU. >>> So the node itself is described bad IMO. >>> >>> Anyway, if the node is set in stone from Linux, there isn;t much we >>> can do. >>> I am also thinking whether we should have the SCMI enabled in SPL as >>> well, but the agent/clock should just access the CRU directly, meaning >>> to do something at the agent level : >>> #ifdef CONFIG_SPL_BUILD >>> -> access CRU >>> #else >>> -> send SCMI message >>> #endif >>> Does this make more sense ? >>> >>> Or have some kind of wrapper driver that would act as a dispatcher >>> depending on the SPL/proper build ? >> >> Looks like the sdmmc clock is controlled by the securecru registers and I >> expect that this can only be configured in secure world, not sure how >> this >> could have been modeled differently. > > Since it can work with a clock given by either scmi or securecru, I > would expect all clocks to be in the list, and the driver could start > the needed clock as per the EL level it's running into :) > > e.g. > clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi > HCLK_SD> , <&cru basic other clocks..> > > Just my vision of how it could be modeled > >> >> With the approach that I took I think the normal clk framework will >> behave >> as such dispatcher and clock gets tied to cru in SPL. >> >> #ifdef CONFIG_SPL_BUILD >> -> bind protocol@14 to rk3588 securecru driver -> read/write securecru >> regs >> #else >> -> bind protocol@14 to scmi clk driver -> send SCMI message >> #endif > > This looks right, just that we would have to bring a lot of bloat to SPL > , firmware subsystem and things we may not want. >> >> Naming and placement of the SPL securecru driver could be improved. >> Not sure if any other soc beside rk3588 will need this at this moment. > > securecru sounds better for me, I don't think we can have a scmi clock > fake driver like in vendor uboot, it has nothing 'scmi' about it. > >> >>> >>> >>> Meanwhile, I will test your patches to see how they work on my setup, I >>> also have some things in progress including pinctrl in SPL for the >>> rock5b. >>> Thanks for your detailed description, >> >> Thanks for the hint at pinctrl, I made some updates to [5] and could >> verify >> that sdmmc works when booting from emmc thanks to your pinctrl commit. >> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe >> and build a spi firmware image. >> >> There is an issue with non-DMA access in sdhci that requires the HACK >> commit >> for proper loading of atf, was hoping to disable SDMA in SPL like what >> is done >> using fifo-mode for sdmmc. But only first sector is read successfully >> then it >> fails, see below. Will take a closer look before I post eMMC series. > > In your patches, I see this : > https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 > Perhaps the fifo-mode is intended for the sdhci(emmc) and not for > sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, > but the sdmmc can ? > one possible workaround is to have DMA to DRAM and then relocate it to > SRAM using the CPU. Having DMA disabled for the whole IP may have > downsides, but this is U-boot, we don't expect to have anything else to > do with the CPU while the DMA master works its magic, and the CPU should > be faster. > > I tested your patches together with the series I sent now > ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to > boot correctly from SD-Card > Once you send them to the ML I can retest them. > Thanks ! Sorry I just realized I had something extra in my tree. It does not work without the dw_mmc reset after data error implementation. I can say that it was the same for me when I had it done on my own. I still have not figured it out why the first attempted SD mode does not work, and after reset, going to legacy mode, SD transfers work. I will send a patch with the dw reset code Eugen > >> >> Trying to boot from MMC2 >> 1 >> - 0 'mmc@fe2c0000' >> - 1 'mmc@fe2e0000' >> - found >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >> clock is disabled (0Hz) >> clock is enabled (400000Hz) >> clk_set_rate(clk=5000d8, rate=400000) >> size=200, ptr=3b8, limit=100000: 5001b8 >> clock is enabled (25000000Hz) >> clk_set_rate(clk=5000d8, rate=25000000) >> clk_set_rate(clk=5000d8, rate=25000000) >> clock is enabled (52000000Hz) >> clk_set_rate(clk=5000d8, rate=52000000) >> spl: mmc boot mode: raw >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >> hdr read sector 4000, count=1 >> Found FIT >> size=a00, ptr=dc0, limit=100000: aligned to 5003c0 >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >> fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 >> mmc_load_image_raw_sector: mmc block read error >> spl: mmc boot mode: fs >> Trying to boot from MMC1 >> >> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc >> >> Regards, >> Jonas >> >> >>> Eugen >>> >>>> >>>> Regards, >>>> Jonas >>>> >>>>> >>>>> Thanks, >>>>> Eugen >>>>> >>>>>> >>>>>> Regards, >>>>>> Jonas >>>>>> >>>>>>> >>>>>>> [3] >>>>>>> https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>>>> >>>>>>> Jagan. >>>>>> >>>>> >>>> >>> >> >
> Date: Mon, 13 Mar 2023 16:21:36 +0200 > From: Eugen Hristev <eugen.hristev@collabora.com> > > On 3/13/23 12:00, Jonas Karlman wrote: > > On 2023-03-13 09:42, Eugen Hristev wrote: > >> On 3/13/23 00:34, Jonas Karlman wrote: > >>> Hi Eugen, > >>> > >>> On 2023-03-08 09:57, Eugen Hristev wrote: > >>>> On 1/29/23 11:04, Jonas Karlman wrote: > >>>>> On 2023-01-27 14:21, Jagan Teki wrote: > >>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>> > >>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: > >>>>>>>> Hi Jagan, > >>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: > >>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: > >>>>>>>>>>> Hi Simon, > >>>>>>>>>>> > >>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: > >>>>>>>>>>>>>>> Hi Jonas, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: > >>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are > >>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit > >>>>>>>>>>>>>>>>> messages. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Verified below BL31 versions, > >>>>>>>>>>>>>>>>> bl31-v1.15 > >>>>>>>>>>>>>>>>> bl31-v1.21 > >>>>>>>>>>>>>>>>> bl31-v1.22 > >>>>>>>>>>>>>>>>> bl31-v1.23 > >>>>>>>>>>>>>>>>> bl31-v1.24 > >>>>>>>>>>>>>>>>> bl31-v1.25 > >>>>>>>>>>>>>>>>> bl31-v1.26 > >>>>>>>>>>>>>>>>> > >>>>>> > >>>>>> < snip > > >>>>>> > >>>>>>>> > >>>>>>>> As you can see in the logs above there is timeout waiting for data. > >>>>>>>> > >>>>>>>> I managed to find the issue and have a workaround that gets me longer > >>>>>>>> in the boot process, there still seem to be other issue with the rk3588 > >>>>>>>> startup. > >>>>>>>> > >>>>>>>> -------- > >>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) > >>>>>>>> Trying to boot from MMC1 > >>>>>>>> ## Checking hash(es) for config config-1 ... OK > >>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK > >>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK > >>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK > >>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK > >>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK > >>>>>>>> INFO: Preloader serial: 2 > >>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang > >>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 > >>>>>>>> INFO: spec: 0x1 > >>>>>>>> INFO: ext 32k is not valid > >>>>>>>> INFO: ddr: stride-en 4CH > >>>>>>>> INFO: GICv3 without legacy support detected. > >>>>>>>> INFO: ARM GICv3 driver initialized in EL3 > >>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 > >>>>>>>> INFO: system boots from cpu-hwid-0 > >>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 > >>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz > >>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz > >>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz > >>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz > >>>>>>>> INFO: BL31: Initialising Exception Handling Framework > >>>>>>>> INFO: BL31: Initializing runtime services > >>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK > >>>>>>>> ERROR: Error initializing runtime service opteed_fast > >>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world > >>>>>>>> INFO: Entry point address = 0xa00000 > >>>>>>>> INFO: SPSR = 0x3c9 > >>>>>>>> "Synchronous Abort" handler, esr 0x96000000 > >>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c > >>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 > >>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 > >>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 > >>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 > >>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 > >>>>>>>> x10: 0000000000000034 x11: 0000000000000002 > >>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc > >>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 > >>>>>>>> x16: 0000000000000000 x17: 0000000000000000 > >>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 > >>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 > >>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 > >>>>>>>> x24: 0000000000000000 x25: 0000000000000000 > >>>>>>>> x26: 0000000000000000 x27: 0000000000000000 > >>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 > >>>>>>>> > >>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) > >>>>>>>> Resetting CPU ... > >>>>>>>> -------- > >>>>>>>> > >>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e > >>>>>>>> with some rk3568 patches and this series, see [1]. > >>>>>>>> > >>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. > >>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a > >>>>>>>> fallback to default 400khz clock. > >>>>>>>> > >>>>>>>> Next issue is the sync abort, looks it happens when u-boot > >>>>>>>> tries to set clock rates based on devicetree. this is the > >>>>>>>> last debug line before the crash. > >>>>>>>> > >>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) > >>>>>>> > >>>>>>> With the commit at [2] I can now successfully run U-Boot proper. > >>>>>>> > >>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. > >>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. > >>>>>>> We can just remove it and leave setting a faster cpu rate to linux. > >>>>>>> > >>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, > >>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. > >>>>>>> > >>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 > >>>>>>> > >>>>>> Okay. We need to find a way to handle the clock value 400Khz > >>>>>> generically via the CLK framework or eMMC can be worth checking as it > >>>>>> doesn't involve SCMI and have a working patch set before MW. I did > >>>>>> that and was able to detect eMMC in U-Boot proper but got some issues > >>>>>> while booting from eMMC [3] > >>>>> > >>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. > >>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. > >>>>> > >>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc > >>>>> and I have included some rk3568 patches and your rk3588 rfc series. > >>>>> I added a few fixup on top of that and a few additional patches, please see commit message > >>>>> for a very brief note on why the change was needed. > >>>>> Feel free to squash fixups and pick commits up to and possible including > >>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" > >>>>> for a v2 of this series. > >>>>> > >>>>> The remaining sdhci patches needs a little bit more work, > >>>>> I can send a separate series with emmc patches once they are fully ready. > >>>>> > >>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. > >>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. > >>>>> I think this is a more appropriate way to work around this issue, instead of patching > >>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. > >>>>> > >>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 > >>>>> > >>>> Hello Jagan, Jonas, > >>>> > >>>> I wanted to chip into this discussion, to ask whether you did anything > >>>> more on the SD clock matter ? > >>> > >>> I have been busy this past week but have now had time to take a new look > >>> at the sdmmc issue, along with completing some other fixes. > >>> > >>>> > >>>> I am currently using your workaround that creates dummy clocks in the DT > >>>> and then the SD node just uses those, and it works, in the SPL, if and > >>>> only if the bootrom also loads the SPL from SD. > >>>> I tried to have the SPL inside the SPI flash, or load it to DDR using > >>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails > >>>> utterly, namely, it cannot communicate with the card. > >>>> I did some changes to have the pinctrl working at SPL level, which > >>>> appears to be fine, but I would like to see what can be done about the > >>>> clocks. > >>>> Having the cru and all the required drivers at SPL level is the way to > >>>> go ? The SPL should run before SCMI is required so it should be able to > >>>> change all the clocks at the clock controller level ? > >>> > >>> I fully agree that we should have some sort of scmi clk driver so that > >>> we can control the sdmmc clk in both SPL and U-Boot proper. > >>> > >>> As you have noticed the current workaround only works because bootrom > >>> leave the clocks in a working state after it has loaded TPL/SPL from > >>> the sdmmc device. When TPL/SPL is loaded from any other source it is > >>> not be possible to read from the sdmmc device in u-boot. > >>> > >>> After having played around with the scmi agent driver and being inspired > >>> by the dummy scmi clk driver in vendor u-boot I have managed to create > >>> something that could work. See top three commits from [5] for a working > >>> proof-of-concept. > >>> > >>> What I did was to enable the scmi agent driver for use in u-boot proper > >>> and keeping it disabled in SPL build. Then added a new scmi clk driver > >>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk > >>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops > >>> for the two sdmmc clocks. With this both SPL and U-Boot proper should > >>> be able to configure the sdmmc clocks. > >>> > >>> The initial fixes commits in that branch should hit the list soon. > >>> Will send the sdmmc related commits once I have had some time to do > >>> more testing. > >>> > >>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes > >>> > >> Hi Jonas, > >> > >> I managed to get it working in SPL as well, with a similar code inspired > >> from the vendor tree. > >> About the solution, I am not sure whether it should be called a scmi > >> clock, and whether it should be part of clk_rk3588 > >> Honestly by looking at the DT node, I find the description of the clock > >> tree wrong to begin with. > >> The SD should not have the 2 clocks tied to the scmi agent, because this > >> is true only if it's running in normal world with a secure world agent > >> that will act as a middle man. So if we are running at a higher EL, as > >> in the SPL case, the clocks should be tied directly to the CRU. > >> So the node itself is described bad IMO. > >> > >> Anyway, if the node is set in stone from Linux, there isn;t much we can do. > >> I am also thinking whether we should have the SCMI enabled in SPL as > >> well, but the agent/clock should just access the CRU directly, meaning > >> to do something at the agent level : > >> #ifdef CONFIG_SPL_BUILD > >> -> access CRU > >> #else > >> -> send SCMI message > >> #endif > >> Does this make more sense ? > >> > >> Or have some kind of wrapper driver that would act as a dispatcher > >> depending on the SPL/proper build ? > > > > Looks like the sdmmc clock is controlled by the securecru registers and I > > expect that this can only be configured in secure world, not sure how this > > could have been modeled differently. > > Since it can work with a clock given by either scmi or securecru, I > would expect all clocks to be in the list, and the driver could start > the needed clock as per the EL level it's running into :) > > e.g. > clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi > HCLK_SD> , <&cru basic other clocks..> > > Just my vision of how it could be modeled That is a change to the device tree bindings. The goal is to have U-Boot use the same bindings and device tree as the official device tree (which currently is the one in mainline Linux). > > With the approach that I took I think the normal clk framework will behave > > as such dispatcher and clock gets tied to cru in SPL. > > > > #ifdef CONFIG_SPL_BUILD > > -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs > > #else > > -> bind protocol@14 to scmi clk driver -> send SCMI message > > #endif > > This looks right, just that we would have to bring a lot of bloat to SPL > , firmware subsystem and things we may not want. I don't see why this would bring in a lot of bloat into SPL. The SPL device tree will grow a little bit since it will have to include the scmi nodes. And a little bit of additional code in the rk3588 clock driver. Navigating the Kconfig stuff is a bit hard, but I don't think this needs to pull in the firmware subsystem in SPL. > > > > Naming and placement of the SPL securecru driver could be improved. > > Not sure if any other soc beside rk3588 will need this at this moment. > > securecru sounds better for me, I don't think we can have a scmi clock > fake driver like in vendor uboot, it has nothing 'scmi' about it. > > >> Meanwhile, I will test your patches to see how they work on my setup, I > >> also have some things in progress including pinctrl in SPL for the rock5b. > >> Thanks for your detailed description, > > > > Thanks for the hint at pinctrl, I made some updates to [5] and could verify > > that sdmmc works when booting from emmc thanks to your pinctrl commit. > > Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe > > and build a spi firmware image. > > > > There is an issue with non-DMA access in sdhci that requires the HACK commit > > for proper loading of atf, was hoping to disable SDMA in SPL like what is done > > using fifo-mode for sdmmc. But only first sector is read successfully then it > > fails, see below. Will take a closer look before I post eMMC series. > > In your patches, I see this : > https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 > Perhaps the fifo-mode is intended for the sdhci(emmc) and not for > sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, > but the sdmmc can ? > one possible workaround is to have DMA to DRAM and then relocate it to > SRAM using the CPU. Having DMA disabled for the whole IP may have > downsides, but this is U-boot, we don't expect to have anything else to > do with the CPU while the DMA master works its magic, and the CPU should > be faster. > > I tested your patches together with the series I sent now > ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to > boot correctly from SD-Card > Once you send them to the ML I can retest them. > Thanks ! > > > > > Trying to boot from MMC2 > > 1 > > - 0 'mmc@fe2c0000' > > - 1 'mmc@fe2e0000' > > - found > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > > clock is disabled (0Hz) > > clock is enabled (400000Hz) > > clk_set_rate(clk=5000d8, rate=400000) > > size=200, ptr=3b8, limit=100000: 5001b8 > > clock is enabled (25000000Hz) > > clk_set_rate(clk=5000d8, rate=25000000) > > clk_set_rate(clk=5000d8, rate=25000000) > > clock is enabled (52000000Hz) > > clk_set_rate(clk=5000d8, rate=52000000) > > spl: mmc boot mode: raw > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > > hdr read sector 4000, count=1 > > Found FIT > > size=a00, ptr=dc0, limit=100000: aligned to 5003c0 > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > > blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > > fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 > > mmc_load_image_raw_sector: mmc block read error > > spl: mmc boot mode: fs > > Trying to boot from MMC1 > > > > [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc > > > > Regards, > > Jonas > > > > > >> Eugen > >> > >>> > >>> Regards, > >>> Jonas > >>> > >>>> > >>>> Thanks, > >>>> Eugen > >>>> > >>>>> > >>>>> Regards, > >>>>> Jonas > >>>>> > >>>>>> > >>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f > >>>>>> > >>>>>> Jagan. > >>>>> > >>>> > >>> > >> > > > >
On 3/13/23 17:07, Mark Kettenis wrote: >> Date: Mon, 13 Mar 2023 16:21:36 +0200 >> From: Eugen Hristev <eugen.hristev@collabora.com> >> >> On 3/13/23 12:00, Jonas Karlman wrote: >>> On 2023-03-13 09:42, Eugen Hristev wrote: >>>> On 3/13/23 00:34, Jonas Karlman wrote: >>>>> Hi Eugen, >>>>> >>>>> On 2023-03-08 09:57, Eugen Hristev wrote: >>>>>> On 1/29/23 11:04, Jonas Karlman wrote: >>>>>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>> >>>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>>>>> Hi Jagan, >>>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>> < snip > >>>>>>>> >>>>>>>>>> >>>>>>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>>>>>> >>>>>>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>>>>>> startup. >>>>>>>>>> >>>>>>>>>> -------- >>>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>>>>> Trying to boot from MMC1 >>>>>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>>>>> INFO: Preloader serial: 2 >>>>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>>>>> INFO: spec: 0x1 >>>>>>>>>> INFO: ext 32k is not valid >>>>>>>>>> INFO: ddr: stride-en 4CH >>>>>>>>>> INFO: GICv3 without legacy support detected. >>>>>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>>>>> INFO: BL31: Initializing runtime services >>>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>>>>> INFO: Entry point address = 0xa00000 >>>>>>>>>> INFO: SPSR = 0x3c9 >>>>>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>>>>> >>>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>>>>> Resetting CPU ... >>>>>>>>>> -------- >>>>>>>>>> >>>>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>>>>> >>>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>>>>> fallback to default 400khz clock. >>>>>>>>>> >>>>>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>>>>> last debug line before the crash. >>>>>>>>>> >>>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>>>>> >>>>>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>>>>> >>>>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>>>>> >>>>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>>>>>> >>>>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>>>>> >>>>>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>>>>> generically via the CLK framework or eMMC can be worth checking as it >>>>>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>>>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>>>>>> while booting from eMMC [3] >>>>>>> >>>>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>>>>> >>>>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>>>>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>>>>> I added a few fixup on top of that and a few additional patches, please see commit message >>>>>>> for a very brief note on why the change was needed. >>>>>>> Feel free to squash fixups and pick commits up to and possible including >>>>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>>>>>> for a v2 of this series. >>>>>>> >>>>>>> The remaining sdhci patches needs a little bit more work, >>>>>>> I can send a separate series with emmc patches once they are fully ready. >>>>>>> >>>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>>>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>>>>>> I think this is a more appropriate way to work around this issue, instead of patching >>>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>>>>>> >>>>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>>>>> >>>>>> Hello Jagan, Jonas, >>>>>> >>>>>> I wanted to chip into this discussion, to ask whether you did anything >>>>>> more on the SD clock matter ? >>>>> >>>>> I have been busy this past week but have now had time to take a new look >>>>> at the sdmmc issue, along with completing some other fixes. >>>>> >>>>>> >>>>>> I am currently using your workaround that creates dummy clocks in the DT >>>>>> and then the SD node just uses those, and it works, in the SPL, if and >>>>>> only if the bootrom also loads the SPL from SD. >>>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>>>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails >>>>>> utterly, namely, it cannot communicate with the card. >>>>>> I did some changes to have the pinctrl working at SPL level, which >>>>>> appears to be fine, but I would like to see what can be done about the >>>>>> clocks. >>>>>> Having the cru and all the required drivers at SPL level is the way to >>>>>> go ? The SPL should run before SCMI is required so it should be able to >>>>>> change all the clocks at the clock controller level ? >>>>> >>>>> I fully agree that we should have some sort of scmi clk driver so that >>>>> we can control the sdmmc clk in both SPL and U-Boot proper. >>>>> >>>>> As you have noticed the current workaround only works because bootrom >>>>> leave the clocks in a working state after it has loaded TPL/SPL from >>>>> the sdmmc device. When TPL/SPL is loaded from any other source it is >>>>> not be possible to read from the sdmmc device in u-boot. >>>>> >>>>> After having played around with the scmi agent driver and being inspired >>>>> by the dummy scmi clk driver in vendor u-boot I have managed to create >>>>> something that could work. See top three commits from [5] for a working >>>>> proof-of-concept. >>>>> >>>>> What I did was to enable the scmi agent driver for use in u-boot proper >>>>> and keeping it disabled in SPL build. Then added a new scmi clk driver >>>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >>>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >>>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should >>>>> be able to configure the sdmmc clocks. >>>>> >>>>> The initial fixes commits in that branch should hit the list soon. >>>>> Will send the sdmmc related commits once I have had some time to do >>>>> more testing. >>>>> >>>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >>>>> >>>> Hi Jonas, >>>> >>>> I managed to get it working in SPL as well, with a similar code inspired >>>> from the vendor tree. >>>> About the solution, I am not sure whether it should be called a scmi >>>> clock, and whether it should be part of clk_rk3588 >>>> Honestly by looking at the DT node, I find the description of the clock >>>> tree wrong to begin with. >>>> The SD should not have the 2 clocks tied to the scmi agent, because this >>>> is true only if it's running in normal world with a secure world agent >>>> that will act as a middle man. So if we are running at a higher EL, as >>>> in the SPL case, the clocks should be tied directly to the CRU. >>>> So the node itself is described bad IMO. >>>> >>>> Anyway, if the node is set in stone from Linux, there isn;t much we can do. >>>> I am also thinking whether we should have the SCMI enabled in SPL as >>>> well, but the agent/clock should just access the CRU directly, meaning >>>> to do something at the agent level : >>>> #ifdef CONFIG_SPL_BUILD >>>> -> access CRU >>>> #else >>>> -> send SCMI message >>>> #endif >>>> Does this make more sense ? >>>> >>>> Or have some kind of wrapper driver that would act as a dispatcher >>>> depending on the SPL/proper build ? >>> >>> Looks like the sdmmc clock is controlled by the securecru registers and I >>> expect that this can only be configured in secure world, not sure how this >>> could have been modeled differently. >> >> Since it can work with a clock given by either scmi or securecru, I >> would expect all clocks to be in the list, and the driver could start >> the needed clock as per the EL level it's running into :) >> >> e.g. >> clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi >> HCLK_SD> , <&cru basic other clocks..> >> >> Just my vision of how it could be modeled > > That is a change to the device tree bindings. The goal is to have > U-Boot use the same bindings and device tree as the official device > tree (which currently is the one in mainline Linux). That's right. A change in the bindings (and Linux ) > >>> With the approach that I took I think the normal clk framework will behave >>> as such dispatcher and clock gets tied to cru in SPL. >>> >>> #ifdef CONFIG_SPL_BUILD >>> -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs >>> #else >>> -> bind protocol@14 to scmi clk driver -> send SCMI message >>> #endif >> >> This looks right, just that we would have to bring a lot of bloat to SPL >> , firmware subsystem and things we may not want. > > I don't see why this would bring in a lot of bloat into SPL. The SPL > device tree will grow a little bit since it will have to include the > scmi nodes. And a little bit of additional code in the rk3588 clock > driver. Navigating the Kconfig stuff is a bit hard, but I don't think > this needs to pull in the firmware subsystem in SPL. When I tried this, I noticed that only the scmi agents look inside the subnodes (e.g. protocol@14) which do not have a compatible, and then they bind the subnodes to a driver found by a hardcoded search for 'scmi_clock'. So, to get there, an agent is required, and firmware node, probed firmware, probed 'scmi clock' . Then, probing it requires additional things, because the agent wants a shared memory 'shmem' node, and you end up also probing a SRAM area, allocating it to the space... that's what I mean when I say that there might be much more bloat added than just an additional clock set/get for securecru clocks. > >>> >>> Naming and placement of the SPL securecru driver could be improved. >>> Not sure if any other soc beside rk3588 will need this at this moment. >> >> securecru sounds better for me, I don't think we can have a scmi clock >> fake driver like in vendor uboot, it has nothing 'scmi' about it. >> >>>> Meanwhile, I will test your patches to see how they work on my setup, I >>>> also have some things in progress including pinctrl in SPL for the rock5b. >>>> Thanks for your detailed description, >>> >>> Thanks for the hint at pinctrl, I made some updates to [5] and could verify >>> that sdmmc works when booting from emmc thanks to your pinctrl commit. >>> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe >>> and build a spi firmware image. >>> >>> There is an issue with non-DMA access in sdhci that requires the HACK commit >>> for proper loading of atf, was hoping to disable SDMA in SPL like what is done >>> using fifo-mode for sdmmc. But only first sector is read successfully then it >>> fails, see below. Will take a closer look before I post eMMC series. >> >> In your patches, I see this : >> https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 >> Perhaps the fifo-mode is intended for the sdhci(emmc) and not for >> sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, >> but the sdmmc can ? >> one possible workaround is to have DMA to DRAM and then relocate it to >> SRAM using the CPU. Having DMA disabled for the whole IP may have >> downsides, but this is U-boot, we don't expect to have anything else to >> do with the CPU while the DMA master works its magic, and the CPU should >> be faster. >> >> I tested your patches together with the series I sent now >> ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to >> boot correctly from SD-Card >> Once you send them to the ML I can retest them. >> Thanks ! >> >>> >>> Trying to boot from MMC2 >>> 1 >>> - 0 'mmc@fe2c0000' >>> - 1 'mmc@fe2e0000' >>> - found >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>> clock is disabled (0Hz) >>> clock is enabled (400000Hz) >>> clk_set_rate(clk=5000d8, rate=400000) >>> size=200, ptr=3b8, limit=100000: 5001b8 >>> clock is enabled (25000000Hz) >>> clk_set_rate(clk=5000d8, rate=25000000) >>> clk_set_rate(clk=5000d8, rate=25000000) >>> clock is enabled (52000000Hz) >>> clk_set_rate(clk=5000d8, rate=52000000) >>> spl: mmc boot mode: raw >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>> hdr read sector 4000, count=1 >>> Found FIT >>> size=a00, ptr=dc0, limit=100000: aligned to 5003c0 >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>> fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 >>> mmc_load_image_raw_sector: mmc block read error >>> spl: mmc boot mode: fs >>> Trying to boot from MMC1 >>> >>> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc >>> >>> Regards, >>> Jonas >>> >>> >>>> Eugen >>>> >>>>> >>>>> Regards, >>>>> Jonas >>>>> >>>>>> >>>>>> Thanks, >>>>>> Eugen >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Jonas >>>>>>> >>>>>>>> >>>>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>>>>> >>>>>>>> Jagan. >>>>>>> >>>>>> >>>>> >>>> >>> >> >>
> Date: Mon, 13 Mar 2023 17:21:05 +0200 > From: Eugen Hristev <eugen.hristev@collabora.com> > > On 3/13/23 17:07, Mark Kettenis wrote: > >> Date: Mon, 13 Mar 2023 16:21:36 +0200 > >> From: Eugen Hristev <eugen.hristev@collabora.com> > >> > >> On 3/13/23 12:00, Jonas Karlman wrote: > >>> On 2023-03-13 09:42, Eugen Hristev wrote: > >>>> On 3/13/23 00:34, Jonas Karlman wrote: > >>>>> Hi Eugen, > >>>>> > >>>>> On 2023-03-08 09:57, Eugen Hristev wrote: > >>>>>> On 1/29/23 11:04, Jonas Karlman wrote: > >>>>>>> On 2023-01-27 14:21, Jagan Teki wrote: > >>>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>> > >>>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: > >>>>>>>>>> Hi Jagan, > >>>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: > >>>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: > >>>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: > >>>>>>>>>>>>>>>>> Hi Jonas, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi Jagan, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: > >>>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are > >>>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit > >>>>>>>>>>>>>>>>>>> messages. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so > >>>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Verified below BL31 versions, > >>>>>>>>>>>>>>>>>>> bl31-v1.15 > >>>>>>>>>>>>>>>>>>> bl31-v1.21 > >>>>>>>>>>>>>>>>>>> bl31-v1.22 > >>>>>>>>>>>>>>>>>>> bl31-v1.23 > >>>>>>>>>>>>>>>>>>> bl31-v1.24 > >>>>>>>>>>>>>>>>>>> bl31-v1.25 > >>>>>>>>>>>>>>>>>>> bl31-v1.26 > >>>>>>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>> < snip > > >>>>>>>> > >>>>>>>>>> > >>>>>>>>>> As you can see in the logs above there is timeout waiting for data. > >>>>>>>>>> > >>>>>>>>>> I managed to find the issue and have a workaround that gets me longer > >>>>>>>>>> in the boot process, there still seem to be other issue with the rk3588 > >>>>>>>>>> startup. > >>>>>>>>>> > >>>>>>>>>> -------- > >>>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) > >>>>>>>>>> Trying to boot from MMC1 > >>>>>>>>>> ## Checking hash(es) for config config-1 ... OK > >>>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK > >>>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK > >>>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK > >>>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK > >>>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK > >>>>>>>>>> INFO: Preloader serial: 2 > >>>>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang > >>>>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 > >>>>>>>>>> INFO: spec: 0x1 > >>>>>>>>>> INFO: ext 32k is not valid > >>>>>>>>>> INFO: ddr: stride-en 4CH > >>>>>>>>>> INFO: GICv3 without legacy support detected. > >>>>>>>>>> INFO: ARM GICv3 driver initialized in EL3 > >>>>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 > >>>>>>>>>> INFO: system boots from cpu-hwid-0 > >>>>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 > >>>>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz > >>>>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz > >>>>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz > >>>>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz > >>>>>>>>>> INFO: BL31: Initialising Exception Handling Framework > >>>>>>>>>> INFO: BL31: Initializing runtime services > >>>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK > >>>>>>>>>> ERROR: Error initializing runtime service opteed_fast > >>>>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world > >>>>>>>>>> INFO: Entry point address = 0xa00000 > >>>>>>>>>> INFO: SPSR = 0x3c9 > >>>>>>>>>> "Synchronous Abort" handler, esr 0x96000000 > >>>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c > >>>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 > >>>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 > >>>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 > >>>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 > >>>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 > >>>>>>>>>> x10: 0000000000000034 x11: 0000000000000002 > >>>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc > >>>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 > >>>>>>>>>> x16: 0000000000000000 x17: 0000000000000000 > >>>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 > >>>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 > >>>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 > >>>>>>>>>> x24: 0000000000000000 x25: 0000000000000000 > >>>>>>>>>> x26: 0000000000000000 x27: 0000000000000000 > >>>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 > >>>>>>>>>> > >>>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) > >>>>>>>>>> Resetting CPU ... > >>>>>>>>>> -------- > >>>>>>>>>> > >>>>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e > >>>>>>>>>> with some rk3568 patches and this series, see [1]. > >>>>>>>>>> > >>>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. > >>>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a > >>>>>>>>>> fallback to default 400khz clock. > >>>>>>>>>> > >>>>>>>>>> Next issue is the sync abort, looks it happens when u-boot > >>>>>>>>>> tries to set clock rates based on devicetree. this is the > >>>>>>>>>> last debug line before the crash. > >>>>>>>>>> > >>>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) > >>>>>>>>> > >>>>>>>>> With the commit at [2] I can now successfully run U-Boot proper. > >>>>>>>>> > >>>>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. > >>>>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. > >>>>>>>>> We can just remove it and leave setting a faster cpu rate to linux. > >>>>>>>>> > >>>>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, > >>>>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. > >>>>>>>>> > >>>>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 > >>>>>>>>> > >>>>>>>> Okay. We need to find a way to handle the clock value 400Khz > >>>>>>>> generically via the CLK framework or eMMC can be worth checking as it > >>>>>>>> doesn't involve SCMI and have a working patch set before MW. I did > >>>>>>>> that and was able to detect eMMC in U-Boot proper but got some issues > >>>>>>>> while booting from eMMC [3] > >>>>>>> > >>>>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. > >>>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. > >>>>>>> > >>>>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc > >>>>>>> and I have included some rk3568 patches and your rk3588 rfc series. > >>>>>>> I added a few fixup on top of that and a few additional patches, please see commit message > >>>>>>> for a very brief note on why the change was needed. > >>>>>>> Feel free to squash fixups and pick commits up to and possible including > >>>>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" > >>>>>>> for a v2 of this series. > >>>>>>> > >>>>>>> The remaining sdhci patches needs a little bit more work, > >>>>>>> I can send a separate series with emmc patches once they are fully ready. > >>>>>>> > >>>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. > >>>>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. > >>>>>>> I think this is a more appropriate way to work around this issue, instead of patching > >>>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. > >>>>>>> > >>>>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 > >>>>>>> > >>>>>> Hello Jagan, Jonas, > >>>>>> > >>>>>> I wanted to chip into this discussion, to ask whether you did anything > >>>>>> more on the SD clock matter ? > >>>>> > >>>>> I have been busy this past week but have now had time to take a new look > >>>>> at the sdmmc issue, along with completing some other fixes. > >>>>> > >>>>>> > >>>>>> I am currently using your workaround that creates dummy clocks in the DT > >>>>>> and then the SD node just uses those, and it works, in the SPL, if and > >>>>>> only if the bootrom also loads the SPL from SD. > >>>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using > >>>>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails > >>>>>> utterly, namely, it cannot communicate with the card. > >>>>>> I did some changes to have the pinctrl working at SPL level, which > >>>>>> appears to be fine, but I would like to see what can be done about the > >>>>>> clocks. > >>>>>> Having the cru and all the required drivers at SPL level is the way to > >>>>>> go ? The SPL should run before SCMI is required so it should be able to > >>>>>> change all the clocks at the clock controller level ? > >>>>> > >>>>> I fully agree that we should have some sort of scmi clk driver so that > >>>>> we can control the sdmmc clk in both SPL and U-Boot proper. > >>>>> > >>>>> As you have noticed the current workaround only works because bootrom > >>>>> leave the clocks in a working state after it has loaded TPL/SPL from > >>>>> the sdmmc device. When TPL/SPL is loaded from any other source it is > >>>>> not be possible to read from the sdmmc device in u-boot. > >>>>> > >>>>> After having played around with the scmi agent driver and being inspired > >>>>> by the dummy scmi clk driver in vendor u-boot I have managed to create > >>>>> something that could work. See top three commits from [5] for a working > >>>>> proof-of-concept. > >>>>> > >>>>> What I did was to enable the scmi agent driver for use in u-boot proper > >>>>> and keeping it disabled in SPL build. Then added a new scmi clk driver > >>>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk > >>>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops > >>>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should > >>>>> be able to configure the sdmmc clocks. > >>>>> > >>>>> The initial fixes commits in that branch should hit the list soon. > >>>>> Will send the sdmmc related commits once I have had some time to do > >>>>> more testing. > >>>>> > >>>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes > >>>>> > >>>> Hi Jonas, > >>>> > >>>> I managed to get it working in SPL as well, with a similar code inspired > >>>> from the vendor tree. > >>>> About the solution, I am not sure whether it should be called a scmi > >>>> clock, and whether it should be part of clk_rk3588 > >>>> Honestly by looking at the DT node, I find the description of the clock > >>>> tree wrong to begin with. > >>>> The SD should not have the 2 clocks tied to the scmi agent, because this > >>>> is true only if it's running in normal world with a secure world agent > >>>> that will act as a middle man. So if we are running at a higher EL, as > >>>> in the SPL case, the clocks should be tied directly to the CRU. > >>>> So the node itself is described bad IMO. > >>>> > >>>> Anyway, if the node is set in stone from Linux, there isn;t much we can do. > >>>> I am also thinking whether we should have the SCMI enabled in SPL as > >>>> well, but the agent/clock should just access the CRU directly, meaning > >>>> to do something at the agent level : > >>>> #ifdef CONFIG_SPL_BUILD > >>>> -> access CRU > >>>> #else > >>>> -> send SCMI message > >>>> #endif > >>>> Does this make more sense ? > >>>> > >>>> Or have some kind of wrapper driver that would act as a dispatcher > >>>> depending on the SPL/proper build ? > >>> > >>> Looks like the sdmmc clock is controlled by the securecru registers and I > >>> expect that this can only be configured in secure world, not sure how this > >>> could have been modeled differently. > >> > >> Since it can work with a clock given by either scmi or securecru, I > >> would expect all clocks to be in the list, and the driver could start > >> the needed clock as per the EL level it's running into :) > >> > >> e.g. > >> clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi > >> HCLK_SD> , <&cru basic other clocks..> > >> > >> Just my vision of how it could be modeled > > > > That is a change to the device tree bindings. The goal is to have > > U-Boot use the same bindings and device tree as the official device > > tree (which currently is the one in mainline Linux). > > That's right. A change in the bindings (and Linux ) And OpenBSD, and NetBSD, and ... The device tree bindings are ABI. You can't change them unless they are really relly broken. > >>> With the approach that I took I think the normal clk framework will behave > >>> as such dispatcher and clock gets tied to cru in SPL. > >>> > >>> #ifdef CONFIG_SPL_BUILD > >>> -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs > >>> #else > >>> -> bind protocol@14 to scmi clk driver -> send SCMI message > >>> #endif > >> > >> This looks right, just that we would have to bring a lot of bloat to SPL > >> , firmware subsystem and things we may not want. > > > > I don't see why this would bring in a lot of bloat into SPL. The SPL > > device tree will grow a little bit since it will have to include the > > scmi nodes. And a little bit of additional code in the rk3588 clock > > driver. Navigating the Kconfig stuff is a bit hard, but I don't think > > this needs to pull in the firmware subsystem in SPL. > > When I tried this, I noticed that only the scmi agents look inside the > subnodes (e.g. protocol@14) which do not have a compatible, and then > they bind the subnodes to a driver found by a hardcoded search for > 'scmi_clock'. > So, to get there, an agent is required, and firmware node, probed > firmware, probed 'scmi clock' . Then, probing it requires additional > things, because the agent wants a shared memory 'shmem' node, and you > end up also probing a SRAM area, allocating it to the space... that's > what I mean when I say that there might be much more bloat added than > just an additional clock set/get for securecru clocks. No, Jonas added code to find the clock protocol node and bind the clock driver to it: https://github.com/Kwiboo/u-boot-rockchip/blob/3209167d7a518291f912964186ee90f10b555084/drivers/clk/rockchip/clk_rk3588.c#L1953 > >>> Naming and placement of the SPL securecru driver could be improved. > >>> Not sure if any other soc beside rk3588 will need this at this moment. > >> > >> securecru sounds better for me, I don't think we can have a scmi clock > >> fake driver like in vendor uboot, it has nothing 'scmi' about it. > >> > >>>> Meanwhile, I will test your patches to see how they work on my setup, I > >>>> also have some things in progress including pinctrl in SPL for the rock5b. > >>>> Thanks for your detailed description, > >>> > >>> Thanks for the hint at pinctrl, I made some updates to [5] and could verify > >>> that sdmmc works when booting from emmc thanks to your pinctrl commit. > >>> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe > >>> and build a spi firmware image. > >>> > >>> There is an issue with non-DMA access in sdhci that requires the HACK commit > >>> for proper loading of atf, was hoping to disable SDMA in SPL like what is done > >>> using fifo-mode for sdmmc. But only first sector is read successfully then it > >>> fails, see below. Will take a closer look before I post eMMC series. > >> > >> In your patches, I see this : > >> https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 > >> Perhaps the fifo-mode is intended for the sdhci(emmc) and not for > >> sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, > >> but the sdmmc can ? > >> one possible workaround is to have DMA to DRAM and then relocate it to > >> SRAM using the CPU. Having DMA disabled for the whole IP may have > >> downsides, but this is U-boot, we don't expect to have anything else to > >> do with the CPU while the DMA master works its magic, and the CPU should > >> be faster. > >> > >> I tested your patches together with the series I sent now > >> ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to > >> boot correctly from SD-Card > >> Once you send them to the ML I can retest them. > >> Thanks ! > >> > >>> > >>> Trying to boot from MMC2 > >>> 1 > >>> - 0 'mmc@fe2c0000' > >>> - 1 'mmc@fe2e0000' > >>> - found > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > >>> clock is disabled (0Hz) > >>> clock is enabled (400000Hz) > >>> clk_set_rate(clk=5000d8, rate=400000) > >>> size=200, ptr=3b8, limit=100000: 5001b8 > >>> clock is enabled (25000000Hz) > >>> clk_set_rate(clk=5000d8, rate=25000000) > >>> clk_set_rate(clk=5000d8, rate=25000000) > >>> clock is enabled (52000000Hz) > >>> clk_set_rate(clk=5000d8, rate=52000000) > >>> spl: mmc boot mode: raw > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > >>> hdr read sector 4000, count=1 > >>> Found FIT > >>> size=a00, ptr=dc0, limit=100000: aligned to 5003c0 > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 > >>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 > >>> fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 > >>> mmc_load_image_raw_sector: mmc block read error > >>> spl: mmc boot mode: fs > >>> Trying to boot from MMC1 > >>> > >>> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc > >>> > >>> Regards, > >>> Jonas > >>> > >>> > >>>> Eugen > >>>> > >>>>> > >>>>> Regards, > >>>>> Jonas > >>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> Eugen > >>>>>> > >>>>>>> > >>>>>>> Regards, > >>>>>>> Jonas > >>>>>>> > >>>>>>>> > >>>>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f > >>>>>>>> > >>>>>>>> Jagan. > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >
On 3/13/23 17:34, Mark Kettenis wrote: >> Date: Mon, 13 Mar 2023 17:21:05 +0200 >> From: Eugen Hristev <eugen.hristev@collabora.com> >> >> On 3/13/23 17:07, Mark Kettenis wrote: >>>> Date: Mon, 13 Mar 2023 16:21:36 +0200 >>>> From: Eugen Hristev <eugen.hristev@collabora.com> >>>> >>>> On 3/13/23 12:00, Jonas Karlman wrote: >>>>> On 2023-03-13 09:42, Eugen Hristev wrote: >>>>>> On 3/13/23 00:34, Jonas Karlman wrote: >>>>>>> Hi Eugen, >>>>>>> >>>>>>> On 2023-03-08 09:57, Eugen Hristev wrote: >>>>>>>> On 1/29/23 11:04, Jonas Karlman wrote: >>>>>>>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>> >>>>>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> < snip > >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>>>>>>>> >>>>>>>>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>>>>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>>>>>>>> startup. >>>>>>>>>>>> >>>>>>>>>>>> -------- >>>>>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>>>>>>> Trying to boot from MMC1 >>>>>>>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>>>>>>> INFO: Preloader serial: 2 >>>>>>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>>>>>>> INFO: spec: 0x1 >>>>>>>>>>>> INFO: ext 32k is not valid >>>>>>>>>>>> INFO: ddr: stride-en 4CH >>>>>>>>>>>> INFO: GICv3 without legacy support detected. >>>>>>>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>>>>>>> INFO: BL31: Initializing runtime services >>>>>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>>>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>>>>>>> INFO: Entry point address = 0xa00000 >>>>>>>>>>>> INFO: SPSR = 0x3c9 >>>>>>>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>>>>>>> >>>>>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>>>>>>> Resetting CPU ... >>>>>>>>>>>> -------- >>>>>>>>>>>> >>>>>>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>>>>>>> >>>>>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>>>>>>> fallback to default 400khz clock. >>>>>>>>>>>> >>>>>>>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>>>>>>> last debug line before the crash. >>>>>>>>>>>> >>>>>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>>>>>>> >>>>>>>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>>>>>>> >>>>>>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>>>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>>>>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>>>>>>> >>>>>>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>>>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>>>>>>>> >>>>>>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>>>>>>> >>>>>>>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>>>>>>> generically via the CLK framework or eMMC can be worth checking as it >>>>>>>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>>>>>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>>>>>>>> while booting from eMMC [3] >>>>>>>>> >>>>>>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>>>>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>>>>>>> >>>>>>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>>>>>>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>>>>>>> I added a few fixup on top of that and a few additional patches, please see commit message >>>>>>>>> for a very brief note on why the change was needed. >>>>>>>>> Feel free to squash fixups and pick commits up to and possible including >>>>>>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>>>>>>>> for a v2 of this series. >>>>>>>>> >>>>>>>>> The remaining sdhci patches needs a little bit more work, >>>>>>>>> I can send a separate series with emmc patches once they are fully ready. >>>>>>>>> >>>>>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>>>>>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>>>>>>>> I think this is a more appropriate way to work around this issue, instead of patching >>>>>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>>>>>>>> >>>>>>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>>>>>>> >>>>>>>> Hello Jagan, Jonas, >>>>>>>> >>>>>>>> I wanted to chip into this discussion, to ask whether you did anything >>>>>>>> more on the SD clock matter ? >>>>>>> >>>>>>> I have been busy this past week but have now had time to take a new look >>>>>>> at the sdmmc issue, along with completing some other fixes. >>>>>>> >>>>>>>> >>>>>>>> I am currently using your workaround that creates dummy clocks in the DT >>>>>>>> and then the SD node just uses those, and it works, in the SPL, if and >>>>>>>> only if the bootrom also loads the SPL from SD. >>>>>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>>>>>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails >>>>>>>> utterly, namely, it cannot communicate with the card. >>>>>>>> I did some changes to have the pinctrl working at SPL level, which >>>>>>>> appears to be fine, but I would like to see what can be done about the >>>>>>>> clocks. >>>>>>>> Having the cru and all the required drivers at SPL level is the way to >>>>>>>> go ? The SPL should run before SCMI is required so it should be able to >>>>>>>> change all the clocks at the clock controller level ? >>>>>>> >>>>>>> I fully agree that we should have some sort of scmi clk driver so that >>>>>>> we can control the sdmmc clk in both SPL and U-Boot proper. >>>>>>> >>>>>>> As you have noticed the current workaround only works because bootrom >>>>>>> leave the clocks in a working state after it has loaded TPL/SPL from >>>>>>> the sdmmc device. When TPL/SPL is loaded from any other source it is >>>>>>> not be possible to read from the sdmmc device in u-boot. >>>>>>> >>>>>>> After having played around with the scmi agent driver and being inspired >>>>>>> by the dummy scmi clk driver in vendor u-boot I have managed to create >>>>>>> something that could work. See top three commits from [5] for a working >>>>>>> proof-of-concept. >>>>>>> >>>>>>> What I did was to enable the scmi agent driver for use in u-boot proper >>>>>>> and keeping it disabled in SPL build. Then added a new scmi clk driver >>>>>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >>>>>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >>>>>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should >>>>>>> be able to configure the sdmmc clocks. >>>>>>> >>>>>>> The initial fixes commits in that branch should hit the list soon. >>>>>>> Will send the sdmmc related commits once I have had some time to do >>>>>>> more testing. >>>>>>> >>>>>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >>>>>>> >>>>>> Hi Jonas, >>>>>> >>>>>> I managed to get it working in SPL as well, with a similar code inspired >>>>>> from the vendor tree. >>>>>> About the solution, I am not sure whether it should be called a scmi >>>>>> clock, and whether it should be part of clk_rk3588 >>>>>> Honestly by looking at the DT node, I find the description of the clock >>>>>> tree wrong to begin with. >>>>>> The SD should not have the 2 clocks tied to the scmi agent, because this >>>>>> is true only if it's running in normal world with a secure world agent >>>>>> that will act as a middle man. So if we are running at a higher EL, as >>>>>> in the SPL case, the clocks should be tied directly to the CRU. >>>>>> So the node itself is described bad IMO. >>>>>> >>>>>> Anyway, if the node is set in stone from Linux, there isn;t much we can do. >>>>>> I am also thinking whether we should have the SCMI enabled in SPL as >>>>>> well, but the agent/clock should just access the CRU directly, meaning >>>>>> to do something at the agent level : >>>>>> #ifdef CONFIG_SPL_BUILD >>>>>> -> access CRU >>>>>> #else >>>>>> -> send SCMI message >>>>>> #endif >>>>>> Does this make more sense ? >>>>>> >>>>>> Or have some kind of wrapper driver that would act as a dispatcher >>>>>> depending on the SPL/proper build ? >>>>> >>>>> Looks like the sdmmc clock is controlled by the securecru registers and I >>>>> expect that this can only be configured in secure world, not sure how this >>>>> could have been modeled differently. >>>> >>>> Since it can work with a clock given by either scmi or securecru, I >>>> would expect all clocks to be in the list, and the driver could start >>>> the needed clock as per the EL level it's running into :) >>>> >>>> e.g. >>>> clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi >>>> HCLK_SD> , <&cru basic other clocks..> >>>> >>>> Just my vision of how it could be modeled >>> >>> That is a change to the device tree bindings. The goal is to have >>> U-Boot use the same bindings and device tree as the official device >>> tree (which currently is the one in mainline Linux). >> >> That's right. A change in the bindings (and Linux ) > > And OpenBSD, and NetBSD, and ... > > The device tree bindings are ABI. You can't change them unless they > are really relly broken. For me, it's broken. You can't describe an SD hardware block, with clocks given by SCMI node . It's not right. It's not like this in hardware. It *can* be through SCMI, but it can also be via another way. And the purpose of the ABI is to describe the hardware. Unless we describe all of it, the description is wrong or at least incomplete. > >>>>> With the approach that I took I think the normal clk framework will behave >>>>> as such dispatcher and clock gets tied to cru in SPL. >>>>> >>>>> #ifdef CONFIG_SPL_BUILD >>>>> -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs >>>>> #else >>>>> -> bind protocol@14 to scmi clk driver -> send SCMI message >>>>> #endif >>>> >>>> This looks right, just that we would have to bring a lot of bloat to SPL >>>> , firmware subsystem and things we may not want. >>> >>> I don't see why this would bring in a lot of bloat into SPL. The SPL >>> device tree will grow a little bit since it will have to include the >>> scmi nodes. And a little bit of additional code in the rk3588 clock >>> driver. Navigating the Kconfig stuff is a bit hard, but I don't think >>> this needs to pull in the firmware subsystem in SPL. >> >> When I tried this, I noticed that only the scmi agents look inside the >> subnodes (e.g. protocol@14) which do not have a compatible, and then >> they bind the subnodes to a driver found by a hardcoded search for >> 'scmi_clock'. >> So, to get there, an agent is required, and firmware node, probed >> firmware, probed 'scmi clock' . Then, probing it requires additional >> things, because the agent wants a shared memory 'shmem' node, and you >> end up also probing a SRAM area, allocating it to the space... that's >> what I mean when I say that there might be much more bloat added than >> just an additional clock set/get for securecru clocks. > > No, Jonas added code to find the clock protocol node and bind the > clock driver to it: > > https://github.com/Kwiboo/u-boot-rockchip/blob/3209167d7a518291f912964186ee90f10b555084/drivers/clk/rockchip/clk_rk3588.c#L1953 While this works, I somehow dislike it. I think the parent of the protocol@14 should search and bind the node to a driver, and not the clock driver. I find it odd and might lead to weird situations, e.g. disable the cru node or driver, then a node under firmware will no longer be probed. I think we should respect the parent-child order of probing and not searching through the DT for other nodes and mess with them. > >>>>> Naming and placement of the SPL securecru driver could be improved. >>>>> Not sure if any other soc beside rk3588 will need this at this moment. >>>> >>>> securecru sounds better for me, I don't think we can have a scmi clock >>>> fake driver like in vendor uboot, it has nothing 'scmi' about it. >>>> >>>>>> Meanwhile, I will test your patches to see how they work on my setup, I >>>>>> also have some things in progress including pinctrl in SPL for the rock5b. >>>>>> Thanks for your detailed description, >>>>> >>>>> Thanks for the hint at pinctrl, I made some updates to [5] and could verify >>>>> that sdmmc works when booting from emmc thanks to your pinctrl commit. >>>>> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe >>>>> and build a spi firmware image. >>>>> >>>>> There is an issue with non-DMA access in sdhci that requires the HACK commit >>>>> for proper loading of atf, was hoping to disable SDMA in SPL like what is done >>>>> using fifo-mode for sdmmc. But only first sector is read successfully then it >>>>> fails, see below. Will take a closer look before I post eMMC series. >>>> >>>> In your patches, I see this : >>>> https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 >>>> Perhaps the fifo-mode is intended for the sdhci(emmc) and not for >>>> sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, >>>> but the sdmmc can ? >>>> one possible workaround is to have DMA to DRAM and then relocate it to >>>> SRAM using the CPU. Having DMA disabled for the whole IP may have >>>> downsides, but this is U-boot, we don't expect to have anything else to >>>> do with the CPU while the DMA master works its magic, and the CPU should >>>> be faster. >>>> >>>> I tested your patches together with the series I sent now >>>> ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to >>>> boot correctly from SD-Card >>>> Once you send them to the ML I can retest them. >>>> Thanks ! >>>> >>>>> >>>>> Trying to boot from MMC2 >>>>> 1 >>>>> - 0 'mmc@fe2c0000' >>>>> - 1 'mmc@fe2e0000' >>>>> - found >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>> clock is disabled (0Hz) >>>>> clock is enabled (400000Hz) >>>>> clk_set_rate(clk=5000d8, rate=400000) >>>>> size=200, ptr=3b8, limit=100000: 5001b8 >>>>> clock is enabled (25000000Hz) >>>>> clk_set_rate(clk=5000d8, rate=25000000) >>>>> clk_set_rate(clk=5000d8, rate=25000000) >>>>> clock is enabled (52000000Hz) >>>>> clk_set_rate(clk=5000d8, rate=52000000) >>>>> spl: mmc boot mode: raw >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>> hdr read sector 4000, count=1 >>>>> Found FIT >>>>> size=a00, ptr=dc0, limit=100000: aligned to 5003c0 >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>> fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 >>>>> mmc_load_image_raw_sector: mmc block read error >>>>> spl: mmc boot mode: fs >>>>> Trying to boot from MMC1 >>>>> >>>>> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc >>>>> >>>>> Regards, >>>>> Jonas >>>>> >>>>> >>>>>> Eugen >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Jonas >>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Eugen >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Jonas >>>>>>>>> >>>>>>>>>> >>>>>>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>>>>>>> >>>>>>>>>> Jagan. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >> >>
On 2023-03-13 16:49, Eugen Hristev wrote: > On 3/13/23 17:34, Mark Kettenis wrote: >>> Date: Mon, 13 Mar 2023 17:21:05 +0200 >>> From: Eugen Hristev <eugen.hristev@collabora.com> >>> >>> On 3/13/23 17:07, Mark Kettenis wrote: >>>>> Date: Mon, 13 Mar 2023 16:21:36 +0200 >>>>> From: Eugen Hristev <eugen.hristev@collabora.com> >>>>> >>>>> On 3/13/23 12:00, Jonas Karlman wrote: >>>>>> On 2023-03-13 09:42, Eugen Hristev wrote: >>>>>>> On 3/13/23 00:34, Jonas Karlman wrote: >>>>>>>> Hi Eugen, >>>>>>>> >>>>>>>> On 2023-03-08 09:57, Eugen Hristev wrote: >>>>>>>>> On 1/29/23 11:04, Jonas Karlman wrote: >>>>>>>>>> On 2023-01-27 14:21, Jagan Teki wrote: >>>>>>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote: >>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote: >>>>>>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote: >>>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote: >>>>>>>>>>>>>>>>>>>> Hi Jonas, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Jagan, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote: >>>>>>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are >>>>>>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit >>>>>>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so >>>>>>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Verified below BL31 versions, >>>>>>>>>>>>>>>>>>>>>> bl31-v1.15 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.21 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.22 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.23 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.24 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.25 >>>>>>>>>>>>>>>>>>>>>> bl31-v1.26 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> < snip > >>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> As you can see in the logs above there is timeout waiting for data. >>>>>>>>>>>>> >>>>>>>>>>>>> I managed to find the issue and have a workaround that gets me longer >>>>>>>>>>>>> in the boot process, there still seem to be other issue with the rk3588 >>>>>>>>>>>>> startup. >>>>>>>>>>>>> >>>>>>>>>>>>> -------- >>>>>>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000) >>>>>>>>>>>>> Trying to boot from MMC1 >>>>>>>>>>>>> ## Checking hash(es) for config config-1 ... OK >>>>>>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK >>>>>>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK >>>>>>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK >>>>>>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK >>>>>>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK >>>>>>>>>>>>> INFO: Preloader serial: 2 >>>>>>>>>>>>> NOTICE: BL31: v2.3():v2.3-468-ge529a2760:derrick.huang >>>>>>>>>>>>> NOTICE: BL31: Built : 09:59:49, Nov 21 2022 >>>>>>>>>>>>> INFO: spec: 0x1 >>>>>>>>>>>>> INFO: ext 32k is not valid >>>>>>>>>>>>> INFO: ddr: stride-en 4CH >>>>>>>>>>>>> INFO: GICv3 without legacy support detected. >>>>>>>>>>>>> INFO: ARM GICv3 driver initialized in EL3 >>>>>>>>>>>>> INFO: valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0 >>>>>>>>>>>>> INFO: system boots from cpu-hwid-0 >>>>>>>>>>>>> INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 >>>>>>>>>>>>> INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz >>>>>>>>>>>>> INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz >>>>>>>>>>>>> INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz >>>>>>>>>>>>> INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz >>>>>>>>>>>>> INFO: BL31: Initialising Exception Handling Framework >>>>>>>>>>>>> INFO: BL31: Initializing runtime services >>>>>>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK >>>>>>>>>>>>> ERROR: Error initializing runtime service opteed_fast >>>>>>>>>>>>> INFO: BL31: Preparing for EL3 exit to normal world >>>>>>>>>>>>> INFO: Entry point address = 0xa00000 >>>>>>>>>>>>> INFO: SPSR = 0x3c9 >>>>>>>>>>>>> "Synchronous Abort" handler, esr 0x96000000 >>>>>>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c >>>>>>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3 >>>>>>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0 >>>>>>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88 >>>>>>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044 >>>>>>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000 >>>>>>>>>>>>> x10: 0000000000000034 x11: 0000000000000002 >>>>>>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc >>>>>>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808 >>>>>>>>>>>>> x16: 0000000000000000 x17: 0000000000000000 >>>>>>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8 >>>>>>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00 >>>>>>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000 >>>>>>>>>>>>> x24: 0000000000000000 x25: 0000000000000000 >>>>>>>>>>>>> x26: 0000000000000000 x27: 0000000000000000 >>>>>>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80 >>>>>>>>>>>>> >>>>>>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435) >>>>>>>>>>>>> Resetting CPU ... >>>>>>>>>>>>> -------- >>>>>>>>>>>>> >>>>>>>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e >>>>>>>>>>>>> with some rk3568 patches and this series, see [1]. >>>>>>>>>>>>> >>>>>>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock. >>>>>>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a >>>>>>>>>>>>> fallback to default 400khz clock. >>>>>>>>>>>>> >>>>>>>>>>>>> Next issue is the sync abort, looks it happens when u-boot >>>>>>>>>>>>> tries to set clock rates based on devicetree. this is the >>>>>>>>>>>>> last debug line before the crash. >>>>>>>>>>>>> >>>>>>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000) >>>>>>>>>>>> >>>>>>>>>>>> With the commit at [2] I can now successfully run U-Boot proper. >>>>>>>>>>>> >>>>>>>>>>>> Source of the two main issues to get this series to run have been the scmi clocks. >>>>>>>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks. >>>>>>>>>>>> We can just remove it and leave setting a faster cpu rate to linux. >>>>>>>>>>>> >>>>>>>>>>>> I also noticed that my sdram size series only detect the first two channels of memory, >>>>>>>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory. >>>>>>>>>>>> >>>>>>>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3 >>>>>>>>>>>> >>>>>>>>>>> Okay. We need to find a way to handle the clock value 400Khz >>>>>>>>>>> generically via the CLK framework or eMMC can be worth checking as it >>>>>>>>>>> doesn't involve SCMI and have a working patch set before MW. I did >>>>>>>>>>> that and was able to detect eMMC in U-Boot proper but got some issues >>>>>>>>>>> while booting from eMMC [3] >>>>>>>>>> >>>>>>>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci. >>>>>>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B. >>>>>>>>>> >>>>>>>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc >>>>>>>>>> and I have included some rk3568 patches and your rk3588 rfc series. >>>>>>>>>> I added a few fixup on top of that and a few additional patches, please see commit message >>>>>>>>>> for a very brief note on why the change was needed. >>>>>>>>>> Feel free to squash fixups and pick commits up to and possible including >>>>>>>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig" >>>>>>>>>> for a v2 of this series. >>>>>>>>>> >>>>>>>>>> The remaining sdhci patches needs a little bit more work, >>>>>>>>>> I can send a separate series with emmc patches once they are fully ready. >>>>>>>>>> >>>>>>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT. >>>>>>>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram. >>>>>>>>>> I think this is a more appropriate way to work around this issue, instead of patching >>>>>>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case. >>>>>>>>>> >>>>>>>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2 >>>>>>>>>> >>>>>>>>> Hello Jagan, Jonas, >>>>>>>>> >>>>>>>>> I wanted to chip into this discussion, to ask whether you did anything >>>>>>>>> more on the SD clock matter ? >>>>>>>> >>>>>>>> I have been busy this past week but have now had time to take a new look >>>>>>>> at the sdmmc issue, along with completing some other fixes. >>>>>>>> >>>>>>>>> >>>>>>>>> I am currently using your workaround that creates dummy clocks in the DT >>>>>>>>> and then the SD node just uses those, and it works, in the SPL, if and >>>>>>>>> only if the bootrom also loads the SPL from SD. >>>>>>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using >>>>>>>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails >>>>>>>>> utterly, namely, it cannot communicate with the card. >>>>>>>>> I did some changes to have the pinctrl working at SPL level, which >>>>>>>>> appears to be fine, but I would like to see what can be done about the >>>>>>>>> clocks. >>>>>>>>> Having the cru and all the required drivers at SPL level is the way to >>>>>>>>> go ? The SPL should run before SCMI is required so it should be able to >>>>>>>>> change all the clocks at the clock controller level ? >>>>>>>> >>>>>>>> I fully agree that we should have some sort of scmi clk driver so that >>>>>>>> we can control the sdmmc clk in both SPL and U-Boot proper. >>>>>>>> >>>>>>>> As you have noticed the current workaround only works because bootrom >>>>>>>> leave the clocks in a working state after it has loaded TPL/SPL from >>>>>>>> the sdmmc device. When TPL/SPL is loaded from any other source it is >>>>>>>> not be possible to read from the sdmmc device in u-boot. >>>>>>>> >>>>>>>> After having played around with the scmi agent driver and being inspired >>>>>>>> by the dummy scmi clk driver in vendor u-boot I have managed to create >>>>>>>> something that could work. See top three commits from [5] for a working >>>>>>>> proof-of-concept. >>>>>>>> >>>>>>>> What I did was to enable the scmi agent driver for use in u-boot proper >>>>>>>> and keeping it disabled in SPL build. Then added a new scmi clk driver >>>>>>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk >>>>>>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops >>>>>>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should >>>>>>>> be able to configure the sdmmc clocks. >>>>>>>> >>>>>>>> The initial fixes commits in that branch should hit the list soon. >>>>>>>> Will send the sdmmc related commits once I have had some time to do >>>>>>>> more testing. >>>>>>>> >>>>>>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes >>>>>>>> >>>>>>> Hi Jonas, >>>>>>> >>>>>>> I managed to get it working in SPL as well, with a similar code inspired >>>>>>> from the vendor tree. >>>>>>> About the solution, I am not sure whether it should be called a scmi >>>>>>> clock, and whether it should be part of clk_rk3588 >>>>>>> Honestly by looking at the DT node, I find the description of the clock >>>>>>> tree wrong to begin with. >>>>>>> The SD should not have the 2 clocks tied to the scmi agent, because this >>>>>>> is true only if it's running in normal world with a secure world agent >>>>>>> that will act as a middle man. So if we are running at a higher EL, as >>>>>>> in the SPL case, the clocks should be tied directly to the CRU. >>>>>>> So the node itself is described bad IMO. >>>>>>> >>>>>>> Anyway, if the node is set in stone from Linux, there isn;t much we can do. >>>>>>> I am also thinking whether we should have the SCMI enabled in SPL as >>>>>>> well, but the agent/clock should just access the CRU directly, meaning >>>>>>> to do something at the agent level : >>>>>>> #ifdef CONFIG_SPL_BUILD >>>>>>> -> access CRU >>>>>>> #else >>>>>>> -> send SCMI message >>>>>>> #endif >>>>>>> Does this make more sense ? >>>>>>> >>>>>>> Or have some kind of wrapper driver that would act as a dispatcher >>>>>>> depending on the SPL/proper build ? >>>>>> >>>>>> Looks like the sdmmc clock is controlled by the securecru registers and I >>>>>> expect that this can only be configured in secure world, not sure how this >>>>>> could have been modeled differently. >>>>> >>>>> Since it can work with a clock given by either scmi or securecru, I >>>>> would expect all clocks to be in the list, and the driver could start >>>>> the needed clock as per the EL level it's running into :) >>>>> >>>>> e.g. >>>>> clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi >>>>> HCLK_SD> , <&cru basic other clocks..> >>>>> >>>>> Just my vision of how it could be modeled >>>> >>>> That is a change to the device tree bindings. The goal is to have >>>> U-Boot use the same bindings and device tree as the official device >>>> tree (which currently is the one in mainline Linux). >>> >>> That's right. A change in the bindings (and Linux ) >> >> And OpenBSD, and NetBSD, and ... >> >> The device tree bindings are ABI. You can't change them unless they >> are really relly broken. > > For me, it's broken. You can't describe an SD hardware block, with > clocks given by SCMI node . It's not right. It's not like this in hardware. > It *can* be through SCMI, but it can also be via another way. > And the purpose of the ABI is to describe the hardware. > Unless we describe all of it, the description is wrong or at least > incomplete. > >> >>>>>> With the approach that I took I think the normal clk framework will behave >>>>>> as such dispatcher and clock gets tied to cru in SPL. >>>>>> >>>>>> #ifdef CONFIG_SPL_BUILD >>>>>> -> bind protocol@14 to rk3588 securecru driver -> read/write securecru regs >>>>>> #else >>>>>> -> bind protocol@14 to scmi clk driver -> send SCMI message >>>>>> #endif >>>>> >>>>> This looks right, just that we would have to bring a lot of bloat to SPL >>>>> , firmware subsystem and things we may not want. >>>> >>>> I don't see why this would bring in a lot of bloat into SPL. The SPL >>>> device tree will grow a little bit since it will have to include the >>>> scmi nodes. And a little bit of additional code in the rk3588 clock >>>> driver. Navigating the Kconfig stuff is a bit hard, but I don't think >>>> this needs to pull in the firmware subsystem in SPL. >>> >>> When I tried this, I noticed that only the scmi agents look inside the >>> subnodes (e.g. protocol@14) which do not have a compatible, and then >>> they bind the subnodes to a driver found by a hardcoded search for >>> 'scmi_clock'. >>> So, to get there, an agent is required, and firmware node, probed >>> firmware, probed 'scmi clock' . Then, probing it requires additional >>> things, because the agent wants a shared memory 'shmem' node, and you >>> end up also probing a SRAM area, allocating it to the space... that's >>> what I mean when I say that there might be much more bloat added than >>> just an additional clock set/get for securecru clocks. >> >> No, Jonas added code to find the clock protocol node and bind the >> clock driver to it: >> >> https://github.com/Kwiboo/u-boot-rockchip/blob/3209167d7a518291f912964186ee90f10b555084/drivers/clk/rockchip/clk_rk3588.c#L1953 >> > While this works, I somehow dislike it. > I think the parent of the protocol@14 should search and bind the node to > a driver, and not the clock driver. > > I find it odd and might lead to weird situations, e.g. disable the cru > node or driver, then a node under firmware will no longer be probed. > I think we should respect the parent-child order of probing and not > searching through the DT for other nodes and mess with them. Initially I also tried to re-use existing scmi agent but that had other issues and felt like a big hack. Another approach could be to implement a stub scmi agent driver with same purpose as the function, to probe and bind protocol subnode to a secturecru driver. However, that just adds another driver and options into the mix, use of a more simple function seemed like a good compromise. > >> >>>>>> Naming and placement of the SPL securecru driver could be improved. >>>>>> Not sure if any other soc beside rk3588 will need this at this moment. >>>>> >>>>> securecru sounds better for me, I don't think we can have a scmi clock >>>>> fake driver like in vendor uboot, it has nothing 'scmi' about it. >>>>> >>>>>>> Meanwhile, I will test your patches to see how they work on my setup, I >>>>>>> also have some things in progress including pinctrl in SPL for the rock5b. >>>>>>> Thanks for your detailed description, >>>>>> >>>>>> Thanks for the hint at pinctrl, I made some updates to [5] and could verify >>>>>> that sdmmc works when booting from emmc thanks to your pinctrl commit. >>>>>> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe >>>>>> and build a spi firmware image. >>>>>> >>>>>> There is an issue with non-DMA access in sdhci that requires the HACK commit >>>>>> for proper loading of atf, was hoping to disable SDMA in SPL like what is done >>>>>> using fifo-mode for sdmmc. But only first sector is read successfully then it >>>>>> fails, see below. Will take a closer look before I post eMMC series. >>>>> >>>>> In your patches, I see this : >>>>> https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61 >>>>> Perhaps the fifo-mode is intended for the sdhci(emmc) and not for >>>>> sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM, >>>>> but the sdmmc can ? This was intended for the sdmmc, I had issues loading atf from sdmmc until I added this. Will re-test on top of pinctrl and other patches. It is unclear to me what the exact limitation is in regard to DMA to SRAM, on rk3568 the firewall is programmed to allow DMA to SRAM for both sdmmc and emmc. >>>>> one possible workaround is to have DMA to DRAM and then relocate it to >>>>> SRAM using the CPU. Having DMA disabled for the whole IP may have >>>>> downsides, but this is U-boot, we don't expect to have anything else to >>>>> do with the CPU while the DMA master works its magic, and the CPU should >>>>> be faster. That is how it is done in vendor u-boot, it use a bounce buffer in spl fit load code, it has been submitted in different forms in the past. I have tested a few different approaches, but usually go back to just disable use of DMA in SPL using existing flags/options. The fifo-mode flag seem to be specific to the dw_mmc driver, and using a SPL_MMC_SDHCI_SDMA Kconfig to disable SDMA in SPL seemed like similar approach. Yet, reading more then a single block failed in my testing. >>>>> >>>>> I tested your patches together with the series I sent now >>>>> ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to >>>>> boot correctly from SD-Card >>>>> Once you send them to the ML I can retest them. Thanks, I am hoping to be able to send something out later today and/or tomorrow. Regards, Jonas >>>>> Thanks ! >>>>> >>>>>> >>>>>> Trying to boot from MMC2 >>>>>> 1 >>>>>> - 0 'mmc@fe2c0000' >>>>>> - 1 'mmc@fe2e0000' >>>>>> - found >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>>> clock is disabled (0Hz) >>>>>> clock is enabled (400000Hz) >>>>>> clk_set_rate(clk=5000d8, rate=400000) >>>>>> size=200, ptr=3b8, limit=100000: 5001b8 >>>>>> clock is enabled (25000000Hz) >>>>>> clk_set_rate(clk=5000d8, rate=25000000) >>>>>> clk_set_rate(clk=5000d8, rate=25000000) >>>>>> clock is enabled (52000000Hz) >>>>>> clk_set_rate(clk=5000d8, rate=52000000) >>>>>> spl: mmc boot mode: raw >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>>> hdr read sector 4000, count=1 >>>>>> Found FIT >>>>>> size=a00, ptr=dc0, limit=100000: aligned to 5003c0 >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2c0000.blk, 67, 0 >>>>>> blk_find_device: uclass_id=67, devnum=1: mmc@fe2e0000.blk, 67, 1 >>>>>> fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00 >>>>>> mmc_load_image_raw_sector: mmc block read error >>>>>> spl: mmc boot mode: fs >>>>>> Trying to boot from MMC1 >>>>>> >>>>>> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc >>>>>> >>>>>> Regards, >>>>>> Jonas >>>>>> >>>>>> >>>>>>> Eugen >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Jonas >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Eugen >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Jonas >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f >>>>>>>>>>> >>>>>>>>>>> Jagan. >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>> >>> >