| Message ID | 20180223133742.26044-11-mylene.josserand@bootlin.com |
|---|---|
| State | New |
| Headers | show |
| Series | Sunxi: Add SMP support on A83T | expand |
On Fri, Feb 23, 2018 at 02:37:42PM +0100, Mylène Josserand wrote: > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > It should be done by the bootloader but it is currently not the case, > even for boot CPU because this SoC is booting in secure mode. > It leads to an random offset value meaning that each CPU will have a > different time, which isn't working very well. > > Add assembly code used for boot CPU and secondary CPU cores to make > sure that the CNTVOFF register is initialized. > > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > --- > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > 2 files changed, 25 insertions(+) > > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > index d5c97e945e69..605896251927 100644 > --- a/arch/arm/mach-sunxi/headsmp.S > +++ b/arch/arm/mach-sunxi/headsmp.S > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > first: .word sunxi_mc_smp_first_comer - . > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > > +ENTRY(sunxi_init_cntvoff) > + /* > + * CNTVOFF has to be initialized either from non-secure Hypervisor > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > + * then it should be handled by the secure code > + */ > + cps #MON_MODE > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > + orr r0, r1, #1 > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > + instr_sync > + mov r0, #0 > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > + instr_sync > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > + instr_sync > + cps #SVC_MODE > + ret lr > +ENDPROC(sunxi_init_cntvoff) > + > #ifdef CONFIG_SMP > ENTRY(sunxi_boot) > bl sunxi_mc_smp_cluster_cache_enable > + bl sunxi_init_cntvoff > b secondary_startup > ENDPROC(sunxi_boot) > > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > index 5e9602ce1573..4bb041492b54 100644 > --- a/arch/arm/mach-sunxi/sunxi.c > +++ b/arch/arm/mach-sunxi/sunxi.c > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > }; > > extern void __init sun6i_reset_init(void); > +extern void sunxi_init_cntvoff(void); > + > static void __init sun6i_timer_init(void) > { > + sunxi_init_cntvoff(); > + This would deserve a comment explaining why this is needed in the first place, and this is not correct. All the other SoCs listed in that machine have their CNTVOFF setup correctly on CPU0, and since Linux is booted in HYP, it's probably not even valid to do that. Maxime
On Fri, Feb 23, 2018 at 9:37 PM, Mylène Josserand <mylene.josserand@bootlin.com> wrote: > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > It should be done by the bootloader but it is currently not the case, > even for boot CPU because this SoC is booting in secure mode. > It leads to an random offset value meaning that each CPU will have a > different time, which isn't working very well. > > Add assembly code used for boot CPU and secondary CPU cores to make > sure that the CNTVOFF register is initialized. > > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > --- > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > 2 files changed, 25 insertions(+) > > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > index d5c97e945e69..605896251927 100644 > --- a/arch/arm/mach-sunxi/headsmp.S > +++ b/arch/arm/mach-sunxi/headsmp.S > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > first: .word sunxi_mc_smp_first_comer - . > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > > +ENTRY(sunxi_init_cntvoff) > + /* > + * CNTVOFF has to be initialized either from non-secure Hypervisor > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > + * then it should be handled by the secure code > + */ > + cps #MON_MODE > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > + orr r0, r1, #1 > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > + instr_sync > + mov r0, #0 > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > + instr_sync > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > + instr_sync > + cps #SVC_MODE > + ret lr > +ENDPROC(sunxi_init_cntvoff) There is no need to move all the assembly into a separate file, just to add this function. Everything can be inlined as a naked function. The "instr_sync" macro can be replaced with "isb", which is what it expands to anyway. I really want to keep everything self-contained without global symbols, and in C files if possible. > + > #ifdef CONFIG_SMP > ENTRY(sunxi_boot) > bl sunxi_mc_smp_cluster_cache_enable > + bl sunxi_init_cntvoff > b secondary_startup > ENDPROC(sunxi_boot) > > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > index 5e9602ce1573..4bb041492b54 100644 > --- a/arch/arm/mach-sunxi/sunxi.c > +++ b/arch/arm/mach-sunxi/sunxi.c > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > }; > > extern void __init sun6i_reset_init(void); > +extern void sunxi_init_cntvoff(void); > + > static void __init sun6i_timer_init(void) > { > + sunxi_init_cntvoff(); You should check the enable-method to see if PSCI is set or not, as an indicator whether the kernel is booted secure or non-secure. AFAIK trying to set CNTVOFF under non-secure would be very bad. Regards ChenYu > + > of_clk_init(NULL); > if (IS_ENABLED(CONFIG_RESET_CONTROLLER)) > sun6i_reset_init(); > -- > 2.11.0 >
On Sat, Feb 24, 2018 at 12:17:13AM +0800, Chen-Yu Tsai wrote: > On Fri, Feb 23, 2018 at 9:37 PM, Mylène Josserand > <mylene.josserand@bootlin.com> wrote: > > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > > It should be done by the bootloader but it is currently not the case, > > even for boot CPU because this SoC is booting in secure mode. > > It leads to an random offset value meaning that each CPU will have a > > different time, which isn't working very well. > > > > Add assembly code used for boot CPU and secondary CPU cores to make > > sure that the CNTVOFF register is initialized. > > > > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > > --- > > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > > 2 files changed, 25 insertions(+) > > > > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > > index d5c97e945e69..605896251927 100644 > > --- a/arch/arm/mach-sunxi/headsmp.S > > +++ b/arch/arm/mach-sunxi/headsmp.S > > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > > first: .word sunxi_mc_smp_first_comer - . > > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > > > > +ENTRY(sunxi_init_cntvoff) > > + /* > > + * CNTVOFF has to be initialized either from non-secure Hypervisor > > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > > + * then it should be handled by the secure code > > + */ > > + cps #MON_MODE > > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > > + orr r0, r1, #1 > > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > > + instr_sync > > + mov r0, #0 > > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > > + instr_sync > > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > > + instr_sync > > + cps #SVC_MODE > > + ret lr > > +ENDPROC(sunxi_init_cntvoff) > > There is no need to move all the assembly into a separate file, just > to add this function. Everything can be inlined as a naked function. > The "instr_sync" macro can be replaced with "isb", which is what it > expands to anyway. > > I really want to keep everything self-contained without global symbols, > and in C files if possible. What is the rationale for keeping it in C files (beside the global symbols)? Because the syntax is quite ugly, and it's much easier to read, review and amend using a separate file. > > #ifdef CONFIG_SMP > > ENTRY(sunxi_boot) > > bl sunxi_mc_smp_cluster_cache_enable > > + bl sunxi_init_cntvoff > > b secondary_startup > > ENDPROC(sunxi_boot) > > > > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > > index 5e9602ce1573..4bb041492b54 100644 > > --- a/arch/arm/mach-sunxi/sunxi.c > > +++ b/arch/arm/mach-sunxi/sunxi.c > > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > > }; > > > > extern void __init sun6i_reset_init(void); > > +extern void sunxi_init_cntvoff(void); > > + > > static void __init sun6i_timer_init(void) > > { > > + sunxi_init_cntvoff(); > > You should check the enable-method to see if PSCI is set or not, > as an indicator whether the kernel is booted secure or non-secure. It's an indicator, but it's not really a perfect one. You could very well have your kernel booted in non-secure, without PSCI. Or even with PSCI, but without the SMP ops. We have a quite big number of these cases already, where, depending on the configuration, we might not have access to the device we write to, the number of hacks to just enable that device for non-secure is a good example of that. > AFAIK trying to set CNTVOFF under non-secure would be very bad. Just like any other access we do :/ Maxime
On Mon, Feb 26, 2018 at 6:12 PM, Maxime Ripard <maxime.ripard@bootlin.com> wrote: > On Sat, Feb 24, 2018 at 12:17:13AM +0800, Chen-Yu Tsai wrote: >> On Fri, Feb 23, 2018 at 9:37 PM, Mylène Josserand >> <mylene.josserand@bootlin.com> wrote: >> > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. >> > It should be done by the bootloader but it is currently not the case, >> > even for boot CPU because this SoC is booting in secure mode. >> > It leads to an random offset value meaning that each CPU will have a >> > different time, which isn't working very well. >> > >> > Add assembly code used for boot CPU and secondary CPU cores to make >> > sure that the CNTVOFF register is initialized. >> > >> > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> >> > --- >> > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ >> > arch/arm/mach-sunxi/sunxi.c | 4 ++++ >> > 2 files changed, 25 insertions(+) >> > >> > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S >> > index d5c97e945e69..605896251927 100644 >> > --- a/arch/arm/mach-sunxi/headsmp.S >> > +++ b/arch/arm/mach-sunxi/headsmp.S >> > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) >> > first: .word sunxi_mc_smp_first_comer - . >> > ENDPROC(sunxi_mc_smp_cluster_cache_enable) >> > >> > +ENTRY(sunxi_init_cntvoff) >> > + /* >> > + * CNTVOFF has to be initialized either from non-secure Hypervisor >> > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled >> > + * then it should be handled by the secure code >> > + */ >> > + cps #MON_MODE >> > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ >> > + orr r0, r1, #1 >> > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ >> > + instr_sync >> > + mov r0, #0 >> > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ >> > + instr_sync >> > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ >> > + instr_sync >> > + cps #SVC_MODE >> > + ret lr >> > +ENDPROC(sunxi_init_cntvoff) >> >> There is no need to move all the assembly into a separate file, just >> to add this function. Everything can be inlined as a naked function. >> The "instr_sync" macro can be replaced with "isb", which is what it >> expands to anyway. >> >> I really want to keep everything self-contained without global symbols, >> and in C files if possible. > > What is the rationale for keeping it in C files (beside the global > symbols)? Because the syntax is quite ugly, and it's much easier to > read, review and amend using a separate file. Global symbols and keeping everything in one place I guess. This symbol would be used in a few places, so I suppose having it in a separate assembly file would be OK. >> > #ifdef CONFIG_SMP >> > ENTRY(sunxi_boot) >> > bl sunxi_mc_smp_cluster_cache_enable >> > + bl sunxi_init_cntvoff >> > b secondary_startup >> > ENDPROC(sunxi_boot) >> > >> > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c >> > index 5e9602ce1573..4bb041492b54 100644 >> > --- a/arch/arm/mach-sunxi/sunxi.c >> > +++ b/arch/arm/mach-sunxi/sunxi.c >> > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { >> > }; >> > >> > extern void __init sun6i_reset_init(void); >> > +extern void sunxi_init_cntvoff(void); >> > + >> > static void __init sun6i_timer_init(void) >> > { >> > + sunxi_init_cntvoff(); >> >> You should check the enable-method to see if PSCI is set or not, >> as an indicator whether the kernel is booted secure or non-secure. > > It's an indicator, but it's not really a perfect one. You could very > well have your kernel booted in non-secure, without PSCI. Or even with > PSCI, but without the SMP ops. > > We have a quite big number of these cases already, where, depending on > the configuration, we might not have access to the device we write to, > the number of hacks to just enable that device for non-secure is a > good example of that. I wouldn't consider them hacks though. The hardware gives the option to have control of many devices delegated solely to secure-only, or secure/non-secure. Our present model is to support everything we can in Linux directly, instead of through some firmware interface to a non-existent firmware. ChenYu >> AFAIK trying to set CNTVOFF under non-secure would be very bad. > > Just like any other access we do :/ > > Maxime > > -- > Maxime Ripard, Bootlin (formerly Free Electrons) > Embedded Linux and Kernel engineering > https://bootlin.com
Hello, On Mon, 26 Feb 2018 18:25:10 +0800 Chen-Yu Tsai <wens@csie.org> wrote: > On Mon, Feb 26, 2018 at 6:12 PM, Maxime Ripard > <maxime.ripard@bootlin.com> wrote: > > On Sat, Feb 24, 2018 at 12:17:13AM +0800, Chen-Yu Tsai wrote: > >> On Fri, Feb 23, 2018 at 9:37 PM, Mylène Josserand > >> <mylene.josserand@bootlin.com> wrote: > >> > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > >> > It should be done by the bootloader but it is currently not the case, > >> > even for boot CPU because this SoC is booting in secure mode. > >> > It leads to an random offset value meaning that each CPU will have a > >> > different time, which isn't working very well. > >> > > >> > Add assembly code used for boot CPU and secondary CPU cores to make > >> > sure that the CNTVOFF register is initialized. > >> > > >> > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > >> > --- > >> > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > >> > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > >> > 2 files changed, 25 insertions(+) > >> > > >> > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > >> > index d5c97e945e69..605896251927 100644 > >> > --- a/arch/arm/mach-sunxi/headsmp.S > >> > +++ b/arch/arm/mach-sunxi/headsmp.S > >> > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > >> > first: .word sunxi_mc_smp_first_comer - . > >> > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > >> > > >> > +ENTRY(sunxi_init_cntvoff) > >> > + /* > >> > + * CNTVOFF has to be initialized either from non-secure Hypervisor > >> > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > >> > + * then it should be handled by the secure code > >> > + */ > >> > + cps #MON_MODE > >> > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > >> > + orr r0, r1, #1 > >> > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > >> > + instr_sync > >> > + mov r0, #0 > >> > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > >> > + instr_sync > >> > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > >> > + instr_sync > >> > + cps #SVC_MODE > >> > + ret lr > >> > +ENDPROC(sunxi_init_cntvoff) > >> > >> There is no need to move all the assembly into a separate file, just > >> to add this function. Everything can be inlined as a naked function. > >> The "instr_sync" macro can be replaced with "isb", which is what it > >> expands to anyway. > >> > >> I really want to keep everything self-contained without global symbols, > >> and in C files if possible. > > > > What is the rationale for keeping it in C files (beside the global > > symbols)? Because the syntax is quite ugly, and it's much easier to > > read, review and amend using a separate file. > > Global symbols and keeping everything in one place I guess. > This symbol would be used in a few places, so I suppose having it > in a separate assembly file would be OK. Okay so I will keep it in a separate file. > > >> > #ifdef CONFIG_SMP > >> > ENTRY(sunxi_boot) > >> > bl sunxi_mc_smp_cluster_cache_enable > >> > + bl sunxi_init_cntvoff > >> > b secondary_startup > >> > ENDPROC(sunxi_boot) > >> > > >> > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > >> > index 5e9602ce1573..4bb041492b54 100644 > >> > --- a/arch/arm/mach-sunxi/sunxi.c > >> > +++ b/arch/arm/mach-sunxi/sunxi.c > >> > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > >> > }; > >> > > >> > extern void __init sun6i_reset_init(void); > >> > +extern void sunxi_init_cntvoff(void); > >> > + > >> > static void __init sun6i_timer_init(void) > >> > { > >> > + sunxi_init_cntvoff(); > >> > >> You should check the enable-method to see if PSCI is set or not, > >> as an indicator whether the kernel is booted secure or non-secure. > > > > It's an indicator, but it's not really a perfect one. You could very > > well have your kernel booted in non-secure, without PSCI. Or even with > > PSCI, but without the SMP ops. > > > > We have a quite big number of these cases already, where, depending on > > the configuration, we might not have access to the device we write to, > > the number of hacks to just enable that device for non-secure is a > > good example of that. > > I wouldn't consider them hacks though. The hardware gives the option > to have control of many devices delegated solely to secure-only, or > secure/non-secure. Our present model is to support everything we can > in Linux directly, instead of through some firmware interface to a > non-existent firmware. I am not sure to understand what is the conclusion about it. Should I use "psci"/enable-method or should I use another mechanism to detect we are in secure/non-secure (if it exists)? Otherwise, for the moment, I can use machine-compatible on sun8i-a83t and we will see later how we can handle it in a better way. Thank you in advance, Best regards, Mylène
On Mon, Mar 05, 2018 at 08:51:48AM +0100, Mylène Josserand wrote: > > >> > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > > >> > index 5e9602ce1573..4bb041492b54 100644 > > >> > --- a/arch/arm/mach-sunxi/sunxi.c > > >> > +++ b/arch/arm/mach-sunxi/sunxi.c > > >> > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > > >> > }; > > >> > > > >> > extern void __init sun6i_reset_init(void); > > >> > +extern void sunxi_init_cntvoff(void); > > >> > + > > >> > static void __init sun6i_timer_init(void) > > >> > { > > >> > + sunxi_init_cntvoff(); > > >> > > >> You should check the enable-method to see if PSCI is set or not, > > >> as an indicator whether the kernel is booted secure or non-secure. > > > > > > It's an indicator, but it's not really a perfect one. You could very > > > well have your kernel booted in non-secure, without PSCI. Or even with > > > PSCI, but without the SMP ops. > > > > > > We have a quite big number of these cases already, where, depending on > > > the configuration, we might not have access to the device we write to, > > > the number of hacks to just enable that device for non-secure is a > > > good example of that. > > > > I wouldn't consider them hacks though. The hardware gives the option > > to have control of many devices delegated solely to secure-only, or > > secure/non-secure. Our present model is to support everything we can > > in Linux directly, instead of through some firmware interface to a > > non-existent firmware. > > I am not sure to understand what is the conclusion about it. > Should I use "psci"/enable-method or should I use another mechanism to > detect we are in secure/non-secure (if it exists)? > > Otherwise, for the moment, I can use machine-compatible on sun8i-a83t > and we will see later how we can handle it in a better way. Can't we have another approach here? If we use an enable-method (and we should), instead of having it tied to the machine compatible, the SMP setup code will run only if our enable-method is the one we set up. If PSCI is in use, the enable-method is not going to be the one defined here, and the code will not run. So why not just move that call to the SMP ops setup function, just like renesas does? Maxime
Hello, On Mon, 5 Mar 2018 09:31:14 +0100 Maxime Ripard <maxime.ripard@bootlin.com> wrote: > On Mon, Mar 05, 2018 at 08:51:48AM +0100, Mylène Josserand wrote: > > > >> > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > > > >> > index 5e9602ce1573..4bb041492b54 100644 > > > >> > --- a/arch/arm/mach-sunxi/sunxi.c > > > >> > +++ b/arch/arm/mach-sunxi/sunxi.c > > > >> > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > > > >> > }; > > > >> > > > > >> > extern void __init sun6i_reset_init(void); > > > >> > +extern void sunxi_init_cntvoff(void); > > > >> > + > > > >> > static void __init sun6i_timer_init(void) > > > >> > { > > > >> > + sunxi_init_cntvoff(); > > > >> > > > >> You should check the enable-method to see if PSCI is set or not, > > > >> as an indicator whether the kernel is booted secure or non-secure. > > > > > > > > It's an indicator, but it's not really a perfect one. You could very > > > > well have your kernel booted in non-secure, without PSCI. Or even with > > > > PSCI, but without the SMP ops. > > > > > > > > We have a quite big number of these cases already, where, depending on > > > > the configuration, we might not have access to the device we write to, > > > > the number of hacks to just enable that device for non-secure is a > > > > good example of that. > > > > > > I wouldn't consider them hacks though. The hardware gives the option > > > to have control of many devices delegated solely to secure-only, or > > > secure/non-secure. Our present model is to support everything we can > > > in Linux directly, instead of through some firmware interface to a > > > non-existent firmware. > > > > I am not sure to understand what is the conclusion about it. > > Should I use "psci"/enable-method or should I use another mechanism to > > detect we are in secure/non-secure (if it exists)? > > > > Otherwise, for the moment, I can use machine-compatible on sun8i-a83t > > and we will see later how we can handle it in a better way. > > Can't we have another approach here? > > If we use an enable-method (and we should), instead of having it tied > to the machine compatible, the SMP setup code will run only if our > enable-method is the one we set up. If PSCI is in use, the > enable-method is not going to be the one defined here, and the code > will not run. > > So why not just move that call to the SMP ops setup function, just > like renesas does? > > Maxime > Okay, I will update my series and handle the differences using enable-method instead of machine-compatible. Best regards,
On 23/02/18 16:17, Chen-Yu Tsai wrote: > On Fri, Feb 23, 2018 at 9:37 PM, Mylène Josserand > <mylene.josserand@bootlin.com> wrote: >> On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. >> It should be done by the bootloader but it is currently not the case, >> even for boot CPU because this SoC is booting in secure mode. >> It leads to an random offset value meaning that each CPU will have a >> different time, which isn't working very well. >> >> Add assembly code used for boot CPU and secondary CPU cores to make >> sure that the CNTVOFF register is initialized. >> >> Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> >> --- >> arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ >> arch/arm/mach-sunxi/sunxi.c | 4 ++++ >> 2 files changed, 25 insertions(+) >> >> diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S >> index d5c97e945e69..605896251927 100644 >> --- a/arch/arm/mach-sunxi/headsmp.S >> +++ b/arch/arm/mach-sunxi/headsmp.S >> @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) >> first: .word sunxi_mc_smp_first_comer - . >> ENDPROC(sunxi_mc_smp_cluster_cache_enable) >> >> +ENTRY(sunxi_init_cntvoff) >> + /* >> + * CNTVOFF has to be initialized either from non-secure Hypervisor >> + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled >> + * then it should be handled by the secure code >> + */ >> + cps #MON_MODE >> + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ >> + orr r0, r1, #1 >> + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ >> + instr_sync >> + mov r0, #0 >> + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ >> + instr_sync >> + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ >> + instr_sync >> + cps #SVC_MODE >> + ret lr >> +ENDPROC(sunxi_init_cntvoff) > > There is no need to move all the assembly into a separate file, just > to add this function. Everything can be inlined as a naked function. > The "instr_sync" macro can be replaced with "isb", which is what it > expands to anyway. > > I really want to keep everything self-contained without global symbols, > and in C files if possible. > >> + >> #ifdef CONFIG_SMP >> ENTRY(sunxi_boot) >> bl sunxi_mc_smp_cluster_cache_enable >> + bl sunxi_init_cntvoff >> b secondary_startup >> ENDPROC(sunxi_boot) >> >> diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c >> index 5e9602ce1573..4bb041492b54 100644 >> --- a/arch/arm/mach-sunxi/sunxi.c >> +++ b/arch/arm/mach-sunxi/sunxi.c >> @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { >> }; >> >> extern void __init sun6i_reset_init(void); >> +extern void sunxi_init_cntvoff(void); >> + >> static void __init sun6i_timer_init(void) >> { >> + sunxi_init_cntvoff(); > > You should check the enable-method to see if PSCI is set or not, > as an indicator whether the kernel is booted secure or non-secure. > AFAIK trying to set CNTVOFF under non-secure would be very bad. Absolutely not. CNTVOFF *is* a non-secure register. The fact that it is not accessible from NS-PL1 is another problem. Please don't conflate the two things together. Thanks, M.
On 23/02/18 13:37, Mylène Josserand wrote: > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. Only on A7? Is that specific to your platform? > It should be done by the bootloader but it is currently not the case, > even for boot CPU because this SoC is booting in secure mode. > It leads to an random offset value meaning that each CPU will have a > different time, which isn't working very well. > > Add assembly code used for boot CPU and secondary CPU cores to make > sure that the CNTVOFF register is initialized. > > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > --- > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > 2 files changed, 25 insertions(+) > > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > index d5c97e945e69..605896251927 100644 > --- a/arch/arm/mach-sunxi/headsmp.S > +++ b/arch/arm/mach-sunxi/headsmp.S > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > first: .word sunxi_mc_smp_first_comer - . > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > > +ENTRY(sunxi_init_cntvoff) > + /* > + * CNTVOFF has to be initialized either from non-secure Hypervisor > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > + * then it should be handled by the secure code > + */ > + cps #MON_MODE > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > + orr r0, r1, #1 > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > + instr_sync Since these CPUs are all ARMv7, you can use isb directly. Nobody wants to see more of the CP15 barriers. > + mov r0, #0 > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > + instr_sync > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > + instr_sync > + cps #SVC_MODE > + ret lr Given that this code is identical to the shmobile hack, it'd be good to make it common, one way or another. > +ENDPROC(sunxi_init_cntvoff) > + > #ifdef CONFIG_SMP > ENTRY(sunxi_boot) > bl sunxi_mc_smp_cluster_cache_enable > + bl sunxi_init_cntvoff > b secondary_startup > ENDPROC(sunxi_boot) > > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > index 5e9602ce1573..4bb041492b54 100644 > --- a/arch/arm/mach-sunxi/sunxi.c > +++ b/arch/arm/mach-sunxi/sunxi.c > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > }; > > extern void __init sun6i_reset_init(void); > +extern void sunxi_init_cntvoff(void); > + > static void __init sun6i_timer_init(void) > { > + sunxi_init_cntvoff(); > + It is slightly odd that some CPUs get initialized from the early asm code, and some others do a detour via some C code. I'm sure this could be made to work > of_clk_init(NULL); > if (IS_ENABLED(CONFIG_RESET_CONTROLLER)) > sun6i_reset_init(); > Thanks, M.
Hello Mark, Please, excuse me for this late answer and thank you for the review! On Wed, 7 Mar 2018 12:18:33 +0000 Marc Zyngier <marc.zyngier@arm.com> wrote: > On 23/02/18 13:37, Mylène Josserand wrote: > > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > > Only on A7? Is that specific to your platform? I do not really know other Allwinner's platforms about this subject. At least, the sun9i-a80 which is a Cortex-a15/a7 does not need that but it is necessary for sun8i-a83t which is a cortex-a7. Maybe, Chen-Yu or Maxime could help us on it. > > > It should be done by the bootloader but it is currently not the case, > > even for boot CPU because this SoC is booting in secure mode. > > It leads to an random offset value meaning that each CPU will have a > > different time, which isn't working very well. > > > > Add assembly code used for boot CPU and secondary CPU cores to make > > sure that the CNTVOFF register is initialized. > > > > Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> > > --- > > arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ > > arch/arm/mach-sunxi/sunxi.c | 4 ++++ > > 2 files changed, 25 insertions(+) > > > > diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S > > index d5c97e945e69..605896251927 100644 > > --- a/arch/arm/mach-sunxi/headsmp.S > > +++ b/arch/arm/mach-sunxi/headsmp.S > > @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) > > first: .word sunxi_mc_smp_first_comer - . > > ENDPROC(sunxi_mc_smp_cluster_cache_enable) > > > > +ENTRY(sunxi_init_cntvoff) > > + /* > > + * CNTVOFF has to be initialized either from non-secure Hypervisor > > + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled > > + * then it should be handled by the secure code > > + */ > > + cps #MON_MODE > > + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ > > + orr r0, r1, #1 > > + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ > > + instr_sync > > Since these CPUs are all ARMv7, you can use isb directly. Nobody wants > to see more of the CP15 barriers. Okay, thanks, I will update that. > > > + mov r0, #0 > > + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ > > + instr_sync > > + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ > > + instr_sync > > + cps #SVC_MODE > > + ret lr > > Given that this code is identical to the shmobile hack, it'd be good to > make it common, one way or another. Sure, I will try that. May you have some hints to give me on how to implement it? > > > +ENDPROC(sunxi_init_cntvoff) > > + > > #ifdef CONFIG_SMP > > ENTRY(sunxi_boot) > > bl sunxi_mc_smp_cluster_cache_enable > > + bl sunxi_init_cntvoff > > b secondary_startup > > ENDPROC(sunxi_boot) > > > > diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c > > index 5e9602ce1573..4bb041492b54 100644 > > --- a/arch/arm/mach-sunxi/sunxi.c > > +++ b/arch/arm/mach-sunxi/sunxi.c > > @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { > > }; > > > > extern void __init sun6i_reset_init(void); > > +extern void sunxi_init_cntvoff(void); > > + > > static void __init sun6i_timer_init(void) > > { > > + sunxi_init_cntvoff(); > > + > > It is slightly odd that some CPUs get initialized from the early asm > code, and some others do a detour via some C code. I'm sure this could > be made to work Renesa's code was also doing that so I thought it could be ok to do it. Without this code in this timer_init function, it fails to initialize cntvoff correctly: http://code.bulix.org/i9fhc9-302612?raw How should I implement that? > > > of_clk_init(NULL); > > if (IS_ENABLED(CONFIG_RESET_CONTROLLER)) > > sun6i_reset_init(); > > > > Thanks, > > M. Thanks, Best regards,
On Mon, Mar 19, 2018 at 3:07 AM, Mylène Josserand <mylene.josserand@bootlin.com> wrote: > Hello Mark, > > Please, excuse me for this late answer and thank you for the review! > > On Wed, 7 Mar 2018 12:18:33 +0000 > Marc Zyngier <marc.zyngier@arm.com> wrote: > >> On 23/02/18 13:37, Mylène Josserand wrote: >> > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. >> >> Only on A7? Is that specific to your platform? > > I do not really know other Allwinner's platforms about this subject. At > least, the sun9i-a80 which is a Cortex-a15/a7 does not need that but it > is necessary for sun8i-a83t which is a cortex-a7. Maybe, Chen-Yu or > Maxime could help us on it. AFAIK all Allwinner CPUs need it if there isn't a firmware (PSCI) layer beneath the kernel that will do the setup. We just have "arm,cpu-registers-not-fw-configured" set for all the other SoCs that have in-kernel SMP support, which includes the A31, A23, A33 and A80. ChenYu
Hi Mylène, On 18/03/18 19:07, Mylène Josserand wrote: > Hello Mark, > > Please, excuse me for this late answer and thank you for the review! No worries. > > On Wed, 7 Mar 2018 12:18:33 +0000 > Marc Zyngier <marc.zyngier@arm.com> wrote: > >> On 23/02/18 13:37, Mylène Josserand wrote: >>> On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. >> >> Only on A7? Is that specific to your platform? > > I do not really know other Allwinner's platforms about this subject. At > least, the sun9i-a80 which is a Cortex-a15/a7 does not need that but it > is necessary for sun8i-a83t which is a cortex-a7. Maybe, Chen-Yu or > Maxime could help us on it. My point was that having an uninitialized CNTVOFF is hardly a property of Cortex-A7. The same behaviour exists on all CPUs that implement the arch timers. What you have here is more a property of the platform, and its lack of firmware support. >> >>> It should be done by the bootloader but it is currently not the case, >>> even for boot CPU because this SoC is booting in secure mode. >>> It leads to an random offset value meaning that each CPU will have a >>> different time, which isn't working very well. >>> >>> Add assembly code used for boot CPU and secondary CPU cores to make >>> sure that the CNTVOFF register is initialized. >>> >>> Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> >>> --- >>> arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ >>> arch/arm/mach-sunxi/sunxi.c | 4 ++++ >>> 2 files changed, 25 insertions(+) >>> >>> diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S >>> index d5c97e945e69..605896251927 100644 >>> --- a/arch/arm/mach-sunxi/headsmp.S >>> +++ b/arch/arm/mach-sunxi/headsmp.S >>> @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) >>> first: .word sunxi_mc_smp_first_comer - . >>> ENDPROC(sunxi_mc_smp_cluster_cache_enable) >>> >>> +ENTRY(sunxi_init_cntvoff) >>> + /* >>> + * CNTVOFF has to be initialized either from non-secure Hypervisor >>> + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled >>> + * then it should be handled by the secure code >>> + */ >>> + cps #MON_MODE >>> + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ >>> + orr r0, r1, #1 >>> + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ >>> + instr_sync >> >> Since these CPUs are all ARMv7, you can use isb directly. Nobody wants >> to see more of the CP15 barriers. > > Okay, thanks, I will update that. > >> >>> + mov r0, #0 >>> + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ >>> + instr_sync >>> + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ >>> + instr_sync >>> + cps #SVC_MODE >>> + ret lr >> >> Given that this code is identical to the shmobile hack, it'd be good to >> make it common, one way or another. > > Sure, I will try that. > May you have some hints to give me on how to implement it? You could move it to arch/arm/common, for example. > >> >>> +ENDPROC(sunxi_init_cntvoff) >>> + >>> #ifdef CONFIG_SMP >>> ENTRY(sunxi_boot) >>> bl sunxi_mc_smp_cluster_cache_enable >>> + bl sunxi_init_cntvoff >>> b secondary_startup >>> ENDPROC(sunxi_boot) >>> >>> diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c >>> index 5e9602ce1573..4bb041492b54 100644 >>> --- a/arch/arm/mach-sunxi/sunxi.c >>> +++ b/arch/arm/mach-sunxi/sunxi.c >>> @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { >>> }; >>> >>> extern void __init sun6i_reset_init(void); >>> +extern void sunxi_init_cntvoff(void); >>> + >>> static void __init sun6i_timer_init(void) >>> { >>> + sunxi_init_cntvoff(); >>> + >> >> It is slightly odd that some CPUs get initialized from the early asm >> code, and some others do a detour via some C code. I'm sure this could >> be made to work > > Renesa's code was also doing that so I thought it could be ok to do > it. It is not that it isn't OK, it is just a slightly bizarre construct. > Without this code in this timer_init function, it fails to initialize > cntvoff correctly: > http://code.bulix.org/i9fhc9-302612?raw > > How should I implement that? I would make it part of a path that all CPUs have to hit at bring-up time. You already have such a path in your SMP init code, so why not take advantage of that, making it completely uniform across the board? Thanks, M.
On Sun, Mar 18, 2018 at 08:07:15PM +0100, Mylène Josserand wrote: > Hello Mark, > > Please, excuse me for this late answer and thank you for the review! > > On Wed, 7 Mar 2018 12:18:33 +0000 > Marc Zyngier <marc.zyngier@arm.com> wrote: > > > On 23/02/18 13:37, Mylène Josserand wrote: > > > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > > > > Only on A7? Is that specific to your platform? > > I do not really know other Allwinner's platforms about this subject. At > least, the sun9i-a80 which is a Cortex-a15/a7 does not need that but it > is necessary for sun8i-a83t which is a cortex-a7. Maybe, Chen-Yu or > Maxime could help us on it. This is not related to the CPU. On all the other SoCs but the A80 and the A83t, U-Boot will boot the system in HYP, and while switching to non-secure will setup CNTVOFF on the boot CPU. On the A80 and A83t, U-Boot will execute the kernel in secure, it will not switch to non-secure, and CNTVOFF will not be set on the boot CPU. Maxime
On Mon, Mar 19, 2018 at 10:14:19AM +0800, Chen-Yu Tsai wrote: > On Mon, Mar 19, 2018 at 3:07 AM, Mylène Josserand > <mylene.josserand@bootlin.com> wrote: > > Hello Mark, > > > > Please, excuse me for this late answer and thank you for the review! > > > > On Wed, 7 Mar 2018 12:18:33 +0000 > > Marc Zyngier <marc.zyngier@arm.com> wrote: > > > >> On 23/02/18 13:37, Mylène Josserand wrote: > >> > On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. > >> > >> Only on A7? Is that specific to your platform? > > > > I do not really know other Allwinner's platforms about this subject. At > > least, the sun9i-a80 which is a Cortex-a15/a7 does not need that but it > > is necessary for sun8i-a83t which is a cortex-a7. Maybe, Chen-Yu or > > Maxime could help us on it. > > AFAIK all Allwinner CPUs need it if there isn't a firmware (PSCI) layer > beneath the kernel that will do the setup. We just have > "arm,cpu-registers-not-fw-configured" set for all the other SoCs that > have in-kernel SMP support, which includes the A31, A23, A33 and A80. Most of these ones are here for historical reasons though. Now that we have U-Boot properly setting it up, we could probably remove it. Maxime
diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S index d5c97e945e69..605896251927 100644 --- a/arch/arm/mach-sunxi/headsmp.S +++ b/arch/arm/mach-sunxi/headsmp.S @@ -65,9 +65,30 @@ ENTRY(sunxi_mc_smp_cluster_cache_enable) first: .word sunxi_mc_smp_first_comer - . ENDPROC(sunxi_mc_smp_cluster_cache_enable) +ENTRY(sunxi_init_cntvoff) + /* + * CNTVOFF has to be initialized either from non-secure Hypervisor + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled + * then it should be handled by the secure code + */ + cps #MON_MODE + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ + orr r0, r1, #1 + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ + instr_sync + mov r0, #0 + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ + instr_sync + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ + instr_sync + cps #SVC_MODE + ret lr +ENDPROC(sunxi_init_cntvoff) + #ifdef CONFIG_SMP ENTRY(sunxi_boot) bl sunxi_mc_smp_cluster_cache_enable + bl sunxi_init_cntvoff b secondary_startup ENDPROC(sunxi_boot) diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c index 5e9602ce1573..4bb041492b54 100644 --- a/arch/arm/mach-sunxi/sunxi.c +++ b/arch/arm/mach-sunxi/sunxi.c @@ -37,8 +37,12 @@ static const char * const sun6i_board_dt_compat[] = { }; extern void __init sun6i_reset_init(void); +extern void sunxi_init_cntvoff(void); + static void __init sun6i_timer_init(void) { + sunxi_init_cntvoff(); + of_clk_init(NULL); if (IS_ENABLED(CONFIG_RESET_CONTROLLER)) sun6i_reset_init();
On Cortex-A7, the CNTVOFF register from arch timer is uninitialized. It should be done by the bootloader but it is currently not the case, even for boot CPU because this SoC is booting in secure mode. It leads to an random offset value meaning that each CPU will have a different time, which isn't working very well. Add assembly code used for boot CPU and secondary CPU cores to make sure that the CNTVOFF register is initialized. Signed-off-by: Mylène Josserand <mylene.josserand@bootlin.com> --- arch/arm/mach-sunxi/headsmp.S | 21 +++++++++++++++++++++ arch/arm/mach-sunxi/sunxi.c | 4 ++++ 2 files changed, 25 insertions(+)