[2/3,v3] linux: kconfig needs host-{flex, bison} to build the configurators

Message ID c85b3d3994261356bedfa3daeaddbc4329836fe9.1533327111.git.yann.morin.1998@free.fr
State Changes Requested
Headers show
Series
  • [1/3,v3] core/pkg-kconfig: allow dependencies before configurators
Related show

Commit Message

Yann E. MORIN Aug. 3, 2018, 8:16 p.m.
Take this opportunity to also drop the append-assignment as the first
assignment to LINUX_DEPENDENCIES.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
Changes v2 -> v3:
  - also fix the append-assignment
---
 linux/linux.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Comments

Jan Kundrát Aug. 7, 2018, 10:21 a.m. | #1
On pátek 3. srpna 2018 22:16:55 CEST, Yann E. MORIN wrote:
> Take this opportunity to also drop the append-assignment as the first
> assignment to LINUX_DEPENDENCIES.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
> ---
> Changes v2 -> v3:
>   - also fix the append-assignment
> ---
>  linux/linux.mk | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 7527b11673..60dad3a249 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -62,7 +62,11 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
>  LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
>  
>  LINUX_INSTALL_IMAGES = YES
> -LINUX_DEPENDENCIES += host-bison host-flex host-kmod
> +LINUX_DEPENDENCIES = host-kmod
> +
> +# Starting with 4.16, the generated kconfig code is no longer
> +# shipped with the kernel sources, so we need flex and bison.
> +LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
>  
>  # host tools needed for kernel compression
>  ifeq ($(BR2_LINUX_KERNEL_LZ4),y)

Tested-by: Jan Kundrát <jan.kundrat@cesnet.cz>
Thomas Petazzoni Aug. 14, 2018, 2:21 p.m. | #2
Hello,

On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
> Take this opportunity to also drop the append-assignment as the first
> assignment to LINUX_DEPENDENCIES.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

I am a bit confused. In this series, you suggest to make
host-flex/host-bison dependencies of the configuration step of Linux.
But in another series
(http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
suggest to make flex/bison hard requirements of Buildroot, so that we
don't have to build them.

Could you clarify in which direction you would like to go ? Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a
clear rule on whether we keep or not our host-bison/host-flex packages,
and if we keep them, when they should be used vs. when the system
bison/flex should be used.

Thanks!

Thomas
Yann E. MORIN Aug. 14, 2018, 3:27 p.m. | #3
Thomas, All,

On 2018-08-14 16:21 +0200, Thomas Petazzoni spake thusly:
> On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
> > Take this opportunity to also drop the append-assignment as the first
> > assignment to LINUX_DEPENDENCIES.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> I am a bit confused. In this series, you suggest to make
> host-flex/host-bison dependencies of the configuration step of Linux.
> But in another series
> (http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
> suggest to make flex/bison hard requirements of Buildroot, so that we
> don't have to build them.
> 
> Could you clarify in which direction you would like to go ?

This very series is about solving a technical problem, that
kconfig-based packages may have dependencies required even before
running the configurators.

On the other hand, that other series is about changing our requirements.

If we decide to require flex+bison, so in this series, only patch 2 can
be dropped.

> Also, if we
> make bison/flex hard requirements of Buildroot, I'd like to have a
> clear rule on whether we keep or not our host-bison/host-flex packages,
> and if we keep them, when they should be used vs. when the system
> bison/flex should be used.

My position is that we will always want to have our host flex and bison
to build packages that install things in host/, in staging/ or in
target/.

For the linux kernel and other kconfig-based packages, we don't care
which flex/bison are used, because the resulting binaries are not
installed, unless those packages also generate code eventually installed
in host/, staging/ or target/

Regards,
Yann E. MORIN.
Thomas Petazzoni Aug. 14, 2018, 7:53 p.m. | #4
Hello,

On Tue, 14 Aug 2018 17:27:43 +0200, Yann E. MORIN wrote:

> > Could you clarify in which direction you would like to go ?  
> 
> This very series is about solving a technical problem, that
> kconfig-based packages may have dependencies required even before
> running the configurators.
> 
> On the other hand, that other series is about changing our requirements.
> 
> If we decide to require flex+bison, so in this series, only patch 2 can
> be dropped.

OK.

> > Also, if we
> > make bison/flex hard requirements of Buildroot, I'd like to have a
> > clear rule on whether we keep or not our host-bison/host-flex packages,
> > and if we keep them, when they should be used vs. when the system
> > bison/flex should be used.  
> 
> My position is that we will always want to have our host flex and bison
> to build packages that install things in host/, in staging/ or in
> target/.
> 
> For the linux kernel and other kconfig-based packages, we don't care
> which flex/bison are used, because the resulting binaries are not
> installed, unless those packages also generate code eventually installed
> in host/, staging/ or target/

So the system-wide bison/flex would only be used for the kconfig
programs built by Buildroot itself (if we move to a newer kconfig
version that doesn't have the pre-generated files) or by
Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
host-flex and host-bison. That's your proposal ?

Best regards,

Thomas
Yann E. MORIN Aug. 14, 2018, 9:03 p.m. | #5
Thomas, All,

On 2018-08-14 21:53 +0200, Thomas Petazzoni spake thusly:
> On Tue, 14 Aug 2018 17:27:43 +0200, Yann E. MORIN wrote:
> > > Also, if we
> > > make bison/flex hard requirements of Buildroot, I'd like to have a
> > > clear rule on whether we keep or not our host-bison/host-flex packages,
> > > and if we keep them, when they should be used vs. when the system
> > > bison/flex should be used.  
> > 
> > My position is that we will always want to have our host flex and bison
> > to build packages that install things in host/, in staging/ or in
> > target/.
> > 
> > For the linux kernel and other kconfig-based packages, we don't care
> > which flex/bison are used, because the resulting binaries are not
> > installed, unless those packages also generate code eventually installed
> > in host/, staging/ or target/
> 
> So the system-wide bison/flex would only be used for the kconfig
> programs built by Buildroot itself (if we move to a newer kconfig
> version that doesn't have the pre-generated files) or by
> Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
> host-flex and host-bison. That's your proposal ?

Yes, that is my proposal.

Regards,
Yann E. MORIN.
Thomas Petazzoni Aug. 14, 2018, 9:12 p.m. | #6
Hello,

On Tue, 14 Aug 2018 23:03:49 +0200, Yann E. MORIN wrote:

> > So the system-wide bison/flex would only be used for the kconfig
> > programs built by Buildroot itself (if we move to a newer kconfig
> > version that doesn't have the pre-generated files) or by
> > Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
> > host-flex and host-bison. That's your proposal ?  
> 
> Yes, that is my proposal.

OK. Let's see what Arnout/Peter have to say about this. If they don't
express any opinion by Thursday, I'll go with your proposal which seems
reasonable to me.

Thanks,

Thomas
Arnout Vandecappelle Aug. 14, 2018, 10:50 p.m. | #7
On 14-08-18 17:27, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2018-08-14 16:21 +0200, Thomas Petazzoni spake thusly:
>> On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
>>> Take this opportunity to also drop the append-assignment as the first
>>> assignment to LINUX_DEPENDENCIES.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Arnout Vandecappelle <arnout@mind.be>
>>> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>>
>> I am a bit confused. In this series, you suggest to make
>> host-flex/host-bison dependencies of the configuration step of Linux.
>> But in another series
>> (http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
>> suggest to make flex/bison hard requirements of Buildroot, so that we
>> don't have to build them.
>>
>> Could you clarify in which direction you would like to go ?
> 
> This very series is about solving a technical problem, that
> kconfig-based packages may have dependencies required even before
> running the configurators.
> 
> On the other hand, that other series is about changing our requirements.
> 
> If we decide to require flex+bison, so in this series, only patch 2 can
> be dropped.
> 
>> Also, if we
>> make bison/flex hard requirements of Buildroot, I'd like to have a
>> clear rule on whether we keep or not our host-bison/host-flex packages,
>> and if we keep them, when they should be used vs. when the system
>> bison/flex should be used.
> 
> My position is that we will always want to have our host flex and bison
> to build packages that install things in host/, in staging/ or in
> target/.

 My position is that we do indeed need our host flex and bison for everything
installed in target/ (which implies everything installed in staging/), but not
for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.

 It would also be nice if dependencies.sh could do a version check and accept
the system flex/bison if it is the right version, like we do for cmake and tar.

 But both of these wishes make things more complicated, so I'm OK with just
keeping all current host-flex/bison dependencies.


> For the linux kernel and other kconfig-based packages, we don't care
> which flex/bison are used, because the resulting binaries are not
> installed, unless those packages also generate code eventually installed
> in host/, staging/ or target/

 Note that this means you'll still need host-flex/bison for the kernel, because
the kernel may build dtc and that gets installed in host...

 Regards,
 Arnout

> 
> Regards,
> Yann E. MORIN.
>
Thomas Petazzoni Aug. 15, 2018, 12:18 p.m. | #8
Hello,

On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:

>  My position is that we do indeed need our host flex and bison for everything
> installed in target/ (which implies everything installed in staging/), but not
> for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.
> 
>  It would also be nice if dependencies.sh could do a version check and accept
> the system flex/bison if it is the right version, like we do for cmake and tar.
> 
>  But both of these wishes make things more complicated, so I'm OK with just
> keeping all current host-flex/bison dependencies.

In fact, I think adding flex and bison as mandatory dependencies of the
system is going to cause too much disruption. All the autobuilders
would start complaining, all the Travis/Gitlab CI setups would have to
be adjusted, and everyone building in containers/minimal environments
will be affected.

So instead, I'd like to detect if flex/bison are available on the
system. If they are available, then we use them for the Linux kernel
configuration process. If they are not available, we build them.

Instead of:

  LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex

I'd prefer to see:

  LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

And the usual logic in support/dependencies/ to check for bison/flex.
This will avoid disrupting all existing setups, and still provide the
speed-up that we don't need to build flex/bison when doing "make
linux-menuconfig".

> > For the linux kernel and other kconfig-based packages, we don't care
> > which flex/bison are used, because the resulting binaries are not
> > installed, unless those packages also generate code eventually installed
> > in host/, staging/ or target/  
> 
>  Note that this means you'll still need host-flex/bison for the kernel, because
> the kernel may build dtc and that gets installed in host...

Are you sure flex/bison are used when building the DTC copy in the
Linux kernel ? Indeed we only added host-bison/host-flex as
dependencies of linux in commit
1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
its pre-generated files. How could DTC be built before
1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

Best regards,

Thomas
Yann E. MORIN Aug. 15, 2018, 4 p.m. | #9
Thomas, All,

On 2018-08-15 14:18 +0200, Thomas Petazzoni spake thusly:
> On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:
> >  My position is that we do indeed need our host flex and bison for everything
> > installed in target/ (which implies everything installed in staging/), but not
> > for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.
> > 
> >  It would also be nice if dependencies.sh could do a version check and accept
> > the system flex/bison if it is the right version, like we do for cmake and tar.
> > 
> >  But both of these wishes make things more complicated, so I'm OK with just
> > keeping all current host-flex/bison dependencies.
> 
> In fact, I think adding flex and bison as mandatory dependencies of the
> system is going to cause too much disruption. All the autobuilders
> would start complaining, all the Travis/Gitlab CI setups would have to
> be adjusted, and everyone building in containers/minimal environments
> will be affected.

Yes, they will be affected. If not today, then later, because at some
point we will also bump to require new versions of componentns. Oh,
wait, we already bumped to gcc-4.4 (even if that is mostly transparent).

So I don't think we should refrain from updating, or adding to, our
requirements just because some people will be affected...

> So instead, I'd like to detect if flex/bison are available on the
> system. If they are available, then we use them for the Linux kernel
> configuration process. If they are not available, we build them.
> 
> Instead of:
>   LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
> I'd prefer to see:
>   LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

Yeah, why not. That is easy enough.

> And the usual logic in support/dependencies/ to check for bison/flex.
> This will avoid disrupting all existing setups, and still provide the
> speed-up that we don't need to build flex/bison when doing "make
> linux-menuconfig".

Except we will soon to have to build the toolchain to be able to run
linux-menuconfig anyway... So, the flex/bison build time will not be
noticeable, except for external toolchains.

> > > For the linux kernel and other kconfig-based packages, we don't care
> > > which flex/bison are used, because the resulting binaries are not
> > > installed, unless those packages also generate code eventually installed
> > > in host/, staging/ or target/  
> > 
> >  Note that this means you'll still need host-flex/bison for the kernel, because
> > the kernel may build dtc and that gets installed in host...

But they would not be a LINUX_KCONFIG_DEPENDENCIES, just a plain
LINUX_DEPENDENCIES.

Regards,
Yann E. MORIN.

> Are you sure flex/bison are used when building the DTC copy in the
> Linux kernel ? Indeed we only added host-bison/host-flex as
> dependencies of linux in commit
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
> its pre-generated files. How could DTC be built before
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com
Arnout Vandecappelle Aug. 16, 2018, 3:50 p.m. | #10
On 15/08/2018 14:18, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:
[snip]
> I'd prefer to see:
> 
>   LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)
> 
> And the usual logic in support/dependencies/ to check for bison/flex.
> This will avoid disrupting all existing setups, and still provide the
> speed-up that we don't need to build flex/bison when doing "make
> linux-menuconfig".

 +1. It also makes it easy to add the feature "build flex is system version is
not OK" later.


>>> For the linux kernel and other kconfig-based packages, we don't care
>>> which flex/bison are used, because the resulting binaries are not
>>> installed, unless those packages also generate code eventually installed
>>> in host/, staging/ or target/  
>>
>>  Note that this means you'll still need host-flex/bison for the kernel, because
>> the kernel may build dtc and that gets installed in host...
> 
> Are you sure flex/bison are used when building the DTC copy in the
> Linux kernel ? Indeed we only added host-bison/host-flex as
> dependencies of linux in commit
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
> its pre-generated files. How could DTC be built before
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

 The shipped files for DTC were removed in
e039139be8c25145b103ab365ff1bd4a540066a3 which is v4.17-rc1~106^2~8^2~2 (while
the kconfig stuff was removed in 29c833061c1d8c2d1d23a62e7061561eadd76cdb which
is v4.16-rc1~105^2~26). So we already had the dependencies when the shipped DTC
files were removed from the kernel. And as Yann said, they're not needed for
kconfig itself, only for building.

 Regards,
 Arnout

> 
> Best regards,
> 
> Thomas
>

Patch

diff --git a/linux/linux.mk b/linux/linux.mk
index 7527b11673..60dad3a249 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -62,7 +62,11 @@  LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 
 LINUX_INSTALL_IMAGES = YES
-LINUX_DEPENDENCIES += host-bison host-flex host-kmod
+LINUX_DEPENDENCIES = host-kmod
+
+# Starting with 4.16, the generated kconfig code is no longer
+# shipped with the kernel sources, so we need flex and bison.
+LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
 
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)