| Message ID | 1373222539-8915-1-git-send-email-s.martin49@gmail.com |
|---|---|
| State | Rejected |
| Headers | show |
Dear Samuel Martin, On Sun, 7 Jul 2013 20:42:19 +0200, Samuel Martin wrote: > MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. > To avoid this, unset the MAKEFLAGS environment variable. Does unsetting MAKEFLAGS really allows to prevent top-level parallel build on Buildroot? Or does it only prevents MAKEFLAGS from leaking to the sub-make calls that we do to build packages? Thomas
>>>>> "Samuel" == Samuel Martin <s.martin49@gmail.com> writes:
Samuel> MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'.
Samuel> To avoid this, unset the MAKEFLAGS environment variable.
Unfortunately this also drops '-s' from child make invocations, making
silent builds significantly more noisy.
We have .NOTPARALLEL: in the top level Makefile, isn't that enough to
ensure funky MAKEFLAGS settings? Anything else you encountered?
Samuel> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Samuel> ---
Samuel> Makefile | 1 +
Samuel> 1 file changed, 1 insertion(+)
Samuel> diff --git a/Makefile b/Makefile
Samuel> index 015fbdf..11a7b70 100644
Samuel> --- a/Makefile
Samuel> +++ b/Makefile
Samuel> @@ -207,6 +207,7 @@ unexport CFLAGS
Samuel> unexport CXXFLAGS
Samuel> unexport GREP_OPTIONS
Samuel> unexport CONFIG_SITE
Samuel> +unexport MAKEFLAGS
Samuel> unexport QMAKESPEC
Samuel> unexport TERMINFO
Samuel> --
Samuel> 1.8.3.2
Samuel> _______________________________________________
Samuel> buildroot mailing list
Samuel> buildroot@busybox.net
Samuel> http://lists.busybox.net/mailman/listinfo/buildroot
Hi Thomas, 2013/7/7 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>: > Dear Samuel Martin, > > On Sun, 7 Jul 2013 20:42:19 +0200, Samuel Martin wrote: >> MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. >> To avoid this, unset the MAKEFLAGS environment variable. > > Does unsetting MAKEFLAGS really allows to prevent top-level parallel > build on Buildroot? It does, but as Peter points out, this also disables some other options we don't want to see disabled. So, forget about this patch. > Or does it only prevents MAKEFLAGS from leaking to > the sub-make calls that we do to build packages? > > Thomas > -- > Thomas Petazzoni, Free Electrons > Kernel, drivers, real-time and embedded Linux > development, consulting, training and support. > http://free-electrons.com
Hi Peter, 2013/7/7 Peter Korsgaard <jacmet@uclibc.org>: >>>>>> "Samuel" == Samuel Martin <s.martin49@gmail.com> writes: > > Samuel> MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. > Samuel> To avoid this, unset the MAKEFLAGS environment variable. > > Unfortunately this also drops '-s' from child make invocations, making > silent builds significantly more noisy. > > We have .NOTPARALLEL: in the top level Makefile, isn't that enough to > ensure funky MAKEFLAGS settings? Anything else you encountered? Seems not since, some days ago, some people complained about it on the irc channel. I will update the doc, instead.
Hi, On Sun, Jul 7, 2013 at 8:42 PM, Samuel Martin <s.martin49@gmail.com> wrote: > MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. > To avoid this, unset the MAKEFLAGS environment variable. > > Signed-off-by: Samuel Martin <s.martin49@gmail.com> > --- > Makefile | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/Makefile b/Makefile > index 015fbdf..11a7b70 100644 > --- a/Makefile > +++ b/Makefile > @@ -207,6 +207,7 @@ unexport CFLAGS > unexport CXXFLAGS > unexport GREP_OPTIONS > unexport CONFIG_SITE > +unexport MAKEFLAGS > unexport QMAKESPEC > unexport TERMINFO > What is the strategy with respect to cleaning up the user's environment when building buildroot? Because there are a number of other variables that users can have (and do have) that corrupt the build, for example: C_INCLUDE_PATH CPLUS_INCLUDE_PATH LIBRARY_PATH LD_LIBRARY_PATH PERL5LIB GCC_EXEC_PREFIX In the twisted environments that I'm working in, I'm unsetting these from a wrapper around buildroot make. However, it seems that there already are a number of cleanups done inside buildroot itself, so it makes sense to add the above variables to the list. What do you think? Best regards, Thomas
On 11 July 2013 11:37, Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> wrote: > Hi, > > On Sun, Jul 7, 2013 at 8:42 PM, Samuel Martin <s.martin49@gmail.com> wrote: >> MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. >> To avoid this, unset the MAKEFLAGS environment variable. >> >> Signed-off-by: Samuel Martin <s.martin49@gmail.com> >> --- >> Makefile | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/Makefile b/Makefile >> index 015fbdf..11a7b70 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -207,6 +207,7 @@ unexport CFLAGS >> unexport CXXFLAGS >> unexport GREP_OPTIONS >> unexport CONFIG_SITE >> +unexport MAKEFLAGS >> unexport QMAKESPEC >> unexport TERMINFO >> > > What is the strategy with respect to cleaning up the user's > environment when building buildroot? > Because there are a number of other variables that users can have (and > do have) that corrupt the build, for example: > > C_INCLUDE_PATH > CPLUS_INCLUDE_PATH > LIBRARY_PATH > LD_LIBRARY_PATH > PERL5LIB > GCC_EXEC_PREFIX > > In the twisted environments that I'm working in, I'm unsetting these > from a wrapper around buildroot make. However, it seems that there > already are a number of cleanups done inside buildroot itself, so it > makes sense to add the above variables to the list. > > What do you think? How about going all the way: clean out the environment completely, and explicitly add env vars to builders as needed. That would result in a more deterministic build environment, IMHO. Best regards, Bjørn Forsman
Dear Thomas De Schampheleire, On Thu, 11 Jul 2013 11:37:10 +0200, Thomas De Schampheleire wrote: > What is the strategy with respect to cleaning up the user's > environment when building buildroot? > Because there are a number of other variables that users can have (and > do have) that corrupt the build, for example: > > C_INCLUDE_PATH > CPLUS_INCLUDE_PATH > LIBRARY_PATH > LD_LIBRARY_PATH > PERL5LIB > GCC_EXEC_PREFIX > > In the twisted environments that I'm working in, I'm unsetting these > from a wrapper around buildroot make. However, it seems that there > already are a number of cleanups done inside buildroot itself, so it > makes sense to add the above variables to the list. > > What do you think? We have are a bit inconsistent on this. Some variables are unset from the main Makefile, and a bunch of others are checked in support/dependencies/dependencies.sh. For example, LD_LIBRARY_PATH which you mentioned get checked in support/dependencies/dependencies.sh. Looks like a bit of cleanup in this area might be useful. Thomas
Hi Bjørn, On Thu, Jul 11, 2013 at 12:33 PM, Bjørn Forsman <bjorn.forsman@gmail.com> wrote: > On 11 July 2013 11:37, Thomas De Schampheleire > <patrickdepinguin+buildroot@gmail.com> wrote: >> Hi, >> >> On Sun, Jul 7, 2013 at 8:42 PM, Samuel Martin <s.martin49@gmail.com> wrote: >>> MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. >>> To avoid this, unset the MAKEFLAGS environment variable. >>> >>> Signed-off-by: Samuel Martin <s.martin49@gmail.com> >>> --- >>> Makefile | 1 + >>> 1 file changed, 1 insertion(+) >>> >>> diff --git a/Makefile b/Makefile >>> index 015fbdf..11a7b70 100644 >>> --- a/Makefile >>> +++ b/Makefile >>> @@ -207,6 +207,7 @@ unexport CFLAGS >>> unexport CXXFLAGS >>> unexport GREP_OPTIONS >>> unexport CONFIG_SITE >>> +unexport MAKEFLAGS >>> unexport QMAKESPEC >>> unexport TERMINFO >>> >> >> What is the strategy with respect to cleaning up the user's >> environment when building buildroot? >> Because there are a number of other variables that users can have (and >> do have) that corrupt the build, for example: >> >> C_INCLUDE_PATH >> CPLUS_INCLUDE_PATH >> LIBRARY_PATH >> LD_LIBRARY_PATH >> PERL5LIB >> GCC_EXEC_PREFIX >> >> In the twisted environments that I'm working in, I'm unsetting these >> from a wrapper around buildroot make. However, it seems that there >> already are a number of cleanups done inside buildroot itself, so it >> makes sense to add the above variables to the list. >> >> What do you think? > > How about going all the way: clean out the environment completely, and > explicitly add env vars to builders as needed. That would result in a > more deterministic build environment, IMHO. It's true that it would be deterministic, but I'm not sure if it's always correct. There are some environment variables that could be needed for a correct build, like proxy settings. Clearing them out will be a problem in such cases. So either we should work with a whitelist of safe flags, or keep the existing environment but delete items on the blacklist.
Hi Thomas, On Thu, Jul 11, 2013 at 1:01 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Dear Thomas De Schampheleire, > > On Thu, 11 Jul 2013 11:37:10 +0200, Thomas De Schampheleire wrote: > >> What is the strategy with respect to cleaning up the user's >> environment when building buildroot? >> Because there are a number of other variables that users can have (and >> do have) that corrupt the build, for example: >> >> C_INCLUDE_PATH >> CPLUS_INCLUDE_PATH >> LIBRARY_PATH >> LD_LIBRARY_PATH >> PERL5LIB >> GCC_EXEC_PREFIX >> >> In the twisted environments that I'm working in, I'm unsetting these >> from a wrapper around buildroot make. However, it seems that there >> already are a number of cleanups done inside buildroot itself, so it >> makes sense to add the above variables to the list. >> >> What do you think? > > We have are a bit inconsistent on this. Some variables are unset from > the main Makefile, and a bunch of others are checked in > support/dependencies/dependencies.sh. For example, LD_LIBRARY_PATH > which you mentioned get checked in support/dependencies/dependencies.sh. > > Looks like a bit of cleanup in this area might be useful. dependencies.sh does not complain on the existence of LD_LIBRARY_PATH in the environment, only on the presence of the current working directory inside of it. But of course, if we decide to unexport this variable, then checking its contents is no longer needed. Is there any valid use case for someone setting LD_LIBRARY_PATH during a buildroot build? For the check on the presence of the current working dir in PATH, we could question whether it makes sense to raise a message and exit, or to manipulate PATH during the build and continue. Another variable checked in dependencies.sh is PERL_MM_OPT. This seems a good candidate to add to the existing list in Makefile. Is there any particular reason why some unexports are done always, and some are done inside a check on BR2_HAVE_DOT_CONFIG ?
Dear Bjørn Forsman, On Thu, 11 Jul 2013 12:33:42 +0200, Bjørn Forsman wrote: > How about going all the way: clean out the environment completely, and > explicitly add env vars to builders as needed. That would result in a > more deterministic build environment, IMHO. I don't think this works: there are a bunch of environment variables that we accept (to tune the uClibc config file, the Busybox config file, the host gcc to use, etc.). And we also accept the feature that all BR2_<foo> values can be overridden using environment variables. So I don't think wiping out the entire environment is possible. However, we could /maybe/ (if that's possible), wipe out the entire environment *except* the BR2_<foo> variables, and the few other variables that we explicitly accept. But I'm not sure how to achieve that. Best regards, Thomas
Thomas, All, On Thursday 11 July 2013 13:46:46 Thomas Petazzoni wrote: > On Thu, 11 Jul 2013 12:33:42 +0200, Bjørn Forsman wrote: > > How about going all the way: clean out the environment completely, and > > explicitly add env vars to builders as needed. That would result in a > > more deterministic build environment, IMHO. > > I don't think this works: there are a bunch of environment variables > that we accept (to tune the uClibc config file, the Busybox config > file, the host gcc to use, etc.). And we also accept the feature that > all BR2_<foo> values can be overridden using environment variables. So > I don't think wiping out the entire environment is possible. However, > we could /maybe/ (if that's possible), wipe out the entire environment > *except* the BR2_<foo> variables, and the few other variables that we > explicitly accept. But I'm not sure how to achieve that. Even that would not work. For example, I have LD_PRELOAD set to tsocksify all network connections, without which I could not download anything. So, even though LD_PRELOAD looks like a good candidate to dump, it can be valid in some cases. I believe it is much better that Buildroot chokes on a select list of variables, warn about a few others, and accept the rest. Regards, Yann E. MORIN.
Hi Yann, On Thu, Jul 11, 2013 at 2:02 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Thomas, All, > > On Thursday 11 July 2013 13:46:46 Thomas Petazzoni wrote: >> On Thu, 11 Jul 2013 12:33:42 +0200, Bjørn Forsman wrote: >> > How about going all the way: clean out the environment completely, and >> > explicitly add env vars to builders as needed. That would result in a >> > more deterministic build environment, IMHO. >> >> I don't think this works: there are a bunch of environment variables >> that we accept (to tune the uClibc config file, the Busybox config >> file, the host gcc to use, etc.). And we also accept the feature that >> all BR2_<foo> values can be overridden using environment variables. So >> I don't think wiping out the entire environment is possible. However, >> we could /maybe/ (if that's possible), wipe out the entire environment >> *except* the BR2_<foo> variables, and the few other variables that we >> explicitly accept. But I'm not sure how to achieve that. > > Even that would not work. For example, I have LD_PRELOAD set to tsocksify > all network connections, without which I could not download anything. > > So, even though LD_PRELOAD looks like a good candidate to dump, it can be > valid in some cases. > > I believe it is much better that Buildroot chokes on a select list of > variables, warn about a few others, and accept the rest. But with this policy, all existing 'unexports' in the Makefile should be removed, and it's up to the user to fix his stuff. In corporate environments, for one, this is not always feasible. Many developers will build with buildroot as part of an overall build process, and will not know anything about buildroot itself nor its requirements. So either the overall build process should clear the user's environment before starting the buildroot build, or buildroot itself should clean up what it knows is problematic. Since there already are several variables cleared in that way, I'm more in favor in going down that path and adding some more troublesome variables, rather than removing these bits altogether.
On 07/11/2013 09:08 AM, Thomas De Schampheleire wrote: > But with this policy, all existing 'unexports' in the Makefile should > be removed, and it's up to the user to fix his stuff. > In corporate environments, for one, this is not always feasible. Many > developers will build with buildroot as part of an overall build > process, and will not know anything about buildroot itself nor its > requirements. So either the overall build process should clear the > user's environment before starting the buildroot build, or buildroot > itself should clean up what it knows is problematic. > > Since there already are several variables cleared in that way, I'm > more in favor in going down that path and adding some more troublesome > variables, rather than removing these bits altogether. I agree. At the moment it's working quite well as it is, maybe we should add some extra unexports in the main Makefile for things that are known to be troublesome. For instance on Gentoo RUBYOPT is set, and when buildroot builds host-ruby this causes problems. The best solution would be to unexport RUBYOPT (and RUBYLIB while at it), i just didn't send a patch yet because i'm looking at the thread to see how it pans out :) Regards.
Dear Thomas De Schampheleire, On Thu, 11 Jul 2013 13:36:28 +0200, Thomas De Schampheleire wrote: > dependencies.sh does not complain on the existence of LD_LIBRARY_PATH > in the environment, only on the presence of the current working > directory inside of it. But of course, if we decide to unexport this > variable, then checking its contents is no longer needed. > Is there any valid use case for someone setting LD_LIBRARY_PATH during > a buildroot build? I guess it could be used if someone is building manually some host tools needed by Buildroot, and a special LD_LIBRARY_PATH is needed to run such tools. But it really seems like a corner case. > For the check on the presence of the current working dir in PATH, we > could question whether it makes sense to raise a message and exit, or > to manipulate PATH during the build and continue. True. > Another variable checked in dependencies.sh is PERL_MM_OPT. This seems > a good candidate to add to the existing list in Makefile. Indeed. I'm the one to blame here, I'm the one who added this variable check in dependencies.sh, if I remember correctly :) > Is there any particular reason why some unexports are done always, and > some are done inside a check on BR2_HAVE_DOT_CONFIG ? So, PKG_CONFIG_PATH, PKG_CONFIG_SYSROOT_DIR and DESTDIR get unset outside of the BR2_HAVE_DOT_CONFIG conditional, all the other are inside. I don't really see why we're doing that. Thomas
On 11 July 2013 14:02, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Thomas, All, > > On Thursday 11 July 2013 13:46:46 Thomas Petazzoni wrote: >> On Thu, 11 Jul 2013 12:33:42 +0200, Bjørn Forsman wrote: >> > How about going all the way: clean out the environment completely, and >> > explicitly add env vars to builders as needed. That would result in a >> > more deterministic build environment, IMHO. >> >> I don't think this works: there are a bunch of environment variables >> that we accept (to tune the uClibc config file, the Busybox config >> file, the host gcc to use, etc.). And we also accept the feature that >> all BR2_<foo> values can be overridden using environment variables. So >> I don't think wiping out the entire environment is possible. However, >> we could /maybe/ (if that's possible), wipe out the entire environment >> *except* the BR2_<foo> variables, and the few other variables that we >> explicitly accept. But I'm not sure how to achieve that. > > Even that would not work. For example, I have LD_PRELOAD set to tsocksify > all network connections, without which I could not download anything. > > So, even though LD_PRELOAD looks like a good candidate to dump, it can be > valid in some cases. > > I believe it is much better that Buildroot chokes on a select list of > variables, warn about a few others, and accept the rest. If you think about it, the downloader is actually a bit different from the builder. At some point Buildroot will (like all other build systems) grow support for checking each downloaded file against checksums specified in the package build file. And once you have that, it doesn't really matter how dirty the environment in the *downloader* is, because it will be guaranteed to match the checksum, or fail. It is only in the *builder* that the environment must be clean. Are there any reasons for allowing env vars, other than BR2_*, to slip through from host and into the Buildroot *builder* processes? I cannot think of any. Best regards, Bjørn Forsman
Dear Bjørn Forsman, On Fri, 12 Jul 2013 19:07:58 +0200, Bjørn Forsman wrote: > > I believe it is much better that Buildroot chokes on a select list of > > variables, warn about a few others, and accept the rest. > > If you think about it, the downloader is actually a bit different from > the builder. At some point Buildroot will (like all other build > systems) grow support for checking each downloaded file against > checksums specified in the package build file. And once you have that, > it doesn't really matter how dirty the environment in the *downloader* > is, because it will be guaranteed to match the checksum, or fail. It > is only in the *builder* that the environment must be clean. I am not sure to understand why you separate the download side and the build side here. Both are done with Buildroot makefiles, there is nothing different in terms of environment variables between those steps. > Are there any reasons for allowing env vars, other than BR2_*, to slip > through from host and into the Buildroot *builder* processes? I cannot > think of any. Yes, we allow passing HOSTCC, UCLIBC_CONFIG_FILE, BUSYBOX_CONFIG_FILE, and a bunch of other environment variables (see the Buildroot manual). Also, some environment variables like LD_LIBRARY_PATH or LD_PRELOAD should just be usable by the user, if he has some funky setup that requires those environment variables to hold special values. Best regards, Thomas
On 13 July 2013 16:07, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Dear Bjørn Forsman, > > On Fri, 12 Jul 2013 19:07:58 +0200, Bjørn Forsman wrote: > >> > I believe it is much better that Buildroot chokes on a select list of >> > variables, warn about a few others, and accept the rest. >> >> If you think about it, the downloader is actually a bit different from >> the builder. At some point Buildroot will (like all other build >> systems) grow support for checking each downloaded file against >> checksums specified in the package build file. And once you have that, >> it doesn't really matter how dirty the environment in the *downloader* >> is, because it will be guaranteed to match the checksum, or fail. It >> is only in the *builder* that the environment must be clean. > > I am not sure to understand why you separate the download side and the > build side here. Both are done with Buildroot makefiles, there is > nothing different in terms of environment variables between those > steps. They are not separate right now, but they could be. Because of the possibility to use checksums to verify the downloader, and because that is not possible to do with the builder, I think the two can be seen as quite different things. Even though they are both implemented in makefiles, I don't think it's difficult to run them in different environments. man env ? >> Are there any reasons for allowing env vars, other than BR2_*, to slip >> through from host and into the Buildroot *builder* processes? I cannot >> think of any. > > Yes, we allow passing HOSTCC, UCLIBC_CONFIG_FILE, BUSYBOX_CONFIG_FILE, > and a bunch of other environment variables (see the Buildroot manual). Ok. Those variables are special to Buildroot, similar to BR2_* vars. So I think they fall under the same category. In fact, they *could* be renamed to BR2_*, because the are non-standard anyways, unlike LD_LIBRARY_PATH etc. Note that I'm not suggesting that they are renamed, just that they are Buildroot specific. > Also, some environment variables like LD_LIBRARY_PATH or LD_PRELOAD > should just be usable by the user, if he has some funky setup that > requires those environment variables to hold special values. Are you thinking of Yann's use case? That was for the download phase, which I agree could run in an unmodified host environment. I was asking if it would make any sense to allow LD_LIBRARY_PATH etc. into the *build* phase (configure && make). I think the build phase is the one where most bugs happen because of different host environments :-) Anyways, I was just throwing my idea out there. I've been playing with nixos[1] lately, and have been inspired by its "pure" build system. It tries very hard to prevent the host environment from affecting the build result. [1] https://nixos.org/ Best regards, Bjørn Forsman
diff --git a/Makefile b/Makefile index 015fbdf..11a7b70 100644 --- a/Makefile +++ b/Makefile @@ -207,6 +207,7 @@ unexport CFLAGS unexport CXXFLAGS unexport GREP_OPTIONS unexport CONFIG_SITE +unexport MAKEFLAGS unexport QMAKESPEC unexport TERMINFO
MAKEFLAGS can carry options that make Buildroot failed, eg. '-jN'. To avoid this, unset the MAKEFLAGS environment variable. Signed-off-by: Samuel Martin <s.martin49@gmail.com> --- Makefile | 1 + 1 file changed, 1 insertion(+)