| Message ID | 1447095621-32080-1-git-send-email-yann.morin.1998@free.fr |
|---|---|
| State | Accepted |
| Commit | 91ea9331275d5074be9202a6d866f6a6a860b8d9 |
| Headers | show |
On 09-11-15 20:00, Yann E. MORIN wrote: > Currently, we set HOSTARCH to the output of `uname -m`. This gives us > the architecture as seen by the running kernel. For example, we would > end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor. > > We use that value to determine whether we can run some binary tools, > like our pre-configured external toolchains. > > However, one may be running a userland in a different bitness than that > of the running kernel. For example, one may run in a 32-bit chroot, even > though the kernel is running in 64-bit. > > Up until recently, this was not an issue because the pre-configured > external toolchains were all requiring an i386 (x86 in Buildroot > parlance). > > But since w eintroduced the latest Linaro toolchains, we now have > toolchains that require a 64-bit userland. > > So, when running on a 64-bit kernel, we believe those toolchains are > available, even when the user is running a 32-bit userland. This causes > build failures for our autobuilders, like so: > > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ > > with the following symptoms: > > >>> toolchain-external undefined Configuring > Cannot execute cross-compiler '/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc' > > So, instead of relying on the output of `uname -r`, look for the host > gcc and extract the target it was configured to generate code for. > > Fixes: > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) > http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) > and many more... > > Besides fixing those issues, it will also allow us to add the 64-bit > variants of toolchains when they exist, like the upcoming Codescape > MTI and IMG toolchains for MIPS from Imagination Technologies. > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> But see below. > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > Cc: Vicente Olivert Riera <Vincent.Riera@imgtec.com> > --- > Makefile | 34 +++++++++++++++++++++++++--------- > 1 file changed, 25 insertions(+), 9 deletions(-) > > diff --git a/Makefile b/Makefile > index 80c264f..4c7ddf5 100644 > --- a/Makefile > +++ b/Makefile > @@ -52,15 +52,31 @@ ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA > $(error You have make '$(RUNNING_MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required) > endif > > -export HOSTARCH := $(shell uname -m | \ > - sed -e s/i.86/x86/ \ > - -e s/sun4u/sparc64/ \ > - -e s/arm.*/arm/ \ > - -e s/sa110/arm/ \ > - -e s/ppc64/powerpc64/ \ > - -e s/ppc/powerpc/ \ > - -e s/macppc/powerpc/\ > - -e s/sh.*/sh/) > +# Determine the userland we are running on. > +# > +# Note that, despite its name, we are not interested in the actual > +# architecture name. This is mostly used to determine whether some > +# of the binary tools (e.g. pre-built external toolchains) can run > +# on the current host. So we need to know if the userland we're > +# running on can actually run those toolchains. > +# > +# For example, a 64-bit prebuilt toolchain will not run on a 64-bit > +# kernel if the userland is 32-bit (e.g. in a chroot for example). > +# > +# So, we extract the first part of the tuple the host gcc was > +# configured to generate code for; we assume this is our userland. > +# > +export HOSTARCH := $(shell gcc -v 2>&1 | \ > + sed -e '/^Target: \([^-]*\).*/!d' \ > + -e 's//\1/' \ > + -e 's/i.86/x86/' \ > + -e 's/sun4u/sparc64/' \ > + -e 's/arm.*/arm/' \ > + -e 's/sa110/arm/' \ > + -e 's/ppc64/powerpc64/' \ > + -e 's/ppc/powerpc/' \ > + -e 's/macppc/powerpc/' \ > + -e 's/sh.*/sh/' ) Since what we get here already are gcc tuples, the de-mangling shouldn't be needed, right? Regards, Arnout > > # Parallel execution of this Makefile is disabled because it changes > # the packages building order, that can be a problem for two reasons: >
Arnout, All, On 2015-11-09 23:01 +0100, Arnout Vandecappelle spake thusly: > On 09-11-15 20:00, Yann E. MORIN wrote: > > Currently, we set HOSTARCH to the output of `uname -m`. This gives us > > the architecture as seen by the running kernel. For example, we would > > end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor. [--SNIP--] > > So, instead of relying on the output of `uname -r`, look for the host > > gcc and extract the target it was configured to generate code for. [--SNIP--] > > +export HOSTARCH := $(shell gcc -v 2>&1 | \ > > + sed -e '/^Target: \([^-]*\).*/!d' \ > > + -e 's//\1/' \ > > + -e 's/i.86/x86/' \ > > + -e 's/sun4u/sparc64/' \ > > + -e 's/arm.*/arm/' \ > > + -e 's/sa110/arm/' \ > > + -e 's/ppc64/powerpc64/' \ > > + -e 's/ppc/powerpc/' \ > > + -e 's/macppc/powerpc/' \ > > + -e 's/sh.*/sh/' ) > > Since what we get here already are gcc tuples, the de-mangling shouldn't be > needed, right? I am not sure either way. Note that I only constrained the check on how to retrieve the value, and expressely did not address the way we played with it, so as to follow the path of least surprise. However, we still have to de-mangle i.86 into x86 because that what we're using everywhere. Also, we need to demangle arm.* into arm, because arm may have some trailing stuff (like armv6) which still means it's an arm. Ditto sh I guess. macppc I would guess (y the name of it) that we really don't care about. sa110: are we really expecting to run on that machine anyway? We can further reduce the de-mangling in a later patch, however, it that makes sense... Regards, Yann E. MORIN.
On 09-11-15 23:17, Yann E. MORIN wrote: > Arnout, All, > > On 2015-11-09 23:01 +0100, Arnout Vandecappelle spake thusly: >> On 09-11-15 20:00, Yann E. MORIN wrote: >>> Currently, we set HOSTARCH to the output of `uname -m`. This gives us >>> the architecture as seen by the running kernel. For example, we would >>> end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor. > [--SNIP--] >>> So, instead of relying on the output of `uname -r`, look for the host >>> gcc and extract the target it was configured to generate code for. > [--SNIP--] >>> +export HOSTARCH := $(shell gcc -v 2>&1 | \ >>> + sed -e '/^Target: \([^-]*\).*/!d' \ >>> + -e 's//\1/' \ >>> + -e 's/i.86/x86/' \ >>> + -e 's/sun4u/sparc64/' \ >>> + -e 's/arm.*/arm/' \ >>> + -e 's/sa110/arm/' \ >>> + -e 's/ppc64/powerpc64/' \ >>> + -e 's/ppc/powerpc/' \ >>> + -e 's/macppc/powerpc/' \ >>> + -e 's/sh.*/sh/' ) >> >> Since what we get here already are gcc tuples, the de-mangling shouldn't be >> needed, right? > > I am not sure either way. Note that I only constrained the check on how > to retrieve the value, and expressely did not address the way we played > with it, so as to follow the path of least surprise. > > However, we still have to de-mangle i.86 into x86 because that what > we're using everywhere. Right, we'd have to change that. > Also, we need to demangle arm.* into arm, > because arm may have some trailing stuff (like armv6) which still means > it's an arm. Really? For the kernel arch (uname -m) that's true, but also for the gcc tuple? But armeb is indeed possible. > Ditto sh I guess. macppc I would guess (y the name of it) > that we really don't care about. sa110: are we really expecting to run > on that machine anyway? > > We can further reduce the de-mangling in a later patch, however, it that > makes sense... Of course. However, I'm not sure if it makes sense to copy the mangling for the uname to the mangling of the gcc tuple... Regards, Arnout > > Regards, > Yann E. MORIN. >
Yann, Peter, On Mon, 9 Nov 2015 20:00:21 +0100, Yann E. MORIN wrote: > Currently, we set HOSTARCH to the output of `uname -m`. This gives us > But since w eintroduced the latest Linaro toolchains, we now have > toolchains that require a 64-bit userland. > > So, when running on a 64-bit kernel, we believe those toolchains are > available, even when the user is running a 32-bit userland. This causes > build failures for our autobuilders, like so: > > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits. However, the real problem is that the new Linaro toolchains have been built against glibc 2.14 (for the host), while my chroot only uses glibc 2.11 (from Debian 6.0). I'm not sure how to handle this. Some old distros may not have glibc 2.14. Suggestions? In any case, I believe the rework of how we get HOSTARCH is not really needed IMO: 1/ it is not the source of the problem for the Linaro toolchain and 2/ it is most likely appropriate to use setarch/linux32 when running a 32 bits chroot under a 64 bits kernel. Best regards, Thomas
Thomas, All, On 2015-11-11 23:53 +0100, Thomas Petazzoni spake thusly: > On Mon, 9 Nov 2015 20:00:21 +0100, Yann E. MORIN wrote: > > Currently, we set HOSTARCH to the output of `uname -m`. This gives us > > > But since w eintroduced the latest Linaro toolchains, we now have > > toolchains that require a 64-bit userland. > > > > So, when running on a 64-bit kernel, we believe those toolchains are > > available, even when the user is running a 32-bit userland. This causes > > build failures for our autobuilders, like so: > > > > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ > > Those build failures are not caused by a 32 bits vs. 64 bits issue. In > fact, I started running my chroot under "linux32", but it was even > worse. Because indeed my chroots *are* 64 bits. OK, but still, in a 32-bit chroot, in the linux32 personality, we were trying to use the 64-bit variant of the toolchain. And I observed the same symptom as the autobuild failures. So, two different problems, with a single symptom. This patch adresses the 32-bit vs. 64-bit issue I encountered. Regards, Yann E. MORIN.
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: >> Those build failures are not caused by a 32 bits vs. 64 bits issue. In >> fact, I started running my chroot under "linux32", but it was even >> worse. Because indeed my chroots *are* 64 bits. > OK, but still, in a 32-bit chroot, in the linux32 personality, we were > trying to use the 64-bit variant of the toolchain. And I observed the > same symptom as the autobuild failures. How can that be? With linux32 uname -m returns 'ix86', right?
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes: Hi, > Those build failures are not caused by a 32 bits vs. 64 bits issue. In > fact, I started running my chroot under "linux32", but it was even > worse. Because indeed my chroots *are* 64 bits. > However, the real problem is that the new Linaro toolchains have been > built against glibc 2.14 (for the host), while my chroot only uses > glibc 2.11 (from Debian 6.0). Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on Enterprise distributions might end up with the same problems. > I'm not sure how to handle this. Some old distros may not have glibc > 2.14. I don't think there's much we can do, besides perhaps adding a comment in the help text. I'm not sure how realistic it is to convince the Linaro people to build their toolchains on ancient distributions. > In any case, I believe the rework of how we get HOSTARCH is not really > needed IMO: 1/ it is not the source of the problem for the Linaro > toolchain and 2/ it is most likely appropriate to use setarch/linux32 > when running a 32 bits chroot under a 64 bits kernel. I agree about linux32, yes.
Hello, On Thu, 12 Nov 2015 09:11:52 +0100, Peter Korsgaard wrote: > > Those build failures are not caused by a 32 bits vs. 64 bits issue. In > > fact, I started running my chroot under "linux32", but it was even > > worse. Because indeed my chroots *are* 64 bits. > > > However, the real problem is that the new Linaro toolchains have been > > built against glibc 2.14 (for the host), while my chroot only uses > > glibc 2.11 (from Debian 6.0). > > Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on > Enterprise distributions might end up with the same problems. Indeed, and that's why I'd like to keep my old distro. > > I'm not sure how to handle this. Some old distros may not have glibc > > 2.14. > > I don't think there's much we can do, besides perhaps adding a comment > in the help text. > > I'm not sure how realistic it is to convince the Linaro people to build > their toolchains on ancient distributions. So I guess that the only reasonable option here is that I had some logic in autobuild-run to not use this toolchain when the host C library is too old. This way, we keep using the Linaro toolchain on build slaves that are modern enough, and we don't have build failures on the others. Thoughts? Thomas
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes: Hi, >> Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on >> Enterprise distributions might end up with the same problems. > Indeed, and that's why I'd like to keep my old distro. Good! > So I guess that the only reasonable option here is that I had some > logic in autobuild-run to not use this toolchain when the host C > library is too old. This way, we keep using the Linaro toolchain on > build slaves that are modern enough, and we don't have build failures > on the others. > Thoughts? Yes, that makes sense.
Peter, All, On 2015-11-12 08:52 +0100, Peter Korsgaard spake thusly: > >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > > >> Those build failures are not caused by a 32 bits vs. 64 bits issue. In > >> fact, I started running my chroot under "linux32", but it was even > >> worse. Because indeed my chroots *are* 64 bits. > > > OK, but still, in a 32-bit chroot, in the linux32 personality, we were > > trying to use the 64-bit variant of the toolchain. And I observed the > > same symptom as the autobuild failures. > > How can that be? With linux32 uname -m returns 'ix86', right? Sorry, my bad. I meant: *without* the linux32 personality. Regards, Yann E. MORIN.
On Mon, Nov 9, 2015 at 8:00 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Currently, we set HOSTARCH to the output of `uname -m`. This gives us > the architecture as seen by the running kernel. For example, we would > end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor. > > We use that value to determine whether we can run some binary tools, > like our pre-configured external toolchains. > > However, one may be running a userland in a different bitness than that > of the running kernel. For example, one may run in a 32-bit chroot, even > though the kernel is running in 64-bit. > > Up until recently, this was not an issue because the pre-configured > external toolchains were all requiring an i386 (x86 in Buildroot > parlance). > > But since w eintroduced the latest Linaro toolchains, we now have > toolchains that require a 64-bit userland. > > So, when running on a 64-bit kernel, we believe those toolchains are > available, even when the user is running a 32-bit userland. This causes > build failures for our autobuilders, like so: > > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ > > with the following symptoms: > > >>> toolchain-external undefined Configuring > Cannot execute cross-compiler '/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc' > > So, instead of relying on the output of `uname -r`, look for the host > gcc and extract the target it was configured to generate code for. > > Fixes: > http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) > http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) > and many more... > > Besides fixing those issues, it will also allow us to add the 64-bit > variants of toolchains when they exist, like the upcoming Codescape > MTI and IMG toolchains for MIPS from Imagination Technologies. > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > Cc: Vicente Olivert Riera <Vincent.Riera@imgtec.com> > --- > Makefile | 34 +++++++++++++++++++++++++--------- > 1 file changed, 25 insertions(+), 9 deletions(-) Committed after changing it to use HOSTCC_NOCCACHE instead of 'gcc', thanks! -- Bye, Peter Korsgaard
diff --git a/Makefile b/Makefile index 80c264f..4c7ddf5 100644 --- a/Makefile +++ b/Makefile @@ -52,15 +52,31 @@ ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA $(error You have make '$(RUNNING_MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required) endif -export HOSTARCH := $(shell uname -m | \ - sed -e s/i.86/x86/ \ - -e s/sun4u/sparc64/ \ - -e s/arm.*/arm/ \ - -e s/sa110/arm/ \ - -e s/ppc64/powerpc64/ \ - -e s/ppc/powerpc/ \ - -e s/macppc/powerpc/\ - -e s/sh.*/sh/) +# Determine the userland we are running on. +# +# Note that, despite its name, we are not interested in the actual +# architecture name. This is mostly used to determine whether some +# of the binary tools (e.g. pre-built external toolchains) can run +# on the current host. So we need to know if the userland we're +# running on can actually run those toolchains. +# +# For example, a 64-bit prebuilt toolchain will not run on a 64-bit +# kernel if the userland is 32-bit (e.g. in a chroot for example). +# +# So, we extract the first part of the tuple the host gcc was +# configured to generate code for; we assume this is our userland. +# +export HOSTARCH := $(shell gcc -v 2>&1 | \ + sed -e '/^Target: \([^-]*\).*/!d' \ + -e 's//\1/' \ + -e 's/i.86/x86/' \ + -e 's/sun4u/sparc64/' \ + -e 's/arm.*/arm/' \ + -e 's/sa110/arm/' \ + -e 's/ppc64/powerpc64/' \ + -e 's/ppc/powerpc/' \ + -e 's/macppc/powerpc/' \ + -e 's/sh.*/sh/' ) # Parallel execution of this Makefile is disabled because it changes # the packages building order, that can be a problem for two reasons:
Currently, we set HOSTARCH to the output of `uname -m`. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor. We use that value to determine whether we can run some binary tools, like our pre-configured external toolchains. However, one may be running a userland in a different bitness than that of the running kernel. For example, one may run in a 32-bit chroot, even though the kernel is running in 64-bit. Up until recently, this was not an issue because the pre-configured external toolchains were all requiring an i386 (x86 in Buildroot parlance). But since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland. So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so: http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ with the following symptoms: >>> toolchain-external undefined Configuring Cannot execute cross-compiler '/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc' So, instead of relying on the output of `uname -r`, look for the host gcc and extract the target it was configured to generate code for. Fixes: http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) and many more... Besides fixing those issues, it will also allow us to add the 64-bit variants of toolchains when they exist, like the upcoming Codescape MTI and IMG toolchains for MIPS from Imagination Technologies. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Vicente Olivert Riera <Vincent.Riera@imgtec.com> --- Makefile | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)