| Message ID | 20180730053936.26460-1-sam@mendozajonas.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [RFC] host-localedef: Compile against glibc | expand |
On 30 July 2018 at 15:09, Samuel Mendoza-Jonas <sam@mendozajonas.com> wrote: > In glibc 2.27 the following change occured: > "Statically compiled applications attempting to load locales compiled > for the GNU C Library version 2.27 will fail and fall back to the > builtin C/POSIX locale." > > This impacts us since upstream buildroot uses a localdef built against > an older eglibc release [0]. > Update the host-localedef package to build against glibc 2.27. > Unfortunately we can't compile just the localedef utility, but even so > this does not increase the build time too significantly. eg: > > With glibc localedef: > real 33m11.278s > user 171m51.684s > sys 19m46.528s > > With eglibc localedef: > real 31m12.715s > user 155m13.616s > sys 16m23.504s > > [0] https://bugs.busybox.net/show_bug.cgi?id=11096 > > Signed-off-by: Samuel Mendoza-Jonas <sam@mendozajonas.com> > --- > This is a patch we're using in our own Buildroot branch to keep us going > until a solution is found for the above bug report. While it does add a > few minutes to our example build it avoids hacking glibc to only build > localedef which is a solution some other users have gone for. > Definitely an RFC as I'm sure some of the configuration in localedef.mk > could be trimmed down but I took the safer bet of basing it off the > existing glibc.mk. Thanks for doing this and submitting it upstream Sam. I think this is idea preferable to the other solution we have, as it is easier to maintain. The implementation might be able to be tweaked a little, but I will defer to our buildroot maintainers on that. Cheers, Joel > > > .../0001-Don-t-include-xlocale.h.patch | 50 ----------- > package/localedef/localedef.hash | 2 +- > package/localedef/localedef.mk | 87 +++++++++++++++++-- > 3 files changed, 79 insertions(+), 60 deletions(-) > delete mode 100644 package/localedef/0001-Don-t-include-xlocale.h.patch > > diff --git a/package/localedef/0001-Don-t-include-xlocale.h.patch b/package/localedef/0001-Don-t-include-xlocale.h.patch > deleted file mode 100644 > index 9a6c2f8e90..0000000000 > --- a/package/localedef/0001-Don-t-include-xlocale.h.patch > +++ /dev/null > @@ -1,50 +0,0 @@ > -From: Bernhard Walle <bernhard@bwalle.de> > -Date: Mon, 2 Oct 2017 16:55:23 +0200 > -Subject: [PATCH] Don't include <xlocale.h> > - > -This header has been removed in glibc 2.26: > - > -https://abi-laboratory.pro/tracker/changelog/glibc/2.26/log.html > - > ------------------------- >8 ------------------------ > -* The nonstandard header <xlocale.h> has been removed. Most programs should > - use <locale.h> instead. If you have a specific need for the definition of > - locale_t with no other declarations, please contact > - libc-alpha@sourceware.org and explain. > ------------------------- 8< ------------------------ > - > -Signed-off-by: Bernhard Walle <bernhard@bwalle.de> > - > -Upstream: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-eglibc-2.14.1-r17443-ptx1/0003-Don-t-include-xlocale.h.patch > -Signed-off-by: Peter Seiderer <ps.report@gmx.net> > ---- > - eglibc/locale/langinfo.h | 2 +- > - eglibc/locale/locale.h | 2 +- > - 2 files changed, 2 insertions(+), 2 deletions(-) > - > -diff --git a/eglibc/locale/langinfo.h b/eglibc/locale/langinfo.h > -index 0a5336507196..76707a4584c1 100644 > ---- a/eglibc/locale/langinfo.h > -+++ b/eglibc/locale/langinfo.h > -@@ -589,7 +589,7 @@ extern char *nl_langinfo (nl_item __item) __THROW; > - more information. */ > - > - /* Get locale datatype definition. */ > --# include <xlocale.h> > -+# include <locale.h> > - > - /* Just like nl_langinfo but get the information from the locale object L. */ > - extern char *nl_langinfo_l (nl_item __item, __locale_t __l); > -diff --git a/eglibc/locale/locale.h b/eglibc/locale/locale.h > -index 2aa19e76acb2..18be711a2d40 100644 > ---- a/eglibc/locale/locale.h > -+++ b/eglibc/locale/locale.h > -@@ -143,7 +143,7 @@ __END_NAMESPACE_STD > - This is a proof-of-concept implementation. */ > - > - /* Get locale datatype definition. */ > --# include <xlocale.h> > -+# include_next <locale.h> > - > - /* Return a reference to a data structure representing a set of locale > - datasets. Unlike for the CATEGORY parameter for `setlocale' the > diff --git a/package/localedef/localedef.hash b/package/localedef/localedef.hash > index ee14fb16b8..6f2d6313b6 100644 > --- a/package/localedef/localedef.hash > +++ b/package/localedef/localedef.hash > @@ -1,2 +1,2 @@ > # Locally calculated > -sha256 9a60f7cdab6fb39adf23a12102f2d950d5f07f0cd7e51e85ec327e07440a79c6 localedef-eglibc-2.14.1-r17443-ptx1.tar.bz2 > +sha256 33189b3f10c88730a1f686fac794bc01f31765f12ffd75bc5e8a0f2a690d217a localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e.tar.gz > diff --git a/package/localedef/localedef.mk b/package/localedef/localedef.mk > index 11d9ba3848..c93132cc3f 100644 > --- a/package/localedef/localedef.mk > +++ b/package/localedef/localedef.mk > @@ -4,19 +4,88 @@ > # > ################################################################################ > > -LOCALEDEF_VERSION = 2.14.1-r17443-ptx1 > -LOCALEDEF_SOURCE = localedef-eglibc-$(LOCALEDEF_VERSION).tar.bz2 > -LOCALEDEF_SITE = http://www.pengutronix.de/software/ptxdist/temporary-src > +# Use the same VERSION and SITE as target glibc > +LOCALEDEF_VERSION = glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e > +LOCALEDEF_SITE = $(call github,bminor,glibc,$(GLIBC_VERSION)) > > -HOST_LOCALEDEF_CONF_OPTS += \ > - --prefix=/usr \ > - --with-glibc=./eglibc > +HOST_LOCALEDEF_LICENSE = GPL-2.0+ (programs), LGPL-2.1+, BSD-3-Clause, MIT (library) > +HOST_LOCALEDEF_LICENSE_FILES = COPYING COPYING.LIB LICENSES > > -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline" > +# glibc is part of the toolchain so disable the toolchain dependency > +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO > > -# The makefile does not implement an install target > +HOST_LOCALEDEF_SUBDIR = build > + > +# Thumb build is broken, build in ARM mode > +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm > +endif > + > +# MIPS64 defaults to n32 so pass the correct -mabi if > +# we are using a different ABI. OABI32 is also used > +# in MIPS so we pass -mabi=32 in this case as well > +# even though it's not strictly necessary. > +ifeq ($(BR2_MIPS_NABI64),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64 > +else ifeq ($(BR2_MIPS_OABI32),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32 > +endif > + > +ifeq ($(BR2_ENABLE_DEBUG),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -g > +endif > + > +# The stubs.h header is not installed by install-headers, but is > +# needed for the gcc build. An empty stubs.h will work, as explained > +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick > +# is used by Crosstool-NG. > +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y) > +define HOST_LOCALEDEF_ADD_MISSING_STUB_H > + mkdir -p $(STAGING_DIR)/usr/include/gnu > + touch $(STAGING_DIR)/usr/include/gnu/stubs.h > +endef > +endif > + > +# Even though we use the autotools-package infrastructure, we have to > +# override the default configure commands for several reasons: > +# > +# 1. We have to build out-of-tree, but we can't use the same > +# 'symbolic link to configure' used with the gcc packages. > +# > +# 2. We have to execute the configure script with bash and not sh. > +# > +# Note that as mentionned in > +# http://patches.openembedded.org/patch/38849/, glibc must be > +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below. > +define HOST_LOCALEDEF_CONFIGURE_CMDS > + mkdir -p $(@D)/build > + # Do the configuration > + (cd $(@D)/build; \ > + $(HOST_CONFIGURE_OPTS) \ > + CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \ > + CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \ > + $(SHELL) $(@D)/configure \ > + ac_cv_path_BASH_SHELL=/bin/bash \ > + libc_cv_forced_unwind=yes \ > + libc_cv_ssp=no \ > + --target=$(GNU_HOST_NAME) \ > + --host=$(GNU_HOST_NAME) \ > + --build=$(GNU_HOST_NAME) \ > + --prefix=/usr \ > + $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \ > + $(if $(BR2_x86_64),--enable-lock-elision) \ > + --with-pkgversion="Buildroot" \ > + --without-cvs \ > + --disable-profile \ > + --without-gd \ > + --enable-obsolete-rpc) > + $(HOST_LOCALEDEF_ADD_MISSING_STUB_H) > +endef > + > +$(eval $(autotools-package)) > +# The makefile does not implement an install target for localedef > define HOST_LOCALEDEF_INSTALL_CMDS > - $(INSTALL) -D -m 0755 $(@D)/localedef $(HOST_DIR)/bin/localedef > + $(INSTALL) -D -m 0755 $(@D)/build/locale/localedef $(HOST_DIR)/bin/localedef > endef > > $(eval $(host-autotools-package)) > -- > 2.18.0 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Hello Samuel, Thanks for taking a look into this! On Mon, 30 Jul 2018 15:39:36 +1000, Samuel Mendoza-Jonas wrote: > In glibc 2.27 the following change occured: > "Statically compiled applications attempting to load locales compiled > for the GNU C Library version 2.27 will fail and fall back to the > builtin C/POSIX locale." I don't really understand this explanation. Isn't the problem that glibc 2.27 doesn't read properly locale files generated by localedef from glibc 2.24 ? I kind of fail to parse the explanation given here :-/ > This impacts us since upstream buildroot uses a localdef built against > an older eglibc release [0]. > Update the host-localedef package to build against glibc 2.27. > Unfortunately we can't compile just the localedef utility, but even so > this does not increase the build time too significantly. eg: > > With glibc localedef: > real 33m11.278s > user 171m51.684s > sys 19m46.528s > > With eglibc localedef: > real 31m12.715s > user 155m13.616s > sys 16m23.504s Meh, two more minutes just to build some not so complicated user-space program :-/ > -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline" > +# glibc is part of the toolchain so disable the toolchain dependency > +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO Not needed for host-localedef. > -# The makefile does not implement an install target > +HOST_LOCALEDEF_SUBDIR = build > + > +# Thumb build is broken, build in ARM mode > +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y) This is a target related option... > +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm ... so it is not relevant to observe it when building a host program. > +endif > + > +# MIPS64 defaults to n32 so pass the correct -mabi if > +# we are using a different ABI. OABI32 is also used > +# in MIPS so we pass -mabi=32 in this case as well > +# even though it's not strictly necessary. > +ifeq ($(BR2_MIPS_NABI64),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64 > +else ifeq ($(BR2_MIPS_OABI32),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32 > +endif Same comment. > +ifeq ($(BR2_ENABLE_DEBUG),y) > +HOST_LOCALEDEF_EXTRA_CFLAGS += -g > +endif Same comment. > + > +# The stubs.h header is not installed by install-headers, but is > +# needed for the gcc build. An empty stubs.h will work, as explained > +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick > +# is used by Crosstool-NG. > +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y) > +define HOST_LOCALEDEF_ADD_MISSING_STUB_H > + mkdir -p $(STAGING_DIR)/usr/include/gnu > + touch $(STAGING_DIR)/usr/include/gnu/stubs.h STAGING_DIR is observed when building target packages, not host packages, so this seems useless. > +endef > +endif > + > +# Even though we use the autotools-package infrastructure, we have to > +# override the default configure commands for several reasons: > +# > +# 1. We have to build out-of-tree, but we can't use the same > +# 'symbolic link to configure' used with the gcc packages. > +# > +# 2. We have to execute the configure script with bash and not sh. > +# > +# Note that as mentionned in > +# http://patches.openembedded.org/patch/38849/, glibc must be > +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below. > +define HOST_LOCALEDEF_CONFIGURE_CMDS > + mkdir -p $(@D)/build > + # Do the configuration > + (cd $(@D)/build; \ > + $(HOST_CONFIGURE_OPTS) \ > + CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \ > + CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \ > + $(SHELL) $(@D)/configure \ > + ac_cv_path_BASH_SHELL=/bin/bash \ > + libc_cv_forced_unwind=yes \ > + libc_cv_ssp=no \ > + --target=$(GNU_HOST_NAME) \ > + --host=$(GNU_HOST_NAME) \ > + --build=$(GNU_HOST_NAME) \ > + --prefix=/usr \ Should be --prefix=$(HOST_DIR) > + $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \ > + $(if $(BR2_x86_64),--enable-lock-elision) \ Both BR2_SOFT_FLOAT and BR2_x86_64 are target options, so observing them when building a host package is almost always wrong. > + --with-pkgversion="Buildroot" \ > + --without-cvs \ > + --disable-profile \ > + --without-gd \ > + --enable-obsolete-rpc) > + $(HOST_LOCALEDEF_ADD_MISSING_STUB_H) > +endef With all this, aren't we building a full glibc for the host ? Can you use the locale/others and locale/install-others make targets instead ? See how they are doing it in PTXdist: https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make. Most likely using the locale/others make target will help cut the build time significantly. Thanks! Thomas
On Tue, 2018-07-31 at 22:55 +0200, Thomas Petazzoni wrote: > Hello Samuel, > > Thanks for taking a look into this! > > On Mon, 30 Jul 2018 15:39:36 +1000, Samuel Mendoza-Jonas wrote: > > In glibc 2.27 the following change occured: > > "Statically compiled applications attempting to load locales compiled > > for the GNU C Library version 2.27 will fail and fall back to the > > builtin C/POSIX locale." > > I don't really understand this explanation. Isn't the problem that > glibc 2.27 doesn't read properly locale files generated by localedef > from glibc 2.24 ? > > I kind of fail to parse the explanation given here :-/ I think it's a bit of both; 2.27 introduces new locale descriptions which localedef can't handle, eg: Generating locale ru_RU.UTF-8 /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error And we fall back to the C locale at runtime (at least in my environment). I'll stare at the glibc release notes / bug report a bit more and see if I can make a more legible explanation. > > > This impacts us since upstream buildroot uses a localdef built against > > an older eglibc release [0]. > > Update the host-localedef package to build against glibc 2.27. > > Unfortunately we can't compile just the localedef utility, but even so > > this does not increase the build time too significantly. eg: > > > > With glibc localedef: > > real 33m11.278s > > user 171m51.684s > > sys 19m46.528s > > > > With eglibc localedef: > > real 31m12.715s > > user 155m13.616s > > sys 16m23.504s > > Meh, two more minutes just to build some not so complicated user-space > program :-/ > > > -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline" > > +# glibc is part of the toolchain so disable the toolchain dependency > > +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO > > Not needed for host-localedef. > Thanks for noting this and the others below; I was sure a lot of this was target-only but I figured someone would be able to notice a few of them better than me :) > > -# The makefile does not implement an install target > > +HOST_LOCALEDEF_SUBDIR = build > > + > > +# Thumb build is broken, build in ARM mode > > +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y) > > This is a target related option... > > > +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm > > ... so it is not relevant to observe it when building a host program. > > > +endif > > + > > +# MIPS64 defaults to n32 so pass the correct -mabi if > > +# we are using a different ABI. OABI32 is also used > > +# in MIPS so we pass -mabi=32 in this case as well > > +# even though it's not strictly necessary. > > +ifeq ($(BR2_MIPS_NABI64),y) > > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64 > > +else ifeq ($(BR2_MIPS_OABI32),y) > > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32 > > +endif > > Same comment. > > > +ifeq ($(BR2_ENABLE_DEBUG),y) > > +HOST_LOCALEDEF_EXTRA_CFLAGS += -g > > +endif > > Same comment. > > > + > > +# The stubs.h header is not installed by install-headers, but is > > +# needed for the gcc build. An empty stubs.h will work, as explained > > +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick > > +# is used by Crosstool-NG. > > +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y) > > +define HOST_LOCALEDEF_ADD_MISSING_STUB_H > > + mkdir -p $(STAGING_DIR)/usr/include/gnu > > + touch $(STAGING_DIR)/usr/include/gnu/stubs.h > > STAGING_DIR is observed when building target packages, not host > packages, so this seems useless. > > > +endef > > +endif > > + > > +# Even though we use the autotools-package infrastructure, we have to > > +# override the default configure commands for several reasons: > > +# > > +# 1. We have to build out-of-tree, but we can't use the same > > +# 'symbolic link to configure' used with the gcc packages. > > +# > > +# 2. We have to execute the configure script with bash and not sh. > > +# > > +# Note that as mentionned in > > +# http://patches.openembedded.org/patch/38849/, glibc must be > > +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below. > > +define HOST_LOCALEDEF_CONFIGURE_CMDS > > + mkdir -p $(@D)/build > > + # Do the configuration > > + (cd $(@D)/build; \ > > + $(HOST_CONFIGURE_OPTS) \ > > + CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \ > > + CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \ > > + $(SHELL) $(@D)/configure \ > > + ac_cv_path_BASH_SHELL=/bin/bash \ > > + libc_cv_forced_unwind=yes \ > > + libc_cv_ssp=no \ > > + --target=$(GNU_HOST_NAME) \ > > + --host=$(GNU_HOST_NAME) \ > > + --build=$(GNU_HOST_NAME) \ > > + --prefix=/usr \ > > Should be --prefix=$(HOST_DIR) Oh yes, will fix. > > > + $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \ > > + $(if $(BR2_x86_64),--enable-lock-elision) \ > > Both BR2_SOFT_FLOAT and BR2_x86_64 are target options, so observing > them when building a host package is almost always wrong. > > > + --with-pkgversion="Buildroot" \ > > + --without-cvs \ > > + --disable-profile \ > > + --without-gd \ > > + --enable-obsolete-rpc) > > + $(HOST_LOCALEDEF_ADD_MISSING_STUB_H) > > +endef > > With all this, aren't we building a full glibc for the host ? > > Can you use the locale/others and locale/install-others make targets > instead ? > > See how they are doing it in PTXdist: > https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make. > Most likely using the locale/others make target will help cut the build > time significantly. Yes this is the contentious part, we're building a whole glibc. Unfortunately if we want to build just the locale/others target like PTXdist we also need their bonus patch to glibc: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch Otherwise the build fails with, for example: make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'. Stop. Makefile:215: recipe for target 'locale/others' failed make[3]: *** [locale/others] Error 2 Makefile:9: recipe for target 'locale/others' failed make[2]: *** [locale/others] Error 2 So the tradeoff is adding a few minutes of extra build time versus carrying a bit of a hack to enable just the localedef/others target. > > Thanks! > > Thomas
Hello, On Wed, 01 Aug 2018 11:31:16 +1000, Samuel Mendoza-Jonas wrote: > I think it's a bit of both; 2.27 introduces new locale descriptions which > localedef can't handle, eg: > > Generating locale ru_RU.UTF-8 > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error Ah, so I guess this update of localedef is going to fix https://bugs.busybox.net/show_bug.cgi?id=11096. > And we fall back to the C locale at runtime (at least in my environment). > I'll stare at the glibc release notes / bug report a bit more and see if > I can make a more legible explanation. Would be nice, though it generally makes sense to have a localedef implementation that is in sync with glibc. > > See how they are doing it in PTXdist: > > https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make. > > Most likely using the locale/others make target will help cut the build > > time significantly. > > Yes this is the contentious part, we're building a whole glibc. Unfortunately > if we want to build just the locale/others target like PTXdist we also need > their bonus patch to glibc: > https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch > > Otherwise the build fails with, for example: > > make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'. Stop. > Makefile:215: recipe for target 'locale/others' failed > make[3]: *** [locale/others] Error 2 > Makefile:9: recipe for target 'locale/others' failed > make[2]: *** [locale/others] Error 2 > > So the tradeoff is adding a few minutes of extra build time versus carrying > a bit of a hack to enable just the localedef/others target. Their patch is not that crazy IMO, it should be too complicated to maintain. Perhaps we could even improve it a bit further to make it acceptable upstream ? Best regards, Thomas
Hello *, On Wed, 1 Aug 2018 08:39:38 +0200, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Hello, > > On Wed, 01 Aug 2018 11:31:16 +1000, Samuel Mendoza-Jonas wrote: > > > I think it's a bit of both; 2.27 introduces new locale descriptions which > > localedef can't handle, eg: > > > > Generating locale ru_RU.UTF-8 > > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error > > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error > > Ah, so I guess this update of localedef is going to fix > https://bugs.busybox.net/show_bug.cgi?id=11096. > > > And we fall back to the C locale at runtime (at least in my environment). > > I'll stare at the glibc release notes / bug report a bit more and see if > > I can make a more legible explanation. > > Would be nice, though it generally makes sense to have a localedef > implementation that is in sync with glibc. > > > > See how they are doing it in PTXdist: > > > https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make. > > > Most likely using the locale/others make target will help cut the build > > > time significantly. > > > > Yes this is the contentious part, we're building a whole glibc. Unfortunately > > if we want to build just the locale/others target like PTXdist we also need > > their bonus patch to glibc: > > https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch > > > > Otherwise the build fails with, for example: > > > > make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'. Stop. > > Makefile:215: recipe for target 'locale/others' failed > > make[3]: *** [locale/others] Error 2 > > Makefile:9: recipe for target 'locale/others' failed > > make[2]: *** [locale/others] Error 2 > > > > So the tradeoff is adding a few minutes of extra build time versus carrying > > a bit of a hack to enable just the localedef/others target. > > Their patch is not that crazy IMO, it should be too complicated to > maintain. Perhaps we could even improve it a bit further to make it > acceptable upstream ? > > Best regards, > l > Thomas Just stumbled over the same (or similar) locale problem with buildroot-2018.11/glibc-2.28, fixed by the suggested patch. Just send out a follow up RFC patch with various comments addressed, fixes and updates, see [1]... Regards, Peter [1] https://patchwork.ozlabs.org/patch/1008473/
diff --git a/package/localedef/0001-Don-t-include-xlocale.h.patch b/package/localedef/0001-Don-t-include-xlocale.h.patch deleted file mode 100644 index 9a6c2f8e90..0000000000 --- a/package/localedef/0001-Don-t-include-xlocale.h.patch +++ /dev/null @@ -1,50 +0,0 @@ -From: Bernhard Walle <bernhard@bwalle.de> -Date: Mon, 2 Oct 2017 16:55:23 +0200 -Subject: [PATCH] Don't include <xlocale.h> - -This header has been removed in glibc 2.26: - -https://abi-laboratory.pro/tracker/changelog/glibc/2.26/log.html - ------------------------- >8 ------------------------ -* The nonstandard header <xlocale.h> has been removed. Most programs should - use <locale.h> instead. If you have a specific need for the definition of - locale_t with no other declarations, please contact - libc-alpha@sourceware.org and explain. ------------------------- 8< ------------------------ - -Signed-off-by: Bernhard Walle <bernhard@bwalle.de> - -Upstream: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-eglibc-2.14.1-r17443-ptx1/0003-Don-t-include-xlocale.h.patch -Signed-off-by: Peter Seiderer <ps.report@gmx.net> ---- - eglibc/locale/langinfo.h | 2 +- - eglibc/locale/locale.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -diff --git a/eglibc/locale/langinfo.h b/eglibc/locale/langinfo.h -index 0a5336507196..76707a4584c1 100644 ---- a/eglibc/locale/langinfo.h -+++ b/eglibc/locale/langinfo.h -@@ -589,7 +589,7 @@ extern char *nl_langinfo (nl_item __item) __THROW; - more information. */ - - /* Get locale datatype definition. */ --# include <xlocale.h> -+# include <locale.h> - - /* Just like nl_langinfo but get the information from the locale object L. */ - extern char *nl_langinfo_l (nl_item __item, __locale_t __l); -diff --git a/eglibc/locale/locale.h b/eglibc/locale/locale.h -index 2aa19e76acb2..18be711a2d40 100644 ---- a/eglibc/locale/locale.h -+++ b/eglibc/locale/locale.h -@@ -143,7 +143,7 @@ __END_NAMESPACE_STD - This is a proof-of-concept implementation. */ - - /* Get locale datatype definition. */ --# include <xlocale.h> -+# include_next <locale.h> - - /* Return a reference to a data structure representing a set of locale - datasets. Unlike for the CATEGORY parameter for `setlocale' the diff --git a/package/localedef/localedef.hash b/package/localedef/localedef.hash index ee14fb16b8..6f2d6313b6 100644 --- a/package/localedef/localedef.hash +++ b/package/localedef/localedef.hash @@ -1,2 +1,2 @@ # Locally calculated -sha256 9a60f7cdab6fb39adf23a12102f2d950d5f07f0cd7e51e85ec327e07440a79c6 localedef-eglibc-2.14.1-r17443-ptx1.tar.bz2 +sha256 33189b3f10c88730a1f686fac794bc01f31765f12ffd75bc5e8a0f2a690d217a localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e.tar.gz diff --git a/package/localedef/localedef.mk b/package/localedef/localedef.mk index 11d9ba3848..c93132cc3f 100644 --- a/package/localedef/localedef.mk +++ b/package/localedef/localedef.mk @@ -4,19 +4,88 @@ # ################################################################################ -LOCALEDEF_VERSION = 2.14.1-r17443-ptx1 -LOCALEDEF_SOURCE = localedef-eglibc-$(LOCALEDEF_VERSION).tar.bz2 -LOCALEDEF_SITE = http://www.pengutronix.de/software/ptxdist/temporary-src +# Use the same VERSION and SITE as target glibc +LOCALEDEF_VERSION = glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e +LOCALEDEF_SITE = $(call github,bminor,glibc,$(GLIBC_VERSION)) -HOST_LOCALEDEF_CONF_OPTS += \ - --prefix=/usr \ - --with-glibc=./eglibc +HOST_LOCALEDEF_LICENSE = GPL-2.0+ (programs), LGPL-2.1+, BSD-3-Clause, MIT (library) +HOST_LOCALEDEF_LICENSE_FILES = COPYING COPYING.LIB LICENSES -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline" +# glibc is part of the toolchain so disable the toolchain dependency +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO -# The makefile does not implement an install target +HOST_LOCALEDEF_SUBDIR = build + +# Thumb build is broken, build in ARM mode +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y) +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm +endif + +# MIPS64 defaults to n32 so pass the correct -mabi if +# we are using a different ABI. OABI32 is also used +# in MIPS so we pass -mabi=32 in this case as well +# even though it's not strictly necessary. +ifeq ($(BR2_MIPS_NABI64),y) +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64 +else ifeq ($(BR2_MIPS_OABI32),y) +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32 +endif + +ifeq ($(BR2_ENABLE_DEBUG),y) +HOST_LOCALEDEF_EXTRA_CFLAGS += -g +endif + +# The stubs.h header is not installed by install-headers, but is +# needed for the gcc build. An empty stubs.h will work, as explained +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick +# is used by Crosstool-NG. +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y) +define HOST_LOCALEDEF_ADD_MISSING_STUB_H + mkdir -p $(STAGING_DIR)/usr/include/gnu + touch $(STAGING_DIR)/usr/include/gnu/stubs.h +endef +endif + +# Even though we use the autotools-package infrastructure, we have to +# override the default configure commands for several reasons: +# +# 1. We have to build out-of-tree, but we can't use the same +# 'symbolic link to configure' used with the gcc packages. +# +# 2. We have to execute the configure script with bash and not sh. +# +# Note that as mentionned in +# http://patches.openembedded.org/patch/38849/, glibc must be +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below. +define HOST_LOCALEDEF_CONFIGURE_CMDS + mkdir -p $(@D)/build + # Do the configuration + (cd $(@D)/build; \ + $(HOST_CONFIGURE_OPTS) \ + CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \ + CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \ + $(SHELL) $(@D)/configure \ + ac_cv_path_BASH_SHELL=/bin/bash \ + libc_cv_forced_unwind=yes \ + libc_cv_ssp=no \ + --target=$(GNU_HOST_NAME) \ + --host=$(GNU_HOST_NAME) \ + --build=$(GNU_HOST_NAME) \ + --prefix=/usr \ + $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \ + $(if $(BR2_x86_64),--enable-lock-elision) \ + --with-pkgversion="Buildroot" \ + --without-cvs \ + --disable-profile \ + --without-gd \ + --enable-obsolete-rpc) + $(HOST_LOCALEDEF_ADD_MISSING_STUB_H) +endef + +$(eval $(autotools-package)) +# The makefile does not implement an install target for localedef define HOST_LOCALEDEF_INSTALL_CMDS - $(INSTALL) -D -m 0755 $(@D)/localedef $(HOST_DIR)/bin/localedef + $(INSTALL) -D -m 0755 $(@D)/build/locale/localedef $(HOST_DIR)/bin/localedef endef $(eval $(host-autotools-package))
In glibc 2.27 the following change occured: "Statically compiled applications attempting to load locales compiled for the GNU C Library version 2.27 will fail and fall back to the builtin C/POSIX locale." This impacts us since upstream buildroot uses a localdef built against an older eglibc release [0]. Update the host-localedef package to build against glibc 2.27. Unfortunately we can't compile just the localedef utility, but even so this does not increase the build time too significantly. eg: With glibc localedef: real 33m11.278s user 171m51.684s sys 19m46.528s With eglibc localedef: real 31m12.715s user 155m13.616s sys 16m23.504s [0] https://bugs.busybox.net/show_bug.cgi?id=11096 Signed-off-by: Samuel Mendoza-Jonas <sam@mendozajonas.com> --- This is a patch we're using in our own Buildroot branch to keep us going until a solution is found for the above bug report. While it does add a few minutes to our example build it avoids hacking glibc to only build localedef which is a solution some other users have gone for. Definitely an RFC as I'm sure some of the configuration in localedef.mk could be trimmed down but I took the safer bet of basing it off the existing glibc.mk. .../0001-Don-t-include-xlocale.h.patch | 50 ----------- package/localedef/localedef.hash | 2 +- package/localedef/localedef.mk | 87 +++++++++++++++++-- 3 files changed, 79 insertions(+), 60 deletions(-) delete mode 100644 package/localedef/0001-Don-t-include-xlocale.h.patch