| Message ID | 20201017221743.160029-2-nolange79@gmail.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [v2,1/4] package/Makefile.in: use gcc wrappers for binutils tools | expand |
Nrobert, All, On 2020-10-18 00:17 +0200, Norbert Lange spake thusly: > This will use gcc-ar, gcc-nm and gcc-ranlib instead of the > normal binutils tools. The difference is that with the > wrappers, gcc plugins will be automatically picked up, > which might be necessary with some flags. > > Signed-off-by: Norbert Lange <nolange79@gmail.com> > --- > v1->v2: > * Only set those wrappers if gc 4.7 or newer > --- > package/gcc/gcc.mk | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk > index 114c2887a0..a24ffe3ee2 100644 > --- a/package/gcc/gcc.mk > +++ b/package/gcc/gcc.mk > @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error > endif > > # Propagate options used for target software building to GCC target libs > +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y) > +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib So we were a bit skeptical about this one. First, we believe that only the final gcc should be using those. Indeed, we do not need much of the initial gcc, except libgcc.a, which anyway gets overwitten by the final gcc, so we do not care that the libgcc.a is built with LTO (and thus the gcc wrappers). Second, gcc initial is only used to build glibc, and as your patch 4 states, glibc must not be built with LTO, we we also do not care that the initial gcc uses its wrappers to build libgcc. Third, shouldn't we be using the tupple-prefixed wrappers? Or does it just happens that gcc will internally add its build directory to PATH so that it can find the un-prefixed just-built ones? For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH, that are just simple wrapper scripts that trace their being called, and then call to the actual gcca-r et al.: #!/bin/sh me="${0}" my_name="${0##*/}" found=false for i in $(which -a "${my_name}"); do if [ "${i}" -ef "${me}" ]; then found=true elif ${found}; then me="${i}" fi done if ! ${found}; then exit 42 fi printf "YEM: wrapped %s\n" "${me}" exec "${me}" "${@}" Turns out that the 'YEM: wrapped' string occurs nowhere in the build directory (so it does not end up in config.log or whatever), and it does not even appear on the stdout/stderr either. So I am wondering how we can exercise these... Any hint? Regards, Yann E. MORIN. > +endif > HOST_GCC_COMMON_CONF_ENV += CFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CFLAGS)" > HOST_GCC_COMMON_CONF_ENV += CXXFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CXXFLAGS)" > > -- > 2.28.0
Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN <yann.morin.1998@free.fr>: > > Nrobert, All, > > On 2020-10-18 00:17 +0200, Norbert Lange spake thusly: > > This will use gcc-ar, gcc-nm and gcc-ranlib instead of the > > normal binutils tools. The difference is that with the > > wrappers, gcc plugins will be automatically picked up, > > which might be necessary with some flags. > > > > Signed-off-by: Norbert Lange <nolange79@gmail.com> > > --- > > v1->v2: > > * Only set those wrappers if gc 4.7 or newer > > --- > > package/gcc/gcc.mk | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk > > index 114c2887a0..a24ffe3ee2 100644 > > --- a/package/gcc/gcc.mk > > +++ b/package/gcc/gcc.mk > > @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error > > endif > > > > # Propagate options used for target software building to GCC target libs > > +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y) > > +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib > > So we were a bit skeptical about this one. > > First, we believe that only the final gcc should be using those. Indeed, > we do not need much of the initial gcc, except libgcc.a, which anyway > gets overwitten by the final gcc, so we do not care that the libgcc.a is > built with LTO (and thus the gcc wrappers). > > Second, gcc initial is only used to build glibc, and as your patch 4 > states, glibc must not be built with LTO, we we also do not care that > the initial gcc uses its wrappers to build libgcc. > > Third, shouldn't we be using the tupple-prefixed wrappers? Or does it > just happens that gcc will internally add its build directory to PATH so > that it can find the un-prefixed just-built ones? > > For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH, > that are just simple wrapper scripts that trace their being called, and > then call to the actual gcca-r et al.: > > #!/bin/sh > me="${0}" > my_name="${0##*/}" > found=false > for i in $(which -a "${my_name}"); do > if [ "${i}" -ef "${me}" ]; then > found=true > elif ${found}; then > me="${i}" > fi > done > if ! ${found}; then > exit 42 > fi > printf "YEM: wrapped %s\n" "${me}" > exec "${me}" "${@}" > > Turns out that the 'YEM: wrapped' string occurs nowhere in the build > directory (so it does not end up in config.log or whatever), and it > does not even appear on the stdout/stderr either. > > So I am wondering how we can exercise these... Any hint? Can't tell anymore, I went to crosstool-ng for the compiler some time ago, so unlike my other patches I dont use them anymore. From my memory, it affected some of the additional libs gcc builds, libstdc++, some sanitizers maybe. there was not absolute paths or tuples as the various stages should use the wrappers from the previous ones. I believe the final compiler ignores PATH and uses the stage2-compiler with abs path to build the included target libraries. I am just sure it was very deliberately done *this* way. Norbert
On 09/01/2022 22:08, Norbert Lange wrote: > Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN > <yann.morin.1998@free.fr>: >> >> Nrobert, All, >> >> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly: >>> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the >>> normal binutils tools. The difference is that with the >>> wrappers, gcc plugins will be automatically picked up, >>> which might be necessary with some flags. >>> >>> Signed-off-by: Norbert Lange <nolange79@gmail.com> >>> --- >>> v1->v2: >>> * Only set those wrappers if gc 4.7 or newer >>> --- >>> package/gcc/gcc.mk | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk >>> index 114c2887a0..a24ffe3ee2 100644 >>> --- a/package/gcc/gcc.mk >>> +++ b/package/gcc/gcc.mk >>> @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error >>> endif >>> >>> # Propagate options used for target software building to GCC target libs >>> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y) >>> +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib >> >> So we were a bit skeptical about this one. >> >> First, we believe that only the final gcc should be using those. Indeed, >> we do not need much of the initial gcc, except libgcc.a, which anyway >> gets overwitten by the final gcc, so we do not care that the libgcc.a is >> built with LTO (and thus the gcc wrappers). >> >> Second, gcc initial is only used to build glibc, and as your patch 4 >> states, glibc must not be built with LTO, we we also do not care that >> the initial gcc uses its wrappers to build libgcc. Although I may have been the one to make this obvservation to begin with, I'm starting to come back on that. Even though indeed it is probably not very useful to apply the wrappers to gcc-initial, it doesn't hurt either. And by putting it in _COMMON_CONF_ENV, we make sure that (almost) all of the options are together in gcc.mk. Note that --enable-lto is also in COMMON_OPTS. >> >> Third, shouldn't we be using the tupple-prefixed wrappers? Or does it >> just happens that gcc will internally add its build directory to PATH so >> that it can find the un-prefixed just-built ones? >> >> For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH, >> that are just simple wrapper scripts that trace their being called, and >> then call to the actual gcca-r et al.: >> >> #!/bin/sh >> me="${0}" >> my_name="${0##*/}" >> found=false >> for i in $(which -a "${my_name}"); do >> if [ "${i}" -ef "${me}" ]; then >> found=true >> elif ${found}; then >> me="${i}" >> fi >> done >> if ! ${found}; then >> exit 42 >> fi >> printf "YEM: wrapped %s\n" "${me}" >> exec "${me}" "${@}" >> >> Turns out that the 'YEM: wrapped' string occurs nowhere in the build >> directory (so it does not end up in config.log or whatever), and it >> does not even appear on the stdout/stderr either. >> >> So I am wondering how we can exercise these... Any hint? > > Can't tell anymore, I went to crosstool-ng for the compiler some time ago, > so unlike my other patches I dont use them anymore. > > From my memory, it affected some of the additional libs gcc builds, > libstdc++, some sanitizers maybe. > there was not absolute paths or tuples as the various stages should > use the wrappers from the previous ones. I believe the final compiler > ignores PATH and uses the stage2-compiler with abs path to build the > included target libraries. That it doesn't look in PATH indeed sounds feasible. However, what is very surprising is that GCC doesn't automatically use the plugins when --enable-lto is passed. Before blindly adding some random options, we want to really understand why it is needed, in case maybe things are already broken in some other way, and to be able in the future to make changes and understand if they're OK or not. That said, I have checked and indeed, without these options it seems to use plain ar instead of gcc-ar. So I looked a bit deeper, and found this really old bug about enabling lto for libgcc [1]. It refers to a bunch of other issues found while doing this. For me, however, the conclusion is that the patch should be applied, but with better explanation. That way, you can set BR2_TARGET_OPTIMIZATION to -flto and get an LTO'ed libgcc/libstdc++/etc. If LTO is not used, it doesn't harm to use the wrapper. There's just one little concern though: I think the wrappers aren't even generated unless LTO is enabled. But that point is moot - since GCC N (where N > 4.9 and < 9), LTO is enabled by default, and we don't pass --disable-lto if LTO is not enabled. In other words, nowadays the BR2_GCC_ENABLE_LTO symbol is useless. Regards, Arnout [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59893 > > I am just sure it was very deliberately done *this* way. > > Norbert > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot >
On 10/01/2022 11:43, Arnout Vandecappelle wrote: > > > On 09/01/2022 22:08, Norbert Lange wrote: >> Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN >> <yann.morin.1998@free.fr>: >>> >>> Nrobert, All, >>> >>> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly: >>>> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the >>>> normal binutils tools. The difference is that with the >>>> wrappers, gcc plugins will be automatically picked up, >>>> which might be necessary with some flags. >>>> >>>> Signed-off-by: Norbert Lange <nolange79@gmail.com> >>>> --- >>>> v1->v2: >>>> * Only set those wrappers if gc 4.7 or newer >>>> --- >>>> package/gcc/gcc.mk | 3 +++ >>>> 1 file changed, 3 insertions(+) >>>> >>>> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk >>>> index 114c2887a0..a24ffe3ee2 100644 >>>> --- a/package/gcc/gcc.mk >>>> +++ b/package/gcc/gcc.mk >>>> @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error >>>> endif >>>> >>>> # Propagate options used for target software building to GCC target libs >>>> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y) This is actually not needed - the lowest GCC version we have for the internal toolchain is 8. >>>> +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm >>>> RANLIB_FOR_TARGET=gcc-ranlib >>> >>> So we were a bit skeptical about this one. >>> >>> First, we believe that only the final gcc should be using those. Indeed, >>> we do not need much of the initial gcc, except libgcc.a, which anyway >>> gets overwitten by the final gcc, so we do not care that the libgcc.a is >>> built with LTO (and thus the gcc wrappers). >> >>> Second, gcc initial is only used to build glibc, and as your patch 4 >>> states, glibc must not be built with LTO, we we also do not care that >>> the initial gcc uses its wrappers to build libgcc. > > Although I may have been the one to make this obvservation to begin with, I'm > starting to come back on that. Even though indeed it is probably not very > useful to apply the wrappers to gcc-initial, it doesn't hurt either. And by > putting it in _COMMON_CONF_ENV, we make sure that (almost) all of the options > are together in gcc.mk. Note that --enable-lto is also in COMMON_OPTS. > >>> >>> Third, shouldn't we be using the tupple-prefixed wrappers? Or does it >>> just happens that gcc will internally add its build directory to PATH so >>> that it can find the un-prefixed just-built ones? >>> >>> For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH, >>> that are just simple wrapper scripts that trace their being called, and >>> then call to the actual gcca-r et al.: >>> >>> #!/bin/sh >>> me="${0}" >>> my_name="${0##*/}" >>> found=false >>> for i in $(which -a "${my_name}"); do >>> if [ "${i}" -ef "${me}" ]; then >>> found=true >>> elif ${found}; then >>> me="${i}" >>> fi >>> done >>> if ! ${found}; then >>> exit 42 >>> fi >>> printf "YEM: wrapped %s\n" "${me}" >>> exec "${me}" "${@}" >>> >>> Turns out that the 'YEM: wrapped' string occurs nowhere in the build >>> directory (so it does not end up in config.log or whatever), and it >>> does not even appear on the stdout/stderr either. >>> >>> So I am wondering how we can exercise these... Any hint? >> >> Can't tell anymore, I went to crosstool-ng for the compiler some time ago, >> so unlike my other patches I dont use them anymore. > >> From my memory, it affected some of the additional libs gcc builds, >> libstdc++, some sanitizers maybe. >> there was not absolute paths or tuples as the various stages should >> use the wrappers from the previous ones. I believe the final compiler >> ignores PATH and uses the stage2-compiler with abs path to build the >> included target libraries. > > That it doesn't look in PATH indeed sounds feasible. > > However, what is very surprising is that GCC doesn't automatically use the > plugins when --enable-lto is passed. Before blindly adding some random > options, we want to really understand why it is needed, in case maybe things > are already broken in some other way, and to be able in the future to make > changes and understand if they're OK or not. > > That said, I have checked and indeed, without these options it seems to use > plain ar instead of gcc-ar. > > So I looked a bit deeper, and found this really old bug about enabling lto > for libgcc [1]. It refers to a bunch of other issues found while doing this. > > For me, however, the conclusion is that the patch should be applied, but with > better explanation. That way, you can set BR2_TARGET_OPTIMIZATION to -flto and > get an LTO'ed libgcc/libstdc++/etc. If LTO is not used, it doesn't harm to use > the wrapper. I summarized this discussion and applied to master, thanks. Regards, Arnout > > There's just one little concern though: I think the wrappers aren't even > generated unless LTO is enabled. But that point is moot - since GCC N (where N > > 4.9 and < 9), LTO is enabled by default, and we don't pass --disable-lto if > LTO is not enabled. In other words, nowadays the BR2_GCC_ENABLE_LTO symbol is > useless. > > > Regards, > Arnout > > > [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59893 > > > >> >> I am just sure it was very deliberately done *this* way. >> >> Norbert >> _______________________________________________ >> buildroot mailing list >> buildroot@buildroot.org >> https://lists.buildroot.org/mailman/listinfo/buildroot >>
diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk index 114c2887a0..a24ffe3ee2 100644 --- a/package/gcc/gcc.mk +++ b/package/gcc/gcc.mk @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error endif # Propagate options used for target software building to GCC target libs +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y) +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib +endif HOST_GCC_COMMON_CONF_ENV += CFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CFLAGS)" HOST_GCC_COMMON_CONF_ENV += CXXFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CXXFLAGS)"
This will use gcc-ar, gcc-nm and gcc-ranlib instead of the normal binutils tools. The difference is that with the wrappers, gcc plugins will be automatically picked up, which might be necessary with some flags. Signed-off-by: Norbert Lange <nolange79@gmail.com> --- v1->v2: * Only set those wrappers if gc 4.7 or newer --- package/gcc/gcc.mk | 3 +++ 1 file changed, 3 insertions(+)