Message ID | 20200411232340.135070-1-maskray@google.com |
---|---|
State | New |
Headers | show |
Series | install: Delete scripts/output-format.sed and support lld | expand |
On 2020-04-11, Fangrui Song wrote: >GNU ld and gold have supported --print-output-format since 2011. glibc >requires binutils>=2.25 (2015), so if LD is GNU ld or gold, we can >assume the option is supported. > >lld is by default a cross linker supporting multiple targets. It auto >detects the file format and does not need OUTPUT_FORMAT. It does not >support --print-output-format, so we can repurpose the >output-format=unknown branch to make format.lds empty. > >GNU ld and gold can skip a script with incompatible OUTPUT_FORMAT and >find the next script in the search path. So libc.so linked with lld >cannot be skipped automatically. However, this is considered a minor >functionality loss by lld maintainers. >--- > Makerules | 12 +++---- > .../strcoll-inputs/filelist#en_US.UTF-8 | 1 - > scripts/output-format.sed | 35 ------------------- > 3 files changed, 4 insertions(+), 44 deletions(-) > delete mode 100644 scripts/output-format.sed > >diff --git a/Makerules b/Makerules >index 1e9c18f0d8..6a6e5579ae 100644 >--- a/Makerules >+++ b/Makerules >@@ -1067,18 +1067,14 @@ install: $(inst_slibdir)/libc.so$(libc.so-version) > # for the configuration we are building. We put this statement into > # the linker scripts we install for -lc et al so that they will not be > # used by a link for a different format on a multi-architecture system. >-$(common-objpfx)format.lds: $(..)scripts/output-format.sed \ >- $(common-objpfx)config.make \ >+$(common-objpfx)format.lds: $(common-objpfx)config.make \ > $(common-objpfx)config.h $(..)Makerules > ifneq (unknown,$(output-format)) > echo > $@.new 'OUTPUT_FORMAT($(output-format))' > else >- $(LINK.o) -shared $(sysdep-LDFLAGS) $(rtld-LDFLAGS) \ >- $(LDFLAGS.so) $(LDFLAGS-lib.so) \ >- -x c /dev/null -o $@.so -Wl,--verbose -v 2>&1 \ >- | sed -n -f $< > $@.new >- test -s $@.new >- rm -f $@.so >+# This branch is taken by lld, which auto detects the file format and does not >+# need OUTPUT_FORMAT. >+ echo > $@.new > endif > mv -f $@.new $@ > common-generated += format.lds >diff --git a/benchtests/strcoll-inputs/filelist#en_US.UTF-8 b/benchtests/strcoll-inputs/filelist#en_US.UTF-8 >index aa44107ad6..cb981de88c 100644 >--- a/benchtests/strcoll-inputs/filelist#en_US.UTF-8 >+++ b/benchtests/strcoll-inputs/filelist#en_US.UTF-8 >@@ -9450,7 +9450,6 @@ move-if-change > check-execstack.awk > pylint > pylintrc >-output-format.sed > merge-test-results.sh > update-copyrights > config-uname.sh >diff --git a/scripts/output-format.sed b/scripts/output-format.sed >deleted file mode 100644 >index 364f52059f..0000000000 >--- a/scripts/output-format.sed >+++ /dev/null >@@ -1,35 +0,0 @@ >-/ld.*[ ]-E[BL]/b f >-/collect.*[ ]-E[BL]/b f >-/OUTPUT_FORMAT[^)]*$/{N >-s/\n[ ]*/ / >-} >-t o >-: o >-s/^.*OUTPUT_FORMAT(\([^,]*\), \1, \1).*$/OUTPUT_FORMAT(\1)/ >-t q >-s/^.*OUTPUT_FORMAT(\([^,]*\), \([^,]*\), \([^,]*\)).*$/\1,\2,\3/ >-t s >-s/^.*OUTPUT_FORMAT(\([^,)]*\).*$)/OUTPUT_FORMAT(\1)/ >-t q >-d >-: s >-s/"//g >-G >-s/\n// >-s/^\([^,]*\),\([^,]*\),\([^,]*\),B/OUTPUT_FORMAT(\2)/p >-s/^\([^,]*\),\([^,]*\),\([^,]*\),L/OUTPUT_FORMAT(\3)/p >-s/^\([^,]*\),\([^,]*\),\([^,]*\)/OUTPUT_FORMAT(\1)/p >-/,/s|^|*** BUG in libc/scripts/output-format.sed *** |p >-q >-: q >-s/"//g >-p >-q >-: f >-s/^.*[ ]-E\([BL]\)[ ].*$/,\1/ >-t h >-s/^.*[ ]-E\([BL]\)$/,\1/ >-t h >-d >-: h >-h >-- >2.26.0.110.g2183baf09c-goog > Ping :)
* Fangrui Song via Libc-alpha: > GNU ld and gold can skip a script with incompatible OUTPUT_FORMAT and > find the next script in the search path. So libc.so linked with lld > cannot be skipped automatically. However, this is considered a minor > functionality loss by lld maintainers. That completely depends on how typical installations set up their linker search paths. Figuring that out is likely more work than providing the correct OUTPUT_FORMAT information in the linker script, so I'm not sure if the patch goes in the right direction. In any case, the choice of linker for building glibc should result in as little semantic difference in the produced artifacts, to avoid inflating the compatibility matrix. If OUTPUT_FORMAT is indeed unnecessary (based on the evaluation I mentioned), it should be removed unconditionally. But as I said, it's probably better to find an alternative way to generate the right OUTPUT_FORMAT directive for the linker script.
On 2020-05-03, Florian Weimer wrote: >* Fangrui Song via Libc-alpha: > >> GNU ld and gold can skip a script with incompatible OUTPUT_FORMAT and >> find the next script in the search path. So libc.so linked with lld >> cannot be skipped automatically. However, this is considered a minor >> functionality loss by lld maintainers. > >That completely depends on how typical installations set up their >linker search paths. Figuring that out is likely more work than >providing the correct OUTPUT_FORMAT information in the linker script, >so I'm not sure if the patch goes in the right direction. For typical installations, OUTPUT_FORMAT is not useful. Even in multilib installations, OUTPUT_FORMAT is not useful unless, for example, /usr/lib is in both -m32 and -m64's library search paths (this is likely a misconfiguration). `OUTPUT_FORMAT(elf64-x86-64)` can provide some protection for GNU ld and gold. >But as I said, it's probably better to find an alternative way to >generate the right OUTPUT_FORMAT directive for the linker script. We (LLD developers) consider OUTPUT_FORMAT with GNU ld/gold semantic has very little value. See https://bugs.llvm.org/show_bug.cgi?id=37432 and https://bugs.llvm.org/show_bug.cgi?id=43740 >In any case, the choice of linker for building glibc should result in >as little semantic difference in the produced artifacts, to avoid >inflating the compatibility matrix. If OUTPUT_FORMAT is indeed >unnecessary (based on the evaluation I mentioned), it should be >removed unconditionally. If there is concern about the aforementioned value, then we probably should not drop OUTPUT_FORMAT for now for GNU ld/gold. If people think we can remove OUTPUT_FORMAT as well, I will be happy to create a follow-up patch.
On Sun, May 3, 2020 at 1:54 PM Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> wrote: > > On 2020-05-03, Florian Weimer wrote: > >* Fangrui Song via Libc-alpha: > > > >> GNU ld and gold can skip a script with incompatible OUTPUT_FORMAT and > >> find the next script in the search path. So libc.so linked with lld > >> cannot be skipped automatically. However, this is considered a minor > >> functionality loss by lld maintainers. > > > >That completely depends on how typical installations set up their > >linker search paths. Figuring that out is likely more work than > >providing the correct OUTPUT_FORMAT information in the linker script, > >so I'm not sure if the patch goes in the right direction. > > For typical installations, OUTPUT_FORMAT is not useful. > Even in multilib installations, OUTPUT_FORMAT is not useful unless, for > example, /usr/lib is in both -m32 and -m64's library search paths (this > is likely a misconfiguration). > > `OUTPUT_FORMAT(elf64-x86-64)` can provide some protection for GNU ld and > gold. > > >But as I said, it's probably better to find an alternative way to > >generate the right OUTPUT_FORMAT directive for the linker script. > > We (LLD developers) consider OUTPUT_FORMAT with GNU ld/gold semantic has > very little value. See https://bugs.llvm.org/show_bug.cgi?id=37432 and > https://bugs.llvm.org/show_bug.cgi?id=43740 > > >In any case, the choice of linker for building glibc should result in > >as little semantic difference in the produced artifacts, to avoid > >inflating the compatibility matrix. If OUTPUT_FORMAT is indeed > >unnecessary (based on the evaluation I mentioned), it should be > >removed unconditionally. > > If there is concern about the aforementioned value, then we probably > should not drop OUTPUT_FORMAT for now for GNU ld/gold. If people think > we can remove OUTPUT_FORMAT as well, I will be happy to create a > follow-up patch. Historical usage of OUTPUT_FORMAT was used to link object files of non-Linux/x86 formats and generate Linux/x86 output. It was very useful during the days when many binary only libraries weren't available for Linux/x86. It is still somewhat useful to link ELF object files for a different OS. In both cases, the format of the first input file isn't the format of the output. That is where OUTPUT_FORMAT comes in.
On Sun, May 3, 2020 at 2:11 PM H.J. Lu <hjl.tools@gmail.com> wrote: > > On Sun, May 3, 2020 at 1:54 PM Fangrui Song via Libc-alpha > <libc-alpha@sourceware.org> wrote: > > > > On 2020-05-03, Florian Weimer wrote: > > >* Fangrui Song via Libc-alpha: > > > > > >> GNU ld and gold can skip a script with incompatible OUTPUT_FORMAT and > > >> find the next script in the search path. So libc.so linked with lld > > >> cannot be skipped automatically. However, this is considered a minor > > >> functionality loss by lld maintainers. > > > > > >That completely depends on how typical installations set up their > > >linker search paths. Figuring that out is likely more work than > > >providing the correct OUTPUT_FORMAT information in the linker script, > > >so I'm not sure if the patch goes in the right direction. > > > > For typical installations, OUTPUT_FORMAT is not useful. > > Even in multilib installations, OUTPUT_FORMAT is not useful unless, for > > example, /usr/lib is in both -m32 and -m64's library search paths (this > > is likely a misconfiguration). > > > > `OUTPUT_FORMAT(elf64-x86-64)` can provide some protection for GNU ld and > > gold. > > > > >But as I said, it's probably better to find an alternative way to > > >generate the right OUTPUT_FORMAT directive for the linker script. > > > > We (LLD developers) consider OUTPUT_FORMAT with GNU ld/gold semantic has > > very little value. See https://bugs.llvm.org/show_bug.cgi?id=37432 and > > https://bugs.llvm.org/show_bug.cgi?id=43740 > > > > >In any case, the choice of linker for building glibc should result in > > >as little semantic difference in the produced artifacts, to avoid > > >inflating the compatibility matrix. If OUTPUT_FORMAT is indeed > > >unnecessary (based on the evaluation I mentioned), it should be > > >removed unconditionally. > > > > If there is concern about the aforementioned value, then we probably > > should not drop OUTPUT_FORMAT for now for GNU ld/gold. If people think > > we can remove OUTPUT_FORMAT as well, I will be happy to create a > > follow-up patch. > > Historical usage of OUTPUT_FORMAT was used to link object files of > non-Linux/x86 formats and generate Linux/x86 output. It was very useful > during the days when many binary only libraries weren't available for > Linux/x86. It is still somewhat useful to link ELF object files for a different > OS. In both cases, the format of the first input file isn't the format of the > output. That is where OUTPUT_FORMAT comes in. > > > -- > H.J. LLD will not support --print-output-format. OUTPUT_FORMAT might be useful at some point and is still needed for unified i386/x86-64 directory trees, but not when distributions use different directory trees for multiarch. Ping
* Fāng-ruì Sòng: > LLD will not support --print-output-format. > > OUTPUT_FORMAT might be useful at some point and is still needed for > unified i386/x86-64 directory trees, but not when distributions use > different directory trees for multiarch. Sorry, I think it's not worthwhile to support LLD for building glibc if it produces output that offers different functionality. Bitwise identical output is of course not necessary, but disabling parts of the build output because it is inconvenient to build them using LLD is not the right direction, in my opinion.
On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: > > * Fāng-ruì Sòng: > > > LLD will not support --print-output-format. > > > > OUTPUT_FORMAT might be useful at some point and is still needed for > > unified i386/x86-64 directory trees, but not when distributions use > > different directory trees for multiarch. > > Sorry, I think it's not worthwhile to support LLD for building glibc > if it produces output that offers different functionality. Bitwise > identical output is of course not necessary, but disabling parts of > the build output because it is inconvenient to build them using LLD is > not the right direction, in my opinion. Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? Or are you simply against removing scripts/output-format.sed
On Mon, Aug 31, 2020 at 11:52 AM Fāng-ruì Sòng <maskray@google.com> wrote: > > On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: > > > > * Fāng-ruì Sòng: > > > > > LLD will not support --print-output-format. > > > > > > OUTPUT_FORMAT might be useful at some point and is still needed for > > > unified i386/x86-64 directory trees, but not when distributions use > > > different directory trees for multiarch. > > > > Sorry, I think it's not worthwhile to support LLD for building glibc > > if it produces output that offers different functionality. Bitwise > > identical output is of course not necessary, but disabling parts of > > the build output because it is inconvenient to build them using LLD is > > not the right direction, in my opinion. > > Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? > > Or are you simply against removing scripts/output-format.sed I think you should open a glibc bug against building with lld.
On Mon, Aug 31, 2020 at 4:11 PM H.J. Lu <hjl.tools@gmail.com> wrote: > > On Mon, Aug 31, 2020 at 11:52 AM Fāng-ruì Sòng <maskray@google.com> wrote: > > > > On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: > > > > > > * Fāng-ruì Sòng: > > > > > > > LLD will not support --print-output-format. > > > > > > > > OUTPUT_FORMAT might be useful at some point and is still needed for > > > > unified i386/x86-64 directory trees, but not when distributions use > > > > different directory trees for multiarch. > > > > > > Sorry, I think it's not worthwhile to support LLD for building glibc > > > if it produces output that offers different functionality. Bitwise > > > identical output is of course not necessary, but disabling parts of > > > the build output because it is inconvenient to build them using LLD is > > > not the right direction, in my opinion. > > > > Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? > > > > Or are you simply against removing scripts/output-format.sed > > I think you should open a glibc bug against building with lld. > > -- > H.J. Filed https://sourceware.org/bugzilla/show_bug.cgi?id=26559 Get OUTPUT_FORMAT from objdump -f to support LLD https://sourceware.org/bugzilla/show_bug.cgi?id=26558 [meta] Build glibc with LLD
* Fāng-ruì Sòng via Libc-alpha: > On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: >> >> * Fāng-ruì Sòng: >> >> > LLD will not support --print-output-format. >> > >> > OUTPUT_FORMAT might be useful at some point and is still needed for >> > unified i386/x86-64 directory trees, but not when distributions use >> > different directory trees for multiarch. >> >> Sorry, I think it's not worthwhile to support LLD for building glibc >> if it produces output that offers different functionality. Bitwise >> identical output is of course not necessary, but disabling parts of >> the build output because it is inconvenient to build them using LLD is >> not the right direction, in my opinion. > > Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? It sounds reasonable. Does LLVM have its own objdump implementation which provides this functionality? Thanks, Florian
On Tue, Sep 1, 2020 at 3:38 AM Florian Weimer <fweimer@redhat.com> wrote: > > * Fāng-ruì Sòng via Libc-alpha: > > > On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: > >> > >> * Fāng-ruì Sòng: > >> > >> > LLD will not support --print-output-format. > >> > > >> > OUTPUT_FORMAT might be useful at some point and is still needed for > >> > unified i386/x86-64 directory trees, but not when distributions use > >> > different directory trees for multiarch. > >> > >> Sorry, I think it's not worthwhile to support LLD for building glibc > >> if it produces output that offers different functionality. Bitwise > >> identical output is of course not necessary, but disabling parts of > >> the build output because it is inconvenient to build them using LLD is > >> not the right direction, in my opinion. > > > > Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? > > It sounds reasonable. Does LLVM have its own objdump implementation > which provides this functionality? > > Thanks, > Florian > Yes. I changed llvm-objdump to use bfd names in LLVM>=11.0.0 (https://reviews.llvm.org/D76046) % llvm-objdump -f /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6: file format elf64-x86-64 ###### here architecture: x86_64 start address: 0x0000000000026e00
* Fāng-ruì Sòng: > On Tue, Sep 1, 2020 at 3:38 AM Florian Weimer <fweimer@redhat.com> wrote: >> >> * Fāng-ruì Sòng via Libc-alpha: >> >> > On Mon, Aug 31, 2020 at 11:07 AM Florian Weimer <fw@deneb.enyo.de> wrote: >> >> >> >> * Fāng-ruì Sòng: >> >> >> >> > LLD will not support --print-output-format. >> >> > >> >> > OUTPUT_FORMAT might be useful at some point and is still needed for >> >> > unified i386/x86-64 directory trees, but not when distributions use >> >> > different directory trees for multiarch. >> >> >> >> Sorry, I think it's not worthwhile to support LLD for building glibc >> >> if it produces output that offers different functionality. Bitwise >> >> identical output is of course not necessary, but disabling parts of >> >> the build output because it is inconvenient to build them using LLD is >> >> not the right direction, in my opinion. >> > >> > Grabbing elf64-x86-64 from objdump -f libc.so.6 . Is this acceptable? >> >> It sounds reasonable. Does LLVM have its own objdump implementation >> which provides this functionality? >> >> Thanks, >> Florian >> > > Yes. I changed llvm-objdump to use bfd names in LLVM>=11.0.0 > (https://reviews.llvm.org/D76046) > > % llvm-objdump -f /lib/x86_64-linux-gnu/libc.so.6 > > /lib/x86_64-linux-gnu/libc.so.6: file format elf64-x86-64 ###### here Okay, then a patch along these lines would make sense. Thanks, Florian
diff --git a/Makerules b/Makerules index 1e9c18f0d8..6a6e5579ae 100644 --- a/Makerules +++ b/Makerules @@ -1067,18 +1067,14 @@ install: $(inst_slibdir)/libc.so$(libc.so-version) # for the configuration we are building. We put this statement into # the linker scripts we install for -lc et al so that they will not be # used by a link for a different format on a multi-architecture system. -$(common-objpfx)format.lds: $(..)scripts/output-format.sed \ - $(common-objpfx)config.make \ +$(common-objpfx)format.lds: $(common-objpfx)config.make \ $(common-objpfx)config.h $(..)Makerules ifneq (unknown,$(output-format)) echo > $@.new 'OUTPUT_FORMAT($(output-format))' else - $(LINK.o) -shared $(sysdep-LDFLAGS) $(rtld-LDFLAGS) \ - $(LDFLAGS.so) $(LDFLAGS-lib.so) \ - -x c /dev/null -o $@.so -Wl,--verbose -v 2>&1 \ - | sed -n -f $< > $@.new - test -s $@.new - rm -f $@.so +# This branch is taken by lld, which auto detects the file format and does not +# need OUTPUT_FORMAT. + echo > $@.new endif mv -f $@.new $@ common-generated += format.lds diff --git a/benchtests/strcoll-inputs/filelist#en_US.UTF-8 b/benchtests/strcoll-inputs/filelist#en_US.UTF-8 index aa44107ad6..cb981de88c 100644 --- a/benchtests/strcoll-inputs/filelist#en_US.UTF-8 +++ b/benchtests/strcoll-inputs/filelist#en_US.UTF-8 @@ -9450,7 +9450,6 @@ move-if-change check-execstack.awk pylint pylintrc -output-format.sed merge-test-results.sh update-copyrights config-uname.sh diff --git a/scripts/output-format.sed b/scripts/output-format.sed deleted file mode 100644 index 364f52059f..0000000000 --- a/scripts/output-format.sed +++ /dev/null @@ -1,35 +0,0 @@ -/ld.*[ ]-E[BL]/b f -/collect.*[ ]-E[BL]/b f -/OUTPUT_FORMAT[^)]*$/{N -s/\n[ ]*/ / -} -t o -: o -s/^.*OUTPUT_FORMAT(\([^,]*\), \1, \1).*$/OUTPUT_FORMAT(\1)/ -t q -s/^.*OUTPUT_FORMAT(\([^,]*\), \([^,]*\), \([^,]*\)).*$/\1,\2,\3/ -t s -s/^.*OUTPUT_FORMAT(\([^,)]*\).*$)/OUTPUT_FORMAT(\1)/ -t q -d -: s -s/"//g -G -s/\n// -s/^\([^,]*\),\([^,]*\),\([^,]*\),B/OUTPUT_FORMAT(\2)/p -s/^\([^,]*\),\([^,]*\),\([^,]*\),L/OUTPUT_FORMAT(\3)/p -s/^\([^,]*\),\([^,]*\),\([^,]*\)/OUTPUT_FORMAT(\1)/p -/,/s|^|*** BUG in libc/scripts/output-format.sed *** |p -q -: q -s/"//g -p -q -: f -s/^.*[ ]-E\([BL]\)[ ].*$/,\1/ -t h -s/^.*[ ]-E\([BL]\)$/,\1/ -t h -d -: h -h