| Message ID | 20220429064547.2334280-1-aik@ozlabs.ru (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [kernel] powerpc/llvm/lto: Allow LLVM LTO builds | expand |
On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote: > > This enables LTO_CLANG builds on POWER with the upstream version of > LLVM. > > LTO optimizes the output vmlinux binary and this may affect the FTP > alternative section if alt branches use "bc" (Branch Conditional) which > is limited by 16 bit offsets. This shows up in errors like: > > ld.lld: error: InputSection too large for range extension thunk vmlinux.o:(__ftr_alt_97+0xF0) > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with > "b" which allows 26 bit offsets. > > This catches the problem instructions in vmlinux.o before it LTO'ed: > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' > 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> > f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> > > This allows LTO builds for ppc64le_defconfig plus LTO options. > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds > but this is not POWERPC-specific. $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig $ ARCH=powerpc make LLVM=1 -j72 menuconfig <disable FTRACE, enable LTO_CLANG_THIN> $ ARCH=powerpc make LLVM=1 -j72 ... VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg /usr/bin/powerpc64le-linux-gnu-ld: /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error loading plugin: /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory clang-15: error: linker command failed with exit code 1 (use -v to see invocation) make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 Looks like LLD isn't being invoked correctly to link the vdso. Probably need to revisit https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ How were you working around this issue? Perhaps you built clang to default to LLD? (there's a cmake option for that) Perhaps for now I should just send: ``` diff --git a/arch/powerpc/kernel/vdso/Makefile b/arch/powerpc/kernel/vdso/Makefile index 954974287ee7..8762e6513683 100644 --- a/arch/powerpc/kernel/vdso/Makefile +++ b/arch/powerpc/kernel/vdso/Makefile @@ -55,6 +55,11 @@ AS32FLAGS := -D__VDSO32__ -s CC64FLAGS := -Wl,-soname=linux-vdso64.so.1 AS64FLAGS := -D__VDSO64__ -s +ifneq ($(LLVM),) +CC32FLAGS += -fuse-ld=lld +CC64FLAGS += -fuse-ld=lld +endif + targets += vdso32.lds CPPFLAGS_vdso32.lds += -P -C -Upowerpc targets += vdso64.lds ``` > > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> > --- > > Note 1: > This is further development of > https://lore.kernel.org/all/20220211023125.1790960-1-aik@ozlabs.ru/T/ > > Note 2: > CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS must be both "m" or "y" > or it won't link. For details: > https://lore.kernel.org/lkml/20220428043850.1706973-1-aik@ozlabs.ru/T/ Yeah, I just hit this: ``` LTO vmlinux.o LLVM ERROR: Function Import: link error: linking module flags 'Code Model': IDs have conflicting values in 'lib/built-in.a(entropy_common.o at 5782)' and 'lib/built-in.a(zstd_decompress_block.o at 6202)' PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace. LLVM ERROR: Failed to rename temporary file .thinlto-cache/Thin-96f93f.tmp.o to .thinlto-cache/llvmcache-A5B351EA452D46A86980E29C78B7260673348AAF: No such file or directory scripts/link-vmlinux.sh: line 76: 1240312 Aborted ${LD} ${KBUILD_LDFLAGS} -r -o ${1} ${lds} ${objects} make: *** [Makefile:1158: vmlinux] Error 134 ``` These two configs aren't easily modified in menuconfig. Perhaps you could find the concise set of configs that need to be disabled for this to be buildable? At least so others can test more easily, or even so we can update Kconfig checks. > --- > arch/powerpc/Kconfig | 2 ++ > arch/powerpc/kernel/exceptions-64s.S | 4 +++- > arch/powerpc/lib/copyuser_64.S | 3 ++- > arch/powerpc/lib/feature-fixups-test.S | 3 +-- > arch/powerpc/lib/memcpy_64.S | 3 ++- > 5 files changed, 10 insertions(+), 5 deletions(-) > > diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig > index 174edabb74fa..e2c7b5c1d0a6 100644 > --- a/arch/powerpc/Kconfig > +++ b/arch/powerpc/Kconfig > @@ -158,6 +158,8 @@ config PPC > select ARCH_WANT_IRQS_OFF_ACTIVATE_MM > select ARCH_WANT_LD_ORPHAN_WARN > select ARCH_WEAK_RELEASE_ACQUIRE > + select ARCH_SUPPORTS_LTO_CLANG > + select ARCH_SUPPORTS_LTO_CLANG_THIN > select BINFMT_ELF > select BUILDTIME_TABLE_SORT > select CLONE_BACKWARDS > diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S > index b66dd6f775a4..5b783bd51260 100644 > --- a/arch/powerpc/kernel/exceptions-64s.S > +++ b/arch/powerpc/kernel/exceptions-64s.S > @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) > .if IHSRR_IF_HVMODE > BEGIN_FTR_SECTION > bne masked_Hinterrupt > + b 4f > FTR_SECTION_ELSE > - bne masked_interrupt > ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) > + bne masked_interrupt > +4: > .elseif IHSRR > bne masked_Hinterrupt > .else > diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S > index db8719a14846..d07f95eebc65 100644 > --- a/arch/powerpc/lib/copyuser_64.S > +++ b/arch/powerpc/lib/copyuser_64.S > @@ -75,10 +75,11 @@ _GLOBAL(__copy_tofrom_user_base) > * set is Power6. > */ > test_feature = (SELFTEST_CASE == 1) > + beq .Ldst_aligned > BEGIN_FTR_SECTION > nop > FTR_SECTION_ELSE > - bne .Ldst_unaligned > + b .Ldst_unaligned > ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ > CPU_FTR_UNALIGNED_LD_STD) > .Ldst_aligned: > diff --git a/arch/powerpc/lib/feature-fixups-test.S b/arch/powerpc/lib/feature-fixups-test.S > index 480172fbd024..2751e42a9fd7 100644 > --- a/arch/powerpc/lib/feature-fixups-test.S > +++ b/arch/powerpc/lib/feature-fixups-test.S > @@ -145,7 +145,6 @@ BEGIN_FTR_SECTION > FTR_SECTION_ELSE > 2: or 2,2,2 > PPC_LCMPI r3,1 > - beq 3f > blt 2b > b 3f > b 1b > @@ -160,10 +159,10 @@ globl(ftr_fixup_test6_expected) > 1: or 1,1,1 > 2: or 2,2,2 > PPC_LCMPI r3,1 > - beq 3f > blt 2b > b 3f > b 1b > + nop > 3: or 1,1,1 > or 2,2,2 > or 3,3,3 > diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S > index 016c91e958d8..286c7e2d0883 100644 > --- a/arch/powerpc/lib/memcpy_64.S > +++ b/arch/powerpc/lib/memcpy_64.S > @@ -50,10 +50,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) > At the time of writing the only CPU that has this combination of bits > set is Power6. */ > test_feature = (SELFTEST_CASE == 1) > + beq .ldst_aligned > BEGIN_FTR_SECTION > nop > FTR_SECTION_ELSE > - bne .Ldst_unaligned > + b .Ldst_unaligned > ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ > CPU_FTR_UNALIGNED_LD_STD) > .Ldst_aligned: > -- > 2.30.2 >
On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote: > > diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S > index b66dd6f775a4..5b783bd51260 100644 > --- a/arch/powerpc/kernel/exceptions-64s.S > +++ b/arch/powerpc/kernel/exceptions-64s.S > @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) > .if IHSRR_IF_HVMODE > BEGIN_FTR_SECTION > bne masked_Hinterrupt > + b 4f > FTR_SECTION_ELSE Do you need to have the ELSE even if there's nothing in it; should it have a nop? The rest of the assembler changes LGTM, but withholding RB tag until we have Kconfig dependencies in better shape. > - bne masked_interrupt > ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) > + bne masked_interrupt > +4: > .elseif IHSRR > bne masked_Hinterrupt > .else
On 5/4/22 07:24, Nick Desaulniers wrote: > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote: >> >> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S >> index b66dd6f775a4..5b783bd51260 100644 >> --- a/arch/powerpc/kernel/exceptions-64s.S >> +++ b/arch/powerpc/kernel/exceptions-64s.S >> @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) >> .if IHSRR_IF_HVMODE >> BEGIN_FTR_SECTION >> bne masked_Hinterrupt >> + b 4f >> FTR_SECTION_ELSE > > Do you need to have the ELSE even if there's nothing in it; should it > have a nop? The rest of the assembler changes LGTM, but withholding > RB tag until we have Kconfig dependencies in better shape. The FTR patcher will add the necessary amount of "nop"s there and dropping "FTR_SECTION_ELSE" breaks the build as it does some "pushsection" magic. > >> - bne masked_interrupt >> ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) >> + bne masked_interrupt >> +4: >> .elseif IHSRR >> bne masked_Hinterrupt >> .else >
On 5/4/22 07:21, Nick Desaulniers wrote: > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote: >> >> This enables LTO_CLANG builds on POWER with the upstream version of >> LLVM. >> >> LTO optimizes the output vmlinux binary and this may affect the FTP >> alternative section if alt branches use "bc" (Branch Conditional) which >> is limited by 16 bit offsets. This shows up in errors like: >> >> ld.lld: error: InputSection too large for range extension thunk vmlinux.o:(__ftr_alt_97+0xF0) >> >> This works around the issue by replacing "bc" in FTR_SECTION_ELSE with >> "b" which allows 26 bit offsets. >> >> This catches the problem instructions in vmlinux.o before it LTO'ed: >> >> $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' >> 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> >> f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> >> >> This allows LTO builds for ppc64le_defconfig plus LTO options. >> Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds >> but this is not POWERPC-specific. > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig > $ ARCH=powerpc make LLVM=1 -j72 menuconfig > <disable FTRACE, enable LTO_CLANG_THIN> > $ ARCH=powerpc make LLVM=1 -j72 > ... > VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg > /usr/bin/powerpc64le-linux-gnu-ld: > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error > loading plugin: > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open > shared object file: No such file or directory > clang-15: error: linker command failed with exit code 1 (use -v to see > invocation) > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 > > Looks like LLD isn't being invoked correctly to link the vdso. > Probably need to revisit > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ > > How were you working around this issue? Perhaps you built clang to > default to LLD? (there's a cmake option for that) I was not. Just did clean build like this: mkdir ~/pbuild/llvm/llvm-lto-latest-cleanbuild cd ~/pbuild/llvm/llvm-lto-latest-cleanbuild CC='clang' CXX='clang++' cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD=PowerPC ~/p/llvm/llvm-latest/llvm/ -DLLVM_ENABLE_LTO=ON -DLLVM_ENABLE_LLD=ON -DLLVM_BINUTILS_INCDIR=/usr/include -DCMAKE_BUILD_TYPE=Release ninja -j 50 It builds fine: [fstn1-p1 ~/p/kernels-llvm/llvm]$ find /home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/ -iname LLVMgold.so -exec ls -l {} \; -rwxrwxr-x 1 aik aik 39032840 May 4 13:06 /home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/lib/LLVMgold.so and then in the kernel tree: PATH=/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/bin:$PATH make -j64 O=/home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/ ARCH=powerpc LLVM_IAS=1 CC=clang LLVM=1 ppc64le_defconfig then enabled LTO in that .config and then just built "vmlinux": [fstn1-p1 ~/p/kernels-llvm/llvm]$ ls -l /home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/vmlinux -rwxrwxr-x 1 aik aik 48145272 May 4 17:00 /home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/vmlinux which boots under qemu, the kernel version is: Preparing to boot Linux version 5.18.0-rc2_0bb153baeff0_a+fstn1 (aik@fstn1-p1) (clang version 15.0.0 (https://github.com/llvm/llvm-proje ct.git e29dc0c6fde284e7f05aa5f45b05c629c9fad295), LLD 15.0.0) #1 SMP Wed May 4 16:54:16 AEST 2022 Before I got to this point, I did many unspeakable things to that build system so may be it is screwed in some way but I cannot pinpoint it. The installed clang/lld is 12.0.0-3ubuntu1~21.04.2 and -DLLVM_ENABLE_LLD=ON from cmake is to accelerate rebuilding of LLVM (for bisecting). I'll try without it now, just takes ages to complete. > > Perhaps for now I should just send: > ``` > diff --git a/arch/powerpc/kernel/vdso/Makefile > b/arch/powerpc/kernel/vdso/Makefile > index 954974287ee7..8762e6513683 100644 > --- a/arch/powerpc/kernel/vdso/Makefile > +++ b/arch/powerpc/kernel/vdso/Makefile > @@ -55,6 +55,11 @@ AS32FLAGS := -D__VDSO32__ -s > CC64FLAGS := -Wl,-soname=linux-vdso64.so.1 > AS64FLAGS := -D__VDSO64__ -s > > +ifneq ($(LLVM),) > +CC32FLAGS += -fuse-ld=lld > +CC64FLAGS += -fuse-ld=lld > +endif > + > targets += vdso32.lds > CPPFLAGS_vdso32.lds += -P -C -Upowerpc > targets += vdso64.lds > ``` > > >> >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >> --- >> >> Note 1: >> This is further development of >> https://lore.kernel.org/all/20220211023125.1790960-1-aik@ozlabs.ru/T/ >> >> Note 2: >> CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS must be both "m" or "y" >> or it won't link. For details: >> https://lore.kernel.org/lkml/20220428043850.1706973-1-aik@ozlabs.ru/T/ > > Yeah, I just hit this: > ``` > LTO vmlinux.o > LLVM ERROR: Function Import: link error: linking module flags 'Code > Model': IDs have conflicting values in > 'lib/built-in.a(entropy_common.o at 5782)' and > 'lib/built-in.a(zstd_decompress_block.o at 6202)' > PLEASE submit a bug report to > https://github.com/llvm/llvm-project/issues/ and include the crash > backtrace. > LLVM ERROR: Failed to rename temporary file > .thinlto-cache/Thin-96f93f.tmp.o to > .thinlto-cache/llvmcache-A5B351EA452D46A86980E29C78B7260673348AAF: No > such file or directory > scripts/link-vmlinux.sh: line 76: 1240312 Aborted > ${LD} ${KBUILD_LDFLAGS} -r -o ${1} ${lds} ${objects} > make: *** [Makefile:1158: vmlinux] Error 134 > ``` > These two configs aren't easily modified in menuconfig. Perhaps you > could find the concise set of configs that need to be disabled for > this to be buildable? At least so others can test more easily, or even > so we can update Kconfig checks. I have posted v2: https://lore.kernel.org/lkml/20220429053329.2278740-1-aik@ozlabs.ru/T/ with this one, the ppc64 defconfigs should just work. >> --- >> arch/powerpc/Kconfig | 2 ++ >> arch/powerpc/kernel/exceptions-64s.S | 4 +++- >> arch/powerpc/lib/copyuser_64.S | 3 ++- >> arch/powerpc/lib/feature-fixups-test.S | 3 +-- >> arch/powerpc/lib/memcpy_64.S | 3 ++- >> 5 files changed, 10 insertions(+), 5 deletions(-) >> >> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig >> index 174edabb74fa..e2c7b5c1d0a6 100644 >> --- a/arch/powerpc/Kconfig >> +++ b/arch/powerpc/Kconfig >> @@ -158,6 +158,8 @@ config PPC >> select ARCH_WANT_IRQS_OFF_ACTIVATE_MM >> select ARCH_WANT_LD_ORPHAN_WARN >> select ARCH_WEAK_RELEASE_ACQUIRE >> + select ARCH_SUPPORTS_LTO_CLANG >> + select ARCH_SUPPORTS_LTO_CLANG_THIN >> select BINFMT_ELF >> select BUILDTIME_TABLE_SORT >> select CLONE_BACKWARDS >> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S >> index b66dd6f775a4..5b783bd51260 100644 >> --- a/arch/powerpc/kernel/exceptions-64s.S >> +++ b/arch/powerpc/kernel/exceptions-64s.S >> @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) >> .if IHSRR_IF_HVMODE >> BEGIN_FTR_SECTION >> bne masked_Hinterrupt >> + b 4f >> FTR_SECTION_ELSE >> - bne masked_interrupt >> ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) >> + bne masked_interrupt >> +4: >> .elseif IHSRR >> bne masked_Hinterrupt >> .else >> diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S >> index db8719a14846..d07f95eebc65 100644 >> --- a/arch/powerpc/lib/copyuser_64.S >> +++ b/arch/powerpc/lib/copyuser_64.S >> @@ -75,10 +75,11 @@ _GLOBAL(__copy_tofrom_user_base) >> * set is Power6. >> */ >> test_feature = (SELFTEST_CASE == 1) >> + beq .Ldst_aligned >> BEGIN_FTR_SECTION >> nop >> FTR_SECTION_ELSE >> - bne .Ldst_unaligned >> + b .Ldst_unaligned >> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >> CPU_FTR_UNALIGNED_LD_STD) >> .Ldst_aligned: >> diff --git a/arch/powerpc/lib/feature-fixups-test.S b/arch/powerpc/lib/feature-fixups-test.S >> index 480172fbd024..2751e42a9fd7 100644 >> --- a/arch/powerpc/lib/feature-fixups-test.S >> +++ b/arch/powerpc/lib/feature-fixups-test.S >> @@ -145,7 +145,6 @@ BEGIN_FTR_SECTION >> FTR_SECTION_ELSE >> 2: or 2,2,2 >> PPC_LCMPI r3,1 >> - beq 3f >> blt 2b >> b 3f >> b 1b >> @@ -160,10 +159,10 @@ globl(ftr_fixup_test6_expected) >> 1: or 1,1,1 >> 2: or 2,2,2 >> PPC_LCMPI r3,1 >> - beq 3f >> blt 2b >> b 3f >> b 1b >> + nop >> 3: or 1,1,1 >> or 2,2,2 >> or 3,3,3 >> diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S >> index 016c91e958d8..286c7e2d0883 100644 >> --- a/arch/powerpc/lib/memcpy_64.S >> +++ b/arch/powerpc/lib/memcpy_64.S >> @@ -50,10 +50,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) >> At the time of writing the only CPU that has this combination of bits >> set is Power6. */ >> test_feature = (SELFTEST_CASE == 1) >> + beq .ldst_aligned >> BEGIN_FTR_SECTION >> nop >> FTR_SECTION_ELSE >> - bne .Ldst_unaligned >> + b .Ldst_unaligned >> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >> CPU_FTR_UNALIGNED_LD_STD) >> .Ldst_aligned: >> -- >> 2.30.2 >> > >
On 04/05/2022 17:11, Alexey Kardashevskiy wrote: > > > On 5/4/22 07:21, Nick Desaulniers wrote: >> On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> >> wrote: >>> >>> This enables LTO_CLANG builds on POWER with the upstream version of >>> LLVM. >>> >>> LTO optimizes the output vmlinux binary and this may affect the FTP >>> alternative section if alt branches use "bc" (Branch Conditional) which >>> is limited by 16 bit offsets. This shows up in errors like: >>> >>> ld.lld: error: InputSection too large for range extension thunk >>> vmlinux.o:(__ftr_alt_97+0xF0) >>> >>> This works around the issue by replacing "bc" in FTR_SECTION_ELSE with >>> "b" which allows 26 bit offsets. >>> >>> This catches the problem instructions in vmlinux.o before it LTO'ed: >>> >>> $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' >>> 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> >>> f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> >>> >>> This allows LTO builds for ppc64le_defconfig plus LTO options. >>> Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds >>> but this is not POWERPC-specific. >> >> $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig >> $ ARCH=powerpc make LLVM=1 -j72 menuconfig >> <disable FTRACE, enable LTO_CLANG_THIN> >> $ ARCH=powerpc make LLVM=1 -j72 >> ... >> VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg >> /usr/bin/powerpc64le-linux-gnu-ld: >> /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error >> loading plugin: >> /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open >> shared object file: No such file or directory >> clang-15: error: linker command failed with exit code 1 (use -v to see >> invocation) >> make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: >> arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 >> >> Looks like LLD isn't being invoked correctly to link the vdso. >> Probably need to revisit >> https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ >> >> How were you working around this issue? Perhaps you built clang to >> default to LLD? (there's a cmake option for that) > > > I was not. > > Just did clean build like this: > > mkdir ~/pbuild/llvm/llvm-lto-latest-cleanbuild > > cd ~/pbuild/llvm/llvm-lto-latest-cleanbuild > > CC='clang' CXX='clang++' cmake -G Ninja > -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD=PowerPC > ~/p/llvm/llvm-latest/llvm/ -DLLVM_ENABLE_LTO=ON -DLLVM_ENABLE_LLD=ON > -DLLVM_BINUTILS_INCDIR=/usr/include -DCMAKE_BUILD_TYPE=Release > > ninja -j 50 > > It builds fine: > > [fstn1-p1 ~/p/kernels-llvm/llvm]$ find > /home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/ -iname LLVMgold.so > -exec ls -l {} \; > -rwxrwxr-x 1 aik aik 39032840 May 4 13:06 > /home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/lib/LLVMgold.so > > > > > and then in the kernel tree: > > > PATH=/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/bin:$PATH make > -j64 > O=/home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/ ARCH=powerpc LLVM_IAS=1 CC=clang LLVM=1 ppc64le_defconfig > > then enabled LTO in that .config and then just built "vmlinux": > > > [fstn1-p1 ~/p/kernels-llvm/llvm]$ ls -l > /home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/vmlinux > -rwxrwxr-x 1 aik aik 48145272 May 4 17:00 > /home/aik/pbuild/kernels-llvm/llvm-wip-llvm-latest-lto-full-cleanbuild/vmlinux > > which boots under qemu, the kernel version is: > > Preparing to boot Linux version 5.18.0-rc2_0bb153baeff0_a+fstn1 > (aik@fstn1-p1) (clang version 15.0.0 (https://github.com/llvm/llvm-proje > ct.git e29dc0c6fde284e7f05aa5f45b05c629c9fad295), LLD 15.0.0) #1 SMP Wed > May 4 16:54:16 AEST 2022 > > > > Before I got to this point, I did many unspeakable things to that build > system so may be it is screwed in some way but I cannot pinpoint it. > > The installed clang/lld is 12.0.0-3ubuntu1~21.04.2 and > -DLLVM_ENABLE_LLD=ON from cmake is to accelerate rebuilding of LLVM (for > bisecting). I'll try without it now, just takes ages to complete. And I just did. clang built with gcc simply crashes while building kernel's scripts/basic/fixdep :-D I may have to file a bug against clang now :-/ > >> >> Perhaps for now I should just send: >> ``` >> diff --git a/arch/powerpc/kernel/vdso/Makefile >> b/arch/powerpc/kernel/vdso/Makefile >> index 954974287ee7..8762e6513683 100644 >> --- a/arch/powerpc/kernel/vdso/Makefile >> +++ b/arch/powerpc/kernel/vdso/Makefile >> @@ -55,6 +55,11 @@ AS32FLAGS := -D__VDSO32__ -s >> CC64FLAGS := -Wl,-soname=linux-vdso64.so.1 >> AS64FLAGS := -D__VDSO64__ -s >> >> +ifneq ($(LLVM),) >> +CC32FLAGS += -fuse-ld=lld >> +CC64FLAGS += -fuse-ld=lld >> +endif >> + >> targets += vdso32.lds >> CPPFLAGS_vdso32.lds += -P -C -Upowerpc >> targets += vdso64.lds >> ``` >> >> >>> >>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >>> --- >>> >>> Note 1: >>> This is further development of >>> https://lore.kernel.org/all/20220211023125.1790960-1-aik@ozlabs.ru/T/ >>> >>> Note 2: >>> CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS must be both "m" or "y" >>> or it won't link. For details: >>> https://lore.kernel.org/lkml/20220428043850.1706973-1-aik@ozlabs.ru/T/ >> >> Yeah, I just hit this: >> ``` >> LTO vmlinux.o >> LLVM ERROR: Function Import: link error: linking module flags 'Code >> Model': IDs have conflicting values in >> 'lib/built-in.a(entropy_common.o at 5782)' and >> 'lib/built-in.a(zstd_decompress_block.o at 6202)' >> PLEASE submit a bug report to >> https://github.com/llvm/llvm-project/issues/ and include the crash >> backtrace. >> LLVM ERROR: Failed to rename temporary file >> .thinlto-cache/Thin-96f93f.tmp.o to >> .thinlto-cache/llvmcache-A5B351EA452D46A86980E29C78B7260673348AAF: No >> such file or directory >> scripts/link-vmlinux.sh: line 76: 1240312 Aborted >> ${LD} ${KBUILD_LDFLAGS} -r -o ${1} ${lds} ${objects} >> make: *** [Makefile:1158: vmlinux] Error 134 >> ``` >> These two configs aren't easily modified in menuconfig. Perhaps you >> could find the concise set of configs that need to be disabled for >> this to be buildable? At least so others can test more easily, or even >> so we can update Kconfig checks. > > > I have posted v2: > > https://lore.kernel.org/lkml/20220429053329.2278740-1-aik@ozlabs.ru/T/ > > with this one, the ppc64 defconfigs should just work. > > >>> --- >>> arch/powerpc/Kconfig | 2 ++ >>> arch/powerpc/kernel/exceptions-64s.S | 4 +++- >>> arch/powerpc/lib/copyuser_64.S | 3 ++- >>> arch/powerpc/lib/feature-fixups-test.S | 3 +-- >>> arch/powerpc/lib/memcpy_64.S | 3 ++- >>> 5 files changed, 10 insertions(+), 5 deletions(-) >>> >>> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig >>> index 174edabb74fa..e2c7b5c1d0a6 100644 >>> --- a/arch/powerpc/Kconfig >>> +++ b/arch/powerpc/Kconfig >>> @@ -158,6 +158,8 @@ config PPC >>> select ARCH_WANT_IRQS_OFF_ACTIVATE_MM >>> select ARCH_WANT_LD_ORPHAN_WARN >>> select ARCH_WEAK_RELEASE_ACQUIRE >>> + select ARCH_SUPPORTS_LTO_CLANG >>> + select ARCH_SUPPORTS_LTO_CLANG_THIN >>> select BINFMT_ELF >>> select BUILDTIME_TABLE_SORT >>> select CLONE_BACKWARDS >>> diff --git a/arch/powerpc/kernel/exceptions-64s.S >>> b/arch/powerpc/kernel/exceptions-64s.S >>> index b66dd6f775a4..5b783bd51260 100644 >>> --- a/arch/powerpc/kernel/exceptions-64s.S >>> +++ b/arch/powerpc/kernel/exceptions-64s.S >>> @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) >>> .if IHSRR_IF_HVMODE >>> BEGIN_FTR_SECTION >>> bne masked_Hinterrupt >>> + b 4f >>> FTR_SECTION_ELSE >>> - bne masked_interrupt >>> ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | >>> CPU_FTR_ARCH_206) >>> + bne masked_interrupt >>> +4: >>> .elseif IHSRR >>> bne masked_Hinterrupt >>> .else >>> diff --git a/arch/powerpc/lib/copyuser_64.S >>> b/arch/powerpc/lib/copyuser_64.S >>> index db8719a14846..d07f95eebc65 100644 >>> --- a/arch/powerpc/lib/copyuser_64.S >>> +++ b/arch/powerpc/lib/copyuser_64.S >>> @@ -75,10 +75,11 @@ _GLOBAL(__copy_tofrom_user_base) >>> * set is Power6. >>> */ >>> test_feature = (SELFTEST_CASE == 1) >>> + beq .Ldst_aligned >>> BEGIN_FTR_SECTION >>> nop >>> FTR_SECTION_ELSE >>> - bne .Ldst_unaligned >>> + b .Ldst_unaligned >>> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >>> CPU_FTR_UNALIGNED_LD_STD) >>> .Ldst_aligned: >>> diff --git a/arch/powerpc/lib/feature-fixups-test.S >>> b/arch/powerpc/lib/feature-fixups-test.S >>> index 480172fbd024..2751e42a9fd7 100644 >>> --- a/arch/powerpc/lib/feature-fixups-test.S >>> +++ b/arch/powerpc/lib/feature-fixups-test.S >>> @@ -145,7 +145,6 @@ BEGIN_FTR_SECTION >>> FTR_SECTION_ELSE >>> 2: or 2,2,2 >>> PPC_LCMPI r3,1 >>> - beq 3f >>> blt 2b >>> b 3f >>> b 1b >>> @@ -160,10 +159,10 @@ globl(ftr_fixup_test6_expected) >>> 1: or 1,1,1 >>> 2: or 2,2,2 >>> PPC_LCMPI r3,1 >>> - beq 3f >>> blt 2b >>> b 3f >>> b 1b >>> + nop >>> 3: or 1,1,1 >>> or 2,2,2 >>> or 3,3,3 >>> diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S >>> index 016c91e958d8..286c7e2d0883 100644 >>> --- a/arch/powerpc/lib/memcpy_64.S >>> +++ b/arch/powerpc/lib/memcpy_64.S >>> @@ -50,10 +50,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) >>> At the time of writing the only CPU that has this combination of >>> bits >>> set is Power6. */ >>> test_feature = (SELFTEST_CASE == 1) >>> + beq .ldst_aligned >>> BEGIN_FTR_SECTION >>> nop >>> FTR_SECTION_ELSE >>> - bne .Ldst_unaligned >>> + b .Ldst_unaligned >>> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >>> CPU_FTR_UNALIGNED_LD_STD) >>> .Ldst_aligned: >>> -- >>> 2.30.2 >>> >> >>
On 5/4/22 07:21, Nick Desaulniers wrote: > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote: >> >> This enables LTO_CLANG builds on POWER with the upstream version of >> LLVM. >> >> LTO optimizes the output vmlinux binary and this may affect the FTP >> alternative section if alt branches use "bc" (Branch Conditional) which >> is limited by 16 bit offsets. This shows up in errors like: >> >> ld.lld: error: InputSection too large for range extension thunk vmlinux.o:(__ftr_alt_97+0xF0) >> >> This works around the issue by replacing "bc" in FTR_SECTION_ELSE with >> "b" which allows 26 bit offsets. >> >> This catches the problem instructions in vmlinux.o before it LTO'ed: >> >> $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' >> 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> >> f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> >> >> This allows LTO builds for ppc64le_defconfig plus LTO options. >> Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds >> but this is not POWERPC-specific. > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig > $ ARCH=powerpc make LLVM=1 -j72 menuconfig > <disable FTRACE, enable LTO_CLANG_THIN> > $ ARCH=powerpc make LLVM=1 -j72 > ... > VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg > /usr/bin/powerpc64le-linux-gnu-ld: > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error > loading plugin: > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open > shared object file: No such file or directory > clang-15: error: linker command failed with exit code 1 (use -v to see > invocation) > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 > > Looks like LLD isn't being invoked correctly to link the vdso. > Probably need to revisit > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ > > How were you working around this issue? Perhaps you built clang to > default to LLD? (there's a cmake option for that) What option is that? I only add -DLLVM_ENABLE_LLD=ON which (I think) tells cmake to use lld to link the LLVM being built but does not seem to tell what the built clang should do. Without -DLLVM_ENABLE_LLD=ON, building just fails: [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100 [619/3501] Linking CXX executable bin/not FAILED: bin/not : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -flto -O3 -DNDEBUG -flto -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libLLVMSupport.a -lrt -ldl -lpthread -lm /usr/lib/powerpc64le-linux-gnu/libz.so /usr/lib/powerpc64le-linux-gnu/libtinfo.so lib/libLLVMDemangle.a && : /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no index; run ranlib to add one clang: error: linker command failed with exit code 1 (use -v to see invocation) [701/3501] Building CXX object utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o ninja: build stopped: subcommand failed. My head hurts :( The above example is running on PPC. Now I am trying x86 box: [2693/3505] Linking CXX shared library lib/libLTO.so.15git FAILED: lib/libLTO.so.15git : && /usr/bin/clang++ -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -flto -O3 -DNDEBUG -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=ld -flto -Wl,-rpath-link,/home/aik/llvm-build/./lib -Wl,--gc-sections -Wl,--version-script,"/home/aik/llvm-build/tools/lto/LTO.exports" -shared -Wl,-soname,libLTO.so.15git -o lib/libLTO.so.15git tools/lto/CMakeFiles/LTO.dir/LTODisassembler.cpp.o tools/lto/CMakeFiles/LTO.dir/lto.cpp.o -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a lib/libLLVMBitReader.a lib/libLLVMCore.a lib/libLLVMCodeGen.a lib/libLLVMLTO.a lib/libLLVMMC.a lib/libLLVMMCDisassembler.a lib/libLLVMSupport.a lib/libLLVMTarget.a lib/libLLVMAsmPrinter.a lib/libLLVMGlobalISel.a lib/libLLVMSelectionDAG.a lib/libLLVMCodeGen.a lib/libLLVMExtensions.a lib/libLLVMPasses.a lib/libLLVMTarget.a lib/libLLVMObjCARCOpts.a lib/libLLVMCoroutines.a lib/libLLVMipo.a lib/libLLVMBitWriter.a lib/libLLVMInstrumentation.a lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMVectorize.a lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMProfileData.a lib/libLLVMSymbolize.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMDebugInfoPDB.a lib/libLLVMDebugInfoMSF.a lib/libLLVMObject.a lib/libLLVMBitReader.a lib/libLLVMCore.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMTextAPI.a lib/libLLVMBinaryFormat.a lib/libLLVMSupport.a -lrt -ldl -lm /usr/lib64/libz.so /usr/lib64/libtinfo.so lib/libLLVMDemangle.a && : /usr/bin/ld: lib/libLLVMObject.a: error adding symbols: file format not recognized clang-13: error: linker command failed with exit code 1 (use -v to see invocation) Why is this? cmake is invoked as: CC='clang' CXX='clang++' cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD=PowerPC ~/llvm-project//llvm -DLLVM_ENABLE_LTO=ON -DLLVM_USE_LINKER=ld -DLLVM_BINUTILS_INCDIR=/usr/lib/gcc/powerpc64le-linux-gnu/11/plugin/include/ -DCMAKE_BUILD_TYPE=Release > Perhaps for now I should just send: > ``` > diff --git a/arch/powerpc/kernel/vdso/Makefile > b/arch/powerpc/kernel/vdso/Makefile > index 954974287ee7..8762e6513683 100644 > --- a/arch/powerpc/kernel/vdso/Makefile > +++ b/arch/powerpc/kernel/vdso/Makefile > @@ -55,6 +55,11 @@ AS32FLAGS := -D__VDSO32__ -s > CC64FLAGS := -Wl,-soname=linux-vdso64.so.1 > AS64FLAGS := -D__VDSO64__ -s > > +ifneq ($(LLVM),) > +CC32FLAGS += -fuse-ld=lld > +CC64FLAGS += -fuse-ld=lld > +endif > + Can LLVM_IAS=1 still work with bfd's LD? Thanks, > targets += vdso32.lds > CPPFLAGS_vdso32.lds += -P -C -Upowerpc > targets += vdso64.lds > ``` > > >> >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >> --- >> >> Note 1: >> This is further development of >> https://lore.kernel.org/all/20220211023125.1790960-1-aik@ozlabs.ru/T/ >> >> Note 2: >> CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS must be both "m" or "y" >> or it won't link. For details: >> https://lore.kernel.org/lkml/20220428043850.1706973-1-aik@ozlabs.ru/T/ > > Yeah, I just hit this: > ``` > LTO vmlinux.o > LLVM ERROR: Function Import: link error: linking module flags 'Code > Model': IDs have conflicting values in > 'lib/built-in.a(entropy_common.o at 5782)' and > 'lib/built-in.a(zstd_decompress_block.o at 6202)' > PLEASE submit a bug report to > https://github.com/llvm/llvm-project/issues/ and include the crash > backtrace. > LLVM ERROR: Failed to rename temporary file > .thinlto-cache/Thin-96f93f.tmp.o to > .thinlto-cache/llvmcache-A5B351EA452D46A86980E29C78B7260673348AAF: No > such file or directory > scripts/link-vmlinux.sh: line 76: 1240312 Aborted > ${LD} ${KBUILD_LDFLAGS} -r -o ${1} ${lds} ${objects} > make: *** [Makefile:1158: vmlinux] Error 134 > ``` > These two configs aren't easily modified in menuconfig. Perhaps you > could find the concise set of configs that need to be disabled for > this to be buildable? At least so others can test more easily, or even > so we can update Kconfig checks. > >> --- >> arch/powerpc/Kconfig | 2 ++ >> arch/powerpc/kernel/exceptions-64s.S | 4 +++- >> arch/powerpc/lib/copyuser_64.S | 3 ++- >> arch/powerpc/lib/feature-fixups-test.S | 3 +-- >> arch/powerpc/lib/memcpy_64.S | 3 ++- >> 5 files changed, 10 insertions(+), 5 deletions(-) >> >> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig >> index 174edabb74fa..e2c7b5c1d0a6 100644 >> --- a/arch/powerpc/Kconfig >> +++ b/arch/powerpc/Kconfig >> @@ -158,6 +158,8 @@ config PPC >> select ARCH_WANT_IRQS_OFF_ACTIVATE_MM >> select ARCH_WANT_LD_ORPHAN_WARN >> select ARCH_WEAK_RELEASE_ACQUIRE >> + select ARCH_SUPPORTS_LTO_CLANG >> + select ARCH_SUPPORTS_LTO_CLANG_THIN >> select BINFMT_ELF >> select BUILDTIME_TABLE_SORT >> select CLONE_BACKWARDS >> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S >> index b66dd6f775a4..5b783bd51260 100644 >> --- a/arch/powerpc/kernel/exceptions-64s.S >> +++ b/arch/powerpc/kernel/exceptions-64s.S >> @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) >> .if IHSRR_IF_HVMODE >> BEGIN_FTR_SECTION >> bne masked_Hinterrupt >> + b 4f >> FTR_SECTION_ELSE >> - bne masked_interrupt >> ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) >> + bne masked_interrupt >> +4: >> .elseif IHSRR >> bne masked_Hinterrupt >> .else >> diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S >> index db8719a14846..d07f95eebc65 100644 >> --- a/arch/powerpc/lib/copyuser_64.S >> +++ b/arch/powerpc/lib/copyuser_64.S >> @@ -75,10 +75,11 @@ _GLOBAL(__copy_tofrom_user_base) >> * set is Power6. >> */ >> test_feature = (SELFTEST_CASE == 1) >> + beq .Ldst_aligned >> BEGIN_FTR_SECTION >> nop >> FTR_SECTION_ELSE >> - bne .Ldst_unaligned >> + b .Ldst_unaligned >> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >> CPU_FTR_UNALIGNED_LD_STD) >> .Ldst_aligned: >> diff --git a/arch/powerpc/lib/feature-fixups-test.S b/arch/powerpc/lib/feature-fixups-test.S >> index 480172fbd024..2751e42a9fd7 100644 >> --- a/arch/powerpc/lib/feature-fixups-test.S >> +++ b/arch/powerpc/lib/feature-fixups-test.S >> @@ -145,7 +145,6 @@ BEGIN_FTR_SECTION >> FTR_SECTION_ELSE >> 2: or 2,2,2 >> PPC_LCMPI r3,1 >> - beq 3f >> blt 2b >> b 3f >> b 1b >> @@ -160,10 +159,10 @@ globl(ftr_fixup_test6_expected) >> 1: or 1,1,1 >> 2: or 2,2,2 >> PPC_LCMPI r3,1 >> - beq 3f >> blt 2b >> b 3f >> b 1b >> + nop >> 3: or 1,1,1 >> or 2,2,2 >> or 3,3,3 >> diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S >> index 016c91e958d8..286c7e2d0883 100644 >> --- a/arch/powerpc/lib/memcpy_64.S >> +++ b/arch/powerpc/lib/memcpy_64.S >> @@ -50,10 +50,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) >> At the time of writing the only CPU that has this combination of bits >> set is Power6. */ >> test_feature = (SELFTEST_CASE == 1) >> + beq .ldst_aligned >> BEGIN_FTR_SECTION >> nop >> FTR_SECTION_ELSE >> - bne .Ldst_unaligned >> + b .Ldst_unaligned >> ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ >> CPU_FTR_UNALIGNED_LD_STD) >> .Ldst_aligned: >> -- >> 2.30.2 >> > >
On 5/9/22 15:18, Alexey Kardashevskiy wrote: > > > On 5/4/22 07:21, Nick Desaulniers wrote: >> On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy <aik@ozlabs.ru> >> wrote: >>> >>> This enables LTO_CLANG builds on POWER with the upstream version of >>> LLVM. >>> >>> LTO optimizes the output vmlinux binary and this may affect the FTP >>> alternative section if alt branches use "bc" (Branch Conditional) which >>> is limited by 16 bit offsets. This shows up in errors like: >>> >>> ld.lld: error: InputSection too large for range extension thunk >>> vmlinux.o:(__ftr_alt_97+0xF0) >>> >>> This works around the issue by replacing "bc" in FTR_SECTION_ELSE with >>> "b" which allows 26 bit offsets. >>> >>> This catches the problem instructions in vmlinux.o before it LTO'ed: >>> >>> $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' >>> 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> >>> f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> >>> >>> This allows LTO builds for ppc64le_defconfig plus LTO options. >>> Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds >>> but this is not POWERPC-specific. >> >> $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig >> $ ARCH=powerpc make LLVM=1 -j72 menuconfig >> <disable FTRACE, enable LTO_CLANG_THIN> >> $ ARCH=powerpc make LLVM=1 -j72 >> ... >> VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg >> /usr/bin/powerpc64le-linux-gnu-ld: >> /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error >> loading plugin: >> /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open >> shared object file: No such file or directory >> clang-15: error: linker command failed with exit code 1 (use -v to see >> invocation) >> make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: >> arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 >> >> Looks like LLD isn't being invoked correctly to link the vdso. >> Probably need to revisit >> https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ >> >> How were you working around this issue? Perhaps you built clang to >> default to LLD? (there's a cmake option for that) > > > What option is that? I only add -DLLVM_ENABLE_LLD=ON which (I think) > tells cmake to use lld to link the LLVM being built but does not seem to > tell what the built clang should do. > > Without -DLLVM_ENABLE_LLD=ON, building just fails: > > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100 > [619/3501] Linking CXX executable bin/not > FAILED: bin/not > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra > -Wno-unused-parameter -Wwrite-strings -Wcast-qual > -Wmissing-field-initializers -pedantic -Wno-long-long > -Wc++98-compat-extra-semi -Wimplicit-fallthrough > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections > -fdata-sections -flto -O3 -DNDEBUG -flto > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not > -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libLLVMSupport.a -lrt > -ldl -lpthread -lm /usr/lib/powerpc64le-linux-gnu/libz.so > /usr/lib/powerpc64le-linux-gnu/libtinfo.so lib/libLLVMDemangle.a && : > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no > index; run ranlib to add one > clang: error: linker command failed with exit code 1 (use -v to see > invocation) > [701/3501] Building CXX object > utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o > ninja: build stopped: subcommand failed. > > > > My head hurts :( > The above example is running on PPC. Now I am trying x86 box: > A bit of progress. cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGET_ARCH=PowerPC -DLLVM_TARGETS_TO_BUILD=PowerPC ~/llvm-project//llvm -DLLVM_ENABLE_LTO=ON -DLLVM_BINUTILS_INCDIR=/usr/lib/gcc/powerpc64le-linux-gnu/11/plugin/include/ -DCMAKE_BUILD_TYPE=Release produces: -- Native target architecture is PowerPC .... -- LLVM host triple: x86_64-unknown-linux-gnu -- LLVM default target triple: x86_64-unknown-linux-gnu and the resulting "clang" can only to "Target: x86_64-unknown-linux-gnu", how do you build LLVM exactly? Thanks,
Hi Alexey, On Mon, May 09, 2022 at 05:42:59PM +1000, Alexey Kardashevskiy wrote: > > > On 5/9/22 15:18, Alexey Kardashevskiy wrote: > > > > > > On 5/4/22 07:21, Nick Desaulniers wrote: > > > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy > > > <aik@ozlabs.ru> wrote: > > > > > > > > This enables LTO_CLANG builds on POWER with the upstream version of > > > > LLVM. > > > > > > > > LTO optimizes the output vmlinux binary and this may affect the FTP > > > > alternative section if alt branches use "bc" (Branch Conditional) which > > > > is limited by 16 bit offsets. This shows up in errors like: > > > > > > > > ld.lld: error: InputSection too large for range extension thunk > > > > vmlinux.o:(__ftr_alt_97+0xF0) > > > > > > > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with > > > > "b" which allows 26 bit offsets. > > > > > > > > This catches the problem instructions in vmlinux.o before it LTO'ed: > > > > > > > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' > > > > 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> > > > > f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> > > > > > > > > This allows LTO builds for ppc64le_defconfig plus LTO options. > > > > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds > > > > but this is not POWERPC-specific. > > > > > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig > > > $ ARCH=powerpc make LLVM=1 -j72 menuconfig > > > <disable FTRACE, enable LTO_CLANG_THIN> > > > $ ARCH=powerpc make LLVM=1 -j72 > > > ... > > > VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg > > > /usr/bin/powerpc64le-linux-gnu-ld: > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error > > > loading plugin: > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open > > > shared object file: No such file or directory > > > clang-15: error: linker command failed with exit code 1 (use -v to see > > > invocation) > > > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: > > > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 > > > > > > Looks like LLD isn't being invoked correctly to link the vdso. > > > Probably need to revisit > > > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ > > > > > > How were you working around this issue? Perhaps you built clang to > > > default to LLD? (there's a cmake option for that) > > > > > > What option is that? I only add -DLLVM_ENABLE_LLD=ON which (I think) The option Nick is referring to here is CLANG_DEFAULT_LINKER, which sets the default linker when clang is invoked as the linker driver, which the PowerPC vDSO Makefile does. We have been trying to move all parts of the kernel to compile with $(CC) and link with $(LD) so that the default linker invoked by the compiler is taken out of the equation. For what it's worth, I think that the error Nick is seeing is due to a lack of the LLVMgold.so plugin on his system, which is built when -DLLVM_BINUTILS_INCDIR=... is included in the list of CMake variables, which you have. The LLVMgold.so plugin is needed when doing LTO with GNU ld, which is the case with the vDSO currently for the reason I mentioned above. We could work around this with Nick's proposed '-fuse-ld=lld' patch or just disable LTO for the vDSO. https://llvm.org/docs/GoldPlugin.html My version of the hack would probably be: ccflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld asflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld > > tells cmake to use lld to link the LLVM being built but does not seem to > > tell what the built clang should do. Right, -DLLVM_ENABLE_LLD=ON is equivalent to -DLLVM_USE_LINKER=lld, except when doing a multi-stage build: https://llvm.org/docs/CMake.html#llvm-related-variables > > > > Without -DLLVM_ENABLE_LLD=ON, building just fails: > > > > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100 > > [619/3501] Linking CXX executable bin/not > > FAILED: bin/not > > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden > > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra > > -Wno-unused-parameter -Wwrite-strings -Wcast-qual > > -Wmissing-field-initializers -pedantic -Wno-long-long > > -Wc++98-compat-extra-semi -Wimplicit-fallthrough > > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor > > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion > > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections > > -fdata-sections -flto -O3 -DNDEBUG -flto > > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib > > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not > > -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libLLVMSupport.a -lrt > > -ldl -lpthread -lm /usr/lib/powerpc64le-linux-gnu/libz.so > > /usr/lib/powerpc64le-linux-gnu/libtinfo.so lib/libLLVMDemangle.a && : > > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no > > index; run ranlib to add one > > clang: error: linker command failed with exit code 1 (use -v to see > > invocation) > > [701/3501] Building CXX object > > utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o > > ninja: build stopped: subcommand failed. > > > > > > > > My head hurts :( > > The above example is running on PPC. Now I am trying x86 box: > > > > A bit of progress. > > cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGET_ARCH=PowerPC > -DLLVM_TARGETS_TO_BUILD=PowerPC ~/llvm-project//llvm -DLLVM_ENABLE_LTO=ON > -DLLVM_BINUTILS_INCDIR=/usr/lib/gcc/powerpc64le-linux-gnu/11/plugin/include/ > -DCMAKE_BUILD_TYPE=Release > > produces: > > -- Native target architecture is PowerPC > > .... > -- LLVM host triple: x86_64-unknown-linux-gnu > -- LLVM default target triple: x86_64-unknown-linux-gnu > > > and the resulting "clang" can only to "Target: x86_64-unknown-linux-gnu", > how do you build LLVM exactly? Thanks, I have never tried to cross compile LLVM but you can use LLVM_DEFAULT_TARGET_TRIPLE to change the default target triple. From what I can tell, the way you are building LLVM on your PowerPC host is fine. Cheers, Nathan
On Mon, May 9, 2022 at 11:08 AM Nathan Chancellor <nathan@kernel.org> wrote: > > Hi Alexey, > > On Mon, May 09, 2022 at 05:42:59PM +1000, Alexey Kardashevskiy wrote: > > > > > > On 5/9/22 15:18, Alexey Kardashevskiy wrote: > > > > > > > > > On 5/4/22 07:21, Nick Desaulniers wrote: > > > > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy > > > > <aik@ozlabs.ru> wrote: > > > > > > > > > > This enables LTO_CLANG builds on POWER with the upstream version of > > > > > LLVM. > > > > > > > > > > LTO optimizes the output vmlinux binary and this may affect the FTP > > > > > alternative section if alt branches use "bc" (Branch Conditional) which > > > > > is limited by 16 bit offsets. This shows up in errors like: > > > > > > > > > > ld.lld: error: InputSection too large for range extension thunk > > > > > vmlinux.o:(__ftr_alt_97+0xF0) > > > > > > > > > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with > > > > > "b" which allows 26 bit offsets. > > > > > > > > > > This catches the problem instructions in vmlinux.o before it LTO'ed: > > > > > > > > > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' > > > > > 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> > > > > > f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> > > > > > > > > > > This allows LTO builds for ppc64le_defconfig plus LTO options. > > > > > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds > > > > > but this is not POWERPC-specific. > > > > > > > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig > > > > $ ARCH=powerpc make LLVM=1 -j72 menuconfig > > > > <disable FTRACE, enable LTO_CLANG_THIN> > > > > $ ARCH=powerpc make LLVM=1 -j72 > > > > ... > > > > VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg > > > > /usr/bin/powerpc64le-linux-gnu-ld: > > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error > > > > loading plugin: > > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open > > > > shared object file: No such file or directory > > > > clang-15: error: linker command failed with exit code 1 (use -v to see > > > > invocation) > > > > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: > > > > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 > > > > > > > > Looks like LLD isn't being invoked correctly to link the vdso. > > > > Probably need to revisit > > > > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ > > > > > > > > How were you working around this issue? Perhaps you built clang to > > > > default to LLD? (there's a cmake option for that) > > > > > > > > > What option is that? I only add -DLLVM_ENABLE_LLD=ON which (I think) > > The option Nick is referring to here is CLANG_DEFAULT_LINKER, which sets > the default linker when clang is invoked as the linker driver, which the > PowerPC vDSO Makefile does. We have been trying to move all parts of the > kernel to compile with $(CC) and link with $(LD) so that the default > linker invoked by the compiler is taken out of the equation. > > For what it's worth, I think that the error Nick is seeing is due to a > lack of the LLVMgold.so plugin on his system, which is built when > -DLLVM_BINUTILS_INCDIR=... is included in the list of CMake variables, > which you have. The LLVMgold.so plugin is needed when doing LTO with > GNU ld, which is the case with the vDSO currently for the reason I > mentioned above. We could work around this with Nick's proposed > '-fuse-ld=lld' patch or just disable LTO for the vDSO. > > https://llvm.org/docs/GoldPlugin.html Ah, and ld.gold is currently banned by commit 75959d44f9dc ("kbuild: Fail if gold linker is detected") which landed in v5.4-rc1. So the ppc vdso build isn't quite as hermetic as we'd like. > > My version of the hack would probably be: > > ccflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld > asflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld LGTM; want to send that as a formal patch? That would also address https://github.com/ClangBuiltLinux/linux/issues/774. > > > > tells cmake to use lld to link the LLVM being built but does not seem to > > > tell what the built clang should do. > > Right, -DLLVM_ENABLE_LLD=ON is equivalent to -DLLVM_USE_LINKER=lld, > except when doing a multi-stage build: > > https://llvm.org/docs/CMake.html#llvm-related-variables > > > > > > > Without -DLLVM_ENABLE_LLD=ON, building just fails: > > > > > > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100 > > > [619/3501] Linking CXX executable bin/not > > > FAILED: bin/not > > > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden > > > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra > > > -Wno-unused-parameter -Wwrite-strings -Wcast-qual > > > -Wmissing-field-initializers -pedantic -Wno-long-long > > > -Wc++98-compat-extra-semi -Wimplicit-fallthrough > > > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor > > > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion > > > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections > > > -fdata-sections -flto -O3 -DNDEBUG -flto > > > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib > > > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not > > > -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libLLVMSupport.a -lrt > > > -ldl -lpthread -lm /usr/lib/powerpc64le-linux-gnu/libz.so > > > /usr/lib/powerpc64le-linux-gnu/libtinfo.so lib/libLLVMDemangle.a && : > > > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no > > > index; run ranlib to add one Sounds like a bug with llvm cmake on ppc hosts. The other error about > /usr/bin/ld: lib/libLLVMObject.a: error adding symbols: file format not recognized Is likely because GNU ld doesn't recognize LLVM IR, which is what's passed to the linker when you build LLVM itself with LTO via `-DLLVM_ENABLE_LTO=ON`.
On 2022-05-09, Nick Desaulniers wrote: >On Mon, May 9, 2022 at 11:08 AM Nathan Chancellor <nathan@kernel.org> wrote: >> >> Hi Alexey, >> >> On Mon, May 09, 2022 at 05:42:59PM +1000, Alexey Kardashevskiy wrote: >> > >> > >> > On 5/9/22 15:18, Alexey Kardashevskiy wrote: >> > > >> > > >> > > On 5/4/22 07:21, Nick Desaulniers wrote: >> > > > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy >> > > > <aik@ozlabs.ru> wrote: >> > > > > >> > > > > This enables LTO_CLANG builds on POWER with the upstream version of >> > > > > LLVM. >> > > > > >> > > > > LTO optimizes the output vmlinux binary and this may affect the FTP >> > > > > alternative section if alt branches use "bc" (Branch Conditional) which >> > > > > is limited by 16 bit offsets. This shows up in errors like: >> > > > > >> > > > > ld.lld: error: InputSection too large for range extension thunk >> > > > > vmlinux.o:(__ftr_alt_97+0xF0) >> > > > > >> > > > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with >> > > > > "b" which allows 26 bit offsets. >> > > > > >> > > > > This catches the problem instructions in vmlinux.o before it LTO'ed: >> > > > > >> > > > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' >> > > > > 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> >> > > > > f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> >> > > > > >> > > > > This allows LTO builds for ppc64le_defconfig plus LTO options. >> > > > > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds >> > > > > but this is not POWERPC-specific. >> > > > >> > > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig >> > > > $ ARCH=powerpc make LLVM=1 -j72 menuconfig >> > > > <disable FTRACE, enable LTO_CLANG_THIN> >> > > > $ ARCH=powerpc make LLVM=1 -j72 >> > > > ... >> > > > VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg >> > > > /usr/bin/powerpc64le-linux-gnu-ld: >> > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error >> > > > loading plugin: >> > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open >> > > > shared object file: No such file or directory >> > > > clang-15: error: linker command failed with exit code 1 (use -v to see >> > > > invocation) >> > > > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67: >> > > > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1 >> > > > >> > > > Looks like LLD isn't being invoked correctly to link the vdso. >> > > > Probably need to revisit >> > > > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/ >> > > > >> > > > How were you working around this issue? Perhaps you built clang to >> > > > default to LLD? (there's a cmake option for that) >> > > >> > > >> > > What option is that? I only add -DLLVM_ENABLE_LLD=ON which (I think) >> >> The option Nick is referring to here is CLANG_DEFAULT_LINKER, which sets >> the default linker when clang is invoked as the linker driver, which the >> PowerPC vDSO Makefile does. We have been trying to move all parts of the >> kernel to compile with $(CC) and link with $(LD) so that the default >> linker invoked by the compiler is taken out of the equation. >> >> For what it's worth, I think that the error Nick is seeing is due to a >> lack of the LLVMgold.so plugin on his system, which is built when >> -DLLVM_BINUTILS_INCDIR=... is included in the list of CMake variables, >> which you have. The LLVMgold.so plugin is needed when doing LTO with >> GNU ld, which is the case with the vDSO currently for the reason I >> mentioned above. We could work around this with Nick's proposed >> '-fuse-ld=lld' patch or just disable LTO for the vDSO. >> >> https://llvm.org/docs/GoldPlugin.html > >Ah, and ld.gold is currently banned by >commit 75959d44f9dc ("kbuild: Fail if gold linker is detected") >which landed in v5.4-rc1. So the ppc vdso build isn't quite as >hermetic as we'd like. > >> >> My version of the hack would probably be: >> >> ccflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld >> asflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld > >LGTM; want to send that as a formal patch? That would also address >https://github.com/ClangBuiltLinux/linux/issues/774. > >> >> > > tells cmake to use lld to link the LLVM being built but does not seem to >> > > tell what the built clang should do. >> >> Right, -DLLVM_ENABLE_LLD=ON is equivalent to -DLLVM_USE_LINKER=lld, >> except when doing a multi-stage build: >> >> https://llvm.org/docs/CMake.html#llvm-related-variables >> >> > > >> > > Without -DLLVM_ENABLE_LLD=ON, building just fails: >> > > >> > > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100 >> > > [619/3501] Linking CXX executable bin/not >> > > FAILED: bin/not >> > > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden >> > > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra >> > > -Wno-unused-parameter -Wwrite-strings -Wcast-qual >> > > -Wmissing-field-initializers -pedantic -Wno-long-long >> > > -Wc++98-compat-extra-semi -Wimplicit-fallthrough >> > > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor >> > > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion >> > > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections >> > > -fdata-sections -flto -O3 -DNDEBUG -flto >> > > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib >> > > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not >> > > -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libLLVMSupport.a -lrt >> > > -ldl -lpthread -lm /usr/lib/powerpc64le-linux-gnu/libz.so >> > > /usr/lib/powerpc64le-linux-gnu/libtinfo.so lib/libLLVMDemangle.a && : >> > > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no >> > > index; run ranlib to add one > >Sounds like a bug with llvm cmake on ppc hosts. > >The other error about > >> /usr/bin/ld: lib/libLLVMObject.a: error adding symbols: file format not >recognized > >Is likely because GNU ld doesn't recognize LLVM IR, which is what's >passed to the linker when you build LLVM itself with LTO via >`-DLLVM_ENABLE_LTO=ON`. GNU ld needs -plugin path/to/LLVMgold.so to recognize LLVM bitcode files. As discussed, LLVMgold.so is not specified... `error adding symbols: archive has no index` indicates another problem. An archive needs to be built with llvm-ar or `ar -plugin path/to/LLVMgold.so` to include LLVM IR symbols in the archive symbol table. ld.lld < 14.0.0 does not support such archives. See https://maskray.me/blog/2022-01-16-archives-and-start-lib#thin-archives-without-a-symbol-table While named LLVMgold.so, the plugin supports all of gold, ld.bfd, ar, and ranlib.
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 174edabb74fa..e2c7b5c1d0a6 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -158,6 +158,8 @@ config PPC select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WEAK_RELEASE_ACQUIRE + select ARCH_SUPPORTS_LTO_CLANG + select ARCH_SUPPORTS_LTO_CLANG_THIN select BINFMT_ELF select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index b66dd6f775a4..5b783bd51260 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -476,9 +476,11 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real, text) .if IHSRR_IF_HVMODE BEGIN_FTR_SECTION bne masked_Hinterrupt + b 4f FTR_SECTION_ELSE - bne masked_interrupt ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) + bne masked_interrupt +4: .elseif IHSRR bne masked_Hinterrupt .else diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S index db8719a14846..d07f95eebc65 100644 --- a/arch/powerpc/lib/copyuser_64.S +++ b/arch/powerpc/lib/copyuser_64.S @@ -75,10 +75,11 @@ _GLOBAL(__copy_tofrom_user_base) * set is Power6. */ test_feature = (SELFTEST_CASE == 1) + beq .Ldst_aligned BEGIN_FTR_SECTION nop FTR_SECTION_ELSE - bne .Ldst_unaligned + b .Ldst_unaligned ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ CPU_FTR_UNALIGNED_LD_STD) .Ldst_aligned: diff --git a/arch/powerpc/lib/feature-fixups-test.S b/arch/powerpc/lib/feature-fixups-test.S index 480172fbd024..2751e42a9fd7 100644 --- a/arch/powerpc/lib/feature-fixups-test.S +++ b/arch/powerpc/lib/feature-fixups-test.S @@ -145,7 +145,6 @@ BEGIN_FTR_SECTION FTR_SECTION_ELSE 2: or 2,2,2 PPC_LCMPI r3,1 - beq 3f blt 2b b 3f b 1b @@ -160,10 +159,10 @@ globl(ftr_fixup_test6_expected) 1: or 1,1,1 2: or 2,2,2 PPC_LCMPI r3,1 - beq 3f blt 2b b 3f b 1b + nop 3: or 1,1,1 or 2,2,2 or 3,3,3 diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S index 016c91e958d8..286c7e2d0883 100644 --- a/arch/powerpc/lib/memcpy_64.S +++ b/arch/powerpc/lib/memcpy_64.S @@ -50,10 +50,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) At the time of writing the only CPU that has this combination of bits set is Power6. */ test_feature = (SELFTEST_CASE == 1) + beq .ldst_aligned BEGIN_FTR_SECTION nop FTR_SECTION_ELSE - bne .Ldst_unaligned + b .Ldst_unaligned ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \ CPU_FTR_UNALIGNED_LD_STD) .Ldst_aligned:
This enables LTO_CLANG builds on POWER with the upstream version of LLVM. LTO optimizes the output vmlinux binary and this may affect the FTP alternative section if alt branches use "bc" (Branch Conditional) which is limited by 16 bit offsets. This shows up in errors like: ld.lld: error: InputSection too large for range extension thunk vmlinux.o:(__ftr_alt_97+0xF0) This works around the issue by replacing "bc" in FTR_SECTION_ELSE with "b" which allows 26 bit offsets. This catches the problem instructions in vmlinux.o before it LTO'ed: $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>' 30: 00 00 82 40 bc 4,eq,30 <__ftr_alt_97+0x30> f0: 00 00 82 40 bc 4,eq,f0 <__ftr_alt_97+0xf0> This allows LTO builds for ppc64le_defconfig plus LTO options. Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds but this is not POWERPC-specific. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> --- Note 1: This is further development of https://lore.kernel.org/all/20220211023125.1790960-1-aik@ozlabs.ru/T/ Note 2: CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS must be both "m" or "y" or it won't link. For details: https://lore.kernel.org/lkml/20220428043850.1706973-1-aik@ozlabs.ru/T/ --- arch/powerpc/Kconfig | 2 ++ arch/powerpc/kernel/exceptions-64s.S | 4 +++- arch/powerpc/lib/copyuser_64.S | 3 ++- arch/powerpc/lib/feature-fixups-test.S | 3 +-- arch/powerpc/lib/memcpy_64.S | 3 ++- 5 files changed, 10 insertions(+), 5 deletions(-)