defconfig fails if CROSS_COMPILE is set while cross-gcc is not avaialble

Message ID 0854f35e92c95e552721dc6a46f4ee2bc4020bc1.camel@synopsys.com
State New
Headers show
Series
  • defconfig fails if CROSS_COMPILE is set while cross-gcc is not avaialble
Related show

Commit Message

Alexey Brodkin Sept. 12, 2018, 12:08 p.m.
Hello Masahiro-san,

Starting from kernel v4.17 it is no longer possible to install kernel headers
for ARC architecture if there's no cross-toolchain in PATH.

Note installation of headers is just one of use-cases when we might not
have cross-tools available but still want to run "make xxx_defconfig".
I.e. the problem is "make xxx_defconfig" with no cross-toolchain.

That's what I see:
----------------------------->8--------------------------
./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
make: arc-linux-gcc: Command not found
/bin/sh: arc-linux-gcc: command not found
/bin/sh: arc-linux-gcc: command not found
*** Default configuration is based on 'nsim_700_defconfig'
./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 29: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 29: arc-linux-gcc: command not found
init/Kconfig:17: syntax error
init/Kconfig:16: invalid option
./scripts/clang-version.sh: line 15: arc-linux-gcc: command not found
./scripts/gcc-plugin.sh: line 11: arc-linux-gcc: command not found
make[1]: *** [scripts/kconfig/Makefile:91: defconfig] Error 1
make: *** [Makefile:531: defconfig] Error 2
----------------------------->8--------------------------

That doesn't happen for ARM and other arches simply because for ARC
we define CROSS_COMPILE if it is not set by user, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n9

Still if CROSS_COMPILE is set before execution of "make defconfig"
then the same problem happens for others.

I was able to find a series of commits which cause this problem,
here they are:
----------------------------->8--------------------------
59f53855babf - gcc-plugins: test plugin support in Kconfig and clean up Makefile
469cb7376c06 - kconfig: add CC_IS_CLANG and CLANG_VERSION
a4353898980c - kconfig: add CC_IS_GCC and GCC_VERSION
----------------------------->8--------------------------

What happen is "$(CC)" is passed as an argument and in its turn CC is
"$(CROSS_COMPILE)gcc", see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Makefile#n385


So if I substitute "CC" with "HOSTCC" in a couple of places (see below)
the problem goes away. But I'm not really sure if what I do is correct.
I.e. when we're interested in CC for target and when only host CC is of
our interest.

Would be interesting to know your opinion here.

-Alexey

-------------------------------------->8---------------------------------------------
-------------------------------------->8---------------------------------------------

Comments

Masahiro Yamada Sept. 12, 2018, 12:53 p.m. | #1
Hi Alexey.

2018-09-12 21:08 GMT+09:00 Alexey Brodkin <Alexey.Brodkin@synopsys.com>:
> Hello Masahiro-san,
>
> Starting from kernel v4.17 it is no longer possible to install kernel headers
> for ARC architecture if there's no cross-toolchain in PATH.


Really?

I can do 'make ARCH=arc headers_install'
with the latest Linus' tree.

I see the following warnings, but possible to install kernel headers.

./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found


If ARC compiler is not found in PATH,
those warnings are displayed with v4.16 or older as well.

I do not see much difference in the behavior.


I am trying to fix even those warnings
by eliminating unneeded compiler invocation.  It is WIP.



> Note installation of headers is just one of use-cases when we might not
> have cross-tools available but still want to run "make xxx_defconfig".
> I.e. the problem is "make xxx_defconfig" with no cross-toolchain.
>
> That's what I see:
> ----------------------------->8--------------------------
> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
> make: arc-linux-gcc: Command not found
> /bin/sh: arc-linux-gcc: command not found
> /bin/sh: arc-linux-gcc: command not found
> *** Default configuration is based on 'nsim_700_defconfig'
> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 29: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 29: arc-linux-gcc: command not found
> init/Kconfig:17: syntax error
> init/Kconfig:16: invalid option
> ./scripts/clang-version.sh: line 15: arc-linux-gcc: command not found
> ./scripts/gcc-plugin.sh: line 11: arc-linux-gcc: command not found
> make[1]: *** [scripts/kconfig/Makefile:91: defconfig] Error 1
> make: *** [Makefile:531: defconfig] Error 2
> ----------------------------->8--------------------------
>
> That doesn't happen for ARM and other arches simply because for ARC
> we define CROSS_COMPILE if it is not set by user, see:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n9

I do not think it is a good practice
to forcibly set CROSS_COMPILE that users may not have.

arch/m68k/Makefile uses cc-cross-prefix helper
to set the first found compiler.


> Still if CROSS_COMPILE is set before execution of "make defconfig"
> then the same problem happens for others.
>
> I was able to find a series of commits which cause this problem,
> here they are:
> ----------------------------->8--------------------------
> 59f53855babf - gcc-plugins: test plugin support in Kconfig and clean up Makefile
> 469cb7376c06 - kconfig: add CC_IS_CLANG and CLANG_VERSION
> a4353898980c - kconfig: add CC_IS_GCC and GCC_VERSION
> ----------------------------->8--------------------------
>
> What happen is "$(CC)" is passed as an argument and in its turn CC is
> "$(CROSS_COMPILE)gcc", see:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Makefile#n385
>
>
> So if I substitute "CC" with "HOSTCC" in a couple of places (see below)
> the problem goes away. But I'm not really sure if what I do is correct.
> I.e. when we're interested in CC for target and when only host CC is of
> our interest.

I think the log of commit 316d55d55f49eca44
is a good source to know the background of this change.

Now Kconfig requires the target compiler.
So, it does not make sense to
do defconfig with non-existing compiler.



> Would be interesting to know your opinion here.

I'd recommend to not hard-code CROSS_COMPILE.



Thanks.
Alexey Brodkin Sept. 12, 2018, 1:43 p.m. | #2
Hi Masahiro,

On Wed, 2018-09-12 at 21:53 +0900, Masahiro Yamada wrote:
> Hi Alexey.
> 
> 2018-09-12 21:08 GMT+09:00 Alexey Brodkin <Alexey.Brodkin@synopsys.com>:
> > Hello Masahiro-san,
> > 
> > Starting from kernel v4.17 it is no longer possible to install kernel headers
> > for ARC architecture if there's no cross-toolchain in PATH.
> 
> 
> Really?
> 
> I can do 'make ARCH=arc headers_install'
> with the latest Linus' tree.

Indeed "headers_install" works that way but is it OK to install headers
without previous configuration. In my experiment I don't see any differences
though for example OpenEmbedded on configuration step of "linux-libc-headers"
do "ARCH=xxx make allnoconfig", see 
https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc#L57

That might be some legacy but I'm not really sure,
worth asking OE people.

For the record in Buildroot no configuration happens for Linux headers,
see https://git.buildroot.org/buildroot/tree/package/linux-headers/linux-headers.mk#n106

> I see the following warnings, but possible to install kernel headers.
> 
> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found

Right... that happens because we're doing 2 hackish things:
1. Checking if current toolchain is configured for ARCompact (ARCv1) ISA
   or ARCv2. That's because we use toolchain's libgcc and so we want to warn
   a user if wrong toolchain is used early instead of leaving him to deal with
   final linkage failure.
   See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n23

2. Asking CC for a path to libgcc
   See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n82

[snip]

> I am trying to fix even those warnings
> by eliminating unneeded compiler invocation.  It is WIP.

And if (1) is not super important (in fact we used to live without it for years)
(2) requires implementation of libgcc in kernel sources (which BTW will make (1)
obsolete immediately). We do have it on our todo list though...

[snip]

> > That doesn't happen for ARM and other arches simply because for ARC
> > we define CROSS_COMPILE if it is not set by user, see:
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_torvalds_linux.git_tree_arch_arc_Makefile-23n9&d=DwIBaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=lqdeeSSEes0GFDDl656eViXO7breS55ytWkhpk5R81I&m=zJsa5MrAtDiYEWr5PZNS1aGk21ETa-qXzLPIddwvoQI&s=BU0ltPPoRLn3np_ba4HRHfH9i141uZjoS8jsJrg8SNc&e=
> 
> I do not think it is a good practice
> to forcibly set CROSS_COMPILE that users may not have.

Maybe so. Still some arches or random platforms do that.

> arch/m68k/Makefile uses cc-cross-prefix helper
> to set the first found compiler.

Well I'm not really sure we need to mess with CROSS_COMPILE in kernel.
I.e. there might be tons of variations depending on who toolchain was built
etc. So I'd better get rid of CROSS_COMPILE setup in our Makefile.

> > Still if CROSS_COMPILE is set before execution of "make defconfig"
> > then the same problem happens for others.
> > 
> > I was able to find a series of commits which cause this problem,
> > here they are:
> > ----------------------------->8--------------------------
> > 59f53855babf - gcc-plugins: test plugin support in Kconfig and clean up Makefile
> > 469cb7376c06 - kconfig: add CC_IS_CLANG and CLANG_VERSION
> > a4353898980c - kconfig: add CC_IS_GCC and GCC_VERSION
> > ----------------------------->8--------------------------
> > 
> > What happen is "$(CC)" is passed as an argument and in its turn CC is
> > "$(CROSS_COMPILE)gcc", see:
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_torvalds_linux.git_tree_Makefile-23n385&d=DwIBaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=lqdeeSSEes0GFDDl656eViXO7breS55ytWkhpk5R81I&m=zJsa5MrAtDiYEWr5PZNS1aGk21ETa-qXzLPIddwvoQI&s=S7zeHBpIp1P94XbyoMsaSnJWRqDGn69yGyeob-jzMJ4&e=
> > 
> > 
> > So if I substitute "CC" with "HOSTCC" in a couple of places (see below)
> > the problem goes away. But I'm not really sure if what I do is correct.
> > I.e. when we're interested in CC for target and when only host CC is of
> > our interest.
> 
> I think the log of commit 316d55d55f49eca44
> is a good source to know the background of this change.
> 
> Now Kconfig requires the target compiler.
> So, it does not make sense to
> do defconfig with non-existing compiler.

Agree, so I'll check with OpenEmbedded people if there's a reason to do "make allnoconfig"
if it's just a legacy code we'll fix it there.

Thanks a lot for all the input, much appreciated!

-Alexey
Masahiro Yamada Sept. 12, 2018, 2:59 p.m. | #3
Hi Alexey,

2018-09-12 22:43 GMT+09:00 Alexey Brodkin <Alexey.Brodkin@synopsys.com>:
> Hi Masahiro,
>
> On Wed, 2018-09-12 at 21:53 +0900, Masahiro Yamada wrote:
>> Hi Alexey.
>>
>> 2018-09-12 21:08 GMT+09:00 Alexey Brodkin <Alexey.Brodkin@synopsys.com>:
>> > Hello Masahiro-san,
>> >
>> > Starting from kernel v4.17 it is no longer possible to install kernel headers
>> > for ARC architecture if there's no cross-toolchain in PATH.
>>
>>
>> Really?
>>
>> I can do 'make ARCH=arc headers_install'
>> with the latest Linus' tree.
>
> Indeed "headers_install" works that way but is it OK to install headers
> without previous configuration. In my experiment I don't see any differences
> though for example OpenEmbedded on configuration step of "linux-libc-headers"
> do "ARCH=xxx make allnoconfig", see
> https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc#L57
>
> That might be some legacy but I'm not really sure,
> worth asking OE people.
>
> For the record in Buildroot no configuration happens for Linux headers,
> see https://git.buildroot.org/buildroot/tree/package/linux-headers/linux-headers.mk#n106


Buildroot is right.

The exported headers should be independent of the kernel configuration.

The interface between user-space and kernel must be stable.
It is strange if it is changed depending on how you configure the kernel.





>> I see the following warnings, but possible to install kernel headers.
>>
>> ./scripts/gcc-version.sh: line 26: arc-linux-gcc: command not found
>> ./scripts/gcc-version.sh: line 27: arc-linux-gcc: command not found
>
> Right... that happens because we're doing 2 hackish things:
> 1. Checking if current toolchain is configured for ARCompact (ARCv1) ISA
>    or ARCv2. That's because we use toolchain's libgcc and so we want to warn
>    a user if wrong toolchain is used early instead of leaving him to deal with
>    final linkage failure.
>    See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n23
>
> 2. Asking CC for a path to libgcc
>    See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arc/Makefile#n82
>
> [snip]
>
>> I am trying to fix even those warnings
>> by eliminating unneeded compiler invocation.  It is WIP.
>
> And if (1) is not super important (in fact we used to live without it for years)
> (2) requires implementation of libgcc in kernel sources (which BTW will make (1)
> obsolete immediately). We do have it on our todo list though...
> [snip]
>
>> > That doesn't happen for ARM and other arches simply because for ARC
>> > we define CROSS_COMPILE if it is not set by user, see:
>> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_torvalds_linux.git_tree_arch_arc_Makefile-23n9&d=DwIBaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=lqdeeSSEes0GFDDl656eViXO7breS55ytWkhpk5R81I&m=zJsa5MrAtDiYEWr5PZNS1aGk21ETa-qXzLPIddwvoQI&s=BU0ltPPoRLn3np_ba4HRHfH9i141uZjoS8jsJrg8SNc&e=
>>
>> I do not think it is a good practice
>> to forcibly set CROSS_COMPILE that users may not have.
>
> Maybe so. Still some arches or random platforms do that.
>
>> arch/m68k/Makefile uses cc-cross-prefix helper
>> to set the first found compiler.
>
> Well I'm not really sure we need to mess with CROSS_COMPILE in kernel.
> I.e. there might be tons of variations depending on who toolchain was built
> etc. So I'd better get rid of CROSS_COMPILE setup in our Makefile.


Agree.  Getting rid of CROSS_COMPILE is better.



>> > Still if CROSS_COMPILE is set before execution of "make defconfig"
>> > then the same problem happens for others.
>> >
>> > I was able to find a series of commits which cause this problem,
>> > here they are:
>> > ----------------------------->8--------------------------
>> > 59f53855babf - gcc-plugins: test plugin support in Kconfig and clean up Makefile
>> > 469cb7376c06 - kconfig: add CC_IS_CLANG and CLANG_VERSION
>> > a4353898980c - kconfig: add CC_IS_GCC and GCC_VERSION
>> > ----------------------------->8--------------------------
>> >
>> > What happen is "$(CC)" is passed as an argument and in its turn CC is
>> > "$(CROSS_COMPILE)gcc", see:
>> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_torvalds_linux.git_tree_Makefile-23n385&d=DwIBaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=lqdeeSSEes0GFDDl656eViXO7breS55ytWkhpk5R81I&m=zJsa5MrAtDiYEWr5PZNS1aGk21ETa-qXzLPIddwvoQI&s=S7zeHBpIp1P94XbyoMsaSnJWRqDGn69yGyeob-jzMJ4&e=
>> >
>> >
>> > So if I substitute "CC" with "HOSTCC" in a couple of places (see below)
>> > the problem goes away. But I'm not really sure if what I do is correct.
>> > I.e. when we're interested in CC for target and when only host CC is of
>> > our interest.
>>
>> I think the log of commit 316d55d55f49eca44
>> is a good source to know the background of this change.
>>
>> Now Kconfig requires the target compiler.
>> So, it does not make sense to
>> do defconfig with non-existing compiler.
>
> Agree, so I'll check with OpenEmbedded people if there's a reason to do "make allnoconfig"
> if it's just a legacy code we'll fix it there.
>
> Thanks a lot for all the input, much appreciated!
>
> -Alexey

Patch

diff --git a/arch/Kconfig b/arch/Kconfig
index d1f2ed462ac8..53c8c7994e6b 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -412,7 +412,7 @@  preferred-plugin-hostcc := $(if-success,[ $(gcc-version) -ge 40800 ],$(HOSTCXX),
 
 config PLUGIN_HOSTCC
        string
-       default "$(shell,$(srctree)/scripts/gcc-plugin.sh "$(preferred-plugin-hostcc)" "$(HOSTCXX)" "$(CC)")"
+       default "$(shell,$(srctree)/scripts/gcc-plugin.sh "$(preferred-plugin-hostcc)" "$(HOSTCXX)" "$(HOSTCC)")"
        help
          Host compiler used to build GCC plugins.  This can be $(HOSTCXX),
          $(HOSTCC), or a null string if GCC plugin is unsupported.
diff --git a/init/Kconfig b/init/Kconfig
index 2a48832fd40e..a0094ddb8244 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -13,7 +13,7 @@  config CC_IS_GCC
 
 config GCC_VERSION
        int
-       default $(shell,$(srctree)/scripts/gcc-version.sh -p $(CC) | sed 's/^0*//') if CC_IS_GCC
+       default $(shell,$(srctree)/scripts/gcc-version.sh -p $(HOSTCC) | sed 's/^0*//') if CC_IS_GCC
        default 0
 
 config CC_IS_CLANG
@@ -21,7 +21,7 @@  config CC_IS_CLANG
 
 config CLANG_VERSION
        int
-       default $(shell,$(srctree)/scripts/clang-version.sh $(CC))
+       default $(shell,$(srctree)/scripts/clang-version.sh $(HOSTCC))
 
 config CONSTRUCTORS
        bool
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 86321f06461e..684cd645eeb1 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -109,7 +109,7 @@  as-instr = $(call try-run,\
        printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
 
 # __cc-option
-# Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
+# Usage: MY_CFLAGS += $(call __cc-option,$(HOSTCC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
 __cc-option = $(call try-run,\
        $(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
 
@@ -143,11 +143,11 @@  cc-disable-warning = $(call try-run,\
 cc-name = $(shell $(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
 
 # cc-version
-cc-version = $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
+cc-version = $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(HOSTCC))
 
 # cc-fullversion
 cc-fullversion = $(shell $(CONFIG_SHELL) \
-       $(srctree)/scripts/gcc-version.sh -p $(CC))
+       $(srctree)/scripts/gcc-version.sh -p $(HOSTCC))
 
 # cc-ifversion
 # Usage:  EXTRA_CFLAGS += $(call cc-ifversion, -lt, 0402, -O1)
diff --git a/scripts/Kconfig.include b/scripts/Kconfig.include
index dad5583451af..3b28bf94cb24 100644
--- a/scripts/Kconfig.include
+++ b/scripts/Kconfig.include
@@ -27,4 +27,4 @@  cc-option = $(success,$(CC) -Werror $(1) -E -x c /dev/null -o /dev/null)
 ld-option = $(success,$(LD) -v $(1))
 
 # gcc version including patch level
-gcc-version := $(shell,$(srctree)/scripts/gcc-version.sh -p $(CC) | sed 's/^0*//')
+gcc-version := $(shell,$(srctree)/scripts/gcc-version.sh -p $(HOSTCC) | sed 's/^0*//')