| Message ID | 20210622024337.3009417-1-christian@paral.in |
|---|---|
| State | Changes Requested, archived |
| Headers | show |
| Series | [v1,1/1] package/go: use host compiler when go-bootstrap unsupported | expand |
Hi Christian, On Tue, Jun 22 2021, Christian Stewart wrote: > All Go compiler versions > 1.4.x (old) are written in Go, and require a existing > compiled Go version to use to build from source. > > https://golang.org/doc/install/source#bootstrapFromSource > > The process for "bootstrapping" the Go compiler in Buildroot is: > > 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain. > 2. Build go-bootstrap (which is Go 1.4.x and written in C) > 3. Build go 1.16.x (written in Go) using go-bootstrap. > > The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. > The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit). > > This means that arm64 host machines will skip building Go and all go-based > packages like Docker. > > This patch instead uses the host Go compiler to bootstrap host-go when > BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how > the host GCC is used to bootstrap the Buildroot toolchain. > > Signed-off-by: Christian Stewart <christian@paral.in> > --- > package/go-bootstrap/go-bootstrap.mk | 6 ++++++ > package/go/Config.in.host | 2 -- > package/go/go.mk | 4 ++++ > 3 files changed, 10 insertions(+), 2 deletions(-) > > diff --git a/package/go-bootstrap/go-bootstrap.mk b/package/go-bootstrap/go-bootstrap.mk > index 6710e31561..8324a67795 100644 > --- a/package/go-bootstrap/go-bootstrap.mk > +++ b/package/go-bootstrap/go-bootstrap.mk > @@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE > # host-go-bootstrap. > HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain > > +# If we do not support this architecture with go-bootstrap, depend on the host > +# Go compiler to bootstrap the host-go compiler instead. > +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) > HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION) > +else > +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go I think we need a check for host go existence and maybe also version under support/dependencies/. baruch > +endif > > # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE > # here. See https://github.com/golang/go/issues/11685. > diff --git a/package/go/Config.in.host b/package/go/Config.in.host > index e82ab6e81a..c07605dd02 100644 > --- a/package/go/Config.in.host > +++ b/package/go/Config.in.host > @@ -2,7 +2,6 @@ > config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS > bool > default y > - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS > depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \ > || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \ > || BR2_mips64 || BR2_mips64el || BR2_s390x > @@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS > config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS > bool > default y > - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS > diff --git a/package/go/go.mk b/package/go/go.mk > index 4252691343..5f501d2d8c 100644 > --- a/package/go/go.mk > +++ b/package/go/go.mk > @@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause > GO_LICENSE_FILES = LICENSE > GO_CPE_ID_VENDOR = golang > > +# Depend on the host Go compiler if go-bootstrap is not available. > +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) > HOST_GO_DEPENDENCIES = host-go-bootstrap > +endif > + > HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path > HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache > HOST_GO_ROOT = $(HOST_DIR)/lib/go
Hi Baruch, On Mon, Jun 21, 2021 at 8:10 PM Baruch Siach <baruch@tkos.co.il> wrote: > > +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go > > I think we need a check for host go existence and maybe also version under > support/dependencies/. > > baruch I had a look at those scripts and am not familiar enough to know the correct way to only check for host-go existence on the system only if all of: - BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS=y - BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set - BR2_PACKAGE_HOST_GO=y If any of these conditions are not met, it's not necessary to have Go installed on the host system. Best, Christian
Hello Christian, On Mon, 21 Jun 2021 19:43:37 -0700 Christian Stewart <christian@paral.in> wrote: > All Go compiler versions > 1.4.x (old) are written in Go, and require a existing > compiled Go version to use to build from source. > > https://golang.org/doc/install/source#bootstrapFromSource > > The process for "bootstrapping" the Go compiler in Buildroot is: > > 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain. > 2. Build go-bootstrap (which is Go 1.4.x and written in C) > 3. Build go 1.16.x (written in Go) using go-bootstrap. > > The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. > The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit). So perhaps my brain is not smart enough here, but how is the Go compiler then build on 64-bit ARM ? > Signed-off-by: Christian Stewart <christian@paral.in> > --- > package/go-bootstrap/go-bootstrap.mk | 6 ++++++ > package/go/Config.in.host | 2 -- > package/go/go.mk | 4 ++++ > 3 files changed, 10 insertions(+), 2 deletions(-) Implementation-wise, I agree with the feedback from Baruch, we need some checks in support/dependencies/. Normally, the way we do things is that packages express their dependency on a particular system tool by selecting a hidden boolean. For example packages that need java on the host will select BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if BR2_NEEDS_HOST_JAVA=y, we check if java is installed. The other model that we have is for optional host tools, where we can use the system-provided one or fallback to building our own if not. That's for example what support/dependencies/check-host-lzip.{mk,sh} is doing. check-host-lzip.mk calls suitable-host-package, which itself calls the check-host-lzip.sh script to verify if lzip is installed. If there is no suitable lzip available on the host, then check-host-lzip.mk defines BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be empty if the system has a lzip program, or it will be set to host-lzip if Buildroot needs to build its own. Another feature for the lzip situation is that we can force Buildroot to build the host packages even if the tool is found on the machine, using BR2_FORCE_HOST_BUILD=y. The thing is that your host-go-bootstrap situation does not exactly fit these situations. I think the easiest would be to do like we do for Java, but only select BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support building host-go-bootstrap. If we go this route, then it should look like this: * Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean * support/dependencies/dependencies.sh should test if a functional Go compiler is available on the machine if BR2_NEEDS_HOST_GO is defined, and if not bail out with a hard error * BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have something like: select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS * go.mk should be modified to not use host-go-bootstrap if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think host-go-bootstrap should be modified like you did however, the changes should be contained into package/go/go.mk. The one thing that bothers me however is the hardcoding of /usr/lib/go in your proposal as the location of the Go compiler. I guess we could have some logic in the top-level Makefile that looks for the path of the host Go compiler perhaps ? Does this make sense ? Best regards, Thomas
Hi Thomas, all, On Wed, Jul 21, 2021 at 1:49 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Christian Stewart <christian@paral.in> wrote: > > The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. > > The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit). > > So perhaps my brain is not smart enough here, but how is the Go > compiler then build on 64-bit ARM ? It has to be bootstrapped using an existing arm64 go compiler binary, or cross compiled. Presumably a bootstrap process which produces first an armv7 Go compiler, and then uses it to build a arm64 Go compiler, is also possible. With gccgo, it should be possible to bootstrap using the Gcc version of the Go compiler (which is included with the Buildroot toolchain today, but not enabled.) > The one thing that bothers me however is the hardcoding of /usr/lib/go > in your proposal as the location of the Go compiler. I guess we could > have some logic in the top-level Makefile that looks for the path of > the host Go compiler perhaps ? Yes, it could be done like this: PATH_TO_GOROOT=$(go env GOROOT) > Does this make sense ? Yes, this should work fine. Eventually the gccgo based bootstrap approach could also be added as another alternative. Best regards, Christian Stewart
Hi Thomas, I have implemented your suggestions below & sent as a V2. On Wed, Jul 21, 2021 at 1:49 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Normally, the way we do things is that packages express their > dependency on a particular system tool by selecting a hidden boolean. > For example packages that need java on the host will select > BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if > BR2_NEEDS_HOST_JAVA=y, we check if java is installed. > > The other model that we have is for optional host tools, where we can > use the system-provided one or fallback to building our own if not. > That's for example what support/dependencies/check-host-lzip.{mk,sh} is > doing. > > check-host-lzip.mk calls suitable-host-package, which itself calls the > check-host-lzip.sh script to verify if lzip is installed. If there is > no suitable lzip available on the host, then check-host-lzip.mk defines > BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the wo > lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be > empty if the system has a lzip program, or it will be set to host-lzip > if Buildroot needs to build its own. > > Another feature for the lzip situation is that we can force Buildroot > to build the host packages even if the tool is found on the machine, > using BR2_FORCE_HOST_BUILD=y. > > The thing is that your host-go-bootstrap situation does not exactly fit > these situations. > > I think the easiest would be to do like we do for Java, but only select > BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support > building host-go-bootstrap. > > If we go this route, then it should look like this: > > * Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean > > * support/dependencies/dependencies.sh should test if a functional Go > compiler is available on the machine if BR2_NEEDS_HOST_GO is > defined, and if not bail out with a hard error > > * BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and > BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends > on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have > something like: > > select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS > > * go.mk should be modified to not use host-go-bootstrap if > !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think > host-go-bootstrap should be modified like you did however, the > changes should be contained into package/go/go.mk. Thanks & best, Christian Stewart
diff --git a/package/go-bootstrap/go-bootstrap.mk b/package/go-bootstrap/go-bootstrap.mk index 6710e31561..8324a67795 100644 --- a/package/go-bootstrap/go-bootstrap.mk +++ b/package/go-bootstrap/go-bootstrap.mk @@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE # host-go-bootstrap. HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain +# If we do not support this architecture with go-bootstrap, depend on the host +# Go compiler to bootstrap the host-go compiler instead. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION) +else +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go +endif # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE # here. See https://github.com/golang/go/issues/11685. diff --git a/package/go/Config.in.host b/package/go/Config.in.host index e82ab6e81a..c07605dd02 100644 --- a/package/go/Config.in.host +++ b/package/go/Config.in.host @@ -2,7 +2,6 @@ config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \ || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \ || BR2_mips64 || BR2_mips64el || BR2_s390x @@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS diff --git a/package/go/go.mk b/package/go/go.mk index 4252691343..5f501d2d8c 100644 --- a/package/go/go.mk +++ b/package/go/go.mk @@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause GO_LICENSE_FILES = LICENSE GO_CPE_ID_VENDOR = golang +# Depend on the host Go compiler if go-bootstrap is not available. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_DEPENDENCIES = host-go-bootstrap +endif + HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache HOST_GO_ROOT = $(HOST_DIR)/lib/go
All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source. https://golang.org/doc/install/source#bootstrapFromSource The process for "bootstrapping" the Go compiler in Buildroot is: 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain. 2. Build go-bootstrap (which is Go 1.4.x and written in C) 3. Build go 1.16.x (written in Go) using go-bootstrap. The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit). This means that arm64 host machines will skip building Go and all go-based packages like Docker. This patch instead uses the host Go compiler to bootstrap host-go when BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how the host GCC is used to bootstrap the Buildroot toolchain. Signed-off-by: Christian Stewart <christian@paral.in> --- package/go-bootstrap/go-bootstrap.mk | 6 ++++++ package/go/Config.in.host | 2 -- package/go/go.mk | 4 ++++ 3 files changed, 10 insertions(+), 2 deletions(-)