[v7,4/4] package/go: use host compiler when go-bootstrap unsupported

All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
compiled Go version to use to build from source.

https://golang.org/doc/install/source#bootstrapFromSource

The process for "bootstrapping" the Go compiler in Buildroot is:

1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap-stage1 (which is Go 1.4.x and written in C)
3. Build go-bootstrap-stage2 (which is Go 1.19.x and written in Go)
3. Build go 1.20 (written in Go) using go-bootstrap-stage2.

go-bootstrap-stage1 does not work on 64-bit arm. The Go 1.4.x bootstrap compiler
is compatible with x86, x86_64, and arm (32 bit) only.

This patch adds a fallback to require a host Go compiler to build host-go when
BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE2_ARCH_SUPPORTS is not set.

Signed-off-by: Christian Stewart <christian@aperture.us>

---

changes prior to inclusion in this series:

 - thanks Thomas for the review & suggestions
 - added NEEDS_HOST_GO boolean
 - added dependency checks to support/dependencies/dependencies.sh
 - removed unnecessary changes to go-bootstrap package
 - add dependency on toolchain if Cgo is enabled
 - updates for go1.20
 - updates for go-bootstrap-stage{1,2}

changes from v1 -> v2:

 - remove whitespace fix in bootstrap stage2

Signed-off-by: Christian Stewart <christian@aperture.us>
---
 Config.in                            |  4 ++++
 package/go/Config.in.host            |  2 +-
 package/go/go.mk                     | 14 ++++++++++++--
 support/dependencies/dependencies.sh |  4 ++++
 4 files changed, 21 insertions(+), 3 deletions(-)

Hello Christian,

On Tue, 11 Jul 2023 15:08:49 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
> compiled Go version to use to build from source.
> 
> https://golang.org/doc/install/source#bootstrapFromSource
> 
> The process for "bootstrapping" the Go compiler in Buildroot is:
> 
> 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> 2. Build go-bootstrap-stage1 (which is Go 1.4.x and written in C)
> 3. Build go-bootstrap-stage2 (which is Go 1.19.x and written in Go)
> 3. Build go 1.20 (written in Go) using go-bootstrap-stage2.
> 
> go-bootstrap-stage1 does not work on 64-bit arm. The Go 1.4.x bootstrap compiler
> is compatible with x86, x86_64, and arm (32 bit) only.
> 
> This patch adds a fallback to require a host Go compiler to build host-go when
> BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE2_ARCH_SUPPORTS is not set.
> 
> Signed-off-by: Christian Stewart <christian@aperture.us>

I understand the need, but I'd like to explore other options, namely
using pre-compiled Go compilers, which we would also like to support in
order to have the ability to run "make legal-info", or build Go
packages without necessarily having to build a Go compiler from
scratch. Like what we do for Rust, where you can chose to either use a
pre-built Rust compiler, or build your own with Buildroot.

For Go, I think we could imagine two options:

(a) You use a pre-built Go compiler as a replacement for package/go.
    This would work for architectures/configurations for which
    pre-built Go compilers are available. We would then move package/go
    as a virtual package, with package/go-src being the provider that
    builds the Go compiler from source, and package/go-bin being the
    provider that downloads/installs a pre-built Go compiler.

(b) You use a pre-built Go compiler as a replacement for
    package/go-bootstrap-stage2, i.e you still build the final Go
    compiler from source, but all the bootstrapping is skipped by using
    a pre-built Go compiler for the host. Would this option also
    resolve your use-case of building on ARM64? In other words, is
    there a Go compiler available pre-built for ARM64?

Best regards,

Thomas

Thomas,



On Sat, Jul 22, 2023, 2:13 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com>
wrote:

> Hello Christian,
>
> On Tue, 11 Jul 2023 15:08:49 -0700
> Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
>
> > All Go compiler versions > 1.4.x (old) are written in Go, and require a
> existing
> > compiled Go version to use to build from source.
> >
> > https://golang.org/doc/install/source#bootstrapFromSource
> >
> > The process for "bootstrapping" the Go compiler in Buildroot is:
> >
> > 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> > 2. Build go-bootstrap-stage1 (which is Go 1.4.x and written in C)
> > 3. Build go-bootstrap-stage2 (which is Go 1.19.x and written in Go)
> > 3. Build go 1.20 (written in Go) using go-bootstrap-stage2.
> >
> > go-bootstrap-stage1 does not work on 64-bit arm. The Go 1.4.x bootstrap
> compiler
> > is compatible with x86, x86_64, and arm (32 bit) only.
> >
> > This patch adds a fallback to require a host Go compiler to build
> host-go when
> > BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE2_ARCH_SUPPORTS is not set.
> >
> > Signed-off-by: Christian Stewart <christian@aperture.us>
>
> I understand the need, but I'd like to explore other options, namely
> using pre-compiled Go compilers, which we would also like to support in
> order to have the ability to run "make legal-info", or build Go
> packages without necessarily having to build a Go compiler from
> scratch. Like what we do for Rust, where you can chose to either use a
> pre-built Rust compiler, or build your own with Buildroot.
>

The main difference is the Go compiler takes only a minute or so of build
time. Bootstrapping it is fast and produces reproducible results across any
of the architectures Buildroot supports, including those for which there
currently are not any binary Go releases.

Therefore I submit that it is always worth the extra build time to
bootstrap the Go compiler from source as part of the build as opposed to
using a precompiled binary. Also it is less maintenance overhead (updating
hashes for all the platforms) and as the one maintaining this process I
appreciate that.

Today we download an external toolchain but still depend on a gcc package
to be installed on the host as well. Depending on the host Go to be
installed or otherwise provided for the bootstrap process is a decent
approach for architectures that we can't bootstrap with the C compiler
alone.

I can see a way we could add an option to do the binary downloads but would
prefer to keep the logic to depend on the host compiler as a fallback to
use for bootstrapping as well.

This is fine for legal-info as the bootstrap process builds the compiler
with itself several times to ensure that only the version built from source
is used.

What do you think?

Thanks,
Christian


> For Go, I think we could imagine two options:
>
> (a) You use a pre-built Go compiler as a replacement for package/go.
>     This would work for architectures/configurations for which
>     pre-built Go compilers are available. We would then move package/go
>     as a virtual package, with package/go-src being the provider that
>     builds the Go compiler from source, and package/go-bin being the
>     provider that downloads/installs a pre-built Go compiler.
>
> (b) You use a pre-built Go compiler as a replacement for
>     package/go-bootstrap-stage2, i.e you still build the final Go
>     compiler from source, but all the bootstrapping is skipped by using
>     a pre-built Go compiler for the host. Would this option also
>     resolve your use-case of building on ARM64? In other words, is
>     there a Go compiler available pre-built for ARM64?
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
>

Hello Christian,

On Sat, 22 Jul 2023 15:01:05 -0700
Christian Stewart <christian@aperture.us> wrote:

> The main difference is the Go compiler takes only a minute or so of build
> time. Bootstrapping it is fast and produces reproducible results across any
> of the architectures Buildroot supports, including those for which there
> currently are not any binary Go releases.
> 
> Therefore I submit that it is always worth the extra build time to
> bootstrap the Go compiler from source as part of the build as opposed to
> using a precompiled binary. Also it is less maintenance overhead (updating
> hashes for all the platforms) and as the one maintaining this process I
> appreciate that.
> 
> Today we download an external toolchain but still depend on a gcc package
> to be installed on the host as well. Depending on the host Go to be
> installed or otherwise provided for the bootstrap process is a decent
> approach for architectures that we can't bootstrap with the C compiler
> alone.
> 
> I can see a way we could add an option to do the binary downloads but would
> prefer to keep the logic to depend on the host compiler as a fallback to
> use for bootstrapping as well.
> 
> This is fine for legal-info as the bootstrap process builds the compiler
> with itself several times to ensure that only the version built from source
> is used.

Thanks for your feedback. What I meant by "legal-info" probably was
misunderstood. I very commonly run "make <foo>-legal-info" after
merging a version bump of a package. For 99% of our packages, this is a
trivial operation: it downloads the new tarball, extracts it, and
checks the hashes. For golang-package, it is a non-trivial operation as
it requires building go-bootstrap-stage1, go-bootstrap-stage2 and go,
before the package can be downloaded and vendored using "go".

Thomas

Thomas,



On Sat, Jul 22, 2023, 3:28 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com>
wrote:

> Hello Christian,
>
> On Sat, 22 Jul 2023 15:01:05 -0700
> Christian Stewart <christian@aperture.us> wrote:
>
> > The main difference is the Go compiler takes only a minute or so of build
> > time. Bootstrapping it is fast and produces reproducible results across
> any
> > of the architectures Buildroot supports, including those for which there
> > currently are not any binary Go releases.
> >
> > Therefore I submit that it is always worth the extra build time to
> > bootstrap the Go compiler from source as part of the build as opposed to
> > using a precompiled binary. Also it is less maintenance overhead
> (updating
> > hashes for all the platforms) and as the one maintaining this process I
> > appreciate that.
> >
> > Today we download an external toolchain but still depend on a gcc package
> > to be installed on the host as well. Depending on the host Go to be
> > installed or otherwise provided for the bootstrap process is a decent
> > approach for architectures that we can't bootstrap with the C compiler
> > alone.
> >
> > I can see a way we could add an option to do the binary downloads but
> would
> > prefer to keep the logic to depend on the host compiler as a fallback to
> > use for bootstrapping as well.
> >
> > This is fine for legal-info as the bootstrap process builds the compiler
> > with itself several times to ensure that only the version built from
> source
> > is used.
>
> Thanks for your feedback. What I meant by "legal-info" probably was
> misunderstood. I very commonly run "make <foo>-legal-info" after
> merging a version bump of a package. For 99% of our packages, this is a
> trivial operation: it downloads the new tarball, extracts it, and
> checks the hashes. For golang-package, it is a non-trivial operation as
> it requires building go-bootstrap-stage1, go-bootstrap-stage2 and go,
> before the package can be downloaded and vendored using "go".
>

Okay, I understand your use case.

What if we do the following:

 1. Merge this patch which enables using the host Go compiler on
architectures that aren't supported by host go bootstrap.
 2. Author and merge another series which adds the binary version of
host-go, still falling back on #1 when neither the bootstrap nor the binary
download (external toolchain) is supported.

I am willing to work on #2 as well.

Best,
Christian

On Sat, 22 Jul 2023 20:22:18 -0700
Christian Stewart <christian@aperture.us> wrote:

> What if we do the following:
> 
>  1. Merge this patch which enables using the host Go compiler on
> architectures that aren't supported by host go bootstrap.
>  2. Author and merge another series which adds the binary version of
> host-go, still falling back on #1 when neither the bootstrap nor the binary
> download (external toolchain) is supported.

I think like we do for Rust, we want to give the user the choice of
using a Go compiler built from source, or a pre-compiled Go compiler.

Other than that, your plan looks OK to me!

Thomas