diff mbox series

[1/1] package/go-native: add a new download only package for go compilers

Message ID 20230707185106.98208-1-ckhardin@gmail.com
State New
Headers show
Series [1/1] package/go-native: add a new download only package for go compilers | expand

Commit Message

Charles Hardin July 7, 2023, 6:51 p.m. UTC
The go-native is mainly for the aarch64 support since that was
added after the go-bootstrap written in C was dropped. So, to
get a cross compiler for go a host/native compiler needs to be
available to generate the setup for the needed target. This is
using the same go version as the expected cross-compiler to
keep the packages in sync. A specific version could be used.

Signed-off-by: Charles Hardin <ckhardin@gmail.com>
---
 package/Config.in.host           |  1 +
 package/go-native/Config.in.host | 12 ++++++++
 package/go-native/go-native.hash |  5 ++++
 package/go-native/go-native.mk   | 48 ++++++++++++++++++++++++++++++++
 package/go/Config.in.host        |  6 ++--
 package/go/go.mk                 | 12 +++++++-
 6 files changed, 81 insertions(+), 3 deletions(-)
 create mode 100644 package/go-native/Config.in.host
 create mode 100644 package/go-native/go-native.hash
 create mode 100644 package/go-native/go-native.mk

Comments

Christian Stewart July 7, 2023, 7:04 p.m. UTC | #1
Hi Charles,

This is not necessary with this patch series:

https://patchwork.ozlabs.org/project/buildroot/list/?series=358451

We can correctly bootstrap on arm64.

On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com> wrote:
>
> The go-native is mainly for the aarch64 support since that was
> added after the go-bootstrap written in C was dropped. So, to
> get a cross compiler for go a host/native compiler needs to be
> available to generate the setup for the needed target. This is
> using the same go version as the expected cross-compiler to
> keep the packages in sync. A specific version could be used.

Thanks,
Christian Stewart
Christian Stewart July 7, 2023, 7:07 p.m. UTC | #2
Hi Charles,


On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us> wrote:
> This is not necessary with this patch series:
>
> https://patchwork.ozlabs.org/project/buildroot/list/?series=358451
>
> We can correctly bootstrap on arm64.

I should clarify: actually go bootstrap stage1 cannot work on arm64
(you're correct on this).

I had submitted this patch as well to use the host "go" in this case:

https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/

Similar to how we use host gcc.

However, I understand that your series intends to add downloading the
binaries for the Go compiler to Buildroot as well.

This is potentially useful but I still submit that we need to merge
the above series updating to Go 1.20 and splitting the bootstrap into
two phases before merging the go-native series.

I'll start testing your patch as well.

Thanks,
Christian

> On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com> wrote:
> >
> > The go-native is mainly for the aarch64 support since that was
> > added after the go-bootstrap written in C was dropped. So, to
> > get a cross compiler for go a host/native compiler needs to be
> > available to generate the setup for the needed target. This is
> > using the same go version as the expected cross-compiler to
> > keep the packages in sync. A specific version could be used.
Charles Hardin July 7, 2023, 8:05 p.m. UTC | #3
Christian

On Fri, Jul 7, 2023 at 12:08 PM Christian Stewart <christian@aperture.us>
wrote:

> Hi Charles,
>
>
> On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us>
> wrote:
> > This is not necessary with this patch series:
> >
> > https://patchwork.ozlabs.org/project/buildroot/list/?series=358451
> >
> > We can correctly bootstrap on arm64.
>
> I should clarify: actually go bootstrap stage1 cannot work on arm64
> (you're correct on this).
>
> I had submitted this patch as well to use the host "go" in this case:
>
>
> https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/
>
> Similar to how we use host gcc.
>
>
Weird - i was looking for a fix and never saw - perhaps i didn't have the
magic google search keywords. So, this was "our fix" to just get something
working specifically on a aarch64 builds. This is our annual timeline of
let's take our patches we have locally and send them upstream to see what
happens and see if we can drop our local patches in favor of upstream fixes.


> However, I understand that your series intends to add downloading the
> binaries for the Go compiler to Buildroot as well.
>
> This is potentially useful but I still submit that we need to merge
> the above series updating to Go 1.20 and splitting the bootstrap into
> two phases before merging the go-native series.
>
>
I just went for a quick and dirty to just use a native instead of doing the
bootstrap - we have a major issue in that we don't have many x86_64 build
systems anymore and we are having to move to aarch64 builds almost
exclusively. So, not having a way to get the compiler going was becoming a
headache.


> I'll start testing your patch as well.
>
>
This is not meant to replace - more of a - here is a way that worked for
us, if there is a better way that fits into buildroot then please just
merge that. Right now we don't have anything in the proper mainline so
everything is a deviation at this time.


> Thanks,
> Christian
>
> > On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com>
> wrote:
> > >
> > > The go-native is mainly for the aarch64 support since that was
> > > added after the go-bootstrap written in C was dropped. So, to
> > > get a cross compiler for go a host/native compiler needs to be
> > > available to generate the setup for the needed target. This is
> > > using the same go version as the expected cross-compiler to
> > > keep the packages in sync. A specific version could be used.
>
Christian Stewart July 7, 2023, 8:19 p.m. UTC | #4
Hi Charles,

On Fri, Jul 7, 2023, 1:05 PM Charles Hardin <ckhardin@gmail.com> wrote:

> Christian
>
> On Fri, Jul 7, 2023 at 12:08 PM Christian Stewart <christian@aperture.us>
> wrote:
>
>> Hi Charles,
>>
>>
>> On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us>
>> wrote:
>> > This is not necessary with this patch series:
>> >
>> > https://patchwork.ozlabs.org/project/buildroot/list/?series=358451
>> >
>> > We can correctly bootstrap on arm64.
>>
>> I should clarify: actually go bootstrap stage1 cannot work on arm64
>> (you're correct on this).
>>
>> I had submitted this patch as well to use the host "go" in this case:
>>
>>
>> https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/
>>
>> Similar to how we use host gcc.
>>
>>
> Weird - i was looking for a fix and never saw - perhaps i didn't have the
> magic google search keywords. So, this was "our fix" to just get something
> working specifically on a aarch64 builds. This is our annual timeline of
> let's take our patches we have locally and send them upstream to see what
> happens and see if we can drop our local patches in favor of upstream fixes.
>
>
>> However, I understand that your series intends to add downloading the
>> binaries for the Go compiler to Buildroot as well.
>>
>> This is potentially useful but I still submit that we need to merge
>> the above series updating to Go 1.20 and splitting the bootstrap into
>> two phases before merging the go-native series.
>>
>>
> I just went for a quick and dirty to just use a native instead of doing
> the bootstrap - we have a major issue in that we don't have many x86_64
> build systems anymore and we are having to move to aarch64 builds almost
> exclusively. So, not having a way to get the compiler going was becoming a
> headache.
>

Ah - okay this makes sense. In this case I recommend using my patch to use
the host Go compiler to bootstrap the Go compiler.

This still compiles whatever version of Go buildroot has in the tree from
source using the host Go compiler to bootstrap.

Thomas, all; I do request that we merge the go 1.20 series and the "use
host compiler to bootstrap on unsupported architectures" patch asap as
these are significant blockers for some users (as seen here)

Thanks!

Christian
diff mbox series

Patch

diff --git a/package/Config.in.host b/package/Config.in.host
index dcadbfdfc1..d7932f048b 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -41,6 +41,7 @@  menu "Host utilities"
 	source "package/gnupg/Config.in.host"
 	source "package/go/Config.in.host"
 	source "package/go-bootstrap/Config.in.host"
+	source "package/go-native/Config.in.host"
 	source "package/google-breakpad/Config.in.host"
 	source "package/gptfdisk/Config.in.host"
 	source "package/imagemagick/Config.in.host"
diff --git a/package/go-native/Config.in.host b/package/go-native/Config.in.host
new file mode 100644
index 0000000000..7e978e02b5
--- /dev/null
+++ b/package/go-native/Config.in.host
@@ -0,0 +1,12 @@ 
+config BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS
+	bool
+	# See https://go.dev/dl/ for supported architecture downloads
+	default y if BR2_HOSTARCH = "x86"
+	default y if BR2_HOSTARCH = "x86_64"
+	default y if BR2_HOSTARCH = "arm"
+	default y if BR2_HOSTARCH = "aarch64"
+
+config BR2_PACKAGE_HOST_GO_NATIVE
+	bool "host go-native"
+	depends on BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS
+	default y if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
diff --git a/package/go-native/go-native.hash b/package/go-native/go-native.hash
new file mode 100644
index 0000000000..3af2f51abc
--- /dev/null
+++ b/package/go-native/go-native.hash
@@ -0,0 +1,5 @@ 
+# From https://go.dev/dl/
+sha256  45a34313664d66087d49b46ca63a3ecf292f56e9e63bd439829b5eaf0270586d  go1.19.10.linux-386.tar.gz
+sha256  8b045a483d3895c6edba2e90a9189262876190dbbd21756870cdd63821810677  go1.19.10.linux-amd64.tar.gz
+sha256  df98698821211c819e8b2420c77a0f802d989e377718578a31b1f91f6be2c5b4  go1.19.10.linux-arm64.tar.gz
+sha256  937f885c35f191be4c6a618868ae0dbbb15b0213925a1a104d122eba02622356  go1.19.10.linux-armv6l.tar.gz
diff --git a/package/go-native/go-native.mk b/package/go-native/go-native.mk
new file mode 100644
index 0000000000..4b541cce73
--- /dev/null
+++ b/package/go-native/go-native.mk
@@ -0,0 +1,48 @@ 
+################################################################################
+#
+# go-native
+#
+################################################################################
+
+# Fixup the host arch as needed
+ifeq ($(HOSTARCH),x86)
+GO_NATIVE_HOST_ARCH = 386
+else ifeq ($(HOSTARCH),x86_64)
+GO_NATIVE_HOST_ARCH = amd64
+else ifeq ($(HOSTARCH),arm)
+GO_NATIVE_HOST_ARCH = armv6l
+else ifeq ($(HOSTARCH),aarch64)
+GO_NATIVE_HOST_ARCH = arm64
+else
+GO_NATIVE_HOST_ARCH = $(HOSTARCH)
+endif
+
+# Download the native tools from as the same go version from golang
+GO_NATIVE_VERSION = $(GO_VERSION)
+GO_NATIVE_SITE = https://go.dev/dl
+GO_NATIVE_LICENSE = BSD-3-Clause
+GO_NATIVE_LICENSE_FILES = LICENSE
+
+HOST_GO_NATIVE_SOURCE = go$(GO_NATIVE_VERSION).linux-$(GO_NATIVE_HOST_ARCH).tar.gz
+
+# To build programs that need cgo support the toolchain needs to be
+# available, so the toolchain is not needed to build host-go-native
+# itself, but needed by other packages that depend on
+# host-go-native.
+HOST_GO_NATIVE_DEPENDENCIES = toolchain
+
+HOST_GO_NATIVE_ROOT = $(HOST_DIR)/lib/go-$(GO_NATIVE_VERSION)
+
+define HOST_GO_NATIVE_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/bin/go $(HOST_GO_NATIVE_ROOT)/bin/go
+	$(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(HOST_GO_NATIVE_ROOT)/bin/gofmt
+
+	cp -a $(@D)/lib $(HOST_GO_NATIVE_ROOT)/
+	cp -a $(@D)/pkg $(HOST_GO_NATIVE_ROOT)/
+
+	# There is a known issue which requires the go sources to be installed
+	# https://golang.org/issue/2775
+	cp -a $(@D)/src $(HOST_GO_NATIVE_ROOT)/
+endef
+
+$(eval $(host-generic-package))
diff --git a/package/go/Config.in.host b/package/go/Config.in.host
index ded02d3b3a..4dbbe9a221 100644
--- a/package/go/Config.in.host
+++ b/package/go/Config.in.host
@@ -2,7 +2,8 @@ 
 config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS \
+		|| BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS
 	depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
 		|| BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
 		|| BR2_mips64 || BR2_mips64el || BR2_riscv || BR2_s390x
@@ -28,4 +29,5 @@  config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS \
+		|| BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS
diff --git a/package/go/go.mk b/package/go/go.mk
index 545d2117b7..af2e3b2f74 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -12,7 +12,11 @@  GO_LICENSE = BSD-3-Clause
 GO_LICENSE_FILES = LICENSE
 GO_CPE_ID_VENDOR = golang
 
+ifeq ($(BR2_PACKAGE_HOST_GO_NATIVE),y)
+HOST_GO_DEPENDENCIES = host-go-native
+else
 HOST_GO_DEPENDENCIES = host-go-bootstrap
+endif
 HOST_GO_GOPATH = $(HOST_DIR)/share/go-path
 HOST_GO_HOST_CACHE = $(HOST_DIR)/share/host-go-cache
 HOST_GO_ROOT = $(HOST_DIR)/lib/go
@@ -123,10 +127,16 @@  HOST_GO_HOST_ENV = \
 
 # The go build system is not compatible with ccache, so use
 # HOSTCC_NOCCACHE.  See https://github.com/golang/go/issues/11685.
+ifeq ($(BR2_PACKAGE_HOST_GO_NATIVE),y)
 HOST_GO_MAKE_ENV = \
+	GOROOT_BOOTSTRAP=$(HOST_GO_NATIVE_ROOT)
+else
+HOST_GO_MAKE_ENV = \
+	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT)
+endif
+HOST_GO_MAKE_ENV += \
 	GO111MODULE=off \
 	GOCACHE=$(HOST_GO_HOST_CACHE) \
-	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT) \
 	GOROOT_FINAL=$(HOST_GO_ROOT) \
 	GOROOT="$(@D)" \
 	GOBIN="$(@D)/bin" \