| Message ID | 20190226034554.24429-1-christian@paral.in |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [1/1] package/go: bump to version 1.12 | expand |
Hello Christian, Thanks for your patch. On Mon, 25 Feb 2019 19:45:54 -0800 Christian Stewart <christian@paral.in> wrote: > Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12 > > Additional notes on how Go modules will evolve in 2019 are here: > https://blog.golang.org/modules2019 > > In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled > for deprecation in 1.13, however. The Buildroot Go package infrastructure should > therefore aim to migrate to a new, currently undefined, Go module friendly > compilation approach before Go 1.13 is released. I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ? Best regards, Thomas
Hi Thomas, On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > I am puzzled by your statement "currently undefined". Is it undefined > in the sense "it is not defined yet how Buildroot will handle this" or > "it is not defined in Go yet how the module compilation will work in > 1.13" ? The Buildroot approach to handling it is currently not defined. Go Modules are built around fetching from the network and verifying downloads within the Go tool itself, while Buildroot likely will avoid any network calls outside of the official Buildroot download mechanism.So, something else will have to be designed to fix this. Best, Christian
Hi Christian, On Tue, Feb 26, 2019 at 8:46 AM Christian Stewart <christian@paral.in> wrote: > > Hi Thomas, > > On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: > > I am puzzled by your statement "currently undefined". Is it undefined > > in the sense "it is not defined yet how Buildroot will handle this" or > > "it is not defined in Go yet how the module compilation will work in > > 1.13" ? > > The Buildroot approach to handling it is currently not defined. Go > Modules are built around fetching from the network and verifying > downloads within the Go tool itself, while Buildroot likely will avoid > any network calls outside of the official Buildroot download > mechanism.So, something else will have to be designed to fix this. It's less of a problem with module support for go now, because the dependencies are proxyed and verified before being delivered to the go download mechanism, so the reproducibility is somewhat guaranteed. The problem remains for breaking the buildroot internal download mechanism and for not a so probably clear licensing of downloaded modules (I have to investigate this). I proposed the approach of tainting a buildroot configuration in such cases but the series was not accepted... > > Best, > Christian
Hi all, Angelo Compagnucci <angelo@amarulasolutions.com> writes: >> On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni >> <thomas.petazzoni@bootlin.com> wrote: >> > I am puzzled by your statement "currently undefined". Is it undefined >> > in the sense "it is not defined yet how Buildroot will handle this" or >> > "it is not defined in Go yet how the module compilation will work in >> > 1.13" ? > > It's less of a problem with module support for go now, because the > dependencies are proxyed and verified before being delivered to the go > download mechanism, so the reproducibility is somewhat guaranteed. > The problem remains for breaking the buildroot internal download > mechanism and for not a so probably clear licensing of downloaded > modules (I have to investigate this). I'm beginning to work on this now, would it be acceptable if Buildroot adopted the following approach: - Remove GOPATH - Configure Go to download module files to buildroot/dl/go-modules - Set GOPROXY to allow network during download step only. - Require go.mod and go.sum hashes for all Go packages. - During extract step, run "go mod vendor" to extract code to build/ - Use vendored building mode "-mod=vendor" This should allow Go to download and verify modules correctly during the download step, to the Buildroot dl/ tree, as well as disallowing network fetching during the build phase, and extracting versioned module code to the build directory. Best, Christian
Christian, All, On 2019-03-10 01:23 -0800, Christian Stewart spake thusly: > Angelo Compagnucci <angelo@amarulasolutions.com> writes: > >> On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni > >> <thomas.petazzoni@bootlin.com> wrote: > >> > I am puzzled by your statement "currently undefined". Is it undefined > >> > in the sense "it is not defined yet how Buildroot will handle this" or > >> > "it is not defined in Go yet how the module compilation will work in > >> > 1.13" ? > > > > It's less of a problem with module support for go now, because the > > dependencies are proxyed and verified before being delivered to the go > > download mechanism, so the reproducibility is somewhat guaranteed. > > The problem remains for breaking the buildroot internal download > > mechanism and for not a so probably clear licensing of downloaded > > modules (I have to investigate this). > > I'm beginning to work on this now, would it be acceptable if Buildroot > adopted the following approach: > > - Remove GOPATH > - Configure Go to download module files to buildroot/dl/go-modules How would you get the modules list and their URIs, when we are just downloading? The package sources are not yet extracted. > - Set GOPROXY to allow network during download step only. > - Require go.mod and go.sum hashes for all Go packages. Not sure what go.mod and go.sum are, but if you're suggesting that we have hashes in foo.hash for all modules in package foo, that would be nice, yes! > - During extract step, run "go mod vendor" to extract code to build/ ... getting the modules from $(BR2_DL_DIR) instead of downloading them that is? > - Use vendored building mode "-mod=vendor" > > This should allow Go to download and verify modules correctly during the > download step, to the Buildroot dl/ tree, as well as disallowing network > fetching during the build phase, and extracting versioned module code to > the build directory. If that were possible, I would like that, yes. That'd be good! However, I'd suggest that this extra module vendoring dance only occur if the package has set a specific variable, like we do for git submodules: FOO_GO_MODULES = YES (I almost wrote GOO_MODULES ;-) ) That way, if a package does not explicitly request them, it would fail at build time since they would not be available and would not be already downloaded. Of course, if we expect the majority of go packages to have modules, that variable would be moot... Regards, Yann E. MORIN.
Hi Yann, "Yann E. MORIN" <yann.morin.1998@free.fr> writes: >> I'm beginning to work on this now, would it be acceptable if Buildroot >> adopted the following approach: Here's a brief explanation of what Go modules are from the perspective of a Buildroot developer. Reference: https://github.com/golang/go/wiki/Modules Go code uses a package-based imports system. Imports are relative to a root directory, previously called GOPATH. import "github.com/docker/docker/pkg/myutils" This would resolve to $GOPATH/src/github.com/docker/docker/pkg/myutils. Note, although Github is in the import, it is not necessary to fetch the code from Github. This is just a naming convention. Buildroot packages currently use an additional feature in the Go tool which allows packages to avoid using GOPATH by "vendoring" dependencies - copying the code directly into the Git repository. vendor/github.com/docker/docker/pkg/myutils Today all of our Buildroot Go modules use "vendor" to avoid downloading dependencies. But, this requires that the Go projects we add to Buildroot include all of their dependencies in their repositories, which is a mess. It also does not allow us the opportunity to validate dependency changes when upgrading packages in Buildroot. The Go module system aims to completely replace the GOPATH mechanism by allowing the Go tool to reason about package versioning, code fetching, proxying, and checksumming. A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers. module mymodule require ( github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible ) The Go tool creates a go.sum file next to the go.mod file. The go.sum file is akin to Node's package-shrinkwrap.json. github.com/aws/aws-sdk-go v1.15.31/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= github.com/aws/aws-sdk-go v1.17.12 h1:jMFwRUaM0LcfdenfvbDLePNoWSoCdOHqF4RCvSB4xNQ= github.com/aws/aws-sdk-go v1.17.12/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible h1:8fBbhRkI5/0ocLFbrhPgnGUm0ogc+Gko1cRodPWDKX4= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= The file contains import paths, versions, and file hashes for the download as well as the contained go.mod file. With replace directives, it's possible to link together local copies of modules: replace ( k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi ) When GO111MODULE=on, all of the Go tool commands become "module-aware." GOPATH is no longer required. Running "go build" for example will fetch dependencies from the Internet, checksum them, and extract to a staging directory (defaulting currently to GOPATH/pkg/gomod or so). Imports are automatically resolved to the appropriately versioned staging directory. There are various ways to control the download / extract behavior: The GOPROXY environment variable allows further control over the download source. If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems. Setting GOPROXY to "off" disallows downloading modules from any source. Otherwise, GOPROXY is expected to be the URL of a module proxy, in which case the go command will fetch all modules from that proxy. No matter the source of the modules, downloaded modules must match existing entries in go.sum... Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy. GOPROXY additionally supports file:// URLs. --- With the above in mind, here's how I believe we could integrate this with Buildroot: >> - Remove GOPATH GOPATH is deprecated anyway so this will be necessary. I'd strongly recommend that Buildroot make a hard requirement that all Go-based packages use the go.mod and go.sum construction, and configure Go to ignore the existing vendor/ trees. This would allow us to pin versioning and tightly control download checksumming through the package.hash -> go.sum merkle tree. If a package does not have a go.mod or go.sum file, a patch would be added to Buildroot to include the files with the correct pinned dependencies. >> - Configure Go to download module files to buildroot/dl/go-modules We can set the paths such that the $GOPATH/pkg/mod/cache/download directory resolves to buildroot/dl/go-modules. According to the above documentation, a Buildroot user could then serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture. Initially, the desired behavior would be implemented by setting GOPROXY to "direct" (or perhaps the Buildroot download mirrors?) during the downloading phase. > How would you get the modules list and their URIs, when we are just > downloading? The package sources are not yet extracted. This is a good question. There are a couple of options. One conservative approach is to download and extract, and use the "extract" step to do the second stage of module downloading after extracting the package sources. A more radical approach, which would perhaps be cleaner, would be to disable the Buildroot download mechanism completely for Go packages, include the root go.mod and go.sum in Buildroot next to the .mk and Config files, and allow Go to manage fetching all of the dependencies including the root package. >> - Set GOPROXY to allow network during download step only. This would be, during the download phase, execute "go mod download" During the extract phase, we would create the build/package/ dir under the output directory as usual in Buildroot, then execute "go mod vendor" to extract the code sources into build/package/vendor During the build phase, we would then disable the Go moduling system and compile from the extracted "vendor" tree. This would help improve build stability and determinism by only engaging the Module code during download / extract. >> - Require go.mod and go.sum hashes for all Go packages. > Not sure what go.mod and go.sum are, but if you're suggesting that we > have hashes in foo.hash for all modules in package foo, that would be > nice, yes! They are explained above - these files specify the required dependency versions, and hash the downloaded code artifacts to ensure integrity. >> - During extract step, run "go mod vendor" to extract code to build/ > ... getting the modules from $(BR2_DL_DIR) instead of downloading them > that is? Exactly. >> - Use vendored building mode "-mod=vendor" > > If that were possible, I would like that, yes. That'd be good! > > However, I'd suggest that this extra module vendoring dance only occur > if the package has set a specific variable, like we do for git > submodules: > > FOO_GO_MODULES = YES > > (I almost wrote GOO_MODULES ;-) ) > > That way, if a package does not explicitly request them, it would fail > at build time since they would not be available and would not be already > downloaded. The only situation where we would not want to use Go modules is if the package has no external dependencies, or if we want to use the old-fashioned vendor/ tree approach. I can prototype this in a way where the old behavior will be kept without a FOO_GO_MODULES flag, if this is something that we want. > Of course, if we expect the majority of go packages to have modules, > that variable would be moot... Best regards, Christian Stewart
Hi Christian, Thanks for this excellent write-up! It certainly clarifies things. On 11/03/2019 01:08, Christian Stewart wrote: [snip] > A project can contain any number of go.mod files. A go.mod file is akin > to Node's package.json file. It specifies all direct and indirect > dependencies of all Go packages in the subtree below the file. The Go > tool can manage this file automatically if desired, and specifies a > required format. Go.mod additionally requires dependency versions to be > explicitly specified. There is no semantic versioning or asterisk-based > version specifiers. > > module mymodule > > require ( > github.com/aws/aws-sdk-go v1.17.12 // indirect > github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible > ) So, since the version is explicit, it's actually completely equivalent to the vendoring: it's still impossible to treat the modules as packages in the Buildroot sense, since Buildroot doesn't allow the same package to have different versions. It's essentially just making our life more difficult :-(. That said, at least it's a consistent way for Go packages to work. Another thing: the module specification is just a path component, not a full URL. Is the https:// implicit? Does it always pull from git? Or does it have some kind of autoconversion similar to our github helper? [snip] > With the above in mind, here's how I believe we could integrate this > with Buildroot: > >>> - Remove GOPATH > > GOPATH is deprecated anyway so this will be necessary. I'd strongly > recommend that Buildroot make a hard requirement that all Go-based > packages use the go.mod and go.sum construction, and configure Go to > ignore the existing vendor/ trees. I've taken a look at the upstream master branch of the go packages we currently have, and none of them seems to have a go.mod yet... So I think this is a little bit premature. > This would allow us to pin versioning > and tightly control download checksumming through the package.hash -> > go.sum merkle tree. I may miss something, but if it's in vendor/ in the upstream git repo (either as a submodule or as a copy), it's also tightly controlled and checksummed, right? So I don't think this is really bringing any value. For us, I mean - I imagine it *does* bring value for people developing Go packages. > If a package does not have a go.mod or go.sum file, a patch would be > added to Buildroot to include the files with the correct pinned > dependencies. > >>> - Configure Go to download module files to buildroot/dl/go-modules If all modules go together in a single location, there is a risk that there will be two modules with the same name and version that are in reality different modules. Or are the full paths repeated there? I.e., does the zip file end up under a github.com/foo/bar directory? > > We can set the paths such that the $GOPATH/pkg/mod/cache/download > directory resolves to buildroot/dl/go-modules. According to the above > documentation, a Buildroot user could then serve the dl/go-modules > directory directly with a HTTP server and set GOPROXY such that all > module downloads come from that server. This could be configurable > eventually via the Buildroot KConfig architecture. No need for it to be configurable, it could be the BR2_PRIMARY_SITE. However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the full solution would be to add something like go download infra that first sets GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally $(BR2_SECONDARY_SITE)/go-modules. Except that I have another idea, see below. > Initially, the desired behavior would be implemented by setting GOPROXY > to "direct" (or perhaps the Buildroot download mirrors?) during the > downloading phase. > >> How would you get the modules list and their URIs, when we are just >> downloading? The package sources are not yet extracted. > > This is a good question. There are a couple of options. One conservative > approach is to download and extract, and use the "extract" step to do > the second stage of module downloading after extracting the package > sources. > > A more radical approach, which would perhaps be cleaner, would be to > disable the Buildroot download mechanism completely for Go packages, > include the root go.mod and go.sum in Buildroot next to the .mk and > Config files, and allow Go to manage fetching all of the dependencies > including the root package. That, combined with the go download infra I mentioned above, would indeed be a feasible approach. Ideally, the go.mod would include the package itself too, so we don't need to download in two steps (i.e. first the package itself, then the modules it uses). But that might be a bit more difficult. But again, see below for an alternative (cliffhanger...) >>> - Set GOPROXY to allow network during download step only. > > This would be, during the download phase, execute "go mod download" > > During the extract phase, we would create the build/package/ dir under > the output directory as usual in Buildroot, then execute "go mod vendor" > to extract the code sources into build/package/vendor Alternatively, we could add a post-extract hook that executes "go mod vendor" with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant. > During the build phase, we would then disable the Go moduling system and disable == GOPROXY=off, right? > compile from the extracted "vendor" tree. This would help improve build > stability and determinism by only engaging the Module code during > download / extract. So this gives me the idea of the alternative... If we anyway end up with everything extracted in the vendor tree, we can just as well do that during the *download* step. Similar like we do for VCS downloads: we get the *entire* source tree, including go modules, and tar that up. We'd still have a go SITE_METHOD and corresponding download helper. This one would first use another download method to get the base tarball, then extract it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball. This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make source-check' would all still work. So it would be (I think) the least invasive way to introduce this. [snip] >> However, I'd suggest that this extra module vendoring dance only occur >> if the package has set a specific variable, like we do for git >> submodules: >> >> FOO_GO_MODULES = YES With my proposal, this would be FOO_SITE_METHOD = go Oh BTW we'd probably also need to support something like FOO_SITE_METHOD = go+git. dl-wrapper will (should) strip off the first +, the go download helper would have to handle the second +. But that can be done in a follow-up patch :-) >> >> (I almost wrote GOO_MODULES ;-) ) >> >> That way, if a package does not explicitly request them, it would fail >> at build time since they would not be available and would not be already >> downloaded. > > The only situation where we would not want to use Go modules is if the > package has no external dependencies, or if we want to use the > old-fashioned vendor/ tree approach. I can prototype this in a way where > the old behavior will be kept without a FOO_GO_MODULES flag, if this is > something that we want. I think it's worth prototyping something that is not complete yet, and send it to the list with a lot of comments in the commit message about what is still missing. That's probably a better basis for discussion that English text. Regards, Arnout > >> Of course, if we expect the majority of go packages to have modules, >> that variable would be moot... > > Best regards, > Christian Stewart > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot >
Hi Arnout, Arnout Vandecappelle <arnout@mind.be> writes: > So, since the version is explicit, it's actually completely equivalent to the > vendoring: it's still impossible to treat the modules as packages in the > Buildroot sense, since Buildroot doesn't allow the same package to have > different versions. It's essentially just making our life more difficult :-(. I don't think it makes sense to treat them as packages in this sense anyway. > That said, at least it's a consistent way for Go packages to work. This also seems to all be in line with the Go developers' intent and philosophy around dependency management: they prioritize compatibility and robust operation over deduplicating dependency instances. > Another thing: the module specification is just a path component, not a full > URL. Is the https:// implicit? Does it always pull from git? Or does it have > some kind of autoconversion similar to our github helper? The Go code fetching mechanisms are complex, but heavily used, and I would assume they always use https shallow Git clones of the desired versions. The go.sum checksumming system ensures everything is consistent as well. >> GOPATH is deprecated anyway so this will be necessary. I'd strongly >> recommend that Buildroot make a hard requirement that all Go-based >> packages use the go.mod and go.sum construction, and configure Go to >> ignore the existing vendor/ trees. > > I've taken a look at the upstream master branch of the go packages we currently > have, and none of them seems to have a go.mod yet... So I think this is a little > bit premature. Removing GOPATH is not as extreme as it sounds. This is just a re-organization of the build code to simplify it quite a bit. Today, the build code tries to synthesize a gopath (pkg-golang.mk): $(2)_WORKSPACE ?= _gopath $(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE)) $(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR) mkdir -p $$(dir $$($(2)_SRC_PATH)) ln -sf $$(@D) $$($(2)_SRC_PATH) GOPATH="$$(@D)/$$($(2)_WORKSPACE)" This creates the following directory structure: $(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE Then the Go tool is configured to use pkg_gopath as the GOPATH. With the Go modules system, even if a Go project is not configured to use Go modules at all, it no longer is necessary to build a GOPATH. With the absence of a go.mod and go.sum file, and the presence of a vendor/ dir, the Go tool will now correctly recognize the vendor/ and build outside GOPATH. This defaults GO111MODULE=on when Go is invoked outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at least, deprecated). In buildroot, we can simplify the code significantly with this new construction. We can simply extract code to $(@D), enable GO111MODULE, and the build will function correctly. >> This would allow us to pin versioning >> and tightly control download checksumming through the package.hash -> >> go.sum merkle tree. > > I may miss something, but if it's in vendor/ in the upstream git repo (either > as a submodule or as a copy), it's also tightly controlled and checksummed, > right? The build would be executed with go build -mod=vendor, which tells the Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is already present in the upstream Git repo, we can skip the "go mod download" and "go mod vendor" steps. The result is that the Go tool uses the following directory layout regardless of if Go modules download/extract was used during the download phase: - $(@D)/vendor: contains all deps extracted - $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule" > So I don't think this is really bringing any value. For us, I mean - I > imagine it *does* bring value for people developing Go packages. If we ignore vendor/ in the upstream Git repository and introduce our own go.mod, it would allow us to tightly control changes to dependencies (including security updates) within the Buildroot project, even if upstream maintainers do not update their vendor trees. This is especially important when we are using older revisions of projects that do not maintain release branches with vendored code. In short, this would allow us to centralize the responsibility of Go dependency management in the Go module system, store all dependencies in the GOPROXY mechanism / Buildroot download directory, and force upgrades of dependencies for older revisions of projects without introducing large code vendoring patches into the Buildroot tree. >> If a package does not have a go.mod or go.sum file, a patch would be >> added to Buildroot to include the files with the correct pinned >> dependencies. This seems like a viable solution - 1. Copy the Buildroot package directory go.mod, go.sum in if exists. 2. If no go.mod exists in the project, write one with the minimum "module github.com/myorg/mymodule" line. 3. Execute "go mod download" and "go mod vendor" as discussed. In point #1, if vendor/ already exists, and we indicate we want to override vendor/, then we can delete the upstream Git vendor/ tree in this step. >>>> - Configure Go to download module files to buildroot/dl/go-modules > > If all modules go together in a single location, there is a risk that there > will be two modules with the same name and version that are in reality different > modules. Or are the full paths repeated there? I.e., does the zip file end up > under a github.com/foo/bar directory? No. The Go tool uses an intricate path scheme to avoid collisions, and does not assume integrity of the on-disk cache (as far as I know). >> We can set the paths such that the $GOPATH/pkg/mod/cache/download >> directory resolves to buildroot/dl/go-modules. According to the above >> documentation, a Buildroot user could then serve the dl/go-modules >> directory directly with a HTTP server and set GOPROXY such that all >> module downloads come from that server. This could be configurable >> eventually via the Buildroot KConfig architecture. > > No need for it to be configurable, it could be the BR2_PRIMARY_SITE. This seems like the best approach. Here is some reference from go help goproxy: A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) can be a module proxy. The GET requests sent to a Go module proxy are: GET $GOPROXY/<module>/@v/list returns a list of all known versions of the given module, one per line. GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata about that version of the given module. GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file for that version of the given module. GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive for that version of the given module. To avoid problems when serving from case-sensitive file systems, the <module> and <version> elements are case-encoded, replacing every uppercase letter with an exclamation mark followed by the corresponding lower-case letter: github.com/Azure encodes as github.com/!azure. So perhaps we would want to add some suffix to PRIMARY_SITE. Something further down the pipe, but worth looking at for those interested, is the upcoming Go code notary project: https://go.googlesource.com/proposal/+/master/design/25530-notary.md > However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the > full solution would be to add something like go download infra that first sets > GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to > $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally > $(BR2_SECONDARY_SITE)/go-modules. https://github.com/golang/go/issues/26334 Looks like indeed, it doesn't have a fallback yet. However, this is a bit awkward: what if our GOPROXY doesn't have just a single dependency? The download tree looks like: ├── github.com │ ├── davecgh │ │ └── go-spew │ │ └── @v │ │ ├── list │ │ ├── list.lock │ │ ├── v1.1.1.info │ │ └── v1.1.1.mod └── golang.org └── x ├── crypto │ └── @v │ ├── list │ ├── list.lock │ ├── v0.0.0-20180904163835-0709b304e793.info │ ├── v0.0.0-20180904163835-0709b304e793.lock │ ├── v0.0.0-20180904163835-0709b304e793.mod │ ├── v0.0.0-20180904163835-0709b304e793.zip │ └── v0.0.0-20180904163835-0709b304e793.ziphash I tried downloading just one dependency, then copying the $GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy. Then, invoking Go build: go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod Okay, this worked as expected. Now, adding a new dependency: Fetching file://goproxy/github.com/urfave/cli/@v/list Fetching file://goproxy/github.com/urfave/@v/list Fetching file://goproxy/github.com/@v/list build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli Hmm. Looks like it doesn't have a fallback yet. Athens is another project addressing this area: https://github.com/gomods/athens > Except that I have another idea, see below. >> A more radical approach, which would perhaps be cleaner, would be to >> disable the Buildroot download mechanism completely for Go packages, >> include the root go.mod and go.sum in Buildroot next to the .mk and >> Config files, and allow Go to manage fetching all of the dependencies >> including the root package. > > That, combined with the go download infra I mentioned above, would indeed be a > feasible approach. > > Ideally, the go.mod would include the package itself too, so we don't need to > download in two steps (i.e. first the package itself, then the modules it uses). > But that might be a bit more difficult. This is completely doable. This approach actually opens up a few other interesting possibilties. We can add a command to set the environment variables up for the Go tool, and then execute the tool to automatically maintain the go.mod file for a particular package. With no package arguments, 'go get' applies to the main module, and to the Go package in the current directory, if any. In particular, 'go get -u' and 'go get -u=patch' update all the dependencies of the main module. If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod described above. Instead, it fails when any changes to go.mod are needed. This setting is most useful to check that go.mod does not need updates, such as in a continuous integration and testing system. The "go get" command remains permitted to update go.mod even with -mod=readonly, and the "go mod" commands do not take the -mod flag (or any other build flags). Of course, this is speculative and not necessary for an initial implementation. > Alternatively, we could add a post-extract hook that executes "go mod vendor" > with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant. This is correct. >> During the build phase, we would then disable the Go moduling system and > > disable == GOPROXY=off, right? GOPROXY=off in every situation other than when we want to download code. Additionally, during build, we would use "-mod=vendor." To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -mod=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored. > If we anyway end up with everything extracted in the vendor tree, we can just > as well do that during the *download* step. Similar like we do for VCS > downloads: we get the *entire* source tree, including go modules, and tar that up. > > We'd still have a go SITE_METHOD and corresponding download helper. This one > would first use another download method to get the base tarball, then extract > it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball. > > This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make > source-check' would all still work. So it would be (I think) the least invasive > way to introduce this. This seems like the right approach. This all feels similar to how Buildroot uses the Git tool to clone a repository, check out a revision, and bundle up a tarball today. > With my proposal, this would be FOO_SITE_METHOD = go > > Oh BTW we'd probably also need to support something like FOO_SITE_METHOD = > go+git. dl-wrapper will (should) strip off the first +, the go download helper > would have to handle the second +. But that can be done in a follow-up patch :-) This is way too fancy for me to implement :) > I think it's worth prototyping something that is not complete yet, and send it > to the list with a lot of comments in the commit message about what is still > missing. That's probably a better basis for discussion that English text. Something to note: you can set GOMOD variable to the path to a go.mod file. Okay, I'll have a look at: - use vendor/ with GO111MODULE=on, remove gopath - use -mod=vendor during build phase - set GOCACHE properly (what is it set to today??) - set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod And (pick one): - post-extract hook with optional go.mod in repository, "go mod vendor" - additional site method which "go mod vendor" and re-compress to tarball Best regards, Christian Stewart
On 12/03/2019 04:35, Christian Stewart wrote: > Hi Arnout, > > Arnout Vandecappelle <arnout@mind.be> writes: >> So, since the version is explicit, it's actually completely equivalent to the >> vendoring: it's still impossible to treat the modules as packages in the >> Buildroot sense, since Buildroot doesn't allow the same package to have >> different versions. It's essentially just making our life more difficult :-(. > > I don't think it makes sense to treat them as packages in this sense anyway. Actually, below, you do seem to suggest something that goes in that direction... [snip] > Removing GOPATH is not as extreme as it sounds. This is just a > re-organization of the build code to simplify it quite a bit. > > Today, the build code tries to synthesize a gopath (pkg-golang.mk): > > $(2)_WORKSPACE ?= _gopath > > $(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE)) > $(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) > $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) > $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) > $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR) > > mkdir -p $$(dir $$($(2)_SRC_PATH)) > ln -sf $$(@D) $$($(2)_SRC_PATH) > > GOPATH="$$(@D)/$$($(2)_WORKSPACE)" > > This creates the following directory structure: > > $(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE > > Then the Go tool is configured to use pkg_gopath as the GOPATH. > > With the Go modules system, even if a Go project is not configured to > use Go modules at all, it no longer is necessary to build a GOPATH. > > With the absence of a go.mod and go.sum file, and the presence of a > vendor/ dir, the Go tool will now correctly recognize the vendor/ and > build outside GOPATH. This defaults GO111MODULE=on when Go is invoked > outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at > least, deprecated). > > In buildroot, we can simplify the code significantly with this new > construction. We can simply extract code to $(@D), enable GO111MODULE, > and the build will function correctly. Hell yeah! As a first step, that sounds like a great move. So we'd have: 1. Bump package/go 2. Set GO111MODULE=on and remove GOPATH from pkg-golang 3. Remove now-unused PKG_SRC_{DOMAIN,VENDOR,...} from packages 4. Remove this stuff from documentation Step 2 should probably also set GOPROXY=off in all steps to avoid accidental downloads. Maybe we should even export GOPROXY=off and enable it only in the specific cases where it needs to be enabled? That would be a fifth patch. >>> This would allow us to pin versioning >>> and tightly control download checksumming through the package.hash -> >>> go.sum merkle tree. >> >> I may miss something, but if it's in vendor/ in the upstream git repo (either >> as a submodule or as a copy), it's also tightly controlled and checksummed, >> right? > > The build would be executed with go build -mod=vendor, which tells the > Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is > already present in the upstream Git repo, we can skip the "go mod > download" and "go mod vendor" steps. So that would be the current sitatuation then. > The result is that the Go tool uses the following directory layout > regardless of if Go modules download/extract was used during the > download phase: > > - $(@D)/vendor: contains all deps extracted > - $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule" Is this go.mod then autogenerated by 'go build -mod=vendor' or do we need to generate it in a post-patch hook? >> So I don't think this is really bringing any value. For us, I mean - I >> imagine it *does* bring value for people developing Go packages. > > If we ignore vendor/ in the upstream Git repository and introduce our > own go.mod, it would allow us to tightly control changes to dependencies > (including security updates) within the Buildroot project, even if > upstream maintainers do not update their vendor trees. This is > especially important when we are using older revisions of projects that > do not maintain release branches with vendored code. > > In short, this would allow us to centralize the responsibility of Go > dependency management in the Go module system, store all dependencies in > the GOPROXY mechanism / Buildroot download directory, and force upgrades > of dependencies for older revisions of projects without introducing > large code vendoring patches into the Buildroot tree. It sounds to me like a bit too big of an endeavor for the small Buildroot project to take on... Upstream Go just doesn't have a decent security story for not-well-maintained packages, and I don't think that there's much we can do about it. >>> If a package does not have a go.mod or go.sum file, a patch would be >>> added to Buildroot to include the files with the correct pinned >>> dependencies. > > This seems like a viable solution - > > 1. Copy the Buildroot package directory go.mod, go.sum in if exists. > 2. If no go.mod exists in the project, write one with the minimum > "module github.com/myorg/mymodule" line. > 3. Execute "go mod download" and "go mod vendor" as discussed. > > In point #1, if vendor/ already exists, and we indicate we want to > override vendor/, then we can delete the upstream Git vendor/ tree in > this step. Yes, sounds good to me. Though I would first like to see if there actually is any project that has both a go.mod and an existing vendor/ tree. What does "go mod vendor" do if a vendor/ tree already exists? Maybe we just have to keep that behaviour, i.e. not delete the existing vendor tree and let the go tool take care of things. It should in principle do the right thing. >>>>> - Configure Go to download module files to buildroot/dl/go-modules >> >> If all modules go together in a single location, there is a risk that there >> will be two modules with the same name and version that are in reality different >> modules. Or are the full paths repeated there? I.e., does the zip file end up >> under a github.com/foo/bar directory? > > No. The Go tool uses an intricate path scheme to avoid collisions, and > does not assume integrity of the on-disk cache (as far as I know). > >>> We can set the paths such that the $GOPATH/pkg/mod/cache/download >>> directory resolves to buildroot/dl/go-modules. According to the above >>> documentation, a Buildroot user could then serve the dl/go-modules >>> directory directly with a HTTP server and set GOPROXY such that all >>> module downloads come from that server. This could be configurable >>> eventually via the Buildroot KConfig architecture. >> >> No need for it to be configurable, it could be the BR2_PRIMARY_SITE. > > This seems like the best approach. > > Here is some reference from go help goproxy: > > A Go module proxy is any web server that can respond to GET requests for > URLs of a specified form. The requests have no query parameters, so even > a site serving from a fixed file system (including a file:/// URL) > can be a module proxy. > > The GET requests sent to a Go module proxy are: > > GET $GOPROXY/<module>/@v/list returns a list of all known versions of the > given module, one per line. > > GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata > about that version of the given module. > > GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file > for that version of the given module. > > GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive > for that version of the given module. > > To avoid problems when serving from case-sensitive file systems, > the <module> and <version> elements are case-encoded, replacing every > uppercase letter with an exclamation mark followed by the corresponding > lower-case letter: github.com/Azure encodes as github.com/!azure. > > So perhaps we would want to add some suffix to PRIMARY_SITE. That's what you proposed: go-modules. But I'm not sure now: does the go mod download generate these auxiliary files automatically in pkg/mod/cache/download? I.e. can we just serve $(BR2_DL_DIR) as PRIMARY, set GOPROXY to $(BR2_PRIMARY_SITE)/go-modules, and Bob's your uncle? > Something further down the pipe, but worth looking at for those > interested, is the upcoming Go code notary project: > > https://go.googlesource.com/proposal/+/master/design/25530-notary.md > >> However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the >> full solution would be to add something like go download infra that first sets >> GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to >> $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally >> $(BR2_SECONDARY_SITE)/go-modules. > > https://github.com/golang/go/issues/26334 > > Looks like indeed, it doesn't have a fallback yet. However, this is a > bit awkward: what if our GOPROXY doesn't have just a single dependency? > > The download tree looks like: > > ├── github.com > │ ├── davecgh > │ │ └── go-spew > │ │ └── @v > │ │ ├── list > │ │ ├── list.lock > │ │ ├── v1.1.1.info > │ │ └── v1.1.1.mod > └── golang.org > └── x > ├── crypto > │ └── @v > │ ├── list > │ ├── list.lock > │ ├── v0.0.0-20180904163835-0709b304e793.info > │ ├── v0.0.0-20180904163835-0709b304e793.lock > │ ├── v0.0.0-20180904163835-0709b304e793.mod > │ ├── v0.0.0-20180904163835-0709b304e793.zip > │ └── v0.0.0-20180904163835-0709b304e793.ziphash > > I tried downloading just one dependency, then copying the > $GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy. > > Then, invoking Go build: > > go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 > Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info > Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod > > Okay, this worked as expected. Now, adding a new dependency: > > Fetching file://goproxy/github.com/urfave/cli/@v/list > Fetching file://goproxy/github.com/urfave/@v/list > Fetching file://goproxy/github.com/@v/list > build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli > > Hmm. Looks like it doesn't have a fallback yet. Heh, I didn't think of that: the tool tries to download *all* modules... What would help is if there was some kind of keep-going option. Then we can do all the fallback on the Buildroot side, like I mentioned: first try with PRIMARY (and download some dependencies and put it in the cache), then try with direct, then try with SECONDARY. The last one should succeed fully, reusing things from the cache if they were already downloaded before, and fetching them from direct or secondary if not. > > Athens is another project addressing this area: https://github.com/gomods/athens [snip] >> If we anyway end up with everything extracted in the vendor tree, we can just >> as well do that during the *download* step. Similar like we do for VCS >> downloads: we get the *entire* source tree, including go modules, and tar that up. >> >> We'd still have a go SITE_METHOD and corresponding download helper. This one >> would first use another download method to get the base tarball, then extract >> it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball. >> >> This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make >> source-check' would all still work. So it would be (I think) the least invasive >> way to introduce this. > > This seems like the right approach. This all feels similar to how > Buildroot uses the Git tool to clone a repository, check out a revision, > and bundle up a tarball today. And I realize now: it would also handle legal-info, which would not be handled correctly with everything that was proposed up to now (because the go modules are not part of $(PKG)_ALL_DOWNLOADS, they don't get copied to legal-info output). > >> With my proposal, this would be FOO_SITE_METHOD = go >> >> Oh BTW we'd probably also need to support something like FOO_SITE_METHOD = >> go+git. dl-wrapper will (should) strip off the first +, the go download helper >> would have to handle the second +. But that can be done in a follow-up patch :-) > > This is way too fancy for me to implement :) This would in fact not be needed if we would have the go.mod in the Buildroot tree and use go mod to get the source itself too. > >> I think it's worth prototyping something that is not complete yet, and send it >> to the list with a lot of comments in the commit message about what is still >> missing. That's probably a better basis for discussion that English text. > > Something to note: you can set GOMOD variable to the path to a go.mod file. > > Okay, I'll have a look at: > > - use vendor/ with GO111MODULE=on, remove gopath That one is not controversial for sure. > - use -mod=vendor during build phase Ditto. It should essentially change nothing for our current packages, right? > - set GOCACHE properly (what is it set to today??) You already sent a patch for that :-) > - set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod That one seems useless - with the steps above, there should never be any download done by go mod. > And (pick one): > > - post-extract hook with optional go.mod in repository, "go mod vendor" > - additional site method which "go mod vendor" and re-compress to tarball So these are the more controversial ones. I think you should pick the one you like most, make a quick prototype that works for one specific package, note what is missing in the commit log, and send it to the list for discussion. Thanks for this work! Regards, Arnout
diff --git a/package/go/go.hash b/package/go/go.hash index 29891399c3..e78cfe410c 100644 --- a/package/go/go.hash +++ b/package/go/go.hash @@ -1,3 +1,3 @@ # From https://golang.org/dl/ -sha256 bc1ef02bb1668835db1390a2e478dcbccb5dd16911691af9d75184bbe5aa943e go1.11.5.src.tar.gz +sha256 09c43d3336743866f2985f566db0520b36f4992aea2b4b2fd9f52f17049e88f2 go1.12.src.tar.gz sha256 2d36597f7117c38b006835ae7f537487207d8ec407aa9d9980794b2030cbc067 LICENSE diff --git a/package/go/go.mk b/package/go/go.mk index 7755a3f785..8709198b63 100644 --- a/package/go/go.mk +++ b/package/go/go.mk @@ -4,7 +4,7 @@ # ################################################################################ -GO_VERSION = 1.11.5 +GO_VERSION = 1.12 GO_SITE = https://storage.googleapis.com/golang GO_SOURCE = go$(GO_VERSION).src.tar.gz
Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12 Additional notes on how Go modules will evolve in 2019 are here: https://blog.golang.org/modules2019 In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled for deprecation in 1.13, however. The Buildroot Go package infrastructure should therefore aim to migrate to a new, currently undefined, Go module friendly compilation approach before Go 1.13 is released. Signed-off-by: Christian Stewart <christian@paral.in> --- package/go/go.hash | 2 +- package/go/go.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)