| Message ID | 20250504163611.141654-1-Stefan.Nickl@gmail.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | [1/1] Add gh site method for GitHub cli | expand |
On 04/05/2025 18:36, stefan.nickl@gmail.com wrote: > From: Stefan Nickl <Stefan.Nickl@gmail.com> > > Hey there, > I would like to propose adding a download/site method "gh" based on the > GitHub CLI tool https://cli.github.com/ > > The main advantage is that it also works for private GitHub repositories > after logging in with "gh auth login". Having something github specific is not very nice... We should then also provide the equivalent for gitlab, bitbucket, ... > It is intended as an alternative to pulling with the "git" method, which > has the drawback of requiring separate ssh keys I guess you mean that you need to generate ssh keys for CI? > and resulting in > different archives depending on the buildroot version, How does gh help with that? The tarballs it produces are not reproducible at all... The ones we use happen to be the same all the time because we re-request them all the time so they're cached, but there's actually no guarantee for that. > and passing an > authentication token on the wget command line, which risks leaking the > token to third-party servers. Well, that's not really a good reason, there are better ways to do that. I've created a documentation patch [1] to explain it and put you in Cc. Regards, Arnout [1] https://patchwork.ozlabs.org/project/buildroot/patch/20250516211429.763211-1-arnout@rnout.be/ > > Signed-off-by: Stefan Nickl <Stefan.Nickl@gmail.com> > --- > Config.in | 4 ++++ > package/pkg-download.mk | 1 + > support/download/dl-wrapper | 2 +- > support/download/gh | 43 +++++++++++++++++++++++++++++++++++++ > 4 files changed, 49 insertions(+), 1 deletion(-) > create mode 100755 support/download/gh > > diff --git a/Config.in b/Config.in > index d730f2034b..cb178ee8d2 100644 > --- a/Config.in > +++ b/Config.in > @@ -172,6 +172,10 @@ config BR2_HG > string "Mercurial (hg) command" > default "hg" > > +config BR2_GH > + string "GitHub cli command" > + default "gh" > + > config BR2_ZCAT > string "zcat command" > default "gzip -d -c" > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > index cf5959ea95..08b0d420c7 100644 > --- a/package/pkg-download.mk > +++ b/package/pkg-download.mk > @@ -111,6 +111,7 @@ define DOWNLOAD > CVS="$(call qstrip,$(BR2_CVS))" \ > GIT="$(call qstrip,$(BR2_GIT))" \ > HG="$(call qstrip,$(BR2_HG))" \ > + GH="$(call qstrip,$(BR2_GH))" \ > LOCALFILES="$(call qstrip,$(BR2_LOCALFILES))" \ > SCP="$(call qstrip,$(BR2_SCP))" \ > SFTP="$(call qstrip,$(BR2_SFTP))" \ > diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper > index 5445aad5a7..a209665b0f 100755 > --- a/support/download/dl-wrapper > +++ b/support/download/dl-wrapper > @@ -95,7 +95,7 @@ main() { > backend_urlencode="${uri%%+*}" > backend="${backend_urlencode%|*}" > case "${backend}" in > - git|svn|cvs|bzr|file|scp|hg|sftp) ;; > + git|svn|cvs|bzr|file|scp|hg|gh|sftp) ;; > ftp|ftps) backend="curl" ;; > *) backend="wget" ;; > esac > diff --git a/support/download/gh b/support/download/gh > new file mode 100755 > index 0000000000..5c36855ccf > --- /dev/null > +++ b/support/download/gh > @@ -0,0 +1,43 @@ > +#!/usr/bin/env bash > + > +# We want to catch any unexpected failure, and exit immediately > +set -e > + > +# Download helper for GitHub CLI, to be called from the download wrapper script > +# > +# Options: > +# -o FILE Save into file FILE. > +# -f FILENAME The filename of the tarball to get at URL > +# -u URL Download file at URL. > +# > +# Environment: > +# GH : the gh command to call > + > +quiet= > +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do > + case "${OPT}" in > + o) output="${OPTARG}";; > + f) filename="${OPTARG}";; > + u) url="${OPTARG}";; > + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; > + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; > + esac > +done > + > +shift $((OPTIND-1)) # Get rid of our options > + > +# Caller needs to single-quote its arguments to prevent them from > +# being expanded a second time (in case there are spaces in them) > +_gh() { > + if [ -z "${quiet}" ]; then > + printf '%s ' ${GH} "${@}"; printf '\n' > + fi > + _plain_gh "$@" > +} > +# Note: please keep command below aligned with what is printed above > +_plain_gh() { > + eval ${GH} "${@}" > +} > + > +_gh api -H "'Accept: application/vnd.github.raw+json'" "${@}" \ > + "'${url}/${filename}'" ">'${output}'"
On Fri, 2025-05-16 at 23:18 +0200, Arnout Vandecappelle wrote: > Having something github specific is not very nice... We should then > also > provide the equivalent for gitlab, bitbucket, ... Thanks for looking into this. There's a tool https://github.com/gruntwork-io/fetch but I haven't given it further consideration because it doesn't seem to be packaged in Fedora or Debian. > > It is intended as an alternative to pulling with the "git" method, > > which > > has the drawback of requiring separate ssh keys > > I guess you mean that you need to generate ssh keys for CI? Other people may have different experiences, but mine is that if you're working within a bigger organization you want to stick to https because otherwise you'll be constantly fighting the admins about getting port 22 unblocked and keeping it that way. > > and resulting in > > different archives depending on the buildroot version, > > How does gh help with that? The tarballs it produces are not > reproducible at > all... The ones we use happen to be the same all the time because we > re-request > them all the time so they're cached, but there's actually no > guarantee for that. I agree that it is an issue, but we have an internal primary set up that keeps copies of the archives indefinitely. Out of curiosity, are you sure you're hitting GitHub that often? Even for rarely used packages cached by sources.buildroot.net? From where I stand, it's more of a hypothetical issue as compared to the practical one that some users are adding packages with "br1" hashes to our shared external while others add "git4" hashes. > > and passing an > > authentication token on the wget command line, which risks leaking > > the > > token to third-party servers. > > Well, that's not really a good reason, there are better ways to do > that. I've > created a documentation patch [1] to explain it and put you in Cc. Good addition to the documentation, however I've already used all those tricks and still here we are. The .netrc method unfortunately does not work because the forges generally require a token header instead of http basic authentication. And because GitHub might change things in the future, going with the official cli tool seems like a relatively safe bet. Regards, Stefan > [1] > https://patchwork.ozlabs.org/project/buildroot/patch/20250516211429.763211-1-arnout@rnout.be/ > > > > > > > Signed-off-by: Stefan Nickl <Stefan.Nickl@gmail.com> > > --- > > Config.in | 4 ++++ > > package/pkg-download.mk | 1 + > > support/download/dl-wrapper | 2 +- > > support/download/gh | 43 > > +++++++++++++++++++++++++++++++++++++ > > 4 files changed, 49 insertions(+), 1 deletion(-) > > create mode 100755 support/download/gh > > > > diff --git a/Config.in b/Config.in > > index d730f2034b..cb178ee8d2 100644 > > --- a/Config.in > > +++ b/Config.in > > @@ -172,6 +172,10 @@ config BR2_HG > > string "Mercurial (hg) command" > > default "hg" > > > > +config BR2_GH > > + string "GitHub cli command" > > + default "gh" > > + > > config BR2_ZCAT > > string "zcat command" > > default "gzip -d -c" > > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > > index cf5959ea95..08b0d420c7 100644 > > --- a/package/pkg-download.mk > > +++ b/package/pkg-download.mk > > @@ -111,6 +111,7 @@ define DOWNLOAD > > CVS="$(call qstrip,$(BR2_CVS))" \ > > GIT="$(call qstrip,$(BR2_GIT))" \ > > HG="$(call qstrip,$(BR2_HG))" \ > > + GH="$(call qstrip,$(BR2_GH))" \ > > LOCALFILES="$(call qstrip,$(BR2_LOCALFILES))" \ > > SCP="$(call qstrip,$(BR2_SCP))" \ > > SFTP="$(call qstrip,$(BR2_SFTP))" \ > > diff --git a/support/download/dl-wrapper b/support/download/dl- > > wrapper > > index 5445aad5a7..a209665b0f 100755 > > --- a/support/download/dl-wrapper > > +++ b/support/download/dl-wrapper > > @@ -95,7 +95,7 @@ main() { > > backend_urlencode="${uri%%+*}" > > backend="${backend_urlencode%|*}" > > case "${backend}" in > > - git|svn|cvs|bzr|file|scp|hg|sftp) ;; > > + git|svn|cvs|bzr|file|scp|hg|gh|sftp) ;; > > ftp|ftps) backend="curl" ;; > > *) backend="wget" ;; > > esac > > diff --git a/support/download/gh b/support/download/gh > > new file mode 100755 > > index 0000000000..5c36855ccf > > --- /dev/null > > +++ b/support/download/gh > > @@ -0,0 +1,43 @@ > > +#!/usr/bin/env bash > > + > > +# We want to catch any unexpected failure, and exit immediately > > +set -e > > + > > +# Download helper for GitHub CLI, to be called from the download > > wrapper script > > +# > > +# Options: > > +# -o FILE Save into file FILE. > > +# -f FILENAME The filename of the tarball to get at URL > > +# -u URL Download file at URL. > > +# > > +# Environment: > > +# GH : the gh command to call > > + > > +quiet= > > +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do > > + case "${OPT}" in > > + o) output="${OPTARG}";; > > + f) filename="${OPTARG}";; > > + u) url="${OPTARG}";; > > + :) printf "option '%s' expects a mandatory argument\n" > > "${OPTARG}"; exit 1;; > > + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; > > + esac > > +done > > + > > +shift $((OPTIND-1)) # Get rid of our options > > + > > +# Caller needs to single-quote its arguments to prevent them from > > +# being expanded a second time (in case there are spaces in them) > > +_gh() { > > + if [ -z "${quiet}" ]; then > > + printf '%s ' ${GH} "${@}"; printf '\n' > > + fi > > + _plain_gh "$@" > > +} > > +# Note: please keep command below aligned with what is printed > > above > > +_plain_gh() { > > + eval ${GH} "${@}" > > +} > > + > > +_gh api -H "'Accept: application/vnd.github.raw+json'" "${@}" \ > > + "'${url}/${filename}'" ">'${output}'"
diff --git a/Config.in b/Config.in index d730f2034b..cb178ee8d2 100644 --- a/Config.in +++ b/Config.in @@ -172,6 +172,10 @@ config BR2_HG string "Mercurial (hg) command" default "hg" +config BR2_GH + string "GitHub cli command" + default "gh" + config BR2_ZCAT string "zcat command" default "gzip -d -c" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index cf5959ea95..08b0d420c7 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -111,6 +111,7 @@ define DOWNLOAD CVS="$(call qstrip,$(BR2_CVS))" \ GIT="$(call qstrip,$(BR2_GIT))" \ HG="$(call qstrip,$(BR2_HG))" \ + GH="$(call qstrip,$(BR2_GH))" \ LOCALFILES="$(call qstrip,$(BR2_LOCALFILES))" \ SCP="$(call qstrip,$(BR2_SCP))" \ SFTP="$(call qstrip,$(BR2_SFTP))" \ diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 5445aad5a7..a209665b0f 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -95,7 +95,7 @@ main() { backend_urlencode="${uri%%+*}" backend="${backend_urlencode%|*}" case "${backend}" in - git|svn|cvs|bzr|file|scp|hg|sftp) ;; + git|svn|cvs|bzr|file|scp|hg|gh|sftp) ;; ftp|ftps) backend="curl" ;; *) backend="wget" ;; esac diff --git a/support/download/gh b/support/download/gh new file mode 100755 index 0000000000..5c36855ccf --- /dev/null +++ b/support/download/gh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# We want to catch any unexpected failure, and exit immediately +set -e + +# Download helper for GitHub CLI, to be called from the download wrapper script +# +# Options: +# -o FILE Save into file FILE. +# -f FILENAME The filename of the tarball to get at URL +# -u URL Download file at URL. +# +# Environment: +# GH : the gh command to call + +quiet= +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + f) filename="${OPTARG}";; + u) url="${OPTARG}";; + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done + +shift $((OPTIND-1)) # Get rid of our options + +# Caller needs to single-quote its arguments to prevent them from +# being expanded a second time (in case there are spaces in them) +_gh() { + if [ -z "${quiet}" ]; then + printf '%s ' ${GH} "${@}"; printf '\n' + fi + _plain_gh "$@" +} +# Note: please keep command below aligned with what is printed above +_plain_gh() { + eval ${GH} "${@}" +} + +_gh api -H "'Accept: application/vnd.github.raw+json'" "${@}" \ + "'${url}/${filename}'" ">'${output}'"