| Message ID | 1471469135-19009-1-git-send-email-kamath.ben@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
Benjmin, All, On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: > From: Benjamin Kamath <bkamath@spaceflight.com> > > Attempt to do a remote archive since it shortcuts us past a few steps when > available. Additionally. if the git server has uploadArchive.allowUnreachable > set to true, then this method can also work on arbitrary sha1s, offering a huge > speed advantage over a full clone. Do you know of a publicly-accessible server that allows remote archives? I've tried with a github-hosted package, but githib won;t allow remote archives, it seems... Regards, Yann E. MORIN. > Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com> > --- > Changes v1 -> v2: > - Skip remote archive for repositories with submodules (suggested by Yann) > > support/download/git | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/support/download/git b/support/download/git > index 416cd1b..d060cbd 100755 > --- a/support/download/git > +++ b/support/download/git > @@ -36,6 +36,17 @@ _git() { > eval ${GIT} "${@}" > } > > +# Try a remote archive, since it is as fast as a shallow clone and can give us > +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true > +# on the remote, this will also work for arbitrary sha1s, and will offer a > +# considerable speedup over a full clone. This will not work for submodules. > +if [ ${recurse} -eq 0 ]; then > + printf "Doing remote archive\n" > + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then > + exit 0 > + fi > +fi > + > # Try a shallow clone, since it is faster than a full clone - but that only > # works if the version is a ref (tag or branch). Before trying to do a shallow > # clone we check if ${cset} is in the list provided by git ls-remote. If not > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Benjmin, All, > > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: >> From: Benjamin Kamath <bkamath@spaceflight.com> >> >> Attempt to do a remote archive since it shortcuts us past a few steps when >> available. Additionally. if the git server has uploadArchive.allowUnreachable >> set to true, then this method can also work on arbitrary sha1s, offering a huge >> speed advantage over a full clone. > > Do you know of a publicly-accessible server that allows remote archives? > > I've tried with a github-hosted package, but githib won;t allow remote > archives, it seems... I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default: git@gitlab.com:gitlab-org/gitlab-ce.git > > Regards, > Yann E. MORIN. > >> Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com> >> --- >> Changes v1 -> v2: >> - Skip remote archive for repositories with submodules (suggested by Yann) >> >> support/download/git | 11 +++++++++++ >> 1 file changed, 11 insertions(+) >> >> diff --git a/support/download/git b/support/download/git >> index 416cd1b..d060cbd 100755 >> --- a/support/download/git >> +++ b/support/download/git >> @@ -36,6 +36,17 @@ _git() { >> eval ${GIT} "${@}" >> } >> >> +# Try a remote archive, since it is as fast as a shallow clone and can give us >> +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true >> +# on the remote, this will also work for arbitrary sha1s, and will offer a >> +# considerable speedup over a full clone. This will not work for submodules. >> +if [ ${recurse} -eq 0 ]; then >> + printf "Doing remote archive\n" >> + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then >> + exit 0 >> + fi >> +fi >> + >> # Try a shallow clone, since it is faster than a full clone - but that only >> # works if the version is a ref (tag or branch). Before trying to do a shallow >> # clone we check if ${cset} is in the list provided by git ls-remote. If not >> -- >> 2.7.4 >> >> _______________________________________________ >> buildroot mailing list >> buildroot@busybox.net >> http://lists.busybox.net/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Benjamin, All, On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly: > On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Benjmin, All, > > > > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: > >> From: Benjamin Kamath <bkamath@spaceflight.com> > >> > >> Attempt to do a remote archive since it shortcuts us past a few steps when > >> available. Additionally. if the git server has uploadArchive.allowUnreachable > >> set to true, then this method can also work on arbitrary sha1s, offering a huge > >> speed advantage over a full clone. > > > > Do you know of a publicly-accessible server that allows remote archives? > > > > I've tried with a github-hosted package, but githib won;t allow remote > > archives, it seems... > I just tried quite a few and it doesn't seem to be frequently > supported. gitlab-ce does seem to have it enabled by default: > > git@gitlab.com:gitlab-org/gitlab-ce.git OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or git:// Which means we have no package in Buildroot that can use this. Do you have a need for that for an internal package? Regards, Yann E. MORIN.
On Thu, Aug 18, 2016 at 2:10 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Benjamin, All, > > On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly: >> On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: >> > Benjmin, All, >> > >> > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: >> >> From: Benjamin Kamath <bkamath@spaceflight.com> >> >> >> >> Attempt to do a remote archive since it shortcuts us past a few steps when >> >> available. Additionally. if the git server has uploadArchive.allowUnreachable >> >> set to true, then this method can also work on arbitrary sha1s, offering a huge >> >> speed advantage over a full clone. >> > >> > Do you know of a publicly-accessible server that allows remote archives? >> > >> > I've tried with a github-hosted package, but githib won;t allow remote >> > archives, it seems... >> I just tried quite a few and it doesn't seem to be frequently >> supported. gitlab-ce does seem to have it enabled by default: >> >> git@gitlab.com:gitlab-org/gitlab-ce.git > > OK, it does work, as long as one uses the ssh protocol. It does not work > via http:// or git:// > OK. I had not noticed that before, and as such that makes this patch much less useful. > Which means we have no package in Buildroot that can use this. > > Do you have a need for that for an internal package? Yes, but we can create a local patch for it. On dev-branches of our project we frequently have internal packages that point directly at sha1s. I mistakenly thought more servers would support remote archive. I suppose we should reject this patch. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Benjamin, All, On 2016-08-18 02:26 -0700, Benjamin Kamath spake thusly: > On Thu, Aug 18, 2016 at 2:10 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Benjamin, All, > > > > On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly: > >> On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > >> > Benjmin, All, > >> > > >> > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: > >> >> From: Benjamin Kamath <bkamath@spaceflight.com> > >> >> > >> >> Attempt to do a remote archive since it shortcuts us past a few steps when > >> >> available. Additionally. if the git server has uploadArchive.allowUnreachable > >> >> set to true, then this method can also work on arbitrary sha1s, offering a huge > >> >> speed advantage over a full clone. > >> > > >> > Do you know of a publicly-accessible server that allows remote archives? > >> > > >> > I've tried with a github-hosted package, but githib won;t allow remote > >> > archives, it seems... > >> I just tried quite a few and it doesn't seem to be frequently > >> supported. gitlab-ce does seem to have it enabled by default: > >> > >> git@gitlab.com:gitlab-org/gitlab-ce.git > > > > OK, it does work, as long as one uses the ssh protocol. It does not work > > via http:// or git:// > > > OK. I had not noticed that before, and as such that makes this patch > much less useful. > > > Which means we have no package in Buildroot that can use this. > > > > Do you have a need for that for an internal package? > Yes, but we can create a local patch for it. On dev-branches of our > project we frequently have internal packages that point directly at > sha1s. I mistakenly thought more servers would support remote archive. > I suppose we should reject this patch. I'm OK with adding this, as long as it does not break existing behaviour. From my little tests, it shows that it works. Regards, Yann E. MORIN.
Benjamin, All, On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly: > From: Benjamin Kamath <bkamath@spaceflight.com> > > Attempt to do a remote archive since it shortcuts us past a few steps when > available. Additionally. if the git server has uploadArchive.allowUnreachable > set to true, then this method can also work on arbitrary sha1s, offering a huge > speed advantage over a full clone. > > Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com> > --- > Changes v1 -> v2: > - Skip remote archive for repositories with submodules (suggested by Yann) > > support/download/git | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/support/download/git b/support/download/git > index 416cd1b..d060cbd 100755 > --- a/support/download/git > +++ b/support/download/git > @@ -36,6 +36,17 @@ _git() { > eval ${GIT} "${@}" > } > > +# Try a remote archive, since it is as fast as a shallow clone and can give us > +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true > +# on the remote, this will also work for arbitrary sha1s, and will offer a > +# considerable speedup over a full clone. This will not work for submodules. > +if [ ${recurse} -eq 0 ]; then > + printf "Doing remote archive\n" > + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then > + exit 0 I would have also done: else printf "Remote archive failed, falling back to cloning\n" Like we do for the shallow clone. Also, why do you redirect stderr? We don't do it in the clone commands. We only do it to fetch the special references (gitHub PRs of Gerrit changes) because it does not make sense to display an spurious message when the reference is in fact not special. Regards, Yann E. MORIN. > + fi > +fi > + > # Try a shallow clone, since it is faster than a full clone - but that only > # works if the version is a ref (tag or branch). Before trying to do a shallow > # clone we check if ${cset} is in the list provided by git ls-remote. If not > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
diff --git a/support/download/git b/support/download/git index 416cd1b..d060cbd 100755 --- a/support/download/git +++ b/support/download/git @@ -36,6 +36,17 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then + exit 0 + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If not