diff mbox series

[RFC,1/3] download/git: fix fetch all refs for old git

Message ID 20180412092855.30621-2-ricardo.martincoski@gmail.com
State Changes Requested
Headers show
Series fix some corner cases for download/git v1 | expand

Commit Message

Ricardo Martincoski April 12, 2018, 9:28 a.m. UTC
With old versions of git, when the shallow fetch was not enough, the
download fails like this:
 Fetching all references
 Could not fetch special ref 'sha1'; assuming it is not special.
 fatal: reference is not a tree: sha1

At git version 1.9.0, the semantics of the option -t for git fetch
changed, see upstream commit [1]:
< 1.9.0: "fetch tags without also fetching other references";
>=1.9.0: "fetch tags *in addition to* other stuff".

Fall back to explicit refspec to support distros that use ancient
versions of git.

Fixes:
http://autobuild.buildroot.net/results/7a88d1aa9ea155f1527d2fa141207c676af85298

[1] https://github.com/git/git/commit/c5a84e92a2fe9e8748e32341c344d7a6c0f52a50

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
WARNING: please review carefully, especially related to the plus sign.

I created a special docker image replacing git 2.11.0 with 1.8.3 [2].
Using it together to a special .gitlab-ci.yml [3], I could test in
Gitlab CI the download, without previous git cache and without backup
site, of all packages in the tree that use site method git in these
combinations:
- [4] patch above master (20c1647f) using git 1.8.3;
- [5] patch above master (20c1647f) using git 2.11.0;
- [6] master (20c1647f) using git 1.8.3;
- [7] master (20c1647f) using git 2.11.0;
- [8] 2018.02.1 using git 1.8.3;
- [9] 2018.02.1 using git 2.11.0.

These packages fail with 'fatal: reference is not a tree' in the master
branch using git 1.8.3 but do not fail for the other 5 combinations above:
aer-inject
armbian-firmware
boot-wrapper-aarch64
dbus-triggerd
dropwatch
edid-decode
expedite
flashbench
gst1-interpipe
host-mxsldr
host-pseudo
host-squashfs
libbroadvoice
libg7221
libilbc
libsilk
libsoundtouch
libubox
libuci
libyuv
linux-firmware
linux@beaglebone_qt5_defconfig
ltrace
mmc-utils
net-tools
open-lldp
psplash
rtmpdump
slirp
squashfs
ti-sgx-demos
ti-sgx-km
ti-sgx-um
uboot@ci20_defconfig
uboot@roseapplepi_defconfig
ubus
uclibc-ng-test
uhttpd
ustream-ssl
x264
xdriver_xf86-video-intel
xdriver_xf86-video-voodoo
yavta

[2] https://gitlab.com/RicardoMartincoski/buildroot/commit/d293ba50fa8cd40eb5c396b2aa234c07e2118020
[3] http://patchwork.ozlabs.org/patch/896012/
[4] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20328681
[5] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20328707
[6] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20328727
[7] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20328745
[8] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20352543
[9] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/20356539
---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Yann E. MORIN April 12, 2018, 5:42 p.m. UTC | #1
Ricardo, All,

On 2018-04-12 06:28 -0300, Ricardo Martincoski spake thusly:
> With old versions of git, when the shallow fetch was not enough, the
> download fails like this:
>  Fetching all references
>  Could not fetch special ref 'sha1'; assuming it is not special.
>  fatal: reference is not a tree: sha1
> 
> At git version 1.9.0, the semantics of the option -t for git fetch
> changed, see upstream commit [1]:
> < 1.9.0: "fetch tags without also fetching other references";
> >=1.9.0: "fetch tags *in addition to* other stuff".
> 
> Fall back to explicit refspec to support distros that use ancient
> versions of git.
[--SNIP--]
> diff --git a/support/download/git b/support/download/git
> index 381f3ceeb3..1b09e5c750 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -76,7 +76,7 @@ if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then
>  fi
>  if [ ${git_done} -eq 0 ]; then
>      printf "Fetching all references\n"
> -    _git fetch origin -t
> +    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"

Why don't we just run:

    git fetch origin
    git fetch origin -t

As I understand it:

                    ancient git         | new git
--------------------------------------------------------------------
git fetch     | fetch all refs but tags | fetches all refs but tags
git fetch -t  | fetches only tags       | fetch all refs and tags

Right?

Regards,
Yann E. MORIN.

>  fi
>  
>  # Try to get the special refs exposed by some forges (pull-requests for
> -- 
> 2.14.1
>
Ricardo Martincoski April 13, 2018, 6:23 p.m. UTC | #2
Hello,

Thank you for your review of the series.

On Thu, Apr 12, 2018 at 02:42 PM, Yann E. MORIN wrote:

>> -    _git fetch origin -t
>> +    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"
> 
> Why don't we just run:
> 
>     git fetch origin
>     git fetch origin -t

One extra connection to the server ...
but much more readable/maintainable!
I will do. I will change this patch to Changes Requested.

> As I understand it:
> 
>                     ancient git         | new git
> --------------------------------------------------------------------
> git fetch     | fetch all refs but tags | fetches all refs but tags
> git fetch -t  | fetches only tags       | fetch all refs and tags
> 
> Right?

Yes.
Nit: actually 'git fetch' does fetch some tags, but not necessarily all. Only
tags in commits also reachable by branches are fetched.


Regards,
Ricardo
Yann E. MORIN April 15, 2018, 12:12 p.m. UTC | #3
Ricardo, All,

On 2018-04-13 15:23 -0300, Ricardo Martincoski spake thusly:
> On Thu, Apr 12, 2018 at 02:42 PM, Yann E. MORIN wrote:
> >> -    _git fetch origin -t
> >> +    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"
> > Why don't we just run:
> >     git fetch origin
> >     git fetch origin -t
> 
> One extra connection to the server ...
> but much more readable/maintainable!
> I will do. I will change this patch to Changes Requested.

Thanks. I don't care much about an extra round-trip to the server,
especially if that makes the code easier to read.

> > As I understand it:
> >                     ancient git         | new git
> > --------------------------------------------------------------------
> > git fetch     | fetch all refs but tags | fetches all refs but tags
> > git fetch -t  | fetches only tags       | fetch all refs and tags
> Nit: actually 'git fetch' does fetch some tags, but not necessarily all. Only
> tags in commits also reachable by branches are fetched.

OK, but overall, we would cover all cases with the two git-fetch,
whether we use an old git or a newer one, and that's the important
info.

Thanks! :-)

Regards,
Yann E. MORIN.
Ricardo Martincoski April 16, 2018, 2:17 a.m. UTC | #4
Hello,

> On 2018-04-13 15:23 -0300, Ricardo Martincoski spake thusly:
>> On Thu, Apr 12, 2018 at 02:42 PM, Yann E. MORIN wrote:
>> >> -    _git fetch origin -t
>> >> +    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"
>> > Why don't we just run:
>> >     git fetch origin
>> >     git fetch origin -t
>> 
>> One extra connection to the server ...
>> but much more readable/maintainable!
>> I will do. I will change this patch to Changes Requested.
> 
> Thanks. I don't care much about an extra round-trip to the server,
> especially if that makes the code easier to read.

I agree.
And a few years from now, when we stop supporting distros with git <1.9.0 we
can remove 'git fetch origin' if we wish.

BTW, the order of the commands is important here. Some dumb http servers don't
work well with old git versions if 'git fetch origin -t' is used first, see:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/62895510
But in the order you propose (and I see you used in your series) it works fine:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/62897927
So we are good.

> OK, but overall, we would cover all cases with the two git-fetch,
> whether we use an old git or a newer one, and that's the important
> info.

Agreed.


Regards,
Ricardo
diff mbox series

Patch

diff --git a/support/download/git b/support/download/git
index 381f3ceeb3..1b09e5c750 100755
--- a/support/download/git
+++ b/support/download/git
@@ -76,7 +76,7 @@  if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Fetching all references\n"
-    _git fetch origin -t
+    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"
 fi
 
 # Try to get the special refs exposed by some forges (pull-requests for