[ovs-dev] travis: Fix kernel download retry.
diff mbox series

Message ID 20200319073240.32718-1-david.marchand@redhat.com
State New
Headers show
Series
  • [ovs-dev] travis: Fix kernel download retry.
Related show

Commit Message

David Marchand March 19, 2020, 7:32 a.m. UTC
wget stops retrying to download a file when hitting fatal http errors
like 503.
But if a previous try had resulted in a partially downloaded ${file}, the
next wget call tries to download to ${file}.1.

Example:
+wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
--2020-03-18 20:51:42--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 103076276 (98M) [application/x-xz]
Saving to: ‘linux-4.16.18.tar.xz’

linux-4.16.18.tar.x   0%[                    ]  13.07K  --.-KB/s    in 0s

2020-03-18 20:54:44 (133 MB/s) - Read error at byte 13383/103076276 (Connection reset by peer). Retrying.

--2020-03-18 20:54:45--  (try: 2)  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
HTTP request sent, awaiting response... 503 first byte timeout
2020-03-18 20:55:46 ERROR 503: first byte timeout.

+wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
--2020-03-18 20:55:46--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 103076276 (98M) [application/x-xz]
Saving to: ‘linux-4.16.18.tar.xz.1’

linux-4.16.18.tar.x 100%[===================>]  98.30M   186MB/s    in 0.5s

2020-03-18 20:55:56 (186 MB/s) - ‘linux-4.16.18.tar.xz.1’ saved [103076276/103076276]

Fixes: 048674b45f4b ("travis: Retry kernel download on 503 first byte timeout.")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .travis/linux-build.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comments

Ilya Maximets March 19, 2020, 11:44 a.m. UTC | #1
On 3/19/20 8:32 AM, David Marchand wrote:
> wget stops retrying to download a file when hitting fatal http errors
> like 503.
> But if a previous try had resulted in a partially downloaded ${file}, the
> next wget call tries to download to ${file}.1.
> 
> Example:
> +wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> --2020-03-18 20:51:42--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
> Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 103076276 (98M) [application/x-xz]
> Saving to: ‘linux-4.16.18.tar.xz’
> 
> linux-4.16.18.tar.x   0%[                    ]  13.07K  --.-KB/s    in 0s
> 
> 2020-03-18 20:54:44 (133 MB/s) - Read error at byte 13383/103076276 (Connection reset by peer). Retrying.
> 
> --2020-03-18 20:54:45--  (try: 2)  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> HTTP request sent, awaiting response... 503 first byte timeout
> 2020-03-18 20:55:46 ERROR 503: first byte timeout.
> 
> +wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> --2020-03-18 20:55:46--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
> Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 103076276 (98M) [application/x-xz]
> Saving to: ‘linux-4.16.18.tar.xz.1’
> 
> linux-4.16.18.tar.x 100%[===================>]  98.30M   186MB/s    in 0.5s
> 
> 2020-03-18 20:55:56 (186 MB/s) - ‘linux-4.16.18.tar.xz.1’ saved [103076276/103076276]
> 
> Fixes: 048674b45f4b ("travis: Retry kernel download on 503 first byte timeout.")
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  .travis/linux-build.sh | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
> index 359f7773ba..74e1028573 100755
> --- a/.travis/linux-build.sh
> +++ b/.travis/linux-build.sh
> @@ -35,7 +35,9 @@ function install_kernel()
>  
>      url="${base_url}/linux-${version}.tar.xz"
>      # Download kernel sources. Try direct link on CDN failure.
> -    wget ${url} || wget ${url} || wget ${url/cdn/www}
> +    wget ${url} ||
> +    (rm -f linux-${version}.tar.xz && wget ${url}) ||
> +    (rm -f linux-${version}.tar.xz && wget ${url/cdn/www})

How about using 'wget -c' instead?
David Marchand March 19, 2020, 12:01 p.m. UTC | #2
On Thu, Mar 19, 2020 at 12:44 PM Ilya Maximets <i.maximets@ovn.org> wrote:
>
> On 3/19/20 8:32 AM, David Marchand wrote:
> > wget stops retrying to download a file when hitting fatal http errors
> > like 503.
> > But if a previous try had resulted in a partially downloaded ${file}, the
> > next wget call tries to download to ${file}.1.
> >
> > Example:
> > +wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> > --2020-03-18 20:51:42--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> > Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
> > Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 103076276 (98M) [application/x-xz]
> > Saving to: ‘linux-4.16.18.tar.xz’
> >
> > linux-4.16.18.tar.x   0%[                    ]  13.07K  --.-KB/s    in 0s
> >
> > 2020-03-18 20:54:44 (133 MB/s) - Read error at byte 13383/103076276 (Connection reset by peer). Retrying.
> >
> > --2020-03-18 20:54:45--  (try: 2)  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> > Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> > HTTP request sent, awaiting response... 503 first byte timeout
> > 2020-03-18 20:55:46 ERROR 503: first byte timeout.
> >
> > +wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> > --2020-03-18 20:55:46--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.18.tar.xz
> > Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176, 151.101.65.176, 151.101.129.176, ...
> > Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 103076276 (98M) [application/x-xz]
> > Saving to: ‘linux-4.16.18.tar.xz.1’
> >
> > linux-4.16.18.tar.x 100%[===================>]  98.30M   186MB/s    in 0.5s
> >
> > 2020-03-18 20:55:56 (186 MB/s) - ‘linux-4.16.18.tar.xz.1’ saved [103076276/103076276]
> >
> > Fixes: 048674b45f4b ("travis: Retry kernel download on 503 first byte timeout.")
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  .travis/linux-build.sh | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
> > index 359f7773ba..74e1028573 100755
> > --- a/.travis/linux-build.sh
> > +++ b/.travis/linux-build.sh
> > @@ -35,7 +35,9 @@ function install_kernel()
> >
> >      url="${base_url}/linux-${version}.tar.xz"
> >      # Download kernel sources. Try direct link on CDN failure.
> > -    wget ${url} || wget ${url} || wget ${url/cdn/www}
> > +    wget ${url} ||
> > +    (rm -f linux-${version}.tar.xz && wget ${url}) ||
> > +    (rm -f linux-${version}.tar.xz && wget ${url/cdn/www})
>
> How about using 'wget -c' instead?

This should work.
Restarting from zero seems safer to me.

Your choice.

Patch
diff mbox series

diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
index 359f7773ba..74e1028573 100755
--- a/.travis/linux-build.sh
+++ b/.travis/linux-build.sh
@@ -35,7 +35,9 @@  function install_kernel()
 
     url="${base_url}/linux-${version}.tar.xz"
     # Download kernel sources. Try direct link on CDN failure.
-    wget ${url} || wget ${url} || wget ${url/cdn/www}
+    wget ${url} ||
+    (rm -f linux-${version}.tar.xz && wget ${url}) ||
+    (rm -f linux-${version}.tar.xz && wget ${url/cdn/www})
 
     tar xvf linux-${version}.tar.xz > /dev/null
     pushd linux-${version}