diff mbox series

[ovs-dev,1/2] travis: Retry kernel download on 503 first byte timeout.

Message ID 20190520132939.14196-2-i.maximets@samsung.com
State Accepted
Headers show
Series travis: CDN download failures + latest stable kernels. | expand

Commit Message

Ilya Maximets May 20, 2019, 1:29 p.m. UTC 
Sometimes it takes to long for CDN to reply in case of downloading
of not frequently used kernels.
For example, even on my local PC it fails to download linux-4.19.29
at the first try:

  $ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.29.tar.xz
  Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176
  Connecting to cdn.kernel.org |151.101.1.176|:443... connected.
  HTTP request sent, awaiting response... 503 first byte timeout

Example from TravisCI:
  https://travis-ci.org/openvswitch/ovs/jobs/530247589

It seems that CDN downloads the tar for that time to the nearby
server and instant retry usually succeeds immediately.

503 is not a "fatal error" for wget and, unfortunately, wget on
TravisCI is too old and we can't just use "--retry-on-http-error=503"
to avoid failures in this case. So, let's just retry unconditionally.
Fallback to the direct download if CDN fails twice.

Fixes: ae6e4f12fcab ("travis: Speed up linux kernel downloads.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
 .travis/linux-build.sh | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Comments

Ben Pfaff May 20, 2019, 11:13 p.m. UTC | #1 
On Mon, May 20, 2019 at 04:29:38PM +0300, Ilya Maximets wrote:
> Sometimes it takes to long for CDN to reply in case of downloading
> of not frequently used kernels.
> For example, even on my local PC it fails to download linux-4.19.29
> at the first try:
> 
>   $ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.29.tar.xz
>   Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176
>   Connecting to cdn.kernel.org |151.101.1.176|:443... connected.
>   HTTP request sent, awaiting response... 503 first byte timeout
> 
> Example from TravisCI:
>   https://travis-ci.org/openvswitch/ovs/jobs/530247589
> 
> It seems that CDN downloads the tar for that time to the nearby
> server and instant retry usually succeeds immediately.
> 
> 503 is not a "fatal error" for wget and, unfortunately, wget on
> TravisCI is too old and we can't just use "--retry-on-http-error=503"
> to avoid failures in this case. So, let's just retry unconditionally.
> Fallback to the direct download if CDN fails twice.
> 
> Fixes: ae6e4f12fcab ("travis: Speed up linux kernel downloads.")
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
>  .travis/linux-build.sh | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
> index d869713f7..c8c9748ac 100755
> --- a/.travis/linux-build.sh
> +++ b/.travis/linux-build.sh
> @@ -19,7 +19,11 @@ function install_kernel()
>          PREFIX="v2.6/longterm/v2.6.32"
>      fi
>  
> -    wget https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
> +    cdn="https://cdn."
> +    direct="https://www."
> +    link="kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz"
> +    # Download kernel sources. Try direct link on CDN failure.
> +    wget ${cdn}${link} || wget ${cdn}${link} || wget ${direct}${link}
>      tar xvf linux-${1}.tar.xz > /dev/null
>      cd linux-${1}
>      make allmodconfig

Acked-by: Ben Pfaff <blp@ovn.org>

Since this is a bash-only script, the bash hacker in me wants to write
it something like:

url=https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
wget $url || wget $url || wget ${url/cdn/www}

Maybe that is too clever though.
Ilya Maximets May 21, 2019, 12:15 p.m. UTC | #2 
On 21.05.2019 2:13, Ben Pfaff wrote:
> On Mon, May 20, 2019 at 04:29:38PM +0300, Ilya Maximets wrote:
>> Sometimes it takes to long for CDN to reply in case of downloading
>> of not frequently used kernels.
>> For example, even on my local PC it fails to download linux-4.19.29
>> at the first try:
>>
>>   $ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.29.tar.xz
>>   Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176
>>   Connecting to cdn.kernel.org |151.101.1.176|:443... connected.
>>   HTTP request sent, awaiting response... 503 first byte timeout
>>
>> Example from TravisCI:
>>   https://travis-ci.org/openvswitch/ovs/jobs/530247589
>>
>> It seems that CDN downloads the tar for that time to the nearby
>> server and instant retry usually succeeds immediately.
>>
>> 503 is not a "fatal error" for wget and, unfortunately, wget on
>> TravisCI is too old and we can't just use "--retry-on-http-error=503"
>> to avoid failures in this case. So, let's just retry unconditionally.
>> Fallback to the direct download if CDN fails twice.
>>
>> Fixes: ae6e4f12fcab ("travis: Speed up linux kernel downloads.")
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>  .travis/linux-build.sh | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
>> index d869713f7..c8c9748ac 100755
>> --- a/.travis/linux-build.sh
>> +++ b/.travis/linux-build.sh
>> @@ -19,7 +19,11 @@ function install_kernel()
>>          PREFIX="v2.6/longterm/v2.6.32"
>>      fi
>>  
>> -    wget https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
>> +    cdn="https://cdn."
>> +    direct="https://www."
>> +    link="kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz"
>> +    # Download kernel sources. Try direct link on CDN failure.
>> +    wget ${cdn}${link} || wget ${cdn}${link} || wget ${direct}${link}
>>      tar xvf linux-${1}.tar.xz > /dev/null
>>      cd linux-${1}
>>      make allmodconfig
> 
> Acked-by: Ben Pfaff <blp@ovn.org>
> 
> Since this is a bash-only script, the bash hacker in me wants to write
> it something like:
> 
> url=https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
> wget $url || wget $url || wget ${url/cdn/www}
> 
> Maybe that is too clever though.

I like your version. It's shorter and doesn't require additional variables.
I updated the patch with that suggestion and pushed to master. Thanks.

Best regards, Ilya Maximets.
Ben Pfaff May 21, 2019, 3:59 p.m. UTC | #3 
On Tue, May 21, 2019 at 03:15:17PM +0300, Ilya Maximets wrote:
> On 21.05.2019 2:13, Ben Pfaff wrote:
> > On Mon, May 20, 2019 at 04:29:38PM +0300, Ilya Maximets wrote:
> >> Sometimes it takes to long for CDN to reply in case of downloading
> >> of not frequently used kernels.
> >> For example, even on my local PC it fails to download linux-4.19.29
> >> at the first try:
> >>
> >>   $ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.29.tar.xz
> >>   Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.1.176
> >>   Connecting to cdn.kernel.org |151.101.1.176|:443... connected.
> >>   HTTP request sent, awaiting response... 503 first byte timeout
> >>
> >> Example from TravisCI:
> >>   https://travis-ci.org/openvswitch/ovs/jobs/530247589
> >>
> >> It seems that CDN downloads the tar for that time to the nearby
> >> server and instant retry usually succeeds immediately.
> >>
> >> 503 is not a "fatal error" for wget and, unfortunately, wget on
> >> TravisCI is too old and we can't just use "--retry-on-http-error=503"
> >> to avoid failures in this case. So, let's just retry unconditionally.
> >> Fallback to the direct download if CDN fails twice.
> >>
> >> Fixes: ae6e4f12fcab ("travis: Speed up linux kernel downloads.")
> >> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >> ---
> >>  .travis/linux-build.sh | 6 +++++-
> >>  1 file changed, 5 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
> >> index d869713f7..c8c9748ac 100755
> >> --- a/.travis/linux-build.sh
> >> +++ b/.travis/linux-build.sh
> >> @@ -19,7 +19,11 @@ function install_kernel()
> >>          PREFIX="v2.6/longterm/v2.6.32"
> >>      fi
> >>  
> >> -    wget https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
> >> +    cdn="https://cdn."
> >> +    direct="https://www."
> >> +    link="kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz"
> >> +    # Download kernel sources. Try direct link on CDN failure.
> >> +    wget ${cdn}${link} || wget ${cdn}${link} || wget ${direct}${link}
> >>      tar xvf linux-${1}.tar.xz > /dev/null
> >>      cd linux-${1}
> >>      make allmodconfig
> > 
> > Acked-by: Ben Pfaff <blp@ovn.org>
> > 
> > Since this is a bash-only script, the bash hacker in me wants to write
> > it something like:
> > 
> > url=https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
> > wget $url || wget $url || wget ${url/cdn/www}
> > 
> > Maybe that is too clever though.
> 
> I like your version. It's shorter and doesn't require additional variables.
> I updated the patch with that suggestion and pushed to master. Thanks.

OK, now I feel clever in a good way ;-)
diff mbox series

Patch

diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
index d869713f7..c8c9748ac 100755
--- a/.travis/linux-build.sh
+++ b/.travis/linux-build.sh
@@ -19,7 +19,11 @@  function install_kernel()
         PREFIX="v2.6/longterm/v2.6.32"
     fi
 
-    wget https://cdn.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz
+    cdn="https://cdn."
+    direct="https://www."
+    link="kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.xz"
+    # Download kernel sources. Try direct link on CDN failure.
+    wget ${cdn}${link} || wget ${cdn}${link} || wget ${direct}${link}
     tar xvf linux-${1}.tar.xz > /dev/null
     cd linux-${1}
     make allmodconfig