| Message ID | 20210930231424.249681-1-giulio.benetti@benettiengineering.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | [buildroot-test] scripts/autobuild-run: extend timeout from 120 to 180 minutes | expand |
Hello Giulio, On Fri, 1 Oct 2021 01:14:24 +0200 Giulio Benetti <giulio.benetti@benettiengineering.com> wrote: > During the last months some package started to timeout, but this is not due > to toolchain infinite loop. I think it's due to the increase of the source > code files' length and to complexity of toolchains or even slow compiling > time. For example python-uvloop and python-lxml only take long time to > build with -Os optimization and this makes it seem a TIMEOUT but it's not. > So let's extend timeout setting from 120 to 180 minutes to avoid false > TIMEOUTs. > > Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com> Do you have a specific example where you have seen a failure? Please note that this HUNG_BUILD_TIMEOUT value is the per-step timeout: it verifies that one step of one particular package doesn't take more than 120 minutes. Unless you have a specific example where you have seen this timeout causing false positives, I'm not really keen on applying such a patch as it would cause situations where the build is really stuck to be stuck for 3 hours instead of 2. Did you identify that the python-uvloop and python-lxml packages really took more than 2 hours to build? I did see at http://autobuild.buildroot.net/?status=TIMEOUT those timeouts dating back from ~November. Did you reproduce the issue, and verified that the build indeed completes at some point? In general, if a simple package such as python-uvloop or python-lxml takes more than 2 hours to build, it means that the compiler (or linker) has gone crazy in an infinite loop, and the build will in fact never finish. Best regards, Thomas
Hi Thomas, On 01/01/22 11:50, Thomas Petazzoni wrote: > Hello Giulio, > > On Fri, 1 Oct 2021 01:14:24 +0200 > Giulio Benetti <giulio.benetti@benettiengineering.com> wrote: > >> During the last months some package started to timeout, but this is not due >> to toolchain infinite loop. I think it's due to the increase of the source >> code files' length and to complexity of toolchains or even slow compiling >> time. For example python-uvloop and python-lxml only take long time to >> build with -Os optimization and this makes it seem a TIMEOUT but it's not. >> So let's extend timeout setting from 120 to 180 minutes to avoid false >> TIMEOUTs. >> >> Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com> > > Do you have a specific example where you have seen a failure? Not anymore, it's been only in September. 2 days ago I was right thinking about dropping this patch when checking patchwork. > Please > note that this HUNG_BUILD_TIMEOUT value is the per-step timeout: it > verifies that one step of one particular package doesn't take more than > 120 minutes. Ah, I thought the entire build, but that makes no sense since some build can take way more than 2 hours. > Unless you have a specific example where you have seen this timeout > causing false positives, I'm not really keen on applying such a patch > as it would cause situations where the build is really stuck to be > stuck for 3 hours instead of 2. Yes, I see you've already marked it as rejected. > Did you identify that the python-uvloop and python-lxml packages really > took more than 2 hours to build? I did see at > http://autobuild.buildroot.net/?status=TIMEOUT those timeouts dating > back from ~November. Did you reproduce the issue, and verified that the > build indeed completes at some point? In general, if a simple package > such as python-uvloop or python-lxml takes more than 2 hours to build, > it means that the compiler (or linker) has gone crazy in an infinite > loop, and the build will in fact never finish. It was only my suspicious as specified in commit log. I couldn't reproduce it and I thought it was due to the autobuilder's host. And yes, now that I know it's a per-step timeout it doesn't make sense at all for such little packages. Thank you Best regards
Hello, On Sat, 1 Jan 2022 16:10:09 +0100 Giulio Benetti <giulio.benetti@benettiengineering.com> wrote: > > Please > > note that this HUNG_BUILD_TIMEOUT value is the per-step timeout: it > > verifies that one step of one particular package doesn't take more than > > 120 minutes. > > Ah, I thought the entire build, but that makes no sense since some build > can take way more than 2 hours. There used to be a global timeout for the build, but we had false positives when really large configurations were built. Due to this, commit 86892c63f754b4f6c9a4a517fde30160127b3506 change to a more fine grained timeout: we monitor the file build-time.log (which gets appended everytime the step of a package starts or ends), and we time out after HUNG_BUILD_TIMEOUT minutes if that file hasn't changed; > > Did you identify that the python-uvloop and python-lxml packages really > > took more than 2 hours to build? I did see at > > http://autobuild.buildroot.net/?status=TIMEOUT those timeouts dating > > back from ~November. Did you reproduce the issue, and verified that the > > build indeed completes at some point? In general, if a simple package > > such as python-uvloop or python-lxml takes more than 2 hours to build, > > it means that the compiler (or linker) has gone crazy in an infinite > > loop, and the build will in fact never finish. > > It was only my suspicious as specified in commit log. I couldn't > reproduce it and I thought it was due to the autobuilder's host. > > And yes, now that I know it's a per-step timeout it doesn't make sense > at all for such little packages. Yes, for such small packages, it clearly hints at a compiler/linker bug that stays stuck in an infinite loop. Thomas
On 01/01/22 21:09, Thomas Petazzoni wrote: > Hello, > > On Sat, 1 Jan 2022 16:10:09 +0100 > Giulio Benetti <giulio.benetti@benettiengineering.com> wrote: > >>> Please >>> note that this HUNG_BUILD_TIMEOUT value is the per-step timeout: it >>> verifies that one step of one particular package doesn't take more than >>> 120 minutes. >> >> Ah, I thought the entire build, but that makes no sense since some build >> can take way more than 2 hours. > > There used to be a global timeout for the build, but we had false > positives when really large configurations were built. Due to this, > commit 86892c63f754b4f6c9a4a517fde30160127b3506 change to a more fine > grained timeout: we monitor the file build-time.log (which gets > appended everytime the step of a package starts or ends), and we time > out after HUNG_BUILD_TIMEOUT minutes if that file hasn't changed; Thank you for the explanation >>> Did you identify that the python-uvloop and python-lxml packages really >>> took more than 2 hours to build? I did see at >>> http://autobuild.buildroot.net/?status=TIMEOUT those timeouts dating >>> back from ~November. Did you reproduce the issue, and verified that the >>> build indeed completes at some point? In general, if a simple package >>> such as python-uvloop or python-lxml takes more than 2 hours to build, >>> it means that the compiler (or linker) has gone crazy in an infinite >>> loop, and the build will in fact never finish. >> >> It was only my suspicious as specified in commit log. I couldn't >> reproduce it and I thought it was due to the autobuilder's host. >> >> And yes, now that I know it's a per-step timeout it doesn't make sense >> at all for such little packages. > > Yes, for such small packages, it clearly hints at a compiler/linker bug > that stays stuck in an infinite loop. Yes, that was the reason I was looking for in the beginning :-) Best regards
diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 346928f..0636368 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -176,7 +176,7 @@ else: # A number of packages can take > 60mins of build time (e.g # gst-ffmpeg, qt5webkit, mimic) -HUNG_BUILD_TIMEOUT = 120 # mins +HUNG_BUILD_TIMEOUT = 180 # mins VERSION = 1 def rm_ro(f, p, _):
During the last months some package started to timeout, but this is not due to toolchain infinite loop. I think it's due to the increase of the source code files' length and to complexity of toolchains or even slow compiling time. For example python-uvloop and python-lxml only take long time to build with -Os optimization and this makes it seem a TIMEOUT but it's not. So let's extend timeout setting from 120 to 180 minutes to avoid false TIMEOUTs. Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com> --- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)