| Message ID | 20221015005611.4054933-1-james.hilliard1@gmail.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [1/1] package/Makefile.in: set .NOTPARALLEL for MAKE1 | expand |
James, All, On 2022-10-14 18:56 -0600, James Hilliard spake thusly: > Make 4.4 introduces a shuffle mode which randomizes prerequisites > in order to better flush out issues with parallel builds, as this > mode randomizes prerequisites even when using -j1 builds we must > instead explicitely mark parallel incompatible builds using the > .NOTPARALLEL special target which disables prerequisites > randomization when running in shuffle mode. > > Signed-off-by: James Hilliard <james.hilliard1@gmail.com> > --- > package/Makefile.in | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/package/Makefile.in b/package/Makefile.in > index 43d214bcbe..2327849a1f 100644 > --- a/package/Makefile.in > +++ b/package/Makefile.in > @@ -17,7 +17,7 @@ else > PARALLEL_JOBS := $(BR2_JLEVEL) > endif > > -MAKE1 := $(HOSTMAKE) -j1 > +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: Packages that do not build in parallel do not because their rules assume the ordering is kept. I don't see how replacing -j1 with --eval .NOTPARALLEL: would cause the ordering to be restored. Instead, I would have expected we do something like: # Only build one job at a time, *and* to not randomise goals and # prerequisites ordering in make 4.4+ MAKE1 := $(HOSTMAKE) -j1 $(if $(findstring --shuffle,$(MAKEFLAGS)),--shuffle=none) Regards, Yann E. MORIN. > override MAKE = $(HOSTMAKE) \ > $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS)) > > -- > 2.34.1 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
On 2022-10-16 18:26 +0200, Yann E. MORIN spake thusly: > James, All, > > On 2022-10-14 18:56 -0600, James Hilliard spake thusly: > > Make 4.4 introduces a shuffle mode which randomizes prerequisites > > in order to better flush out issues with parallel builds, as this > > mode randomizes prerequisites even when using -j1 builds we must > > instead explicitely mark parallel incompatible builds using the > > .NOTPARALLEL special target which disables prerequisites > > randomization when running in shuffle mode. > > > > Signed-off-by: James Hilliard <james.hilliard1@gmail.com> > > --- > > package/Makefile.in | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/package/Makefile.in b/package/Makefile.in > > index 43d214bcbe..2327849a1f 100644 > > --- a/package/Makefile.in > > +++ b/package/Makefile.in > > @@ -17,7 +17,7 @@ else > > PARALLEL_JOBS := $(BR2_JLEVEL) > > endif > > > > -MAKE1 := $(HOSTMAKE) -j1 > > +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: > > Packages that do not build in parallel do not because their rules assume > the ordering is kept. > > I don't see how replacing -j1 with --eval .NOTPARALLEL: would cause the > ordering to be restored. Ah, it's in the --shuffle documentation, not in the .NOTPARALLEL one (and reading .texi source is not my main ability). Still, I think my proposal is better, because it is more explicit. Regards, Yann E. MORIN. > Instead, I would have expected we do something like: > > # Only build one job at a time, *and* to not randomise goals and > # prerequisites ordering in make 4.4+ > MAKE1 := $(HOSTMAKE) -j1 $(if $(findstring --shuffle,$(MAKEFLAGS)),--shuffle=none) > > Regards, > Yann E. MORIN. > > > override MAKE = $(HOSTMAKE) \ > > $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS)) > > > > -- > > 2.34.1 > > > > _______________________________________________ > > buildroot mailing list > > buildroot@buildroot.org > > https://lists.buildroot.org/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
On Sun, Oct 16, 2022 at 12:35 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > On 2022-10-16 18:26 +0200, Yann E. MORIN spake thusly: > > James, All, > > > > On 2022-10-14 18:56 -0600, James Hilliard spake thusly: > > > Make 4.4 introduces a shuffle mode which randomizes prerequisites > > > in order to better flush out issues with parallel builds, as this > > > mode randomizes prerequisites even when using -j1 builds we must > > > instead explicitely mark parallel incompatible builds using the > > > .NOTPARALLEL special target which disables prerequisites > > > randomization when running in shuffle mode. > > > > > > Signed-off-by: James Hilliard <james.hilliard1@gmail.com> > > > --- > > > package/Makefile.in | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/package/Makefile.in b/package/Makefile.in > > > index 43d214bcbe..2327849a1f 100644 > > > --- a/package/Makefile.in > > > +++ b/package/Makefile.in > > > @@ -17,7 +17,7 @@ else > > > PARALLEL_JOBS := $(BR2_JLEVEL) > > > endif > > > > > > -MAKE1 := $(HOSTMAKE) -j1 > > > +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: > > > > Packages that do not build in parallel do not because their rules assume > > the ordering is kept. > > > > I don't see how replacing -j1 with --eval .NOTPARALLEL: would cause the > > ordering to be restored. > > Ah, it's in the --shuffle documentation, not in the .NOTPARALLEL one (and > reading .texi source is not my main ability). > > Still, I think my proposal is better, because it is more explicit. We don't want to modify the shuffle value as it's useful for reproducing failures, although there's currently a bug which doesn't set the seed correctly: https://savannah.gnu.org/bugs/index.php?63215 Using .NOTPARALLEL should ensure the shuffle value is still passed down so that it appears in the failure message(once that bug is fixed) while still disabling parallel build for the parallel incompatible package. We want to be able to reproduce failures by passing the shuffle seed to the top level make invocation. > > Regards, > Yann E. MORIN. > > > Instead, I would have expected we do something like: > > > > # Only build one job at a time, *and* to not randomise goals and > > # prerequisites ordering in make 4.4+ > > MAKE1 := $(HOSTMAKE) -j1 $(if $(findstring --shuffle,$(MAKEFLAGS)),--shuffle=none) > > > > Regards, > > Yann E. MORIN. > > > > > override MAKE = $(HOSTMAKE) \ > > > $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS)) > > > > > > -- > > > 2.34.1 > > > > > > _______________________________________________ > > > buildroot mailing list > > > buildroot@buildroot.org > > > https://lists.buildroot.org/mailman/listinfo/buildroot > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' > > _______________________________________________ > > buildroot mailing list > > buildroot@buildroot.org > > https://lists.buildroot.org/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
James, All, On 2022-10-16 12:53 -0400, James Hilliard spake thusly: > On Sun, Oct 16, 2022 at 12:35 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: [--SNIP--] > > > > -MAKE1 := $(HOSTMAKE) -j1 > > > > +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: > > > > > > Packages that do not build in parallel do not because their rules assume > > > the ordering is kept. > > > > > > I don't see how replacing -j1 with --eval .NOTPARALLEL: would cause the > > > ordering to be restored. > > > > Ah, it's in the --shuffle documentation, not in the .NOTPARALLEL one (and > > reading .texi source is not my main ability). > > > > Still, I think my proposal is better, because it is more explicit. > > We don't want to modify the shuffle value In this case, yes, we do want to disable shuffling. Packages that do not build in parallel, do not exactly because of the build ordering. So far, we resorted to just forcing -j1 for those packages, because -j1 hid away the build order issue. So, shuffling them even in -j1, would cause build failures. And as you said, and as the make manual states, .NOTPARALLEL disables shuffling. I.e., technically, those two lines are equivalent (assuming make 4.4): MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: MAKE1 := $(HOSTMAKE) -j1 --shuffle=none Except the latter is explicit about its goals: single job and no shuffling, while the former is only explicit about not parallel (also I do not like that --eval, it feels horribly weird, especially with the trailing colon). What I am saying, is that I prefer we explicitly disable shuffling, rather than rely on the implicit behaviour of .NOTPARALLEL. Finally, my proposal only appends --shuffle=none if it finds --shuffle in MAKEFLAGS. If it is not present, we do not need it, either because we are make 4.3-or-earlier, or we are make 4.4 without shuffling. > as it's useful for > reproducing failures, This is irrelevant to set MAKE1, because with MAKE1 we know the build ordering matters, and that we explicitly state that we do not want to be bothered by it. > although there's currently a bug which doesn't set the seed correctly: > https://savannah.gnu.org/bugs/index.php?63215 > > Using .NOTPARALLEL should ensure the shuffle value is still passed down Sorry, but .NOTPARALLEL disables shuffling, so I am lost: why would it matter to have the seed passed down? > so that it appears in the failure message(once that bug is fixed) while > still disabling parallel build for the parallel incompatible package. Parallel-incompatible packages are incompatible exactly because the build ordering does matter to them, so we do not want to shuffle them. > We want to be able to reproduce failures by passing the shuffle seed to the > top level make invocation. This is about debugging our dependency issues, not about the packages' Makefiles. Regards, Yann E. MORIN.
On Sun, Oct 16, 2022 at 2:46 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > James, All, > > On 2022-10-16 12:53 -0400, James Hilliard spake thusly: > > On Sun, Oct 16, 2022 at 12:35 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > [--SNIP--] > > > > > -MAKE1 := $(HOSTMAKE) -j1 > > > > > +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: > > > > > > > > Packages that do not build in parallel do not because their rules assume > > > > the ordering is kept. > > > > > > > > I don't see how replacing -j1 with --eval .NOTPARALLEL: would cause the > > > > ordering to be restored. > > > > > > Ah, it's in the --shuffle documentation, not in the .NOTPARALLEL one (and > > > reading .texi source is not my main ability). > > > > > > Still, I think my proposal is better, because it is more explicit. > > > > We don't want to modify the shuffle value > > In this case, yes, we do want to disable shuffling. > > Packages that do not build in parallel, do not exactly because of the > build ordering. So far, we resorted to just forcing -j1 for those > packages, because -j1 hid away the build order issue. > > So, shuffling them even in -j1, would cause build failures. > > And as you said, and as the make manual states, .NOTPARALLEL disables > shuffling. > > I.e., technically, those two lines are equivalent (assuming make 4.4): > > MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: > MAKE1 := $(HOSTMAKE) -j1 --shuffle=none > > Except the latter is explicit about its goals: single job and no > shuffling, while the former is only explicit about not parallel (also I > do not like that --eval, it feels horribly weird, especially with the > trailing colon). > > What I am saying, is that I prefer we explicitly disable shuffling, > rather than rely on the implicit behaviour of .NOTPARALLEL. > > Finally, my proposal only appends --shuffle=none if it finds --shuffle > in MAKEFLAGS. If it is not present, we do not need it, either because > we are make 4.3-or-earlier, or we are make 4.4 without shuffling. I'll test a little more and send a patch based on that. > > > as it's useful for > > reproducing failures, > > This is irrelevant to set MAKE1, because with MAKE1 we know the build > ordering matters, and that we explicitly state that we do not want to > be bothered by it. > > > although there's currently a bug which doesn't set the seed correctly: > > https://savannah.gnu.org/bugs/index.php?63215 > > > > Using .NOTPARALLEL should ensure the shuffle value is still passed down > > Sorry, but .NOTPARALLEL disables shuffling, so I am lost: why would it > matter to have the seed passed down? Nevermind, it seems --shuffle=none works the same, I assumed it would cause the submake to drop the initial seed value(for printing in the event of an error) but it apparently doesn't at least with this patch: https://savannah.gnu.org/bugs/index.php?63215#comment0 > > > so that it appears in the failure message(once that bug is fixed) while > > still disabling parallel build for the parallel incompatible package. > > Parallel-incompatible packages are incompatible exactly because the > build ordering does matter to them, so we do not want to shuffle them. Right, but we still want to see the seed value on error, even if the package is using MAKE1, seems to work fine with --shuffle=none though. > > > We want to be able to reproduce failures by passing the shuffle seed to the > > top level make invocation. > > This is about debugging our dependency issues, not about the packages' > Makefiles. Well seed is supposed to get passed down all the way, otherwise it wouldn't properly test parallel issues as those can affect both cases. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
diff --git a/package/Makefile.in b/package/Makefile.in index 43d214bcbe..2327849a1f 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -17,7 +17,7 @@ else PARALLEL_JOBS := $(BR2_JLEVEL) endif -MAKE1 := $(HOSTMAKE) -j1 +MAKE1 := $(HOSTMAKE) --eval .NOTPARALLEL: override MAKE = $(HOSTMAKE) \ $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS))
Make 4.4 introduces a shuffle mode which randomizes prerequisites in order to better flush out issues with parallel builds, as this mode randomizes prerequisites even when using -j1 builds we must instead explicitely mark parallel incompatible builds using the .NOTPARALLEL special target which disables prerequisites randomization when running in shuffle mode. Signed-off-by: James Hilliard <james.hilliard1@gmail.com> --- package/Makefile.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)