| Message ID | 56618964.3020209@arm.com |
|---|---|
| State | New |
| Headers | show |
On 04/12/15 12:39, Szabolcs Nagy wrote: > As described in pr other/67627, the all-multi target can be > built in parallel with the %_.lo targets which generate make > dependencies that are parsed during the build of all-multi. > > gcc -MD does not generate the makefile dependencies in an > atomic way so make can fail if it concurrently parses those > half-written files. > (not observed on x86, but happens on arm native builds.) > > this workaround forces all-multi to only run after the *_.lo > targets are done, but there might be a better solution using > automake properly. (automake should know about the generated > make dependency files that are included into the makefile so > no manual tinkering is needed to get the right build order, > but i don't know how to do that.) > > 2015-12-04 Szabolcs Nagy <szabolcs.nagy@arm.com> > > PR other/67627 > * Makefile.am (all-multi): Add dependency. > * Makefile.in: Regenerate. > ping and cc rth (as his name is on this makefile). > > diff --git a/libatomic/Makefile.am b/libatomic/Makefile.am > index bd0ab29..38c635f 100644 > --- a/libatomic/Makefile.am > +++ b/libatomic/Makefile.am > @@ -139,3 +139,10 @@ endif > > libatomic_convenience_la_SOURCES = $(libatomic_la_SOURCES) > libatomic_convenience_la_LIBADD = $(libatomic_la_LIBADD) > + > +# Override the automake generated all-multi rule to guarantee that all-multi > +# is not run in parallel with the %_.lo rules which generate $(DEPDIR)/*.Ppo > +# makefile fragments to avoid broken *.Ppo getting included into the Makefile > +# when it is reloaded during the build of all-multi. > +all-multi: $(libatomic_la_LIBADD) > + $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) > diff --git a/libatomic/Makefile.in b/libatomic/Makefile.in > index b696d55..a083d87 100644 > --- a/libatomic/Makefile.in > +++ b/libatomic/Makefile.in > @@ -496,12 +496,6 @@ clean-libtool: > > distclean-libtool: > -rm -f libtool config.lt > - > -# GNU Make needs to see an explicit $(MAKE) variable in the command it > -# runs to enable its job server during parallel builds. Hence the > -# comments below. > -all-multi: > - $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) > install-multi: > $(MULTIDO) $(AM_MAKEFLAGS) DO=install multi-do # $(MAKE) > > @@ -800,6 +794,13 @@ vpath % $(strip $(search_path)) > %_.lo: Makefile > $(LTCOMPILE) $(M_DEPS) $(M_SIZE) $(M_IFUNC) -c -o $@ $(M_SRC) > > +# Override the automake generated all-multi rule to guarantee that all-multi > +# is not run in parallel with the %_.lo rules which generate $(DEPDIR)/*.Ppo > +# makefile fragments to avoid broken *.Ppo getting included into the Makefile > +# when it is reloaded during the build of all-multi. > +all-multi: $(libatomic_la_LIBADD) > + $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) > + > # Tell versions [3.59,3.63) of GNU make to not export all variables. > # Otherwise a system limit (for SysV at least) may be exceeded. > .NOEXPORT: >
On 12/04/2015 05:39 AM, Szabolcs Nagy wrote: > As described in pr other/67627, the all-multi target can be > built in parallel with the %_.lo targets which generate make > dependencies that are parsed during the build of all-multi. > > gcc -MD does not generate the makefile dependencies in an > atomic way so make can fail if it concurrently parses those > half-written files. > (not observed on x86, but happens on arm native builds.) > > this workaround forces all-multi to only run after the *_.lo > targets are done, but there might be a better solution using > automake properly. (automake should know about the generated > make dependency files that are included into the makefile so > no manual tinkering is needed to get the right build order, > but i don't know how to do that.) > > 2015-12-04 Szabolcs Nagy <szabolcs.nagy@arm.com> > > PR other/67627 > * Makefile.am (all-multi): Add dependency. > * Makefile.in: Regenerate. ISTM that many of the libraries have this problem. [law@localhost gcc]$ grep all-am: */Makefile.in | grep -v install boehm-gc/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi gotools/Makefile.in:all-am: Makefile $(PROGRAMS) $(MANS) libatomic/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi auto-config.h libbacktrace/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi config.h libcc1/Makefile.in:all-am: Makefile $(LTLIBRARIES) cc1plugin-config.h libcilkrts/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi $(HEADERS) libffi/Makefile.in:all-am: Makefile $(INFO_DEPS) $(LTLIBRARIES) all-multi $(DATA) \ libgfortran/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi $(DATA) $(HEADERS) config.h libgo/Makefile.in:all-am: Makefile $(LIBRARIES) $(LTLIBRARIES) all-multi $(DATA) \ libgomp/Makefile.in:all-am: Makefile $(INFO_DEPS) $(LTLIBRARIES) all-multi $(HEADERS) \ libitm/Makefile.in:all-am: Makefile $(INFO_DEPS) $(LTLIBRARIES) all-multi $(HEADERS) \ libjava/Makefile.in:all-am: Makefile $(LTLIBRARIES) $(PROGRAMS) $(SCRIPTS) all-multi \ libmpx/Makefile.in:all-am: Makefile all-multi $(HEADERS) config.h liboffloadmic/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi $(HEADERS) all-local libquadmath/Makefile.in:all-am: Makefile $(INFO_DEPS) $(LTLIBRARIES) all-multi $(HEADERS) \ libsanitizer/Makefile.in:all-am: Makefile all-multi $(HEADERS) config.h libssp/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi $(HEADERS) config.h libstdc++-v3/Makefile.in:all-am: Makefile all-multi config.h libvtv/Makefile.in:all-am: Makefile $(LTLIBRARIES) all-multi $(HEADERS) lto-plugin/Makefile.in:all-am: Makefile $(LTLIBRARIES) config.h all-local zlib/Makefile.in:all-am: Makefile $(LIBRARIES) $(LTLIBRARIES) all-multi And if you look at the all-multi targets in each of those, all-multi has no dependencies. So any of them which use auto-dependency generation are potentially going to bump into this problem. We can't fix it by twiddling Makefile.in as that's a generated file. I think it has to happen at the automake level. jeff
On 12/04/2015 05:39 AM, Szabolcs Nagy wrote: > As described in pr other/67627, the all-multi target can be > built in parallel with the %_.lo targets which generate make > dependencies that are parsed during the build of all-multi. > > gcc -MD does not generate the makefile dependencies in an > atomic way so make can fail if it concurrently parses those > half-written files. > (not observed on x86, but happens on arm native builds.) > > this workaround forces all-multi to only run after the *_.lo > targets are done, but there might be a better solution using > automake properly. (automake should know about the generated > make dependency files that are included into the makefile so > no manual tinkering is needed to get the right build order, > but i don't know how to do that.) > > 2015-12-04 Szabolcs Nagy <szabolcs.nagy@arm.com> > > PR other/67627 > * Makefile.am (all-multi): Add dependency. > * Makefile.in: Regenerate. So looking at the patch, it looks like you're adding a dependency in Makefile.am to pass it through to Makefile.in, which is fine. So I think you just need to replicate that fix across the other libraries which have this problem. jeff
On 16/12/15 17:06, Jeff Law wrote: > On 12/04/2015 05:39 AM, Szabolcs Nagy wrote: >> As described in pr other/67627, the all-multi target can be >> built in parallel with the %_.lo targets which generate make >> dependencies that are parsed during the build of all-multi. >> >> gcc -MD does not generate the makefile dependencies in an >> atomic way so make can fail if it concurrently parses those >> half-written files. >> (not observed on x86, but happens on arm native builds.) >> >> this workaround forces all-multi to only run after the *_.lo >> targets are done, but there might be a better solution using >> automake properly. (automake should know about the generated >> make dependency files that are included into the makefile so >> no manual tinkering is needed to get the right build order, >> but i don't know how to do that.) >> >> 2015-12-04 Szabolcs Nagy <szabolcs.nagy@arm.com> >> >> PR other/67627 >> * Makefile.am (all-multi): Add dependency. >> * Makefile.in: Regenerate. > So looking at the patch, it looks like you're adding > a dependency in Makefile.am to pass it through to > Makefile.in, which is fine. > > So I think you just need to replicate that fix across > the other libraries which have this problem. > i don't see other libraries that use all-multi and include auto-dependency generated makefiles as well. only libatomic has both. is it ok to commit and backport? (gcc-5 and 4.9 have the same issue)
On 12/17/2015 02:51 AM, Szabolcs Nagy wrote: > On 16/12/15 17:06, Jeff Law wrote: >> On 12/04/2015 05:39 AM, Szabolcs Nagy wrote: >>> As described in pr other/67627, the all-multi target can be >>> built in parallel with the %_.lo targets which generate make >>> dependencies that are parsed during the build of all-multi. >>> >>> gcc -MD does not generate the makefile dependencies in an >>> atomic way so make can fail if it concurrently parses those >>> half-written files. >>> (not observed on x86, but happens on arm native builds.) >>> >>> this workaround forces all-multi to only run after the *_.lo >>> targets are done, but there might be a better solution using >>> automake properly. (automake should know about the generated >>> make dependency files that are included into the makefile so >>> no manual tinkering is needed to get the right build order, >>> but i don't know how to do that.) >>> >>> 2015-12-04 Szabolcs Nagy <szabolcs.nagy@arm.com> >>> >>> PR other/67627 >>> * Makefile.am (all-multi): Add dependency. >>> * Makefile.in: Regenerate. >> So looking at the patch, it looks like you're adding >> a dependency in Makefile.am to pass it through to >> Makefile.in, which is fine. >> >> So I think you just need to replicate that fix across >> the other libraries which have this problem. >> > > i don't see other libraries that use all-multi and > include auto-dependency generated makefiles as well. > > only libatomic has both. > > is it ok to commit and backport? > (gcc-5 and 4.9 have the same issue) Sorry for the delay. Yes, you're correct, it appears that only libatomic has this issue. The patch is fine for the trunk and any release branches you want to backport to. Thanks, jeff
diff --git a/libatomic/Makefile.am b/libatomic/Makefile.am index bd0ab29..38c635f 100644 --- a/libatomic/Makefile.am +++ b/libatomic/Makefile.am @@ -139,3 +139,10 @@ endif libatomic_convenience_la_SOURCES = $(libatomic_la_SOURCES) libatomic_convenience_la_LIBADD = $(libatomic_la_LIBADD) + +# Override the automake generated all-multi rule to guarantee that all-multi +# is not run in parallel with the %_.lo rules which generate $(DEPDIR)/*.Ppo +# makefile fragments to avoid broken *.Ppo getting included into the Makefile +# when it is reloaded during the build of all-multi. +all-multi: $(libatomic_la_LIBADD) + $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) diff --git a/libatomic/Makefile.in b/libatomic/Makefile.in index b696d55..a083d87 100644 --- a/libatomic/Makefile.in +++ b/libatomic/Makefile.in @@ -496,12 +496,6 @@ clean-libtool: distclean-libtool: -rm -f libtool config.lt - -# GNU Make needs to see an explicit $(MAKE) variable in the command it -# runs to enable its job server during parallel builds. Hence the -# comments below. -all-multi: - $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) install-multi: $(MULTIDO) $(AM_MAKEFLAGS) DO=install multi-do # $(MAKE) @@ -800,6 +794,13 @@ vpath % $(strip $(search_path)) %_.lo: Makefile $(LTCOMPILE) $(M_DEPS) $(M_SIZE) $(M_IFUNC) -c -o $@ $(M_SRC) +# Override the automake generated all-multi rule to guarantee that all-multi +# is not run in parallel with the %_.lo rules which generate $(DEPDIR)/*.Ppo +# makefile fragments to avoid broken *.Ppo getting included into the Makefile +# when it is reloaded during the build of all-multi. +all-multi: $(libatomic_la_LIBADD) + $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE) + # Tell versions [3.59,3.63) of GNU make to not export all variables. # Otherwise a system limit (for SysV at least) may be exceeded. .NOEXPORT: