--enable-link-serialization support

On Thu, Sep 03, 2020 at 03:53:35PM +0200, Richard Biener wrote:
> On Thu, 3 Sep 2020, Jakub Jelinek wrote:
> But is that an issue in practice?  I usually do not do make -j32 cc1plus
> in a tree that was configured for bootstrap, nor do I use
> --enable-link-serialization in that case.

Guess most often true, but one could still do it when debugging some
particular problem during bootstrap.

> Btw, do you plan to keep --enable-link-mutex the way it is?  What it
> provides ontop of --enable-link-serialization is that progress
> metering for very long running LTO link stages.  Not sure if that
> works for the last LTO link or only when there's some other link
> waiting.

I kept it as is for now.  Primarily I didn't want to use
--enable-linux-mutex for the new thing when it has nothing to do with a
mutex.  I think with both options together it will just print useless
message that it acquired the lock immediately in each case.
And not really sure how and what should we print as progress indicator,
perhaps Make-hooks could contain in some variable number of the languages
(well, one can use $(words $(CONFIG_LANGUAGES)) for that purpose)
and in other vars record for each FE its serial number) and then add
	@echo Linking C++ - $(c++.idx) out of $(words $(CONFIG_LANGUAGES))
right before the $(LLINK) lines, perhaps only added if $(DO_LINK_SERIALIZATION)
is true.  Or perhaps do some ANSI colors progress bar for that too.

Anyway, here is the updated patch that doesn't use recursive make for that,
I think that could run into latent Makefile dependency issues and the like.

For new FEs, the new requirements are basically that one should add
$lang.serial $lang.prev to .PHONY, make $lang.serial depend on the binary
linked with $(LLINKER) and add $lang.prev to the dependencies of that
binary.  Unless, as in the lto special case, there are multiple such
binaries, then $lang.serial should depend on the last one, $lang.prev
should be the dependy of the first one and there should be dependencies
between the binaries guarded on $(DO_LINK_SERIALIZATION) equal to true.

2020-09-03  Jakub Jelinek  <jakub@redhat.com>

	* configure.ac (--enable-link-serialization): New configure option.
	* Makefile.tpl (EXTRA_GCC_FLAGS): Pass DO_LINK_SERIALIZATION=true
	or false to recursive make in gcc subdirectory.
	* configure: Regenerated.
	* Makefile.in: Regenerated.
gcc/
	* configure.ac: Add $lang.prev: $prev.serial or $lang.prev: rules to
	Make-hooks.
	* doc/install.texi (--enable-link-serialization): Document.
	* configure: Regenerated.
gcc/c/
	* Make-lang.in (c.serial): New goal.
	(.PHONY): Add c.serial c.prev.
gcc/cp/
	* Make-lang.in (c++.serial): New goal.
	(.PHONY): Add c++.serial c++.prev.
	(cc1plus$(exeext)): Depend on c++.prev.
gcc/fortran/
	* Make-lang.in (fortran.serial): New goal.
	(.PHONY): Add fortran.serial fortran.prev.
	(f951$(exeext)): Depend on fortran.prev.
gcc/lto/
	* Make-lang.in (lto, lto.serial): New goals.
	(.PHONY): Add fortran.serial fortran.prev.
	(lto.all.cross, lto.start.encap): Remove dependencies.
	($(LTO_EXE)): Depend on lto.prev.
	(LTO_DUMP_EXE_PREV): New variable.
	($(LTO_DUMP_EXE)): Depend on $(LTO_DUMP_EXE_PREV).
gcc/objc/
	* Make-lang.in (objc.serial): New goal.
	(.PHONY): Add objc.serial objc.prev.
	(cc1obj$(exeext)): Depend on objc.prev.
gcc/objcp/
	* Make-lang.in (obj-c++.serial): New goal.
	(.PHONY): Add obj-c++.serial obj-c++.prev.
	(cc1objplus$(exeext)): Depend on obj-c++.prev.
gcc/ada/
	* gcc-interface/Make-lang.in (ada.serial): New goal.
	(.PHONY): Add ada.serial ada.prev.
	(gnat1$(exeext)): Depend on ada.prev.
gcc/brig/
	* Make-lang.in (brig.serial): New goal.
	(.PHONY): Add brig.serial brig.prev.
	(brig1$(exeext)): Depend on brig.prev.
gcc/go/
	* Make-lang.in (go.serial): New goal.
	(.PHONY): Add go.serial go.prev.
	(go1$(exeext)): Depend on go.prev.
gcc/jit/
	* Make-lang.in (jit.serial): New goal.
	(.PHONY): Add jit.serial jit.prev.
	($(LIBGCCJIT_FILENAME)): Depend on jit.prev.
gcc/d/
	* Make-lang.in (d.serial): New goal.
	(.PHONY): Add d.serial d.prev.
	(d21$(exeext)): Depend on d.prev.
	


	Jakub

Hi!

CCing build maintainers now.

On Thu, Sep 03, 2020 at 04:49:09PM +0200, Jakub Jelinek via Gcc-patches wrote:
> On Thu, Sep 03, 2020 at 03:53:35PM +0200, Richard Biener wrote:
> > No review on the actual patch - it looks like what I'd have tried
> > but I'm no make expert ;)

Successfully bootstrapped/regtested now, once on x86_64-linux with:
../configure --enable-languages=default,ada,obj-c++,lto,go,brig,d --enable-link-serialization --with-build-config=bootstrap-lto --enable-checking=yes,rtl,extra && make -j16 bootstrap > LOG 2>&1 && GXX_TESTSUITE_STDS=98,11,14,17,2a make -j32 -k check > LOGC 2>&1; ../contrib/test_summary > LOGT 2>&1
i.e. with LTO bootstrap and this serialization on and verified that all the
large program LTO links were serialized, and normally on x86_64-linux
and i686-linux (i.e. with serialization turned off, non-lto bootstraps)
and verified that all the links were concurrent.

Ok for trunk?

If we wanted to go further, it could have also arguments,
--disable-link-serialization (the default, and =no, and =0) would mean allow all LLINKERs
concurrently,
--enable-link-serialization (and =yes, =1) would serialize and
--enable-link-serialization=N for N > 0 would ensure at most N concurrent
LLINKER jobs.
This could be done by adjusting the patch, so that configure knows there are
two jobs for lto and use lto1.{prev,serial} and lto2.{prev,serial}, and do
emit into Make-hooks:
ifeq ($(DO_LINK_SERIALIZATION),)
SERIAL_LIST =
else
# Note, configure would put the list sorted backwards and without the last one (obj-c++.serial in my case)
SERIAL_LIST = $(wordlist $(DO_LINK_SERIALIZATION),10,objc.serial lto2.serial lto1.serial go.serial fortran.serial d.serial c++.serial brig.serial ada.serial c.serial)
endif
ada.prev: $(word 10,$(SERIAL_LIST))
brig.prev: $(word 9,$(SERIAL_LIST))
c++.prev: $(word 8,$(SERIAL_LIST))
d.prev: $(word 7,$(SERIAL_LIST))
fortran.prev: $(word 6,$(SERIAL_LIST))
go.prev: $(word 5,$(SERIAL_LIST))
lto1.prev: $(word 4,$(SERIAL_LIST))
lto2.prev: $(word 3,$(SERIAL_LIST))
objc.prev: $(word 2,$(SERIAL_LIST))
obj-c++.prev: $(word 1,$(SERIAL_LIST))
The toplevel would then pass DO_LINK_SERIALIZATION=1024 or something
similarly high (or DO_LINK_SERIALIZATION= ) for the default
--disable-link-serialization case, 1 for the yes or 1 case etc.
I think this would arrange the same thing as the patch does now for
the DO_LINK_SERIALIZATION= and DO_LINK_SERIALIZATION=1 (currently false and
true) cases, ada.prev would depend on c.serial, brig.prev on ada.prev etc.,
but e.g. for =2 ada.prev wouldn't depend on anything (i.e. it can be linked
concurretly with c), brig.prev would depend on c.prev (so it needs to wait
until cc1 is done), c++.prev would depend on ada.prev (so it would wait
until gnat1 is done), etc.

> > On Thu, 3 Sep 2020, Jakub Jelinek wrote:
> > But is that an issue in practice?  I usually do not do make -j32 cc1plus
> > in a tree that was configured for bootstrap, nor do I use
> > --enable-link-serialization in that case.
> 
> Guess most often true, but one could still do it when debugging some
> particular problem during bootstrap.
> 
> > Btw, do you plan to keep --enable-link-mutex the way it is?  What it
> > provides ontop of --enable-link-serialization is that progress
> > metering for very long running LTO link stages.  Not sure if that
> > works for the last LTO link or only when there's some other link
> > waiting.
> 
> I kept it as is for now.  Primarily I didn't want to use
> --enable-linux-mutex for the new thing when it has nothing to do with a
> mutex.  I think with both options together it will just print useless
> message that it acquired the lock immediately in each case.
> And not really sure how and what should we print as progress indicator,
> perhaps Make-hooks could contain in some variable number of the languages
> (well, one can use $(words $(CONFIG_LANGUAGES)) for that purpose)
> and in other vars record for each FE its serial number) and then add
> 	@echo Linking C++ - $(c++.idx) out of $(words $(CONFIG_LANGUAGES))
> right before the $(LLINK) lines, perhaps only added if $(DO_LINK_SERIALIZATION)
> is true.  Or perhaps do some ANSI colors progress bar for that too.
> 
> Anyway, here is the updated patch that doesn't use recursive make for that,
> I think that could run into latent Makefile dependency issues and the like.
> 
> For new FEs, the new requirements are basically that one should add
> $lang.serial $lang.prev to .PHONY, make $lang.serial depend on the binary
> linked with $(LLINKER) and add $lang.prev to the dependencies of that
> binary.  Unless, as in the lto special case, there are multiple such
> binaries, then $lang.serial should depend on the last one, $lang.prev
> should be the dependy of the first one and there should be dependencies
> between the binaries guarded on $(DO_LINK_SERIALIZATION) equal to true.
> 
> 2020-09-03  Jakub Jelinek  <jakub@redhat.com>
> 
> 	* configure.ac (--enable-link-serialization): New configure option.
> 	* Makefile.tpl (EXTRA_GCC_FLAGS): Pass DO_LINK_SERIALIZATION=true
> 	or false to recursive make in gcc subdirectory.
> 	* configure: Regenerated.
> 	* Makefile.in: Regenerated.
> gcc/
> 	* configure.ac: Add $lang.prev: $prev.serial or $lang.prev: rules to
> 	Make-hooks.
> 	* doc/install.texi (--enable-link-serialization): Document.
> 	* configure: Regenerated.
> gcc/c/
> 	* Make-lang.in (c.serial): New goal.
> 	(.PHONY): Add c.serial c.prev.
> gcc/cp/
> 	* Make-lang.in (c++.serial): New goal.
> 	(.PHONY): Add c++.serial c++.prev.
> 	(cc1plus$(exeext)): Depend on c++.prev.
> gcc/fortran/
> 	* Make-lang.in (fortran.serial): New goal.
> 	(.PHONY): Add fortran.serial fortran.prev.
> 	(f951$(exeext)): Depend on fortran.prev.
> gcc/lto/
> 	* Make-lang.in (lto, lto.serial): New goals.
> 	(.PHONY): Add fortran.serial fortran.prev.
> 	(lto.all.cross, lto.start.encap): Remove dependencies.
> 	($(LTO_EXE)): Depend on lto.prev.
> 	(LTO_DUMP_EXE_PREV): New variable.
> 	($(LTO_DUMP_EXE)): Depend on $(LTO_DUMP_EXE_PREV).
> gcc/objc/
> 	* Make-lang.in (objc.serial): New goal.
> 	(.PHONY): Add objc.serial objc.prev.
> 	(cc1obj$(exeext)): Depend on objc.prev.
> gcc/objcp/
> 	* Make-lang.in (obj-c++.serial): New goal.
> 	(.PHONY): Add obj-c++.serial obj-c++.prev.
> 	(cc1objplus$(exeext)): Depend on obj-c++.prev.
> gcc/ada/
> 	* gcc-interface/Make-lang.in (ada.serial): New goal.
> 	(.PHONY): Add ada.serial ada.prev.
> 	(gnat1$(exeext)): Depend on ada.prev.
> gcc/brig/
> 	* Make-lang.in (brig.serial): New goal.
> 	(.PHONY): Add brig.serial brig.prev.
> 	(brig1$(exeext)): Depend on brig.prev.
> gcc/go/
> 	* Make-lang.in (go.serial): New goal.
> 	(.PHONY): Add go.serial go.prev.
> 	(go1$(exeext)): Depend on go.prev.
> gcc/jit/
> 	* Make-lang.in (jit.serial): New goal.
> 	(.PHONY): Add jit.serial jit.prev.
> 	($(LIBGCCJIT_FILENAME)): Depend on jit.prev.
> gcc/d/
> 	* Make-lang.in (d.serial): New goal.
> 	(.PHONY): Add d.serial d.prev.
> 	(d21$(exeext)): Depend on d.prev.
> 	
> --- configure.ac.jj	2020-08-24 10:00:01.248259486 +0200
> +++ configure.ac	2020-09-03 16:09:51.010782912 +0200
> @@ -1866,6 +1866,24 @@ AC_ARG_ENABLE(linker-plugin-flags,
>    extra_linker_plugin_flags=)
>  AC_SUBST(extra_linker_plugin_flags)
>  
> +dnl Whether to prevent multiple GCC front-ends from linking at the same time
> +
> +AC_MSG_CHECKING([whether to serialize linking of multiple front-ends])
> +  AC_ARG_ENABLE(link-serialization,
> +[AS_HELP_STRING([--enable-link-serialization],
> +		[avoid linking multiple GCC front-ends at once using make
> +		 dependencies to avoid thrashing on the build machine])],
> +      do_link_serialization=$enableval,
> +      do_link_serialization=no)
> +AC_MSG_RESULT($do_link_serialization)
> +
> +if test "$do_link_serialization" = "yes"; then
> +   DO_LINK_SERIALIZATION=true
> +else
> +   DO_LINK_SERIALIZATION=false
> +fi
> +AC_SUBST(DO_LINK_SERIALIZATION)
> +
>  # Enable --enable-host-shared.
>  # Checked early to determine whether jit is an 'all' language
>  AC_ARG_ENABLE(host-shared,
> --- Makefile.tpl.jj	2020-01-12 11:54:35.751423396 +0100
> +++ Makefile.tpl	2020-09-03 16:14:03.381097293 +0200
> @@ -734,7 +734,8 @@ TARGET_FLAGS_TO_PASS = $(BASE_FLAGS_TO_P
>  EXTRA_GCC_FLAGS = \
>  	"GCC_FOR_TARGET=$(GCC_FOR_TARGET)" \
>  	"`echo 'STMP_FIXPROTO=$(STMP_FIXPROTO)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`" \
> -	"`echo 'LIMITS_H_TEST=$(LIMITS_H_TEST)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`"
> +	"`echo 'LIMITS_H_TEST=$(LIMITS_H_TEST)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`" \
> +	"DO_LINK_SERIALIZATION=@DO_LINK_SERIALIZATION@"
>  
>  GCC_FLAGS_TO_PASS = $(BASE_FLAGS_TO_PASS) $(EXTRA_HOST_FLAGS) $(EXTRA_GCC_FLAGS)
>  
> --- gcc/configure.ac.jj	2020-08-26 17:09:45.829254709 +0200
> +++ gcc/configure.ac	2020-09-03 16:05:26.985638746 +0200
> @@ -6540,6 +6540,22 @@ do
>  	echo "lang.$t: $x" >> Make-hooks
>  done
>  
> +echo "ifeq (\$(DO_LINK_SERIALIZATION),true)" >> Make-hooks
> +prev=c
> +for lang in $all_selected_languages
> +do
> +	test $lang = c && continue
> +	echo "$lang.prev: $prev.serial" >> Make-hooks
> +	prev=$lang
> +done
> +echo else >> Make-hooks
> +for lang in $all_selected_languages
> +do
> +	test $lang = c && continue
> +	echo "$lang.prev:" >> Make-hooks
> +done
> +echo endif >> Make-hooks
> +
>  # --------
>  # Option include files
>  # --------
> --- gcc/c/Make-lang.in.jj	2020-07-28 15:39:09.669760889 +0200
> +++ gcc/c/Make-lang.in	2020-09-03 14:24:41.861122758 +0200
> @@ -37,9 +37,10 @@
>  #
>  # Define the names for selecting c in LANGUAGES.
>  c: cc1$(exeext)
> +c.serial: c
>  
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: c gcc
> +.PHONY: c gcc c.serial
>  
>  # The C front end driver.  This is different from the drivers for other
>  # front ends, because there is no C language specific driver (i.e. nothing
> --- gcc/cp/Make-lang.in.jj	2020-07-28 15:39:09.770759500 +0200
> +++ gcc/cp/Make-lang.in	2020-09-03 15:46:34.413189898 +0200
> @@ -47,9 +47,10 @@ CP_PLUGIN_HEADERS := cp-tree.h cxx-prett
>  # into the C++ rule, but that needs a little bit of work
>  # to do the right thing within all.cross.
>  c++: cc1plus$(exeext)
> +c++.serial: c++
>  
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: c++
> +.PHONY: c++ c++.serial c++.prev
>  
>  CFLAGS-cp/g++spec.o += $(DRIVER_DEFINES)
>  
> @@ -116,7 +117,7 @@ cc1plus-checksum.c : build/genchecksum$(
>  	  $(srcdir)/../move-if-change cc1plus-checksum.c.tmp cc1plus-checksum.c; \
>  	fi
>  
> -cc1plus$(exeext): $(CXX_OBJS) cc1plus-checksum.o $(BACKEND) $(LIBDEPS)
> +cc1plus$(exeext): $(CXX_OBJS) cc1plus-checksum.o $(BACKEND) $(LIBDEPS) c++.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  	      $(CXX_OBJS) cc1plus-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
>  
> --- gcc/fortran/Make-lang.in.jj	2020-01-12 11:54:36.582410859 +0100
> +++ gcc/fortran/Make-lang.in	2020-09-03 15:52:35.805901622 +0200
> @@ -72,9 +72,10 @@ fortran_OBJS = $(F95_OBJS) fortran/gfort
>  #
>  # Define the names for selecting gfortran in LANGUAGES.
>  fortran: f951$(exeext)
> +fortran.serial: fortran
>  
>  # Tell GNU make to ignore files by these names if they exist.
> -.PHONY: fortran
> +.PHONY: fortran fortran.serial fortran.prev
>  
>  CFLAGS-fortran/gfortranspec.o += $(DRIVER_DEFINES)
>  
> @@ -92,8 +93,7 @@ gfortran-cross$(exeext): gfortran$(exeex
>  	cp gfortran$(exeext) gfortran-cross$(exeext)
>  
>  # The compiler itself is called f951.
> -f951$(exeext): $(F95_OBJS) \
> -		$(BACKEND) $(LIBDEPS) attribs.o
> +f951$(exeext): $(F95_OBJS) $(BACKEND) $(LIBDEPS) attribs.o fortran.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  		$(F95_OBJS) $(BACKEND) $(ZLIB) $(LIBS) attribs.o \
>  		$(BACKENDLIBS)
> --- gcc/lto/Make-lang.in.jj	2020-01-12 11:54:36.677409426 +0100
> +++ gcc/lto/Make-lang.in	2020-09-03 16:04:44.140264456 +0200
> @@ -41,10 +41,15 @@ lto_dump_OBJS = $(LTO_DUMP_OBJS)
>  
>  # Rules
>  
> +lto: $(LTO_EXE) $(LTO_DUMP_EXE)
> +lto.serial: $(LTO_DUMP_EXE)
> +
> +.PHONY: lto lto.serial lto.prev
> +
>  # These hooks are used by the main GCC Makefile.  Consult that
>  # Makefile for documentation.
> -lto.all.cross: $(LTO_EXE) $(LTO_DUMP_EXE)
> -lto.start.encap: $(LTO_EXE) $(LTO_DUMP_EXE)
> +lto.all.cross:
> +lto.start.encap:
>  lto.rest.encap:
>  lto.tags:
>  lto.install-common: installdirs
> @@ -84,11 +89,16 @@ lto.stagefeedback:
>  # Use strict warnings for this front end.
>  lto-warn = $(STRICT_WARN)
>  
> -$(LTO_EXE): $(LTO_OBJS) $(BACKEND) $(LIBDEPS)
> +$(LTO_EXE): $(LTO_OBJS) $(BACKEND) $(LIBDEPS) lto.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  		$(LTO_OBJS) $(BACKEND) $(BACKENDLIBS) $(LIBS)
>  
> -$(LTO_DUMP_EXE): $(LTO_DUMP_OBJS) $(BACKEND) $(LIBDEPS)
> +ifeq ($(DO_LINK_SERIALIZATION),true)
> +LTO_DUMP_EXE_PREV = $(LTO_EXE)
> +else
> +LTO_DUMP_EXE_PREV =
> +endif
> +$(LTO_DUMP_EXE): $(LTO_DUMP_OBJS) $(BACKEND) $(LIBDEPS) $(LTO_DUMP_EXE_PREV)
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  		$(LTO_DUMP_OBJS) $(BACKEND) $(BACKENDLIBS) $(LIBS)
>  
> --- gcc/objc/Make-lang.in.jj	2020-01-12 11:54:36.681409365 +0100
> +++ gcc/objc/Make-lang.in	2020-09-03 15:50:47.542485854 +0200
> @@ -38,9 +38,10 @@
>  #
>  # Define the names for selecting Objective-C in LANGUAGES.
>  objc: cc1obj$(exeext)
> +objc.serial: objc
>  
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: objc
> +.PHONY: objc objc.serial objc.prev
>  
>  # Use maximal warnings for this front end.
>  objc-warn = $(STRICT_WARN)
> @@ -62,7 +63,8 @@ cc1obj-checksum.c : build/genchecksum$(b
>          $(BACKEND) $(LIBDEPS) checksum-options > cc1obj-checksum.c.tmp && \
>  	$(srcdir)/../move-if-change cc1obj-checksum.c.tmp cc1obj-checksum.c
>  
> -cc1obj$(exeext): $(OBJC_OBJS) $(C_AND_OBJC_OBJS) cc1obj-checksum.o $(BACKEND) $(LIBDEPS)
> +cc1obj$(exeext): $(OBJC_OBJS) $(C_AND_OBJC_OBJS) cc1obj-checksum.o $(BACKEND) \
> +		 $(LIBDEPS) objc.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  	      $(OBJC_OBJS) $(C_AND_OBJC_OBJS) cc1obj-checksum.o \
>  	      $(BACKEND) $(LIBS) $(BACKENDLIBS)
> --- gcc/objcp/Make-lang.in.jj	2020-01-12 11:54:36.685409305 +0100
> +++ gcc/objcp/Make-lang.in	2020-09-03 15:49:40.975459927 +0200
> @@ -39,9 +39,10 @@
>  #
>  # Define the names for selecting Objective-C++ in LANGUAGES.
>  obj-c++: cc1objplus$(exeext)
> +obj-c++.serial: obj-c++
>  
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: obj-c++
> +.PHONY: obj-c++ obj-c++.serial obj-c++.prev
>  
>  # Use maximal warnings for this front end.  Also, make ObjC and C++
>  # headers accessible.
> @@ -66,7 +67,8 @@ cc1objplus-checksum.c : build/genchecksu
>  	$(srcdir)/../move-if-change cc1objplus-checksum.c.tmp \
>  	cc1objplus-checksum.c
>  
> -cc1objplus$(exeext): $(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBDEPS)
> +cc1objplus$(exeext): $(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) \
> +		     $(LIBDEPS) obj-c++.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  		$(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
>  
> --- gcc/ada/gcc-interface/Make-lang.in.jj	2020-07-28 15:39:09.470763625 +0200
> +++ gcc/ada/gcc-interface/Make-lang.in	2020-09-03 15:55:04.140734768 +0200
> @@ -146,9 +146,10 @@ endif
>  
>  # Define the names for selecting Ada in LANGUAGES.
>  ada: gnat1$(exeext) gnatbind$(exeext)
> +ada.serial: gnat1$(exeext)
>  
>  # Tell GNU Make to ignore these, if they exist.
> -.PHONY: ada
> +.PHONY: ada ada.serial ada.prev
>  
>  # Compute the FLAGS to pass for gnattools, now linked with a C++ driver as
>  # we're linking against at least libcommon which contains C++ compiled code.
> @@ -666,7 +667,8 @@ ada/libgnat/s-excmac.adb: $(srcdir)/ada/
>  # Needs to be built with CC=gcc
>  # Since the RTL should be built with the latest compiler, remove the
>  #  stamp target in the parent directory whenever gnat1 is rebuilt
> -gnat1$(exeext): $(TARGET_ADA_SRCS) $(GNAT1_OBJS) $(ADA_BACKEND) libcommon-target.a $(LIBDEPS)
> +gnat1$(exeext): $(TARGET_ADA_SRCS) $(GNAT1_OBJS) $(ADA_BACKEND) libcommon-target.a \
> +		$(LIBDEPS) ada.prev
>  	+$(GCC_LLINK) -o $@ $(GNAT1_OBJS) $(ADA_BACKEND) \
>  	  libcommon-target.a $(LIBS) $(SYSLIBS) $(BACKENDLIBS) $(CFLAGS)
>  	$(RM) stamp-gnatlib2-rts stamp-tools
> --- gcc/brig/Make-lang.in.jj	2020-01-12 11:54:36.193416728 +0100
> +++ gcc/brig/Make-lang.in	2020-09-03 15:53:16.563305843 +0200
> @@ -29,8 +29,9 @@ GCCBRIG_TARGET_INSTALL_NAME := $(target_
>  
>  # The name for selecting brig in LANGUAGES.
>  brig: brig1$(exeext)
> +brig.serial: brig
>  
> -.PHONY: brig
> +.PHONY: brig brig.serial brig.prev
>  
>  CFLAGS-brig/brigspec.o += $(DRIVER_DEFINES)
>  
> @@ -81,12 +82,7 @@ BRIG_OBJS = \
>  
>  brig_OBJS = $(BRIG_OBJS) brig/brigspec.o
>  
> -# brig1$(exeext): $(BRIG_OBJS) attribs.o $(BACKEND) $(LIBDEPS)
> -# 	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
> -# 	      $(BRIG_OBJS) attribs.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
> -
> -
> -brig1$(exeext): $(BRIG_OBJS) attribs.o $(BACKEND) $(LIBDEPS)
> +brig1$(exeext): $(BRIG_OBJS) attribs.o $(BACKEND) $(LIBDEPS) brig.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  	      $(BRIG_OBJS) attribs.o $(BACKEND) $(LIBS) \
>  		  $(BACKENDLIBS)
> --- gcc/go/Make-lang.in.jj	2020-01-12 11:54:36.630410135 +0100
> +++ gcc/go/Make-lang.in	2020-09-03 15:54:02.497635018 +0200
> @@ -27,8 +27,9 @@ GCCGO_TARGET_INSTALL_NAME := $(target_no
>  
>  # The name for selecting go in LANGUAGES.
>  go: go1$(exeext)
> +go.serial: go
>  
> -.PHONY: go
> +.PHONY: go go.serial go.prev
>  
>  CFLAGS-go/gospec.o += $(DRIVER_DEFINES)
>  
> @@ -78,7 +79,7 @@ GO_OBJS = \
>  
>  go_OBJS = $(GO_OBJS) go/gospec.o
>  
> -go1$(exeext): $(GO_OBJS) attribs.o $(BACKEND) $(LIBDEPS)
> +go1$(exeext): $(GO_OBJS) attribs.o $(BACKEND) $(LIBDEPS) go.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  	      $(GO_OBJS) attribs.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
>  
> --- gcc/jit/Make-lang.in.jj	2020-07-28 15:39:09.956756942 +0200
> +++ gcc/jit/Make-lang.in	2020-09-03 16:00:58.490559851 +0200
> @@ -81,8 +81,10 @@ jit: $(LIBGCCJIT_FILENAME) \
>  	$(FULL_DRIVER_NAME)
>  endif
>  
> +jit.serial: $(LIBGCCJIT_FILENAME)
> +
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: jit
> +.PHONY: jit jit.serial jit.prev
>  
>  jit_OBJS = attribs.o \
>  	jit/dummy-frontend.o \
> @@ -117,7 +119,7 @@ $(LIBGCCJIT_FILENAME): $(jit_OBJS) \
>  	libbackend.a libcommon-target.a libcommon.a \
>  	$(CPPLIB) $(LIBDECNUMBER) \
>  	$(LIBDEPS) $(srcdir)/jit/libgccjit.map \
> -	$(EXTRA_GCC_OBJS)
> +	$(EXTRA_GCC_OBJS) jit.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ -shared \
>  	     $(jit_OBJS) libbackend.a libcommon-target.a libcommon.a \
>  	     $(CPPLIB) $(LIBDECNUMBER) $(EXTRA_GCC_LIBS) $(LIBS) $(BACKENDLIBS) \
> --- gcc/d/Make-lang.in.jj	2020-08-26 10:27:10.690552343 +0200
> +++ gcc/d/Make-lang.in	2020-09-03 15:48:15.010717845 +0200
> @@ -27,9 +27,10 @@ D_LIBPHOBOS = -DLIBPHOBOS=\"gphobos\"
>  
>  # The name for selecting d in LANGUAGES.
>  d: d21$(exeext)
> +d.serial: d
>  
>  # Tell GNU make to ignore these if they exist.
> -.PHONY: d
> +.PHONY: d d.serial d.prev
>  
>  # Create the compiler driver for D.
>  CFLAGS-d/d-spec.o += $(DRIVER_DEFINES) $(D_LIBPHOBOS)
> @@ -162,7 +163,7 @@ D_ALL_OBJS = $(D_FRONTEND_OBJS) $(D_GENE
>  
>  d_OBJS = $(D_ALL_OBJS) d/d-spec.o
>  
> -d21$(exeext): $(D_ALL_OBJS) attribs.o $(BACKEND) $(LIBDEPS)
> +d21$(exeext): $(D_ALL_OBJS) attribs.o $(BACKEND) $(LIBDEPS) d.prev
>  	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
>  		$(D_ALL_OBJS) attribs.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
>  
> --- gcc/doc/install.texi.jj	2020-08-25 07:15:43.982903777 +0200
> +++ gcc/doc/install.texi	2020-09-03 16:16:41.403789530 +0200
> @@ -1533,6 +1533,13 @@ When building GCC, use a mutex to avoid
>  multiple languages at the same time, to avoid thrashing on build
>  systems with limited free memory.  The default is not to use such a mutex.
>  
> +@item --enable-link-serialization
> +When building GCC, use make dependencies to serialize linking the compilers for
> +multiple languages, to avoid thrashing on build
> +systems with limited free memory.  The default is not to add such
> +dependencies and thus with parallel make potentially link different
> +compilers concurrently.
> +
>  @item --enable-maintainer-mode
>  The build rules that regenerate the Autoconf and Automake output files as
>  well as the GCC master message catalog @file{gcc.pot} are normally
> --- configure.jj	2020-08-24 10:00:01.233259699 +0200
> +++ configure	2020-09-03 16:09:58.121679071 +0200
> @@ -680,6 +680,7 @@ extra_host_zlib_configure_flags
>  extra_host_libiberty_configure_flags
>  stage1_languages
>  host_shared
> +DO_LINK_SERIALIZATION
>  extra_linker_plugin_flags
>  extra_linker_plugin_configure_flags
>  islinc
> @@ -819,6 +820,7 @@ enable_isl_version_check
>  enable_lto
>  enable_linker_plugin_configure_flags
>  enable_linker_plugin_flags
> +enable_link_serialization
>  enable_host_shared
>  enable_stage1_languages
>  enable_objc_gc
> @@ -1538,6 +1540,10 @@ Optional Features:
>    --enable-linker-plugin-flags=FLAGS
>                            additional flags for configuring and building linker
>                            plugins [none]
> +  --enable-link-serialization
> +                          avoid linking multiple GCC front-ends at once using
> +                          make dependencies to avoid thrashing on the build
> +                          machine
>    --enable-host-shared    build host code as shared libraries
>    --enable-stage1-languages[=all]
>                            choose additional languages to build during stage1.
> @@ -8264,6 +8270,26 @@ fi
>  
>  
>  
> +
> +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to serialize linking of multiple front-ends" >&5
> +$as_echo_n "checking whether to serialize linking of multiple front-ends... " >&6; }
> +  # Check whether --enable-link-serialization was given.
> +if test "${enable_link_serialization+set}" = set; then :
> +  enableval=$enable_link_serialization; do_link_serialization=$enableval
> +else
> +  do_link_serialization=no
> +fi
> +
> +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $do_link_serialization" >&5
> +$as_echo "$do_link_serialization" >&6; }
> +
> +if test "$do_link_serialization" = "yes"; then
> +   DO_LINK_SERIALIZATION=true
> +else
> +   DO_LINK_SERIALIZATION=false
> +fi
> +
> +
>  # Enable --enable-host-shared.
>  # Checked early to determine whether jit is an 'all' language
>  # Check whether --enable-host-shared was given.
> --- Makefile.in.jj	2020-04-09 21:19:04.790224960 +0200
> +++ Makefile.in	2020-09-03 16:14:29.543715216 +0200
> @@ -979,7 +979,8 @@ TARGET_FLAGS_TO_PASS = $(BASE_FLAGS_TO_P
>  EXTRA_GCC_FLAGS = \
>  	"GCC_FOR_TARGET=$(GCC_FOR_TARGET)" \
>  	"`echo 'STMP_FIXPROTO=$(STMP_FIXPROTO)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`" \
> -	"`echo 'LIMITS_H_TEST=$(LIMITS_H_TEST)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`"
> +	"`echo 'LIMITS_H_TEST=$(LIMITS_H_TEST)' | sed -e s'/[^=][^=]*=$$/XFOO=/'`" \
> +	"DO_LINK_SERIALIZATION=@DO_LINK_SERIALIZATION@"
>  
>  GCC_FLAGS_TO_PASS = $(BASE_FLAGS_TO_PASS) $(EXTRA_HOST_FLAGS) $(EXTRA_GCC_FLAGS)
>  
> --- gcc/configure.jj	2020-08-26 17:09:45.828254723 +0200
> +++ gcc/configure	2020-09-03 16:05:40.070447656 +0200
> @@ -30209,6 +30209,22 @@ do
>  	echo "lang.$t: $x" >> Make-hooks
>  done
>  
> +echo "ifeq (\$(DO_LINK_SERIALIZATION),true)" >> Make-hooks
> +prev=c
> +for lang in $all_selected_languages
> +do
> +	test $lang = c && continue
> +	echo "$lang.prev: $prev.serial" >> Make-hooks
> +	prev=$lang
> +done
> +echo else >> Make-hooks
> +for lang in $all_selected_languages
> +do
> +	test $lang = c && continue
> +	echo "$lang.prev:" >> Make-hooks
> +done
> +echo endif >> Make-hooks
> +
>  # --------
>  # Option include files
>  # --------
> 
> 

	Jakub

On Thu, Sep 3, 2020 at 10:49 AM Jakub Jelinek via Gcc-patches
<gcc-patches@gcc.gnu.org> wrote:
>
> On Thu, Sep 03, 2020 at 03:53:35PM +0200, Richard Biener wrote:
> > On Thu, 3 Sep 2020, Jakub Jelinek wrote:
> > But is that an issue in practice?  I usually do not do make -j32 cc1plus
> > in a tree that was configured for bootstrap, nor do I use
> > --enable-link-serialization in that case.
>
> Guess most often true, but one could still do it when debugging some
> particular problem during bootstrap.
>
> > Btw, do you plan to keep --enable-link-mutex the way it is?  What it
> > provides ontop of --enable-link-serialization is that progress
> > metering for very long running LTO link stages.  Not sure if that
> > works for the last LTO link or only when there's some other link
> > waiting.

Only when there's another link waiting.

> I kept it as is for now.  Primarily I didn't want to use
> --enable-linux-mutex for the new thing when it has nothing to do with a
> mutex.  I think with both options together it will just print useless
> message that it acquired the lock immediately in each case.

Yes.

For this to replace --enable-link-mutex for me, I'd want it to work
with plain 'make' in the gcc directory, not just at top level.

Jason