| Message ID | 20181020132535.24508-1-thomas.petazzoni@bootlin.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | qt5: use ccache for building host code | expand |
Hello Peter, On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote: > qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache > when building all its host code (especially qmake). This means that > even with ccache enabled and a hot cache, it still takes a long time > to build qt5base. > > Before this patch, building qt5base takes: > > - 446 seconds with a cold ccache > - 185 seconds with a hot ccache > > This is because the ccache is not used for host code. > > After this patch, building qt5base takes: > > - 450 seconds with a cold ccache > - 15 seconds with a hot ccache > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Any comment on this patch ? Thanks! Thomas
Hello Thomas, On Sat, 3 Nov 2018 14:49:17 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Hello Peter, > > On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote: > > qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache > > when building all its host code (especially qmake). This means that > > even with ccache enabled and a hot cache, it still takes a long time > > to build qt5base. > > > > Before this patch, building qt5base takes: > > > > - 446 seconds with a cold ccache > > - 185 seconds with a hot ccache > > > > This is because the ccache is not used for host code. > > > > After this patch, building qt5base takes: > > > > - 450 seconds with a cold ccache > > - 15 seconds with a hot ccache > > > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > > Any comment on this patch ? Sorry, not an regular ccache user (maybe an mistake ;-) ), so I did not examine this patch yet, nice speedup, some questions at he patch reply... Regards, Peter > > Thanks! > > Thomas
Hello Thomas, On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache > when building all its host code (especially qmake). This means that > even with ccache enabled and a hot cache, it still takes a long time > to build qt5base. > > Before this patch, building qt5base takes: > > - 446 seconds with a cold ccache > - 185 seconds with a hot ccache > > This is because the ccache is not used for host code. > > After this patch, building qt5base takes: > > - 450 seconds with a cold ccache > - 15 seconds with a hot ccache > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/qt5/qt5base/qt5base.mk | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk > index ab2b883fc4..d9f7af7196 100644 > --- a/package/qt5/qt5base/qt5base.mk > +++ b/package/qt5/qt5base/qt5base.mk > @@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG > endef > endif > > +# This allows to use ccache when available > +define QT5BASE_CONFIGURE_HOSTCC > + $(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > + $(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > + $(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf > +endef > + With this sed commands applied, a qmake host compile is always using ccache (same behavior already for the cross/target compile?)? Regards, Peter > define QT5BASE_CONFIGURE_CMDS > mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/ > sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \ > @@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS > $(QT5BASE_CONFIGURE_CONFIG_FILE) > touch $(QT5BASE_ARCH_CONFIG_FILE) > $(QT5BASE_CONFIGURE_ARCH_CONFIG) > + $(QT5BASE_CONFIGURE_HOSTCC) > (cd $(@D); \ > $(TARGET_MAKE_ENV) \ > PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
Hello Peter, Thanks for looking at this patch! On Sun, 4 Nov 2018 22:12:20 +0100, Peter Seiderer wrote: > > +# This allows to use ccache when available > > +define QT5BASE_CONFIGURE_HOSTCC > > + $(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > > + $(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > > + $(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf > > +endef > > + > > With this sed commands applied, a qmake host compile is always using ccache (same > behavior already for the cross/target compile?)? I'm not sure to understand what you mean here. This change ensures that qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++. HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y, in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++". So basically, this patch ensures that qmake is built with what Buildroot considers to be the "host compiler", which may or may not have ccache enabled. Does this answer your question, or perhaps I misunderstood it completely ? Thanks! Thomas
Hello Thomas, On Sun, 4 Nov 2018 22:25:01 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Hello Peter, > > Thanks for looking at this patch! > > On Sun, 4 Nov 2018 22:12:20 +0100, Peter Seiderer wrote: > > > > +# This allows to use ccache when available > > > +define QT5BASE_CONFIGURE_HOSTCC > > > + $(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > > > + $(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf > > > + $(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf > > > +endef > > > + > > > > With this sed commands applied, a qmake host compile is always using ccache (same > > behavior already for the cross/target compile?)? > > I'm not sure to understand what you mean here. This change ensures that > qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++. > > HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y, > in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++". > > So basically, this patch ensures that qmake is built with what > Buildroot considers to be the "host compiler", which may or may not > have ccache enabled. > > Does this answer your question, or perhaps I misunderstood it > completely ? Thanks for the answer (you got it right)...., so in case I use the buildroot-generated qmake I will always use ccache (if configured), even if used outside of the buildroot build system, right? The $(HOSTCC) is expanded at the sed command running time, g++-base.conf will contain 'QMAKE_COMPILER=gcc' or 'QMAKE_COMPILER=ccache gcc'... Not sure if this matters for some use cases... Regards, Peter > > Thanks! > > Thomas
Hello Peter, On Sun, 4 Nov 2018 22:46:38 +0100, Peter Seiderer wrote: > Thanks for the answer (you got it right)...., so in case I use the > buildroot-generated qmake I will always use ccache (if configured), > even if used outside of the buildroot build system, right? Well, in the context of Buildroot, qmake is essentially used to build code for the target, using the cross-compiler. In this case, ccache was already used, because invoking ccache is hidden inside the toolchain wrapper, so there's no way around it :-) However, I guess with the right qmake invocations, you can make it build code for the host as well. And then yes, in this case, I assume it would now start using ccache. > The $(HOSTCC) is expanded at the sed command running time, > g++-base.conf will contain 'QMAKE_COMPILER=gcc' or > 'QMAKE_COMPILER=ccache gcc'... > > Not sure if this matters for some use cases... What are you worried about? The fact that there is a space in the QMAKE_COMPILER variable value ? Best regards, Thomas
Hello Thomas, On Mon, 5 Nov 2018 08:49:34 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Hello Peter, > > On Sun, 4 Nov 2018 22:46:38 +0100, Peter Seiderer wrote: > > > Thanks for the answer (you got it right)...., so in case I use the > > buildroot-generated qmake I will always use ccache (if configured), > > even if used outside of the buildroot build system, right? > > Well, in the context of Buildroot, qmake is essentially used to build > code for the target, using the cross-compiler. In this case, ccache was > already used, because invoking ccache is hidden inside the toolchain > wrapper, so there's no way around it :-) > > However, I guess with the right qmake invocations, you can make it > build code for the host as well. And then yes, in this case, I assume > it would now start using ccache. > So host ccache usage will behave the same as target usage..., O.k. > > The $(HOSTCC) is expanded at the sed command running time, > > g++-base.conf will contain 'QMAKE_COMPILER=gcc' or > > 'QMAKE_COMPILER=ccache gcc'... > > Ups, missed the context here, the above was only to be sure it is expanded at configure time... > > Not sure if this matters for some use cases... And this should be at the context host ccache usage... > > What are you worried about? The fact that there is a space in the > QMAKE_COMPILER variable value ? No worries, you can add my: Reviewed-by: Peter Seiderer <ps.report@gmx.net> Regards, Peter > > Best regards, > > Thomas
Hello, On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote: > qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache > when building all its host code (especially qmake). This means that > even with ccache enabled and a hot cache, it still takes a long time > to build qt5base. > > Before this patch, building qt5base takes: > > - 446 seconds with a cold ccache > - 185 seconds with a hot ccache > > This is because the ccache is not used for host code. > > After this patch, building qt5base takes: > > - 450 seconds with a cold ccache > - 15 seconds with a hot ccache > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/qt5/qt5base/qt5base.mk | 8 ++++++++ > 1 file changed, 8 insertions(+) Applied to next. Thomas
diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk index ab2b883fc4..d9f7af7196 100644 --- a/package/qt5/qt5base/qt5base.mk +++ b/package/qt5/qt5base/qt5base.mk @@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG endef endif +# This allows to use ccache when available +define QT5BASE_CONFIGURE_HOSTCC + $(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf + $(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf + $(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf +endef + define QT5BASE_CONFIGURE_CMDS mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/ sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \ @@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS $(QT5BASE_CONFIGURE_CONFIG_FILE) touch $(QT5BASE_ARCH_CONFIG_FILE) $(QT5BASE_CONFIGURE_ARCH_CONFIG) + $(QT5BASE_CONFIGURE_HOSTCC) (cd $(@D); \ $(TARGET_MAKE_ENV) \ PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base. Before this patch, building qt5base takes: - 446 seconds with a cold ccache - 185 seconds with a hot ccache This is because the ccache is not used for host code. After this patch, building qt5base takes: - 450 seconds with a cold ccache - 15 seconds with a hot ccache Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- package/qt5/qt5base/qt5base.mk | 8 ++++++++ 1 file changed, 8 insertions(+)