| Message ID | 1430309142-36242-1-git-send-email-kaszak@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
Karoly, On Wed, Apr 29, 2015 at 8:05 AM, Karoly Kasza <kaszak@gmail.com> wrote: > This patch adds a wrapper script to call ccache, which will differentiate > if the HOST or either internal or external TARGET compiler is called. > > Ccache needs to know what compiler we use, but can be simply fooled to fetch > invalid objects. To circumvent this, it uses different algorythms to > distiguish the compilers. This needs to be tuned for Buildroot's usage, > as the internal toolchains' characteristics change with every recompilation, > while ccache can not directly access the external toolchains (symlinks). > > This patch does the following: > - Reenable using mtime (hash of binary size and modification date) as the > default compiler_check algorythm. This will be used for the host compiler. > - Renames the ccache binary, so the wrapper script can be installed on it's > place. > - Creates a wrapper script with hardcoded values for performance, which will > disable the mtime compiler check and add an extra file to hash if it is called > for a target compiler (either internal or external). It is basically all the > same if we are using internal or external toolchains, ccache only sees a path. > - Creates a checksum file for the above method from the extract of the current > Buildroot config, which should not change when configuring packages or > recompiling the whole project. However it will change if the compiler options > change, thus safely pointing to cached objects. > > Signed-off-by: Karoly Kasza <kaszak@gmail.com> > --- > Config.in | 7 ------- > package/ccache/ccache.mk | 25 ++++++++++++++++++++----- > 2 files changed, 20 insertions(+), 12 deletions(-) > > diff --git a/Config.in b/Config.in > index 2b39d6a..6e7f722 100644 > --- a/Config.in > +++ b/Config.in > @@ -257,13 +257,6 @@ config BR2_CCACHE > up future builds. By default, the cache is stored in > $HOME/.buildroot-ccache. > > - Note that Buildroot does not try to invalidate the cache > - contents when the compiler changes in an incompatible > - way. Therefore, if you make a change to the compiler version > - and/or configuration, you are responsible for purging the > - ccache cache by removing the $HOME/.buildroot-ccache > - directory. > - > if BR2_CCACHE > > config BR2_CCACHE_DIR > diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk > index 52b5c67..ee9085d 100644 > --- a/package/ccache/ccache.mk > +++ b/package/ccache/ccache.mk > @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib > # is already used by autotargets for the ccache package. > # BR_CACHE_DIR is exported by Makefile based on config option > # BR2_CCACHE_DIR. > -# - ccache shouldn't use the compiler binary mtime to detect a change in > -# the compiler, because in the context of Buildroot, that completely > -# defeats the purpose of ccache. Of course, that leaves the user > -# responsible for purging its cache when the compiler changes. > # - Change hard-coded last-ditch default to match path in .config, to avoid > # the need to specify BR_CACHE_DIR when invoking ccache directly. > +# - Rename ccache binary to ccache.bin, so later the wrapper can be safely > +# named ccache, as referred everywhere > define HOST_CCACHE_PATCH_CONFIGURATION > sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c > - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c > sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c > + sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h > endef > > HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION > @@ -58,6 +56,23 @@ endef > HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP > endif > > +# Install a wrapper script, which, when using the TARGET_ toolchain, will make > +# ccache use the hash of the toolchain options instead of the compiler's mtime > +define HOST_CCACHE_INSTALL_WRAPPER > + mv $(CCACHE) $(CCACHE).bin > + echo "#!/bin/sh" > $(CCACHE) > + echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE) > + echo " [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE) > + echo " export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE) > + echo "fi" >> $(CCACHE) > + echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE) > + grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \ > + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash Does the list used for grep need these as well, as they influence the ext-toolchain-wrapper.c? * BR2_TARGET_OPTIMIZATION * BR2_SOFT_FLOAT Danomi - > + chmod 755 $(CCACHE) > +endef > + > +HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER > + > $(eval $(host-autotools-package)) > > ifeq ($(BR2_CCACHE),y) > -- > 1.7.10.4 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Hi Danomi, > + grep > "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" > $(BR2_CONFIG) | \ > > + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash > > Does the list used for grep need these as well, as they influence the > ext-toolchain-wrapper.c? > > * BR2_TARGET_OPTIMIZATION > * BR2_SOFT_FLOAT > > This patch don't really calculates with ext-toolchain-wrapper.c, but it's a good question if these config options should be factored in, I am not sure. If these are passed at compilation time to the target compiler (by the wrapper binary), then theoretically ccache gets them and calculates them in, as ccache is not influenced by the wrapper (mtime turned off) - so we don't need them in the hash file. If the target compiler binary changes based on their values, then they should be calculated in - I think this doesn't happen, as the external binaries won't be changed. Opinions? (It's not a be a big deal to add them also into the grep line, but if we restrict the hash too much, then it won't match the cache hits even when it should. If we don't restrict it enough, it may generate false cache hits and through that corrupted binaries.) BR Karoly
On 04/30/15 20:17, Károly Kasza wrote: > Hi Danomi, > > > + grep > "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" > $(BR2_CONFIG) | \ > > + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash > > Does the list used for grep need these as well, as they influence the > ext-toolchain-wrapper.c? > > * BR2_TARGET_OPTIMIZATION > * BR2_SOFT_FLOAT > > > This patch don't really calculates with ext-toolchain-wrapper.c, but it's a good > question if these config options should be factored in, I am not sure. If these > are passed at compilation time to the target compiler (by the wrapper binary), > then theoretically ccache gets them and calculates them in, as ccache is not > influenced by the wrapper (mtime turned off) - so we don't need them in the hash > file. The wrapper adds these values, and ccache calls the wrapper. So if these values change, ccache won't notice it (at least if the wrapper is used). The list of affected options is actually a lot longer I'm afraid. I think this is the full list of options affecting the compiler. Some of these are Config.in options, some of these are options calculated in the makefiles, but all of them are available to make. Options that affect the build of the compiler (internal toolchain): BR2_GCC_TARGET_ARCH BR2_GCC_TARGET_ABI BR2_GCC_TARGET_CPU BR2_GCC_TARGET_CPU_REVISION BR2_GCC_TARGET_FPU BR2_GCC_TARGET_FLOAT_ABI BR2_GCC_TARGET_MODE BR2_EXTRA_GCC_CONFIG_OPTIONS BR2_SOFT_FLOAT BR2_PTHREADS_* BR2_TOOLCHAIN_BUILDROOT_LIBC BR2_TOOLCHAIN_BUILDROOT_WCHAR BR2_GCC_ENABLE_TLS BR2_GCC_ENABLE_LTO BR2_GCC_ENABLE_GRAPHITE BR2_GCC_ENABLE_OPENMP (probably not needed, only affects headers/libraries) BR2_GCC_ENABLE_LIBMUDFLAP (probably not needed, only affects headers/libraries) GCC_VERSION The contents of package/gcc/$(GCC_VERSION)/* The contents of $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\ $(BR2_XTENSA_CORE_NAME)).tar The contents of package/gcc/gcc.mk (although that may be too conservative) The contents of package/gcc/gcc-final.mk (although that may be too conservative) Note that the libc options are not relevant, since they will affect only libraries and header files. ccache doesn't deal with libraries, and ccache will still fingerprint header files. A simpler way of getting the correct fingerprint would be to look at HOST_GCC_FINAL_CONF_OPTS, after removing all occurences of $(HOST_DIR) and $(TARGET_DIR) from it. It's slightly too conservative since some unimportant options (e.g. BR2_INSTALL_LIBSTDCPP) still appear in there, but on the other hand it's more future safe since any option that affects the compiler will appear in it. Options that affect external toolchains: The BR2_GCC_TARGET_* options mentioned above BR2_TARGET_OPTIMIZATION BR2_SOFT_FLOAT The contents of $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) The contents of $(BR2_TOOLCHAIN_EXTERNAL_PATH) (if preinstalled) As it happens, all of the flags that affect the toolchain wrapper are already in TOOLCHAIN_EXTERNAL_CFLAGS. So I think the easiest solution is to calculate the ccache hash in two places: - In HOST_GCC_FINAL_POST_PATCH_HOOKS, calculate a hash of the contents of $(@D) and $(subst $(HOST_DIR),@,$(HOST_GCC_FINAL_CONF_OPTS)) - In TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER, calculate a hash of $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) or of $(BR2_TOOLCHAIN_EXTERNAL_PATH) and $(TOOLCHAIN_EXTERNAL_CFLAGS) Obviously, this needs a bit of experimentation to see if it really works. Regards, Arnout > If the target compiler binary changes based on their values, then they should be > calculated in - I think this doesn't happen, as the external binaries won't be > changed. > Opinions? > > (It's not a be a big deal to add them also into the grep line, but if we > restrict the hash too much, then it won't match the cache hits even when it > should. If we don't restrict it enough, it may generate false cache hits and > through that corrupted binaries.) > > BR > Karoly > > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot >
On 04/29/15 14:05, Karoly Kasza wrote: > This patch adds a wrapper script to call ccache, which will differentiate > if the HOST or either internal or external TARGET compiler is called. > > Ccache needs to know what compiler we use, but can be simply fooled to fetch > invalid objects. To circumvent this, it uses different algorythms to > distiguish the compilers. This needs to be tuned for Buildroot's usage, > as the internal toolchains' characteristics change with every recompilation, > while ccache can not directly access the external toolchains (symlinks). > > This patch does the following: > - Reenable using mtime (hash of binary size and modification date) as the > default compiler_check algorythm. This will be used for the host compiler. > - Renames the ccache binary, so the wrapper script can be installed on it's > place. > - Creates a wrapper script with hardcoded values for performance, which will > disable the mtime compiler check and add an extra file to hash if it is called > for a target compiler (either internal or external). It is basically all the > same if we are using internal or external toolchains, ccache only sees a path. > - Creates a checksum file for the above method from the extract of the current > Buildroot config, which should not change when configuring packages or > recompiling the whole project. However it will change if the compiler options > change, thus safely pointing to cached objects. > > Signed-off-by: Karoly Kasza <kaszak@gmail.com> > --- > Config.in | 7 ------- > package/ccache/ccache.mk | 25 ++++++++++++++++++++----- > 2 files changed, 20 insertions(+), 12 deletions(-) > > diff --git a/Config.in b/Config.in > index 2b39d6a..6e7f722 100644 > --- a/Config.in > +++ b/Config.in > @@ -257,13 +257,6 @@ config BR2_CCACHE > up future builds. By default, the cache is stored in > $HOME/.buildroot-ccache. > > - Note that Buildroot does not try to invalidate the cache > - contents when the compiler changes in an incompatible > - way. Therefore, if you make a change to the compiler version > - and/or configuration, you are responsible for purging the > - ccache cache by removing the $HOME/.buildroot-ccache > - directory. > - > if BR2_CCACHE > > config BR2_CCACHE_DIR > diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk > index 52b5c67..ee9085d 100644 > --- a/package/ccache/ccache.mk > +++ b/package/ccache/ccache.mk > @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib > # is already used by autotargets for the ccache package. > # BR_CACHE_DIR is exported by Makefile based on config option > # BR2_CCACHE_DIR. > -# - ccache shouldn't use the compiler binary mtime to detect a change in > -# the compiler, because in the context of Buildroot, that completely > -# defeats the purpose of ccache. Of course, that leaves the user > -# responsible for purging its cache when the compiler changes. > # - Change hard-coded last-ditch default to match path in .config, to avoid > # the need to specify BR_CACHE_DIR when invoking ccache directly. > +# - Rename ccache binary to ccache.bin, so later the wrapper can be safely > +# named ccache, as referred everywhere > define HOST_CCACHE_PATCH_CONFIGURATION > sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c > - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c > sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c > + sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h > endef > > HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION > @@ -58,6 +56,23 @@ endef > HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP > endif > > +# Install a wrapper script, which, when using the TARGET_ toolchain, will make > +# ccache use the hash of the toolchain options instead of the compiler's mtime > +define HOST_CCACHE_INSTALL_WRAPPER > + mv $(CCACHE) $(CCACHE).bin > + echo "#!/bin/sh" > $(CCACHE) > + echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE) > + echo " [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE) > + echo " export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE) > + echo "fi" >> $(CCACHE) > + echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE) I don't really like this additional wrapper script. Can't we just define two variables, HOST_CCACHE and TARGET_CCACHE: HOST_CCACHE = $(HOST_DIR)/usr/bin/ccache TARGET_CCACHE = \ CCACHE_COMPILERCHECK=none \ CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash \ $(HOST_DIR)/usr/bin/ccache TARGET_CC := $(TARGET_CCACHE) $(TARGET_CC) (and subsitute the other references to $(CCACHE) as appropriate) The handling of cmake and qmake will require a bit of thinking however. If you do keep the wrapper script, then I'd prefer if you create it as package/ccache/ccache_wrapper.in and fill in the relevant variables with sed. > + grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \ > + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash Since CCACHE_EXTRAFILES is anyway taken together with the input file into one big hash, I wouldn't bother with hashing it here, but just put the actual options to be hashed in that file. That gives a good reference when you want to debug ccache, and the speed penalty is negligible (the overhead of opening the file is larger than reading and hashing it). Regards, Arnout > + chmod 755 $(CCACHE) > +endef > + > +HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER > + > $(eval $(host-autotools-package)) > > ifeq ($(BR2_CCACHE),y) >
Hi Karoly, On Fri, May 1, 2015 at 1:05 AM, Arnout Vandecappelle <arnout@mind.be> wrote: > On 04/29/15 14:05, Karoly Kasza wrote: >> This patch adds a wrapper script to call ccache, which will differentiate >> if the HOST or either internal or external TARGET compiler is called. >> >> Ccache needs to know what compiler we use, but can be simply fooled to fetch >> invalid objects. To circumvent this, it uses different algorythms to >> distiguish the compilers. This needs to be tuned for Buildroot's usage, >> as the internal toolchains' characteristics change with every recompilation, >> while ccache can not directly access the external toolchains (symlinks). >> >> This patch does the following: >> - Reenable using mtime (hash of binary size and modification date) as the >> default compiler_check algorythm. This will be used for the host compiler. >> - Renames the ccache binary, so the wrapper script can be installed on it's >> place. >> - Creates a wrapper script with hardcoded values for performance, which will >> disable the mtime compiler check and add an extra file to hash if it is called >> for a target compiler (either internal or external). It is basically all the >> same if we are using internal or external toolchains, ccache only sees a path. >> - Creates a checksum file for the above method from the extract of the current >> Buildroot config, which should not change when configuring packages or >> recompiling the whole project. However it will change if the compiler options >> change, thus safely pointing to cached objects. >> >> Signed-off-by: Karoly Kasza <kaszak@gmail.com> >> --- >> Config.in | 7 ------- >> package/ccache/ccache.mk | 25 ++++++++++++++++++++----- >> 2 files changed, 20 insertions(+), 12 deletions(-) >> >> diff --git a/Config.in b/Config.in >> index 2b39d6a..6e7f722 100644 >> --- a/Config.in >> +++ b/Config.in >> @@ -257,13 +257,6 @@ config BR2_CCACHE >> up future builds. By default, the cache is stored in >> $HOME/.buildroot-ccache. >> >> - Note that Buildroot does not try to invalidate the cache >> - contents when the compiler changes in an incompatible >> - way. Therefore, if you make a change to the compiler version >> - and/or configuration, you are responsible for purging the >> - ccache cache by removing the $HOME/.buildroot-ccache >> - directory. >> - >> if BR2_CCACHE >> >> config BR2_CCACHE_DIR >> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk >> index 52b5c67..ee9085d 100644 >> --- a/package/ccache/ccache.mk >> +++ b/package/ccache/ccache.mk >> @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib >> # is already used by autotargets for the ccache package. >> # BR_CACHE_DIR is exported by Makefile based on config option >> # BR2_CCACHE_DIR. >> -# - ccache shouldn't use the compiler binary mtime to detect a change in >> -# the compiler, because in the context of Buildroot, that completely >> -# defeats the purpose of ccache. Of course, that leaves the user >> -# responsible for purging its cache when the compiler changes. >> # - Change hard-coded last-ditch default to match path in .config, to avoid >> # the need to specify BR_CACHE_DIR when invoking ccache directly. >> +# - Rename ccache binary to ccache.bin, so later the wrapper can be safely >> +# named ccache, as referred everywhere >> define HOST_CCACHE_PATCH_CONFIGURATION >> sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c >> - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c >> sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c >> + sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h >> endef >> >> HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION >> @@ -58,6 +56,23 @@ endef >> HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP >> endif >> >> +# Install a wrapper script, which, when using the TARGET_ toolchain, will make >> +# ccache use the hash of the toolchain options instead of the compiler's mtime >> +define HOST_CCACHE_INSTALL_WRAPPER >> + mv $(CCACHE) $(CCACHE).bin >> + echo "#!/bin/sh" > $(CCACHE) >> + echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE) >> + echo " [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE) >> + echo " export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE) >> + echo "fi" >> $(CCACHE) >> + echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE) > > I don't really like this additional wrapper script. Can't we just define two > variables, HOST_CCACHE and TARGET_CCACHE: > > HOST_CCACHE = $(HOST_DIR)/usr/bin/ccache > TARGET_CCACHE = \ > CCACHE_COMPILERCHECK=none \ > CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash \ > $(HOST_DIR)/usr/bin/ccache > > TARGET_CC := $(TARGET_CCACHE) $(TARGET_CC) > > (and subsitute the other references to $(CCACHE) as appropriate) > > The handling of cmake and qmake will require a bit of thinking however. > > > If you do keep the wrapper script, then I'd prefer if you create it as > package/ccache/ccache_wrapper.in and fill in the relevant variables with sed. > > >> + grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \ >> + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash > > Since CCACHE_EXTRAFILES is anyway taken together with the input file into one > big hash, I wouldn't bother with hashing it here, but just put the actual > options to be hashed in that file. That gives a good reference when you want to > debug ccache, and the speed penalty is negligible (the overhead of opening the > file is larger than reading and hashing it). > > Did you have a chance to look any further into this topic ? Thanks, Thomas
Hi Thomas, all I had (have) an IRL interrupt, but I still plan to take this topic further, based on Arnout's comments. Also, ccache released a new minor version, I'll send a bump in a few days, as soon as I have time to test it properly. The new CCACHE_COMPILERCHECK=string:<value> option could provide useful after the bump. Kind regards, Karoly > > Did you have a chance to look any further into this topic ? > > Thanks, > Thomas
diff --git a/Config.in b/Config.in index 2b39d6a..6e7f722 100644 --- a/Config.in +++ b/Config.in @@ -257,13 +257,6 @@ config BR2_CCACHE up future builds. By default, the cache is stored in $HOME/.buildroot-ccache. - Note that Buildroot does not try to invalidate the cache - contents when the compiler changes in an incompatible - way. Therefore, if you make a change to the compiler version - and/or configuration, you are responsible for purging the - ccache cache by removing the $HOME/.buildroot-ccache - directory. - if BR2_CCACHE config BR2_CCACHE_DIR diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk index 52b5c67..ee9085d 100644 --- a/package/ccache/ccache.mk +++ b/package/ccache/ccache.mk @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib # is already used by autotargets for the ccache package. # BR_CACHE_DIR is exported by Makefile based on config option # BR2_CCACHE_DIR. -# - ccache shouldn't use the compiler binary mtime to detect a change in -# the compiler, because in the context of Buildroot, that completely -# defeats the purpose of ccache. Of course, that leaves the user -# responsible for purging its cache when the compiler changes. # - Change hard-coded last-ditch default to match path in .config, to avoid # the need to specify BR_CACHE_DIR when invoking ccache directly. +# - Rename ccache binary to ccache.bin, so later the wrapper can be safely +# named ccache, as referred everywhere define HOST_CCACHE_PATCH_CONFIGURATION sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c + sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h endef HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION @@ -58,6 +56,23 @@ endef HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP endif +# Install a wrapper script, which, when using the TARGET_ toolchain, will make +# ccache use the hash of the toolchain options instead of the compiler's mtime +define HOST_CCACHE_INSTALL_WRAPPER + mv $(CCACHE) $(CCACHE).bin + echo "#!/bin/sh" > $(CCACHE) + echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE) + echo " [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE) + echo " export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE) + echo "fi" >> $(CCACHE) + echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE) + grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \ + grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash + chmod 755 $(CCACHE) +endef + +HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER + $(eval $(host-autotools-package)) ifeq ($(BR2_CCACHE),y)
This patch adds a wrapper script to call ccache, which will differentiate if the HOST or either internal or external TARGET compiler is called. Ccache needs to know what compiler we use, but can be simply fooled to fetch invalid objects. To circumvent this, it uses different algorythms to distiguish the compilers. This needs to be tuned for Buildroot's usage, as the internal toolchains' characteristics change with every recompilation, while ccache can not directly access the external toolchains (symlinks). This patch does the following: - Reenable using mtime (hash of binary size and modification date) as the default compiler_check algorythm. This will be used for the host compiler. - Renames the ccache binary, so the wrapper script can be installed on it's place. - Creates a wrapper script with hardcoded values for performance, which will disable the mtime compiler check and add an extra file to hash if it is called for a target compiler (either internal or external). It is basically all the same if we are using internal or external toolchains, ccache only sees a path. - Creates a checksum file for the above method from the extract of the current Buildroot config, which should not change when configuring packages or recompiling the whole project. However it will change if the compiler options change, thus safely pointing to cached objects. Signed-off-by: Karoly Kasza <kaszak@gmail.com> --- Config.in | 7 ------- package/ccache/ccache.mk | 25 ++++++++++++++++++++----- 2 files changed, 20 insertions(+), 12 deletions(-)