| Message ID | 546E4C82.2010305@freescale.com |
|---|---|
| State | RFC |
| Delegated to: | Simon Glass |
| Headers | show |
On 11/20/2014 12:18 PM, York Sun wrote: > > I don't see ARCH defined anywhere. > Simon, I think ARCH is defined in Kconfig as SYS_ARCH. Is it possible buildman runs out-of-order? York
On 11/20/2014 01:04 PM, York Sun wrote: > On 11/20/2014 12:18 PM, York Sun wrote: >> >> I don't see ARCH defined anywhere. >> > > Simon, > > I think ARCH is defined in Kconfig as SYS_ARCH. Is it possible buildman runs > out-of-order? > There is something. When the server is under heavy load, this error doesn't show. When the server is mostly idling, the error comes. Any suggestion to debug it? York
On 11/20/2014 01:33 PM, York Sun wrote: > On 11/20/2014 01:04 PM, York Sun wrote: >> On 11/20/2014 12:18 PM, York Sun wrote: >>> >>> I don't see ARCH defined anywhere. >>> >> >> Simon, >> >> I think ARCH is defined in Kconfig as SYS_ARCH. Is it possible buildman runs >> out-of-order? >> > > There is something. When the server is under heavy load, this error doesn't > show. When the server is mostly idling, the error comes. Any suggestion to debug it? Scratch that. It is not consistent. York
Hi York, On 20 November 2014 21:34, York Sun <yorksun@freescale.com> wrote: > On 11/20/2014 01:33 PM, York Sun wrote: >> On 11/20/2014 01:04 PM, York Sun wrote: >>> On 11/20/2014 12:18 PM, York Sun wrote: >>>> >>>> I don't see ARCH defined anywhere. >>>> >>> >>> Simon, >>> >>> I think ARCH is defined in Kconfig as SYS_ARCH. Is it possible buildman runs >>> out-of-order? >>> >> >> There is something. When the server is under heavy load, this error doesn't >> show. When the server is mostly idling, the error comes. Any suggestion to debug it? > > Scratch that. It is not consistent. It's a mystery to me. I haven't seen this problem in much usage and can only suggest digging in to figure it out. Regards, Simon
On 11/20/2014 01:46 PM, Simon Glass wrote: > Hi York, > > On 20 November 2014 21:34, York Sun <yorksun@freescale.com> wrote: >> On 11/20/2014 01:33 PM, York Sun wrote: >>> On 11/20/2014 01:04 PM, York Sun wrote: >>>> On 11/20/2014 12:18 PM, York Sun wrote: >>>>> >>>>> I don't see ARCH defined anywhere. >>>>> >>>> >>>> Simon, >>>> >>>> I think ARCH is defined in Kconfig as SYS_ARCH. Is it possible buildman runs >>>> out-of-order? >>>> >>> >>> There is something. When the server is under heavy load, this error doesn't >>> show. When the server is mostly idling, the error comes. Any suggestion to debug it? >> >> Scratch that. It is not consistent. > > It's a mystery to me. I haven't seen this problem in much usage and > can only suggest digging in to figure it out. > Found something but not the root cause yet. For the failure run, it runs in Makefile, passed include/config/uboot.release, but not anything under "prepare3:". For the success run, it continues to run. -bash-3.2$ more log make[1]: Entering directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting- pull/working_upstream_qoriq/.bm-work/00/build' GEN ./Makefile scripts/kconfig/conf --silentoldconfig Kconfig CHK include/config.h UPD include/config.h GEN include/autoconf.mk GEN include/autoconf.mk.dep make[1]: Leaving directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-p ull/working_upstream_qoriq/.bm-work/00/build' make[1]: Entering directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting- pull/working_upstream_qoriq/.bm-work/00/build' echo "York debug here" York debug here CHK include/config/uboot.release CHK include/generated/timestamp_autogenerated.h GEN ./Makefile UPD include/generated/timestamp_autogenerated.h UPD include/config/uboot.release echo "Done uboot.release" Done uboot.release (note, for the success run, the log would have "Using .. as source for U-Boot" here) make[1]: Leaving directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-p ull/working_upstream_qoriq/.bm-work/00/build' Not sure if this is still buildman related, but only visible when running with buildman. York
On 11/20/2014 02:25 PM, York Sun wrote: > Not sure if this is still buildman related, but only visible when running with > buildman. Simon, It seems related to out-of-tree building. For the failure case, the dependency "include/config/uboot.release" completes but not continue to run. It seems to be related to out of tree building. When I use buildman, it use out-of-tree building by default. I can reproduce the issue by using make directly with O=<path>. I don't understand how out-of-tree building would inconsistently fail. -bash-3.2$ CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all make[1]: Entering directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' CLEAN tools CLEAN tools/lib tools/common CLEAN scripts/basic CLEAN scripts/kconfig CLEAN u-boot.lds CLEAN include/config include/generated CLEAN .config include/autoconf.mk include/autoconf.mk.dep include/config.h HOSTCC scripts/basic/fixdep GEN ./Makefile HOSTCC scripts/kconfig/conf.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/zconf.lex.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c: In function ‘get_symbol_str’: /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c:551: warning: ‘jump’ may be used uninitialized in this function HOSTLD scripts/kconfig/conf # # configuration written to .config # GEN ./Makefile scripts/kconfig/conf --silentoldconfig Kconfig CHK include/config.h UPD include/config.h GEN include/autoconf.mk GEN include/autoconf.mk.dep CHK include/config/uboot.release GEN ./Makefile CHK include/generated/timestamp_autogenerated.h make[2]: *** No rule to make target `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/arch//cpu/u-boot.lds', needed by `u-boot.lds'. Stop. make[2]: *** Waiting for unfinished jobs.... UPD include/generated/timestamp_autogenerated.h UPD include/config/uboot.release make[1]: *** [__build_one_by_one] Error 2 make[1]: Leaving directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' make: *** [sub-make] Error 2 For the passed case, the log is -bash-3.2$ CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all make[1]: Entering directory `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' CLEAN scripts/basic CLEAN scripts/kconfig CLEAN include/config include/generated CLEAN .config include/autoconf.mk include/autoconf.mk.dep include/config.h HOSTCC scripts/basic/fixdep GEN ./Makefile HOSTCC scripts/kconfig/conf.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/zconf.lex.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c: In function ‘get_symbol_str’: /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c:551: warning: ‘jump’ may be used uninitialized in this function HOSTLD scripts/kconfig/conf # # configuration written to .config # GEN ./Makefile scripts/kconfig/conf --silentoldconfig Kconfig CHK include/config.h UPD include/config.h GEN include/autoconf.mk GEN include/autoconf.mk.dep CHK include/config/uboot.release CHK include/generated/timestamp_autogenerated.h GEN ./Makefile UPD include/generated/timestamp_autogenerated.h UPD include/config/uboot.release Using /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace as source for U-Boot CHK include/generated/version_autogenerated.h UPD include/generated/version_autogenerated.h CC lib/asm-offsets.s GEN include/generated/generic-asm-offsets.h LDS u-boot.lds HOSTCC tools/envcrc.o WRAP tools/lib/crc32.c WRAP tools/common/env_embedded.c WRAP tools/lib/sha1.c HOSTCC tools/gen_eth_addr HOSTCC tools/img2srec HOSTCC tools/mkenvimage.o HOSTCC tools/os_support.o (and keep going) York
+Masahiro who may have some ideas Hi York, On 21 November 2014 00:35, York Sun <yorksun@freescale.com> wrote: > On 11/20/2014 02:25 PM, York Sun wrote: >> Not sure if this is still buildman related, but only visible when running with >> buildman. > > Simon, > > It seems related to out-of-tree building. > > For the failure case, the dependency "include/config/uboot.release" completes > but not continue to run. It seems to be related to out of tree building. When I > use buildman, it use out-of-tree building by default. I can reproduce the issue > by using make directly with O=<path>. I don't understand how out-of-tree > building would inconsistently fail. > > > -bash-3.2$ > CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- > make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all > make[1]: Entering directory > `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' > CLEAN tools > CLEAN tools/lib tools/common > CLEAN scripts/basic > CLEAN scripts/kconfig > CLEAN u-boot.lds > CLEAN include/config include/generated > CLEAN .config include/autoconf.mk include/autoconf.mk.dep include/config.h > HOSTCC scripts/basic/fixdep > GEN ./Makefile > HOSTCC scripts/kconfig/conf.o > SHIPPED scripts/kconfig/zconf.tab.c > SHIPPED scripts/kconfig/zconf.lex.c > SHIPPED scripts/kconfig/zconf.hash.c > HOSTCC scripts/kconfig/zconf.tab.o > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c: > In function ‘get_symbol_str’: > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c:551: > warning: ‘jump’ may be used uninitialized in this function > HOSTLD scripts/kconfig/conf > # > # configuration written to .config > # > GEN ./Makefile > scripts/kconfig/conf --silentoldconfig Kconfig > CHK include/config.h > UPD include/config.h > GEN include/autoconf.mk > GEN include/autoconf.mk.dep > CHK include/config/uboot.release > GEN ./Makefile > CHK include/generated/timestamp_autogenerated.h > make[2]: *** No rule to make target > `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/arch//cpu/u-boot.lds', > needed by `u-boot.lds'. Stop. > make[2]: *** Waiting for unfinished jobs.... > UPD include/generated/timestamp_autogenerated.h > UPD include/config/uboot.release > make[1]: *** [__build_one_by_one] Error 2 > make[1]: Leaving directory > `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' > make: *** [sub-make] Error 2 > > > > For the passed case, the log is > > -bash-3.2$ > CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- > make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all > make[1]: Entering directory > `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/working_upstream_qoriq' > CLEAN scripts/basic > CLEAN scripts/kconfig > CLEAN include/config include/generated > CLEAN .config include/autoconf.mk include/autoconf.mk.dep include/config.h > HOSTCC scripts/basic/fixdep > GEN ./Makefile > HOSTCC scripts/kconfig/conf.o > SHIPPED scripts/kconfig/zconf.tab.c > SHIPPED scripts/kconfig/zconf.lex.c > SHIPPED scripts/kconfig/zconf.hash.c > HOSTCC scripts/kconfig/zconf.tab.o > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c: > In function ‘get_symbol_str’: > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c:551: > warning: ‘jump’ may be used uninitialized in this function > HOSTLD scripts/kconfig/conf > # > # configuration written to .config > # > GEN ./Makefile > scripts/kconfig/conf --silentoldconfig Kconfig > CHK include/config.h > UPD include/config.h > GEN include/autoconf.mk > GEN include/autoconf.mk.dep > CHK include/config/uboot.release > CHK include/generated/timestamp_autogenerated.h > GEN ./Makefile > UPD include/generated/timestamp_autogenerated.h > UPD include/config/uboot.release > Using > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace as > source for U-Boot > CHK include/generated/version_autogenerated.h > UPD include/generated/version_autogenerated.h > CC lib/asm-offsets.s > GEN include/generated/generic-asm-offsets.h > LDS u-boot.lds > HOSTCC tools/envcrc.o > WRAP tools/lib/crc32.c > WRAP tools/common/env_embedded.c > WRAP tools/lib/sha1.c > HOSTCC tools/gen_eth_addr > HOSTCC tools/img2srec > HOSTCC tools/mkenvimage.o > HOSTCC tools/os_support.o > (and keep going) This may be unrelated to buildman, but to check you can use the -i flag to enable an in-tree build. Regards, Simon
Hi York, Simon, On Fri, 21 Nov 2014 05:14:02 +0100 Simon Glass <sjg@chromium.org> wrote: > +Masahiro who may have some ideas > > Hi York, > > On 21 November 2014 00:35, York Sun <yorksun@freescale.com> wrote: > > On 11/20/2014 02:25 PM, York Sun wrote: > >> Not sure if this is still buildman related, but only visible when running with > >> buildman. > > > > Simon, > > > > It seems related to out-of-tree building. > > > > For the failure case, the dependency "include/config/uboot.release" completes > > but not continue to run. It seems to be related to out of tree building. When I > > use buildman, it use out-of-tree building by default. I can reproduce the issue > > by using make directly with O=<path>. I don't understand how out-of-tree > > building would inconsistently fail. > > > > > > -bash-3.2$ > > CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- > > make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all Please do not run "mrproper" with "*_defconfig" and "all". Doing "_defconfig" and "all" at one time is supported, but you must "clean" or "mrproper" separately. This is the upstream limitation. Linux Kernel does not support it either. Without -j* option, "make mrproper *_defconfig all" works, of course, because they are run one by one in the given order. With -j* option, it would not work because the build system removes some files while it compiles other objects. I guess this is the root cause of the inconsistent fails you have seen. Best Regards Masahiro Yamada
Hi York, Simon, On Fri, 21 Nov 2014 13:42:05 +0900 Masahiro Yamada <yamada.m@jp.panasonic.com> wrote: > > Hi York, > > > > On 21 November 2014 00:35, York Sun <yorksun@freescale.com> wrote: > > > On 11/20/2014 02:25 PM, York Sun wrote: > > >> Not sure if this is still buildman related, but only visible when running with > > >> buildman. > > > > > > Simon, > > > > > > It seems related to out-of-tree building. > > > > > > For the failure case, the dependency "include/config/uboot.release" completes > > > but not continue to run. It seems to be related to out of tree building. When I > > > use buildman, it use out-of-tree building by default. I can reproduce the issue > > > by using make directly with O=<path>. I don't understand how out-of-tree > > > building would inconsistently fail. > > > > > > > > > -bash-3.2$ > > > CROSS_COMPILE=/opt/fsl-networking/QorIQ-SDK-V1.4/sysroots/x86_64-fsl_networking_sdk-linux/usr/bin/ppce500mc-fsl_networking-linux/powerpc-fsl_networking-linux- > > > make -j24 O=../working_upstream_qoriq/ mrproper MPC832XEMDS_HOST_66_defconfig all > > > Please do not run "mrproper" with "*_defconfig" and "all". > > > Doing "_defconfig" and "all" at one time is supported, > but you must "clean" or "mrproper" separately. > This is the upstream limitation. Linux Kernel does not support it either. > > > Without -j* option, "make mrproper *_defconfig all" works, of course, > because they are run one by one in the given order. > > With -j* option, it would not work because > the build system removes some files while it compiles other objects. > > I guess this is the root cause of the inconsistent fails you have seen. Sorry, this was my misunderstanding. "make mrproper *_defconfig all" works. Then, it is still a mystery... Best Regards Masahiro Yamada
Thanks for the hint. I should point out buildman does the three steps separately, and only apply -j to the last step. But I still see the failure. York
Hi York, I have not been able to reproduce this issue on my box, but the cause surely resides on Kbuild itself. It is not related to buildman at all. > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c: > In function 組et_symbol_str・ > /local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/scripts/kconfig/menu.c:551: > warning: 阻ump・may be used uninitialized in this function > HOSTLD scripts/kconfig/conf > # > # configuration written to .config > # > GEN ./Makefile > scripts/kconfig/conf --silentoldconfig Kconfig > CHK include/config.h > UPD include/config.h > GEN include/autoconf.mk > GEN include/autoconf.mk.dep > CHK include/config/uboot.release > GEN ./Makefile > CHK include/generated/timestamp_autogenerated.h > make[2]: *** No rule to make target > `/local/jenkins/jobs/york-scm-test-upstream-before-requesting-pull/workspace/arch//cpu/u-boot.lds', > needed by `u-boot.lds'. Stop. According to this fail log, "ARCH" has not been defined yet when LDSCRIPT is evaluated. # If there is no specified link script, we look in a number of places for it ifndef LDSCRIPT ifeq ($(wildcard $(LDSCRIPT)),) LDSCRIPT := $(srctree)/board/$(BOARDDIR)/u-boot.lds endif ifeq ($(wildcard $(LDSCRIPT)),) LDSCRIPT := $(srctree)/$(CPUDIR)/u-boot.lds endif ifeq ($(wildcard $(LDSCRIPT)),) LDSCRIPT := $(srctree)/arch/$(ARCH)/cpu/u-boot.lds endif endif I am not sure the real cause but it looks like the following code does not work on *some* situation. autoconf_is_current := $(if $(wildcard $(KCONFIG_CONFIG)),$(shell find . \ -path ./include/config/auto.conf -newer $(KCONFIG_CONFIG))) ifneq ($(autoconf_is_current),) include $(srctree)/config.mk endif Now I have realized that it is the problem I am responsible to fix, but it might not be easy enough to change LDSCRIPT stuff. So, my question is, how much time can you give me to fix it? Can you bare with it one month or so? Best Regards Masahiro Yamada
Masahiro, Sorry for repeatly sending this email. My previous reply was rejected by the mailing list due to some weird base64 encoding. I tend to agree with you since I could reproduce it using make. I want to share some of my debug. There is a label in Makefile, prepare3. When it runs correctly, the code under that label runs after the dependency "include/config/uboot.release". I can inject some echo to see the progress. However, for the failed run (no change of code or command, just repeatedly run), nothing runs after the dependency " include/config/uboot.release". The host is Ubuntu 12.04 x86_64. I am not sure if the inconsistency is cause by Kbuild. York
On 11/20/2014 10:47 PM, York Sun wrote: > Masahiro, > > Sorry for repeatly sending this email. My previous reply was rejected by the mailing list due to some weird base64 encoding. > > I tend to agree with you since I could reproduce it using make. > I want to share some of my debug. There is a label in Makefile, prepare3. When it runs correctly, the code under that label runs after the dependency "include/config/uboot.release". I can inject some echo to see the progress. However, for the failed run (no change of code or command, just repeatedly run), nothing runs after the dependency " > include/config/uboot.release". The host is Ubuntu 12.04 x86_64. I am not sure if the inconsistency is cause by Kbuild. > Sorry, I was mistaken about the host OS. It is CentOS 5.11, if that makes any difference. York
On 11/20/2014 08:14 PM, Simon Glass wrote: >> (and keep going) > > This may be unrelated to buildman, but to check you can use the -i > flag to enable an in-tree build. > Thanks for the tip, but it didn't help. I repeatedly run 10 times. 3 out of 10 failed. FWIW, MAKEALL fails the same way. Is there anything funny about CentOS 5.11? York
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index 8155c16..0411959 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -324,6 +324,7 @@ class Builder: kwargs: Arguments to pass to command.RunPipe() """ cmd = [self.gnu_make] + list(args) + print cmd result = command.RunPipe([cmd], capture=True, capture_stderr=True, cwd=cwd, raise_on_error=False, **kwargs) return result diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index a9cf68a..d9526e6 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -197,7 +197,7 @@ class BuilderThread(threading.Thread): src_dir = os.getcwd() else: args.append('O=build') - args.append('-s') +# args.append('-s') if self.builder.num_jobs is not None: args.extend(['-j', str(self.builder.num_jobs)]) config_args = ['%s_defconfig' % brd.target]