| Message ID | 20200407200724.8308-1-cotequeiroz@gmail.com |
|---|---|
| Headers | show |
| Series | build: update scritps/config to kconfig-v5.6 | expand |
Hi Eneas, I am sorry but I had to completely revert the kconfig bump. It thoroughly broke the package repository builds since multiple days and the fatal recursive dependencies make it a no-go for master, at least as far as our build infrastructure is concerned. Right now, a single malformed feed package can entirely break the builder, across all architectures. I am happy to reapply it but first we have to figure out why things like http://builds.openwrt.org/master/packages/builders/arm_fa526/builds/224/steps/compile/logs/stdio happen and how we can prevent them in the future. I'm also strongly in favor of making recursive deps a warning, at least when CONFIG_BUILDBOT=y. Regards, Jo
On Sat, Apr 11, 2020 at 7:23 PM Jo-Philipp Wich <jo@mein.io> wrote: > > Hi Eneas, > > I am sorry but I had to completely revert the kconfig bump. It thoroughly > broke the package repository builds since multiple days and the fatal > recursive dependencies make it a no-go for master, at least as far as our > build infrastructure is concerned. > > Right now, a single malformed feed package can entirely break the builder, > across all architectures. > > I am happy to reapply it but first we have to figure out why things like > http://builds.openwrt.org/master/packages/builders/arm_fa526/builds/224/steps/compile/logs/stdio > happen and how we can prevent them in the future. > > I'm also strongly in favor of making recursive deps a warning, at least when > CONFIG_BUILDBOT=y. > > Regards, > Jo Hi I'm really sorry--and embarrassed, really--to have caused all this trouble. I'll see what I can do from here, but I'm not familiar enough with the build bot system to do much on my own--and that was the origin of all problems. As for the recursive dependency warning/error, at first I thought about using CONFIG_BUILDBOT but then scripts/config/conf may be built before .config even exist. I added a make option, so the bots could just add WARN_RECURSIVE_DEPS=1 to the make args. Even changing the recursive dep to a warning would not have been enough to overcome this, for example: feeds/base/package/utils/busybox/config/Config.in:712: invalid statement I'm not sure if the feeds/base/package structure is the same as $(TOPDIR)/package, but 1da014f should have quoted the source filenames in package/utils/busybox/config/Config.in. Does anybody have any suggestion about how this could be moved forward? Eneas
Hi Eneas, > I'm really sorry--and embarrassed, really--to have caused all this > trouble. I'll see what I can do from here, but I'm not familiar > enough with the build bot system to do much on my own--and that was > the origin of all problems. no need to be embarrassed, I just figured that reverting for now might be the better course forward, compared to working on fixes under pressure. Theoretically you should be able to setup a fully working buildbot setup yourself by cloning https://git.openwrt.org/?p=buildbot.git;a=summary and using the docker-compose file to start it. It is quite a time and resource consuming endeavor though since it first needs to build all phase1 targets (images + SDK) and then the phase2 package builds using the previously generated SDKs. A simpler way to reproduce may be building your own SDK (CONFIG_SDK=y), then unpack the resulting tarball somewhere, install *all* feeds and *all* packages into it: tar -xzf openwrt-sdk-*.tar.xz cd openwrt-sdk-* rm -f .config && make defconfig ./scripts/feeds update -f ./scripts/feeds install -a make That should theoretically already replicate the issue occurring on the builders. In case you cannot replicate it with a self-built SDK, I uploaded an affected one here before it gets replaced with a reverted version: http://mein.io/openwrt-sdk-bcm47xx-generic_gcc-8.4.0_musl.Linux-x86_64.tar.xz SHA256: f9855ae67f19a28b584d2b0af844b6565c91dd7d2eaf196151ba8c45215d52c5 > As for the recursive dependency warning/error, at first I thought > about using CONFIG_BUILDBOT but then scripts/config/conf may be built > before .config even exist. I added a make option, so the bots could > just add WARN_RECURSIVE_DEPS=1 to the make args. Even changing the > recursive dep to a warning would not have been enough to overcome > this, for example: Right, I see that too now. I know that it will not have fixed the particular issues we've seen on the builders, but the issue still stands. With circular dependency warnings-as-errors, a sole broken DEPENDS:= in a random feed package will DoS the entire build pipeline. > feeds/base/package/utils/busybox/config/Config.in:712: invalid statement > > I'm not sure if the feeds/base/package structure is the same as > $(TOPDIR)/package, but 1da014f should have quoted the source filenames > in package/utils/busybox/config/Config.in. Indeed and it appears to have worked. The log I quoted was a rather old one. Here's one showing a failing build due to recusive deps: http://builds.openwrt.org/master/packages/builders/i386_pentium4/builds/237/steps/compile/logs/stdio > Does anybody have any suggestion about how this could be moved forward? Hm, what about reapplying the update but with the recursive dependency warning logic inverted? Means always just warn by default and offer an environment variable to make them strict errors. CI tests or whoever is interested could opt into the strict checking, others are unaffected. Regards, Jo
The full cover-letter is in the original series. V2 is about shipping pre-generated c/h files to avoid depending on bison & flex. The _shipped files are gone upstream, so I did not use the previous scheme, of copying *_shipped files. Instead, I'm shipping the *.[ch] files directly. I've made it easier to generate the files if desired, by running make with BUILD_SHIPPED_FILES=1 (the actual value does not matter). It is documented in the README file. Defining the variable changes make clean to remove the generated files as well. -- Changelog v1->v2: - Added pre-generated *.tab.[ch] *.lex.c files to avoid depending on flex & bison Eneas U de Queiroz (6): kernel: add @IPV6 dependency to ipv6 modules busybox: quote 'source' filenames in Config.in build: define RTC_SUPPORT as a bool build: simplify building *config targets build: scripts/config - update to kconfig-v5.6 build: add option to warn on recursive dependency include/toplevel.mk | 15 +- package/kernel/linux/modules/netfilter.mk | 13 +- package/kernel/linux/modules/netsupport.mk | 6 +- package/utils/busybox/config/Config.in | 44 +- .../utils/busybox/config/networking/Config.in | 2 +- .../utils/busybox/config/util-linux/Config.in | 2 +- scripts/config/.gitignore | 29 +- scripts/config/Makefile | 179 +- scripts/config/README | 30 +- scripts/config/conf.c | 248 +- scripts/config/confdata.c | 533 +- scripts/config/expr.c | 216 +- scripts/config/expr.h | 110 +- scripts/config/images.c | 34 +- scripts/config/images.h | 33 + scripts/config/{zconf.l => lexer.l} | 340 +- scripts/config/lexer.lex.c | 4499 +++++++++++++++++ scripts/config/list.h | 1 + scripts/config/lkc.h | 58 +- scripts/config/lkc_proto.h | 21 +- scripts/config/lxdialog/.gitignore | 2 - scripts/config/lxdialog/check-lxdialog.sh | 91 - scripts/config/lxdialog/checklist.c | 19 +- scripts/config/lxdialog/dialog.h | 23 +- scripts/config/lxdialog/inputbox.c | 22 +- scripts/config/lxdialog/menubox.c | 25 +- scripts/config/lxdialog/textbox.c | 17 +- scripts/config/lxdialog/util.c | 15 +- scripts/config/lxdialog/yesno.c | 19 +- scripts/config/mconf-cfg.sh | 50 + scripts/config/mconf.c | 179 +- scripts/config/menu.c | 451 +- .../{zconf.tab.c_shipped => parser.tab.c} | 939 ++-- scripts/config/parser.tab.h | 129 + scripts/config/{zconf.y => parser.y} | 429 +- scripts/config/preprocess.c | 575 +++ scripts/config/qconf-cfg.sh | 32 + scripts/config/qconf.cc | 174 +- scripts/config/qconf.h | 3 +- scripts/config/symbol.c | 268 +- scripts/config/util.c | 86 +- scripts/config/zconf.gperf | 49 - scripts/config/zconf.hash.c_shipped | 250 - scripts/config/zconf.lex.c_shipped | 2533 ---------- target/Config.in | 3 +- 45 files changed, 7798 insertions(+), 4998 deletions(-) create mode 100644 scripts/config/images.h rename scripts/config/{zconf.l => lexer.l} (50%) create mode 100644 scripts/config/lexer.lex.c delete mode 100644 scripts/config/lxdialog/.gitignore delete mode 100644 scripts/config/lxdialog/check-lxdialog.sh create mode 100755 scripts/config/mconf-cfg.sh rename scripts/config/{zconf.tab.c_shipped => parser.tab.c} (73%) create mode 100644 scripts/config/parser.tab.h rename scripts/config/{zconf.y => parser.y} (64%) create mode 100644 scripts/config/preprocess.c create mode 100755 scripts/config/qconf-cfg.sh delete mode 100644 scripts/config/zconf.gperf delete mode 100644 scripts/config/zconf.hash.c_shipped delete mode 100644 scripts/config/zconf.lex.c_shipped