| Message ID | cover.1515421834.git.mschiffer@universe-factory.net |
|---|---|
| Headers | show |
| Series | Metadata generator refactoring, cleanup and PROVIDES improvements | expand |
On 01/08/2018 03:52 PM, Matthias Schiffer wrote: > When attempting to fix FS#837, I decided to treat our metadata generator to > a greater refactoring, which resulted in this patchset. FS#837 is fixed in > patch 13. > > I also removed two seemingly obsolete features, "preconfig" and "package > features", which are not used by any package in the OpenWrt/LEDE base or > our default feeds. > > The changes in this patchset also improve dependency diagnostics and > clearly separate between build depends (on source packages) and runtime > depends (on binary packages, implying build depends). This requires some > fixes in package Makefiles with broken dependencies (patch 10, [1], [2]). > > There are also broken runtime depends in many feed packages, which I did > not fix, but which will be shown more prominently during prepare-tmpinfo > now. I'm not sure what the best way to handle such broken dependencies is, > as they might also be caused by missing feeds, e.g. LuCI depending on > telephony packages. > - Keep the warning, but try to build the packages as before (current > solution in this patchset)? > - Hide the warning again (old state)? > - Ignore packages with missing build dependencies altogether (just show a > warning in the metadata generators)? > > As hinted in patch 15, I have further refactoring in mind, but let's try to > get this set reviewed and committed first... > > Regards, > Matthias I've pushed an updated branch fixing the mistake Jo pointed out and some other stylistic improvements to the metadata2 branch of my staging tree [3]. > [1] https://github.com/openwrt/packages/pull/5370 > [2] https://github.com/openwrt-routing/packages/pull/345 [3] https://git.lede-project.org/?p=openwrt/staging/neoraider.git;a=shortlog;h=refs/heads/metadata2 > > > Matthias Schiffer (15): > metadata: remove 'base-files' special case > metadata: make srcpackage extensible > metadata: move 'buildtypes' from binary to source packages > metadata: move 'builddepends' from binary to source packages > metadata: interate over source packages when generating Makefile > metadata: change pkg->{src} field to hold a reference > metadata: replace %subdir hash with a path field in source packages > metadata: remove redundant fields from package hash > build: remove package preconfig feature > treewide: fix build depends to refer to source package names > metadata: handle target build depends together with host build depends > metadata: simplify generation of build depends from runtime depends > metadata: always resolve dependencies through provides list > build: remove obsolete "package feature" feature > include/package-dumpinfo.mk: don't duplicate source package > information for every binary package > > include/autotools.mk | 4 +- > include/nls.mk | 2 +- > include/package-dumpinfo.mk | 53 +---- > package/Makefile | 3 - > package/network/config/ltq-adsl-app/Makefile | 2 +- > package/network/config/ltq-vdsl-app/Makefile | 2 +- > scripts/feeds | 28 +-- > scripts/metadata.pm | 101 +++------ > scripts/package-metadata.pl | 299 +++++++++------------------ > 9 files changed, 158 insertions(+), 336 deletions(-) >
On 8 January 2018 at 14:02, Matthias Schiffer <mschiffer@universe-factory.net> wrote: > I've pushed an updated branch fixing the mistake Jo pointed out and some > other stylistic improvements to the metadata2 branch of my staging tree [3]. > > >> [1] https://github.com/openwrt/packages/pull/5370 >> [2] https://github.com/openwrt-routing/packages/pull/345 > > [3] > https://git.lede-project.org/?p=openwrt/staging/neoraider.git;a=shortlog;h=refs/heads/metadata2 So the base repo works: $ make Checking 'working-make'... ok. Checking 'case-sensitive-fs'... ok. Checking 'proper-umask'... ok. Checking 'gcc'... ok. Checking 'working-gcc'... ok. Checking 'g++'... ok. Checking 'working-g++'... ok. Checking 'ncurses'... ok. Checking 'zlib'... ok. Checking 'perl-thread-queue'... ok. Checking 'tar'... ok. Checking 'find'... ok. Checking 'bash'... ok. Checking 'patch'... ok. Checking 'diff'... ok. Checking 'cp'... ok. Checking 'seq'... ok. Checking 'awk'... ok. Checking 'grep'... ok. Checking 'getopt'... ok. Checking 'stat'... ok. Checking 'unzip'... ok. Checking 'bzip2'... ok. Checking 'wget'... ok. Checking 'perl'... ok. Checking 'python'... ok. Checking 'git'... ok. Checking 'file'... ok. Checking 'ldconfig-stub'... ok. Collecting package info: done Collecting target info: done WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on 'libpam', which does not exist WARNING: Makefile 'package/utils/busybox/Makefile' has a build dependency on 'libpam', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libgnutls', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libopenldap', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libidn', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libssh2', which does not exist WARNING: Makefile 'package/boot/kexec-tools/Makefile' has a dependency on 'liblzma', which does not exist WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on 'kmod-usb-musb-tusb6010', which does not exist WARNING: Makefile 'package/network/services/lldpd/Makefile' has a dependency on 'libnetsnmp', which does not exist make[2]: Entering directory '/home/weedy/projects/openwrt-lede/scripts/config' conf.c: In function 'check_stdin': conf.c:78:3: warning: format not a string literal and no format arguments [-Wformat-security] printf(_("aborted!\n\n")); ^~~~~~ conf.c:79:3: warning: format not a string literal and no format arguments [-Wformat-security] printf(_("Console input/output is redirected. ")); ^~~~~~ conf.c:80:3: warning: format not a string literal and no format arguments [-Wformat-security] printf(_("Run 'make oldconfig' to update configuration.\n\n")); ^~~~~~ conf.c: In function 'conf_askvalue': conf.c:90:3: warning: format not a string literal and no format arguments [-Wformat-security] printf(_("(NEW) ")); ^~~~~~ conf.c: In function 'conf_choice': conf.c:291:5: warning: format not a string literal and no format arguments [-Wformat-security] printf(_(" (NEW)")); ^~~~~~ conf.c: In function 'check_conf': conf.c:439:6: warning: format not a string literal and no format arguments [-Wformat-security] printf(_("*\n* Restart config...\n*\n")); ^~~~~~ conf.c: In function 'main': conf.c:620:6: warning: format not a string literal and no format arguments [-Wformat-security] _("\n*** The configuration requires explicit update.\n\n")); ^ conf.c:674:4: warning: format not a string literal and no format arguments [-Wformat-security] fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n")); ^~~~~~~ conf.c:678:4: warning: format not a string literal and no format arguments [-Wformat-security] fprintf(stderr, _("\n*** Error during update of the configuration.\n\n")); ^~~~~~~ conf.c:689:4: warning: format not a string literal and no format arguments [-Wformat-security] fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n")); ^~~~~~~ make[2]: Leaving directory '/home/weedy/projects/openwrt-lede/scripts/config' make[2]: Entering directory '/home/weedy/projects/openwrt-lede' make[3]: Entering directory '/home/weedy/projects/openwrt-lede/scripts/config' make[3]: Leaving directory '/home/weedy/projects/openwrt-lede/scripts/config' make[3]: Entering directory '/home/weedy/projects/openwrt-lede' make[3]: Leaving directory '/home/weedy/projects/openwrt-lede' WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on 'libpam', which does not exist WARNING: Makefile 'package/utils/busybox/Makefile' has a build dependency on 'libpam', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libgnutls', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libopenldap', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libidn', which does not exist WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libssh2', which does not exist WARNING: Makefile 'package/boot/kexec-tools/Makefile' has a dependency on 'liblzma', which does not exist WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on 'kmod-usb-musb-tusb6010', which does not exist WARNING: Makefile 'package/network/services/lldpd/Makefile' has a dependency on 'libnetsnmp', which does not exist configuration written to .config *** End of the configuration. *** Execute 'make' to start the build or try 'make help'. make[2]: Leaving directory '/home/weedy/projects/openwrt-lede' make[1] world make[2] tools/compile But adding feeds pukes: .. Installing package 'znc' from openwrtPackages Installing package 'zoneinfo' from openwrtPackages Installing package 'zsh' from openwrtPackages + make V=99 oldconfig Collecting package info: done WARNING: Makefile 'package/feeds/openwrtPackages/acme/Makefile' has a dependency on 'luci-base', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/acme/Makefile' has a dependency on 'luci-app-uhttpd', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/coova-chilli/Makefile' has a dependency on 'libmatrixssl', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/dynapoint/Makefile' has a dependency on 'luci-lib-nixio', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/libiio/Makefile' has a dependency on 'libavahi-common', which does not exist WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on 'kmod-usb-musb-tusb6010', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/python/Makefile' has a dependency on 'python-pip-conf-src', which does not exist WARNING: Makefile 'package/feeds/openwrtPackages/sqm-scripts/Makefile' has a dependency on 'luci-base', which does not exist [ -L .config ] && export KCONFIG_OVERWRITECONFIG=1; \ scripts/config/conf --oldconfig Config.in tmp/.config-package.in:37300: syntax error tmp/.config-package.in:37299: invalid option tmp/.config-package.in:37318: syntax error tmp/.config-package.in:37317: invalid option make: *** [/home/weedy/projects/openwrt-lede/include/toplevel.mk:132: oldconfig] Error 1 How do you want me to debug this?
On 14 January 2018 at 06:21, Weedy <weedy2887@gmail.com> wrote: > On 8 January 2018 at 14:02, Matthias Schiffer > <mschiffer@universe-factory.net> wrote: >> I've pushed an updated branch fixing the mistake Jo pointed out and some >> other stylistic improvements to the metadata2 branch of my staging tree [3]. >> >> >>> [1] https://github.com/openwrt/packages/pull/5370 >>> [2] https://github.com/openwrt-routing/packages/pull/345 >> >> [3] >> https://git.lede-project.org/?p=openwrt/staging/neoraider.git;a=shortlog;h=refs/heads/metadata2 > > > So the base repo works: > $ make > Checking 'working-make'... ok. > Checking 'case-sensitive-fs'... ok. > Checking 'proper-umask'... ok. > Checking 'gcc'... ok. > Checking 'working-gcc'... ok. > Checking 'g++'... ok. > Checking 'working-g++'... ok. > Checking 'ncurses'... ok. > Checking 'zlib'... ok. > Checking 'perl-thread-queue'... ok. > Checking 'tar'... ok. > Checking 'find'... ok. > Checking 'bash'... ok. > Checking 'patch'... ok. > Checking 'diff'... ok. > Checking 'cp'... ok. > Checking 'seq'... ok. > Checking 'awk'... ok. > Checking 'grep'... ok. > Checking 'getopt'... ok. > Checking 'stat'... ok. > Checking 'unzip'... ok. > Checking 'bzip2'... ok. > Checking 'wget'... ok. > Checking 'perl'... ok. > Checking 'python'... ok. > Checking 'git'... ok. > Checking 'file'... ok. > Checking 'ldconfig-stub'... ok. > Collecting package info: done > Collecting target info: done > WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on > 'libpam', which does not exist > WARNING: Makefile 'package/utils/busybox/Makefile' has a build > dependency on 'libpam', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libgnutls', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libopenldap', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libidn', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libssh2', which does not exist > WARNING: Makefile 'package/boot/kexec-tools/Makefile' has a dependency > on 'liblzma', which does not exist > WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on > 'kmod-usb-musb-tusb6010', which does not exist > WARNING: Makefile 'package/network/services/lldpd/Makefile' has a > dependency on 'libnetsnmp', which does not exist > make[2]: Entering directory '/home/weedy/projects/openwrt-lede/scripts/config' > conf.c: In function 'check_stdin': > conf.c:78:3: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_("aborted!\n\n")); > ^~~~~~ > conf.c:79:3: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_("Console input/output is redirected. ")); > ^~~~~~ > conf.c:80:3: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_("Run 'make oldconfig' to update configuration.\n\n")); > ^~~~~~ > conf.c: In function 'conf_askvalue': > conf.c:90:3: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_("(NEW) ")); > ^~~~~~ > conf.c: In function 'conf_choice': > conf.c:291:5: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_(" (NEW)")); > ^~~~~~ > conf.c: In function 'check_conf': > conf.c:439:6: warning: format not a string literal and no format > arguments [-Wformat-security] > printf(_("*\n* Restart config...\n*\n")); > ^~~~~~ > conf.c: In function 'main': > conf.c:620:6: warning: format not a string literal and no format > arguments [-Wformat-security] > _("\n*** The configuration requires explicit update.\n\n")); > ^ > conf.c:674:4: warning: format not a string literal and no format > arguments [-Wformat-security] > fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n")); > ^~~~~~~ > conf.c:678:4: warning: format not a string literal and no format > arguments [-Wformat-security] > fprintf(stderr, _("\n*** Error during update of the configuration.\n\n")); > ^~~~~~~ > conf.c:689:4: warning: format not a string literal and no format > arguments [-Wformat-security] > fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n")); > ^~~~~~~ > make[2]: Leaving directory '/home/weedy/projects/openwrt-lede/scripts/config' > make[2]: Entering directory '/home/weedy/projects/openwrt-lede' > make[3]: Entering directory '/home/weedy/projects/openwrt-lede/scripts/config' > make[3]: Leaving directory '/home/weedy/projects/openwrt-lede/scripts/config' > make[3]: Entering directory '/home/weedy/projects/openwrt-lede' > make[3]: Leaving directory '/home/weedy/projects/openwrt-lede' > WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on > 'libpam', which does not exist > WARNING: Makefile 'package/utils/busybox/Makefile' has a build > dependency on 'libpam', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libgnutls', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libopenldap', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libidn', which does not exist > WARNING: Makefile 'package/network/utils/curl/Makefile' has a > dependency on 'libssh2', which does not exist > WARNING: Makefile 'package/boot/kexec-tools/Makefile' has a dependency > on 'liblzma', which does not exist > WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on > 'kmod-usb-musb-tusb6010', which does not exist > WARNING: Makefile 'package/network/services/lldpd/Makefile' has a > dependency on 'libnetsnmp', which does not exist > configuration written to .config > > *** End of the configuration. > *** Execute 'make' to start the build or try 'make help'. > > make[2]: Leaving directory '/home/weedy/projects/openwrt-lede' > make[1] world > make[2] tools/compile > > > But adding feeds pukes: > .. > Installing package 'znc' from openwrtPackages > Installing package 'zoneinfo' from openwrtPackages > Installing package 'zsh' from openwrtPackages > + make V=99 oldconfig > Collecting package info: done > WARNING: Makefile 'package/feeds/openwrtPackages/acme/Makefile' has a > dependency on 'luci-base', which does not exist > WARNING: Makefile 'package/feeds/openwrtPackages/acme/Makefile' has a > dependency on 'luci-app-uhttpd', which does not exist > WARNING: Makefile > 'package/feeds/openwrtPackages/coova-chilli/Makefile' has a dependency > on 'libmatrixssl', which does not exist > WARNING: Makefile 'package/feeds/openwrtPackages/dynapoint/Makefile' > has a dependency on 'luci-lib-nixio', which does not exist > WARNING: Makefile 'package/feeds/openwrtPackages/libiio/Makefile' has > a dependency on 'libavahi-common', which does not exist > WARNING: Makefile 'package/kernel/linux/Makefile' has a dependency on > 'kmod-usb-musb-tusb6010', which does not exist > WARNING: Makefile 'package/feeds/openwrtPackages/python/Makefile' has > a dependency on 'python-pip-conf-src', which does not exist > WARNING: Makefile 'package/feeds/openwrtPackages/sqm-scripts/Makefile' > has a dependency on 'luci-base', which does not exist > [ -L .config ] && export KCONFIG_OVERWRITECONFIG=1; \ > scripts/config/conf --oldconfig Config.in > tmp/.config-package.in:37300: syntax error > tmp/.config-package.in:37299: invalid option > tmp/.config-package.in:37318: syntax error > tmp/.config-package.in:37317: invalid option > make: *** [/home/weedy/projects/openwrt-lede/include/toplevel.mk:132: > oldconfig] Error 1 > > > How do you want me to debug this? tmp/.config-package.in:37317: invalid option select PACKAGE_sqm-scripts depends on <--- this line depends on PACKAGE_lua tmp/.config-package.in:37299: invalid option select PACKAGE_libpthread if USE_GLIBC depends on <--- this line depends on PACKAGE_lua select PACKAGE_libc
Hi Weedy, can you provide some more context lines to see which package those broken depends belong to? Thanks, Jo
On 2018-01-14 08:12 AM, Jo-Philipp Wich wrote: > Hi Weedy, > > can you provide some more context lines to see which package those broken depends belong to? > > Thanks, > Jo > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel > menu "LuCI" menu "3. Applications" config PACKAGE_luci-app-acme tristate "luci-app-acme.............................. ACME package - LuCI interface" default y if DEFAULT_luci-app-acme default m if ALL select PACKAGE_libc select PACKAGE_librt if USE_GLIBC *** depends on depends on PACKAGE_lua select PACKAGE_libpthread if USE_GLIBC select PACKAGE_acme select PACKAGE_libssp if SSP_SUPPORT help Control the ACME Letsencrypt certificate interface Toke Høiland-Jørgensen <toke@toke.dk> config PACKAGE_luci-app-sqm tristate "luci-app-sqm................................ SQM Scripts - LuCI interface" default y if DEFAULT_luci-app-sqm default m if ALL select PACKAGE_libc *** depends on select PACKAGE_librt if USE_GLIBC depends on PACKAGE_lua depends on !LINUX_3_18 select PACKAGE_libssp if SSP_SUPPORT select PACKAGE_libpthread if USE_GLIBC select PACKAGE_sqm-scripts help Luci interface for the SQM scripts queue management configuration package. Toke Høiland-Jørgensen <toke@toke.dk> endmenu endmenu
When attempting to fix FS#837, I decided to treat our metadata generator to a greater refactoring, which resulted in this patchset. FS#837 is fixed in patch 13. I also removed two seemingly obsolete features, "preconfig" and "package features", which are not used by any package in the OpenWrt/LEDE base or our default feeds. The changes in this patchset also improve dependency diagnostics and clearly separate between build depends (on source packages) and runtime depends (on binary packages, implying build depends). This requires some fixes in package Makefiles with broken dependencies (patch 10, [1], [2]). There are also broken runtime depends in many feed packages, which I did not fix, but which will be shown more prominently during prepare-tmpinfo now. I'm not sure what the best way to handle such broken dependencies is, as they might also be caused by missing feeds, e.g. LuCI depending on telephony packages. - Keep the warning, but try to build the packages as before (current solution in this patchset)? - Hide the warning again (old state)? - Ignore packages with missing build dependencies altogether (just show a warning in the metadata generators)? As hinted in patch 15, I have further refactoring in mind, but let's try to get this set reviewed and committed first... Regards, Matthias [1] https://github.com/openwrt/packages/pull/5370 [2] https://github.com/openwrt-routing/packages/pull/345 Matthias Schiffer (15): metadata: remove 'base-files' special case metadata: make srcpackage extensible metadata: move 'buildtypes' from binary to source packages metadata: move 'builddepends' from binary to source packages metadata: interate over source packages when generating Makefile metadata: change pkg->{src} field to hold a reference metadata: replace %subdir hash with a path field in source packages metadata: remove redundant fields from package hash build: remove package preconfig feature treewide: fix build depends to refer to source package names metadata: handle target build depends together with host build depends metadata: simplify generation of build depends from runtime depends metadata: always resolve dependencies through provides list build: remove obsolete "package feature" feature include/package-dumpinfo.mk: don't duplicate source package information for every binary package include/autotools.mk | 4 +- include/nls.mk | 2 +- include/package-dumpinfo.mk | 53 +---- package/Makefile | 3 - package/network/config/ltq-adsl-app/Makefile | 2 +- package/network/config/ltq-vdsl-app/Makefile | 2 +- scripts/feeds | 28 +-- scripts/metadata.pm | 101 +++------ scripts/package-metadata.pl | 299 +++++++++------------------ 9 files changed, 158 insertions(+), 336 deletions(-)