[OpenWrt-Devel,v2] build: allow openwrt.git packages to be replaced by feeds
diff mbox

Message ID 1421895307-28550-1-git-send-email-mathieu@qca.qualcomm.com
State Accepted
Headers show

Commit Message

Mathieu Olivari Jan. 22, 2015, 2:55 a.m. UTC
Currently, replacing a package available in openwrt.git requires
modifications in openwrt.git, or requires duplicating the package in a
feed but with a different name, which causes all kind of problems
related to dependencies (all packages selecting it would have to be
modified accordingly to select the new package).

With this change, if a package with the same name is present both in
feeds/ and package/ folders, the one in feeds/ can override the one
in package/, both in the menuconfig and during the build, by passing the
"-f" option to "./scripts/feeds install"

This mechanism is particularly useful for vendor tree, or in general for
application which needs to replace one particular package which exists
within openwrt.git by a custom/newer version.

Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>
---
v2:
*fix an issue when trying to override packages which include a dependency in the
 core packages. (ex: lua depends on liblua, which exists in the core, which
 ended-up in the package not being overridden properly).
*address John's concerns about core OpenWrt packages being overriden without
 notice and control. Override is now disable by default, and "-f" needs to be
 passed to "./scripts/feeds install" to allow it.
---
 include/scan.awk |   17 +++++++++++++++++
 include/scan.mk  |    2 +-
 scripts/feeds    |   25 ++++++++++++++++++-------
 3 files changed, 36 insertions(+), 8 deletions(-)
 create mode 100644 include/scan.awk

Comments

John Crispin Jan. 22, 2015, 8:52 a.m. UTC | #1
On 22/01/2015 03:55, Mathieu Olivari wrote:
> Currently, replacing a package available in openwrt.git requires 
> modifications in openwrt.git, or requires duplicating the package
> in a feed but with a different name, which causes all kind of
> problems related to dependencies (all packages selecting it would
> have to be modified accordingly to select the new package).
> 
> With this change, if a package with the same name is present both
> in feeds/ and package/ folders, the one in feeds/ can override the
> one in package/, both in the menuconfig and during the build, by
> passing the "-f" option to "./scripts/feeds install"
> 
> This mechanism is particularly useful for vendor tree, or in
> general for application which needs to replace one particular
> package which exists within openwrt.git by a custom/newer version.
> 
> Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>


Hi Mathieu,

this looks much better. i think we should also add some info in
/etc/openwrt_release in the field DISTRIB_TAINTS='no-all' indicating
that the build uses out of tree vendor packages.

i will merge this patch as is, but could you put the openwrt_release
patch onto your todo list and send it in the nearish future please

	John

> --- v2: *fix an issue when trying to override packages which
> include a dependency in the core packages. (ex: lua depends on
> liblua, which exists in the core, which ended-up in the package not
> being overridden properly). *address John's concerns about core
> OpenWrt packages being overriden without notice and control.
> Override is now disable by default, and "-f" needs to be passed to
> "./scripts/feeds install" to allow it. --- include/scan.awk |   17
> +++++++++++++++++ include/scan.mk  |    2 +- scripts/feeds    |
> 25 ++++++++++++++++++------- 3 files changed, 36 insertions(+), 8
> deletions(-) create mode 100644 include/scan.awk
> 
> diff --git a/include/scan.awk b/include/scan.awk new file mode
> 100644 index 0000000..39b2977 --- /dev/null +++ b/include/scan.awk 
> @@ -0,0 +1,17 @@ +BEGIN { FS="/" } +$1 ~ /^feeds/ { FEEDS[$NF]=$0
> } +$1 !~ /^feeds/ { PKGS[$NF]=$0 } +END { +	# Filter-out OpenWrt
> packages which have a feeds equivalent +	for (pkg in PKGS) +		if
> (pkg in FEEDS) +			delete PKGS[pkg] +	n = asort(PKGS) +	for (i=1; i
> <= n; i++) { +		print PKGS[i] +	} +	n = asort(FEEDS) +	for (i=1; i
> <= n; i++){ +		print FEEDS[i] +	} +} diff --git a/include/scan.mk
> b/include/scan.mk index 0998333..138707d 100644 ---
> a/include/scan.mk +++ b/include/scan.mk @@ -43,7 +43,7 @@ endef
> 
> $(FILELIST): rm -f $(TMP_DIR)/info/.files-$(SCAN_TARGET)-* -	$(call
> FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if
> $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep
> -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' |
> sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq > $@ +
> $(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if
> $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep
> -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' |
> sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -f
> include/scan.awk > $@
> 
> $(TMP_DIR)/info/.files-$(SCAN_TARGET).mk: $(FILELIST) ( \ diff
> --git a/scripts/feeds b/scripts/feeds index 31ad544..6c03375
> 100755 --- a/scripts/feeds +++ b/scripts/feeds @@ -358,6 +358,7 @@
> sub is_core_package($) { sub install_package { my $feed = shift; my
> $name = shift; +	my $force = shift; my $ret = 0;
> 
> $feed = lookup_package($feed, $name); @@ -383,13 +384,22 @@ sub
> install_package { $src or $src = $name;
> 
> # previously installed packages set the runtime package -	# newly
> installed packages set the source package -	$installed{$src} and
> return 0; +	# newly installed packages set the source package to 1 
> +	$installed{$src} and $installed{$src} == 1 and return 0; + +	#
> we'll trigger the override only with the 3 conditions below: +	# -
> override is allowed by command line (-f) +	# - a package with the
> same src exists in the core packages list +	# - the package
> previously installed is not from a feed +	my $override = 1 if
> ($force and is_core_package($src) and !$installed{$src}->{feed});
> 
> # check previously installed packages -	$installed{$name} and
> return 0; +	$installed{$name} and !$override and return 0; 
> $installed{$src} = 1; -	warn "Installing package '$src'\n"; + +
> $override == 1 +		and warn "Overriding package '$src'\n" +		or warn
> "Installing package '$src'\n";
> 
> $install_method{$type} or do { warn "Unknown installation method:
> '$type'\n"; @@ -436,7 +446,7 @@ sub install { my $feed; my $ret =
> 0;
> 
> -	getopts('ap:d:h', \%opts); +	getopts('ap:d:fh', \%opts);
> 
> if ($opts{h}) { usage(); @@ -462,7 +472,7 @@ sub install { my $p =
> $feed_package->{$name}; next if $p->{vdepends}; if( $p->{name} ) { 
> -						install_package($feed, $p->{name}) == 0 or $ret = 1; +
> install_package($feed, $p->{name}, exists($opts{f})) == 0 or $ret =
> 1; get_feed($f->[1]); } } @@ -470,7 +480,7 @@ sub install { } }
> else { while ($name = shift @ARGV) { -			install_package($feed,
> $name) == 0 or $ret = 1; +			install_package($feed, $name,
> exists($opts{f})) == 0 or $ret = 1; } }
> 
> @@ -638,6 +648,7 @@ Commands: -a :           Install all packages
> from all feeds or from the specified feed using the -p option. -p
> <feedname>: Prefer this feed when installing packages. -d <y|m|n>:
> Set default for newly installed packages. +	    -f :
> Install will be forced even if the package exists in core OpenWrt
> (override)
> 
> search [options] <substring>: Search for a package Options:
>
John Crispin Jan. 22, 2015, 1:15 p.m. UTC | #2
On 22/01/2015 03:55, Mathieu Olivari wrote:
> Currently, replacing a package available in openwrt.git requires
> modifications in openwrt.git, or requires duplicating the package in a
> feed but with a different name, which causes all kind of problems
> related to dependencies (all packages selecting it would have to be
> modified accordingly to select the new package).
> 
> With this change, if a package with the same name is present both in
> feeds/ and package/ folders, the one in feeds/ can override the one
> in package/, both in the menuconfig and during the build, by passing the
> "-f" option to "./scripts/feeds install"
> 
> This mechanism is particularly useful for vendor tree, or in general for
> application which needs to replace one particular package which exists
> within openwrt.git by a custom/newer version.
> 

Hi,

./scripts/feeds install -p -a alljoyn
Use of uninitialized value $override in numeric eq (==) at
./scripts/feeds line 400.
Installing package 'alljoyn'

i will revert the patch now and await a fixed version

	John



> Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>
> ---
> v2:
> *fix an issue when trying to override packages which include a dependency in the
>  core packages. (ex: lua depends on liblua, which exists in the core, which
>  ended-up in the package not being overridden properly).
> *address John's concerns about core OpenWrt packages being overriden without
>  notice and control. Override is now disable by default, and "-f" needs to be
>  passed to "./scripts/feeds install" to allow it.
> ---
>  include/scan.awk |   17 +++++++++++++++++
>  include/scan.mk  |    2 +-
>  scripts/feeds    |   25 ++++++++++++++++++-------
>  3 files changed, 36 insertions(+), 8 deletions(-)
>  create mode 100644 include/scan.awk
> 
> diff --git a/include/scan.awk b/include/scan.awk
> new file mode 100644
> index 0000000..39b2977
> --- /dev/null
> +++ b/include/scan.awk
> @@ -0,0 +1,17 @@
> +BEGIN { FS="/" }
> +$1 ~ /^feeds/ { FEEDS[$NF]=$0 }
> +$1 !~ /^feeds/ { PKGS[$NF]=$0 }
> +END {
> +	# Filter-out OpenWrt packages which have a feeds equivalent
> +	for (pkg in PKGS)
> +		if (pkg in FEEDS)
> +			delete PKGS[pkg]
> +	n = asort(PKGS)
> +	for (i=1; i <= n; i++) {
> +		print PKGS[i]
> +	}
> +	n = asort(FEEDS)
> +	for (i=1; i <= n; i++){
> +		print FEEDS[i]
> +	}
> +}
> diff --git a/include/scan.mk b/include/scan.mk
> index 0998333..138707d 100644
> --- a/include/scan.mk
> +++ b/include/scan.mk
> @@ -43,7 +43,7 @@ endef
>  
>  $(FILELIST):
>  	rm -f $(TMP_DIR)/info/.files-$(SCAN_TARGET)-*
> -	$(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq > $@
> +	$(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -f include/scan.awk > $@
>  
>  $(TMP_DIR)/info/.files-$(SCAN_TARGET).mk: $(FILELIST)
>  	( \
> diff --git a/scripts/feeds b/scripts/feeds
> index 31ad544..6c03375 100755
> --- a/scripts/feeds
> +++ b/scripts/feeds
> @@ -358,6 +358,7 @@ sub is_core_package($) {
>  sub install_package {
>  	my $feed = shift;
>  	my $name = shift;
> +	my $force = shift;
>  	my $ret = 0;
>  
>  	$feed = lookup_package($feed, $name);
> @@ -383,13 +384,22 @@ sub install_package {
>  	$src or $src = $name;
>  
>  	# previously installed packages set the runtime package
> -	# newly installed packages set the source package
> -	$installed{$src} and return 0;
> +	# newly installed packages set the source package to 1
> +	$installed{$src} and $installed{$src} == 1 and return 0;
> +
> +	# we'll trigger the override only with the 3 conditions below:
> +	# - override is allowed by command line (-f)
> +	# - a package with the same src exists in the core packages list
> +	# - the package previously installed is not from a feed
> +	my $override = 1 if ($force and is_core_package($src) and !$installed{$src}->{feed});
>  
>  	# check previously installed packages
> -	$installed{$name} and return 0;
> +	$installed{$name} and !$override and return 0;
>  	$installed{$src} = 1;
> -	warn "Installing package '$src'\n";
> +
> +	$override == 1
> +		and warn "Overriding package '$src'\n"
> +		or warn "Installing package '$src'\n";
>  
>  	$install_method{$type} or do {
>  		warn "Unknown installation method: '$type'\n";
> @@ -436,7 +446,7 @@ sub install {
>  	my $feed;
>  	my $ret = 0;
>  
> -	getopts('ap:d:h', \%opts);
> +	getopts('ap:d:fh', \%opts);
>  
>  	if ($opts{h}) {
>  		usage();
> @@ -462,7 +472,7 @@ sub install {
>  					my $p = $feed_package->{$name};
>  					next if $p->{vdepends};
>  					if( $p->{name} ) {
> -						install_package($feed, $p->{name}) == 0 or $ret = 1;
> +						install_package($feed, $p->{name}, exists($opts{f})) == 0 or $ret = 1;
>  						get_feed($f->[1]);
>  					}
>  				}
> @@ -470,7 +480,7 @@ sub install {
>  		}
>  	} else {
>  		while ($name = shift @ARGV) {
> -			install_package($feed, $name) == 0 or $ret = 1;
> +			install_package($feed, $name, exists($opts{f})) == 0 or $ret = 1;
>  		}
>  	}
>  
> @@ -638,6 +648,7 @@ Commands:
>  	    -a :           Install all packages from all feeds or from the specified feed using the -p option.
>  	    -p <feedname>: Prefer this feed when installing packages.
>  	    -d <y|m|n>:    Set default for newly installed packages.
> +	    -f :           Install will be forced even if the package exists in core OpenWrt (override)
>  
>  	search [options] <substring>: Search for a package
>  	Options:
>

Patch
diff mbox

diff --git a/include/scan.awk b/include/scan.awk
new file mode 100644
index 0000000..39b2977
--- /dev/null
+++ b/include/scan.awk
@@ -0,0 +1,17 @@ 
+BEGIN { FS="/" }
+$1 ~ /^feeds/ { FEEDS[$NF]=$0 }
+$1 !~ /^feeds/ { PKGS[$NF]=$0 }
+END {
+	# Filter-out OpenWrt packages which have a feeds equivalent
+	for (pkg in PKGS)
+		if (pkg in FEEDS)
+			delete PKGS[pkg]
+	n = asort(PKGS)
+	for (i=1; i <= n; i++) {
+		print PKGS[i]
+	}
+	n = asort(FEEDS)
+	for (i=1; i <= n; i++){
+		print FEEDS[i]
+	}
+}
diff --git a/include/scan.mk b/include/scan.mk
index 0998333..138707d 100644
--- a/include/scan.mk
+++ b/include/scan.mk
@@ -43,7 +43,7 @@  endef
 
 $(FILELIST):
 	rm -f $(TMP_DIR)/info/.files-$(SCAN_TARGET)-*
-	$(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq > $@
+	$(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -f include/scan.awk > $@
 
 $(TMP_DIR)/info/.files-$(SCAN_TARGET).mk: $(FILELIST)
 	( \
diff --git a/scripts/feeds b/scripts/feeds
index 31ad544..6c03375 100755
--- a/scripts/feeds
+++ b/scripts/feeds
@@ -358,6 +358,7 @@  sub is_core_package($) {
 sub install_package {
 	my $feed = shift;
 	my $name = shift;
+	my $force = shift;
 	my $ret = 0;
 
 	$feed = lookup_package($feed, $name);
@@ -383,13 +384,22 @@  sub install_package {
 	$src or $src = $name;
 
 	# previously installed packages set the runtime package
-	# newly installed packages set the source package
-	$installed{$src} and return 0;
+	# newly installed packages set the source package to 1
+	$installed{$src} and $installed{$src} == 1 and return 0;
+
+	# we'll trigger the override only with the 3 conditions below:
+	# - override is allowed by command line (-f)
+	# - a package with the same src exists in the core packages list
+	# - the package previously installed is not from a feed
+	my $override = 1 if ($force and is_core_package($src) and !$installed{$src}->{feed});
 
 	# check previously installed packages
-	$installed{$name} and return 0;
+	$installed{$name} and !$override and return 0;
 	$installed{$src} = 1;
-	warn "Installing package '$src'\n";
+
+	$override == 1
+		and warn "Overriding package '$src'\n"
+		or warn "Installing package '$src'\n";
 
 	$install_method{$type} or do {
 		warn "Unknown installation method: '$type'\n";
@@ -436,7 +446,7 @@  sub install {
 	my $feed;
 	my $ret = 0;
 
-	getopts('ap:d:h', \%opts);
+	getopts('ap:d:fh', \%opts);
 
 	if ($opts{h}) {
 		usage();
@@ -462,7 +472,7 @@  sub install {
 					my $p = $feed_package->{$name};
 					next if $p->{vdepends};
 					if( $p->{name} ) {
-						install_package($feed, $p->{name}) == 0 or $ret = 1;
+						install_package($feed, $p->{name}, exists($opts{f})) == 0 or $ret = 1;
 						get_feed($f->[1]);
 					}
 				}
@@ -470,7 +480,7 @@  sub install {
 		}
 	} else {
 		while ($name = shift @ARGV) {
-			install_package($feed, $name) == 0 or $ret = 1;
+			install_package($feed, $name, exists($opts{f})) == 0 or $ret = 1;
 		}
 	}
 
@@ -638,6 +648,7 @@  Commands:
 	    -a :           Install all packages from all feeds or from the specified feed using the -p option.
 	    -p <feedname>: Prefer this feed when installing packages.
 	    -d <y|m|n>:    Set default for newly installed packages.
+	    -f :           Install will be forced even if the package exists in core OpenWrt (override)
 
 	search [options] <substring>: Search for a package
 	Options: