diff mbox series

scripts: add size_compare.sh

Message ID 20200731031441.767974-1-mail@aparcar.org
State New
Headers show
Series scripts: add size_compare.sh | expand

Commit Message

Paul Spooren July 31, 2020, 3:14 a.m. UTC
As package size changes are a continuous topic on the mailing list this
scripts helps developers to compare their local package modifications
against latest upstream.

The script downloads the latest package indexes based on env variables
or the `.config` file. The script compares the actual installed size
(data.tar.gz) or the IPK package size.

An example output is found below:

```
user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh
Compare packages of ath79/tiny/mips_24kc:
dropbear busybox iw ubus

Checking installed size

Fetching latest package indexes...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 80634  100 80634    0     0  33499      0  0:00:02  0:00:02 --:--:-- 33485
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54082  100 54082    0     0  24252      0  0:00:02  0:00:02 --:--:-- 24252

Comparing package sizes...
Change 	Local 	Package
611B	208715B	busybox
39B	5612B	ubus
-42B	34940B	iw
-14916B	89265B	dropbear
```

I plan to integrate this script into the CI so we have a summary how
sizes change over different architectures.

Signed-off-by: Paul Spooren <mail@aparcar.org>
---
 scripts/size_compare.sh | 117 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100755 scripts/size_compare.sh

Comments

Adrian Schmutzler Sept. 3, 2020, 1:36 p.m. UTC | #1
> -----Original Message-----
> From: openwrt-devel [mailto:openwrt-devel-bounces@lists.openwrt.org]
> On Behalf Of Paul Spooren
> Sent: Freitag, 31. Juli 2020 05:15
> To: openwrt-devel@lists.openwrt.org
> Cc: freifunk@adrianschmutzler.de; Paul Spooren <mail@aparcar.org>
> Subject: [PATCH] scripts: add size_compare.sh
> 
> As package size changes are a continuous topic on the mailing list this scripts
> helps developers to compare their local package modifications against latest
> upstream.
> 
> The script downloads the latest package indexes based on env variables or
> the `.config` file. The script compares the actual installed size
> (data.tar.gz) or the IPK package size.

Just gave this a quick try, and generally it looks good.

However, I got 82 packages listed for " Compare packages of ath79/generic/mips_24kc", but only 66 were in the final comparison.

Will check the code in detail later. A few comment below.

> 
> An example output is found below:
> 
> ```
> user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare
> packages of ath79/tiny/mips_24kc:
> dropbear busybox iw ubus
> 
> Checking installed size
> 
> Fetching latest package indexes...
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 80634  100 80634    0     0  33499      0  0:00:02  0:00:02 --:--:-- 33485
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 54082  100 54082    0     0  24252      0  0:00:02  0:00:02 --:--:-- 24252
> 
> Comparing package sizes...
> Change 	Local 	Package
> 611B	208715B	busybox
> 39B	5612B	ubus
> -42B	34940B	iw
> -14916B	89265B	dropbear
> ```
> 
> I plan to integrate this script into the CI so we have a summary how sizes
> change over different architectures.
> 
> Signed-off-by: Paul Spooren <mail@aparcar.org>
> ---
>  scripts/size_compare.sh | 117
> ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 117 insertions(+)
>  create mode 100755 scripts/size_compare.sh
> 
> diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode
> 100755 index 0000000000..b310a085a4
> --- /dev/null
> +++ b/scripts/size_compare.sh
> @@ -0,0 +1,117 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul
> +Spooren <mail@aparcar.org> # ### ### size_compare - compare size of
> +OpenWrt packages against upstream ### ### The script compares locally
> +compiled package with the package indexes ### available upstream. This
> +way the storage impact of optimizations or ### feature modifiactions is
> +easy to see.
> +###
> +### If no environmental variables are set the scritp reads the current
> +### .config file. The evaluated env variables are the following:
> +###
> +###   TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL
> CHECK_INSTALLED
> +###
> +### Usage:
> +###   ./scripts/size_compare.sh
> +###
> +### Options:
> +###   -p --package-size 	Check IPK package size and not installed size
> +###   -h --help 		This message
> +
> +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p'
> .config)
> +CONFIG_SUBTARGET=$(sed -n
> 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p'
> +.config) CONFIG_ARCH=$(sed -n
> +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config)
> +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p'
> .config |
> +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n
> +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config)
> +
> +TARGET=${TARGET:-$CONFIG_TARGET}
> +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET}
> +ARCH=${ARCH:-$CONFIG_ARCH}
> +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES}
> +BIN_DIR=${CONFIG_BIN_DIR:-./bin}
> +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}"
> +CHECK_INSTALLED="${CHECK_INSTALLED:-y}"
> +
> +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack
> ages.gz"
> +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz"
> +
> +help() {
> +    sed -rn 's/^### ?//;T;p' "$0"
> +}
> +
> +package_size () {
> +	FOUND_PACKAGE=
> +	if [ -z "$CHECK_INSTALLED" ]; then
> +		SEARCH_PATTERN="Size"
> +	else
> +		SEARCH_PATTERN="Installed-Size"
> +	fi
> +	while IFS= read -r line; do
> +		if [ "$line" = "Package: $2" ]; then
> +			FOUND_PACKAGE=y
> +		fi
> +		if [ -n "$FOUND_PACKAGE" ]; then
> +			case $line in
> +				"$SEARCH_PATTERN"*)
> +					echo "$line" | cut -d ' ' -f 2
> +					break
> +					;;
> +			esac
> +		fi
> +	done < "$1"
> +}
> +
> +compare_sizes () {
> +	for PACKAGE in $PACKAGES; do
> +		if [ "$PACKAGE" = "libc" ]; then
> +			continue
> +		fi
> +		PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/"
> "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk")

When I first run this, I was in x86 target where only a kernel refresh has happened before (so, no build).
This then produces a lot of warning due to missing packages folder:

find: ‘./bin/targets/x86/generic/’: No such file or directory
find: ‘./bin/packages/i386_pentium4/’: No such file or directory

So, one should either suppress the message here for the find command, or alternatively check for the folders' existence beforehand.

> +		if [ -z "$PACKAGE_FILE" ]; then
> +			continue
> +		fi
> +		if [ -z "$CHECK_INSTALLED" ]; then
> +			SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE")
> +		else
> +			SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep
> data.tar.gz | awk '{ print $3 }')
> +		fi
> +		SIZE_UPSTREAM=$(package_size "$TMP_INDEX"
> "$PACKAGE")
> +		SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))"
> +		echo "${SIZE_DIFF}B	${SIZE_LOCAL}B	$PACKAGE"

Personally, I think the "B" looks ugly. As we only have bytes, we could consider to drop it.
One could also consider to add a "+" for the first column when change is positive. Both changes here are lowest priority.

> +	done
> +}
> +
> +if [ "$1" = "-h" ]; then
> +    help
> +    exit 0
> +fi
> +
> +if [ "$1" = "-p" ]; then
> +    CHECK_INSTALLED=
> +fi
> +
> +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH":
> +echo "$PACKAGES"
> +echo
> +
> +if [ -z "$CHECK_INSTALLED" ]; then
> +	echo "Checking IPK package size"
> +else
> +	echo "Checking installed size"
> +fi
> +echo
> +
> +echo "Fetching latest package indexes..."
> +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX)
> +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl
> +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo
> +
> +echo "Comparing package sizes..."
> +echo "Change 	Local 	Package"

Since we have enough space, I'd go for "Change  Local  Remote  Package" here.

Best

Adrian

> +compare_sizes | sort -n -r
> +
> +rm "$TMP_INDEX"
> --
> 2.25.1
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
diff mbox series

Patch

diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh
new file mode 100755
index 0000000000..b310a085a4
--- /dev/null
+++ b/scripts/size_compare.sh
@@ -0,0 +1,117 @@ 
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright (C) 2020 Paul Spooren <mail@aparcar.org>
+#
+###
+### size_compare - compare size of OpenWrt packages against upstream
+###
+### The script compares locally compiled package with the package indexes
+### available upstream. This way the storage impact of optimizations or
+### feature modifiactions is easy to see.
+###
+### If no environmental variables are set the scritp reads the current
+### .config file. The evaluated env variables are the following:
+###
+###   TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL CHECK_INSTALLED
+###
+### Usage:
+###   ./scripts/size_compare.sh
+###
+### Options:
+###   -p --package-size 	Check IPK package size and not installed size
+###   -h --help 		This message
+
+CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' .config)
+CONFIG_SUBTARGET=$(sed -n 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' .config)
+CONFIG_ARCH=$(sed -n 's/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config)
+CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' .config | tr '\n' ' ')
+CONFIG_BIN_DIR=$(sed -n 's/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config)
+
+TARGET=${TARGET:-$CONFIG_TARGET}
+SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET}
+ARCH=${ARCH:-$CONFIG_ARCH}
+PACKAGES=${PACKAGES:-$CONFIG_PACKAGES}
+BIN_DIR=${CONFIG_BIN_DIR:-./bin}
+BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}"
+CHECK_INSTALLED="${CHECK_INSTALLED:-y}"
+
+TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Packages.gz"
+PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz"
+
+help() {
+    sed -rn 's/^### ?//;T;p' "$0"
+}
+
+package_size () {
+	FOUND_PACKAGE=
+	if [ -z "$CHECK_INSTALLED" ]; then
+		SEARCH_PATTERN="Size"
+	else
+		SEARCH_PATTERN="Installed-Size"
+	fi
+	while IFS= read -r line; do
+		if [ "$line" = "Package: $2" ]; then
+			FOUND_PACKAGE=y
+		fi
+		if [ -n "$FOUND_PACKAGE" ]; then
+			case $line in
+				"$SEARCH_PATTERN"*)
+					echo "$line" | cut -d ' ' -f 2
+					break
+					;;
+			esac
+		fi
+	done < "$1"
+}
+
+compare_sizes () {
+	for PACKAGE in $PACKAGES; do
+		if [ "$PACKAGE" = "libc" ]; then
+			continue
+		fi
+		PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk")
+		if [ -z "$PACKAGE_FILE" ]; then
+			continue
+		fi
+		if [ -z "$CHECK_INSTALLED" ]; then
+			SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE")
+		else
+			SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep data.tar.gz | awk '{ print $3 }')
+		fi
+		SIZE_UPSTREAM=$(package_size "$TMP_INDEX" "$PACKAGE")
+		SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))"
+		echo "${SIZE_DIFF}B	${SIZE_LOCAL}B	$PACKAGE"
+	done
+}
+
+if [ "$1" = "-h" ]; then
+    help
+    exit 0
+fi
+
+if [ "$1" = "-p" ]; then
+    CHECK_INSTALLED=
+fi
+
+echo "Compare packages of $TARGET/$SUBTARGET/$ARCH":
+echo "$PACKAGES"
+echo
+
+if [ -z "$CHECK_INSTALLED" ]; then
+	echo "Checking IPK package size"
+else
+	echo "Checking installed size"
+fi
+echo
+
+echo "Fetching latest package indexes..."
+TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX)
+curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1
+curl "$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1
+echo
+
+echo "Comparing package sizes..."
+echo "Change 	Local 	Package"
+compare_sizes | sort -n -r
+
+rm "$TMP_INDEX"