Message ID | 20200731031441.767974-1-mail@aparcar.org |
---|---|
State | Superseded |
Headers | show |
Series | scripts: add size_compare.sh | expand |
> -----Original Message----- > From: openwrt-devel [mailto:openwrt-devel-bounces@lists.openwrt.org] > On Behalf Of Paul Spooren > Sent: Freitag, 31. Juli 2020 05:15 > To: openwrt-devel@lists.openwrt.org > Cc: freifunk@adrianschmutzler.de; Paul Spooren <mail@aparcar.org> > Subject: [PATCH] scripts: add size_compare.sh > > As package size changes are a continuous topic on the mailing list this scripts > helps developers to compare their local package modifications against latest > upstream. > > The script downloads the latest package indexes based on env variables or > the `.config` file. The script compares the actual installed size > (data.tar.gz) or the IPK package size. Just gave this a quick try, and generally it looks good. However, I got 82 packages listed for " Compare packages of ath79/generic/mips_24kc", but only 66 were in the final comparison. Will check the code in detail later. A few comment below. > > An example output is found below: > > ``` > user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare > packages of ath79/tiny/mips_24kc: > dropbear busybox iw ubus > > Checking installed size > > Fetching latest package indexes... > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485 > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252 > > Comparing package sizes... > Change Local Package > 611B 208715B busybox > 39B 5612B ubus > -42B 34940B iw > -14916B 89265B dropbear > ``` > > I plan to integrate this script into the CI so we have a summary how sizes > change over different architectures. > > Signed-off-by: Paul Spooren <mail@aparcar.org> > --- > scripts/size_compare.sh | 117 > ++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 117 insertions(+) > create mode 100755 scripts/size_compare.sh > > diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode > 100755 index 0000000000..b310a085a4 > --- /dev/null > +++ b/scripts/size_compare.sh > @@ -0,0 +1,117 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul > +Spooren <mail@aparcar.org> # ### ### size_compare - compare size of > +OpenWrt packages against upstream ### ### The script compares locally > +compiled package with the package indexes ### available upstream. This > +way the storage impact of optimizations or ### feature modifiactions is > +easy to see. > +### > +### If no environmental variables are set the scritp reads the current > +### .config file. The evaluated env variables are the following: > +### > +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL > CHECK_INSTALLED > +### > +### Usage: > +### ./scripts/size_compare.sh > +### > +### Options: > +### -p --package-size Check IPK package size and not installed size > +### -h --help This message > + > +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' > .config) > +CONFIG_SUBTARGET=$(sed -n > 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' > +.config) CONFIG_ARCH=$(sed -n > +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config) > +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' > .config | > +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n > +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config) > + > +TARGET=${TARGET:-$CONFIG_TARGET} > +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET} > +ARCH=${ARCH:-$CONFIG_ARCH} > +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES} > +BIN_DIR=${CONFIG_BIN_DIR:-./bin} > +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}" > +CHECK_INSTALLED="${CHECK_INSTALLED:-y}" > + > +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack > ages.gz" > +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz" > + > +help() { > + sed -rn 's/^### ?//;T;p' "$0" > +} > + > +package_size () { > + FOUND_PACKAGE= > + if [ -z "$CHECK_INSTALLED" ]; then > + SEARCH_PATTERN="Size" > + else > + SEARCH_PATTERN="Installed-Size" > + fi > + while IFS= read -r line; do > + if [ "$line" = "Package: $2" ]; then > + FOUND_PACKAGE=y > + fi > + if [ -n "$FOUND_PACKAGE" ]; then > + case $line in > + "$SEARCH_PATTERN"*) > + echo "$line" | cut -d ' ' -f 2 > + break > + ;; > + esac > + fi > + done < "$1" > +} > + > +compare_sizes () { > + for PACKAGE in $PACKAGES; do > + if [ "$PACKAGE" = "libc" ]; then > + continue > + fi > + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" > "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk") When I first run this, I was in x86 target where only a kernel refresh has happened before (so, no build). This then produces a lot of warning due to missing packages folder: find: ‘./bin/targets/x86/generic/’: No such file or directory find: ‘./bin/packages/i386_pentium4/’: No such file or directory So, one should either suppress the message here for the find command, or alternatively check for the folders' existence beforehand. > + if [ -z "$PACKAGE_FILE" ]; then > + continue > + fi > + if [ -z "$CHECK_INSTALLED" ]; then > + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE") > + else > + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep > data.tar.gz | awk '{ print $3 }') > + fi > + SIZE_UPSTREAM=$(package_size "$TMP_INDEX" > "$PACKAGE") > + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))" > + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE" Personally, I think the "B" looks ugly. As we only have bytes, we could consider to drop it. One could also consider to add a "+" for the first column when change is positive. Both changes here are lowest priority. > + done > +} > + > +if [ "$1" = "-h" ]; then > + help > + exit 0 > +fi > + > +if [ "$1" = "-p" ]; then > + CHECK_INSTALLED= > +fi > + > +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH": > +echo "$PACKAGES" > +echo > + > +if [ -z "$CHECK_INSTALLED" ]; then > + echo "Checking IPK package size" > +else > + echo "Checking installed size" > +fi > +echo > + > +echo "Fetching latest package indexes..." > +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX) > +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl > +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo > + > +echo "Comparing package sizes..." > +echo "Change Local Package" Since we have enough space, I'd go for "Change Local Remote Package" here. Best Adrian > +compare_sizes | sort -n -r > + > +rm "$TMP_INDEX" > -- > 2.25.1 > > > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
On Thu Sep 3, 2020 at 3:36 AM HST, Adrian Schmutzler wrote: > > -----Original Message----- > > From: openwrt-devel [mailto:openwrt-devel-bounces@lists.openwrt.org] > > On Behalf Of Paul Spooren > > Sent: Freitag, 31. Juli 2020 05:15 > > To: openwrt-devel@lists.openwrt.org > > Cc: freifunk@adrianschmutzler.de; Paul Spooren <mail@aparcar.org> > > Subject: [PATCH] scripts: add size_compare.sh > > > > As package size changes are a continuous topic on the mailing list this scripts > > helps developers to compare their local package modifications against latest > > upstream. > > > > The script downloads the latest package indexes based on env variables or > > the `.config` file. The script compares the actual installed size > > (data.tar.gz) or the IPK package size. > > Just gave this a quick try, and generally it looks good. > > However, I got 82 packages listed for " Compare packages of > ath79/generic/mips_24kc", but only 66 were in the final comparison. The current script also interprets some options as packages, which is couldn't find a clean solution for: CONFIG_PACKAGE_MAC80211_DEBUGFS=y # CONFIG_PACKAGE_MAC80211_TRACING is not set CONFIG_PACKAGE_MAC80211_MESH=y Also, this script only checks for packages of openwrt.git, so if you have packages from other repositories installed, they'd be missing. > > Will check the code in detail later. A few comment below. > > > > > An example output is found below: > > > > ``` > > user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare > > packages of ath79/tiny/mips_24kc: > > dropbear busybox iw ubus > > > > Checking installed size > > > > Fetching latest package indexes... > > % Total % Received % Xferd Average Speed Time Time Time Current > > Dload Upload Total Spent Left Speed > > 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485 > > % Total % Received % Xferd Average Speed Time Time Time Current > > Dload Upload Total Spent Left Speed > > 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252 > > > > Comparing package sizes... > > Change Local Package > > 611B 208715B busybox > > 39B 5612B ubus > > -42B 34940B iw > > -14916B 89265B dropbear > > ``` > > > > I plan to integrate this script into the CI so we have a summary how sizes > > change over different architectures. > > > > Signed-off-by: Paul Spooren <mail@aparcar.org> > > --- > > scripts/size_compare.sh | 117 > > ++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 117 insertions(+) > > create mode 100755 scripts/size_compare.sh > > > > diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode > > 100755 index 0000000000..b310a085a4 > > --- /dev/null > > +++ b/scripts/size_compare.sh > > @@ -0,0 +1,117 @@ > > +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul > > +Spooren <mail@aparcar.org> # ### ### size_compare - compare size of > > +OpenWrt packages against upstream ### ### The script compares locally > > +compiled package with the package indexes ### available upstream. This > > +way the storage impact of optimizations or ### feature modifiactions is > > +easy to see. > > +### > > +### If no environmental variables are set the scritp reads the current > > +### .config file. The evaluated env variables are the following: > > +### > > +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL > > CHECK_INSTALLED > > +### > > +### Usage: > > +### ./scripts/size_compare.sh > > +### > > +### Options: > > +### -p --package-size Check IPK package size and not installed size > > +### -h --help This message > > + > > +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' > > .config) > > +CONFIG_SUBTARGET=$(sed -n > > 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' > > +.config) CONFIG_ARCH=$(sed -n > > +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config) > > +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' > > .config | > > +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n > > +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config) > > + > > +TARGET=${TARGET:-$CONFIG_TARGET} > > +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET} > > +ARCH=${ARCH:-$CONFIG_ARCH} > > +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES} > > +BIN_DIR=${CONFIG_BIN_DIR:-./bin} > > +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}" > > +CHECK_INSTALLED="${CHECK_INSTALLED:-y}" > > + > > +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack > > ages.gz" > > +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz" > > + > > +help() { > > + sed -rn 's/^### ?//;T;p' "$0" > > +} > > + > > +package_size () { > > + FOUND_PACKAGE= > > + if [ -z "$CHECK_INSTALLED" ]; then > > + SEARCH_PATTERN="Size" > > + else > > + SEARCH_PATTERN="Installed-Size" > > + fi > > + while IFS= read -r line; do > > + if [ "$line" = "Package: $2" ]; then > > + FOUND_PACKAGE=y > > + fi > > + if [ -n "$FOUND_PACKAGE" ]; then > > + case $line in > > + "$SEARCH_PATTERN"*) > > + echo "$line" | cut -d ' ' -f 2 > > + break > > + ;; > > + esac > > + fi > > + done < "$1" > > +} > > + > > +compare_sizes () { > > + for PACKAGE in $PACKAGES; do > > + if [ "$PACKAGE" = "libc" ]; then > > + continue > > + fi > > + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" > > "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk") > > When I first run this, I was in x86 target where only a kernel refresh > has happened before (so, no build). > This then produces a lot of warning due to missing packages folder: > > find: ‘./bin/targets/x86/generic/’: No such file or directory > find: ‘./bin/packages/i386_pentium4/’: No such file or directory > > So, one should either suppress the message here for the find command, or > alternatively check for the folders' existence beforehand. I surpressed the error message. > > > + if [ -z "$PACKAGE_FILE" ]; then > > + continue > > + fi > > + if [ -z "$CHECK_INSTALLED" ]; then > > + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE") > > + else > > + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep > > data.tar.gz | awk '{ print $3 }') > > + fi > > + SIZE_UPSTREAM=$(package_size "$TMP_INDEX" > > "$PACKAGE") > > + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))" > > + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE" > > Personally, I think the "B" looks ugly. As we only have bytes, we could > consider to drop it. Dropped. > One could also consider to add a "+" for the first column when change is > positive. Both changes here are lowest priority. Added. I created a updated version on GitHub as I'm not the biggest fan of patchworks: https://github.com/openwrt/openwrt/pull/3599 > > + done > > +} > > + > > +if [ "$1" = "-h" ]; then > > + help > > + exit 0 > > +fi > > + > > +if [ "$1" = "-p" ]; then > > + CHECK_INSTALLED= > > +fi > > + > > +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH": > > +echo "$PACKAGES" > > +echo > > + > > +if [ -z "$CHECK_INSTALLED" ]; then > > + echo "Checking IPK package size" > > +else > > + echo "Checking installed size" > > +fi > > +echo > > + > > +echo "Fetching latest package indexes..." > > +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX) > > +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl > > +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo > > + > > +echo "Comparing package sizes..." > > +echo "Change Local Package" > > Since we have enough space, I'd go for "Change Local Remote Package" > here. > > Best > > Adrian > > > +compare_sizes | sort -n -r > > + > > +rm "$TMP_INDEX" > > -- > > 2.25.1 > > > > > > _______________________________________________ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode 100755 index 0000000000..b310a085a4 --- /dev/null +++ b/scripts/size_compare.sh @@ -0,0 +1,117 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2020 Paul Spooren <mail@aparcar.org> +# +### +### size_compare - compare size of OpenWrt packages against upstream +### +### The script compares locally compiled package with the package indexes +### available upstream. This way the storage impact of optimizations or +### feature modifiactions is easy to see. +### +### If no environmental variables are set the scritp reads the current +### .config file. The evaluated env variables are the following: +### +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL CHECK_INSTALLED +### +### Usage: +### ./scripts/size_compare.sh +### +### Options: +### -p --package-size Check IPK package size and not installed size +### -h --help This message + +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' .config) +CONFIG_SUBTARGET=$(sed -n 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' .config) +CONFIG_ARCH=$(sed -n 's/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config) +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' .config | tr '\n' ' ') +CONFIG_BIN_DIR=$(sed -n 's/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config) + +TARGET=${TARGET:-$CONFIG_TARGET} +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET} +ARCH=${ARCH:-$CONFIG_ARCH} +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES} +BIN_DIR=${CONFIG_BIN_DIR:-./bin} +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}" +CHECK_INSTALLED="${CHECK_INSTALLED:-y}" + +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Packages.gz" +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz" + +help() { + sed -rn 's/^### ?//;T;p' "$0" +} + +package_size () { + FOUND_PACKAGE= + if [ -z "$CHECK_INSTALLED" ]; then + SEARCH_PATTERN="Size" + else + SEARCH_PATTERN="Installed-Size" + fi + while IFS= read -r line; do + if [ "$line" = "Package: $2" ]; then + FOUND_PACKAGE=y + fi + if [ -n "$FOUND_PACKAGE" ]; then + case $line in + "$SEARCH_PATTERN"*) + echo "$line" | cut -d ' ' -f 2 + break + ;; + esac + fi + done < "$1" +} + +compare_sizes () { + for PACKAGE in $PACKAGES; do + if [ "$PACKAGE" = "libc" ]; then + continue + fi + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk") + if [ -z "$PACKAGE_FILE" ]; then + continue + fi + if [ -z "$CHECK_INSTALLED" ]; then + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE") + else + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep data.tar.gz | awk '{ print $3 }') + fi + SIZE_UPSTREAM=$(package_size "$TMP_INDEX" "$PACKAGE") + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))" + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE" + done +} + +if [ "$1" = "-h" ]; then + help + exit 0 +fi + +if [ "$1" = "-p" ]; then + CHECK_INSTALLED= +fi + +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH": +echo "$PACKAGES" +echo + +if [ -z "$CHECK_INSTALLED" ]; then + echo "Checking IPK package size" +else + echo "Checking installed size" +fi +echo + +echo "Fetching latest package indexes..." +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX) +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 +curl "$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 +echo + +echo "Comparing package sizes..." +echo "Change Local Package" +compare_sizes | sort -n -r + +rm "$TMP_INDEX"
As package size changes are a continuous topic on the mailing list this scripts helps developers to compare their local package modifications against latest upstream. The script downloads the latest package indexes based on env variables or the `.config` file. The script compares the actual installed size (data.tar.gz) or the IPK package size. An example output is found below: ``` user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare packages of ath79/tiny/mips_24kc: dropbear busybox iw ubus Checking installed size Fetching latest package indexes... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252 Comparing package sizes... Change Local Package 611B 208715B busybox 39B 5612B ubus -42B 34940B iw -14916B 89265B dropbear ``` I plan to integrate this script into the CI so we have a summary how sizes change over different architectures. Signed-off-by: Paul Spooren <mail@aparcar.org> --- scripts/size_compare.sh | 117 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100755 scripts/size_compare.sh