From patchwork Fri Jul 31 03:14:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Spooren X-Patchwork-Id: 1339210 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2001:8b0:10b:1231::1; helo=merlin.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=aparcar.org Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=merlin.20170209 header.b=UDBdc31o; dkim-atps=neutral Received: from merlin.infradead.org (merlin.infradead.org [IPv6:2001:8b0:10b:1231::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BHss054x8z9sRN for ; Fri, 31 Jul 2020 13:17:07 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=WyLhtyRy9O81CFUUSRlE6uNPqxnQhxGJu98PeYvB/PA=; b=UDBdc31o+cHIf6GEeg1gbOUTxr 4lKsJiVSJcs2eMMfmznNV0eby5aJVWLzZfIxuoEzAx2jFFeTSnPwstDqdYErRTdBrdcA4S7Zz1DQM tgtpaaCZFeLTyNNG1PEP6DuIe4cIoN0DnzEKYO9bRb8cPAY33q2hrswg+nD1Gfpeun9ClWzpTtTq+ g4Qg7HTUEVQXt6GZUz9byZJH1vlpryP14/fjAxD0JuTBRH4r1w6W1qGUeb2GnzL1/f5v1u2HkYqNA DQOejp93q4dI+lXHiDl3uq0F1HGtirrXb6cOhPgCd9zxMOfNkxOTyvqn8SqOtrlmZ9Vu4Ottf/8Bs 1Varg8Mg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k1LVi-0005CV-8r; Fri, 31 Jul 2020 03:15:06 +0000 Received: from relay1-d.mail.gandi.net ([217.70.183.193]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k1LVf-0005Br-Lq for openwrt-devel@lists.openwrt.org; Fri, 31 Jul 2020 03:15:04 +0000 X-Originating-IP: 72.234.141.215 Received: from dawn.lan (udp224251uds.hawaiiantel.net [72.234.141.215]) (Authenticated sender: mail@aparcar.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPA id 0D6A1240009; Fri, 31 Jul 2020 03:14:57 +0000 (UTC) From: Paul Spooren To: openwrt-devel@lists.openwrt.org Subject: [PATCH] scripts: add size_compare.sh Date: Thu, 30 Jul 2020 17:14:42 -1000 Message-Id: <20200731031441.767974-1-mail@aparcar.org> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200730_231503_914555_88722BDD X-CRM114-Status: GOOD ( 13.38 ) X-Spam-Score: -0.7 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [217.70.183.193 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [217.70.183.193 listed in wl.mailspike.net] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: OpenWrt Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: freifunk@adrianschmutzler.de, Paul Spooren Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org As package size changes are a continuous topic on the mailing list this scripts helps developers to compare their local package modifications against latest upstream. The script downloads the latest package indexes based on env variables or the `.config` file. The script compares the actual installed size (data.tar.gz) or the IPK package size. An example output is found below: ``` user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare packages of ath79/tiny/mips_24kc: dropbear busybox iw ubus Checking installed size Fetching latest package indexes... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252 Comparing package sizes... Change Local Package 611B 208715B busybox 39B 5612B ubus -42B 34940B iw -14916B 89265B dropbear ``` I plan to integrate this script into the CI so we have a summary how sizes change over different architectures. Signed-off-by: Paul Spooren --- scripts/size_compare.sh | 117 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100755 scripts/size_compare.sh diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode 100755 index 0000000000..b310a085a4 --- /dev/null +++ b/scripts/size_compare.sh @@ -0,0 +1,117 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2020 Paul Spooren +# +### +### size_compare - compare size of OpenWrt packages against upstream +### +### The script compares locally compiled package with the package indexes +### available upstream. This way the storage impact of optimizations or +### feature modifiactions is easy to see. +### +### If no environmental variables are set the scritp reads the current +### .config file. The evaluated env variables are the following: +### +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL CHECK_INSTALLED +### +### Usage: +### ./scripts/size_compare.sh +### +### Options: +### -p --package-size Check IPK package size and not installed size +### -h --help This message + +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' .config) +CONFIG_SUBTARGET=$(sed -n 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' .config) +CONFIG_ARCH=$(sed -n 's/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config) +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' .config | tr '\n' ' ') +CONFIG_BIN_DIR=$(sed -n 's/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config) + +TARGET=${TARGET:-$CONFIG_TARGET} +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET} +ARCH=${ARCH:-$CONFIG_ARCH} +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES} +BIN_DIR=${CONFIG_BIN_DIR:-./bin} +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}" +CHECK_INSTALLED="${CHECK_INSTALLED:-y}" + +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Packages.gz" +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz" + +help() { + sed -rn 's/^### ?//;T;p' "$0" +} + +package_size () { + FOUND_PACKAGE= + if [ -z "$CHECK_INSTALLED" ]; then + SEARCH_PATTERN="Size" + else + SEARCH_PATTERN="Installed-Size" + fi + while IFS= read -r line; do + if [ "$line" = "Package: $2" ]; then + FOUND_PACKAGE=y + fi + if [ -n "$FOUND_PACKAGE" ]; then + case $line in + "$SEARCH_PATTERN"*) + echo "$line" | cut -d ' ' -f 2 + break + ;; + esac + fi + done < "$1" +} + +compare_sizes () { + for PACKAGE in $PACKAGES; do + if [ "$PACKAGE" = "libc" ]; then + continue + fi + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk") + if [ -z "$PACKAGE_FILE" ]; then + continue + fi + if [ -z "$CHECK_INSTALLED" ]; then + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE") + else + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep data.tar.gz | awk '{ print $3 }') + fi + SIZE_UPSTREAM=$(package_size "$TMP_INDEX" "$PACKAGE") + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))" + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE" + done +} + +if [ "$1" = "-h" ]; then + help + exit 0 +fi + +if [ "$1" = "-p" ]; then + CHECK_INSTALLED= +fi + +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH": +echo "$PACKAGES" +echo + +if [ -z "$CHECK_INSTALLED" ]; then + echo "Checking IPK package size" +else + echo "Checking installed size" +fi +echo + +echo "Fetching latest package indexes..." +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX) +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 +curl "$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 +echo + +echo "Comparing package sizes..." +echo "Change Local Package" +compare_sizes | sort -n -r + +rm "$TMP_INDEX"