From patchwork Tue Dec 4 21:32:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Whitcroft X-Patchwork-Id: 1007883 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 438Znn0Bmwz9sCX; Wed, 5 Dec 2018 08:32:13 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1gUIIZ-0003St-Pw; Tue, 04 Dec 2018 21:32:07 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.86_2) (envelope-from ) id 1gUIIX-0003SG-Ag for kernel-team@lists.ubuntu.com; Tue, 04 Dec 2018 21:32:05 +0000 Received: from 1.general.apw.uk.vpn ([10.172.192.78] helo=localhost) by youngberry.canonical.com with esmtpsa (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1gUIIX-0002b1-1T; Tue, 04 Dec 2018 21:32:05 +0000 From: Andy Whitcroft To: kernel-team@lists.ubuntu.com Subject: [disco/master-next 1/2] UBUNTU: [Packaging] getabis -- handle all known package combinations Date: Tue, 4 Dec 2018 21:32:01 +0000 Message-Id: <20181204213202.18755-2-apw@canonical.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181204213202.18755-1-apw@canonical.com> References: <20181204213202.18755-1-apw@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andy Whitcroft Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" Traditionally we have tried to download all and any packages we can find. If we have any packages we just assume that what we got is a consistent set and use it. This leads to incomplete sets being committed on network failure. Firstly detect and differentiate transport errors and valid missing packages. Secondly switch to analysing known good package set combinations; this relies on the presumption that the publisher only publishes all or none of a binary package set. This lets us throw errors when we are unable to find an internally consistent set of packages. BugLink: http://bugs.launchpad.net/bugs/1806380 Signed-off-by: Andy Whitcroft --- debian/scripts/misc/getabis | 155 +++++++++++++++++++++--------------- 1 file changed, 91 insertions(+), 64 deletions(-) diff --git a/debian/scripts/misc/getabis b/debian/scripts/misc/getabis index 42690b0311e2..edba4b078867 100755 --- a/debian/scripts/misc/getabis +++ b/debian/scripts/misc/getabis @@ -28,7 +28,7 @@ fwinfo=$abidir/fwinfo test -d $tmpdir || mkdir $tmpdir package_prefixes() { - __package_prefixes="$@" + : # no longer used ... } getall() { @@ -39,89 +39,118 @@ getall() { for sub in $@; do if [ -f $abidir/$arch/$sub ]; then - echo "Exists: $sub" + echo "Existing $sub($arch)..." continue fi - echo -n "Fetching $sub($arch)..." - prefixes="" + echo "Fetching $sub($arch)..." + getall_set "linux-buildinfo" "$arch" "$sub" || \ + getall_set "linux-image-unsigned linux-modules linux-modules-extra" "$arch" "$sub" || \ + getall_set "linux-image-unsigned linux-modules" "$arch" "$sub" || \ + getall_set "linux-image linux-modules linux-modules-extra" "$arch" "$sub" || \ + getall_set "linux-image linux-modules" "$arch" "$sub" || \ + getall_set "linux-image linux-image-extra" "$arch" "$sub" || \ + getall_set "linux-image" "$arch" "$sub" || \ + { echo "FAILED"; exit 1; } + done +} +getall_set() +{ + prefixes="$1" + arch="$2" + sub="$3" + ( + echo -n " set:" filenames="" cd $tmpdir - for prefix in $__package_prefixes + found=1 + for prefix in $prefixes do + echo -n " $prefix=" + if [ "$found" = 0 ]; then + echo -n "-" + continue + fi filename=${prefix}-${verabi}-${sub}_${verfull}_${arch}.deb for r in "${repo_list[@]}" do if ! [ -f $filename ]; then $WGET $r/$filename + rc="$?" + # If this was not successful or a valid error + # return from the server all bets are off, bail. + [ "$rc" != 0 -a "$rc" != 8 ] && return 1 fi if [ -f $filename ]; then - prefixes="$prefixes $prefix" + echo -n "y" filenames="$filenames $filename" break fi done + if [ ! -f "$filename" ]; then + echo -n "n" + found=0 + fi done - if [ "$filenames" != "" ]; then - echo -n "extracting$prefixes..." - for filename in $filenames - do - dpkg-deb --extract $filename tmp + echo "" + if [ "$found" = 0 ]; then + return 1 + fi + echo " extracting..." + for filename in $filenames + do + dpkg-deb --extract $filename tmp + done + # FORM 1: linux-image et al extracted here. + if [ -d tmp/boot ]; then + echo " images..." + find tmp -name "*.ko" | while read f; do + modinfo $f | grep ^firmware >> $fwinfo done - # FORM 1: linux-image et al extracted here. - if [ -d tmp/boot ]; then - echo -n "images..." - find tmp -name "*.ko" | while read f; do - modinfo $f | grep ^firmware >> $fwinfo - done - if [ -f tmp/boot/abi-* ]; then - mv tmp/boot/abi-* $abidir/$arch/$sub - else - echo -n "NO ABI FILE..." - fi - if [ -f tmp/boot/retpoline-* ]; then - mv tmp/boot/retpoline-* $abidir/$arch/$sub.retpoline - else - echo -n "NO RETPOLINE FILE..." - fi - (cd tmp; find lib/modules/$verabi-$sub/kernel -name '*.ko') | \ - sed -e 's/.*\/\([^\/]*\)\.ko/\1/' | sort > \ - $abidir/$arch/$sub.modules - ( - cd tmp; - # Prevent exposing some errors when called by python scripts. SIGPIPE seems to get - # exposed when using the `find ...` form of the command. - ko=$(find lib/modules/$verabi-$sub/kernel \ - -name '*.ko' | head -1) - readelf -p .comment "$ko" | gawk ' - ($1 == "[") { - printf("%s", $3); - for (n=4; n<=NF; n++) { - printf(" %s", $n); - } - print "" - }' | sort -u >$abidir/$arch/$sub.compiler - version=`cat $abidir/$arch/$sub.compiler` - echo -n "$version..." - ) - # FORM 2: moduleinfo packages - # cranky-fix -- modinfo supported + if [ -f tmp/boot/abi-* ]; then + mv tmp/boot/abi-* $abidir/$arch/$sub + else + echo " NO ABI FILE" + fi + if [ -f tmp/boot/retpoline-* ]; then + mv tmp/boot/retpoline-* $abidir/$arch/$sub.retpoline else - echo -n "buildinfo..." - base="tmp/usr/lib/linux/${verabi}-${sub}" - mv "$base/abi" "$abidir/$arch/$sub" - for comp in 'modules' 'retpoline' 'compiler' - do - mv "$base/$comp" "$abidir/$arch/$sub.$comp" - done - cat "$base/fwinfo" >>"$fwinfo" + echo " NO RETPOLINE FILE" fi - rm -rf tmp $filenames - echo "done." + (cd tmp; find lib/modules/$verabi-$sub/kernel -name '*.ko') | \ + sed -e 's/.*\/\([^\/]*\)\.ko/\1/' | sort > \ + $abidir/$arch/$sub.modules + ( + cd tmp; + # Prevent exposing some errors when called by python scripts. SIGPIPE seems to get + # exposed when using the `find ...` form of the command. + ko=$(find lib/modules/$verabi-$sub/kernel \ + -name '*.ko' | head -1) + readelf -p .comment "$ko" | gawk ' + ($1 == "[") { + printf("%s", $3); + for (n=4; n<=NF; n++) { + printf(" %s", $n); + } + print "" + }' | sort -u >$abidir/$arch/$sub.compiler + version=`cat $abidir/$arch/$sub.compiler` + echo " $version" + ) + # FORM 2: moduleinfo packages + # cranky-fix -- modinfo supported else - echo "FAILED." + echo " buildinfo..." + base="tmp/usr/lib/linux/${verabi}-${sub}" + mv "$base/abi" "$abidir/$arch/$sub" + for comp in 'modules' 'retpoline' 'compiler' + do + mv "$base/$comp" "$abidir/$arch/$sub.$comp" + done + cat "$base/fwinfo" >>"$fwinfo" fi - cd $origdir - done + rm -rf tmp $filenames + echo " done" + ) } # MAIN @@ -133,8 +162,6 @@ echo $abi > $abidir/abiname # NOTE: The flavours are hardcoded, because they may have changed from the # current build. -__package_prefixes="linux-image" - . $DEBIAN/etc/getabis # Extract compiler source package version from e.g.: