From patchwork Mon Jan 7 22:05:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1021626 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="j9Z9lutf"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43YTxH63Z8z9sD9 for ; Tue, 8 Jan 2019 09:06:11 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 49C078691B; Mon, 7 Jan 2019 22:06:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aWxAQibs6xGM; Mon, 7 Jan 2019 22:06:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 1EB8B85BD6; Mon, 7 Jan 2019 22:06:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id CAD791C296D for ; Mon, 7 Jan 2019 22:06:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CA7088637D for ; Mon, 7 Jan 2019 22:06:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dkWcTKCqw+A6 for ; Mon, 7 Jan 2019 22:05:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by whitealder.osuosl.org (Postfix) with ESMTPS id 7C8B985C88 for ; Mon, 7 Jan 2019 22:05:58 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id a20so2348278edc.8 for ; Mon, 07 Jan 2019 14:05:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=VZcPujwjd1pgMWdOII6VVX9gKfq94g7K/6Y86J9Do2Q=; b=j9Z9lutfsoWxgOT1+KJN6WqdwmHhgWuqz8Mli+vI3cNF4M7qzmbzp23ohzliqXNZ0S LjlWVyFAYU6HXrHuQubJlV091Qqy2fquTlTOGZ1+BcNOOhelj8bpECBJ+AmgbIWppvVY dVXkyFTZfa/5iaThoDHlp9QZNT/iMa7MBGFcNMNQjAWaaPm0JpNzEEnX2guj7QkM8nf6 G9mzdJ3+S3YnAScZfzshFYA6nDLkxDcaSHKuZJQq6uUvCLHHAj9knTziWykXjXz5pQcd gGZj+Fq5cpneqMzBhv+60EbYxTyyRSaue/LGi9G1VHrkQjgWvKvw+i+SX2hov+4wITP0 SQqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=VZcPujwjd1pgMWdOII6VVX9gKfq94g7K/6Y86J9Do2Q=; b=mrs9c23nx9kxM9CsKuSvKjBboqByg++ALNo2PStytYP7C0KGZVnvteSB94ni993HzI QzKyh1JsvYMaoyqOKu9rM8fyH1Wm46HaOaJq6zszrma1hxfi5JFVHkBfHbogB7duioRb QyJlv6BBLoD0gW/OUU50N45/8zvYOXhUO1fef6IRL1tGEreWLhtNtaug8igsVlUz37fT pVMly7U7VUtfTzZOafzr3RjVxm5yiGt+6sh9jY6iZnhGlVQevcxECH1YZ7e5vMILvDJ6 VEUm3pJ1gpZvGkzuQhSGp2a7x3DH7W3tZig23dnQOQ4plo/iHIaQgOMe5G7SbzzkdCpw xO8A== X-Gm-Message-State: AA+aEWbjQI7/mzxN1ZKI143ehjmpM993Vv278U7njCTWTn2EvAhXSCC7 SU9iRfcitt65cFqLXfAMSEIiTeJe X-Google-Smtp-Source: AFSGD/V7MOkqbR5b6+0YY21+9Tzu0Z7KBV4AwIRb/LIVxPvJF+3iyl4r2J+lu2J0zfwYqmCGou79yQ== X-Received: by 2002:a50:aa9b:: with SMTP id q27mr56875087edc.93.1546898756573; Mon, 07 Jan 2019 14:05:56 -0800 (PST) Received: from scaer.home ([2a01:cb19:829a:2800:68e8:7a61:9bb9:12a]) by smtp.gmail.com with ESMTPSA id d56sm31799589ede.76.2019.01.07.14.05.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jan 2019 14:05:55 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 7 Jan 2019 23:05:32 +0100 Message-Id: <638ba4cafb8ab013365ae54a393b49f459bc6e74.1546898693.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 10/19] support: rewrite check-bin-arch in python X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , "Yann E. MORIN" MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The existing check-bin-arch script is written in shell, so it can't make use of all the helpers we have in python, especially the parser for the package-files lists. Although this script is relatively clean as it is, it is not totally fool-proof either, especially against weird filenames (e.g. specially-crafted filenames with \n, or \b, etc...). Also, shell scripts are often frowned upon but for the most mundane processing, and this script is definitely not mundane. Finally, shell scripts are slow, as all the processing the have to do is more often than not done by spawning programs, and that is relatively expensive. Rewrite that script in python. This allows to do cleaner processing and reuse the package-files list parser. There's however a drawback: the script grows substantially, in part because of spawning the actual readelf call (there is no ELF parser in the standard python library), and because we want to keep backward compatible with old pythons that lack proper abstractions like subprocess.DEVNULL et al. Signed-off-by: "Yann E. MORIN" Cc: Thomas De Schampheleire --- support/scripts/check-bin-arch | 205 +++++++++++++++++++++++------------------ 1 file changed, 113 insertions(+), 92 deletions(-) diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch index 66b8d89932..d4902163e7 100755 --- a/support/scripts/check-bin-arch +++ b/support/scripts/check-bin-arch @@ -1,92 +1,113 @@ -#!/usr/bin/env bash - -# List of hardcoded paths that should be ignored, as they may -# contain binaries for an architecture different from the -# architecture of the target. -declare -a IGNORES=( - # Skip firmware files, they could be ELF files for other - # architectures - "/lib/firmware" - "/usr/lib/firmware" - - # Skip kernel modules - # When building a 32-bit userland on 64-bit architectures, the kernel - # and its modules may still be 64-bit. To keep the basic - # check-bin-arch logic simple, just skip this directory. - "/lib/modules" - "/usr/lib/modules" - - # Skip files in /usr/share, several packages (qemu, - # pru-software-support) legitimately install ELF binaries that - # are not for the target architecture - "/usr/share" - - # Skip files in /lib/grub, since it is possible to have it - # for a different architecture (e.g. i386 grub on x86_64). - "/lib/grub" -) - -while getopts p:l:r:a:i: OPT ; do - case "${OPT}" in - p) package="${OPTARG}";; - l) pkg_list="${OPTARG}";; - r) readelf="${OPTARG}";; - a) arch_name="${OPTARG}";; - i) - # Ensure we do have single '/' as separators, - # and that we have a leading and a trailing one. - pattern="$(sed -r -e 's:/+:/:g; s:^/*:/:; s:/*$:/:;' <<<"${OPTARG}")" - IGNORES+=("${pattern}") - ;; - :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; - \?) error "unknown option '%s'\n" "${OPTARG}";; - esac -done - -if test -z "${package}" -o -z "${pkg_list}" -o -z "${readelf}" -o -z "${arch_name}" ; then - echo "Usage: $0 -p -l -r -a [-i PATH ...]" - exit 1 -fi - -exitcode=0 - -# Only split on new lines, for filenames-with-spaces -IFS=" -" - -while read f; do - for ignore in "${IGNORES[@]}"; do - if [[ "${f}" =~ ^"${ignore}" ]]; then - continue 2 - fi - done - - # Skip symlinks. Some symlinks may have absolute paths as - # target, pointing to host binaries while we're building. - if [[ -L "${TARGET_DIR}/${f}" ]]; then - continue - fi - - # Get architecture using readelf. We pipe through 'head -1' so - # that when the file is a static library (.a), we only take - # into account the architecture of the first object file. - arch=$(LC_ALL=C ${readelf} -h "${TARGET_DIR}/${f}" 2>&1 | \ - sed -r -e '/^ Machine: +(.+)/!d; s//\1/;' | head -1) - - # If no architecture found, assume it was not an ELF file - if test "${arch}" = "" ; then - continue - fi - - # Architecture is correct - if test "${arch}" = "${arch_name}" ; then - continue - fi - - printf 'ERROR: architecture for "%s" is "%s", should be "%s"\n' \ - "${f}" "${arch}" "${arch_name}" - - exitcode=1 -done < <( sed -r -e "/^${package},\.(.+)$/!d; s//\1/;" ${pkg_list} ) - -exit ${exitcode} +#!/usr/bin/env python + +import argparse +import os +import re +import subprocess +import sys +from brpkgutil import parse_pkg_file_list as parse_pkg_file_list + + +# List of hardcoded paths that should be ignored, as they may contain +# binaries for an architecture different from the architecture of the +# target +IGNORES = { + # Skip firmware files + # They could be ELF files for other architectures. + '/lib/firmware', + '/usr/lib/firmware', + + # Skip kernel modules + # When building a 32-bit userland on 64-bit architectures, the + # kernel and its modules may still be 64-bit. To keep the basic + # check-bin-arch logic simple, just skip these directories + '/lib/modules', + '/usr/lib/modules', + + # Skip files in /usr/share + # Several packages (qemu, pru-software-support) legitimately install + # ELF binaries that are not for the target architecture + '/usr/share', + + # Skip files in /lib/grub + # It is possible to have it for a different architecture (e.g. i386 + # grub on x86_64) + '/lib/grub', +} + + +ERROR = 'ERROR: architecture for "%s" is "%s", should be "%s"\n' + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('--package', '-p', metavar='PACKAGE', required=True) + parser.add_argument('--pkg-list', '-l', metavar='PKG_LIST', required=True) + parser.add_argument('--readelf', '-r', metavar='READELF', required=True) + parser.add_argument('--arch', '-a', metavar='ARCH', required=True) + parser.add_argument('--ignore', '-i', metavar='PATH', action='append') + args = parser.parse_args() + + if args.ignore is not None: + # Ensure we do have single '/' as separators, and that we have a + # leading and a trailing one, then append to the global list. + for pattern in args.ignore: + IGNORES.add(re.sub('/+', '/', '/{0}/'.format(pattern))) + + ignores_re = set() + for i in IGNORES: + ignores_re.add(re.compile(i)) + + arch_re = re.compile('^ Machine: +(.+)') + + target_dir = os.environ['TARGET_DIR'] + + exit_code = 0 + for record in parse_pkg_file_list(args.pkg_list): + if record['pkg'] != args.package: + continue + + fpath = record['file'] + + ignored = False + for i in ignores_re: + if i.match(fpath): + ignored = True + break + if ignored: + continue + + # Skip symlinks. Some symlinks may have absolute paths as + # target, pointing to host binaries while we're building. + if os.path.islink(os.path.join(target_dir, fpath)): + continue + + cmd = [args.readelf, '-h', os.path.join(target_dir, fpath)] + try: + with open(os.devnull, 'wb') as devnull: + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull, + universal_newlines=True) + stdout = p.communicate()[0].split('\n')[:-1] + ret = p.returncode + except OSError: + ret = 1 + stdout = list() + # If readelf returned in error, or returned nothing on stdout, + # then that was not an ELF file, or we may not yet have readelf + # (e.g. in skeleton, before toolchain) + if ret != 0 or len(stdout) == 0: + continue + + for line in stdout: + if arch_re.match(line): + arch = arch_re.sub(r'\1', line) + if arch != args.arch: + print(ERROR.format(fpath, arch, args.arch)) + exit_code = 1 + break + + return exit_code + + +if __name__ == "__main__": + sys.exit(main())