From patchwork Mon Jan 7 22:05:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1021622 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hzNEkqLt"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43YTx653Gvz9sDT for ; Tue, 8 Jan 2019 09:06:02 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 58A5C85F90; Mon, 7 Jan 2019 22:06:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9ZFXimnXelbg; Mon, 7 Jan 2019 22:05:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8774785F8A; Mon, 7 Jan 2019 22:05:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 485C11C296D for ; Mon, 7 Jan 2019 22:05:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4893421544 for ; Mon, 7 Jan 2019 22:05:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nq9CNtqFGmnp for ; Mon, 7 Jan 2019 22:05:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by silver.osuosl.org (Postfix) with ESMTPS id 8F64520762 for ; Mon, 7 Jan 2019 22:05:55 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id x30so2385409edx.2 for ; Mon, 07 Jan 2019 14:05:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C5ikEtKVws3+DexkOpe0CQXJcIYosiOC9Zp/WS5IqtI=; b=hzNEkqLt0qGcqJ4dxlq5uZyIZPDAZN1swgs6CWF6U7KaADHxUJ0Zi3y95PsZniEeIn htR1qgyqFyDZYuhGvXXtyYzTsRMDBSVz18lSDS5isrymzzNmC50ea6a6R5Jo1IU4EEcU Wq4jzr/ifG0zOl91HuZ2KH4j9QyOK0ZU0ST6D53nD59oM6ZcKzc/R+9xAzXQCmvyt9DI TZbDRa/GmA4IP7cXAF+amXbob/ZY5asJeeOp+lFVDBen2ogSS700cGu259DiS+Na/mwi y8MK7csdcr9bTJfiKjpSihldOlNcu9pgRPxvjS4Q5JaogmDbESTRQKLs2WN4YaKVToGY oVmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=C5ikEtKVws3+DexkOpe0CQXJcIYosiOC9Zp/WS5IqtI=; b=rBlWOUJEMz41yQv9cIj+5WLYi7osSNNui+ENg1TJaxkm4Wo1/hYAs3aHwcX71mAu07 24hJONDFLPMDCPtSBd/PkbmAAM2Y8cwNgYoH1sDKjheUh00AVLAYChPauxwgqVmVE5Ot X8BDv3y20gwNsRCV7qWMGcHTkObSBxjoB5NI+zk2QpLexlxL2fvOZdf0NQbiv6gpDX5O A7pTTXRZVYtDjpi4wtJoR3KTd4C9eDwkOOGI01GCoI9zCM9cJNCuvVsE+TEEWVsy2AXG f4OMOD/9HSIkwxndQyBvgO7rTAUvg1N9iyj4TlVtd6UjXTXquF1ducnu+GXs320jtEjV 6s1Q== X-Gm-Message-State: AA+aEWYmLYFHqnZN04P6YEOnmg1qNx7k6bE605+DCCLYmPVayq3gZvRA ub5uizJwMYfaOaY9ZL78zs8NnYo8 X-Google-Smtp-Source: AFSGD/XX3Q9UPOQ/YYdR0TNJA693pN48xIuYHPY6Z6u3eE+HH3+I2ae4kYYldBMaXWHKEDyUftc+FA== X-Received: by 2002:a50:a1e5:: with SMTP id 92mr58615129edk.181.1546898753812; Mon, 07 Jan 2019 14:05:53 -0800 (PST) Received: from scaer.home ([2a01:cb19:829a:2800:68e8:7a61:9bb9:12a]) by smtp.gmail.com with ESMTPSA id d56sm31799589ede.76.2019.01.07.14.05.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jan 2019 14:05:53 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 7 Jan 2019 23:05:30 +0100 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Buildroot] [PATCH 08/19] support/check-uniq-files: decode as many strings as possible X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Currently, when there is at least one string we can't decode when reporting the file and the packages that touched it, we fallback to not decoding any string at all, which generates a report like: Warning: target file "b'/some/file'" is touched by more than one package: [b'toolchain', b'busybox'] This is not very nice, though, so we introduce a decoder that returns the decoded string if possible, and falls back to returning the repr() of the un-decoded string. Also, using a set as argument to format() further yields a not-so-nice output either (even if the decoding was OK): [u'toolchain', u'busybox'] So, we just join together all the elements of the set into a string, which is what we pass to format(). Now the output is much nicer to look at: Warning: file "/some/file" is touched by more than one package: busybox, toolchain and even in the case of an un-decodable string (with a manually tweaked list, \xbd being œ in iso8859-15, and not a valid UTF-8 encoding): Warning: file "/some/file" is touched by more than one package: 'busyb\xbdx', toolchain Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni --- support/scripts/check-uniq-files | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/support/scripts/check-uniq-files b/support/scripts/check-uniq-files index eb92724e42..e95a134168 100755 --- a/support/scripts/check-uniq-files +++ b/support/scripts/check-uniq-files @@ -7,6 +7,16 @@ from collections import defaultdict warn = 'Warning: {0} file "{1}" is touched by more than one package: {2}\n' +# If possible, try to decode the binary string s with the user's locale. +# If s contains characters that can't be decoded with that locale, return +# the representation (in the user's locale) of the un-decoded string. +def str_decode(s): + try: + return s.decode() + except UnicodeDecodeError: + return repr(s) + + def main(): parser = argparse.ArgumentParser() parser.add_argument('packages_file_list', nargs='*', @@ -32,16 +42,9 @@ def main(): for file in file_to_pkg: if len(file_to_pkg[file]) > 1: - # If possible, try to decode the binary strings with - # the default user's locale - try: - sys.stderr.write(warn.format(args.type, file.decode(), - [p.decode() for p in file_to_pkg[file]])) - except UnicodeDecodeError: - # ... but fallback to just dumping them raw if they - # contain non-representable chars - sys.stderr.write(warn.format(args.type, file, - file_to_pkg[file])) + sys.stderr.write(warn.format(args.type, str_decode(file), + ", ".join([str_decode(p) + for p in file_to_pkg[file]]))) if __name__ == "__main__":