From patchwork Mon Oct 17 13:08:07 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 682944 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3syJRc6SWdz9s8x for ; Tue, 18 Oct 2016 00:08:28 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=tXW8DnO0; dkim-atps=neutral Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 8E6F58CDDC; Mon, 17 Oct 2016 13:08:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8YBiAsoa771v; Mon, 17 Oct 2016 13:08:23 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 4C8428CDBB; Mon, 17 Oct 2016 13:08:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 4686B1C24E7 for ; Mon, 17 Oct 2016 13:08:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 44AFF8CD5D for ; Mon, 17 Oct 2016 13:08:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7ddUFf9W-9zF for ; Mon, 17 Oct 2016 13:08:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-yb0-f193.google.com (mail-yb0-f193.google.com [209.85.213.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id DC8F38CD6B for ; Mon, 17 Oct 2016 13:08:20 +0000 (UTC) Received: by mail-yb0-f193.google.com with SMTP id 63so685825ybs.0 for ; Mon, 17 Oct 2016 06:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=E1F/+3sNfOP1URk8rYtv9Qkaa4/kTNHOtGdzjUMbvMM=; b=tXW8DnO0Av/uQEnmcjymaavtMoatPvRJFHy2BPET4KeZmPBXwVxhQz0TixXPhs5/1E rOzXJxgBulkCyePFJBkDsw+n7l9UeuWGlz3pgSnwevZ49XqTU3URQbWHxmuTeHJqI9GK d/VHClLI+U9x97V38tlfBPszRGicnaWWrLvWIGREGLNgSnKTP6q8/0vuPVGyi9Vyvf+a M+n0tFskDSZGkZ1cOe4eazubAQwvFoKq6zfZPkqtZQN8q+bPnVDogaa9RvwxwBGLUhHO t2GKnXHEGUDuMAXIRkjUHougkDTlWcEyZfc4OLF7NZbGggkRGkqDsX/L5/9Z6bGtckwS TRyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=E1F/+3sNfOP1URk8rYtv9Qkaa4/kTNHOtGdzjUMbvMM=; b=V22dYkO5XfPbG2nu6eIjSed5pqbg0wvgnXQehqcBJzZgF/gDqGt10JOC6EkZrkJq53 Oiywy1oL8/9ilcyeMK9HuUoQ7x4ryNT+AvDyX/Im1VJ/C8lV4AYasIB/mFn9WIdZq364 MbvVmDr488wEDWpJTTPUFqTZyE+rqrcb3+Uj/vQOak6fR03RhXPXFRiSPRXnvMJScGAr 3Oy314Yv1as4NcMdvw0pDZHfbkpkiQziXoFaGPU6JXNzdni80T04zXOOBotto9bEcG0d nr/nP6xzwWMawAmZnwGd/ARFjm35Fe9/b1qFlFah44tE+DI0SB4ZMYLMKvJEDwbB7sYf KsTg== X-Gm-Message-State: AA6/9RmilLP5pF/RnnZTPx6ElHsVUfMUQBueO7WldPMdKNkXXntEW+rQA4PZctKxDaXNCw== X-Received: by 10.37.61.65 with SMTP id k62mr11070366yba.138.1476709699934; Mon, 17 Oct 2016 06:08:19 -0700 (PDT) Received: from argentina.be.alcatel-lucent.com (alc112.alcatel.be. [195.207.101.112]) by smtp.gmail.com with ESMTPSA id k62sm11752772ywe.45.2016.10.17.06.08.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 17 Oct 2016 06:08:18 -0700 (PDT) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Mon, 17 Oct 2016 15:08:07 +0200 Message-Id: <1476709687-21484-1-git-send-email-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.7.3 Cc: yann.morin.1998@free.fr, Thomas De Schampheleire Subject: [Buildroot] [PATCH 1/1] pkg-download: use raw basename for repo archiving to remove host- prefix X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire For packages that use a version control repository rather than a pre-made tarball, the directory prefix used inside the tarball is currently FOO_BASE_NAME, which can be 'foo' or 'host-foo'. This means that the hash of such tarball will be different for target and host packages, even though the contents are exactly the same. Hence, if the hash file is created based on 'foo', and later a fresh build is made where 'host-foo' happens to be built before 'foo' (with a different config, for example), the hash will be detected as incorrect and a new download is started. This problem does not affect many packages/users, due to the number of conditions to be met: - the package should be available for target _and_ host - the package needs to use a VCS download method, e.g. git, hg, svn, ... This does not include standard github downloads, which download a pre-made archive. - there should be a hash file containing the hash of the downloaded archive. Since normally there is no hash file for packages with sources coming from a version control system, this restricts even further. Some examples of packages in this category that do have a hash file (but not necessarily match the earlier conditions): expedite, vexpress-firmware, squashfs, ... - the archive needs to be stored in a 'primary site' after initial archiving and thus be downloaded later using a non-version-controlled method, like wget or scp. This is because the version control download methods do not receive a '-H' parameter pointing to the hash file and thus no hashes are checked at all even if the file is present. While packages matching the third condition could be considered to be 'wrong' and need to be fixed, it does actually makes sense to have a hash file for packages from version control, in particular if they are stored in a primary site as mentioned in the last condition. Regardless of any different opinions on the previous paragraph, it is also not conceptually correct that a tarball of a package source can contain a Buildroot-specific directory prefix 'host-'. Therefore, use FOO_RAW_BASE_NAME instead of FOO_BASE_NAME when calling the dl-wrapper. Example test scenario that exhibits the problem: $ rm -rf /tmp/dl dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz $ make qemu_x86_64_defconfig $ make host-squashfs-dirclean host-squashfs-source $ mkdir /tmp/dl $ mv dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz /tmp/dl/ $ sed -i -e 's,BR2_PRIMARY_SITE=.*,BR2_PRIMARY_SITE="file:///tmp/dl",' \ -e '/BR2_PRIMARY_SITE/aBR2_PRIMARY_SITE_ONLY=y' .config $ make host-squashfs-dirclean host-squashfs-source Signed-off-by: Thomas De Schampheleire Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- package/pkg-download.mk | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 3b6561b..0f542e6 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -81,7 +81,7 @@ define DOWNLOAD_GIT -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ - $($(PKG)_BASE_NAME) \ + $($(PKG)_RAW_BASE_NAME) \ $($(PKG)_DL_OPTS) endef @@ -98,7 +98,7 @@ define DOWNLOAD_BZR -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ - $($(PKG)_BASE_NAME) \ + $($(PKG)_RAW_BASE_NAME) \ $($(PKG)_DL_OPTS) endef @@ -114,7 +114,7 @@ define DOWNLOAD_CVS $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ $($(PKG)_DL_VERSION) \ $($(PKG)_RAWNAME) \ - $($(PKG)_BASE_NAME) \ + $($(PKG)_RAW_BASE_NAME) \ $($(PKG)_DL_OPTS) endef @@ -130,7 +130,7 @@ define DOWNLOAD_SVN -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ - $($(PKG)_BASE_NAME) \ + $($(PKG)_RAW_BASE_NAME) \ $($(PKG)_DL_OPTS) endef @@ -162,7 +162,7 @@ define DOWNLOAD_HG -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ - $($(PKG)_BASE_NAME) \ + $($(PKG)_RAW_BASE_NAME) \ $($(PKG)_DL_OPTS) endef