From patchwork Mon Oct 19 02:41:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383950 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=IRN0IVOX; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Hg5xqtz9sSG for ; Mon, 19 Oct 2020 13:42:07 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E964C823C4; Mon, 19 Oct 2020 04:41:52 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="IRN0IVOX"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 9B5D081F7C; Mon, 19 Oct 2020 04:41:49 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id E3DCC81B79 for ; Mon, 19 Oct 2020 04:41:46 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x133.google.com with SMTP id p9so9021424ilr.1 for ; Sun, 18 Oct 2020 19:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HDQxlp4dzXG1Xc6VNsjJxEZhpUbW3hPaEKytZG+sQs4=; b=IRN0IVOXW+TkrznreJi3BVfQ9WetGu6rELQDMWMyeC3pt9e+Q2ghEO5m5RQOceP8Ou hjvl5PlTxhVUF/+zdTY7YmvEjOmL0EJC6jZN9LItJmg3HYjD7RhiPwRMCScGfI0yY0yG 1FacFIYksR6ucIEtWW7v9eWzc9mshE7ZienzQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HDQxlp4dzXG1Xc6VNsjJxEZhpUbW3hPaEKytZG+sQs4=; b=GdGPQH+fnYa2lCzSzd5f6OEBB6N8+k/j0AICrWgTDGJwtOrRXLTgGOhStxSVwOBhot L5mbZ8sg6Q0PdZrgzEC8dbXrv2gb4zWzDxZm4MNVhYQWR4T8oSbHPO1Mu8rokXBCnd0u 9VF1ozhdu2c+dXsk6IT76me6S3harxFd1JUTLWi4pbaY5ra8UmNQ/lOQuMjPODaqmYQo PSkUMa0ARaY5mla1CHQZtTHNIy17BZZ+A4Bxuu1MfMA6v+RLUIAd6xHqn63OmTiO1F/8 nVLnPtVvvmWeVtIqqjUdVritwS2Gb8n/msJqO0w4e3gfbhhW2wHIGSg2d65fVLEHN15p unTw== X-Gm-Message-State: AOAM533wSL3VVTgRekLynF3nhQJSPAsBjtIUdMFw4SDUPbz1mr0nZlnE JXtLQP7+yt8t1WsUEyg98agD5OgyQI6Y5T4E X-Google-Smtp-Source: ABdhPJzyZVmJx4uK6UHspi3CgIOaPksp7NSL4AQNe8gYubAqO49noorCst2YyEyr/5tM4xrIWk8onA== X-Received: by 2002:a05:6e02:1241:: with SMTP id j1mr9906258ilq.267.1603075305445; Sun, 18 Oct 2020 19:41:45 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:44 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 01/25] binman: Give a sensible error if no command is given Date: Sun, 18 Oct 2020 20:41:14 -0600 Message-Id: <20201019024138.3804540-2-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present if 'binman' is typed on the command line, a strange error about a missing argument is displayed. Fix this. These does not seem to be standard way to add the 'required' argument in all recent Python versions, so set it manually. Signed-off-by: Simon Glass --- tools/binman/cmdline.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/binman/cmdline.py b/tools/binman/cmdline.py index bb4d9d1288b..c007d0a036d 100644 --- a/tools/binman/cmdline.py +++ b/tools/binman/cmdline.py @@ -37,6 +37,7 @@ controlled by a description in the board device tree.''' '3=info, 4=detail, 5=debug') subparsers = parser.add_subparsers(dest='cmd') + subparsers.required = True build_parser = subparsers.add_parser('build', help='Build firmware image') build_parser.add_argument('-a', '--entry-arg', type=str, action='append', From patchwork Mon Oct 19 02:41:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383951 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=J4IgikcA; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Hv23BZz9sSG for ; Mon, 19 Oct 2020 13:42:19 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 8093D823DD; Mon, 19 Oct 2020 04:41:54 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="J4IgikcA"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 9E97A823A7; Mon, 19 Oct 2020 04:41:51 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 19CB481BD2 for ; Mon, 19 Oct 2020 04:41:48 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x136.google.com with SMTP id n5so2344964ile.7 for ; Sun, 18 Oct 2020 19:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mW8s2QEDat8HhwvIBBN/nyzRHZbJ12Qhr4zc0qYPWcE=; b=J4IgikcA66s0Oe3/vFLkuXlY0e9p/8GAarWTcWH/tUiwafMWf9q/xcVvgwZMSJA1CR yl2vyHwyEjaFpCk54cOKpe3LTlQC7QtRYxGFDkL1bTKXePEstLqq1+47yvsjsKRNYYOW pHUb8AfVbmqX8k63KKliUhrR4nc8NJsfvXUO4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mW8s2QEDat8HhwvIBBN/nyzRHZbJ12Qhr4zc0qYPWcE=; b=JB7XQD073JhfZ1SedCGVK3n8FzuKrGnrtfqwbP/NAHQs6lE53uHriAdO93ABKvumvv EhmyGqOpFc7aqDgDIm/fTpoTpci3WOTG4w0KcLw2bRSiCU7UKek3Yh+URoOhuvGILLg5 fZ4H+X0COU+WZodhypo6i1FEcLmoxYjMsyQZmfJ8k3e5YDdncTEzEd53aw5nYOnu/zWp fz8dN/X+FnPN452qeZB4mz1VjNF7We5m+FVIjoytCNNW9Bs66uucWeMesZWOC+T3XFys q8R/l9w5PGLUX9eMBrmrNHir2PuWnudMc3TP+xxSUYAtG03aJSnKxLel4pUuJNKEmuwd pObw== X-Gm-Message-State: AOAM533mdMlZx/3ZI30Fw4Zx/7oogEOi0rZX2s0O/aXb+7skHYmMkK9C l9Yw6o+EOwbtSXi+BiNNWyMQPOKB/Qnu/Fod X-Google-Smtp-Source: ABdhPJxtceGHszQIEUHRP1LyspmjWGYnXgezTqP4pgSTxgGEuV2+5gXLvzzeAHTNWhNzMQdZueasrw== X-Received: by 2002:a92:7914:: with SMTP id u20mr9587179ilc.203.1603075306853; Sun, 18 Oct 2020 19:41:46 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:45 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Bin Meng Subject: [PATCH 02/25] binman: Fix return from u-boot-ucode if there is no DT Date: Sun, 18 Oct 2020 20:41:15 -0600 Message-Id: <20201019024138.3804540-3-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean This should return empty contents, not leave it unset. Fix it. Signed-off-by: Simon Glass --- tools/binman/etype/u_boot_ucode.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/binman/etype/u_boot_ucode.py b/tools/binman/etype/u_boot_ucode.py index 44622936182..b4cb8cdb6e1 100644 --- a/tools/binman/etype/u_boot_ucode.py +++ b/tools/binman/etype/u_boot_ucode.py @@ -81,6 +81,7 @@ class Entry_u_boot_ucode(Entry_blob): if fdt_entry: break if not fdt_entry: + self.data = b'' return True if not fdt_entry.ready: return False From patchwork Mon Oct 19 02:41:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383952 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=Mzx6pSYb; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1J84tWmz9sSG for ; Mon, 19 Oct 2020 13:42:32 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 6350D8241C; Mon, 19 Oct 2020 04:41:56 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="Mzx6pSYb"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 0164482396; Mon, 19 Oct 2020 04:41:52 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x142.google.com (mail-il1-x142.google.com [IPv6:2607:f8b0:4864:20::142]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 0CB27821A4 for ; Mon, 19 Oct 2020 04:41:49 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x142.google.com with SMTP id j13so8989522ilc.4 for ; Sun, 18 Oct 2020 19:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LdDhii/MWmXGRbao/u6KAzTi88VZfQ9YZeRamZWV4xE=; b=Mzx6pSYbCaZoJKJlQPynI9WmzURjo9/RJSAE9WdO5OO3sK0Nk8i4iKmKmErFyHTB71 pf0hdJ1CB2a0GPPrsppaXSmdZBUt3Y8XZbtuJDm6fn1UVsV0eWz6X8itKwuxXnXM8WZe fXGnl52iBXu0KO7zO6lmzWeyvUcadU1qYZwFQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LdDhii/MWmXGRbao/u6KAzTi88VZfQ9YZeRamZWV4xE=; b=R4cpN2nfxDOG2RN7CC+2ptEylOgubobI+/83Qqc4E4fYS7aZrjbM9SKtrgR2AjS0lK 3Z374NI2RDO8C5HS8Cl5R1Y1SYVRhoqgbiDxFPVrlDp1yFkanHmBj00qzOkZgOcR619G qjKl72wIRsbMWbea3jH5+ROVAAjD0prpmP8bLczB8+QM83ndA87fKTJlf4weQieF9ToR d9aDTGx5HT0zEICL3nODYEppPd1aROeov8knXsDyskwRMG9vzW48ieUKxAL6sOrHKuth n4Ez7LXQ8lVYLBTqxuW/luk0YBrAZpwevC0ZCsvHt32qlrpLggmNBXsPkb1xCWY3l/ej nDaQ== X-Gm-Message-State: AOAM530zVz82NtRHqxpOnYlX4/MCyXQQoTu78JdNI8pJs/YuFxjpna79 iEGwhmRmkW8Md1ZPrzw/OxDOAneFRbNjB0NN X-Google-Smtp-Source: ABdhPJyRt/Yuh40YpNgpbmmTvn1nLOoMnN18bR/uhYUOJQIEnhCA6TdkJdultLLEv620IYeKcWE0wQ== X-Received: by 2002:a92:ce09:: with SMTP id b9mr4270110ilo.14.1603075307694; Sun, 18 Oct 2020 19:41:47 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:47 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 03/25] binman: Remove references to 'image' in entry_Section Date: Sun, 18 Oct 2020 20:41:16 -0600 Message-Id: <20201019024138.3804540-4-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean While a section is the base class of Image, it is more correct to refer to sections in most places in this file. Fix these comments. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 515c97f9290..327750461eb 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -56,7 +56,7 @@ class Entry_section(Entry): self._end_4gb = False def ReadNode(self): - """Read properties from the image node""" + """Read properties from the section node""" super().ReadNode() self._pad_byte = fdt_util.GetInt(self._node, 'pad-byte', 0) self._sort = fdt_util.GetBool(self._node, 'sort-by-offset') @@ -183,7 +183,7 @@ class Entry_section(Entry): return super().Pack(offset) def _PackEntries(self): - """Pack all entries into the image""" + """Pack all entries into the section""" offset = self._skip_at_start for entry in self._entries.values(): offset = entry.Pack(offset) @@ -209,7 +209,7 @@ class Entry_section(Entry): self._entries[entry._node.name] = entry def CheckEntries(self): - """Check that entries do not overlap or extend outside the image""" + """Check that entries do not overlap or extend outside the section""" if self._sort: self._SortEntries() self._ExpandEntries() @@ -456,7 +456,7 @@ class Entry_section(Entry): def CheckSize(self): - """Check that the image contents does not exceed its size, etc.""" + """Check that the section contents does not exceed its size, etc.""" contents_size = 0 for entry in self._entries.values(): contents_size = max(contents_size, entry.offset + entry.size) From patchwork Mon Oct 19 02:41:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383953 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=jylbT8o5; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1JR3Y5Kz9sSG for ; Mon, 19 Oct 2020 13:42:47 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id BC69482426; Mon, 19 Oct 2020 04:42:00 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jylbT8o5"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 381EE823DD; Mon, 19 Oct 2020 04:41:53 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x141.google.com (mail-il1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 4F8E881F7C for ; Mon, 19 Oct 2020 04:41:50 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x141.google.com with SMTP id q1so8983641ilt.6 for ; Sun, 18 Oct 2020 19:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wklKVqILWhLLb2nVoPboh3X5v1dBC0hlkg3uzEuutlM=; b=jylbT8o59KlM0kqVKsnGIZsr+Xi6KfhOaGsQKrSwpj6GPuhuLomHw9W7QWljhg1sK8 0dWSpoIZteIimMNeXM+5zUDm7FyNlJnZK3flwwnmgWVvtAEcU3akt4WHMO9Oy3rc1cvR xqQadBCAxW8o7kQRLlo7k8/h+BZTQ7z31YnyA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wklKVqILWhLLb2nVoPboh3X5v1dBC0hlkg3uzEuutlM=; b=GaaJhY/oQ4YGDajpHyPU3eXNfPPZrxLUHBydUXYEMOGhesFC9j1m8dtrplYCwckJrI GdcclWN6VudF6ncfx0luX/wgzqvpZXquNLeskPNAyhedrudNUwfvf+bK+a0K8MDyArJh MWGqN6njngzmUnWt33eEdOz8ktLplXziJYQq+sfoq3HTHvHNBZiH32ZKLj7cGUv9HSp0 1UbLOQgLHKJ/60s4YnN8TBp53Z/y/18HsZwnmGcjNEsFCzVAGqnnxe4poLIsvAMonccH upxAOuUYJ/yqHeo65F27WhytEYHqZBlO+h/XH0wrOeB8k5F/Sb9o2MQXRNoTqLORG9Eh fEmw== X-Gm-Message-State: AOAM531tpsjike57tAwurkxfBAUlurFZIRSMHZNzN29tQZgCkzrCIQH2 r6LtOi/0j8fSDbwdN+mp8nCMI0FCEq7HK6N+ X-Google-Smtp-Source: ABdhPJwS4qBKOFD5GBs8gh92fOTEMcPgtc9JlcerNJ2aZwZZvqQI/F765/9c3SvldA7heq3Hz/1WKw== X-Received: by 2002:a05:6e02:6d1:: with SMTP id p17mr8774199ils.190.1603075309099; Sun, 18 Oct 2020 19:41:49 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:48 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 04/25] binman: Expand the error message for breaching a section Date: Sun, 18 Oct 2020 20:41:17 -0600 Message-Id: <20201019024138.3804540-5-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Add in a few more details to this error message to make it easier to see what is going on. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 10 ++++++---- tools/binman/ftest.py | 5 +++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 327750461eb..a3e37c33c1b 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -220,10 +220,12 @@ class Entry_section(Entry): if (entry.offset < self._skip_at_start or entry.offset + entry.size > self._skip_at_start + self.size): - entry.Raise("Offset %#x (%d) is outside the section starting " - "at %#x (%d)" % - (entry.offset, entry.offset, self._skip_at_start, - self._skip_at_start)) + entry.Raise('Offset %#x (%d) size %#x (%d) is outside the ' + "section '%s' starting at %#x (%d) " + 'of size %#x (%d)' % + (entry.offset, entry.offset, entry.size, entry.size, + self._node.path, self._skip_at_start, + self._skip_at_start, self.size, self.size)) if entry.offset < offset and entry.size: entry.Raise("Offset %#x (%d) overlaps with previous entry '%s' " "ending at %#x (%d)" % diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index b771b9d5df7..8e6222ad785 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -968,8 +968,9 @@ class TestFunctional(unittest.TestCase): """Test that the end-at-4gb property checks for offset boundaries""" with self.assertRaises(ValueError) as e: self._DoTestFile('028_pack_4gb_outside.dts') - self.assertIn("Node '/binman/u-boot': Offset 0x0 (0) is outside " - "the section starting at 0xffffffe0 (4294967264)", + self.assertIn("Node '/binman/u-boot': Offset 0x0 (0) size 0x4 (4) " + "is outside the section '/binman' starting at " + '0xffffffe0 (4294967264) of size 0x20 (32)', str(e.exception)) def testPackX86Rom(self): From patchwork Mon Oct 19 02:41:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383954 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=PlaJ2yva; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Jg3s3Yz9sSG for ; Mon, 19 Oct 2020 13:42:59 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2F11282429; Mon, 19 Oct 2020 04:42:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="PlaJ2yva"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 4CEEC823E0; Mon, 19 Oct 2020 04:41:54 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 307698236F for ; Mon, 19 Oct 2020 04:41:51 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd44.google.com with SMTP id u19so11334694ion.3 for ; Sun, 18 Oct 2020 19:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RaPkjFcn/49WXb8FQG0WHW+lYSv+dXYjEvO4eclAQd0=; b=PlaJ2yvaHLtxRyvOfbL5UdPVi/pmF3I8AEo48AHuzCGFlxMmRU9PSueTqmKPWLJ5MI YMk4jv/v9iWiI4RF4+dKDVjTkagudgk4IVIiNqykiVc0w824ZaEXWxtMJnHkKECgP71+ chAPskvbnmSp94yq2pWDZySb5qpme86ecxD3U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RaPkjFcn/49WXb8FQG0WHW+lYSv+dXYjEvO4eclAQd0=; b=XHVwC7KJcJE2AFdAjVFZYsgJNED+w9z2uFTX/Xl41YWRUhWmocay/jE1xzOCIuOJCd uqNtfjjmM1WDlVXq+0sgmByvCa66+Ihndw/fTs3UXHEJKjTOnflmY17TNMvA4dQYETv1 /3ZrML0n8NlPaz6W8E55JKBno2Yg9+9eYS8Dis64CdrYwZKcb075X9xMalYd39dBwD7y ZC0rCsgjoxA85gZC9HD8e57wTuKsZ1nN36xlEIS0dw8S/Lo4xt4RtX/shKhAPp6Jb/7z 6ariyVzQb4HGQ9Q26iyQmDfQcWKrGTs2sCiOxU3RL8SYr/D5m6YrWqY+TXg+U5o2CvEe cMAA== X-Gm-Message-State: AOAM532Ibt6vUcmCuW6Wwtu8Q6CZVC7bAVwERE8uVR1WMU/NKOHgOitA K/7bEgpSukwOOXU/gp8zP/iOvqZQJ1JaVt82 X-Google-Smtp-Source: ABdhPJwdXEiMCxhkPBBlkprAGWen4bkF/hNEOSx/kYjNvJBLRK71LrgstRW0KhzJH5wY3R0+a5wqaQ== X-Received: by 2002:a02:cce6:: with SMTP id l6mr9939818jaq.32.1603075309930; Sun, 18 Oct 2020 19:41:49 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:49 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Andy Shevchenko , Bin Meng Subject: [PATCH 05/25] binman: Move CompressData() into Entry base class Date: Sun, 18 Oct 2020 20:41:18 -0600 Message-Id: <20201018204115.5.Ic9d88753747c0ece1716ee5b006c38282fc6f040@changeid> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present this is only used by blobs. To allow it to be used by other entry types (such as sections), move it into the base class. Also read the compression type in the base class. Signed-off-by: Simon Glass --- tools/binman/entry.py | 17 +++++++++++++++++ tools/binman/etype/blob.py | 7 ------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/tools/binman/entry.py b/tools/binman/entry.py index f7adc3b1abb..173c9131cbb 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -180,6 +180,9 @@ class Entry(object): self.expand_size = fdt_util.GetBool(self._node, 'expand-size') self.missing_msg = fdt_util.GetString(self._node, 'missing-msg') + # This is only supported by blobs and sections at present + self.compress = fdt_util.GetString(self._node, 'compress', 'none') + def GetDefaultFilename(self): return None @@ -836,3 +839,17 @@ features to produce new behaviours. list of possible tags, most desirable first """ return list(filter(None, [self.missing_msg, self.name, self.etype])) + + def CompressData(self, indata): + """Compress data according to the entry's compression method + + Args: + indata: Data to compress + + Returns: + Compressed data (first word is the compressed size) + """ + if self.compress != 'none': + self.uncomp_size = len(indata) + data = tools.Compress(indata, self.compress) + return data diff --git a/tools/binman/etype/blob.py b/tools/binman/etype/blob.py index ecfb1e476e8..301ac55e3b2 100644 --- a/tools/binman/etype/blob.py +++ b/tools/binman/etype/blob.py @@ -33,7 +33,6 @@ class Entry_blob(Entry): def __init__(self, section, etype, node): super().__init__(section, etype, node) self._filename = fdt_util.GetString(self._node, 'filename', self.etype) - self.compress = fdt_util.GetString(self._node, 'compress', 'none') def ObtainContents(self): self._filename = self.GetDefaultFilename() @@ -48,12 +47,6 @@ class Entry_blob(Entry): self.ReadBlobContents() return True - def CompressData(self, indata): - if self.compress != 'none': - self.uncomp_size = len(indata) - data = tools.Compress(indata, self.compress) - return data - def ReadBlobContents(self): """Read blob contents into memory From patchwork Mon Oct 19 02:41:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383955 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=IxYXcdjh; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Jx5Tcxz9sSG for ; Mon, 19 Oct 2020 13:43:13 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 9ABD78244A; Mon, 19 Oct 2020 04:42:07 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="IxYXcdjh"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 888C582425; Mon, 19 Oct 2020 04:41:55 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 01BA0823A7 for ; Mon, 19 Oct 2020 04:41:52 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x12d.google.com with SMTP id w17so8958512ilg.8 for ; Sun, 18 Oct 2020 19:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E18Q54tya0VSkrbMvzkLRbiBmzfcUfZPAUN6AxEt41s=; b=IxYXcdjhLmwRNRHdoo4HYg6xTuyJgxRSeCKvLO2L5tN4F3l04cZyojulFPw7KdyHIp HciJ5gs+mSHFE2ys48nFgryFo+Y38CkDG94pm51l8g7ZvrD25PUKGsygN9AF4pMajLbo aRXLcNmq8BaBSHcooskCGFwe5kK1deOUcJ/w8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E18Q54tya0VSkrbMvzkLRbiBmzfcUfZPAUN6AxEt41s=; b=dpY0lDZvSNP6BC8Db+4LqsWOrsdowj0dnpe0svu86XyjN3gzdI+KuADzSvGOPODaD2 0SMlmjFNbVQtSIp34d3hUPdizR+TAofOI2VyuNwRnm+Yq9XVTfkQnstICabzpGDkvI3d fmd0Itp/Yw8ayCBOwdw48AbtkJ7Vat4tPPRyAd91HgtJo/V4cKFYu/xrFghLG4IYRNls igYPIBoiRWko5W0UlmIf7eOCazvknBywnI0fgjBjXXaa+tQnH4Ni6YA/03RVb7Nwijwe 7rnxEbBRK9EI7uRLCfNBaFXJggJlD9aMfZzYxl3eVD2W9OvpWhBv3SM19FTC/SuJKW3U JBdw== X-Gm-Message-State: AOAM531X0sRA+l2gtdEI7V/qvqYxNtohAFgZ6HolkTfuR651dVuB6nc2 4GQ6P1FRBijeJImP9hmTUlBs5LgQPCNSzafu X-Google-Smtp-Source: ABdhPJz7xHlgubrqG+oCzcYYSWGjXFY4fQxK1jgFDYI/9Jehfb1NosYY31FGd7O6W5hcBR6T20lnnA== X-Received: by 2002:a92:243:: with SMTP id 64mr8997892ilc.298.1603075310728; Sun, 18 Oct 2020 19:41:50 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:50 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 06/25] binman: Use 'files-compress' to set compression for files Date: Sun, 18 Oct 2020 20:41:19 -0600 Message-Id: <20201019024138.3804540-6-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present we use 'compress' as the property to set the compression of a 'files' entry. But this conflicts with the same property for entries, of which Entry_section is a subclass. Strictly speaking, since Entry_files is in fact a subclass of Entry_section, the files can be compressed individually but also the section (that contains all the files) can itself be compressed. With this change, it is possible to express that. Signed-off-by: Simon Glass --- tools/binman/README.entries | 15 ++++++++++++++- tools/binman/etype/files.py | 7 ++++--- tools/binman/etype/section.py | 4 ++-- tools/binman/test/085_files_compress.dts | 2 +- 4 files changed, 21 insertions(+), 7 deletions(-) diff --git a/tools/binman/README.entries b/tools/binman/README.entries index c1d436563e8..a3a314753c5 100644 --- a/tools/binman/README.entries +++ b/tools/binman/README.entries @@ -299,7 +299,7 @@ Entry: files: Entry containing a set of files Properties / Entry arguments: - pattern: Filename pattern to match the files to include - - compress: Compression algorithm to use: + - files-compress: Compression algorithm to use: none: No compression lz4: Use lz4 compression (via 'lz4' command-line utility) @@ -406,6 +406,10 @@ The 'default' property, if present, will be automatically set to the name if of configuration whose devicetree matches the 'default-dt' entry argument, e.g. with '-a default-dt=sun50i-a64-pine64-lts'. +Available substitutions for '@' property values are: + + DEFAULT-SEQ Sequence number of the default fdt,as provided by the + 'default-dt' entry argument Properties (in the 'fit' node itself): fit,external-offset: Indicates that the contents of the FIT are external @@ -878,6 +882,15 @@ relocated to any address for execution. +Entry: u-boot-env: An entry which contains a U-Boot environment +--------------------------------------------------------------- + +Properties / Entry arguments: + - filename: File containing the environment text, with each line in the + form var=value + + + Entry: u-boot-img: U-Boot legacy image -------------------------------------- diff --git a/tools/binman/etype/files.py b/tools/binman/etype/files.py index 9adb3afeb14..ce3832e3cdd 100644 --- a/tools/binman/etype/files.py +++ b/tools/binman/etype/files.py @@ -19,7 +19,7 @@ class Entry_files(Entry_section): Properties / Entry arguments: - pattern: Filename pattern to match the files to include - - compress: Compression algorithm to use: + - files-compress: Compression algorithm to use: none: No compression lz4: Use lz4 compression (via 'lz4' command-line utility) @@ -36,7 +36,8 @@ class Entry_files(Entry_section): self._pattern = fdt_util.GetString(self._node, 'pattern') if not self._pattern: self.Raise("Missing 'pattern' property") - self._compress = fdt_util.GetString(self._node, 'compress', 'none') + self._files_compress = fdt_util.GetString(self._node, 'files-compress', + 'none') self._require_matches = fdt_util.GetBool(self._node, 'require-matches') @@ -53,7 +54,7 @@ class Entry_files(Entry_section): subnode = state.AddSubnode(self._node, name) state.AddString(subnode, 'type', 'blob') state.AddString(subnode, 'filename', fname) - state.AddString(subnode, 'compress', self._compress) + state.AddString(subnode, 'compress', self._files_compress) # Read entries again, now that we have some self._ReadEntries() diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index a3e37c33c1b..9222042f5d8 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -160,7 +160,7 @@ class Entry_section(Entry): section_data += tools.GetBytes(self._pad_byte, pad) self.Detail('GetData: %d entries, total size %#x' % (len(self._entries), len(section_data))) - return section_data + return self.CompressData(section_data) def GetOffsets(self): """Handle entries that want to set the offset/size of other entries @@ -414,7 +414,7 @@ class Entry_section(Entry): return None def GetEntryContents(self): - """Call ObtainContents() for the section + """Call ObtainContents() for each entry in the section """ todo = self._entries.values() for passnum in range(3): diff --git a/tools/binman/test/085_files_compress.dts b/tools/binman/test/085_files_compress.dts index 847b398bf2b..5aeead2e6e9 100644 --- a/tools/binman/test/085_files_compress.dts +++ b/tools/binman/test/085_files_compress.dts @@ -5,7 +5,7 @@ binman { files { pattern = "files/*.dat"; - compress = "lz4"; + files-compress = "lz4"; }; }; }; From patchwork Mon Oct 19 02:41:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383956 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=EzQvPQno; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1KD0prqz9sT6 for ; Mon, 19 Oct 2020 13:43:28 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id A1B9C82435; Mon, 19 Oct 2020 04:42:09 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="EzQvPQno"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 3A58F82400; Mon, 19 Oct 2020 04:41:56 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id C2908821A4 for ; Mon, 19 Oct 2020 04:41:52 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x12d.google.com with SMTP id p9so9021594ilr.1 for ; Sun, 18 Oct 2020 19:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tgUTxSj6ELacTmMt38+sYwMmDK67sxOD91/1w6VCVrY=; b=EzQvPQnoG8y3x72PC15RkXsK0ixfLFOKapboyFAG6+Xmdpc+Xrcm3M9y4sv7fyHxNH QhuUkisKsPEli9ISEJZ/sy5k98H0pccBewUnqei9lrK67xqMQmxaPZWzx2nuYIeRNjqf OkI8g3534eby89nv+jIfqOQKlv7EKX3Mflq4o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tgUTxSj6ELacTmMt38+sYwMmDK67sxOD91/1w6VCVrY=; b=nstodd6HlVjFzs+YWPrOQRlro//yJTD1iGNS3iII0ygfkqFmvM6Ar6FAj3KAQfqhOR fffJ+A2sXCe+uliZZ4TCXa1tkGb3RNnWQNuXDcrp5sj0jQq1JTjsRkKh/NmWYMPIL66S PVa5K2p8z2KSWmaqHjoZl9E7xhKqV0gkZBqLLSZbCCwJnrIGpZOsFP3yQzm1eYNhpqdR IIOXio4Vj1CeUblcng1QCQMDCcwY/O9ca3GaBI7nqzWTlly5ATAu/aJHurIi5dOd88x+ ICQu505Lv/UcIYhofYwEYamXbkXBE4x8tj2tW1pxGow4jrS0Xp6pGzfQkuD8+Bpr0x9a ypzg== X-Gm-Message-State: AOAM532xx2jTIb6DSv6tc5CIDsxwx6BYYEJPXMPR2Mig+21EFVg6AGAK r+S0S4OEoHodeMw/dcrGInHQ2JQwDEczk6Zk X-Google-Smtp-Source: ABdhPJx2tFIoexu2HxjDGnsfbU5vc5WNI4hYyjysJwPCXMadvTiWfe6YzQ3dpO+f99pcFf4TAYtEcg== X-Received: by 2002:a92:d742:: with SMTP id e2mr9126990ilq.11.1603075311526; Sun, 18 Oct 2020 19:41:51 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:51 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 07/25] binman: Update testPackExtra with more checks Date: Sun, 18 Oct 2020 20:41:20 -0600 Message-Id: <20201019024138.3804540-7-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Check the contents of each section to make sure it is actually in the right place. Also fix a whitespace error in the .dts file. Signed-off-by: Simon Glass --- tools/binman/ftest.py | 27 ++++++++++++++++++++++----- tools/binman/test/009_pack_extra.dts | 2 +- 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 8e6222ad785..643663d44ec 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -783,9 +783,8 @@ class TestFunctional(unittest.TestCase): def testPackExtra(self): """Test that extra packing feature works as expected""" - retcode = self._DoTestFile('009_pack_extra.dts') + data = self._DoReadFile('009_pack_extra.dts') - self.assertEqual(0, retcode) self.assertIn('image', control.images) image = control.images['image'] entries = image.GetEntries() @@ -797,30 +796,48 @@ class TestFunctional(unittest.TestCase): self.assertEqual(0, entry.offset) self.assertEqual(3, entry.pad_before) self.assertEqual(3 + 5 + len(U_BOOT_DATA), entry.size) + self.assertEqual(U_BOOT_DATA, entry.data) + self.assertEqual(tools.GetBytes(0, 3) + U_BOOT_DATA + + tools.GetBytes(0, 5), data[:entry.size]) + pos = entry.size # Second u-boot has an aligned size, but it has no effect self.assertIn('u-boot-align-size-nop', entries) entry = entries['u-boot-align-size-nop'] - self.assertEqual(12, entry.offset) - self.assertEqual(4, entry.size) + self.assertEqual(pos, entry.offset) + self.assertEqual(len(U_BOOT_DATA), entry.size) + self.assertEqual(U_BOOT_DATA, entry.data) + self.assertEqual(U_BOOT_DATA, data[pos:pos + entry.size]) + pos += entry.size # Third u-boot has an aligned size too self.assertIn('u-boot-align-size', entries) entry = entries['u-boot-align-size'] - self.assertEqual(16, entry.offset) + self.assertEqual(pos, entry.offset) self.assertEqual(32, entry.size) + self.assertEqual(U_BOOT_DATA, entry.data) + self.assertEqual(U_BOOT_DATA + tools.GetBytes(0, 32 - len(U_BOOT_DATA)), + data[pos:pos + entry.size]) + pos += entry.size # Fourth u-boot has an aligned end self.assertIn('u-boot-align-end', entries) entry = entries['u-boot-align-end'] self.assertEqual(48, entry.offset) self.assertEqual(16, entry.size) + self.assertEqual(U_BOOT_DATA, entry.data[:len(U_BOOT_DATA)]) + self.assertEqual(U_BOOT_DATA + tools.GetBytes(0, 16 - len(U_BOOT_DATA)), + data[pos:pos + entry.size]) + pos += entry.size # Fifth u-boot immediately afterwards self.assertIn('u-boot-align-both', entries) entry = entries['u-boot-align-both'] self.assertEqual(64, entry.offset) self.assertEqual(64, entry.size) + self.assertEqual(U_BOOT_DATA, entry.data[:len(U_BOOT_DATA)]) + self.assertEqual(U_BOOT_DATA + tools.GetBytes(0, 64 - len(U_BOOT_DATA)), + data[pos:pos + entry.size]) self.CheckNoGaps(entries) self.assertEqual(128, image.size) diff --git a/tools/binman/test/009_pack_extra.dts b/tools/binman/test/009_pack_extra.dts index 0765707dea2..1b315557716 100644 --- a/tools/binman/test/009_pack_extra.dts +++ b/tools/binman/test/009_pack_extra.dts @@ -28,7 +28,7 @@ u-boot-align-both { type = "u-boot"; - align= <64>; + align = <64>; align-end = <128>; }; }; From patchwork Mon Oct 19 02:41:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383957 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=GbcSbKfY; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1KV4lXFz9sSf for ; Mon, 19 Oct 2020 13:43:42 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 145DC8244E; Mon, 19 Oct 2020 04:42:11 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="GbcSbKfY"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 08BDA82429; Mon, 19 Oct 2020 04:41:57 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id AFE3481F7C for ; Mon, 19 Oct 2020 04:41:53 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x129.google.com with SMTP id l16so8932696ilj.9 for ; Sun, 18 Oct 2020 19:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iQ/oaXXbJk8W1GypnBW8cf8TAj8t+UYDXC3OQFK7Jug=; b=GbcSbKfYkr45oIB+UsVlRMBJjniQaVzxm3SfsfaIwQEwNdoIfOHC8tQgljlt2vFoIz WBVKqfOahqzn28rCYMz1/09ZCEDjKBsmyK7oI9pbGSA3m/QuKeQBUwc+ei242KE67POV HduzvXCmYdfW3WiDcQce88LhCmbSrIhJQ5+Q4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iQ/oaXXbJk8W1GypnBW8cf8TAj8t+UYDXC3OQFK7Jug=; b=irrAvGrQnEQAfarMSmOtAOtmWatpiaC9bE32JLTqRmONDOLl0EXY8Mu0Q/ER9XMYRt ZNF7WFfsV16Icv21b6nQHVaaNVTuFwaopc7EQ1SMg7MFHzjnpiKHNJQ415mLFcJ4CWAa v3WqjE4zGtoa3tOMBQAe4jmVsmhXjz/qKD1TqZA15jb6fDE6B1zxzph74o8sbbo4ZF0R fz6EqFx9nbRsemXcNh7qKhHL4GNcIvAualiIswjP7j5QjrF2SvIeFQyd4BWfLCdkuM7f ghtruDW4P/xAdU35+PBKMg8eWJzWnVRWnhNoPXxUGcrMiTQEgNZwq6aixGbCi62kMfia MnRg== X-Gm-Message-State: AOAM530yKmiAyPrGGgu4ZIzwCYwRFrcok4M3w+B+VNArTr9/zXQHLaSL ADTkgAx+cB9EywRD+PmzonLSOkLkOOIF3Skd X-Google-Smtp-Source: ABdhPJzeKHZQLI8s2dxXLURfxWIjqW7lJDQS49y8uqHAH4oeRnd2f5AedzD0KOZDAeU7BuDxnMjoMw== X-Received: by 2002:a92:d5cc:: with SMTP id d12mr9761059ilq.171.1603075312393; Sun, 18 Oct 2020 19:41:52 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:51 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 08/25] binman: Expand docs and test for padding Date: Sun, 18 Oct 2020 20:41:21 -0600 Message-Id: <20201019024138.3804540-8-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Padding becomes part of the entry once the image is written out, but within binman the entry contents does not include the padding. Add documentation to make this clear, as well as a test. Signed-off-by: Simon Glass --- tools/binman/README | 12 +++++++++--- tools/binman/entry.py | 11 ++++++++--- tools/binman/ftest.py | 29 ++++++++++++++++++++++++++++- 3 files changed, 45 insertions(+), 7 deletions(-) diff --git a/tools/binman/README b/tools/binman/README index fbcfdc77c3e..0433cabce4f 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -290,14 +290,20 @@ size: pad-before: Padding before the contents of the entry. Normally this is 0, meaning - that the contents start at the beginning of the entry. This can be - offset the entry contents a little. Defaults to 0. + that the contents start at the beginning of the entry. This can be used + to offset the entry contents a little. While this does not affect the + contents of the entry within binman itself (the padding is performed + only when its parent section is assembled), the end result will be that + the entry starts with the padding bytes, so may grow. Defaults to 0. pad-after: Padding after the contents of the entry. Normally this is 0, meaning that the entry ends at the last byte of content (unless adjusted by other properties). This allows room to be created in the image for - this entry to expand later. Defaults to 0. + this entry to expand later. While this does not affect the contents of + the entry within binman itself (the padding is performed only when its + parent section is assembled), the end result will be that the entry ends + with the padding bytes, so may grow. Defaults to 0. align-size: This sets the alignment of the entry size. For example, to ensure diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 173c9131cbb..e5d0aa52bd6 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -51,9 +51,14 @@ class Entry(object): align: Entry start offset alignment, or None align_size: Entry size alignment, or None align_end: Entry end offset alignment, or None - pad_before: Number of pad bytes before the contents, 0 if none - pad_after: Number of pad bytes after the contents, 0 if none - data: Contents of entry (string of bytes) + pad_before: Number of pad bytes before the contents when it is placed + in the containing section, 0 if none. The pad bytes become part of + the entry. + pad_after: Number of pad bytes after the contents when it is placed in + the containing section, 0 if none. The pad bytes become part of + the entry. + data: Contents of entry (string of bytes). This does not include + padding created by pad_before or pad_after compress: Compression algoithm used (e.g. 'lz4'), 'none' if none orig_offset: Original offset value read from node orig_size: Original size value read from node diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 643663d44ec..74ee7606cca 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -3546,12 +3546,39 @@ class TestFunctional(unittest.TestCase): def testPadInSections(self): """Test pad-before, pad-after for entries in sections""" - data = self._DoReadFile('166_pad_in_sections.dts') + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '166_pad_in_sections.dts', update_dtb=True) expected = (U_BOOT_DATA + tools.GetBytes(ord('!'), 12) + U_BOOT_DATA + tools.GetBytes(ord('!'), 6) + U_BOOT_DATA) self.assertEqual(expected, data) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['size', 'image-pos', 'offset']) + expected = { + 'image-pos': 0, + 'offset': 0, + 'size': 12 + 6 + 3 * len(U_BOOT_DATA), + + 'section:image-pos': 0, + 'section:offset': 0, + 'section:size': 12 + 6 + 3 * len(U_BOOT_DATA), + + 'section/before:image-pos': 0, + 'section/before:offset': 0, + 'section/before:size': len(U_BOOT_DATA), + + 'section/u-boot:image-pos': 4, + 'section/u-boot:offset': 4, + 'section/u-boot:size': 12 + len(U_BOOT_DATA) + 6, + + 'section/after:image-pos': 26, + 'section/after:offset': 26, + 'section/after:size': len(U_BOOT_DATA), + } + self.assertEqual(expected, props) + def testFitImageSubentryAlignment(self): """Test relative alignability of FIT image subentries""" entry_args = { From patchwork Mon Oct 19 02:41:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383958 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=goVUQbH9; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Kn2DFqz9sSG for ; Mon, 19 Oct 2020 13:43:57 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id AAA9C82398; Mon, 19 Oct 2020 04:42:12 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="goVUQbH9"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id B4A0082431; Mon, 19 Oct 2020 04:41:58 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id CA48F823F8 for ; Mon, 19 Oct 2020 04:41:54 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x136.google.com with SMTP id j13so8989683ilc.4 for ; Sun, 18 Oct 2020 19:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cIeJVT0UaO6Z1uL1sxu3kQNK1JKhFB0rGdPRpTsqAdE=; b=goVUQbH94siu4iktKw7EjCjJfh6MPKOj0AU0YakkiM+1WINHqY87hlGaagBFx426HJ jIdnVLppSVjaXOyLbQbaIH1CDqfgYR39XWkHZKmXJSIQm0FjcQ4f4YKLh3ra2jQ0J+Mh vSAQDXap0jweZxpbzkrMKjM8TIp7v87SpV0Hk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cIeJVT0UaO6Z1uL1sxu3kQNK1JKhFB0rGdPRpTsqAdE=; b=D7OBY4S+KyE8qCt0M7MRKrkcmNZPFD+E0pe8UCIiXWprM/U/0WAJmaiyNSaVRCwNHg C0Wgs5oA20eo/8OylQPoT/fRAFwzCYktzkCzldJiJegVGBF5OE4NM/hP+w5/VN8p0mCv HIvKm5Y+MiwNFIiZ9ydyOspXfP6YpzhOI/p6+5pa2jjXLYY0FbG61JDOROVXXAdgg2Yr WgNxyk9obUxIzitAUY193Pg/MoAaiB69w+XctRtWly1MJnO90Ryc/yAyxsO9WrK3DYgy aAoQ3G6XbC/3TvYXX0UqfWwod4QLXj9Q4G1KuQK54VBU5gcbFjtwFu7GIWjy8jKmFcQx jo2g== X-Gm-Message-State: AOAM532gzYGXPvfayN4E43TBsGX/tvos17QZ9qh4RADFPt8Z89PHJyJE lHQNLXF62LCMadiUdhvC6qB/cBZvRBdx1XmY X-Google-Smtp-Source: ABdhPJyr7oKpaOZOzwm6EPCRAOOOGqyxYDnGjok+21n7dTKbdtGw/TTlra10zYjLXaV8VvTvSe1xxA== X-Received: by 2002:a05:6e02:107:: with SMTP id t7mr9465789ilm.174.1603075313213; Sun, 18 Oct 2020 19:41:53 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:52 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 09/25] binman: Expand docs and test for alignment Date: Sun, 18 Oct 2020 20:41:22 -0600 Message-Id: <20201019024138.3804540-9-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Alignment does form part of the entry once the image is written out, but within binman the entry contents does not include the padding. Add documentation to make this clear, as well as a test. Signed-off-by: Simon Glass --- tools/binman/README | 29 ++++++++++++++++++++--------- tools/binman/entry.py | 6 ++++-- tools/binman/ftest.py | 33 ++++++++++++++++++++++++++++++++- 3 files changed, 56 insertions(+), 12 deletions(-) diff --git a/tools/binman/README b/tools/binman/README index 0433cabce4f..0dee71d1b22 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -278,9 +278,12 @@ offset: align: This sets the alignment of the entry. The entry offset is adjusted - so that the entry starts on an aligned boundary within the image. For - example 'align = <16>' means that the entry will start on a 16-byte - boundary. Alignment shold be a power of 2. If 'align' is not + so that the entry starts on an aligned boundary within the containing + section or image. For example 'align = <16>' means that the entry will + start on a 16-byte boundary. This may mean that padding is added before + the entry. The padding is part of the containing section but is not + included in the entry, meaning that an empty space may be created before + the entry starts. Alignment should be a power of 2. If 'align' is not provided, no alignment is performed. size: @@ -308,14 +311,22 @@ pad-after: align-size: This sets the alignment of the entry size. For example, to ensure that the size of an entry is a multiple of 64 bytes, set this to 64. - If 'align-size' is not provided, no alignment is performed. + While this does not affect the contents of the entry within binman + itself (the padding is performed only when its parent section is + assembled), the end result is that the entry ends with the padding + bytes, so may grow. If 'align-size' is not provided, no alignment is + performed. align-end: - This sets the alignment of the end of an entry. Some entries require - that they end on an alignment boundary, regardless of where they - start. This does not move the start of the entry, so the contents of - the entry will still start at the beginning. But there may be padding - at the end. If 'align-end' is not provided, no alignment is performed. + This sets the alignment of the end of an entry with respect to the + containing section. Some entries require that they end on an alignment + boundary, regardless of where they start. This does not move the start + of the entry, so the contents of the entry will still start at the + beginning. But there may be padding at the end. While this does not + affect the contents of the entry within binman itself (the padding is + performed only when its parent section is assembled), the end result + is that the entry ends with the padding bytes, so may grow. + If 'align-end' is not provided, no alignment is performed. filename: For 'blob' types this provides the filename containing the binary to diff --git a/tools/binman/entry.py b/tools/binman/entry.py index e5d0aa52bd6..0421129c031 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -48,9 +48,11 @@ class Entry(object): uncomp_size: Size of uncompressed data in bytes, if the entry is compressed, else None contents_size: Size of contents in bytes, 0 by default - align: Entry start offset alignment, or None + align: Entry start offset alignment relative to the start of the + containing section, or None align_size: Entry size alignment, or None - align_end: Entry end offset alignment, or None + align_end: Entry end offset alignment relative to the start of the + containing section, or None pad_before: Number of pad bytes before the contents when it is placed in the containing section, 0 if none. The pad bytes become part of the entry. diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 74ee7606cca..e265941a392 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -783,7 +783,8 @@ class TestFunctional(unittest.TestCase): def testPackExtra(self): """Test that extra packing feature works as expected""" - data = self._DoReadFile('009_pack_extra.dts') + data, _, _, out_dtb_fname = self._DoReadFileDtb('009_pack_extra.dts', + update_dtb=True) self.assertIn('image', control.images) image = control.images['image'] @@ -842,6 +843,36 @@ class TestFunctional(unittest.TestCase): self.CheckNoGaps(entries) self.assertEqual(128, image.size) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['size', 'offset', 'image-pos']) + expected = { + 'image-pos': 0, + 'offset': 0, + 'size': 128, + + 'u-boot:image-pos': 0, + 'u-boot:offset': 0, + 'u-boot:size': 3 + 5 + len(U_BOOT_DATA), + + 'u-boot-align-size-nop:image-pos': 12, + 'u-boot-align-size-nop:offset': 12, + 'u-boot-align-size-nop:size': 4, + + 'u-boot-align-size:image-pos': 16, + 'u-boot-align-size:offset': 16, + 'u-boot-align-size:size': 32, + + 'u-boot-align-end:image-pos': 48, + 'u-boot-align-end:offset': 48, + 'u-boot-align-end:size': 16, + + 'u-boot-align-both:image-pos': 64, + 'u-boot-align-both:offset': 64, + 'u-boot-align-both:size': 64, + } + self.assertEqual(expected, props) + def testPackAlignPowerOf2(self): """Test that invalid entry alignment is detected""" with self.assertRaises(ValueError) as e: From patchwork Mon Oct 19 02:41:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383959 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=lyXt9DFO; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1L20GGlz9sSf for ; Mon, 19 Oct 2020 13:44:09 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D161182455; Mon, 19 Oct 2020 04:42:13 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="lyXt9DFO"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id B931E8242B; Mon, 19 Oct 2020 04:41:59 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 857B1823E5 for ; Mon, 19 Oct 2020 04:41:55 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd44.google.com with SMTP id b15so7432896iod.13 for ; Sun, 18 Oct 2020 19:41:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lOsi2dLMpkGPOKGr2dbnGDw9jG3HhNbNKkqcUysFstU=; b=lyXt9DFOUAzIQDyKpSlEaFKobXOtWrZuuYav55pi4+KLZfJd9RmSDp1RreQ7qZBK9L 3VqEZjy7qzQVFkp9jLRHy3/1eHtE1PoQTSeEbs171ma4vG1bxFr2W72eXaA9oviaMRzV gSskl9z9smfKHohZMKfq0aXtbrUfMFwbQHPnI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lOsi2dLMpkGPOKGr2dbnGDw9jG3HhNbNKkqcUysFstU=; b=GF7Xt3gEKcHdIRpLhT0xQMnxFoMRzaX6mr9BC4++nmNK0p3V563p0aszZV8Zcv94Cq /64K4HZnndisIWFgyny6DGYt6HUDT+uwOajJWuDdpH2G+8QiTAW6s2t+0NQNaZD9Le+Q fpno/DHt2pydWLzK8oA5ogKdZxW5wzgP/EBLmPMoLo72IPt7gpHnuuWCdEZPs2fyRE7i Obkhb2WZzF9+0FImRp910hlQ9SodqunYybjxCmORcA9yyBN5PN8meB51w46hD+CMJ/k/ a2r9Ua9yLTfanWfKv9fxMIb+ZiQtXvmW+Wmnd9pG3bSUmhqPQIqdkJWI3WT82FTQVhfv GgKA== X-Gm-Message-State: AOAM531jbtiPne+foAb4qqHjXQL0RtTZA7r4Ew1f6aOz+WU0cLaIrFMn kJDULsHn1GJ0apEwS401CvjNHla5PMllHovE X-Google-Smtp-Source: ABdhPJy1XQsqWhXhGNBBTLu5rKigimekdKZs+lxPxNO75FAroyMWULCgH3YC2dJHIkcarVYTwBMtqA== X-Received: by 2002:a05:6638:2494:: with SMTP id x20mr9599378jat.83.1603075314034; Sun, 18 Oct 2020 19:41:54 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:53 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 10/25] binman: Move section-building code into a function Date: Sun, 18 Oct 2020 20:41:23 -0600 Message-Id: <20201019024138.3804540-10-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Create a new _BuildSectionData() to hold the code that is now in GetData(), so that it is clearly separated from entry.GetData() base function. Separate out the 'pad-before' processing to make this easier to understand. Unfortunately this breaks the testDual test. Rather than squash several patches into an un-reviewable glob, disable the test for now. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 35 +++++++++++++++++++++++++++++------ tools/binman/ftest.py | 3 ++- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 9222042f5d8..d05adf00274 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -144,24 +144,47 @@ class Entry_section(Entry): def ObtainContents(self): return self.GetEntryContents() - def GetData(self): + def _BuildSectionData(self): + """Build the contents of a section + + This places all entries at the right place, dealing with padding before + and after entries. It does not do padding for the section itself (the + pad-before and pad-after properties in the section items) since that is + handled by the parent section. + + Returns: + Contents of the section (bytes) + """ section_data = b'' for entry in self._entries.values(): data = entry.GetData() - base = self.pad_before + (entry.offset or 0) - self._skip_at_start - pad = base - len(section_data) + (entry.pad_before or 0) + # Handle empty space before the entry + pad = (entry.offset or 0) - self._skip_at_start - len(section_data) if pad > 0: section_data += tools.GetBytes(self._pad_byte, pad) + + # Handle padding before the entry + if entry.pad_before: + section_data += tools.GetBytes(self._pad_byte, entry.pad_before) + + # Add in the actual entry data section_data += data + + # Handle padding after the entry + if entry.pad_after: + section_data += tools.GetBytes(self._pad_byte, entry.pad_after) + if self.size: - pad = self.size - len(section_data) - if pad > 0: - section_data += tools.GetBytes(self._pad_byte, pad) + section_data += tools.GetBytes(self._pad_byte, + self.size - len(section_data)) self.Detail('GetData: %d entries, total size %#x' % (len(self._entries), len(section_data))) return self.CompressData(section_data) + def GetData(self): + return self._BuildSectionData() + def GetOffsets(self): """Handle entries that want to set the offset/size of other entries diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e265941a392..3c6eb7f6736 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -706,7 +706,8 @@ class TestFunctional(unittest.TestCase): """Test a simple binman run with debugging enabled""" self._DoTestFile('005_simple.dts', debug=True) - def testDual(self): + # Disable for now until padding of images is supported + def xtestDual(self): """Test that we can handle creating two images This also tests image padding. From patchwork Mon Oct 19 02:41:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383962 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=T8qd63Wj; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1LF69h0z9sSf for ; Mon, 19 Oct 2020 13:44:21 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 9070782464; Mon, 19 Oct 2020 04:42:15 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="T8qd63Wj"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id D83CD82429; Mon, 19 Oct 2020 04:42:00 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 62B3B82413 for ; Mon, 19 Oct 2020 04:41:56 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd41.google.com with SMTP id h21so1773212iob.10 for ; Sun, 18 Oct 2020 19:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZOyzyT4wN0Xq0GpLa6GlF60NhUZTRjGHn997yUTIn4k=; b=T8qd63WjlTT6hEAc3djw/yOJfLFqBQyvFodQX7xAh/NIa+fXGnwiiV0ly12WtbpJ4E rQPRdY2kNykVRi+MRlbjeKKb7usURAwveB3W7XSeP41Vg1MeIpk4SehgGQRetzqCCT2V fAf0RNjTLNlbdZfox1cT7Hw6EgtYLywlL5zDA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZOyzyT4wN0Xq0GpLa6GlF60NhUZTRjGHn997yUTIn4k=; b=dr9CW+LCVT0r5+cx2zojXeyPQRwJEYXaY3jQLPBcz1P6AGHUbiY1+xrjwt0c0E5iUX GoR9xA9sl7WZmwXmlGBgrDbi3zy2jmsSF3+naEAv/kEoW0uVqc+yfT1H6CYlmKqaZQpB ByypLZl9I1FIC1ZoRTvcutqJIYYhQPMwtZ17miuKxXocRev39k8fi2h02R45ONtfz08S juAb2ssY0EpAld8CDg8F99N9PZsQvtOapk1exDNOw2J2UMZ4f9Q98YEe0j2jCuEwG+FI H2sqxwrmfGUO9ksRytUMQD3yt9Wf+FQJCXO/7xomMoytOuOrAF8i5ZQdQrVqGHpN3Xlh xXlw== X-Gm-Message-State: AOAM531Y7iI9oLSihgOdDvAS3/yiccTZtRL4X0lxCxBfxxGhDn87m53j U7dh1zHNjSEVJKjfAD2YbvcLI7F49NjFw6/v X-Google-Smtp-Source: ABdhPJwWf9Nf3yIs5VA3MmjkLJwatQSYXDEWv0BnZqrYvrk0ze9ZS+ZxWPE5bIMi55ZDakhnPbefpg== X-Received: by 2002:a02:b388:: with SMTP id p8mr9503068jan.2.1603075314890; Sun, 18 Oct 2020 19:41:54 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:54 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 11/25] binman: Refactor _BuildSectionData() Date: Sun, 18 Oct 2020 20:41:24 -0600 Message-Id: <20201019024138.3804540-11-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present this function does the padding needed around an entry. It is easier to understand what is going on if we have a function that returns the contents of an entry, with padding included. Refactor the code accordingly, adding a new GetPaddedData() method. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 59 +++++++++++++++++++++++++++++------ tools/binman/image.py | 2 +- 2 files changed, 51 insertions(+), 10 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index d05adf00274..f80432914f2 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -16,6 +16,7 @@ from binman.entry import Entry from dtoc import fdt_util from patman import tools from patman import tout +from patman.tools import ToHexSize class Entry_section(Entry): @@ -144,6 +145,36 @@ class Entry_section(Entry): def ObtainContents(self): return self.GetEntryContents() + def GetPaddedDataForEntry(self, entry): + """Get the data for an entry including any padding + + Gets the entry data and uses the section pad-byte value to add padding + before and after as defined by the pad-before and pad-after properties. + This does not consider alignment. + + Args: + entry: Entry to check + + Returns: + Contents of the entry along with any pad bytes before and + after it (bytes) + """ + data = b'' + # Handle padding before the entry + if entry.pad_before: + data += tools.GetBytes(self._pad_byte, entry.pad_before) + + # Add in the actual entry data + data += entry.GetData() + + # Handle padding after the entry + if entry.pad_after: + data += tools.GetBytes(self._pad_byte, entry.pad_after) + + self.Detail('GetPaddedDataForEntry: size %s' % ToHexSize(self.data)) + + return data + def _BuildSectionData(self): """Build the contents of a section @@ -158,23 +189,15 @@ class Entry_section(Entry): section_data = b'' for entry in self._entries.values(): - data = entry.GetData() + data = self.GetPaddedDataForEntry(entry) # Handle empty space before the entry pad = (entry.offset or 0) - self._skip_at_start - len(section_data) if pad > 0: section_data += tools.GetBytes(self._pad_byte, pad) - # Handle padding before the entry - if entry.pad_before: - section_data += tools.GetBytes(self._pad_byte, entry.pad_before) - # Add in the actual entry data section_data += data - # Handle padding after the entry - if entry.pad_after: - section_data += tools.GetBytes(self._pad_byte, entry.pad_after) - if self.size: section_data += tools.GetBytes(self._pad_byte, self.size - len(section_data)) @@ -182,6 +205,24 @@ class Entry_section(Entry): (len(self._entries), len(section_data))) return self.CompressData(section_data) + def GetPaddedData(self): + """Get the data for a section including any padding + + Gets the section data and uses the parent section's pad-byte value to + add padding before and after as defined by the pad-before and pad-after + properties. If this is a top-level section (i.e. an image), this is the + same as GetData(), since padding is not supported. + + This does not consider alignment. + + Returns: + Contents of the section along with any pad bytes before and + after it (bytes) + """ + if self.section: + return super().GetPaddedData() + return self.GetData() + def GetData(self): return self._BuildSectionData() diff --git a/tools/binman/image.py b/tools/binman/image.py index a8772c3763b..d65ab887b80 100644 --- a/tools/binman/image.py +++ b/tools/binman/image.py @@ -146,7 +146,7 @@ class Image(section.Entry_section): fname = tools.GetOutputFilename(self._filename) tout.Info("Writing image to '%s'" % fname) with open(fname, 'wb') as fd: - data = self.GetData() + data = self.GetPaddedData() fd.write(data) tout.Info("Wrote %#x bytes" % len(data)) From patchwork Mon Oct 19 02:41:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383965 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=ljcYzYOM; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1LT392Tz9sSG for ; Mon, 19 Oct 2020 13:44:33 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 3FB3282471; Mon, 19 Oct 2020 04:42:17 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="ljcYzYOM"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id AF9D082432; Mon, 19 Oct 2020 04:42:01 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id F17E78242A for ; Mon, 19 Oct 2020 04:41:56 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x132.google.com with SMTP id l16so8899958ilt.13 for ; Sun, 18 Oct 2020 19:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nTphH1K98F/cGnBR8PcBrhk+QE3fllAD3OHCXvlz2oQ=; b=ljcYzYOM5HNVIJjX4n2+ReVqbL/83TCv3epBBhqGc3SAe3a1N/AjmkAWqJrCNlEeZD FXGtzDWSJ8D2zTI1Kk7Q/lCUvfP2C0kzMR3Px3WTD66S01YXdddEB3pHvB/KAeOAFrrJ buaxM6KhyYgTydU3R0S2r+fcSoaiRv9W0i77A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nTphH1K98F/cGnBR8PcBrhk+QE3fllAD3OHCXvlz2oQ=; b=MiOLI5POcGKmnQHKKy41QDevBDiEimSGZaq8zrvdf+Hxpe3kvwJPKRx/nSwpHlh1om kGIxAipJ+bUyADyqGWC8cejtYtVro3PC2i9iT/h/GBPomUvL1iyQyDjgNnWe/yGoSpV5 tFJKRYnrV/c6l66/jCMjEnBMdradhhrlFc3HVy1t/HLCSJ0KI/YLCNdoTUaVTw3e4dyF vGVtMCb16sOMyrWQ7b0VbUZ9hVzGm8XOZ/XHp6fiBTwieYXK46EemlaJ1gvR8Sj2ESCl 6oGxulhBi5f2Crs3FJy4eN6XfvUcY8vvfGhlT9mPagq/7s330RRWLNwCta9UUsb0VxOy 0KmA== X-Gm-Message-State: AOAM530f/D1GgAlNepleQ7m/oNESQplILH23YvFvOWkaf1M9aJ+1WvOg yZa1VOf3tEBwnMMvrr1K+8y9u1hcCSs8/n4A X-Google-Smtp-Source: ABdhPJxDyWXHXnJkuGoVBQ7XHEKuvHkDXeYadfuClZB/PYo8C9eIAb5ebhqU5JxXzWiJLeDTBr9AUA== X-Received: by 2002:a92:8bc7:: with SMTP id i190mr5262725ild.121.1603075315724; Sun, 18 Oct 2020 19:41:55 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:55 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 12/25] binman: Move section padding to the parent Date: Sun, 18 Oct 2020 20:41:25 -0600 Message-Id: <20201019024138.3804540-12-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Each section is padded up to its size, if the contents are not large enough. Move this logic from _BuildSectionData() to GetPaddedDataForEntry() so that all the padding is in one place. With this, the testDual test is working again, so enable it. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 18 ++++++++++-------- tools/binman/ftest.py | 3 +-- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index f80432914f2..7cbb50057a3 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -159,17 +159,23 @@ class Entry_section(Entry): Contents of the entry along with any pad bytes before and after it (bytes) """ + pad_byte = (entry._pad_byte if isinstance(entry, Entry_section) + else self._pad_byte) + data = b'' # Handle padding before the entry if entry.pad_before: - data += tools.GetBytes(self._pad_byte, entry.pad_before) + data += tools.GetBytes(pad_byte, entry.pad_before) # Add in the actual entry data data += entry.GetData() # Handle padding after the entry if entry.pad_after: - data += tools.GetBytes(self._pad_byte, entry.pad_after) + data += tools.GetBytes(pad_byte, entry.pad_after) + + if entry.size: + data += tools.GetBytes(pad_byte, entry.size - len(data)) self.Detail('GetPaddedDataForEntry: size %s' % ToHexSize(self.data)) @@ -198,9 +204,6 @@ class Entry_section(Entry): # Add in the actual entry data section_data += data - if self.size: - section_data += tools.GetBytes(self._pad_byte, - self.size - len(section_data)) self.Detail('GetData: %d entries, total size %#x' % (len(self._entries), len(section_data))) return self.CompressData(section_data) @@ -219,9 +222,8 @@ class Entry_section(Entry): Contents of the section along with any pad bytes before and after it (bytes) """ - if self.section: - return super().GetPaddedData() - return self.GetData() + section = self.section or self + return section.GetPaddedDataForEntry(self) def GetData(self): return self._BuildSectionData() diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 3c6eb7f6736..e265941a392 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -706,8 +706,7 @@ class TestFunctional(unittest.TestCase): """Test a simple binman run with debugging enabled""" self._DoTestFile('005_simple.dts', debug=True) - # Disable for now until padding of images is supported - def xtestDual(self): + def testDual(self): """Test that we can handle creating two images This also tests image padding. From patchwork Mon Oct 19 02:41:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383966 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=jvfPsHQm; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Lj1b1Sz9sSG for ; Mon, 19 Oct 2020 13:44:45 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 7FEAD82479; Mon, 19 Oct 2020 04:42:18 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jvfPsHQm"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 0B8C782429; Mon, 19 Oct 2020 04:42:03 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x12f.google.com (mail-il1-x12f.google.com [IPv6:2607:f8b0:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id D14ED823A7 for ; Mon, 19 Oct 2020 04:41:57 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x12f.google.com with SMTP id k1so8940608ilc.10 for ; Sun, 18 Oct 2020 19:41:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KFbGjBEOJ1nldYSyjFfOsOygUdGOF5r9FOIvut1ndzk=; b=jvfPsHQm+2V6WpoVPEkox6LNkoDq+2sclN82BcRT2UTyfztC1seLyZFsnvznLciRFc 6cDuJoLYGHMxGiX9eoxZaTfFdB+FpS41cybGrsVOaGkMMVjtWMP1KnM3o17XhPmhwMXN cE2IwmYWdS3gd066hScKNfix2SY/QiwIvl03A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KFbGjBEOJ1nldYSyjFfOsOygUdGOF5r9FOIvut1ndzk=; b=FNf0/CrHhjSL80NepfpavnJoQ4ugaz9mg2R0PMnUcgPvlnqrZUP6kn/d7zYwG2i3iL g0VXyuFFXdL2FrVevv2NwHXHTGfboBWUba96CxcL278cPeHhMh0IRifQ0RMojT98XCr3 sSz2BmyiWg8UB9cUhG2DYzSmPAGASJIEL3Me/HsYwrbw7BSiLbzVdTmSBwSn0NGo/1eX 9AwEgKeyftQMbNnLn2qXaTUJ0JLABaVpXnVIAniyTpNaAsFAWvFJqT5i8zPRhyonmTwm zK8V7LcoJU+gEu5Zd9UQ/rivohFS2klzhK6jNUPJo2nXujS/GbUofNXw0ADyNw4oh3a2 KZ7w== X-Gm-Message-State: AOAM53243/p9JSwBgy+euyloV/lo6nUfi8peo34f4fu+t3ppxtW3rRik nXd1Rr3BZs+OMSvwvEkGhAog4jpQG+8cPYuZ X-Google-Smtp-Source: ABdhPJyjogmJ3VPZk052tQi8ezsSjdyL1qEoqCttxyj5sn/wVXEKv5H0TzVZgnh2RCWYsaASHPmmoQ== X-Received: by 2002:a05:6e02:970:: with SMTP id q16mr4211992ilt.69.1603075316547; Sun, 18 Oct 2020 19:41:56 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:56 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 13/25] binman: Make section padding consistent with other entries Date: Sun, 18 Oct 2020 20:41:26 -0600 Message-Id: <20201019024138.3804540-13-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present padding of sections is inconsistent with other entry types, in that different pad bytes are used. When a normal entry is padded by its parent, the parent's pad byte is used. But for sections, the section's pad byte is used. Adjust logic to always do this the same way. Note there is still a special case in entry_Section.GetPaddedData() where an image is padded with the pad byte of the top-level section. This is necessary since otherwise there would be no way to set the pad byte of the image, without adding a top-level section to every image. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 4 +-- tools/binman/ftest.py | 22 ++++++++++++++++ tools/binman/test/177_section_pad.dts | 27 ++++++++++++++++++++ tools/binman/test/178_section_align.dts | 34 +++++++++++++++++++++++++ 4 files changed, 85 insertions(+), 2 deletions(-) create mode 100644 tools/binman/test/177_section_pad.dts create mode 100644 tools/binman/test/178_section_align.dts diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 7cbb50057a3..c423a22c80f 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -165,14 +165,14 @@ class Entry_section(Entry): data = b'' # Handle padding before the entry if entry.pad_before: - data += tools.GetBytes(pad_byte, entry.pad_before) + data += tools.GetBytes(self._pad_byte, entry.pad_before) # Add in the actual entry data data += entry.GetData() # Handle padding after the entry if entry.pad_after: - data += tools.GetBytes(pad_byte, entry.pad_after) + data += tools.GetBytes(self._pad_byte, entry.pad_after) if entry.size: data += tools.GetBytes(pad_byte, entry.size - len(data)) diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e265941a392..a43f79cc248 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -3821,6 +3821,28 @@ class TestFunctional(unittest.TestCase): self.assertIn("too small to hold data (need %#x more bytes)" % short, str(e.exception)) + def testSectionPad(self): + """Testing padding with sections""" + data = self._DoReadFile('177_section_pad.dts') + expected = (tools.GetBytes(ord('&'), 3) + + tools.GetBytes(ord('!'), 5) + + U_BOOT_DATA + + tools.GetBytes(ord('!'), 1) + + tools.GetBytes(ord('&'), 2)) + self.assertEqual(expected, data) + + def testSectionAlign(self): + """Testing alignment with sections""" + data = self._DoReadFileDtb('178_section_align.dts', map=True)[0] + expected = (b'\0' + # fill section + tools.GetBytes(ord('&'), 1) + # padding to section align + b'\0' + # fill section + tools.GetBytes(ord('!'), 3) + # padding to u-boot align + U_BOOT_DATA + + tools.GetBytes(ord('!'), 4) + # padding to u-boot size + tools.GetBytes(ord('!'), 4)) # padding to section size + self.assertEqual(expected, data) + if __name__ == "__main__": unittest.main() diff --git a/tools/binman/test/177_section_pad.dts b/tools/binman/test/177_section_pad.dts new file mode 100644 index 00000000000..7e4ebf257b8 --- /dev/null +++ b/tools/binman/test/177_section_pad.dts @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0+ + +/dts-v1/; + +/ { + #address-cells = <1>; + #size-cells = <1>; + + binman { + pad-byte = <0x26>; + section@0 { + read-only; + + /* Padding for the section uses the 0x26 pad byte */ + pad-before = <3>; + pad-after = <2>; + + /* Set the padding byte for entries, i.e. u-boot */ + pad-byte = <0x21>; + + u-boot { + pad-before = <5>; + pad-after = <1>; + }; + }; + }; +}; diff --git a/tools/binman/test/178_section_align.dts b/tools/binman/test/178_section_align.dts new file mode 100644 index 00000000000..90795d131b0 --- /dev/null +++ b/tools/binman/test/178_section_align.dts @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0+ + +/dts-v1/; + +/ { + #address-cells = <1>; + #size-cells = <1>; + + binman { + pad-byte = <0x26>; + fill { + size = <1>; + }; + section@1 { + read-only; + + /* Padding for the section uses the 0x26 pad byte */ + align = <2>; + align-size = <0x10>; + + /* Set the padding byte for entries, i.e. u-boot */ + pad-byte = <0x21>; + + fill { + size = <1>; + }; + + u-boot { + align = <4>; + align-size = <8>; + }; + }; + }; +}; From patchwork Mon Oct 19 02:41:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383967 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=kI+9dJGK; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Lx0Rtnz9sSG for ; Mon, 19 Oct 2020 13:44:56 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id AA55B8248B; Mon, 19 Oct 2020 04:42:19 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="kI+9dJGK"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5D869823FC; Mon, 19 Oct 2020 04:42:03 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id C8E77823F8 for ; Mon, 19 Oct 2020 04:41:58 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd35.google.com with SMTP id r4so11364863ioh.0 for ; Sun, 18 Oct 2020 19:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WJ6xUlbL2FCEn9MT/U5ZGCs8DG2eqYbJQTmhhJDZhG0=; b=kI+9dJGKFDSH6q4woOy+BTKfgrgEBqMrJPf9BM6Q5IpvIAESrM3gwEGz/ZyxCpMzFj t0CQH8GXMG2teOX4/j7kqxzoqlslA42GTBYhbpr8RpS9N1BYHnvPvh9k5jOoqYciQCGW D2IUsmRbiBvj/yX3iiEnxWjNOYCW/R+VIuzCk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WJ6xUlbL2FCEn9MT/U5ZGCs8DG2eqYbJQTmhhJDZhG0=; b=V8UpRlGVXMZmDKvgaKAElWc9PWNcKsiUOwW6udZtSDgnFOVynWEXSx+VhtcG4TrJ8q 50pTKeqlAtDhfyYPvc7+fyKfpfau1L16TRvIwQn/dvEEIe2YakCW8mqPioXgXFoWna17 iGEni6swfiq8MPo5s3w4GH2j0Me+Js4R9sgNgxtob4iD3Aet+HzBI8QvofBTeAlSuniB g85+syfKyCcXfrX2ONpf6oonOx0GxaRLioWswOGmluktSfI6KwyaQ04KEgBBL5GcjWFH QHzd7efb1wSY7DqECZYJPxfbtzs76xIOrtw5FXgcbNFkq4xiHRI8g8ajbkCgITS1bd3i iyxQ== X-Gm-Message-State: AOAM532DKT3cOgNslodYBR1PfJb0fL0aRXttU1QPbrRQAzrY01rFxaGa bU1ybmuog78zOWSMEf9HGOiYpXzOrMCaLh4Q X-Google-Smtp-Source: ABdhPJzX5/MxNJHGFWP/qzHOl5WPhZu6g+V7ow4Fe4sof23Lbrhvs1jWquI49B5DrTinK38Zb9CqNw== X-Received: by 2002:a02:6cd0:: with SMTP id w199mr9819858jab.121.1603075317379; Sun, 18 Oct 2020 19:41:57 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:56 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 14/25] binman: Store the original data before compression Date: Sun, 18 Oct 2020 20:41:27 -0600 Message-Id: <20201019024138.3804540-14-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean When compressing an entry, the original uncompressed data is overwritten. Store it so it is available if needed. Signed-off-by: Simon Glass --- tools/binman/entry.py | 7 ++++++- tools/binman/ftest.py | 12 ++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 0421129c031..d701eaff8fd 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -60,7 +60,10 @@ class Entry(object): the containing section, 0 if none. The pad bytes become part of the entry. data: Contents of entry (string of bytes). This does not include - padding created by pad_before or pad_after + padding created by pad_before or pad_after. If the entry is + compressed, this contains the compressed data. + uncomp_data: Original uncompressed data, if this entry is compressed, + else None compress: Compression algoithm used (e.g. 'lz4'), 'none' if none orig_offset: Original offset value read from node orig_size: Original size value read from node @@ -83,6 +86,7 @@ class Entry(object): self.pre_reset_size = None self.uncomp_size = None self.data = None + self.uncomp_data = None self.contents_size = 0 self.align = None self.align_size = None @@ -856,6 +860,7 @@ features to produce new behaviours. Returns: Compressed data (first word is the compressed size) """ + self.uncomp_data = indata if self.compress != 'none': self.uncomp_size = len(indata) data = tools.Compress(indata, self.compress) diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index a43f79cc248..7360e2ebaba 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -1808,6 +1808,18 @@ class TestFunctional(unittest.TestCase): props = self._GetPropTree(dtb, ['size', 'uncomp-size']) orig = self._decompress(data) self.assertEquals(COMPRESS_DATA, orig) + + # Do a sanity check on various fields + image = control.images['image'] + entries = image.GetEntries() + self.assertEqual(1, len(entries)) + + entry = entries['blob'] + self.assertEqual(COMPRESS_DATA, entry.uncomp_data) + self.assertEqual(len(COMPRESS_DATA), entry.uncomp_size) + orig = self._decompress(entry.data) + self.assertEqual(orig, entry.uncomp_data) + expected = { 'blob:uncomp-size': len(COMPRESS_DATA), 'blob:size': len(data), From patchwork Mon Oct 19 02:41:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383968 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=E5YI+Euw; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1MB2tzcz9sSf for ; Mon, 19 Oct 2020 13:45:10 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D697F8247D; Mon, 19 Oct 2020 04:42:26 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="E5YI+Euw"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5DC7082434; Mon, 19 Oct 2020 04:42:04 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 8A4B082423 for ; Mon, 19 Oct 2020 04:41:59 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd42.google.com with SMTP id q9so11275665iow.6 for ; Sun, 18 Oct 2020 19:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ycR2O976HKneSyXCQwU1zAjsG0XS4RpeYTrAG4zig7Y=; b=E5YI+EuwP9jHtJfPgJ2Dgzm6evP0slrNtvbAZS+VMYhveIXK7UVMsyU/48fg01meSK R7ntJNHxDfk5vF5ZVHoqi6/RbSuBjysbXBcWNr+CZjbB9cNS/GWIEhrHU40vJSEPcS2L Y5GzFd2Lr3fXdtTRpdh2qPPd+8N4kteXjMEgo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ycR2O976HKneSyXCQwU1zAjsG0XS4RpeYTrAG4zig7Y=; b=N1OFhu6p5cXWPdzz+mQLiyysbX2rxeOmrX8H0cHjMITKZCwfsKPKjF8uFR8kDFfOOF SZvTWnvbs+CpdYJScJMVDaOUZY2Yezc6sfLA4Hj5BhGoYlnQdewwJG9yTHHRUf+WVGIA 6s0xK62TzqcqciHEnFOeSThrde/VEWxh7ui3PbuL5hdBiziqyy3zwAgxMDq21lx/mcWz 0QpfsWAXHTxuDKJkGGQSkeaQHlZRf48JPTtmf4OippBwsnZ7+UtIAMZW/SLWAr3+WuFh 2vc3Gen443AcVRhE1oavxrlOrSpHuMjllPSb6QdhWxSOx9IgEvcM7cLuLkc8VV3DZBtG JdHA== X-Gm-Message-State: AOAM5311lG0jKGafAabCfAZ8QoCXBqvWkpmlF84vBUAyJeEfidPErZAp Np87nta8KXJVv3RzIleR9O1HtUXRSVJUCc3e X-Google-Smtp-Source: ABdhPJzo5qaGzkWmbRbKQwBqT/ANGP5rn3NaLEAPIou7ueA/vIFizVeOwlUL/hXw/hYkSK4yUkGkpQ== X-Received: by 2002:a6b:1505:: with SMTP id 5mr9258577iov.3.1603075318203; Sun, 18 Oct 2020 19:41:58 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:57 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 15/25] binman: Set section contents in GetData() Date: Sun, 18 Oct 2020 20:41:28 -0600 Message-Id: <20201019024138.3804540-15-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Section contents is not set up when ObtainContents() is called, since packing often changes the layout of the contents. Ensure that the contents are correctly recorded by making this function regenerate the section. It is normally only called by the parent section (when packing) or by the top-level image code, when writing out the image. So the performance impact is fairly small. Signed-off-by: Simon Glass --- tools/binman/entry.py | 6 ++++++ tools/binman/etype/section.py | 14 +++++++++++++- tools/binman/ftest.py | 2 ++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/tools/binman/entry.py b/tools/binman/entry.py index d701eaff8fd..01a5fde84ed 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -437,6 +437,12 @@ class Entry(object): return self._node.path def GetData(self): + """Get the contents of an entry + + Returns: + bytes content of the entry, excluding any padding. If the entry is + compressed, the compressed data is returned + """ self.Detail('GetData: size %s' % ToHexSize(self.data)) return self.data diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index c423a22c80f..6e6f6749727 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -226,7 +226,19 @@ class Entry_section(Entry): return section.GetPaddedDataForEntry(self) def GetData(self): - return self._BuildSectionData() + """Get the contents of an entry + + This builds the contents of the section, stores this as the contents of + the section and returns it + + Returns: + bytes content of the section, made up for all all of its subentries. + This excludes any padding. If the section is compressed, the + compressed data is returned + """ + data = self._BuildSectionData() + self.SetContents(data) + return data def GetOffsets(self): """Handle entries that want to set the offset/size of other entries diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 7360e2ebaba..bcd4482717a 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -1820,6 +1820,8 @@ class TestFunctional(unittest.TestCase): orig = self._decompress(entry.data) self.assertEqual(orig, entry.uncomp_data) + self.assertEqual(image.data, entry.data) + expected = { 'blob:uncomp-size': len(COMPRESS_DATA), 'blob:size': len(data), From patchwork Mon Oct 19 02:41:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383969 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=kluRkDbN; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1MN5KMKz9sSG for ; Mon, 19 Oct 2020 13:45:20 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 3134082495; Mon, 19 Oct 2020 04:42:28 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="kluRkDbN"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 0321E8243D; Mon, 19 Oct 2020 04:42:05 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 79AB0823E5 for ; Mon, 19 Oct 2020 04:42:00 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd43.google.com with SMTP id b15so7433038iod.13 for ; Sun, 18 Oct 2020 19:42:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KNXbN/GuCOi281r6iE9fmAhR4mc8bEqAnHF8g5nt5XE=; b=kluRkDbNmfxZMpYNt/n5brFu4AbbU6ACNegDP3gu8i7dha/PRSH0l0IGX9YsRo2eph kEgiNN6Vjt27SqzM7dNnRvx2czjKkT1RHPa/fi0yOETzlFjdONVNKUIqw8r6pJWvD+I2 GMtHW7cpgDEwyDsC/4fHNVucbuRm7aFx9wG4c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KNXbN/GuCOi281r6iE9fmAhR4mc8bEqAnHF8g5nt5XE=; b=WNz2Jlvv+5E5cVTfTBysLNV8sqw6A6NpdsQrzufbk+28KBUVEoEc50tQzJ0RtPfEde m4LI5dEgqtGPjq+w/GXiU/FcSRxT11J1GbbBA58UAbPQOOPV3TNq5PvlCkdPsTpndl4D bqCUwLzwPIXOJQnbMSptMUFUdLB0ysjSFkuM6fY4T0UF0FsQNgG7iLqw5m+7AK0djNRl XhdbwF2TWnh6n16qrsjwlEehbL8GysY0s+FaGKJbNqBqAx9qfO3eiRKbBninqoZbKQKx c3rODwLitKz8x0mUbBBoZJVxJdhiD1Cm6nRaBbAaIcwIHWWWjjetpgmbqhcic07gff0W wLHg== X-Gm-Message-State: AOAM533YSKTw1FyPtypDcD3R7kGCwR5Na3zJ8RjeD7HcSi5FwRaCfres l3ImII5Z6xYB5Tkiqvs1KvpngsPhc91JxJaW X-Google-Smtp-Source: ABdhPJxupoDO8KLuE0pzwxghGboRMBvGfx6z27mm36FoO3Cm0EVstUlvK2Slm7BOdQVQ8v1nSkPppA== X-Received: by 2002:a5d:8b4c:: with SMTP id c12mr9249937iot.167.1603075319138; Sun, 18 Oct 2020 19:41:59 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:58 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 16/25] binman: Avoid reporting image-pos with compression Date: Sun, 18 Oct 2020 20:41:29 -0600 Message-Id: <20201019024138.3804540-16-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean When a section is compressed, all entries within it are grouped together into a compressed block of data. This obscures the start of each individual child entry. Avoid reporting bogus 'image-pos' properties in this case, since it is not possible to access the entry at the location provided. The entire section must be decompressed first. CBFS does not support compressing whole sections, only individual files, so needs no special handling here. Signed-off-by: Simon Glass --- tools/binman/control.py | 2 +- tools/binman/entry.py | 18 ++++++++++++++---- tools/binman/etype/cbfs.py | 6 +++--- tools/binman/etype/section.py | 13 ++++++++----- 4 files changed, 26 insertions(+), 13 deletions(-) diff --git a/tools/binman/control.py b/tools/binman/control.py index ee5771e7292..26f1cf462ec 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -462,7 +462,7 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt): for image in images.values(): image.ExpandEntries() if update_fdt: - image.AddMissingProperties() + image.AddMissingProperties(True) image.ProcessFdt(dtb) for dtb_item in state.GetAllFdts(): diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 01a5fde84ed..8fa1dcef2da 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -213,11 +213,20 @@ class Entry(object): def ExpandEntries(self): pass - def AddMissingProperties(self): - """Add new properties to the device tree as needed for this entry""" - for prop in ['offset', 'size', 'image-pos']: + def AddMissingProperties(self, have_image_pos): + """Add new properties to the device tree as needed for this entry + + Args: + have_image_pos: True if this entry has an image position. This can + be False if its parent section is compressed, since compression + groups all entries together into a compressed block of data, + obscuring the start of each individual child entry + """ + for prop in ['offset', 'size']: if not prop in self._node.props: state.AddZeroProp(self._node, prop) + if have_image_pos and 'image-pos' not in self._node.props: + state.AddZeroProp(self._node, 'image-pos') if self.GetImage().allow_repack: if self.orig_offset is not None: state.AddZeroProp(self._node, 'orig-offset', True) @@ -235,7 +244,8 @@ class Entry(object): state.SetInt(self._node, 'offset', self.offset) state.SetInt(self._node, 'size', self.size) base = self.section.GetRootSkipAtStart() if self.section else 0 - state.SetInt(self._node, 'image-pos', self.image_pos - base) + if self.image_pos is not None: + state.SetInt(self._node, 'image-pos', self.image_pos) if self.GetImage().allow_repack: if self.orig_offset is not None: state.SetInt(self._node, 'orig-offset', self.orig_offset, True) diff --git a/tools/binman/etype/cbfs.py b/tools/binman/etype/cbfs.py index 650ab2c292f..6cdbaa085f5 100644 --- a/tools/binman/etype/cbfs.py +++ b/tools/binman/etype/cbfs.py @@ -237,10 +237,10 @@ class Entry_cbfs(Entry): if entry._cbfs_compress: entry.uncomp_size = cfile.memlen - def AddMissingProperties(self): - super().AddMissingProperties() + def AddMissingProperties(self, have_image_pos): + super().AddMissingProperties(have_image_pos) for entry in self._cbfs_entries.values(): - entry.AddMissingProperties() + entry.AddMissingProperties(have_image_pos) if entry._cbfs_compress: state.AddZeroProp(entry._node, 'uncomp-size') # Store the 'compress' property, since we don't look at diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 6e6f6749727..2812989ba1a 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -136,11 +136,13 @@ class Entry_section(Entry): for entry in self._entries.values(): entry.ExpandEntries() - def AddMissingProperties(self): + def AddMissingProperties(self, have_image_pos): """Add new properties to the device tree as needed for this entry""" - super().AddMissingProperties() + super().AddMissingProperties(have_image_pos) + if self.compress != 'none': + have_image_pos = False for entry in self._entries.values(): - entry.AddMissingProperties() + entry.AddMissingProperties(have_image_pos) def ObtainContents(self): return self.GetEntryContents() @@ -323,8 +325,9 @@ class Entry_section(Entry): def SetImagePos(self, image_pos): super().SetImagePos(image_pos) - for entry in self._entries.values(): - entry.SetImagePos(image_pos + self.offset) + if self.compress == 'none': + for entry in self._entries.values(): + entry.SetImagePos(image_pos + self.offset) def ProcessContents(self): sizes_ok_base = super(Entry_section, self).ProcessContents() From patchwork Mon Oct 19 02:41:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383970 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=Y0375arZ; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1MZ0STXz9sSG for ; Mon, 19 Oct 2020 13:45:30 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id CF900824A5; Mon, 19 Oct 2020 04:42:29 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="Y0375arZ"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5D86782434; Mon, 19 Oct 2020 04:42:05 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 3764C8242B for ; Mon, 19 Oct 2020 04:42:01 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd41.google.com with SMTP id z5so2708883iob.1 for ; Sun, 18 Oct 2020 19:42:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gVq1SH71fv/Wy/Yr2BrJbqjj8SiDYDR/QeEefDysqyM=; b=Y0375arZKuPWNQOXpZufg6DScsL0Yl6L3HtwRdMD9U9VBFuT3fv/n8KdHs2oLfTtc3 P9140E4qBOHHKxQR5uXKnTC+ICbiT/iDb1Zm4rB5SbuC0BGP+nxV9vsBjP/bqpodhNzX jZGXZ8CxRhhfzauJrCgiPtnQEVhdz/HpYZJjY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gVq1SH71fv/Wy/Yr2BrJbqjj8SiDYDR/QeEefDysqyM=; b=KJ6G+5iPd+2cGcuq928XUS/qb4zyLG1+IF1XN0A6BAzUzVw3l6AW+21L4rS7XdwGH4 2MEv75korkjp81+q5cuET+mjhLuot9SXzTY8OxiM67zOHk27xksD7e+OFiFo0kNOHiiG KGHEnAQ8XJGB2wa4eBKA3s+vDp7qIIpeC6tod2uDDA+tiX1znzNp/fOsNMESMeayj3A7 2XOOPg8fOfu7aPWbULBMA4+gNsYf5SDxj1Xtxqfq6nzPpnQSmIUVHkYeZi+iTT8ENHGo aVoDBxaOLoIn5vNs0z8megQuUwuIJwL5fBCKhSmn98P8csoBAbShNk6MIVfQMrWLRvbz uzmA== X-Gm-Message-State: AOAM530hNWstL753AQXuv+cxRBB9TflizzwJmSRgtQS4EYGdlQAnR3pP ozQ+gc7/qp1c1UCHFBqIek8sTtxmf5uizdEk X-Google-Smtp-Source: ABdhPJxLkv6j2hWJXw8pkYINM7U3kK9N6gOtRemW1hf4d6QPi6iPj4qZFTPJIHi65wAHqBNCxo7/gQ== X-Received: by 2002:a5d:9656:: with SMTP id d22mr3928065ios.50.1603075320010; Sun, 18 Oct 2020 19:42:00 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.41.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:41:59 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 17/25] binman: Drop Entry.CheckOffset() Date: Sun, 18 Oct 2020 20:41:30 -0600 Message-Id: <20201019024138.3804540-17-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean This function just calls CheckEntries() in the only non-trivial implementation. Drop it and use CheckEntries() directly. Signed-off-by: Simon Glass --- tools/binman/entry.py | 2 +- tools/binman/etype/section.py | 5 +---- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 8fa1dcef2da..8946d2bc02f 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -520,7 +520,7 @@ class Entry(object): """ pass - def CheckOffset(self): + def CheckEntries(self): """Check that the entry offsets are correct This is used for entries which have extra offset requirements (other diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 2812989ba1a..fb4bf640bfb 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -296,7 +296,7 @@ class Entry_section(Entry): offset = 0 prev_name = 'None' for entry in self._entries.values(): - entry.CheckOffset() + entry.CheckEntries() if (entry.offset < self._skip_at_start or entry.offset + entry.size > self._skip_at_start + self.size): @@ -337,9 +337,6 @@ class Entry_section(Entry): sizes_ok = False return sizes_ok and sizes_ok_base - def CheckOffset(self): - self.CheckEntries() - def WriteMap(self, fd, indent): """Write a map of the section to a .map file From patchwork Mon Oct 19 02:41:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383971 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=jYnOo2LW; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Mk4HrLz9sSf for ; Mon, 19 Oct 2020 13:45:38 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2F963824AC; Mon, 19 Oct 2020 04:42:31 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jYnOo2LW"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id A72858242B; Mon, 19 Oct 2020 04:42:06 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 0F52082435 for ; Mon, 19 Oct 2020 04:42:02 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd41.google.com with SMTP id b8so11223774ioh.11 for ; Sun, 18 Oct 2020 19:42:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+XsVv/FMe+6rkX5Z6st/FTicLNhQFIBMhtqRwkfC5hE=; b=jYnOo2LWtRYE/jHZcotoUsHNpWPKQwRmUgN2WQmcZfuL+agLFTL9ZiJAyq9AaE0Gw4 FbAJCa84PZqSiZyIwU+DpgQuY1kZUL4PzZihjU3/O21yrjlqsonDxYNDSL0uSXTwGnl+ OabD3bWAhfMCgbOgggQBQC19646EYrmo08Z20= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+XsVv/FMe+6rkX5Z6st/FTicLNhQFIBMhtqRwkfC5hE=; b=j/vH+cI4ooOmNBItWjW8h7nIwijaVuOwkL95nqekWdyz2uUPKmDDwZPzxWlzF3bV0a 3T2RPFrLxy8CZG0rB7RUjQFyzCn0iqMuQ0WyyHXF9vqPUCGN8GSYIqkahlrMYktNKbzt coo2WzqLkeSE0LJ1H27QhdEs7BCtZ4K4Bes9Pqfe2KWPlyC+4n9jd8sOGPLAfrOktt7q sWC18Orr16p7I60AWZVSTVNnzFKNdChTPUqpJFB7O55xzv9ZSRNgB6t2UHlHprgM0JXX dVag4eDUppcXUEiQjO4Kby/uU4TdPTeoDUhGSzWI55XG0w45RnM3Jw846dVc/FUoNmtm jN0Q== X-Gm-Message-State: AOAM531Kky8Ybiu5e+cwp6nwRaNQhSOdSBZ4h2IfVmQn85+MbafmLE+7 0HLaOnIOWUohmfvUy5Jaw6NKJfPCce+DMUcI X-Google-Smtp-Source: ABdhPJwrlzIHYnxhv73va0m4FS3ZRJnYHBsB59h4W9Hop8VTeXwvuVR9aLX6A8FnLYWB/DHt6Kb88w== X-Received: by 2002:a6b:7f43:: with SMTP id m3mr9011791ioq.114.1603075320776; Sun, 18 Oct 2020 19:42:00 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:00 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 18/25] binman: Move sort and expand to the main Pack() function Date: Sun, 18 Oct 2020 20:41:31 -0600 Message-Id: <20201019024138.3804540-18-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present sorting and expanding entries are side-effects of the CheckEntries() function. This is a bit confusing, as 'checking' would not normally involve making changes. Move these steps into the Pack() function instead. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index fb4bf640bfb..c883f0d67c4 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -260,6 +260,10 @@ class Entry_section(Entry): def Pack(self, offset): """Pack all entries into the section""" self._PackEntries() + if self._sort: + self._SortEntries() + self._ExpandEntries() + return super().Pack(offset) def _PackEntries(self): @@ -290,9 +294,6 @@ class Entry_section(Entry): def CheckEntries(self): """Check that entries do not overlap or extend outside the section""" - if self._sort: - self._SortEntries() - self._ExpandEntries() offset = 0 prev_name = 'None' for entry in self._entries.values(): From patchwork Mon Oct 19 02:41:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383974 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=OdDx6Cxa; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1NG1xBpz9sSG for ; Mon, 19 Oct 2020 13:46:05 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D4155824C6; Mon, 19 Oct 2020 04:42:34 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="OdDx6Cxa"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 10E2482435; Mon, 19 Oct 2020 04:42:08 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id E96098243E for ; Mon, 19 Oct 2020 04:42:02 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd2a.google.com with SMTP id q25so11300678ioh.4 for ; Sun, 18 Oct 2020 19:42:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nIkzpi0ZMHYsO3qIMq/mW9/M9wJ2JCAcNvyBO7H2e7c=; b=OdDx6Cxa2qfigIy3yCPvNX6hLDhS3sd4Rh1bVTRCdK10GR0trsRSUKJ8uOrDP/R4YS Owk+gSKRCyoa3fhJB8tJZ6LhY7nTCfS0GDGcdBD38zHXj1vkL3peiuwOIwmYZ5Uzt4Fs A446QLIrAyNmkapSevJ1gysNkOxOM80X4uv5Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nIkzpi0ZMHYsO3qIMq/mW9/M9wJ2JCAcNvyBO7H2e7c=; b=k4/Ewkvqjx1roAGSmV1A8l0TJPGjmX9fT9PzyJSC7LQF0vf7+s7NjViFAY1CAtjq1i lG5rdCgA7LePJaKh9udwHempFv75RUSmUZ/SXeqX5B4NwC83Moy+T3NR5g8q2x4pOBvk 0S2PaiO1F/au7bT+7vHobFgDAsgFVAm/h323GAnB+KgshDXWFusPzc6ru8xWZN+GGDyI s8BpoaZZaLCwDAzwKTux2HRA+19ejRQm1ajI1rp0GHB71aVgkpUq8LSx8swQtAALk3Qo HmSqm/G36/A3iF5R5p5QgC+exgZEIGHN3rv5BJNbz5o/x3h87shpsQQ+3jSO1hRh+i3r ZpFQ== X-Gm-Message-State: AOAM533GhcCE5//KVQADtvcwbOqnQPNg/H+X2bMsxoiCm5K6YgL/DbnM u7itxCbh110okkQPXfSPNsSINOhMdUFzt8P0 X-Google-Smtp-Source: ABdhPJy0F4pNWWoJYPtWZRo81mFkJPmMzvk8ALXe3JXo9ak3K/aem7pACu3K6WtD+3xriAzRWE8SZQ== X-Received: by 2002:a6b:3ac5:: with SMTP id h188mr9583837ioa.173.1603075321580; Sun, 18 Oct 2020 19:42:01 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:01 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 19/25] binman: Drop the Entry.CheckSize() method Date: Sun, 18 Oct 2020 20:41:32 -0600 Message-Id: <20201019024138.3804540-19-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean This is only used by entry_Section and that class already calls it. Avoid calling it twice. Also drop it from the documentation. Signed-off-by: Simon Glass --- tools/binman/README | 21 ++++++++++----------- tools/binman/control.py | 1 - 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/tools/binman/README b/tools/binman/README index 0dee71d1b22..c7c787c99fd 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -712,41 +712,40 @@ size of an entry. The 'current' image offset is passed in, and the function returns the offset immediately after the entry being packed. The default implementation of Pack() is usually sufficient. -6. CheckSize() - checks that the contents of all the entries fits within -the image size. If the image does not have a defined size, the size is set -large enough to hold all the entries. +Note: for sections, this also sets the size of the entry to be large enough +for all entries it contains. -7. CheckEntries() - checks that the entries do not overlap, nor extend +6. CheckEntries() - checks that the entries do not overlap, nor extend outside the image. -8. SetImagePos() - sets the image position of every entry. This is the absolute +7. SetImagePos() - sets the image position of every entry. This is the absolute position 'image-pos', as opposed to 'offset' which is relative to the containing section. This must be done after all offsets are known, which is why it is quite late in the ordering. -9. SetCalculatedProperties() - update any calculated properties in the device +8. SetCalculatedProperties() - update any calculated properties in the device tree. This sets the correct 'offset' and 'size' vaues, for example. -10. ProcessEntryContents() - this calls Entry.ProcessContents() on each entry. +9. ProcessEntryContents() - this calls Entry.ProcessContents() on each entry. The default implementatoin does nothing. This can be overriden to adjust the contents of an entry in some way. For example, it would be possible to create an entry containing a hash of the contents of some other entries. At this stage the offset and size of entries should not be adjusted unless absolutely necessary, since it requires a repack (going back to PackEntries()). -11. ResetForPack() - if the ProcessEntryContents() step failed, in that an entry +10. ResetForPack() - if the ProcessEntryContents() step failed, in that an entry has changed its size, then there is no alternative but to go back to step 5 and try again, repacking the entries with the updated size. ResetForPack() removes the fixed offset/size values added by binman, so that the packing can start from scratch. -12. WriteSymbols() - write the value of symbols into the U-Boot SPL binary. +11. WriteSymbols() - write the value of symbols into the U-Boot SPL binary. See 'Access to binman entry offsets at run time' below for a description of what happens in this stage. -13. BuildImage() - builds the image and writes it to a file +12. BuildImage() - builds the image and writes it to a file -14. WriteMap() - writes a text file containing a map of the image. This is the +13. WriteMap() - writes a text file containing a map of the image. This is the final step. diff --git a/tools/binman/control.py b/tools/binman/control.py index 26f1cf462ec..9eeac5db995 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -513,7 +513,6 @@ def ProcessImage(image, update_fdt, write_map, get_contents=True, for pack_pass in range(passes): try: image.PackEntries() - image.CheckSize() image.CheckEntries() except Exception as e: if write_map: From patchwork Mon Oct 19 02:41:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383972 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=a2T9h4Pg; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Mv04Vpz9sSG for ; Mon, 19 Oct 2020 13:45:46 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 51674824B5; Mon, 19 Oct 2020 04:42:32 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="a2T9h4Pg"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id CF8768244F; Mon, 19 Oct 2020 04:42:07 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id ADDDE823F8 for ; Mon, 19 Oct 2020 04:42:03 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd43.google.com with SMTP id n6so11191695ioc.12 for ; Sun, 18 Oct 2020 19:42:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hI+3tc/3eDiqFoxa94qmA+iQE/NnMhWx6eiDnbCWY+w=; b=a2T9h4PgGHJjcBeLUIM7UqXrb9640E9ncFQY6sBFhJhW2dlVli6yEIsFmayqBmPTEw oUn2QyJMFJysoHpOXx62LYdESQ3elDmW8y2ZIwro5z9eBHCPZRyo6IYkyYj6YSK70H47 qw1gfLkgzpNNWBDcP5QLzfS0Lb7+KHVcE4Kpw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hI+3tc/3eDiqFoxa94qmA+iQE/NnMhWx6eiDnbCWY+w=; b=k2Ern+x+fvEnu9J39bRWKnG8cqKVzMolnVFFTTicS0nXYjJ6iWZV4sraaH5cdfkLjp vJGEWoM640UMkq0otjtM1bY2kJH1iGTHFnzUs/kjPvIeGkHuDRZtCE4/n6D9NpcM/1QB yOQf4aLATAuFZCt4So1Q3r7uk0AaHAUt6SQNR9dUBV09lq/vAUFQpzGVIcbul6a9NCRg 48unnSURN27Vx/g5QUMNZrJ+job2KgoA+U7OdD2cH1nmurwayOpx3jusI0tMMCB6+8bk dquSevIG+V0gVyKyNxE+k65Uku2mnHJ+8nBxpl/KrTzC6UECvYd8G0ue4M9Sl3iAM3gY /n3g== X-Gm-Message-State: AOAM53025CXf6XlZhVDnANT343hXdjyjVfCAfnlnmFAetavCDuwRdzE9 tCgfr+NiIQHtdU4Zx50ACKESN4xdUVIMGFAk X-Google-Smtp-Source: ABdhPJzpBGSz3DbjfvG3xL0yRI6f+oMrO7UMJoKK8wC7/QyIa8bcwJxcOXJqDc00x0MIzpfQgPu5Uw== X-Received: by 2002:a6b:908a:: with SMTP id s132mr9294542iod.62.1603075322471; Sun, 18 Oct 2020 19:42:02 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:02 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 20/25] binman: Call CheckSize() from the section's Pack() method Date: Sun, 18 Oct 2020 20:41:33 -0600 Message-Id: <20201019024138.3804540-20-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present CheckSize() is called from the function that packs the entries. Move it up to the main Pack() function so that _PackEntries() can just do the packing. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index c883f0d67c4..f93469a170b 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -264,6 +264,9 @@ class Entry_section(Entry): self._SortEntries() self._ExpandEntries() + size = self.CheckSize() + self.size = size + return super().Pack(offset) def _PackEntries(self): @@ -271,7 +274,7 @@ class Entry_section(Entry): offset = self._skip_at_start for entry in self._entries.values(): offset = entry.Pack(offset) - self.size = self.CheckSize() + return offset def _ExpandEntries(self): """Expand any entries that are permitted to""" From patchwork Mon Oct 19 02:41:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383973 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=NxMBVuZT; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1N42sknz9sSG for ; Mon, 19 Oct 2020 13:45:56 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id BEF59824BE; Mon, 19 Oct 2020 04:42:33 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="NxMBVuZT"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 4787782449; Mon, 19 Oct 2020 04:42:08 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id D54C68243C for ; Mon, 19 Oct 2020 04:42:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd33.google.com with SMTP id b8so11223843ioh.11 for ; Sun, 18 Oct 2020 19:42:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=psL+DinhR0N85jdVU+1jYJ3ksFsX4H9T8IQYX6MCkeU=; b=NxMBVuZTXxg6M+qkuOQbdaFZZJ5fLTw38krjjMA2EvHagC0lYhJawpCq2yqogdM1eI 7RtUUlE8MVWbp5I/epGpl/VLOuLIL1YbxCJ21oi0m8XKNOhrEhpQ8ztskRVKEdLgFeBj nmXP+4iuF3OCbF63+P128isV4XO+sMeyo0Idw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=psL+DinhR0N85jdVU+1jYJ3ksFsX4H9T8IQYX6MCkeU=; b=rF+Y0yXwwki8cRsNU/25SNXq7wensLktId9zAXdNzLvKLKv8iPlnr0W9cfLkBTnvXf GIq9antPFthADGi8DmVvE5b/ppRftPi1ZPHwk9+zz0fbO52mHV8bchBJLGdSa+vAj7K5 T3k70Cm5wkkXyhrCLv4KfGizqQleUqrpszfj2+namF2CgmkulZG1Gnw9Z+zeG+qgnNoB wunxSfxsL4KPpF3pYq5UoT1LPsF+zBfX83eil8O0fjXNDmsg+SOCcyfFS9HDj2l0IOP7 fUheB1Ennl6x3MUlMW3d+qPYQFNUx+hDa0OH62Q/giPLFFcKddMQbIIzEFT3S9TaI1mT 3mbw== X-Gm-Message-State: AOAM531zKyfOZWxpYZ7DgPqpsPP/lgfxFHMfuLLX+jzMdBYm+0racweL bUSVpaq4Cr7AuF2SukAB/wlBQRbBqNax5ASD X-Google-Smtp-Source: ABdhPJxRvnoOQ+GcWrg5d6Pnj+qHj+Y5a3oqbixU+H91gvz1saIMzOOWq+l0CeXq2LirBL5z5dj0kw== X-Received: by 2002:a05:6638:c3:: with SMTP id w3mr9185471jao.108.1603075323239; Sun, 18 Oct 2020 19:42:03 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:02 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 21/25] binman: Drop CheckEntries() Date: Sun, 18 Oct 2020 20:41:34 -0600 Message-Id: <20201019024138.3804540-21-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean This method introduces a separation between packing and checking that is different for sections. In order to handle compression properly, we need to be able to deal with a section's size being smaller than the uncompressed size of its contents. It is easier to make this work if everything happens in the Pack() method. The only real user of CheckEntries() is entry_Section and it can call it directly. Drop the call from 'control' and handle it locally. Signed-off-by: Simon Glass --- tools/binman/README | 22 ++++++++++------------ tools/binman/control.py | 1 - tools/binman/etype/section.py | 4 +++- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/tools/binman/README b/tools/binman/README index c7c787c99fd..c14cee5d115 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -712,40 +712,38 @@ size of an entry. The 'current' image offset is passed in, and the function returns the offset immediately after the entry being packed. The default implementation of Pack() is usually sufficient. -Note: for sections, this also sets the size of the entry to be large enough -for all entries it contains. +Note: for sections, this also checks that the entries do not overlap, nor extend +outside the section. If the section does not have a defined size, the size is +set large enough to hold all the entries. -6. CheckEntries() - checks that the entries do not overlap, nor extend -outside the image. - -7. SetImagePos() - sets the image position of every entry. This is the absolute +6. SetImagePos() - sets the image position of every entry. This is the absolute position 'image-pos', as opposed to 'offset' which is relative to the containing section. This must be done after all offsets are known, which is why it is quite late in the ordering. -8. SetCalculatedProperties() - update any calculated properties in the device +7. SetCalculatedProperties() - update any calculated properties in the device tree. This sets the correct 'offset' and 'size' vaues, for example. -9. ProcessEntryContents() - this calls Entry.ProcessContents() on each entry. +8. ProcessEntryContents() - this calls Entry.ProcessContents() on each entry. The default implementatoin does nothing. This can be overriden to adjust the contents of an entry in some way. For example, it would be possible to create an entry containing a hash of the contents of some other entries. At this stage the offset and size of entries should not be adjusted unless absolutely necessary, since it requires a repack (going back to PackEntries()). -10. ResetForPack() - if the ProcessEntryContents() step failed, in that an entry +9. ResetForPack() - if the ProcessEntryContents() step failed, in that an entry has changed its size, then there is no alternative but to go back to step 5 and try again, repacking the entries with the updated size. ResetForPack() removes the fixed offset/size values added by binman, so that the packing can start from scratch. -11. WriteSymbols() - write the value of symbols into the U-Boot SPL binary. +10. WriteSymbols() - write the value of symbols into the U-Boot SPL binary. See 'Access to binman entry offsets at run time' below for a description of what happens in this stage. -12. BuildImage() - builds the image and writes it to a file +11. BuildImage() - builds the image and writes it to a file -13. WriteMap() - writes a text file containing a map of the image. This is the +12. WriteMap() - writes a text file containing a map of the image. This is the final step. diff --git a/tools/binman/control.py b/tools/binman/control.py index 9eeac5db995..072417f3644 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -513,7 +513,6 @@ def ProcessImage(image, update_fdt, write_map, get_contents=True, for pack_pass in range(passes): try: image.PackEntries() - image.CheckEntries() except Exception as e: if write_map: fname = image.WriteMap() diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index f93469a170b..1618bebe4be 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -267,7 +267,9 @@ class Entry_section(Entry): size = self.CheckSize() self.size = size - return super().Pack(offset) + offset = super().Pack(offset) + self.CheckEntries() + return offset def _PackEntries(self): """Pack all entries into the section""" From patchwork Mon Oct 19 02:41:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383975 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=aV5bq5Ik; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1NR3Ss0z9sSG for ; Mon, 19 Oct 2020 13:46:15 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id DDA66824B7; Mon, 19 Oct 2020 04:42:35 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="aV5bq5Ik"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 890B48243E; Mon, 19 Oct 2020 04:42:09 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 3E32182432 for ; Mon, 19 Oct 2020 04:42:05 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd42.google.com with SMTP id u19so11335156ion.3 for ; Sun, 18 Oct 2020 19:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EmfJ8rm8w0Uen+uKLa6VBKVEeOd/hggH3awg/OognHo=; b=aV5bq5IkTHmFEGNrYITdN5GTuwwaItsPqqhekFcIzkF9pVJvd1yk0cUmdysqqiPsem vot45U5EFzOsCpLFpK1GdAZse/7mVLEZyJgrvTmV54lwwbmI9OSphVVfDb1Ca7rOPSjR jB5gBiSWta6dM6quFZnnPijduMtPxF4fGsKWA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EmfJ8rm8w0Uen+uKLa6VBKVEeOd/hggH3awg/OognHo=; b=qi4+BjLLUJbd46/SA1nG5jl1bnTRcwMshYwfaO6FELTXYdd1Ket7XV+jkfv6ZQquv/ SyMLXT0VAA8zFwcR2cxw8N7biLNphnluXew/tXetGXdt6h9xt4ZGopUCMRyKz2+73xLX GQoIBFJWZ+0muGHfcVCKbZord20CpiFCIkaAOc9Zl8WxzRixMIjvL8fYyvCueIP1rA+Y RC5YmPPzt2Sll+9zpXZ3zZS81QxPZYupjK8jvfGMNbKSN/K8PYOPWyqZIuyJDidGDN/C u0VdTmcWWJ7L9xjRAi03zVove/yW+6Xlt0ECRJU0rsZAg4nK7fICmGVbrT6cpx5x816N Iv2Q== X-Gm-Message-State: AOAM530XXNW8+M0laQ4hDkmqGMYBDwCzXvfttsqRMyPZeV3TDmNQJGUZ SXb3RLnx+W+sLaL46OipIqlmdH4nvlkn8cvN X-Google-Smtp-Source: ABdhPJz43tOGv5yhP3vXxA6AR1KOBoZ2D1F6t6nYwKI2/VyEkyDCJ24/G1SkQYCEwKM50wEmNGLUBg== X-Received: by 2002:a02:3b57:: with SMTP id i23mr10262313jaf.110.1603075324068; Sun, 18 Oct 2020 19:42:04 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:03 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 22/25] binman: Update CheckEntries() for compressed sections Date: Sun, 18 Oct 2020 20:41:35 -0600 Message-Id: <20201019024138.3804540-22-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present this function assumes that the size of a section is at least as large as its contents. With compression this is often not the case. Relax this constraint by using the uncompressed size, if available. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 1618bebe4be..b146239b779 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -299,19 +299,21 @@ class Entry_section(Entry): def CheckEntries(self): """Check that entries do not overlap or extend outside the section""" + max_size = self.size if self.uncomp_size is None else self.uncomp_size + offset = 0 prev_name = 'None' for entry in self._entries.values(): entry.CheckEntries() if (entry.offset < self._skip_at_start or entry.offset + entry.size > self._skip_at_start + - self.size): + max_size): entry.Raise('Offset %#x (%d) size %#x (%d) is outside the ' "section '%s' starting at %#x (%d) " 'of size %#x (%d)' % (entry.offset, entry.offset, entry.size, entry.size, self._node.path, self._skip_at_start, - self._skip_at_start, self.size, self.size)) + self._skip_at_start, max_size, max_size)) if entry.offset < offset and entry.size: entry.Raise("Offset %#x (%d) overlaps with previous entry '%s' " "ending at %#x (%d)" % From patchwork Mon Oct 19 02:41:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383976 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=F8uCTYm9; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Nc6PqZz9sSG for ; Mon, 19 Oct 2020 13:46:24 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2B69B824C7; Mon, 19 Oct 2020 04:42:37 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="F8uCTYm9"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2EC2A8244C; Mon, 19 Oct 2020 04:42:11 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 405F882440 for ; Mon, 19 Oct 2020 04:42:06 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd44.google.com with SMTP id u19so11335170ion.3 for ; Sun, 18 Oct 2020 19:42:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rbAFEx4IcJQkAE2f8DBJOwGB9EePomks03+vXoRixzQ=; b=F8uCTYm9gOERLN8XqPczw5+k38GecUXu/fCQsiNz6ELcgWlYuLOc/i3THrYsSygGWL fo2uGQVtEbcSSViV7C9xdxHybjpVYUzdO7OWrJ2OOo8a8Pi57qpQwtcKQWljLiLfoDtB Lp7GFvTPBWg73Wi6nBLGAh8RGLZD7arYvML40= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rbAFEx4IcJQkAE2f8DBJOwGB9EePomks03+vXoRixzQ=; b=fg8BBDm62z4b8O9rWBblzmzg/NKFSD4SZcE6ouXP8KR4KP5CgDbeRku+E9tF7WyHbx AcNB+4hB+JD0RXlvel4VLxe8hi3052ZpEoj+pIKs6YaiLCBAChQ52l02GV0DyszIM3V4 bZU0rabKRVZTnk1wWs5jKxPQ7wWiej3aHbDTZexWPmWxU5Lg0OqoG1yYR7ES4QgQW46A MCFJRjKt1cd9KA1MiMw2iT37/mZpPkQ1NoY5LkqoHi2QzkZT4bHDFri+lOItqdH0+MGC Xnxh/rpJVaCiQcwbnn28FqBvOickxp0b2k7UR07R9fEGkMq19NWEjx9xLfutjwIrnd56 Y35g== X-Gm-Message-State: AOAM53174cdGijdaZ7/KlN7M/Nji1zCLNEFrosIxt5g938WebzrS2g7H 8nYA7E4VdO44lePlEJnUEqSLHajnaoVzeu6D X-Google-Smtp-Source: ABdhPJyAZUvxb1cQmBIHLFKA7KwEOqK54ZtGIJ3cWadcEQkVPLRXgsLi4IwG9LDoZvFkJUBLYJB8aw== X-Received: by 2002:a02:7817:: with SMTP id p23mr9583197jac.57.1603075324852; Sun, 18 Oct 2020 19:42:04 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:04 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 23/25] binman: Use the actual contents in CheckSize() Date: Sun, 18 Oct 2020 20:41:36 -0600 Message-Id: <20201019024138.3804540-23-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present this function adds up the total size of entries to work out the size of a section's contents. With compression this is no-longer enough. We may as well bite the bullet and build the section contents instead. Call _BuildSectionData() to get the (possibly compressed) contents and GetPaddedData() to get the same but with padding added. Note that this is inefficient since the section contents is calculated twice. Future work will improve this. This affects testPackOverlapMap() since the error is reported with a different section size now (enough to hold the contents). Update that at the same time. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 11 ++++------- tools/binman/ftest.py | 2 +- 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index b146239b779..570dbfcfd41 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -544,16 +544,13 @@ class Entry_section(Entry): def CheckSize(self): - """Check that the section contents does not exceed its size, etc.""" - contents_size = 0 - for entry in self._entries.values(): - contents_size = max(contents_size, entry.offset + entry.size) - - contents_size -= self._skip_at_start + data = self._BuildSectionData() + contents_size = len(data) size = self.size if not size: - size = self.pad_before + contents_size + self.pad_after + data = self.GetPaddedData() + size = len(data) size = tools.Align(size, self.align_size) if self.size and contents_size > self.size: diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index bcd4482717a..8b7537e4eca 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -2022,7 +2022,7 @@ class TestFunctional(unittest.TestCase): self.assertTrue(os.path.exists(map_fname)) map_data = tools.ReadFile(map_fname, binary=False) self.assertEqual('''ImagePos Offset Size Name - 00000000 00000007 main-section + 00000000 00000008 main-section 00000000 00000004 u-boot 00000003 00000004 u-boot-align ''', map_data) From patchwork Mon Oct 19 02:41:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383977 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=cGR9rYVn; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1Np275Zz9sSG for ; Mon, 19 Oct 2020 13:46:34 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 5919F824D5; Mon, 19 Oct 2020 04:42:38 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="cGR9rYVn"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 02D0882452; Mon, 19 Oct 2020 04:42:12 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 7BECB82443 for ; Mon, 19 Oct 2020 04:42:07 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd43.google.com with SMTP id b8so11223902ioh.11 for ; Sun, 18 Oct 2020 19:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JsPUX/iIZaHHYNdFAHWAuA3epbANvy/l4EA3/PVUcME=; b=cGR9rYVnkTe63ayj7N+bnNxLxhGFI1f3zOGhIYBlSnTNQoJu1JFXVHy22Fhl8091Wq zq/ADr4M4KwuhZNnNOdneyBEnLSfJf9KVXhdtKDd2YzzlCVwy2wQwXlVwL//6bcMr/1P ddhzjuk/AiCvMTfknMgCmRUPaIMB2fIyr8DMQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JsPUX/iIZaHHYNdFAHWAuA3epbANvy/l4EA3/PVUcME=; b=uVfnqKmW6ohWVAzPMl1dq38Pnxf56y3YlLs4w0BKd3B2OWatbT/GhsH0WPbcpBFx2U ROZvNcaelQInrZ2riZjOFZ39EVjvLt1bqbfUbpGP4DuTGZGX2R9h99+zcAFlk6SDc9Fz rxd0ZTeLnOENUlH7NQ9BGvknGZCNOgXSJC/cgqGRpxzLBtr2v0qM/YHwzMr9T/YZ/4Di xQ5L3ebt0e8QL3QqHF5nMjq1fMEp+V/Hmg2XYqljYuiXoUDRJxYHiQQsgz5ZCH/eJvZ5 TsoCw1jAaIOiAZm4X6MQdoIR5XuVuo/OD619BSZYCiFgEUElgXyaPMDMAVBb5uS6XKkX wkdg== X-Gm-Message-State: AOAM533uK541PymwmEYqdRsCHJbqZx1/3wpBoodLE3lw8dzqGvHn2fsi S27ZOiJhj5UCrkfhtUySeBDTrVip7fHTV8HU X-Google-Smtp-Source: ABdhPJzQhpODVrv/FVvI3UHsaZ/FTPNrFGPa5aADfutvyWi9EowK6361bcmCDR3CbC782LDWCNsKYg== X-Received: by 2002:a02:6045:: with SMTP id d5mr9709780jaf.144.1603075325621; Sun, 18 Oct 2020 19:42:05 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:05 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass Subject: [PATCH 24/25] binman: Support compression of sections Date: Sun, 18 Oct 2020 20:41:37 -0600 Message-Id: <20201019024138.3804540-24-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean With the previous changes, it is now possible to compress entire sections. Add some tests to check that compression works correctly, including updating the metadata. Also update the documentation. Signed-off-by: Simon Glass --- tools/binman/README | 8 + tools/binman/ftest.py | 217 ++++++++++++++++++ tools/binman/test/179_compress_image.dts | 14 ++ tools/binman/test/180_compress_image_less.dts | 14 ++ .../binman/test/181_compress_section_size.dts | 17 ++ tools/binman/test/182_compress_section.dts | 16 ++ tools/binman/test/183_compress_extra.dts | 37 +++ 7 files changed, 323 insertions(+) create mode 100644 tools/binman/test/179_compress_image.dts create mode 100644 tools/binman/test/180_compress_image_less.dts create mode 100644 tools/binman/test/181_compress_section_size.dts create mode 100644 tools/binman/test/182_compress_section.dts create mode 100644 tools/binman/test/183_compress_extra.dts diff --git a/tools/binman/README b/tools/binman/README index c14cee5d115..de1eedfc3f7 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -853,6 +853,14 @@ The entry will then contain the compressed data, using the 'lz4' compression algorithm. Currently this is the only one that is supported. The uncompressed size is written to the node in an 'uncomp-size' property, if -u is used. +Compression is also supported for sections. In that case the entire section is +compressed in one block, including all its contents. This means that accessing +an entry from the section required decompressing the entire section. Also, the +size of a section indicates the space that it consumes in its parent section +(and typically the image). With compression, the section may contain more data, +and the uncomp-size property indicates that, as above. The contents of the +section is compressed first, before any padding is added. This ensures that the +padding itself is not compressed, which would be a waste of time. Map files diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 8b7537e4eca..403cd836b71 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -70,6 +70,7 @@ VBLOCK_DATA = b'vblk' FILES_DATA = (b"sorry I'm late\nOh, don't bother apologising, I'm " + b"sorry you're alive\n") COMPRESS_DATA = b'compress xxxxxxxxxxxxxxxxxxxxxx data' +COMPRESS_DATA_BIG = COMPRESS_DATA * 2 REFCODE_DATA = b'refcode' FSP_M_DATA = b'fsp_m' FSP_S_DATA = b'fsp_s' @@ -174,6 +175,7 @@ class TestFunctional(unittest.TestCase): os.path.join(cls._indir, 'files')) TestFunctional._MakeInputFile('compress', COMPRESS_DATA) + TestFunctional._MakeInputFile('compress_big', COMPRESS_DATA_BIG) TestFunctional._MakeInputFile('bl31.bin', ATF_BL31_DATA) # Add a few .dtb files for testing @@ -3857,6 +3859,221 @@ class TestFunctional(unittest.TestCase): tools.GetBytes(ord('!'), 4)) # padding to section size self.assertEqual(expected, data) + def testCompressImage(self): + """Test compression of the entire image""" + self._CheckLz4() + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '179_compress_image.dts', use_real_dtb=True, update_dtb=True) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', + 'uncomp-size']) + orig = self._decompress(data) + self.assertEquals(COMPRESS_DATA + U_BOOT_DATA, orig) + + # Do a sanity check on various fields + image = control.images['image'] + entries = image.GetEntries() + self.assertEqual(2, len(entries)) + + entry = entries['blob'] + self.assertEqual(COMPRESS_DATA, entry.data) + self.assertEqual(len(COMPRESS_DATA), entry.size) + + entry = entries['u-boot'] + self.assertEqual(U_BOOT_DATA, entry.data) + self.assertEqual(len(U_BOOT_DATA), entry.size) + + self.assertEqual(len(data), image.size) + self.assertEqual(COMPRESS_DATA + U_BOOT_DATA, image.uncomp_data) + self.assertEqual(len(COMPRESS_DATA + U_BOOT_DATA), image.uncomp_size) + orig = self._decompress(image.data) + self.assertEqual(orig, image.uncomp_data) + + expected = { + 'blob:offset': 0, + 'blob:size': len(COMPRESS_DATA), + 'u-boot:offset': len(COMPRESS_DATA), + 'u-boot:size': len(U_BOOT_DATA), + 'uncomp-size': len(COMPRESS_DATA + U_BOOT_DATA), + 'offset': 0, + 'image-pos': 0, + 'size': len(data), + } + self.assertEqual(expected, props) + + def testCompressImageLess(self): + """Test compression where compression reduces the image size""" + self._CheckLz4() + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '180_compress_image_less.dts', use_real_dtb=True, update_dtb=True) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', + 'uncomp-size']) + orig = self._decompress(data) + + self.assertEquals(COMPRESS_DATA + COMPRESS_DATA + U_BOOT_DATA, orig) + + # Do a sanity check on various fields + image = control.images['image'] + entries = image.GetEntries() + self.assertEqual(2, len(entries)) + + entry = entries['blob'] + self.assertEqual(COMPRESS_DATA_BIG, entry.data) + self.assertEqual(len(COMPRESS_DATA_BIG), entry.size) + + entry = entries['u-boot'] + self.assertEqual(U_BOOT_DATA, entry.data) + self.assertEqual(len(U_BOOT_DATA), entry.size) + + self.assertEqual(len(data), image.size) + self.assertEqual(COMPRESS_DATA_BIG + U_BOOT_DATA, image.uncomp_data) + self.assertEqual(len(COMPRESS_DATA_BIG + U_BOOT_DATA), + image.uncomp_size) + orig = self._decompress(image.data) + self.assertEqual(orig, image.uncomp_data) + + expected = { + 'blob:offset': 0, + 'blob:size': len(COMPRESS_DATA_BIG), + 'u-boot:offset': len(COMPRESS_DATA_BIG), + 'u-boot:size': len(U_BOOT_DATA), + 'uncomp-size': len(COMPRESS_DATA_BIG + U_BOOT_DATA), + 'offset': 0, + 'image-pos': 0, + 'size': len(data), + } + self.assertEqual(expected, props) + + def testCompressSectionSize(self): + """Test compression of a section with a fixed size""" + self._CheckLz4() + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '181_compress_section_size.dts', use_real_dtb=True, update_dtb=True) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', + 'uncomp-size']) + orig = self._decompress(data) + self.assertEquals(COMPRESS_DATA + U_BOOT_DATA, orig) + expected = { + 'section/blob:offset': 0, + 'section/blob:size': len(COMPRESS_DATA), + 'section/u-boot:offset': len(COMPRESS_DATA), + 'section/u-boot:size': len(U_BOOT_DATA), + 'section:offset': 0, + 'section:image-pos': 0, + 'section:uncomp-size': len(COMPRESS_DATA + U_BOOT_DATA), + 'section:size': 0x30, + 'offset': 0, + 'image-pos': 0, + 'size': 0x30, + } + self.assertEqual(expected, props) + + def testCompressSection(self): + """Test compression of a section with no fixed size""" + self._CheckLz4() + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '182_compress_section.dts', use_real_dtb=True, update_dtb=True) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', + 'uncomp-size']) + orig = self._decompress(data) + self.assertEquals(COMPRESS_DATA + U_BOOT_DATA, orig) + expected = { + 'section/blob:offset': 0, + 'section/blob:size': len(COMPRESS_DATA), + 'section/u-boot:offset': len(COMPRESS_DATA), + 'section/u-boot:size': len(U_BOOT_DATA), + 'section:offset': 0, + 'section:image-pos': 0, + 'section:uncomp-size': len(COMPRESS_DATA + U_BOOT_DATA), + 'section:size': len(data), + 'offset': 0, + 'image-pos': 0, + 'size': len(data), + } + self.assertEqual(expected, props) + + def testCompressExtra(self): + """Test compression of a section with no fixed size""" + self._CheckLz4() + data, _, _, out_dtb_fname = self._DoReadFileDtb( + '183_compress_extra.dts', use_real_dtb=True, update_dtb=True) + dtb = fdt.Fdt(out_dtb_fname) + dtb.Scan() + props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', + 'uncomp-size']) + + base = data[len(U_BOOT_DATA):] + self.assertEquals(U_BOOT_DATA, base[:len(U_BOOT_DATA)]) + rest = base[len(U_BOOT_DATA):] + + # Check compressed data + section1 = self._decompress(rest) + expect1 = tools.Compress(COMPRESS_DATA + U_BOOT_DATA, 'lz4') + self.assertEquals(expect1, rest[:len(expect1)]) + self.assertEquals(COMPRESS_DATA + U_BOOT_DATA, section1) + rest1 = rest[len(expect1):] + + section2 = self._decompress(rest1) + expect2 = tools.Compress(COMPRESS_DATA + COMPRESS_DATA, 'lz4') + self.assertEquals(expect2, rest1[:len(expect2)]) + self.assertEquals(COMPRESS_DATA + COMPRESS_DATA, section2) + rest2 = rest1[len(expect2):] + + expect_size = (len(U_BOOT_DATA) + len(U_BOOT_DATA) + len(expect1) + + len(expect2) + len(U_BOOT_DATA)) + #self.assertEquals(expect_size, len(data)) + + #self.assertEquals(U_BOOT_DATA, rest2) + + self.maxDiff = None + expected = { + 'u-boot:offset': 0, + 'u-boot:image-pos': 0, + 'u-boot:size': len(U_BOOT_DATA), + + 'base:offset': len(U_BOOT_DATA), + 'base:image-pos': len(U_BOOT_DATA), + 'base:size': len(data) - len(U_BOOT_DATA), + 'base/u-boot:offset': 0, + 'base/u-boot:image-pos': len(U_BOOT_DATA), + 'base/u-boot:size': len(U_BOOT_DATA), + 'base/u-boot2:offset': len(U_BOOT_DATA) + len(expect1) + + len(expect2), + 'base/u-boot2:image-pos': len(U_BOOT_DATA) * 2 + len(expect1) + + len(expect2), + 'base/u-boot2:size': len(U_BOOT_DATA), + + 'base/section:offset': len(U_BOOT_DATA), + 'base/section:image-pos': len(U_BOOT_DATA) * 2, + 'base/section:size': len(expect1), + 'base/section:uncomp-size': len(COMPRESS_DATA + U_BOOT_DATA), + 'base/section/blob:offset': 0, + 'base/section/blob:size': len(COMPRESS_DATA), + 'base/section/u-boot:offset': len(COMPRESS_DATA), + 'base/section/u-boot:size': len(U_BOOT_DATA), + + 'base/section2:offset': len(U_BOOT_DATA) + len(expect1), + 'base/section2:image-pos': len(U_BOOT_DATA) * 2 + len(expect1), + 'base/section2:size': len(expect2), + 'base/section2:uncomp-size': len(COMPRESS_DATA + COMPRESS_DATA), + 'base/section2/blob:offset': 0, + 'base/section2/blob:size': len(COMPRESS_DATA), + 'base/section2/blob2:offset': len(COMPRESS_DATA), + 'base/section2/blob2:size': len(COMPRESS_DATA), + + 'offset': 0, + 'image-pos': 0, + 'size': len(data), + } + self.assertEqual(expected, props) + if __name__ == "__main__": unittest.main() diff --git a/tools/binman/test/179_compress_image.dts b/tools/binman/test/179_compress_image.dts new file mode 100644 index 00000000000..4176b7f2e62 --- /dev/null +++ b/tools/binman/test/179_compress_image.dts @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0+ +/dts-v1/; + +/ { + binman { + compress = "lz4"; + blob { + filename = "compress"; + }; + + u-boot { + }; + }; +}; diff --git a/tools/binman/test/180_compress_image_less.dts b/tools/binman/test/180_compress_image_less.dts new file mode 100644 index 00000000000..1d9d57b78c9 --- /dev/null +++ b/tools/binman/test/180_compress_image_less.dts @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0+ +/dts-v1/; + +/ { + binman { + compress = "lz4"; + blob { + filename = "compress_big"; + }; + + u-boot { + }; + }; +}; diff --git a/tools/binman/test/181_compress_section_size.dts b/tools/binman/test/181_compress_section_size.dts new file mode 100644 index 00000000000..95ed30add1a --- /dev/null +++ b/tools/binman/test/181_compress_section_size.dts @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0+ +/dts-v1/; + +/ { + binman { + section { + size = <0x30>; + compress = "lz4"; + blob { + filename = "compress"; + }; + + u-boot { + }; + }; + }; +}; diff --git a/tools/binman/test/182_compress_section.dts b/tools/binman/test/182_compress_section.dts new file mode 100644 index 00000000000..dc3e340c5d6 --- /dev/null +++ b/tools/binman/test/182_compress_section.dts @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0+ +/dts-v1/; + +/ { + binman { + section { + compress = "lz4"; + blob { + filename = "compress"; + }; + + u-boot { + }; + }; + }; +}; diff --git a/tools/binman/test/183_compress_extra.dts b/tools/binman/test/183_compress_extra.dts new file mode 100644 index 00000000000..59aae822638 --- /dev/null +++ b/tools/binman/test/183_compress_extra.dts @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0+ +/dts-v1/; + +/ { + binman { + u-boot { + }; + base { + type = "section"; + u-boot { + }; + section { + compress = "lz4"; + blob { + filename = "compress"; + }; + + u-boot { + }; + }; + section2 { + type = "section"; + compress = "lz4"; + blob { + filename = "compress"; + }; + blob2 { + type = "blob"; + filename = "compress"; + }; + }; + u-boot2 { + type = "u-boot"; + }; + }; + }; +}; From patchwork Mon Oct 19 02:41:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1383978 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=Jrr6bDat; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CF1P053cjz9sSG for ; Mon, 19 Oct 2020 13:46:44 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2466C824DA; Mon, 19 Oct 2020 04:42:40 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="Jrr6bDat"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5068D8244F; Mon, 19 Oct 2020 04:42:12 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 293F08243D for ; Mon, 19 Oct 2020 04:42:08 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd41.google.com with SMTP id k25so11279769ioh.7 for ; Sun, 18 Oct 2020 19:42:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vUVcHQ4Wv1c271lq6r/0FI16lH+cEUMkMm7onKuzqVQ=; b=Jrr6bDatlxxJ7nSke4LoTyVAbKR7nxZM6oDSquYYnrlNrBuGQX5amyyka57AwKs5i/ ImXQJXB0NvaXUy5s0txPe4WSEoIWeNEIYvqJib5vJsSboJnYvmdc/2LFgatQVUYwpTV1 4tWST9jnOXR6+j+QlsvIggkdN2Bj/KXLn6quQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vUVcHQ4Wv1c271lq6r/0FI16lH+cEUMkMm7onKuzqVQ=; b=lcogOBNjVo2/eVrF/h7u72Z1Ea4UlBEZayqU99i+HKVimvXnl4W0Oo5pzif3Yp7Wb2 5gWhse2biJBIPONmOQlxIq05qMOFmQG3m2xZLpT231yLbpEJarSdWUFGQ/maobi7vOKN otog8KHyg76OdFfh+kBhZD2ykh9hD47abHVI2TXb+2hpH+S7yOL6JpLI//B+57sO+csz 9J4BW7qz99LVO92mGRHziWLZ8dpVSzpp2sDcUnmxYM9d7uAN/bRibj1UBfyYgCgoxBsG jJRe2ZK+9iLs+/NzM3on7fYURBxoCiCiRUFwXQ5JyvxJcCB2ghzo2DtO40ZFRib3ClhR d4hQ== X-Gm-Message-State: AOAM533lIrQbKeegJHcEHiHPE0ED3M1UiyGK40oKtQ5+gNv4IAYPsCig LR+XJEbuJnmIuBNy6+CS0Jd6FoMGxh0TqX5q X-Google-Smtp-Source: ABdhPJzZOJ8ov3HoXid+YqlAtNBym/NwK2uQHqOiDtg9D8VUe6RmiLC9gRZXz807/x9s/o5flS6EnQ== X-Received: by 2002:a02:6d61:: with SMTP id e33mr9496091jaf.14.1603075326585; Sun, 18 Oct 2020 19:42:06 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id g185sm9750871ilh.35.2020.10.18.19.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Oct 2020 19:42:06 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Alper Nebi Yasak , Bin Meng Subject: [PATCH 25/25] binman: Avoid calculated section data repeatedly Date: Sun, 18 Oct 2020 20:41:38 -0600 Message-Id: <20201019024138.3804540-25-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201019024138.3804540-1-sjg@chromium.org> References: <20201019024138.3804540-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Refactor the implementation slightly so that section data is not rebuilt when it is already available. We still have GetData() set up to rebuild the section, since we don't currently track when things change that might affect a section. For example, if a blob is updated within a section, we must rebuild it. Tracking that would be possible but is more complex, so it left for another time. Signed-off-by: Simon Glass --- tools/binman/etype/section.py | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/tools/binman/etype/section.py b/tools/binman/etype/section.py index 570dbfcfd41..3dd5f58c4c2 100644 --- a/tools/binman/etype/section.py +++ b/tools/binman/etype/section.py @@ -147,7 +147,7 @@ class Entry_section(Entry): def ObtainContents(self): return self.GetEntryContents() - def GetPaddedDataForEntry(self, entry): + def GetPaddedDataForEntry(self, entry, entry_data): """Get the data for an entry including any padding Gets the entry data and uses the section pad-byte value to add padding @@ -170,7 +170,7 @@ class Entry_section(Entry): data += tools.GetBytes(self._pad_byte, entry.pad_before) # Add in the actual entry data - data += entry.GetData() + data += entry_data # Handle padding after the entry if entry.pad_after: @@ -197,7 +197,7 @@ class Entry_section(Entry): section_data = b'' for entry in self._entries.values(): - data = self.GetPaddedDataForEntry(entry) + data = self.GetPaddedDataForEntry(entry, entry.GetData()) # Handle empty space before the entry pad = (entry.offset or 0) - self._skip_at_start - len(section_data) if pad > 0: @@ -210,7 +210,7 @@ class Entry_section(Entry): (len(self._entries), len(section_data))) return self.CompressData(section_data) - def GetPaddedData(self): + def GetPaddedData(self, data=None): """Get the data for a section including any padding Gets the section data and uses the parent section's pad-byte value to @@ -225,7 +225,9 @@ class Entry_section(Entry): after it (bytes) """ section = self.section or self - return section.GetPaddedDataForEntry(self) + if data is None: + data = self.GetData() + return section.GetPaddedDataForEntry(self, data) def GetData(self): """Get the contents of an entry @@ -264,8 +266,10 @@ class Entry_section(Entry): self._SortEntries() self._ExpandEntries() - size = self.CheckSize() - self.size = size + data = self._BuildSectionData() + self.SetContents(data) + + self.CheckSize() offset = super().Pack(offset) self.CheckEntries() @@ -542,14 +546,12 @@ class Entry_section(Entry): for name, info in offset_dict.items(): self._SetEntryOffsetSize(name, *info) - def CheckSize(self): - data = self._BuildSectionData() - contents_size = len(data) + contents_size = len(self.data) size = self.size if not size: - data = self.GetPaddedData() + data = self.GetPaddedData(self.data) size = len(data) size = tools.Align(size, self.align_size)