[00/15,v2] support/download: extend download features and reproducibility (branch yem/git-attributes)

Hello All!

This series extends the download infra in two ways:
 1. git attributes are emulated
 2- download reproducibility is enhanced

1. git attributes

Git attributes are very similar to the decades-old CVS-style keywords
and placeholders; except they are acted on while generating an archive
with git-archive, rather than on checkout.

Some packages (pcm-tools, luajit) use such markers for versioning
information; pcm-tools needs a (short-)hash, while luajit uses the UNIX
timestamp as its semver patch-level.

This series introduces support for git attributes in the git download
backend, cleans up pcm-tools, converts luajit, and updates all other
packages with the new -br2 git version suffix to filenames of generate
archives.

2. download reproducibility

We are going at great length to try and ensure that the archive we
generate are reproducible, so that we can hash them and validat ethe
hashes on further downloads. We so far had quite a strong reproducility,
except for the mode of the files. This was thought to be OK, because
Buildroot runs under a known umask, and thus files are expected to
always be created with the same modes, whatever the actual umask of the
user.

However, that does not account for ACLs, where two or more users can
share a downlaod directory (e.g. on a shared build server), and set the
ACLs with the 'default' xattr, like so:

    $ setfacl -m default:user::rwx ${BR2_DL_DIR}
    $ setfacl -m default:group::rwx ${BR2_DL_DIR}
    $ setfacl -m default:other::rwx ${BR2_DL_DIR}

This has the side effect that, whatever umaks is in use, all files and
directories created in BR2_DL_DIR will be user, group and, and world
redable and writable, and directories will additioanlly be ugo+x.
Usually, i.e. group and others are not expected to have write
permission.

Thus, when we create an archive in that situation, the modes are not
what is expected, and the hashes differ from the ones generated in a
more traditional setup...

This series addresses that in two ways:
  - first, vendored archive gain a version suffix, like for git and svn
  - second, when creating tarballs, the modes are forced to a sane and
    reproducible value

Changes v1 -> v2;
  - drop applied preparatory patches
  - split the update of hash files and runtime test in separate patches
  - convert luajit
  - add migration section in the manual
  - small coding style in awk script

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (15):
      package/flutter-packages: fix and comment the shared _SOURCE variable
      support/download/git: handle git attributes
      package/*: update hashes for packages with export-subst git attibute
      *: update filename for git-download packages
      package/luajit: drop useless post-extract hook
      package/luajit: use the git download for known-reproducible tarball
      support/testing: add test for export-subst support in git-download
      docs/manual: document git backend handling of export-subst attribute
      package/docker-compose: bump version
      package/pkg-utils: add vendoring mechanism into generated archive filename
      support/download: even more reproducible archives (until next time)
      {boot,package}: update hashes of git, svn, go, and cargo archives
      support/testing: update git tests
      board: update hashes of git archives
      docs/manual: document new archive version suffix

 .checkpackageignore                                |   1 +
 board/radxa/rock5b/patches/linux/linux.hash        |   2 +-
 board/toradex/apalis-imx6/patches/linux/linux.hash |   2 +-
 board/toradex/apalis-imx6/patches/uboot/uboot.hash |   2 +-
 boot/edk2/edk2.hash                                |   2 +-
 boot/vexpress-firmware/vexpress-firmware.hash      |   2 +-
 docs/manual/migrating.adoc                         |  26 +++++++++
 package/abootimg/abootimg.hash                     |   2 +-
 package/aer-inject/aer-inject.hash                 |   2 +-
 package/am33x-cm3/am33x-cm3.hash                   |   2 +-
 package/armbian-firmware/armbian-firmware.hash     |   2 +-
 package/azure-iot-sdk-c/azure-iot-sdk-c.hash       |   2 +-
 package/balena-engine/balena-engine.hash           |   2 +-
 package/bat/bat.hash                               |   2 +-
 package/bayer2rgb-neon/bayer2rgb-neon.hash         |   2 +-
 package/bpftool/bpftool.hash                       |   2 +-
 package/brickd/brickd.hash                         |   2 +-
 package/c-capnproto/c-capnproto.hash               |   2 +-
 package/cni-plugins/cni-plugins.hash               |   2 +-
 package/containerd/containerd.hash                 |   2 +-
 package/crucible/crucible.hash                     |   2 +-
 package/dbus-triggerd/dbus-triggerd.hash           |   2 +-
 package/delve/delve.hash                           |   2 +-
 package/depot-tools/depot-tools.hash               |   2 +-
 package/docker-cli/docker-cli.hash                 |   2 +-
 package/docker-compose/docker-compose.hash         |   2 +-
 package/docker-compose/docker-compose.mk           |   2 +-
 package/docker-engine/docker-engine.hash           |   2 +-
 package/dtv-scan-tables/dtv-scan-tables.hash       |   2 +-
 package/dust/dust.hash                             |   2 +-
 package/edid-decode/edid-decode.hash               |   2 +-
 package/embiggen-disk/embiggen-disk.hash           |   2 +-
 package/eza/eza.hash                               |   2 +-
 package/firmware-utils/firmware-utils.hash         |   2 +-
 package/flannel/flannel.hash                       |   2 +-
 package/flashbench/flashbench.hash                 |   2 +-
 package/flutter-packages/flutter-packages.hash     |   2 +-
 package/flutter-packages/flutter-packages.mk       |   8 ++-
 package/flutter-pi/flutter-pi.hash                 |   2 +-
 package/freescale-imx/imx-lib/imx-lib.hash         |   2 +-
 package/gitlab-runner/gitlab-runner.hash           |   2 +-
 package/gocryptfs/gocryptfs.hash                   |   2 +-
 package/google-breakpad/google-breakpad.hash       |   2 +-
 .../gstreamer1/gst1-interpipe/gst1-interpipe.hash  |   2 +-
 .../gst1-plugins-bayer2rgb-neon.hash               |   2 +-
 package/gstreamer1/gst1-shark/gst1-shark.hash      |   3 +-
 package/hyperfine/hyperfine.hash                   |   2 +-
 package/ibm-sw-tpm2/ibm-sw-tpm2.hash               |   2 +-
 package/ivi-homescreen/ivi-homescreen.hash         |   2 +-
 package/kvmtool/kvmtool.hash                       |   2 +-
 package/libbroadvoice/libbroadvoice.hash           |   2 +-
 package/libcamera/libcamera.hash                   |   2 +-
 package/libdbi-drivers/libdbi-drivers.hash         |   2 +-
 package/libdbi/libdbi.hash                         |   2 +-
 package/libg7221/libg7221.hash                     |   2 +-
 package/libilbc/libilbc.hash                       |   2 +-
 package/libsilk/libsilk.hash                       |   2 +-
 package/libsvgtiny/libsvgtiny.hash                 |   2 +-
 package/libubox/libubox.hash                       |   2 +-
 package/libuci/libuci.hash                         |   2 +-
 package/libxmlrpc/libxmlrpc.hash                   |   2 +-
 package/libyuv/libyuv.hash                         |   2 +-
 .../linux-syscall-support.hash                     |   2 +-
 package/luajit/luajit.hash                         |   2 +-
 package/luajit/luajit.mk                           |  12 +----
 package/mender-artifact/mender-artifact.hash       |   2 +-
 package/mender-connect/mender-connect.hash         |   2 +-
 package/mender/mender.hash                         |   2 +-
 package/mmc-utils/mmc-utils.hash                   |   2 +-
 package/moby-buildkit/moby-buildkit.hash           |   2 +-
 package/mxsldr/mxsldr.hash                         |   2 +-
 package/nerdctl/nerdctl.hash                       |   2 +-
 .../netsurf-buildsystem/netsurf-buildsystem.hash   |   2 +-
 package/nushell/nushell.hash                       |   2 +-
 package/odhcp6c/odhcp6c.hash                       |   2 +-
 package/ogre/ogre.hash                             |   2 +-
 package/open62541/open62541.hash                   |   2 +-
 package/opkg-utils/opkg-utils.hash                 |   2 +-
 package/pcm-tools/pcm-tools.hash                   |   2 +-
 package/pcm-tools/pcm-tools.mk                     |   8 ---
 package/piglit/piglit.hash                         |   2 +-
 package/pkg-download.mk                            |   8 +--
 package/pkg-utils.mk                               |   2 +-
 package/prelink-cross/prelink-cross.hash           |   2 +-
 package/procs/procs.hash                           |   2 +-
 package/psplash/psplash.hash                       |   2 +-
 package/qt-webkit-kiosk/qt-webkit-kiosk.hash       |   2 +-
 package/qt5/qt5coap/qt5coap.hash                   |   2 +-
 package/qt5/qt5knx/qt5knx.hash                     |   2 +-
 package/qt5/qt5location/qt5location.hash           |   2 +-
 package/qt5/qt5mqtt/qt5mqtt.hash                   |   2 +-
 package/qt5/qt5opcua/qt5opcua.hash                 |   2 +-
 .../qt5webengine-chromium-catapult.hash            |   2 +-
 package/ripgrep/ripgrep.hash                       |   2 +-
 package/rockchip-rkbin/rockchip-rkbin.hash         |   2 +-
 package/rtc-tools/rtc-tools.hash                   |   2 +-
 package/rtmpdump/rtmpdump.hash                     |   2 +-
 package/runc/runc.hash                             |   2 +-
 package/rust-bindgen/rust-bindgen.hash             |   2 +-
 package/sentry-cli/sentry-cli.hash                 |   2 +-
 package/signal-estimator/signal-estimator.hash     |   2 +-
 package/sox/sox.hash                               |   2 +-
 package/tealdeer/tealdeer.hash                     |   2 +-
 package/tftpd/tftpd.hash                           |   2 +-
 package/ti-sgx-demos/ti-sgx-demos.hash             |   2 +-
 package/ti-sgx-km/ti-sgx-km.hash                   |   2 +-
 package/ti-sgx-um/ti-sgx-um.hash                   |   2 +-
 package/tinifier/tinifier.hash                     |   2 +-
 package/tl-expected/tl-expected.hash               |   2 +-
 package/tremor/tremor.hash                         |   2 +-
 package/ubus/ubus.hash                             |   2 +-
 package/uclibc-ng-test/uclibc-ng-test.hash         |   2 +-
 package/uemacs/uemacs.hash                         |   2 +-
 package/uhttpd/uhttpd.hash                         |   2 +-
 package/uqmi/uqmi.hash                             |   2 +-
 package/ustream-ssl/ustream-ssl.hash               |   2 +-
 package/vboot-utils/vboot-utils.hash               |   2 +-
 package/wilink-bt-firmware/wilink-bt-firmware.hash |   2 +-
 package/wtfutil/wtfutil.hash                       |   2 +-
 .../xdriver_xf86-video-intel.hash                  |   2 +-
 .../xdriver_xf86-video-openchrome.hash             |   2 +-
 package/x264/x264.hash                             |   2 +-
 package/yavta/yavta.hash                           |   2 +-
 support/download/git                               |  60 +++++++++++++++++++++
 support/download/helpers                           |   2 +-
 .../br2-external/git-hash/package/bad/bad.hash     |   2 +-
 .../package/export-subst/export-subst.hash         |   1 +
 .../git-hash/package/export-subst/export-subst.mk  |  10 ++++
 .../br2-external/git-hash/package/good/good.hash   |   2 +-
 .../git-partial-sha1-branch-head.hash              |   2 +-
 .../git-partial-sha1-reachable-by-branch.hash      |   2 +-
 .../git-partial-sha1-reachable-by-tag.hash         |   2 +-
 .../git-partial-sha1-tag-itself.hash               |   2 +-
 .../git-partial-sha1-tag-points-to.hash            |   2 +-
 .../git-sha1-branch-head/git-sha1-branch-head.hash |   2 +-
 .../git-sha1-reachable-by-branch.hash              |   2 +-
 .../git-sha1-reachable-by-tag.hash                 |   2 +-
 .../git-sha1-tag-itself/git-sha1-tag-itself.hash   |   2 +-
 .../git-sha1-tag-points-to.hash                    |   2 +-
 .../git-submodule-disabled.hash                    |   2 +-
 .../git-submodule-enabled.hash                     |   2 +-
 .../git-refs/package/git-tag/git-tag.hash          |   2 +-
 .../git-wrong-content/git-wrong-content.hash       |   2 +-
 .../05/482df734b3715b849ef4a3147a9b1b1f8cca38      | Bin 0 -> 35 bytes
 .../0f/db95cf4f3c5ed4003287649cabb33c5f843e26      | Bin 0 -> 197 bytes
 .../68/28f88dcb0e88b8cd738ad6044ce74d7a9a13c8      | Bin 0 -> 28 bytes
 .../6d/a12b257e47f6089612fe97a8746d2d9c4ca0e0      | Bin 0 -> 112 bytes
 .../download/git-remote/repo.git/refs/heads/master |   2 +-
 support/testing/tests/download/test_git.py         |   4 ++
 149 files changed, 251 insertions(+), 158 deletions(-)
 create mode 100644 support/testing/tests/download/br2-external/git-hash/package/export-subst/export-subst.hash
 create mode 100644 support/testing/tests/download/br2-external/git-hash/package/export-subst/export-subst.mk
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/05/482df734b3715b849ef4a3147a9b1b1f8cca38
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/0f/db95cf4f3c5ed4003287649cabb33c5f843e26
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/68/28f88dcb0e88b8cd738ad6044ce74d7a9a13c8
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/6d/a12b257e47f6089612fe97a8746d2d9c4ca0e0

--
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Hi Yann,all,

On 27/04/2024 20:38, Yann E. MORIN wrote:
> Hello All!
> 
> This series extends the download infra in two ways:
>   1. git attributes are emulated
>   2- download reproducibility is enhanced

  As discussed privately on IRC: while applying this series, I suddenly 
remembered that we also wanted to update the tar version used by Buildroot to 
something higher than 1.34. We're currently stuck there because of changes in 
the generated tar format, which also changes the tarball hashes, which would 
imply a huge refresh of the hash files that use VCS or vendored sources.

  This series, however, already does such a refresh of all the hash files, so 
it's the ideal occasion to also update tar.

  Therefore, I haven't applied to master. Instead, I've pushed my modifications 
to branch yem/git-attributes on https://gitlab.com/arnout/buildroot and you can 
pick it up from there. And I've marked the series as Changes Requested.

  Regards,
  Arnout

[snip]