[OpenWrt-Devel] include/image.mk: make tar reproducible

several packages reported different file order between builds
make binutils, kmod-sched reproducible

Signed-off-by: Dirk Neukirchen <dirkneukirchen@web.de>
---
 include/image.mk   | 2 +-
 scripts/ipkg-build | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

Hi,

On Sun, Aug 30, 2015 at 6:26 PM, Dirk Neukirchen <dirkneukirchen@web.de> wrote:
> several packages reported different file order between builds
> make binutils, kmod-sched reproducible
>
> Signed-off-by: Dirk Neukirchen <dirkneukirchen@web.de>

These are stil not reproducible for others, because ...
> ---
>  include/image.mk   | 2 +-
>  scripts/ipkg-build | 6 +++---
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/include/image.mk b/include/image.mk
> index 73fc805..f8601f0 100644
> --- a/include/image.mk
> +++ b/include/image.mk
> @@ -245,7 +245,7 @@ define Image/mkfs/cpiogz
>  endef
>
>  define Image/mkfs/targz
> -       $(TAR) -czpf $(BIN_DIR)/$(IMG_PREFIX)$(if $(PROFILE),-$(PROFILE))-rootfs.tar.gz --numeric-owner --owner=0 --group=0 -C $(TARGET_DIR)/ .
> +       $(TAR) -czpf $(BIN_DIR)/$(IMG_PREFIX)$(if $(PROFILE),-$(PROFILE))-rootfs.tar.gz --numeric-owner --owner=0 --group=0 --sort=name -C $(TARGET_DIR)/ .
>  endef
>
>  E2SIZE=$(shell echo $$(($(CONFIG_TARGET_ROOTFS_PARTSIZE)*1024*1024)))
> diff --git a/scripts/ipkg-build b/scripts/ipkg-build
> index cd70c1f..a54e9b5 100755
> --- a/scripts/ipkg-build
> +++ b/scripts/ipkg-build
> @@ -139,20 +139,20 @@ mkdir $tmp_dir
>
>  echo $CONTROL > $tmp_dir/tarX
>  # Preserve permissions (-p) when creating data.tar.gz as non-root user
> -( cd $pkg_dir && $TAR $ogargs -X $tmp_dir/tarX --format=gnu -czpf $tmp_dir/data.tar.gz --mtime="$TIMESTAMP" . )
> +( cd $pkg_dir && $TAR $ogargs -X $tmp_dir/tarX --format=gnu --sort=name -czpf $tmp_dir/data.tar.gz --mtime="$TIMESTAMP" . )
>
>  installed_size=`stat -c "%s" $tmp_dir/data.tar.gz`
>  sed -i -e "s/^Installed-Size: .*/Installed-Size: $installed_size/" \
>         $pkg_dir/$CONTROL/control
>
> -( cd $pkg_dir/$CONTROL && $TAR $ogargs --format=gnu -czf $tmp_dir/control.tar.gz --mtime="$TIMESTAMP" . )
> +( cd $pkg_dir/$CONTROL && $TAR $ogargs --format=gnu --sort=name -czf $tmp_dir/control.tar.gz --mtime="$TIMESTAMP" . )
>  rm $tmp_dir/tarX
>
>  echo "2.0" > $tmp_dir/debian-binary
>
>  pkg_file=$dest_dir/${pkg}_${version}_${arch}.ipk
>  rm -f $pkg_file
> -( cd $tmp_dir && $TAR --format=gnu -zcf $pkg_file --mtime="$TIMESTAMP" ./debian-binary ./data.tar.gz ./control.tar.gz )
> +( cd $tmp_dir && $TAR --format=gnu --sort=name -zcf $pkg_file --mtime="$TIMESTAMP" ./debian-binary ./data.tar.gz ./control.tar.gz )

this $TAR invocation does not use $ogargs, so the outer .ipk will use
the current user as the owner for its contents. I don't think this
omission is intentional, but I haven't tested if anything breaks when
adding it.


>
>  rm $tmp_dir/debian-binary $tmp_dir/data.tar.gz $tmp_dir/control.tar.gz
>  rmdir $tmp_dir


Jonas

On 30.08.2015 19:42, Jonas Gorski wrote:
> Hi,
> 
> On Sun, Aug 30, 2015 at 6:26 PM, Dirk Neukirchen <dirkneukirchen@web.de> wrote:
>> several packages reported different file order between builds
>> make binutils, kmod-sched reproducible
>>
>> Signed-off-by: Dirk Neukirchen <dirkneukirchen@web.de>
> 
> These are stil not reproducible for others, because ...
>> ---
>>  include/image.mk   | 2 +-
>>  scripts/ipkg-build | 6 +++---
>>  2 files changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/image.mk b/include/image.mk
>> index 73fc805..f8601f0 100644
>> --- a/include/image.mk
>> +++ b/include/image.mk
>> @@ -245,7 +245,7 @@ define Image/mkfs/cpiogz
>>  endef
>>
>>  define Image/mkfs/targz
>> -       $(TAR) -czpf $(BIN_DIR)/$(IMG_PREFIX)$(if $(PROFILE),-$(PROFILE))-rootfs.tar.gz --numeric-owner --owner=0 --group=0 -C $(TARGET_DIR)/ .
>> +       $(TAR) -czpf $(BIN_DIR)/$(IMG_PREFIX)$(if $(PROFILE),-$(PROFILE))-rootfs.tar.gz --numeric-owner --owner=0 --group=0 --sort=name -C $(TARGET_DIR)/ .
>>  endef
>>
>>  E2SIZE=$(shell echo $$(($(CONFIG_TARGET_ROOTFS_PARTSIZE)*1024*1024)))
>> diff --git a/scripts/ipkg-build b/scripts/ipkg-build
>> index cd70c1f..a54e9b5 100755
>> --- a/scripts/ipkg-build
>> +++ b/scripts/ipkg-build
>> @@ -139,20 +139,20 @@ mkdir $tmp_dir
>>
>>  echo $CONTROL > $tmp_dir/tarX
>>  # Preserve permissions (-p) when creating data.tar.gz as non-root user
>> -( cd $pkg_dir && $TAR $ogargs -X $tmp_dir/tarX --format=gnu -czpf $tmp_dir/data.tar.gz --mtime="$TIMESTAMP" . )
>> +( cd $pkg_dir && $TAR $ogargs -X $tmp_dir/tarX --format=gnu --sort=name -czpf $tmp_dir/data.tar.gz --mtime="$TIMESTAMP" . )
>>
>>  installed_size=`stat -c "%s" $tmp_dir/data.tar.gz`
>>  sed -i -e "s/^Installed-Size: .*/Installed-Size: $installed_size/" \
>>         $pkg_dir/$CONTROL/control
>>
>> -( cd $pkg_dir/$CONTROL && $TAR $ogargs --format=gnu -czf $tmp_dir/control.tar.gz --mtime="$TIMESTAMP" . )
>> +( cd $pkg_dir/$CONTROL && $TAR $ogargs --format=gnu --sort=name -czf $tmp_dir/control.tar.gz --mtime="$TIMESTAMP" . )
>>  rm $tmp_dir/tarX
>>
>>  echo "2.0" > $tmp_dir/debian-binary
>>
>>  pkg_file=$dest_dir/${pkg}_${version}_${arch}.ipk
>>  rm -f $pkg_file
>> -( cd $tmp_dir && $TAR --format=gnu -zcf $pkg_file --mtime="$TIMESTAMP" ./debian-binary ./data.tar.gz ./control.tar.gz )
>> +( cd $tmp_dir && $TAR --format=gnu --sort=name -zcf $pkg_file --mtime="$TIMESTAMP" ./debian-binary ./data.tar.gz ./control.tar.gz )
> 
> this $TAR invocation does not use $ogargs, so the outer .ipk will use
> the current user as the owner for its contents. I don't think this
> omission is intentional, but I haven't tested if anything breaks when
> adding it.
> 
> 

Adding $ogargs does not seem to break anything - it currently works
with extracting name:ID thepeople/1008 on the device.

Currently OpenWrt reproducible page does not change more between
different builds. 
This is documented (uid is not yet varied between rebuilds of OpenWrt.)

So this still fixes the file ordering for inner and outer tar.
(https://reproducible.debian.net/openwrt/dbd/ramips/base/binutils_2.24-3_ramips.ipk.html)

I sent a separate patch on top of this one to add $oargs.

>>
>>  rm $tmp_dir/debian-binary $tmp_dir/data.tar.gz $tmp_dir/control.tar.gz
>>  rmdir $tmp_dir
> 
> 
> Jonas
>