diff mbox series

Issue after changing from xz to zstd

Message ID 9216a6e7-3ef4-4b65-8997-84f3210f4795@gmail.com
State Not Applicable
Headers show
Series Issue after changing from xz to zstd | expand

Commit Message

Hartmut Birr April 6, 2024, 7:46 p.m. UTC
Hi,

I did change the Makefile for dnsmasq in the way that I give:

PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2024-03-27
PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2	# Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c

Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.

Any idea why VERSION is manipulated?

I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.

dnsmasq/patches/999-dnsmasq-version.patch:

Regards,
Hartmut

Comments

Robert Marko April 6, 2024, 7:50 p.m. UTC | #1
On Sat, 6 Apr 2024 at 21:47, Hartmut Birr <osexpert@gmail.com> wrote:
>
> Hi,
>
> I did change the Makefile for dnsmasq in the way that I give:
>
> PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
> PKG_SOURCE_PROTO:=git
> PKG_SOURCE_DATE:=2024-03-27
> PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2    # Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
> PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c
>
> Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.
>
> Any idea why VERSION is manipulated?
>
> I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.
>
> dnsmasq/patches/999-dnsmasq-version.patch:
> diff --git a/VERSION b/VERSION
> index 998eb1f..9977908 100644
> --- a/VERSION
> +++ b/VERSION
> @@ -1 +1 @@
> -$Format:%d$
> + (master, v2.90deb2-8-g550c368, head)
>

Hi Hartmut,
We are now using git-archive to make a tarball of git repositories in
order to make them reproducible which fixes a long standing issue.
git-archive respects .gitattributes so that along with:
https://github.com/imp/dnsmasq/commit/2aaea18f432374fd062370f2d899e47849558b2f

Is most likely the cause.

Regards,
Robert

> Regards,
> Hartmut
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
e9hack April 6, 2024, 8:35 p.m. UTC | #2
Am 06.04.2024 um 21:50 schrieb Robert Marko:
> On Sat, 6 Apr 2024 at 21:47, Hartmut Birr <osexpert@gmail.com> wrote:
>>
>> Hi,
>>
>> I did change the Makefile for dnsmasq in the way that I give:
>>
>> PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
>> PKG_SOURCE_PROTO:=git
>> PKG_SOURCE_DATE:=2024-03-27
>> PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2    # Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
>> PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c
>>
>> Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.
>>
>> Any idea why VERSION is manipulated?
>>
>> I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.
>>
>> dnsmasq/patches/999-dnsmasq-version.patch:
>> diff --git a/VERSION b/VERSION
>> index 998eb1f..9977908 100644
>> --- a/VERSION
>> +++ b/VERSION
>> @@ -1 +1 @@
>> -$Format:%d$
>> + (master, v2.90deb2-8-g550c368, head)
>>
> 
> Hi Hartmut,
> We are now using git-archive to make a tarball of git repositories in
> order to make them reproducible which fixes a long standing issue.
> git-archive respects .gitattributes so that along with:
> https://github.com/imp/dnsmasq/commit/2aaea18f432374fd062370f2d899e47849558b2f
> 
> Is most likely the cause.

I did add another static patch to solve the issue for me temporary:

--- a/VERSION	2024-03-27 17:00:06.000000000 +0100
+++ b/VERSION	2024-03-27 17:00:06.000000000 +0100
@@ -1 +1 @@
- (HEAD, origin/master, origin/HEAD, master)
+$Format:%d$

I generate 999-dnsmasq-version.patch via the following extension in the Makefile:

define Build/Prepare
	mkdir -p $(TOPDIR)/tmp/dl && \
		cd $(TOPDIR)/tmp/dl && \
		rm -rf $(PKG_SOURCE_SUBDIR) && \
		git clone -4 $(PKG_SOURCE_URL) $(PKG_SOURCE_SUBDIR) --recursive && \
		cd $(PKG_SOURCE_SUBDIR) && \
		git checkout $(PKG_SOURCE_VERSION) && \
		version=`git describe` && \
		echo " (master, $$$$version, head)" > VERSION && \
		git diff VERSION > 999-dnsmasq-version.patch
	$(CP) $(TOPDIR)/tmp/dl/$(PKG_SOURCE_SUBDIR)/999-dnsmasq-version.patch ./patches
	touch -c -r ./Makefile ./patches/999-dnsmasq-version.patch
	cd $(TOPDIR)/tmp/dl && \
		rm -rf $(PKG_SOURCE_SUBDIR)
	$(call Build/Prepare/Default)
endef

Regards,
Hartmut

> 
> Regards,
> Robert
> 
>> Regards,
>> Hartmut
>>
>> _______________________________________________
>> openwrt-devel mailing list
>> openwrt-devel@lists.openwrt.org
>> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Robert Marko April 6, 2024, 9:32 p.m. UTC | #3
On Sat, 6 Apr 2024 at 22:35, e9hack <e9hack@gmail.com> wrote:
>
> Am 06.04.2024 um 21:50 schrieb Robert Marko:
> > On Sat, 6 Apr 2024 at 21:47, Hartmut Birr <osexpert@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I did change the Makefile for dnsmasq in the way that I give:
> >>
> >> PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
> >> PKG_SOURCE_PROTO:=git
> >> PKG_SOURCE_DATE:=2024-03-27
> >> PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2    # Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
> >> PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c
> >>
> >> Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.
> >>
> >> Any idea why VERSION is manipulated?
> >>
> >> I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.
> >>
> >> dnsmasq/patches/999-dnsmasq-version.patch:
> >> diff --git a/VERSION b/VERSION
> >> index 998eb1f..9977908 100644
> >> --- a/VERSION
> >> +++ b/VERSION
> >> @@ -1 +1 @@
> >> -$Format:%d$
> >> + (master, v2.90deb2-8-g550c368, head)
> >>
> >
> > Hi Hartmut,
> > We are now using git-archive to make a tarball of git repositories in
> > order to make them reproducible which fixes a long standing issue.
> > git-archive respects .gitattributes so that along with:
> > https://github.com/imp/dnsmasq/commit/2aaea18f432374fd062370f2d899e47849558b2f
> >
> > Is most likely the cause.
>
> I did add another static patch to solve the issue for me temporary:
>
> --- a/VERSION   2024-03-27 17:00:06.000000000 +0100
> +++ b/VERSION   2024-03-27 17:00:06.000000000 +0100
> @@ -1 +1 @@
> - (HEAD, origin/master, origin/HEAD, master)
> +$Format:%d$
>
> I generate 999-dnsmasq-version.patch via the following extension in the Makefile:
>
> define Build/Prepare
>         mkdir -p $(TOPDIR)/tmp/dl && \
>                 cd $(TOPDIR)/tmp/dl && \
>                 rm -rf $(PKG_SOURCE_SUBDIR) && \
>                 git clone -4 $(PKG_SOURCE_URL) $(PKG_SOURCE_SUBDIR) --recursive && \
>                 cd $(PKG_SOURCE_SUBDIR) && \
>                 git checkout $(PKG_SOURCE_VERSION) && \
>                 version=`git describe` && \
>                 echo " (master, $$$$version, head)" > VERSION && \
>                 git diff VERSION > 999-dnsmasq-version.patch
>         $(CP) $(TOPDIR)/tmp/dl/$(PKG_SOURCE_SUBDIR)/999-dnsmasq-version.patch ./patches
>         touch -c -r ./Makefile ./patches/999-dnsmasq-version.patch
>         cd $(TOPDIR)/tmp/dl && \
>                 rm -rf $(PKG_SOURCE_SUBDIR)
>         $(call Build/Prepare/Default)
> endef

Hi, can I ask what is the point though?
That VERSION file contains a placeholder format specifier that is
supposed to be filled with a version
by git, it was most likely meant to be used with git-archive as
.gitattributes specifies that VERSION should
be updated.

$Format:%d$ is just the format specifier placeholder to be replaced.

Regards,
Robert
>
> Regards,
> Hartmut
>
> >
> > Regards,
> > Robert
> >
> >> Regards,
> >> Hartmut
> >>
> >> _______________________________________________
> >> openwrt-devel mailing list
> >> openwrt-devel@lists.openwrt.org
> >> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
>
Robert Marko April 6, 2024, 9:42 p.m. UTC | #4
On Sat, 6 Apr 2024 at 23:32, Robert Marko <robimarko@gmail.com> wrote:
>
> On Sat, 6 Apr 2024 at 22:35, e9hack <e9hack@gmail.com> wrote:
> >
> > Am 06.04.2024 um 21:50 schrieb Robert Marko:
> > > On Sat, 6 Apr 2024 at 21:47, Hartmut Birr <osexpert@gmail.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I did change the Makefile for dnsmasq in the way that I give:
> > >>
> > >> PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
> > >> PKG_SOURCE_PROTO:=git
> > >> PKG_SOURCE_DATE:=2024-03-27
> > >> PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2    # Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
> > >> PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c
> > >>
> > >> Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.
> > >>
> > >> Any idea why VERSION is manipulated?
> > >>
> > >> I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.
> > >>
> > >> dnsmasq/patches/999-dnsmasq-version.patch:
> > >> diff --git a/VERSION b/VERSION
> > >> index 998eb1f..9977908 100644
> > >> --- a/VERSION
> > >> +++ b/VERSION
> > >> @@ -1 +1 @@
> > >> -$Format:%d$
> > >> + (master, v2.90deb2-8-g550c368, head)
> > >>
> > >
> > > Hi Hartmut,
> > > We are now using git-archive to make a tarball of git repositories in
> > > order to make them reproducible which fixes a long standing issue.
> > > git-archive respects .gitattributes so that along with:
> > > https://github.com/imp/dnsmasq/commit/2aaea18f432374fd062370f2d899e47849558b2f
> > >
> > > Is most likely the cause.
> >
> > I did add another static patch to solve the issue for me temporary:
> >
> > --- a/VERSION   2024-03-27 17:00:06.000000000 +0100
> > +++ b/VERSION   2024-03-27 17:00:06.000000000 +0100
> > @@ -1 +1 @@
> > - (HEAD, origin/master, origin/HEAD, master)
> > +$Format:%d$
> >
> > I generate 999-dnsmasq-version.patch via the following extension in the Makefile:
> >
> > define Build/Prepare
> >         mkdir -p $(TOPDIR)/tmp/dl && \
> >                 cd $(TOPDIR)/tmp/dl && \
> >                 rm -rf $(PKG_SOURCE_SUBDIR) && \
> >                 git clone -4 $(PKG_SOURCE_URL) $(PKG_SOURCE_SUBDIR) --recursive && \
> >                 cd $(PKG_SOURCE_SUBDIR) && \
> >                 git checkout $(PKG_SOURCE_VERSION) && \
> >                 version=`git describe` && \
> >                 echo " (master, $$$$version, head)" > VERSION && \
> >                 git diff VERSION > 999-dnsmasq-version.patch
> >         $(CP) $(TOPDIR)/tmp/dl/$(PKG_SOURCE_SUBDIR)/999-dnsmasq-version.patch ./patches
> >         touch -c -r ./Makefile ./patches/999-dnsmasq-version.patch
> >         cd $(TOPDIR)/tmp/dl && \
> >                 rm -rf $(PKG_SOURCE_SUBDIR)
> >         $(call Build/Prepare/Default)
> > endef
>
> Hi, can I ask what is the point though?
> That VERSION file contains a placeholder format specifier that is
> supposed to be filled with a version
> by git, it was most likely meant to be used with git-archive as
> .gitattributes specifies that VERSION should
> be updated.
>
> $Format:%d$ is just the format specifier placeholder to be replaced.

Hm, I might have understood your previous statement wrong because when
checking dnsmasq 2.90 tarball
on OpenWrt sources VERSION is clearly properly populated so
git-archive is most likely the culprit by
not populating it with the commit, tag etc but by just the remote and
branch names.

Regards,
Robert

>
> Regards,
> Robert
> >
> > Regards,
> > Hartmut
> >
> > >
> > > Regards,
> > > Robert
> > >
> > >> Regards,
> > >> Hartmut
> > >>
> > >> _______________________________________________
> > >> openwrt-devel mailing list
> > >> openwrt-devel@lists.openwrt.org
> > >> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
> >
Christian Marangi April 6, 2024, 10:56 p.m. UTC | #5
Il giorno sab 6 apr 2024 alle ore 21:46 Hartmut Birr
<osexpert@gmail.com> ha scritto:
>
> Hi,
>
> I did change the Makefile for dnsmasq in the way that I give:
>
> PKG_SOURCE_URL:=git://thekelleys.org.uk/dnsmasq.git
> PKG_SOURCE_PROTO:=git
> PKG_SOURCE_DATE:=2024-03-27
> PKG_SOURCE_VERSION:=550c368adea12b312f83686c61f9015c122046c2    # Treat cache insertion failure of DNSKEY and DS records as another resource problem and fail validation with suitable logging.
> PKG_MIRROR_HASH:=284a34bdb967ec8a9dff132df065ca64e9a1819d79bb8cecee1af001e22d626c
>
> Before changing to zstd, the generated source tar ball contains a file 'VERSION' with content '$Format:%d$'. This does match the dnsmasq git repository. After changing to zstd, VERSION contains ' (HEAD, origin/master, origin/HEAD, master)'.
>
> Any idea why VERSION is manipulated?
>
> I generate automatically a patch which modifies VERSION to see the commit hash via logread. Applying the patch doesn't work any more.
>
> dnsmasq/patches/999-dnsmasq-version.patch:
> diff --git a/VERSION b/VERSION
> index 998eb1f..9977908 100644
> --- a/VERSION
> +++ b/VERSION
> @@ -1 +1 @@
> -$Format:%d$
> + (master, v2.90deb2-8-g550c368, head)
>

Hi,
thanks for pointing this out, although your situation is special and a
downstream
situation it was worth investigating it.

We confirmed that the change is correct and this is expected and it was
actually wrong previously.

The format works and requires an associated tag to the commit. Passing
an untagged hash results in what you are observing.

This is git archive handling and nothing done wrong on our hand. Soo your
solution was working due to us doing things in a special way (and ignoring
.gitattributes)

A more robust solution (and what I'm suggesting you to change), is to
make use of the Prepare define and just remove the VERSION file
and generate one of your own. It should permit you to make things
more maintainable and even remove the downstream patch.
diff mbox series

Patch

diff --git a/VERSION b/VERSION
index 998eb1f..9977908 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@ 
-$Format:%d$
+ (master, v2.90deb2-8-g550c368, head)