diff mbox

[03/34] reproducibility: generate SOURCE_DATE_EPOCH

Message ID 1462002570-14706-3-git-send-email-gilles.chanteperdrix@xenomai.org
State Changes Requested
Headers show

Commit Message

Gilles Chanteperdrix April 30, 2016, 7:48 a.m. UTC 
When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
environment variable which contains either the date of buildroot last
commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been change, even though some of them
may not have changed in fact.
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

Comments

Thomas Petazzoni May 7, 2016, 1:20 p.m. UTC | #1 
Hello,

On Sat, 30 Apr 2016 09:48:59 +0200, Gilles Chanteperdrix wrote:
> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH

wanted -> requested

generates -> generate

> environment variable which contains either the date of buildroot last
> commit if running from a git repository, or the latest release date.
> 
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
> may not have changed in fact.

Missing SoB.

> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)

Yirk. I'm wondering if we shouldn't simply have a BR2_RELEASE_DATE
variable next to BR2_VERSION in the top level Makefile that gives the
release date.

Thomas
Arnout Vandecappelle May 7, 2016, 9 p.m. UTC | #2 
On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> environment variable which contains either the date of buildroot last
> commit if running from a git repository, or the latest release date.
>
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
> may not have changed in fact.
> ---
>  Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 86b2ed2..c1d2961 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ=UTC
>  export LANG=C
>  export LC_ALL=C
> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))

  I really don't like exporting a variable that is not well-known and that 
doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in 
Buildroot-specific patches, so we can just as well add the BR_ prefix.

  But actually, we don't want to introduce buildroot-specific (and therefore 
unupstreamable) things in patches either. So there is not much point in 
exporting, except if it's used by scripts.


  Regards,
  Arnout

>  endif
>
>  # To put more focus on warnings, be less verbose as default
>
Gilles Chanteperdrix May 8, 2016, 8:22 p.m. UTC | #3 
On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> > When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> > environment variable which contains either the date of buildroot last
> > commit if running from a git repository, or the latest release date.
> >
> > This means that all packages embedding build dates will appear to
> > have the same build date, so in case of new commit or release, all
> > packages will appear to have been change, even though some of them
> > may not have changed in fact.
> > ---
> >  Makefile | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/Makefile b/Makefile
> > index 86b2ed2..c1d2961 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
> >  export TZ=UTC
> >  export LANG=C
> >  export LC_ALL=C
> > +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> > +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> > +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> > +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> > +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> 
>   I really don't like exporting a variable that is not well-known and that 
> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in 
> Buildroot-specific patches, so we can just as well add the BR_ prefix.

SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
specification:
https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal

> 
>   But actually, we don't want to introduce buildroot-specific (and therefore 
> unupstreamable) things in patches either. So there is not much point in 
> exporting, except if it's used by scripts.

I used it in some patched configure scripts (libnspr, if I remember
correctly). Which is why I exported it.
Arnout Vandecappelle May 9, 2016, 11:28 p.m. UTC | #4 
On 05/08/16 22:22, Gilles Chanteperdrix wrote:
> On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
>> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
>>> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
>>> environment variable which contains either the date of buildroot last
>>> commit if running from a git repository, or the latest release date.
>>>
>>> This means that all packages embedding build dates will appear to
>>> have the same build date, so in case of new commit or release, all
>>> packages will appear to have been change, even though some of them
>>> may not have changed in fact.
>>> ---
>>>  Makefile | 5 +++++
>>>  1 file changed, 5 insertions(+)
>>>
>>> diff --git a/Makefile b/Makefile
>>> index 86b2ed2..c1d2961 100644
>>> --- a/Makefile
>>> +++ b/Makefile
>>> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>>>  export TZ=UTC
>>>  export LANG=C
>>>  export LC_ALL=C
>>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>>> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
>>> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
>>> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
>>> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
>>
>>   I really don't like exporting a variable that is not well-known and that
>> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in
>> Buildroot-specific patches, so we can just as well add the BR_ prefix.
>
> SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
> specification:
> https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal

  Ah, good.

  Actually, a better reference is
https://reproducible-builds.org/specs/source-date-epoch/
(actually the same developers wrote this spec, but it's supposedly not 
debian-specific).

  When you repost, could you make sure that the commit message mentions this 
explicitly?

>
>>
>>   But actually, we don't want to introduce buildroot-specific (and therefore
>> unupstreamable) things in patches either. So there is not much point in
>> exporting, except if it's used by scripts.
>
> I used it in some patched configure scripts (libnspr, if I remember
> correctly). Which is why I exported it.

  As I said: used in patches.

  But indeed, it is well-known. So OK to export it.

  BTW, when it is possible to download a debian patch for a package (by setting 
FOO_PATCH = ...) rather than rolling our own, we prefer that.

  Regards,
  Arnout
Gilles Chanteperdrix May 11, 2016, 7:17 a.m. UTC | #5 
On Tue, May 10, 2016 at 01:28:14AM +0200, Arnout Vandecappelle wrote:
> On 05/08/16 22:22, Gilles Chanteperdrix wrote:
> > On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
> >> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> >>> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> >>> environment variable which contains either the date of buildroot last
> >>> commit if running from a git repository, or the latest release date.
> >>>
> >>> This means that all packages embedding build dates will appear to
> >>> have the same build date, so in case of new commit or release, all
> >>> packages will appear to have been change, even though some of them
> >>> may not have changed in fact.
> >>> ---
> >>>  Makefile | 5 +++++
> >>>  1 file changed, 5 insertions(+)
> >>>
> >>> diff --git a/Makefile b/Makefile
> >>> index 86b2ed2..c1d2961 100644
> >>> --- a/Makefile
> >>> +++ b/Makefile
> >>> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
> >>>  export TZ=UTC
> >>>  export LANG=C
> >>>  export LC_ALL=C
> >>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> >>> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> >>> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> >>> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> >>> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> >>
> >>   I really don't like exporting a variable that is not well-known and that
> >> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in
> >> Buildroot-specific patches, so we can just as well add the BR_ prefix.
> >
> > SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
> > specification:
> > https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal
> 
>   Ah, good.
> 
>   Actually, a better reference is
> https://reproducible-builds.org/specs/source-date-epoch/
> (actually the same developers wrote this spec, but it's supposedly not 
> debian-specific).
> 
>   When you repost, could you make sure that the commit message mentions this 
> explicitly?
> 
> >
> >>
> >>   But actually, we don't want to introduce buildroot-specific (and therefore
> >> unupstreamable) things in patches either. So there is not much point in
> >> exporting, except if it's used by scripts.
> >
> > I used it in some patched configure scripts (libnspr, if I remember
> > correctly). Which is why I exported it.
> 
>   As I said: used in patches.
> 
>   But indeed, it is well-known. So OK to export it.
> 
>   BTW, when it is possible to download a debian patch for a package (by setting 
> FOO_PATCH = ...) rather than rolling our own, we prefer that.

The thing is that trying and understanding what Debian does is not
convenient for me and most of the time a waste of time:
- things that I believe are problems for buildroot may not be
considered problems for Debian (see for instance the libltdl issue,
or the grub2 issue). 
- the Debian wiki has a high noise/signal ratio, and is not always
up to date, and this when I can access it, because the IP addresses
I use for browsing are blacklisted by Debian, so I have to take
special steps to access it.

So, I gave up on Debian pretty early, and I am not interested in
trying that painful exercise again. If someone else wants to do it
though, go ahead.
diff mbox

Patch

diff --git a/Makefile b/Makefile
index 86b2ed2..c1d2961 100644
--- a/Makefile
+++ b/Makefile
@@ -223,6 +223,11 @@  ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
+	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
+	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
 endif
 
 # To put more focus on warnings, be less verbose as default