| Message ID | 1509720041684-0.post@n4.nabble.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | BR2_REPRODUCIBLE issues | expand |
>>>>> "Einar" == Einar Jón Gunnarsson <tolvupostur@gmail.com> writes: > Hello all. > I've been using BR2_REPRODUCIBLE, and it mostly works. > But when it is using a git repo, I have the following issues with creating > SOURCE_DATE_EPOCH: > 1) It uses $(GIT), but the GIT variable is only defined in line 487: > "include package/Makefile.in". > Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes > $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH What make version (and buildroot version) are you using? It works here: strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH 2>&1 | grep log [pid 9504] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 [pid 9504] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 [pid 9505] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 [pid 9505] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 It uses the BR2_GIT setting, E.G.: strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH BR2_GIT=foot 2>&1 | grep log [pid 9880] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 [pid 9881] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 This is with current master and make 4.1. > 2) It always uses the last git commit to calculate the date. So any commit > changes the timestamp, while the non-git timestamp stays the same. > This makes the whole thing almost useless. > If the git log command could take a parameter, you could make reproducible > builds based on e.g. tag 2017.08, which would be much more useful. Well, SOURCE_DATE_EPOCH is meant to refer to the date of the last source code change rather than the build time (E.G. so repeated builds of the same sources would give the same results, but builds of changed sources would return a newer timestamp). So I don't really agree that it useless. What we could perhaps do is to only export our value for SOURCE_DATE_EPOCH if not already set in the enviroment, so you can set it outside Buildroot if you want a different behaviour.
On 03-11-17 21:15, Peter Korsgaard wrote: >>>>>> "Einar" == Einar Jón Gunnarsson <tolvupostur@gmail.com> writes: > > > Hello all. > > I've been using BR2_REPRODUCIBLE, and it mostly works. > > > But when it is using a git repo, I have the following issues with creating > > SOURCE_DATE_EPOCH: > > 1) It uses $(GIT), but the GIT variable is only defined in line 487: > > "include package/Makefile.in". > > Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes > > $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH > > What make version (and buildroot version) are you using? It works here: > > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH 2>&1 | grep log > [pid 9504] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 > [pid 9504] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 > [pid 9505] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 > [pid 9505] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 > > It uses the BR2_GIT setting, E.G.: > > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH BR2_GIT=foot 2>&1 | grep log > [pid 9880] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 > [pid 9881] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 > > This is with current master and make 4.1. But not when there is no .config file: $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH /bin/sh: log: command not found SOURCE_DATE_EPOCH= /bin/sh: log: command not found $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH GIT=git SOURCE_DATE_EPOCH=1509720578 Solution: move the GIT definition to top-level Makefile. >> 2) It always uses the last git commit to calculate the date. So any commit > > changes the timestamp, while the non-git timestamp stays the same. > > This makes the whole thing almost useless. > > If the git log command could take a parameter, you could make reproducible > > builds based on e.g. tag 2017.08, which would be much more useful. > > Well, SOURCE_DATE_EPOCH is meant to refer to the date of the last source > code change rather than the build time (E.G. so repeated builds of the > same sources would give the same results, but builds of changed sources > would return a newer timestamp). > > So I don't really agree that it useless. What we could perhaps do is to > only export our value for SOURCE_DATE_EPOCH if not already set in the > enviroment, so you can set it outside Buildroot if you want a different > behaviour. Well, the proposed solution of using a config variable sounds OK. Only, I think it would make more sense to have a config variable with a date, rather than a commit ID. Regards, Arnout
>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes: Hi, >> This is with current master and make 4.1. > But not when there is no .config file: > $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH > /bin/sh: log: command not found > SOURCE_DATE_EPOCH= > /bin/sh: log: command not found No, but that makes sense as there isn't a BR2_GIT setting, so moving the assignment around won't fix that. Having the SOURCE_DATE_EPOCH assignment fail when you don't have a .config (and hence also no BR2_REPRODUCIBLE=y) is imho not a big deal. > Solution: move the GIT definition to top-level Makefile. Sorry, what does that fix? GIT comes from BR2_GIT, E.G. the .config. >> So I don't really agree that it useless. What we could perhaps do is to >> only export our value for SOURCE_DATE_EPOCH if not already set in the >> enviroment, so you can set it outside Buildroot if you want a different >> behaviour. > Well, the proposed solution of using a config variable sounds OK. Only, I think > it would make more sense to have a config variable with a date, rather than a > commit ID. Agreed, or simply only set SOURCE_DATE_EPOCH if it isn't already set in the environment which imho is a more generic solution.
Hello, On Sat, 4 Nov 2017 20:53:27 +0100, Arnout Vandecappelle wrote: > > So I don't really agree that it useless. What we could perhaps do is to > > only export our value for SOURCE_DATE_EPOCH if not already set in the > > enviroment, so you can set it outside Buildroot if you want a different > > behaviour. > > Well, the proposed solution of using a config variable sounds OK. Only, I think > it would make more sense to have a config variable with a date, rather than a > commit ID. Why would we want a config option, rather than just comply with the SOURCE_DATE_EPOCH environment variable, if already defined in the environment? Thomas
Einar, All, On 2017-11-03 07:40 -0700, Einar Jón Gunnarsson spake thusly: > I've been using BR2_REPRODUCIBLE, and it mostly works. > > But when it is using a git repo, I have the following issues with creating > SOURCE_DATE_EPOCH: > 1) It uses $(GIT), but the GIT variable is only defined in line 487: > "include package/Makefile.in". That's not true. In current master, it is defined in: package/pkg-download.mk:15:export GIT := $(call qstrip,$(BR2_GIT)) And GIT has never been defined in package.Makefile.in, ever. What version of Buildroot are you using? > Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes > $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH No it iis not, because BR2_VERSION_GIT_EPOCH is a recursively-expanded varible, that is its value is only evaluated when the variable is evalauated, not when it is asigned. You can test with the simple Makefile: FOO = $(BAR) BAR = yes, works all: @echo "'$(FOO)'" And BR2_VERSION_GIT_EPOCH is only defined when we BR2_REPRODUCIBLE is defioned, which can be true only when we have a .config file, as BR2_REPRODUCIBLE comes form the .config file. And in that case, BR2_GIT is defined, and thus GIT is defined. Note: if the user defied an empty value for BR2_GIT, then quite a few other things will go south as well... > 2) It always uses the last git commit to calculate the date. So any commit > changes the timestamp, while the non-git timestamp stays the same. > This makes the whole thing almost useless. Absolutely not, this is expected: the source-date-epoch stuff guarantees that, given a set of source files, you always get the same binaries (at elast with rgard to date/time). So you really want to use the date of the commit, not the date of the build. See: https://reproducible-builds.org/specs/source-date-epoch/ So, this patch is NAK from me. However, if you really, like realy-really, want to override it to something else, then just export SOURCE_DATE_EPOCH before calling Buildroot. Regards, Yann E. MORIN. > If the git log command could take a parameter, you could make reproducible > builds based on e.g. tag 2017.08, which would be much more useful. > > Regards, > Einar Jón > > An idea for a patch made against origin/next - feel free to do whatever with > it. > if BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION is empty, the behaviour is unchanged > from before. > Maybe setting the default value of BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION to > the last major tag version might make sense. > ------ > From ba5e36391f698946e6b5e02c4e7ab389f88f1373 Mon Sep 17 00:00:00 2001 > From: Einar Jon Gunnarsson <tolvupostur@gmail.com> > Date: Fri, 3 Nov 2017 15:15:16 +0100 > Subject: [PATCH] BR2_REPRODUCIBLE can use a custom git commit > > --- > Config.in | 12 ++++++++++++ > Makefile | 3 ++- > 2 files changed, 14 insertions(+), 1 deletion(-) > > diff --git a/Config.in b/Config.in > index e395995..faab671 100644 > --- a/Config.in > +++ b/Config.in > @@ -726,6 +726,18 @@ config BR2_REPRODUCIBLE > This is labeled as an experimental feature, as not all > packages behave properly to ensure reproducibility. > > +if BR2_REPRODUCIBLE > + > +config BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION > + string "Git version for reproducible builds" > + help > + Revision to use in the typical format used by Git > + E.G. a sha id, a tag, branch, .. > + Uses the last commit if empty. > + Only useful if TOPDIR is a git repository. > + > +endif > + > endmenu > > endmenu > diff --git a/Makefile b/Makefile > index 6d4aeca..a19d20e 100644 > --- a/Makefile > +++ b/Makefile > @@ -253,7 +253,8 @@ export TZ = UTC > export LANG = C > export LC_ALL = C > export GZIP = -n > -BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 > --format=%at) > +GIT := $(call qstrip,$(BR2_GIT)) > +BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 > --format=%at $(BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION)) > export SOURCE_DATE_EPOCH = $(if $(wildcard > $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) > DEPENDENCIES_HOST_PREREQ += host-fakedate > endif > -- > 1.9.1 > > > > -- > Sent from: http://buildroot-busybox.2317881.n4.nabble.com/ > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Arnout, All, On 2017-11-04 20:53 +0100, Arnout Vandecappelle spake thusly: > On 03-11-17 21:15, Peter Korsgaard wrote: > >>>>>> "Einar" == Einar Jón Gunnarsson <tolvupostur@gmail.com> writes: > > > > > Hello all. > > > I've been using BR2_REPRODUCIBLE, and it mostly works. > > > > > But when it is using a git repo, I have the following issues with creating > > > SOURCE_DATE_EPOCH: > > > 1) It uses $(GIT), but the GIT variable is only defined in line 487: > > > "include package/Makefile.in". > > > Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes > > > $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH > > > > What make version (and buildroot version) are you using? It works here: > > > > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH 2>&1 | grep log > > [pid 9504] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 > > [pid 9504] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 > > [pid 9505] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 > > [pid 9505] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 > > > > It uses the BR2_GIT setting, E.G.: > > > > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH BR2_GIT=foot 2>&1 | grep log > > [pid 9880] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 > > [pid 9881] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 > > > > This is with current master and make 4.1. > > But not when there is no .config file: > > $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH > /bin/sh: log: command not found > SOURCE_DATE_EPOCH= > /bin/sh: log: command not found Uh? Why you want to do that? > $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH GIT=git > SOURCE_DATE_EPOCH=1509720578 If anything, SOURCE_DATE_EPOCH should come from the environment, as specified in the spec (see below). > Solution: move the GIT definition to top-level Makefile. That would not solve anything, because it is still defined from BR2_GIT that comes... from .config . > >> 2) It always uses the last git commit to calculate the date. So any commit > > > changes the timestamp, while the non-git timestamp stays the same. > > > This makes the whole thing almost useless. > > > If the git log command could take a parameter, you could make reproducible > > > builds based on e.g. tag 2017.08, which would be much more useful. > > > > Well, SOURCE_DATE_EPOCH is meant to refer to the date of the last source > > code change rather than the build time (E.G. so repeated builds of the > > same sources would give the same results, but builds of changed sources > > would return a newer timestamp). > > > > So I don't really agree that it useless. What we could perhaps do is to > > only export our value for SOURCE_DATE_EPOCH if not already set in the > > enviroment, so you can set it outside Buildroot if you want a different > > behaviour. > > Well, the proposed solution of using a config variable sounds OK. Only, I think > it would make more sense to have a config variable with a date, rather than a > commit ID. Or let the user define SOURCE_DATE_EPOCH in the environment, as this is what would be used and is how SOURCE_DATE_EPOCH is specified; https://reproducible-builds.org/specs/source-date-epoch/ Build systems MUST NOT overwrite this variable for child processes to consume if it is already present. I still fail to see what is wrong with the current situation... Regards, Yann E. MORIN. > Regards, > Arnout > > > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: Hi, > So, this patch is NAK from me. Agreed. > However, if you really, like realy-really, want to override it to > something else, then just export SOURCE_DATE_EPOCH before calling > Buildroot. We afaik don't support this yet as SOURCE_DATE_EPOCH is unconditionally exported, but that could get fixed naturally.
Peter, All, On 2017-11-05 09:54 +0100, Peter Korsgaard spake thusly: > >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > > However, if you really, like realy-really, want to override it to > > something else, then just export SOURCE_DATE_EPOCH before calling > > Buildroot. > > We afaik don't support this yet as SOURCE_DATE_EPOCH is unconditionally > exported, but that could get fixed naturally. Ah, rioght, we do have a bug here: we must keep it as-is if it set. I'll cook-up a patch... Regards, Yann E. MORIN.
On 5 November 2017 at 09:46, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Arnout, All, > > On 2017-11-04 20:53 +0100, Arnout Vandecappelle spake thusly: >> On 03-11-17 21:15, Peter Korsgaard wrote: >> >>>>>> "Einar" == Einar Jón Gunnarsson <tolvupostur@gmail.com> writes: >> > >> > > Hello all. >> > > I've been using BR2_REPRODUCIBLE, and it mostly works. >> > >> > > But when it is using a git repo, I have the following issues with creating >> > > SOURCE_DATE_EPOCH: >> > > 1) It uses $(GIT), but the GIT variable is only defined in line 487: >> > > "include package/Makefile.in". >> > > Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes >> > > $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH >> > >> > What make version (and buildroot version) are you using? It works here: >> > >> > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH 2>&1 | grep log >> > [pid 9504] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 >> > [pid 9504] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 >> > [pid 9505] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git git log -1 --format=%at"], [/* 46 vars */]) = 0 >> > [pid 9505] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at"], [/* 47 vars */]) = 0 >> > >> > It uses the BR2_GIT setting, E.G.: >> > >> > strace -s 100 -f -e execve make BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH BR2_GIT=foot 2>&1 | grep log >> > [pid 9880] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 >> > [pid 9881] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/peko/source/buildroot/.git foot log -1 --format=%at"], [/* 46 vars */]) = 0 >> > >> > This is with current master and make 4.1. >> >> But not when there is no .config file: >> >> $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH >> /bin/sh: log: command not found >> SOURCE_DATE_EPOCH= >> /bin/sh: log: command not found > > Uh? Why you want to do that? It's a symptom - it's not the only way to get that result. It seems easier to reproduce with an out-of-tree build, but that is surely not the only trigger. I have a hard time reproducing it on master, but in my production Buildroot 2017.02, I get this: $ strace -s 100 -f make O=output_armxl 2>&1 | grep git.*logstrace -s 100 -f make O=output_armxl 2>&1 | grep -e git.*log [pid 27208] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git log -1 --"...], [/* 86 vars */] <unfinished ...> [pid 27208] stat("/home/ejg/git/Platform3/Build/bin/log", 0x7ffd410c57d0) = -1 ENOENT (No such file or directory) [pid 27314] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git git log -1"...], [/* 86 vars */] <unfinished ...> [pid 27314] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 [pid 27385] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git git log -1"...], [/* 86 vars */] <unfinished ...> [pid 27385] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 [pid 27386] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git git log -1"...], [/* 86 vars */] <unfinished ...> [pid 27386] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 $ strace -s 100 -f make O=output_armxl GIT=git 2>&1 | grep git.*log [pid 23959] execve("/bin/bash", ["/bin/bash", "-c", "GIT_DIR=/home/jenkinslinux/jenkins/workspace/buildroot_for_armxl/armxl-buildroot/.git git log -1"...], [/* 87 vars */] <unfinished ...> [pid 23959] execve("/usr/bin/git", ["git", "log", "-1", "--format=%at", "1.0.0.22"], [/* 88 vars */]) = 0 etc... Note that the first run fails. I had a error check that would stop the whole make in that case,which triggered on that. even if the build works on the second run onwards, I can call this a non-issue, > >> $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH GIT=git >> SOURCE_DATE_EPOCH=1509720578 > > If anything, SOURCE_DATE_EPOCH should come from the environment, as > specified in the spec (see below). > >> Solution: move the GIT definition to top-level Makefile. > > That would not solve anything, because it is still defined from BR2_GIT > that comes... from .config . > >> >> 2) It always uses the last git commit to calculate the date. So any commit >> > > changes the timestamp, while the non-git timestamp stays the same. >> > > This makes the whole thing almost useless. >> > > If the git log command could take a parameter, you could make reproducible >> > > builds based on e.g. tag 2017.08, which would be much more useful. >> > >> > Well, SOURCE_DATE_EPOCH is meant to refer to the date of the last source >> > code change rather than the build time (E.G. so repeated builds of the >> > same sources would give the same results, but builds of changed sources >> > would return a newer timestamp). >> > >> > So I don't really agree that it useless. What we could perhaps do is to >> > only export our value for SOURCE_DATE_EPOCH if not already set in the >> > enviroment, so you can set it outside Buildroot if you want a different >> > behaviour. >> >> Well, the proposed solution of using a config variable sounds OK. Only, I think >> it would make more sense to have a config variable with a date, rather than a >> commit ID. > > Or let the user define SOURCE_DATE_EPOCH in the environment, as this is > what would be used and is how SOURCE_DATE_EPOCH is specified; > https://reproducible-builds.org/specs/source-date-epoch/ > > Build systems MUST NOT overwrite this variable for child processes > to consume if it is already present. > > I still fail to see what is wrong with the current situation... The point is that you currently have close to 1900 packages. Most commits only changes 1 package. So you have 1899 packages with the exact same source code, but they produce different binaries because someone changed the 1900th package. Some people see that as an issue. For example we have a setup where we want to update embedded devices in the field. The update should be as small as possible, so we take a diff of the target/ for baseline commit (that BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION points to) and the new build. To fix a bug in a single tiny file that results in a 7.5MB update package if SOURCE_DATE_EPOCH keeps changing, but it can be below 4kb if the SOURCE_DATE_EPOCH is the same. But supplying the SOURCE_DATE_EPOCH from Jenkins is fine by me. So the fix is great. I'll shut up now. Thanks > > Regards, > Yann E. MORIN. > >> Regards, >> Arnout >> >> >> -- >> Arnout Vandecappelle arnout at mind be >> Senior Embedded Software Architect +32-16-286500 >> Essensium/Mind http://www.mind.be >> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven >> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle >> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF >> _______________________________________________ >> buildroot mailing list >> buildroot@busybox.net >> http://lists.busybox.net/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Einar, All, On 2017-11-06 10:43 +0100, Einar Jón spake thusly: > On 5 November 2017 at 09:46, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > I still fail to see what is wrong with the current situation... > > The point is that you currently have close to 1900 packages. Most > commits only changes 1 package. > So you have 1899 packages with the exact same source code, but they > produce different binaries because someone changed the 1900th package. > Some people see that as an issue. But you usually have no way, Buildroot-wise, to know that a change in a package has no impact on other packages. For example, is all you change is a library, and that library now exposes an extra header that can then be detected by another package, then that other package will have to be rebuilt because, although it is the same source tree, the build of that package will be different because it will see new features in that library. And that is not theoretical: there are packages that do things like: checking for libfoo... yes checking if libfoo understand blabla... no And that when you update libfoo, will now behave as: checking for libfoo... yes checking if libfoo understand blabla... yes So, from Buildroot's point of view, and probably from most users' point of view, it is impossible to know if changing one package will have an impact on other packages. So we do really want to have SOURCE_DATE_EPOCH, *as exported by Buildroot*, to contain the date of the last git commit, because that is what makes sense: the whole recipe has changed, because one of its constituent has changed, so the result as a whole will change. Now, that parts of the whole do not change is obviously correct, but then it is so technically complex to *guarantee* a clean separation, that it is nigh impossible. > For example we have a setup where we want to update embedded devices > in the field. > The update should be as small as possible, so we take a diff of the > target/ for baseline commit (that BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION > points to) and the new build. > To fix a bug in a single tiny file that results in a 7.5MB update > package if SOURCE_DATE_EPOCH keeps changing, but it can be below 4kb > if the SOURCE_DATE_EPOCH is the same. I do see your use case, and I do understand it: you basically want to do partial upgrades of your system (for various reasons that are your own right). But Buildroot does not build packages like a distro, like OpenEmbedded or like openWRT (or others) do. Instead, Buildroot builds a complete rootfs; it is the atomic artefact output by Buildroot. As such, what matters for Buildroot is the whole, not the parts. See also the manual, where we explain at length why we are not doing binary packages, especially the last part about our position statement, tangentially related to the explanation of your use-case: https://buildroot.org/downloads/manual/manual.html#faq-no-binary-packages > But supplying the SOURCE_DATE_EPOCH from Jenkins is fine by me. So the > fix is great. > > I'll shut up now. Sorry if you found my and the others' replies to imply that you should "shut up". That was definitely not my intention, and I know Peter, Arnout and Thomas enough to know that it was not theirs as well. Your feedback was valuable; it just does not fit in the Buildroot way of things. Regards, Yann E. MORIN. > Thanks > > > > > Regards, > > Yann E. MORIN. > > > >> Regards, > >> Arnout > >> > >> > >> -- > >> Arnout Vandecappelle arnout at mind be > >> Senior Embedded Software Architect +32-16-286500 > >> Essensium/Mind http://www.mind.be > >> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > >> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > >> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF > >> _______________________________________________ > >> buildroot mailing list > >> buildroot@busybox.net > >> http://lists.busybox.net/mailman/listinfo/buildroot > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' > > > > -- > Regards > Einar Jón > +31 610 957234
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: Hi, > I do see your use case, and I do understand it: you basically want to do > partial upgrades of your system (for various reasons that are your own > right). > But Buildroot does not build packages like a distro, like OpenEmbedded > or like openWRT (or others) do. > Instead, Buildroot builds a complete rootfs; it is the atomic artefact > output by Buildroot. As such, what matters for Buildroot is the whole, > not the parts. > See also the manual, where we explain at length why we are not doing > binary packages, especially the last part about our position statement, > tangentially related to the explanation of your use-case: > https://buildroot.org/downloads/manual/manual.html#faq-no-binary-packages Small deltas might be interesting even though you do full system upgrades, E.G. to save on bandwidth with binary delta tools like xdelta/bsdiff or casync. I believe there was some talk about adding casync support to rauc. > Sorry if you found my and the others' replies to imply that you should > "shut up". That was definitely not my intention, and I know Peter, > Arnout and Thomas enough to know that it was not theirs as well. Exactly!
On 06-11-17 10:43, Einar Jón wrote: > On 5 November 2017 at 09:46, Yann E. MORIN <yann.morin.1998@free.fr> wrote: [snip] > It seems easier to reproduce with an out-of-tree build, but that is > surely not the only trigger. > I have a hard time reproducing it on master, but in my production > Buildroot 2017.02, I get this: > > $ strace -s 100 -f make O=output_armxl 2>&1 | grep git.*logstrace -s > 100 -f make O=output_armxl 2>&1 | grep -e git.*log > [pid 27208] execve("/bin/bash", ["/bin/bash", "-c", > "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git > log -1 --"...], [/* 86 vars */] <unfinished ...> I guess that must be due to some local hacks then, otherwise BR2_REPRODUCIBLE can only be set to y if .config is parsed, and if .config is parsed then the definition of GIT is included as well. > [pid 27208] stat("/home/ejg/git/Platform3/Build/bin/log", > 0x7ffd410c57d0) = -1 ENOENT (No such file or directory) > [pid 27314] execve("/bin/bash", ["/bin/bash", "-c", > "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git > git log -1"...], [/* 86 vars */] <unfinished ...> > [pid 27314] execve("/usr/bin/git", ["git", "log", "-1", > "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 > [pid 27385] execve("/bin/bash", ["/bin/bash", "-c", > "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git > git log -1"...], [/* 86 vars */] <unfinished ...> > [pid 27385] execve("/usr/bin/git", ["git", "log", "-1", > "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 > [pid 27386] execve("/bin/bash", ["/bin/bash", "-c", > "GIT_DIR=/home/jenkinslinux/jenkins/workspace/SWx_buildroot_for_armxl/armxl-buildroot/.git > git log -1"...], [/* 86 vars */] <unfinished ...> > [pid 27386] execve("/usr/bin/git", ["git", "log", "-1", > "--format=%at", "1.0.0.22"], [/* 87 vars */]) = 0 > > $ strace -s 100 -f make O=output_armxl GIT=git 2>&1 | grep git.*log > [pid 23959] execve("/bin/bash", ["/bin/bash", "-c", > "GIT_DIR=/home/jenkinslinux/jenkins/workspace/buildroot_for_armxl/armxl-buildroot/.git > git log -1"...], [/* 87 vars */] <unfinished ...> > [pid 23959] execve("/usr/bin/git", ["git", "log", "-1", > "--format=%at", "1.0.0.22"], [/* 88 vars */]) = 0 > etc... > > Note that the first run fails. I had a error check that would stop the > whole make in that case,which triggered on that. even if the build > works on the second run onwards, I can call this a non-issue, > > > >> >>> $ make O=/tmp/brtest BR2_REPRODUCIBLE=y printvars VARS=SOURCE_DATE_EPOCH GIT=git >>> SOURCE_DATE_EPOCH=1509720578 >> >> If anything, SOURCE_DATE_EPOCH should come from the environment, as >> specified in the spec (see below). >> >>> Solution: move the GIT definition to top-level Makefile. >> >> That would not solve anything, because it is still defined from BR2_GIT >> that comes... from .config . Well, yes, indeed, it would need a default value as well. [snip] > But supplying the SOURCE_DATE_EPOCH from Jenkins is fine by me. So the > fix is great. So, will you make a patch that fixes it? Regards, Arnout
Arnout, All, On 2017-11-06 22:34 +0100, Arnout Vandecappelle spake thusly: > On 06-11-17 10:43, Einar Jón wrote: [--SNIP--] > > But supplying the SOURCE_DATE_EPOCH from Jenkins is fine by me. So the > > fix is great. > > So, will you make a patch that fixes it? It has alreday been applied by Peter yesterday: 0437d2f8f6e core/reproducible: do not override SOURCE_DATE_EPOCH Regards, Yann E. MORIN.
On 06-11-17 18:50, Yann E. MORIN wrote: > Einar, All, > > On 2017-11-06 10:43 +0100, Einar Jón spake thusly: >> On 5 November 2017 at 09:46, Yann E. MORIN <yann.morin.1998@free.fr> wrote: >>> I still fail to see what is wrong with the current situation... >> >> The point is that you currently have close to 1900 packages. Most >> commits only changes 1 package. >> So you have 1899 packages with the exact same source code, but they >> produce different binaries because someone changed the 1900th package. >> Some people see that as an issue. > > But you usually have no way, Buildroot-wise, to know that a change in a > package has no impact on other packages. [snip good explanation of how this can happen] > So we do really want to have SOURCE_DATE_EPOCH, *as exported by Buildroot*, > to contain the date of the last git commit, because that is what makes > sense: the whole recipe has changed, because one of its constituent has > changed, so the result as a whole will change. ... except that SOURCE_DATE_EPOCH is NOT a 'hash' or identifier of the reproducible build. On the contrary, it is just there to make sure that two builds *of the same configuration* at different times will give the same binaries. If the configuration is different, then the binaries will be different whatever the value of SOURCE_DATE_EPOCH. So really, we could just set SOURCE_DATE_EPOCH to 0 (1 Jan 1970) and be done with it. The only reason to have a "reasonable" value of SOURCE_DATE_EPOCH is that __DATE__ and __TIME__ expand to somewhat meaningful values. It would be an entirely different story if we would start using BR2_VERSION_GIT_EPOCH as a kind of version marker to be able to reuse a previously built tarball of the per-package SDK-and-target directories. But we're not quite there yet :-) Regards, Arnout
Arnout, All, On 2017-11-06 23:50 +0100, Arnout Vandecappelle spake thusly: > On 06-11-17 18:50, Yann E. MORIN wrote: > > Einar, All, > > > > On 2017-11-06 10:43 +0100, Einar Jón spake thusly: > >> On 5 November 2017 at 09:46, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > >>> I still fail to see what is wrong with the current situation... > >> > >> The point is that you currently have close to 1900 packages. Most > >> commits only changes 1 package. > >> So you have 1899 packages with the exact same source code, but they > >> produce different binaries because someone changed the 1900th package. > >> Some people see that as an issue. > > > > But you usually have no way, Buildroot-wise, to know that a change in a > > package has no impact on other packages. > [snip good explanation of how this can happen] > > So we do really want to have SOURCE_DATE_EPOCH, *as exported by Buildroot*, > > to contain the date of the last git commit, because that is what makes > > sense: the whole recipe has changed, because one of its constituent has > > changed, so the result as a whole will change. > > ... except that SOURCE_DATE_EPOCH is NOT a 'hash' or identifier of the > reproducible build. On the contrary, it is just there to make sure that two > builds *of the same configuration* at different times will give the same > binaries. If the configuration is different, then the binaries will be different > whatever the value of SOURCE_DATE_EPOCH. Yes... I never said anything implying that given an S.D.E, one could ensure reproducibility of binaries across configurations, no. Reproducibility is only guaranteed when all the follwing conditions are met: - a given Buildroot tree - a given set of packages - a given S.D.E. - probably quite a few other factors (like the path and user) If you change a package version, then nothing can guarantee the reproducibility of that package, obviously, but I argue that it is not even possible to guarantee the reproducibility of other packages, especially those that depend on the changed package. Thus, if S.D.E. has to point at a specific time, it shall be the one when the package was updated. Now, if a specific use case, like the one for Einar, works by setting a constant S.D.E, that is no longer the responsibility of Buildroot to set S.D.E., especially since we now abide by the spec to not override it if already set. > So really, we could just set SOURCE_DATE_EPOCH to 0 (1 Jan 1970) and be done > with it. The only reason to have a "reasonable" value of SOURCE_DATE_EPOCH is > that __DATE__ and __TIME__ expand to somewhat meaningful values. Ah, did you mean that we could just fake a constant forever and be done with it? No we can't use '0', because some packages will detect that the date is uterly wrong, too long in the past, and will refuse to work at all. And any other arbitrary value is no better than the actual last source change. > It would be an entirely different story if we would start using > BR2_VERSION_GIT_EPOCH as a kind of version marker to be able to reuse a > previously built tarball of the per-package SDK-and-target directories. But > we're not quite there yet :-) And we will not go that route, because that would mean we are doing binary packages, which is *way* more complex that just looking at the date. ;-) But for Buildroot, we do really want to set S.D.E. to a meaningful value, yes. And that value is the time of the last change of the Buildroot source tree as a whole. Yes, the spec does not madate that it is; it only says 'should'. But it does not make sense, from the Buildroot point of view, to set it to anything else... Regards, Yann E. MORIN. > Regards, > Arnout > > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: Hi, > But for Buildroot, we do really want to set S.D.E. to a meaningful > value, yes. And that value is the time of the last change of the > Buildroot source tree as a whole. > Yes, the spec does not madate that it is; it only says 'should'. But it > does not make sense, from the Buildroot point of view, to set it to > anything else... I agree.
On 08-11-17 19:00, Yann E. MORIN wrote: > Yes, the spec does not madate that it is; it only says 'should'. But it > does not make sense, from the Buildroot point of view, to set it to > anything else... Absolutely. I just wanted to say: if someone wants to use a different value, that's OK as well. Regards, Arnout
diff --git a/Config.in b/Config.in index e395995..faab671 100644 --- a/Config.in +++ b/Config.in @@ -726,6 +726,18 @@ config BR2_REPRODUCIBLE This is labeled as an experimental feature, as not all packages behave properly to ensure reproducibility. +if BR2_REPRODUCIBLE + +config BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION + string "Git version for reproducible builds" + help + Revision to use in the typical format used by Git + E.G. a sha id, a tag, branch, .. + Uses the last commit if empty. + Only useful if TOPDIR is a git repository. + +endif + endmenu endmenu diff --git a/Makefile b/Makefile index 6d4aeca..a19d20e 100644 --- a/Makefile +++ b/Makefile @@ -253,7 +253,8 @@ export TZ = UTC export LANG = C export LC_ALL = C export GZIP = -n -BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) +GIT := $(call qstrip,$(BR2_GIT)) +BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at $(BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION)) export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) DEPENDENCIES_HOST_PREREQ += host-fakedate
Hello all. I've been using BR2_REPRODUCIBLE, and it mostly works. But when it is using a git repo, I have the following issues with creating SOURCE_DATE_EPOCH: 1) It uses $(GIT), but the GIT variable is only defined in line 487: "include package/Makefile.in". Note that this is done around line 250 in $(TOPDIR)/Makefile, so sometimes $(GIT) is unset then getting BR2_VERSION_GIT_EPOCH 2) It always uses the last git commit to calculate the date. So any commit changes the timestamp, while the non-git timestamp stays the same. This makes the whole thing almost useless. If the git log command could take a parameter, you could make reproducible builds based on e.g. tag 2017.08, which would be much more useful. Regards, Einar Jón An idea for a patch made against origin/next - feel free to do whatever with it. if BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION is empty, the behaviour is unchanged from before. Maybe setting the default value of BR2_REPRODUCIBLE_CUSTOM_REPO_VERSION to the last major tag version might make sense. ------ From ba5e36391f698946e6b5e02c4e7ab389f88f1373 Mon Sep 17 00:00:00 2001 From: Einar Jon Gunnarsson <tolvupostur@gmail.com> Date: Fri, 3 Nov 2017 15:15:16 +0100 Subject: [PATCH] BR2_REPRODUCIBLE can use a custom git commit --- Config.in | 12 ++++++++++++ Makefile | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-) endif -- 1.9.1 -- Sent from: http://buildroot-busybox.2317881.n4.nabble.com/