| Message ID | 20230209231518.9458-1-ckhardin@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [1/4] linux: create a scmversion file based on the custom repo variables | expand |
Hi Charles, First of all, in patchwork [1] there are about twelve patches from you about this subject, but I think all except these four are superseded. Is that correct? Could you update patchwork and mark the old ones as superseded? On 10/02/2023 00:15, ckhardin@gmail.com wrote: > From: Charles Hardin <ckhardin@gmail.com> > > When using the custom repositories, the archive creation will > omit the source history to get reproducible hashes. This has > the side effect of making the setlocalversion scripts within the > builds to not create the tracking information as expected. So, > just generate a .scmversion so that kernels will be labelled > with where buildroot referenced them. This is less than ideal because it's completely different than the normal git describe output. In particular, if you first develop something in an OVERRIDE_SRCDIR (after applying patch 4) and then change the config to point to the exact same commit with BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, it will create a completely different UTS_RELEASE string... I think a better solution would be to extend the git download wrapper with a hook that gets called in the git tree itself. E.g.: LINUX_SCMVERSION_HOOK = ./scripts/setlocalversion --save-scmversion The same hook could then also be used for rsync. And it would be easy to add the same feature to barebox, tf-a, ... Regards, Arnout > > For example (line wrapped) > > #define UTS_RELEASE "5.4.70-repo-version-" \ > "92c23d10d0031e0a13244a1a97e7530d3b53fc6e" > > Signed-off-by: Charles Hardin <ckhardin@gmail.com> > --- > linux/linux.mk | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > > diff --git a/linux/linux.mk b/linux/linux.mk > index 7645b5f507..0b27c8dee2 100644 > --- a/linux/linux.mk > +++ b/linux/linux.mk > @@ -596,6 +596,20 @@ endif > ifeq ($(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL)),) > $(error No custom repo URL set. Check your BR2_LINUX_KERNEL_CUSTOM_REPO_URL setting) > endif > + > +# Create a custom scm version file to reflect the source version since the > +# archive will omit source directories like .git to maintain reproducible > +# hashes for the archives > +LINUX_CUSTOM_REPO_SCMVERSION = \ > + "-repo-version-$(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION))" > +define LINUX_CUSTOM_REPO_SCMVERSION_HOOK > + (cd $(@D); \ > + if [ ! -f .scmversion ]; then \ > + echo $(LINUX_CUSTOM_REPO_SCMVERSION) > .scmversion; \ > + fi) > +endef > + > +LINUX_POST_EXTRACT_HOOKS += LINUX_CUSTOM_REPO_SCMVERSION_HOOK > endif > > ifeq ($(BR_BUILDING),y)
> On Feb 14, 2023, at 1:46 PM, Arnout Vandecappelle <arnout@mind.be> wrote: > > Hi Charles, > > First of all, in patchwork [1] there are about twelve patches from you about this subject, but I think all except these four are superseded. Is that correct? Could you update patchwork and mark the old ones as superseded? > Yeah, sorry - had some mail server changes that kept saying an error but apparently the mail went thru. > > On 10/02/2023 00:15, ckhardin@gmail.com wrote: >> From: Charles Hardin <ckhardin@gmail.com> >> When using the custom repositories, the archive creation will >> omit the source history to get reproducible hashes. This has >> the side effect of making the setlocalversion scripts within the >> builds to not create the tracking information as expected. So, >> just generate a .scmversion so that kernels will be labelled >> with where buildroot referenced them. > > This is less than ideal because it's completely different than the normal git describe output. In particular, if you first develop something in an OVERRIDE_SRCDIR (after applying patch 4) and then change the config to point to the exact same commit with BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, it will create a completely different UTS_RELEASE string... Agreed -I would argue that the OVERRIDE_SRCDIR patch is functionally correct and what is desired particularly for local dev but the CUSTOM_REPO is a bit hackish and broken. Trying to figure out a way to do that. > > I think a better solution would be to extend the git download wrapper with a hook that gets called in the git tree itself. E.g.: > > LINUX_SCMVERSION_HOOK = ./scripts/setlocalversion --save-scmversion > > The same hook could then also be used for rsync. And it would be easy to add the same feature to barebox, tf-a, ... That is a good idea - but, from what I can tell the git download script does a pull and extract all at once with no hooks being run from support/download/git ``` #!/usr/bin/env bash # NOTE: if the output of this backend has to change (e.g. we change what gets # included in the archive (e.g. LFS), or we change the format of the archive # (e.g. tar options, compression ratio or method)), we MUST update the format # version in the variable BR_FMT_VERSION_git, in package/pkg-download.mk. # We want to catch any unexpected failure, and exit immediately set -E … snip snip … Download from git repo in here … snip snip … # Generate the archive. # We do not want the .git dir; we keep other .git files, in case they are the # only files in their directory. # The .git dir would generate non reproducible tarballs as it depends on # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. mk_tar_gz "${git_cache}" "${basename}" "${date}" "${output}" ".git/*” ``` So from the above I don’t see where to put a HOOK in from the make scripts - is there an example for this because we cann’t run the hook from the tarball because there is no “.git” history in the tarball and the hook would need to be done at the creation? right? > > Regards, > Arnout > > >> For example (line wrapped) >> #define UTS_RELEASE "5.4.70-repo-version-" \ >> "92c23d10d0031e0a13244a1a97e7530d3b53fc6e" >> Signed-off-by: Charles Hardin <ckhardin@gmail.com> >> --- >> linux/linux.mk | 14 ++++++++++++++ >> 1 file changed, 14 insertions(+) >> diff --git a/linux/linux.mk b/linux/linux.mk >> index 7645b5f507..0b27c8dee2 100644 >> --- a/linux/linux.mk >> +++ b/linux/linux.mk >> @@ -596,6 +596,20 @@ endif >> ifeq ($(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL)),) >> $(error No custom repo URL set. Check your BR2_LINUX_KERNEL_CUSTOM_REPO_URL setting) >> endif >> + >> +# Create a custom scm version file to reflect the source version since the >> +# archive will omit source directories like .git to maintain reproducible >> +# hashes for the archives >> +LINUX_CUSTOM_REPO_SCMVERSION = \ >> + "-repo-version-$(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION))" >> +define LINUX_CUSTOM_REPO_SCMVERSION_HOOK >> + (cd $(@D); \ >> + if [ ! -f .scmversion ]; then \ >> + echo $(LINUX_CUSTOM_REPO_SCMVERSION) > .scmversion; \ >> + fi) >> +endef >> + >> +LINUX_POST_EXTRACT_HOOKS += LINUX_CUSTOM_REPO_SCMVERSION_HOOK >> endif >> ifeq ($(BR_BUILDING),y)
Adding source control tracking into the linux and uboot packages specifically. The generic problem is that buildroot needs to make reproducible tarballs for verification which means the ".git" directories have to be excluded from the archive since they can change even though the contents from the git reference has not changed. This leads to the problem that a setlocalversion script will not work in the build because after the extract there is no access to the history information given the reasons above. To mitigate this problem and allow change control tracking for local development and remote references a somewhat intruisive change needs to be introduced at the download step for the archive to add an scmversion during the creation when both the revision history in the source tree exists and the archive is being created. Charles Hardin (3): support/download: add a helper scipt to generate scmversions boot/uboot: add a scmversion rsync hook when overriding srcdir linux: add a scmversion rsync hook when overriding srcdir boot/uboot/uboot.mk | 12 +++++++ linux/linux.mk | 12 +++++++ package/pkg-download.mk | 1 + support/download/dl-wrapper | 5 +-- support/download/git | 10 ++++++ support/download/hg | 10 ++++++ support/download/scmversion | 63 +++++++++++++++++++++++++++++++++++++ support/download/svn | 10 ++++++ 8 files changed, 121 insertions(+), 2 deletions(-) create mode 100755 support/download/scmversion
diff --git a/linux/linux.mk b/linux/linux.mk index 7645b5f507..0b27c8dee2 100644 --- a/linux/linux.mk +++ b/linux/linux.mk @@ -596,6 +596,20 @@ endif ifeq ($(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL)),) $(error No custom repo URL set. Check your BR2_LINUX_KERNEL_CUSTOM_REPO_URL setting) endif + +# Create a custom scm version file to reflect the source version since the +# archive will omit source directories like .git to maintain reproducible +# hashes for the archives +LINUX_CUSTOM_REPO_SCMVERSION = \ + "-repo-version-$(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION))" +define LINUX_CUSTOM_REPO_SCMVERSION_HOOK + (cd $(@D); \ + if [ ! -f .scmversion ]; then \ + echo $(LINUX_CUSTOM_REPO_SCMVERSION) > .scmversion; \ + fi) +endef + +LINUX_POST_EXTRACT_HOOKS += LINUX_CUSTOM_REPO_SCMVERSION_HOOK endif ifeq ($(BR_BUILDING),y)