| Message ID | 20191210215952.902477-3-thomas.petazzoni@bootlin.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Ensure extractor dependencies are taken into account | expand |
Le 10/12/2019 à 22:59, Thomas Petazzoni a écrit : > To extract some archive types, if the extracting tool is not available > on the system where Buildroot runs on, we build it using a Buildroot > host package. > > Such dependencies are currently explicitly handled by the > inner-generic-package macro, but in fact we also need to handle them > in all places where the "suitable-extractor" macro is invoked, and > some packages invoke it directly. Otherwise, such packages may be > missing a dependency to the appropriate host Buildroot package > building the extracting tool they need. An example is gettext-tiny, > whose source code is a gzip-compressed tarball, but in addition > manually extracts a xz-compressed tarball. > > This extractor-pkg-dependency macro will be used in follow-up commits > to ensure all the packages that use suitable-extractor properly add > the correct dependencies. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/pkg-utils.mk | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk > index 0f916d81d8..35a441e0aa 100644 > --- a/package/pkg-utils.mk > +++ b/package/pkg-utils.mk > @@ -46,6 +46,14 @@ INFLATE.tar = cat > # suitable-extractor(filename): returns extractor based on suffix > suitable-extractor = $(INFLATE$(suffix $(1))) > What about gzip ? EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY) EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY) Best regards, Romain > +EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY) > +EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY) > +EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY) > + > +# extractor-pkg-dependency(filename): returns a Buildroot package > +# dependency needed to extract file based on suffix > +extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))) > + > # extractor-system-dependency(filename): returns the name of the tool > # needed to extract 'filename', and is meant to be used with > # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is >
On Sat, 14 Dec 2019 22:20:21 +0100 Romain Naour <romain.naour@smile.fr> wrote: > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk > > index 0f916d81d8..35a441e0aa 100644 > > --- a/package/pkg-utils.mk > > +++ b/package/pkg-utils.mk > > @@ -46,6 +46,14 @@ INFLATE.tar = cat > > # suitable-extractor(filename): returns extractor based on suffix > > suitable-extractor = $(INFLATE$(suffix $(1))) > > > > What about gzip ? > > EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY) > EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY) Because oddly enough, we are today not taking into account the dependency on gzip when extracting tarballs. We do take it into account only in: ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),) $(2)_DOWNLOAD_DEPENDENCIES += \ $(BR2_GZIP_HOST_DEPENDENCY) \ $(BR2_TAR_HOST_DEPENDENCY) endif i.e, when we fetch from a version control system, and create our own tarball locally. But if we download directly from http/ftp a .tar.gz or .tgz, we don't rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one from the host machine. This should be fixed, I agree, but it's already broken today I believe. Best regards, Thomas
Thomas, Romain, All, On 2019-12-14 22:42 +0100, Thomas Petazzoni spake thusly: > On Sat, 14 Dec 2019 22:20:21 +0100 > Romain Naour <romain.naour@smile.fr> wrote: > > > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk > > > index 0f916d81d8..35a441e0aa 100644 > > > --- a/package/pkg-utils.mk > > > +++ b/package/pkg-utils.mk > > > @@ -46,6 +46,14 @@ INFLATE.tar = cat > > > # suitable-extractor(filename): returns extractor based on suffix > > > suitable-extractor = $(INFLATE$(suffix $(1))) > > > > > > > What about gzip ? > > > > EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY) > > EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY) > > Because oddly enough, we are today not taking into account the > dependency on gzip when extracting tarballs. > > We do take it into account only in: > > ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),) > $(2)_DOWNLOAD_DEPENDENCIES += \ > $(BR2_GZIP_HOST_DEPENDENCY) \ > $(BR2_TAR_HOST_DEPENDENCY) > endif > > i.e, when we fetch from a version control system, and create our own > tarball locally. > > But if we download directly from http/ftp a .tar.gz or .tgz, we don't > rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one > from the host machine. > > This should be fixed, I agree, but it's already broken today I believe. In fact no. We need to build our own host-gzip only to create archives, because some host have pigz instead of gzip, and this creates non-reproducible archives. However, decompressing with gzip or pigz is exactly the same; we do not need our own host-gzip, and we can rely on the system-provided gzip, whether that be gzip of pigz. So, the current state of affairs is, AFAICS, correct. Regards, Yann E. MORIN. > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index 0f916d81d8..35a441e0aa 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -46,6 +46,14 @@ INFLATE.tar = cat # suitable-extractor(filename): returns extractor based on suffix suitable-extractor = $(INFLATE$(suffix $(1))) +EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY) +EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY) +EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY) + +# extractor-pkg-dependency(filename): returns a Buildroot package +# dependency needed to extract file based on suffix +extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))) + # extractor-system-dependency(filename): returns the name of the tool # needed to extract 'filename', and is meant to be used with # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
To extract some archive types, if the extracting tool is not available on the system where Buildroot runs on, we build it using a Buildroot host package. Such dependencies are currently explicitly handled by the inner-generic-package macro, but in fact we also need to handle them in all places where the "suitable-extractor" macro is invoked, and some packages invoke it directly. Otherwise, such packages may be missing a dependency to the appropriate host Buildroot package building the extracting tool they need. An example is gettext-tiny, whose source code is a gzip-compressed tarball, but in addition manually extracts a xz-compressed tarball. This extractor-pkg-dependency macro will be used in follow-up commits to ensure all the packages that use suitable-extractor properly add the correct dependencies. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- package/pkg-utils.mk | 8 ++++++++ 1 file changed, 8 insertions(+)