[RFCv3,08/15] package/pkg-generic: handle host-xz as an extract dependency

Message ID 20171201205352.24287-9-thomas.petazzoni@free-electrons.com
State New
Headers show
Series
  • Per-package host/target directory support
Related show

Commit Message

Thomas Petazzoni Dec. 1, 2017, 8:53 p.m.
This moves the host-xz dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.

To achieve that, check-host-xz.mk fills in the
BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.

The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:

 - host-xz, because we would otherwise depend on ourself.

 - host-tar, because xz itself is delivered as a tarball, so we need
   to have host-xz depend on host-tar, and not host-tar depend on
   host-xz

 - host-skeleton, because we need to have host-xz depend on
   host-skeleton, and not the opposite.

In addition, we modify xz.mk to explicitly build host-xz without
ccache. We generally took the approach of building host-ccache *after*
all the extractors have been built.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 package/pkg-generic.mk                   | 4 ++++
 package/xz/xz.mk                         | 5 +++++
 support/dependencies/check-host-xzcat.mk | 2 +-
 3 files changed, 10 insertions(+), 1 deletion(-)

Comments

Yann E. MORIN Dec. 2, 2017, 3:08 p.m. | #1
Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:
> 
>  - host-xz, because we would otherwise depend on ourself.
> 
>  - host-tar, because xz itself is delivered as a tarball, so we need
>    to have host-xz depend on host-tar, and not host-tar depend on
>    host-xz
> 
>  - host-skeleton, because we need to have host-xz depend on
>    host-skeleton, and not the opposite.
> 
> In addition, we modify xz.mk to explicitly build host-xz without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - New patch
> ---
>  package/pkg-generic.mk                   | 4 ++++
>  package/xz/xz.mk                         | 5 +++++
>  support/dependencies/check-host-xzcat.mk | 2 +-
>  3 files changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 86dd2561d4..dfd4719d29 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,6 +591,10 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> +endif

So, all packages will now depend on this, even those that are not
compressed with xz?

Currently, filesystems that want to compress with xz all depend on
host-xz unconditioanlly. Do we want to change that as well, so they only
depend on it if needed?

Regards,
Yann E. MORIN.

>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/package/xz/xz.mk b/package/xz/xz.mk
> index e8116f24ae..bcdac13ee3 100644
> --- a/package/xz/xz.mk
> +++ b/package/xz/xz.mk
> @@ -18,5 +18,10 @@ else
>  XZ_CONF_OPTS = --disable-threads
>  endif
>  
> +# we are built before ccache
> +HOST_XZ_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index 9be75c7311..e5d72f95dd 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -2,7 +2,7 @@
>  # If it is not present, build our own host-xzcat
>  
>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-xz
> +BR2_XZCAT_HOST_DEPENDENCY = host-xz
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>  XZCAT = $(HOST_DIR)/bin/xzcat
>  endif
> -- 
> 2.13.6
>
Thomas Petazzoni Dec. 2, 2017, 8:16 p.m. | #2
Hello,

On Sat, 2 Dec 2017 16:08:43 +0100, Yann E. MORIN wrote:

> > +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> > +endif  
> 
> So, all packages will now depend on this, even those that are not
> compressed with xz?

Yes, if there is no xzcat available on the system. This is exactly what
the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
not intend to solve this problem, which is a separate one.

I do believe however that by moving those dependencies (host-xz and
host-lzip) as proper package dependencies, it will be easier to have
follow-up patches that add the dependency only if lzip or xz are
needed. But again, my series does not intend to fix that, it's a
separate problem.

> Currently, filesystems that want to compress with xz all depend on
> host-xz unconditioanlly. Do we want to change that as well, so they only
> depend on it if needed?

I don't understand this part, because filesystems do not depend on
host-xz unconditionally. They only depend on it if xz compression of
the filesystem image is requested. From fs/common.mk:

ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
ROOTFS_$(2)_DEPENDENCIES += host-xz
ROOTFS_$(2)_COMPRESS_EXT = .xz
ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
endif

Best regards,

Thomas Petazzoni
Yann E. MORIN Dec. 3, 2017, 8:32 a.m. | #3
On 2017-12-02 21:16 +0100, Thomas Petazzoni spake thusly:
> Hello,
> 
> On Sat, 2 Dec 2017 16:08:43 +0100, Yann E. MORIN wrote:
> 
> > > +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> > > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> > > +endif  
> > 
> > So, all packages will now depend on this, even those that are not
> > compressed with xz?
> 
> Yes, if there is no xzcat available on the system. This is exactly what
> the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
> not intend to solve this problem, which is a separate one.

I somehow interpreted your patches in my head as "only build them when
needed". ;-)

> I do believe however that by moving those dependencies (host-xz and
> host-lzip) as proper package dependencies, it will be easier to have
> follow-up patches that add the dependency only if lzip or xz are
> needed. But again, my series does not intend to fix that, it's a
> separate problem.

OK, agreed.

> > Currently, filesystems that want to compress with xz all depend on
> > host-xz unconditioanlly. Do we want to change that as well, so they only
> > depend on it if needed?
> 
> I don't understand this part, because filesystems do not depend on
> host-xz unconditionally. They only depend on it if xz compression of
> the filesystem image is requested. From fs/common.mk:
> 
> ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> ROOTFS_$(2)_DEPENDENCIES += host-xz
> ROOTFS_$(2)_COMPRESS_EXT = .xz
> ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> endif

Well, first, let's assume that when the host has a suitable xzcat, it
also has the xz command as well; I think it's not too far-fetched an
assumption.

Then, we have BR2_XZCAT_HOST_DEPENDENCY empty, because we do have a
suitable xzcat, and thus a suitable xz. Yet, when the user requires
their filesystem to be xz-compressed, we still forcefully depend on
host-xz, even though we do have a suitable xz command.

What I mean is that, probably, we will want to make it:

    ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas Petazzoni
> -- 
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
Thomas Petazzoni Dec. 3, 2017, 9:29 a.m. | #4
Hello,

On Sun, 3 Dec 2017 09:32:56 +0100, Yann E. MORIN wrote:

> > Yes, if there is no xzcat available on the system. This is exactly what
> > the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
> > not intend to solve this problem, which is a separate one.  
> 
> I somehow interpreted your patches in my head as "only build them when
> needed". ;-)

Nope, I'm not fixing that (separate) problem. I still want to fix this
problem as well, but not as part of the per-package SDK stuff.

> > > Currently, filesystems that want to compress with xz all depend on
> > > host-xz unconditioanlly. Do we want to change that as well, so they only
> > > depend on it if needed?  
> > 
> > I don't understand this part, because filesystems do not depend on
> > host-xz unconditionally. They only depend on it if xz compression of
> > the filesystem image is requested. From fs/common.mk:
> > 
> > ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> > ROOTFS_$(2)_DEPENDENCIES += host-xz
> > ROOTFS_$(2)_COMPRESS_EXT = .xz
> > ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> > endif  
> 
> Well, first, let's assume that when the host has a suitable xzcat, it
> also has the xz command as well; I think it's not too far-fetched an
> assumption.
> 
> Then, we have BR2_XZCAT_HOST_DEPENDENCY empty, because we do have a
> suitable xzcat, and thus a suitable xz. Yet, when the user requires
> their filesystem to be xz-compressed, we still forcefully depend on
> host-xz, even though we do have a suitable xz command.
> 
> What I mean is that, probably, we will want to make it:
> 
>     ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)

Yes, I thought about this too. But again, it's a separate issue: the
filesystem code already depends on host-xz, regardless of whether xz is
already available on the system.

Thomas
Yann E. MORIN Dec. 3, 2017, 9:34 a.m. | #5
Thomas, All,

On 2017-12-03 10:29 +0100, Thomas Petazzoni spake thusly:
[--SNIP--]
> > > > Currently, filesystems that want to compress with xz all depend on
> > > > host-xz unconditioanlly. Do we want to change that as well, so they only
> > > > depend on it if needed?  
> > > 
> > > I don't understand this part, because filesystems do not depend on
> > > host-xz unconditionally. They only depend on it if xz compression of
> > > the filesystem image is requested. From fs/common.mk:
> > > 
> > > ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> > > ROOTFS_$(2)_DEPENDENCIES += host-xz
> > > ROOTFS_$(2)_COMPRESS_EXT = .xz
> > > ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> > > endif  
[--SNIP--]
> > What I mean is that, probably, we will want to make it:
> > 
> >     ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
> 
> Yes, I thought about this too. But again, it's a separate issue: the
> filesystem code already depends on host-xz, regardless of whether xz is
> already available on the system.

Yes, sure. I was just merely saying that when you eventually make those
dependencies really when-needed, you should also have a look at the
filesystem stuff. ;-)

Regards,
Yann E. MORIN.
Yann E. MORIN Dec. 29, 2017, 3:36 p.m. | #6
Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:
> 
>  - host-xz, because we would otherwise depend on ourself.
> 
>  - host-tar, because xz itself is delivered as a tarball, so we need
>    to have host-xz depend on host-tar, and not host-tar depend on
>    host-xz
> 
>  - host-skeleton, because we need to have host-xz depend on
>    host-skeleton, and not the opposite.
> 
> In addition, we modify xz.mk to explicitly build host-xz without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

My prior questions have been answered, so:

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - New patch
> ---
>  package/pkg-generic.mk                   | 4 ++++
>  package/xz/xz.mk                         | 5 +++++
>  support/dependencies/check-host-xzcat.mk | 2 +-
>  3 files changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 86dd2561d4..dfd4719d29 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,6 +591,10 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/package/xz/xz.mk b/package/xz/xz.mk
> index e8116f24ae..bcdac13ee3 100644
> --- a/package/xz/xz.mk
> +++ b/package/xz/xz.mk
> @@ -18,5 +18,10 @@ else
>  XZ_CONF_OPTS = --disable-threads
>  endif
>  
> +# we are built before ccache
> +HOST_XZ_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index 9be75c7311..e5d72f95dd 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -2,7 +2,7 @@
>  # If it is not present, build our own host-xzcat
>  
>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-xz
> +BR2_XZCAT_HOST_DEPENDENCY = host-xz
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>  XZCAT = $(HOST_DIR)/bin/xzcat
>  endif
> -- 
> 2.13.6
>

Patch

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 86dd2561d4..dfd4719d29 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -591,6 +591,10 @@  ifeq ($(filter host-tar host-skeleton,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
 endif
 
+ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
+$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
diff --git a/package/xz/xz.mk b/package/xz/xz.mk
index e8116f24ae..bcdac13ee3 100644
--- a/package/xz/xz.mk
+++ b/package/xz/xz.mk
@@ -18,5 +18,10 @@  else
 XZ_CONF_OPTS = --disable-threads
 endif
 
+# we are built before ccache
+HOST_XZ_CONF_ENV = \
+	CC="$(HOSTCC_NOCCACHE)" \
+	CXX="$(HOSTCXX_NOCCACHE)"
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
index 9be75c7311..e5d72f95dd 100644
--- a/support/dependencies/check-host-xzcat.mk
+++ b/support/dependencies/check-host-xzcat.mk
@@ -2,7 +2,7 @@ 
 # If it is not present, build our own host-xzcat
 
 ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
-DEPENDENCIES_HOST_PREREQ += host-xz
+BR2_XZCAT_HOST_DEPENDENCY = host-xz
 EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
 XZCAT = $(HOST_DIR)/bin/xzcat
 endif