Patchwork [3/3] initramfs: add support for LZO and XZ compression methods

login
register
mail settings
Submitter Gustavo Zacarias
Date Jan. 21, 2013, 10:55 p.m.
Message ID <1358808943-17063-3-git-send-email-gustavo@zacarias.com.ar>
Download mbox | patch
Permalink /patch/214293/
State Changes Requested
Headers show

Comments

Gustavo Zacarias - Jan. 21, 2013, 10:55 p.m.
Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 fs/common.mk      |  8 +++++++-
 fs/cpio/Config.in | 18 ++++++++++++++----
 2 files changed, 21 insertions(+), 5 deletions(-)
Stefan Fröberg - Jan. 21, 2013, 11:38 p.m.
Hi Gustavo

22.1.2013 0:55, Gustavo Zacarias kirjoitti:
> Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> ---
>  fs/common.mk      |  8 +++++++-
>  fs/cpio/Config.in | 18 ++++++++++++++----
>  2 files changed, 21 insertions(+), 5 deletions(-)
>
> diff --git a/fs/common.mk b/fs/common.mk
> index 8b5b2f2..0c171fc 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \
>  define ROOTFS_TARGET_INTERNAL
>  
>  # extra deps
> -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma)
> +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>  
>  $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
>  	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
> @@ -70,6 +70,12 @@ endif
>  ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y)
>  	$$(LZMA) -9 -c $$@ > $$@.lzma
>  endif
> +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y)
> +	$$(LZOP) -9 -c $$@ > $$@.lzo
> +endif
> +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> +	$$(XZ) -9 -c $$@ > $$@.xz
> +endif

You will probably need --check=crc32 with that xz-compression.
I have had difficulties booting linux 3.3.1 kernel (don't know about
later versions)
with xz-compressed initramfs without that option.

Regards
Stefan
Gustavo Zacarias - Jan. 21, 2013, 11:44 p.m.
On 01/21/2013 08:38 PM, Stefan Fröberg wrote:

> You will probably need --check=crc32 with that xz-compression.
> I have had difficulties booting linux 3.3.1 kernel (don't know about
> later versions)
> with xz-compressed initramfs without that option.

You are 100% correct.
It didn't fail me during testing because buildroot is never using any
compressed cpio archive, thus the previous options (gzip, bzip2) were
never used.
I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig
test run that it never changed much.
New patchset will follow that fixes this and the previous issue.
Regards.
Stefan Fröberg - Jan. 22, 2013, 12:34 a.m.
22.1.2013 1:44, Gustavo Zacarias kirjoitti:
> On 01/21/2013 08:38 PM, Stefan Fröberg wrote:
>
>> You will probably need --check=crc32 with that xz-compression.
>> I have had difficulties booting linux 3.3.1 kernel (don't know about
>> later versions)
>> with xz-compressed initramfs without that option.
> You are 100% correct.
> It didn't fail me during testing because buildroot is never using any
> compressed cpio archive, thus the previous options (gzip, bzip2) were
> never used.

Yeah.
It really doesn't make sense to compress if the initramfs file in
question is embedded directly into
kernel (which at least version 2012.05 and earlier buildroot did)
because usually the kernel
itself would be also compressed. So in that specific case doing double
compression
(once for the initramfs file embedded into kernel and then again for the
whole kernel+initramfs thing)
would be waste of time.

But for initramfs files *outside* kernel these two new compression
methods are very usefull.

lzo being just barely better in compression than gzip but fastest
decompression of all,
while xz being the best compression.
(So lzo replaces gzip and xz replaces bzip2 and lzma)

I use xz for kernel itself and also for outside initramfs

> I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig
> test run that it never changed much.
> New patchset will follow that fixes this and the previous issue.
> Regards.

Stefan
Thomas Petazzoni - Jan. 22, 2013, 5:05 p.m.
Dear Gustavo Zacarias,

On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote:

> +config BR2_TARGET_ROOTFS_CPIO_LZO
> +	bool "lzo"
> +	help
> +	  Do compress the cpio filesystem with lzo.
> +
> +config BR2_TARGET_ROOTFS_CPIO_XZ
> +	bool "xz"
> +	help
> +	  Do compress the cpio filesystem with xz.

I don't remember exactly, but isn't the kernel build process itself
capable of compressing the initramfs already? I remember we had some
discussion about this a while ago, but I don't remember the outcome of
these. Arnout was leading this discussion, IIRC.

Best regards,

Thomas
Stefan Fröberg - Jan. 22, 2013, 6:17 p.m.
22.1.2013 19:05, Thomas Petazzoni kirjoitti:
> Dear Gustavo Zacarias,
>
> On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote:
>
>> +config BR2_TARGET_ROOTFS_CPIO_LZO
>> +	bool "lzo"
>> +	help
>> +	  Do compress the cpio filesystem with lzo.
>> +
>> +config BR2_TARGET_ROOTFS_CPIO_XZ
>> +	bool "xz"
>> +	help
>> +	  Do compress the cpio filesystem with xz.
> I don't remember exactly, but isn't the kernel build process itself
> capable of compressing the initramfs already? I remember we had some
> discussion about this a while ago, but I don't remember the outcome of
> these. Arnout was leading this discussion, IIRC.

If it's embedded initramfs then yes, kernel can handle it all itself
just nicely.
without needing to compress inside initramfs.
(images kernel.png and kernel2.png)

But if outside cpio-archive (aka initramfs) then no. It has to be done
by hand.
And like I said to Gustavo it's bad to do double compression.


Here's are the different combinations:

1 ) compressed initramfs  inside  compressed kernel = bad (waste of time
decompressing twice, and also waste of package file header size overhead)
2 ) compressed initramfs inside uncompressed kernel = better (no time
waste but still space waste)
3 ) uncompressed initramfs inside compressed kernel = best solution for
embedded initramfs (no time waste & time space waste)

4 ) uncompressed outside initramfs and uncompressed kernel (waste of
space and even according to some benchmarks, waste of time)
5 ) compressed outside initramfs and compressed kernel (no wasted time
and no wasted space)

So option 3 is best for embedded initramfs and option 5 for otherwise
for those who want more flexibility with their enivronment.
For example I could have this:

   
-------------------------------------------------------------------------------------------------------------------------------------------------
   |  embedded, uncompressed initramfs insize xz-compressed kernel, with
just busybox and minimal stuff |
  
--------------------------------------------------------------------------------------------------------------------------------------------------
                                                              |
  
---------------------------------------------------------------------------------------------------------------------------------
   | compressed outside initramfs with ncurses & directfb application,
like links web-browser    |
  
---------------------------------------------------------------------------------------------------------------------------------
                                                              |
 
----------------------------------------------------------------------------------------------------------------------------------
  | still another compressed initramfs with Xorg, firefox and all the
heavy stuff                             |
 
----------------------------------------------------------------------------------------------------------------------------------

All stackable one top over another (at least grub legacy allows giving
several initramfs).

And in Grub conf I could have boot entries like:
- Minimal system (just busybox and it's apps)
- Light system (ncurses, directfb and other non-Xorg "graphical" apps)
- Full system  (full Xorg stuff and apps)

Without outside initramfs option I would have to stick kernel (and it's
modules) to each of those initramfs files which would be a waste of space.
Now I have only one kernel (with or without initramfs) and 2 outside
initramfs which I can mix and match like lego blocks.

Stefan














> Best regards,
>
> Thomas
Arnout Vandecappelle - Jan. 24, 2013, 7:09 a.m.
On 01/21/13 23:55, Gustavo Zacarias wrote:
> Signed-off-by: Gustavo Zacarias<gustavo@zacarias.com.ar>
> ---
>   fs/common.mk      |  8 +++++++-
>   fs/cpio/Config.in | 18 ++++++++++++++----
>   2 files changed, 21 insertions(+), 5 deletions(-)
>
> diff --git a/fs/common.mk b/fs/common.mk
> index 8b5b2f2..0c171fc 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \
>   define ROOTFS_TARGET_INTERNAL
>
>   # extra deps
> -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma)
> +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)

  Gustavo,

  If you were to submit a v4, could you split this very long line?

  Regards,
  Arnout
Arnout Vandecappelle - Jan. 24, 2013, 7:11 a.m.
On 01/22/13 19:17, Stefan Fröberg wrote:
[snip]
> Here's are the different combinations:

  Excellent explanation! How about adding this to the documentation?

  Regards,
  Arnout
Gustavo Zacarias - Jan. 24, 2013, 10:48 a.m.
On 01/22/2013 03:17 PM, Stefan Fröberg wrote:
>> I don't remember exactly, but isn't the kernel build process itself
>> capable of compressing the initramfs already? I remember we had some
>> discussion about this a while ago, but I don't remember the outcome of
>> these. Arnout was leading this discussion, IIRC.
> 
> If it's embedded initramfs then yes, kernel can handle it all itself
> just nicely.
> without needing to compress inside initramfs.
> (images kernel.png and kernel2.png)
> 
> But if outside cpio-archive (aka initramfs) then no. It has to be done
> by hand.
> And like I said to Gustavo it's bad to do double compression.

Well, it's not always bad to do double compression.
On some systems where touching the bootloader isn't an option with say,
uboot, if uboot doesn't understand the newer compression method you're
most likely stuck with gzip or none for the uImage.
So doing xz for the initramfs in that case would be a space saver - even
if you're doing double (de)compression.
Granted, you'll pay it dearly in cpu time.
I started looking at this because someone in #elinux started asking
about making the tinyest possible kernel+initramfs for x86 for pxeboot
with some packages and i saw we were lacking options to go for the best.
Also noticed we weren't doing anything about the compressed cpio archive
so the resulting combined kernel was roughly always the same size.
For some unknown reason he wanted everything to fit in a 1.8 MB file -
maybe some PXE limitation, i don't know.
Question is, should we care about those uses cases or not?
Regards.
Gustavo Zacarias - Jan. 24, 2013, 10:57 a.m.
On 01/24/2013 04:09 AM, Arnout Vandecappelle wrote:

>  Gustavo,
> 
>  If you were to submit a v4, could you split this very long line?
> 
>  Regards,
>  Arnout

v4? There's not even a v3 yet! ;-)
There are many fixable lines inside fs/ that are easily wrappable.
For example in ubifs/ubifs.mk, cloop/cloop.mk, initramfs/initramfs.mk
(if we're going for the strict 80 column standard) and others.
But yes, on a new patchset if necessary i could.
Regards.
Stefan Fröberg - Jan. 24, 2013, 11:55 a.m.
24.1.2013 9:11, Arnout Vandecappelle kirjoitti:
> On 01/22/13 19:17, Stefan Fröberg wrote:
> [snip]
>> Here's are the different combinations:
>
>  Excellent explanation! How about adding this to the documentation?
>
>  Regards,
>  Arnout

Well, uh...ok.
But before that I have to finish that <pkg>_CONFIG_FIXUP thing first.
I think ill just make one final, v3 version, of it with fixed doc and
your sed changes
as one single combined patch.
And then Im going to test it one final time before resubmit.


Regards
Stefan
Stefan Fröberg - Jan. 24, 2013, 12:08 p.m.
24.1.2013 12:48, Gustavo Zacarias kirjoitti:
> On 01/22/2013 03:17 PM, Stefan Fröberg wrote:
>>> I don't remember exactly, but isn't the kernel build process itself
>>> capable of compressing the initramfs already? I remember we had some
>>> discussion about this a while ago, but I don't remember the outcome of
>>> these. Arnout was leading this discussion, IIRC.
>> If it's embedded initramfs then yes, kernel can handle it all itself
>> just nicely.
>> without needing to compress inside initramfs.
>> (images kernel.png and kernel2.png)
>>
>> But if outside cpio-archive (aka initramfs) then no. It has to be done
>> by hand.
>> And like I said to Gustavo it's bad to do double compression.
> Well, it's not always bad to do double compression.
> On some systems where touching the bootloader isn't an option with say,
> uboot, if uboot doesn't understand the newer compression method you're
> most likely stuck with gzip or none for the uImage.
> So doing xz for the initramfs in that case would be a space saver - even
> if you're doing double (de)compression.
> Granted, you'll pay it dearly in cpu time.
> I started looking at this because someone in #elinux started asking
> about making the tinyest possible kernel+initramfs for x86 for pxeboot
> with some packages and i saw we were lacking options to go for the best.
> Also noticed we weren't doing anything about the compressed cpio archive
> so the resulting combined kernel was roughly always the same size.
> For some unknown reason he wanted everything to fit in a 1.8 MB file -
> maybe some PXE limitation, i don't know.
> Question is, should we care about those uses cases or not?
> Regards.

I think users should be allowed to do every possible combination they
want to.
But also give them big fat warning in cases when their choices might
drop performance
so that they are perfectly aware of consequences of the choices they
made in config.
So yeah, double (de)compression is option too but with big fat warning
added that might
affect your system speed.

User is the King.

Stefan
Gustavo Zacarias - Jan. 24, 2013, 12:13 p.m.
On 01/24/2013 09:08 AM, Stefan Fröberg wrote:

> I think users should be allowed to do every possible combination they
> want to.
> But also give them big fat warning in cases when their choices might
> drop performance
> so that they are perfectly aware of consequences of the choices they
> made in config.
> So yeah, double (de)compression is option too but with big fat warning
> added that might
> affect your system speed.
> 
> User is the King.

At this point my patchset (v2) does nothing to address the issue, so
it's safe to commit.
I was leaving that functionality for a separate one afterwards if the
consensus is "go for it".
Regards.

Patch

diff --git a/fs/common.mk b/fs/common.mk
index 8b5b2f2..0c171fc 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -39,7 +39,7 @@  ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \
 define ROOTFS_TARGET_INTERNAL
 
 # extra deps
-ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma)
+ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
 
 $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
@@ -70,6 +70,12 @@  endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y)
 	$$(LZMA) -9 -c $$@ > $$@.lzma
 endif
+ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y)
+	$$(LZOP) -9 -c $$@ > $$@.lzo
+endif
+ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
+	$$(XZ) -9 -c $$@ > $$@.xz
+endif
 
 rootfs-$(1)-show-depends:
 	@echo $$(ROOTFS_$(2)_DEPENDENCIES)
diff --git a/fs/cpio/Config.in b/fs/cpio/Config.in
index 0669a44..d9e1733 100644
--- a/fs/cpio/Config.in
+++ b/fs/cpio/Config.in
@@ -18,22 +18,32 @@  choice
 config BR2_TARGET_ROOTFS_CPIO_NONE
         bool "no compression"
         help
-         Do not compress the cpio filesystem.
+          Do not compress the cpio filesystem.
 
 config BR2_TARGET_ROOTFS_CPIO_GZIP
         bool "gzip"
         help
-         Do compress the cpio filesystem with gzip.
+          Do compress the cpio filesystem with gzip.
 
 config BR2_TARGET_ROOTFS_CPIO_BZIP2
         bool "bzip2"
         help
-         Do compress the cpio filesystem with bzip2.
+          Do compress the cpio filesystem with bzip2.
 
 config BR2_TARGET_ROOTFS_CPIO_LZMA
         bool "lzma"
         help
-         Do compress the cpio filesystem with lzma.
+          Do compress the cpio filesystem with lzma.
+
+config BR2_TARGET_ROOTFS_CPIO_LZO
+	bool "lzo"
+	help
+	  Do compress the cpio filesystem with lzo.
+
+config BR2_TARGET_ROOTFS_CPIO_XZ
+	bool "xz"
+	help
+	  Do compress the cpio filesystem with xz.
 
 endchoice