| Message ID | 1358808943-17063-3-git-send-email-gustavo@zacarias.com.ar |
|---|---|
| State | Changes Requested |
| Headers | show |
Hi Gustavo 22.1.2013 0:55, Gustavo Zacarias kirjoitti: > Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar> > --- > fs/common.mk | 8 +++++++- > fs/cpio/Config.in | 18 ++++++++++++++---- > 2 files changed, 21 insertions(+), 5 deletions(-) > > diff --git a/fs/common.mk b/fs/common.mk > index 8b5b2f2..0c171fc 100644 > --- a/fs/common.mk > +++ b/fs/common.mk > @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \ > define ROOTFS_TARGET_INTERNAL > > # extra deps > -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) > +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) > > $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) > @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") > @@ -70,6 +70,12 @@ endif > ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y) > $$(LZMA) -9 -c $$@ > $$@.lzma > endif > +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y) > + $$(LZOP) -9 -c $$@ > $$@.lzo > +endif > +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y) > + $$(XZ) -9 -c $$@ > $$@.xz > +endif You will probably need --check=crc32 with that xz-compression. I have had difficulties booting linux 3.3.1 kernel (don't know about later versions) with xz-compressed initramfs without that option. Regards Stefan
On 01/21/2013 08:38 PM, Stefan Fröberg wrote: > You will probably need --check=crc32 with that xz-compression. > I have had difficulties booting linux 3.3.1 kernel (don't know about > later versions) > with xz-compressed initramfs without that option. You are 100% correct. It didn't fail me during testing because buildroot is never using any compressed cpio archive, thus the previous options (gzip, bzip2) were never used. I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig test run that it never changed much. New patchset will follow that fixes this and the previous issue. Regards.
22.1.2013 1:44, Gustavo Zacarias kirjoitti: > On 01/21/2013 08:38 PM, Stefan Fröberg wrote: > >> You will probably need --check=crc32 with that xz-compression. >> I have had difficulties booting linux 3.3.1 kernel (don't know about >> later versions) >> with xz-compressed initramfs without that option. > You are 100% correct. > It didn't fail me during testing because buildroot is never using any > compressed cpio archive, thus the previous options (gzip, bzip2) were > never used. Yeah. It really doesn't make sense to compress if the initramfs file in question is embedded directly into kernel (which at least version 2012.05 and earlier buildroot did) because usually the kernel itself would be also compressed. So in that specific case doing double compression (once for the initramfs file embedded into kernel and then again for the whole kernel+initramfs thing) would be waste of time. But for initramfs files *outside* kernel these two new compression methods are very usefull. lzo being just barely better in compression than gzip but fastest decompression of all, while xz being the best compression. (So lzo replaces gzip and xz replaces bzip2 and lzma) I use xz for kernel itself and also for outside initramfs > I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig > test run that it never changed much. > New patchset will follow that fixes this and the previous issue. > Regards. Stefan
Dear Gustavo Zacarias, On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote: > +config BR2_TARGET_ROOTFS_CPIO_LZO > + bool "lzo" > + help > + Do compress the cpio filesystem with lzo. > + > +config BR2_TARGET_ROOTFS_CPIO_XZ > + bool "xz" > + help > + Do compress the cpio filesystem with xz. I don't remember exactly, but isn't the kernel build process itself capable of compressing the initramfs already? I remember we had some discussion about this a while ago, but I don't remember the outcome of these. Arnout was leading this discussion, IIRC. Best regards, Thomas
22.1.2013 19:05, Thomas Petazzoni kirjoitti: > Dear Gustavo Zacarias, > > On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote: > >> +config BR2_TARGET_ROOTFS_CPIO_LZO >> + bool "lzo" >> + help >> + Do compress the cpio filesystem with lzo. >> + >> +config BR2_TARGET_ROOTFS_CPIO_XZ >> + bool "xz" >> + help >> + Do compress the cpio filesystem with xz. > I don't remember exactly, but isn't the kernel build process itself > capable of compressing the initramfs already? I remember we had some > discussion about this a while ago, but I don't remember the outcome of > these. Arnout was leading this discussion, IIRC. If it's embedded initramfs then yes, kernel can handle it all itself just nicely. without needing to compress inside initramfs. (images kernel.png and kernel2.png) But if outside cpio-archive (aka initramfs) then no. It has to be done by hand. And like I said to Gustavo it's bad to do double compression. Here's are the different combinations: 1 ) compressed initramfs inside compressed kernel = bad (waste of time decompressing twice, and also waste of package file header size overhead) 2 ) compressed initramfs inside uncompressed kernel = better (no time waste but still space waste) 3 ) uncompressed initramfs inside compressed kernel = best solution for embedded initramfs (no time waste & time space waste) 4 ) uncompressed outside initramfs and uncompressed kernel (waste of space and even according to some benchmarks, waste of time) 5 ) compressed outside initramfs and compressed kernel (no wasted time and no wasted space) So option 3 is best for embedded initramfs and option 5 for otherwise for those who want more flexibility with their enivronment. For example I could have this: ------------------------------------------------------------------------------------------------------------------------------------------------- | embedded, uncompressed initramfs insize xz-compressed kernel, with just busybox and minimal stuff | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | compressed outside initramfs with ncurses & directfb application, like links web-browser | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | still another compressed initramfs with Xorg, firefox and all the heavy stuff | ---------------------------------------------------------------------------------------------------------------------------------- All stackable one top over another (at least grub legacy allows giving several initramfs). And in Grub conf I could have boot entries like: - Minimal system (just busybox and it's apps) - Light system (ncurses, directfb and other non-Xorg "graphical" apps) - Full system (full Xorg stuff and apps) Without outside initramfs option I would have to stick kernel (and it's modules) to each of those initramfs files which would be a waste of space. Now I have only one kernel (with or without initramfs) and 2 outside initramfs which I can mix and match like lego blocks. Stefan > Best regards, > > Thomas
On 01/21/13 23:55, Gustavo Zacarias wrote: > Signed-off-by: Gustavo Zacarias<gustavo@zacarias.com.ar> > --- > fs/common.mk | 8 +++++++- > fs/cpio/Config.in | 18 ++++++++++++++---- > 2 files changed, 21 insertions(+), 5 deletions(-) > > diff --git a/fs/common.mk b/fs/common.mk > index 8b5b2f2..0c171fc 100644 > --- a/fs/common.mk > +++ b/fs/common.mk > @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \ > define ROOTFS_TARGET_INTERNAL > > # extra deps > -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) > +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) Gustavo, If you were to submit a v4, could you split this very long line? Regards, Arnout
On 01/22/13 19:17, Stefan Fröberg wrote:
[snip]
> Here's are the different combinations:
Excellent explanation! How about adding this to the documentation?
Regards,
Arnout
On 01/22/2013 03:17 PM, Stefan Fröberg wrote: >> I don't remember exactly, but isn't the kernel build process itself >> capable of compressing the initramfs already? I remember we had some >> discussion about this a while ago, but I don't remember the outcome of >> these. Arnout was leading this discussion, IIRC. > > If it's embedded initramfs then yes, kernel can handle it all itself > just nicely. > without needing to compress inside initramfs. > (images kernel.png and kernel2.png) > > But if outside cpio-archive (aka initramfs) then no. It has to be done > by hand. > And like I said to Gustavo it's bad to do double compression. Well, it's not always bad to do double compression. On some systems where touching the bootloader isn't an option with say, uboot, if uboot doesn't understand the newer compression method you're most likely stuck with gzip or none for the uImage. So doing xz for the initramfs in that case would be a space saver - even if you're doing double (de)compression. Granted, you'll pay it dearly in cpu time. I started looking at this because someone in #elinux started asking about making the tinyest possible kernel+initramfs for x86 for pxeboot with some packages and i saw we were lacking options to go for the best. Also noticed we weren't doing anything about the compressed cpio archive so the resulting combined kernel was roughly always the same size. For some unknown reason he wanted everything to fit in a 1.8 MB file - maybe some PXE limitation, i don't know. Question is, should we care about those uses cases or not? Regards.
On 01/24/2013 04:09 AM, Arnout Vandecappelle wrote: > Gustavo, > > If you were to submit a v4, could you split this very long line? > > Regards, > Arnout v4? There's not even a v3 yet! ;-) There are many fixable lines inside fs/ that are easily wrappable. For example in ubifs/ubifs.mk, cloop/cloop.mk, initramfs/initramfs.mk (if we're going for the strict 80 column standard) and others. But yes, on a new patchset if necessary i could. Regards.
24.1.2013 9:11, Arnout Vandecappelle kirjoitti: > On 01/22/13 19:17, Stefan Fröberg wrote: > [snip] >> Here's are the different combinations: > > Excellent explanation! How about adding this to the documentation? > > Regards, > Arnout Well, uh...ok. But before that I have to finish that <pkg>_CONFIG_FIXUP thing first. I think ill just make one final, v3 version, of it with fixed doc and your sed changes as one single combined patch. And then Im going to test it one final time before resubmit. Regards Stefan
24.1.2013 12:48, Gustavo Zacarias kirjoitti: > On 01/22/2013 03:17 PM, Stefan Fröberg wrote: >>> I don't remember exactly, but isn't the kernel build process itself >>> capable of compressing the initramfs already? I remember we had some >>> discussion about this a while ago, but I don't remember the outcome of >>> these. Arnout was leading this discussion, IIRC. >> If it's embedded initramfs then yes, kernel can handle it all itself >> just nicely. >> without needing to compress inside initramfs. >> (images kernel.png and kernel2.png) >> >> But if outside cpio-archive (aka initramfs) then no. It has to be done >> by hand. >> And like I said to Gustavo it's bad to do double compression. > Well, it's not always bad to do double compression. > On some systems where touching the bootloader isn't an option with say, > uboot, if uboot doesn't understand the newer compression method you're > most likely stuck with gzip or none for the uImage. > So doing xz for the initramfs in that case would be a space saver - even > if you're doing double (de)compression. > Granted, you'll pay it dearly in cpu time. > I started looking at this because someone in #elinux started asking > about making the tinyest possible kernel+initramfs for x86 for pxeboot > with some packages and i saw we were lacking options to go for the best. > Also noticed we weren't doing anything about the compressed cpio archive > so the resulting combined kernel was roughly always the same size. > For some unknown reason he wanted everything to fit in a 1.8 MB file - > maybe some PXE limitation, i don't know. > Question is, should we care about those uses cases or not? > Regards. I think users should be allowed to do every possible combination they want to. But also give them big fat warning in cases when their choices might drop performance so that they are perfectly aware of consequences of the choices they made in config. So yeah, double (de)compression is option too but with big fat warning added that might affect your system speed. User is the King. Stefan
On 01/24/2013 09:08 AM, Stefan Fröberg wrote: > I think users should be allowed to do every possible combination they > want to. > But also give them big fat warning in cases when their choices might > drop performance > so that they are perfectly aware of consequences of the choices they > made in config. > So yeah, double (de)compression is option too but with big fat warning > added that might > affect your system speed. > > User is the King. At this point my patchset (v2) does nothing to address the issue, so it's safe to commit. I was leaving that functionality for a separate one afterwards if the consensus is "go for it". Regards.
diff --git a/fs/common.mk b/fs/common.mk index 8b5b2f2..0c171fc 100644 --- a/fs/common.mk +++ b/fs/common.mk @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \ define ROOTFS_TARGET_INTERNAL # extra deps -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") @@ -70,6 +70,12 @@ endif ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y) $$(LZMA) -9 -c $$@ > $$@.lzma endif +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y) + $$(LZOP) -9 -c $$@ > $$@.lzo +endif +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y) + $$(XZ) -9 -c $$@ > $$@.xz +endif rootfs-$(1)-show-depends: @echo $$(ROOTFS_$(2)_DEPENDENCIES) diff --git a/fs/cpio/Config.in b/fs/cpio/Config.in index 0669a44..d9e1733 100644 --- a/fs/cpio/Config.in +++ b/fs/cpio/Config.in @@ -18,22 +18,32 @@ choice config BR2_TARGET_ROOTFS_CPIO_NONE bool "no compression" help - Do not compress the cpio filesystem. + Do not compress the cpio filesystem. config BR2_TARGET_ROOTFS_CPIO_GZIP bool "gzip" help - Do compress the cpio filesystem with gzip. + Do compress the cpio filesystem with gzip. config BR2_TARGET_ROOTFS_CPIO_BZIP2 bool "bzip2" help - Do compress the cpio filesystem with bzip2. + Do compress the cpio filesystem with bzip2. config BR2_TARGET_ROOTFS_CPIO_LZMA bool "lzma" help - Do compress the cpio filesystem with lzma. + Do compress the cpio filesystem with lzma. + +config BR2_TARGET_ROOTFS_CPIO_LZO + bool "lzo" + help + Do compress the cpio filesystem with lzo. + +config BR2_TARGET_ROOTFS_CPIO_XZ + bool "xz" + help + Do compress the cpio filesystem with xz. endchoice
Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar> --- fs/common.mk | 8 +++++++- fs/cpio/Config.in | 18 ++++++++++++++---- 2 files changed, 21 insertions(+), 5 deletions(-)