fs: allow extra arguments to common tarball extraction

Message ID 20180603022145.14222-1-casantos@datacom.com.br
State New
Headers show
Series
  • fs: allow extra arguments to common tarball extraction
Related show

Commit Message

Carlos Santos June 3, 2018, 2:21 a.m.
Since commit 118534fe54b (fs: use a common tarball as base for the other
filesystems) Buildroot creates a .tar filesystem image and re-extracts
it in a private directory to create each format-specific image. Add an
option to pass extra arguments to tar when that commom root image is
extracted.

This option is useful when the root filesystem is volatile (e.g. initrd)
or read-only but a read-write subtree is still necessary for persistent
data modified by programs as they run.

For example, one can pass "--exclude='./var/lib/*'" to exclude that path
from the rootfs image and use a post-fakeroot script to make a separate
filesystem image for /var/lib.

Signed-off-by: Carlos Santos <casantos@datacom.com.br>
---
Additional explanation

This change solves a real-life problem of my current project. Before
buildroot 2018.05 I had to use a trickier approach:

1. Add a post-fakeroot script to
   1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib.
   1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR).
   1.3. Create an empty $(TARGET_DIR)/var/lib.
2. Add a post-image script to
   2.1. Copy the rootfs and /var/lib images to the disk image.
   2.2. Remove the empty $(TARGET_DIR)/var/lib.
   2.3. Restore $(TARGET_DIR)/var/lib from the backup path.

That solution is fragile: if something goes wrong during the execution
of the post-image script it can lead to a cripled $(TARGET_DIR) with an
empty var/lib subdir. Rebuilding a package without restoring the backup
can easily cause a kaboom and force me to run a lengthy clean build.

I added some protection mechanisms to stop the build if a stale /var/lib
backup is found but quite frankly the whole thing stinks.

The solution with custom extraction arguments, OTOH, is bullet-proof
because it does not change $(TARGET_DIR), so it is always safe to run
dirty builds.
---
 docs/manual/customize-rootfs.txt | 10 ++++++++++
 fs/Config.in                     | 13 +++++++++++++
 fs/common.mk                     |  3 ++-
 3 files changed, 25 insertions(+), 1 deletion(-)

Comments

Yann E. MORIN June 5, 2018, 5:23 p.m. | #1
Carlos, All,

On 2018-06-02 23:21 -0300, Carlos Santos spake thusly:
> Since commit 118534fe54b (fs: use a common tarball as base for the other
> filesystems) Buildroot creates a .tar filesystem image and re-extracts
> it in a private directory to create each format-specific image. Add an
> option to pass extra arguments to tar when that commom root image is
> extracted.
> 
> This option is useful when the root filesystem is volatile (e.g. initrd)
> or read-only but a read-write subtree is still necessary for persistent
> data modified by programs as they run.
> 
> For example, one can pass "--exclude='./var/lib/*'" to exclude that path
> from the rootfs image and use a post-fakeroot script to make a separate
> filesystem image for /var/lib.

Well, I am not too fond of this extra option. But see below...

> Signed-off-by: Carlos Santos <casantos@datacom.com.br>
> ---
> Additional explanation
> 
> This change solves a real-life problem of my current project. Before
> buildroot 2018.05 I had to use a trickier approach:
> 
> 1. Add a post-fakeroot script to
>    1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib.
>    1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR).
>    1.3. Create an empty $(TARGET_DIR)/var/lib.
> 2. Add a post-image script to
>    2.1. Copy the rootfs and /var/lib images to the disk image.
>    2.2. Remove the empty $(TARGET_DIR)/var/lib.
>    2.3. Restore $(TARGET_DIR)/var/lib from the backup path.
> 
> That solution is fragile: if something goes wrong during the execution
> of the post-image script it can lead to a cripled $(TARGET_DIR) with an
> empty var/lib subdir. Rebuilding a package without restoring the backup
> can easily cause a kaboom and force me to run a lengthy clean build.

What prevents you from providing your own filesystem mechanism at all?

Since you have post-scripts, it means you do have an existing tree
where they are stored (even probably a git trre, at that).

So you could use that location as a br2-external, if that's not what
you already do. And then, you can add a new filesystem type in that
br2-external tree.

Jsut make your filesystem depend on rootfs-tar, then you can basically
do something like:

    $ cat my-mkfs-tool
    #!/bin/sh
    if [ $(id -u) -ne 0 ]; then
        exec fakeroot "${0}" "${@}"
    fi

    mktempdir && cd tempdir # You get the idea
    tar xf $(IMAGES_DIR)/rootfs.tar
    do-soemthing

And then you have a mechanism that cleanly and reliably fits in
Buildroot's filesystem infra.

I don't think the overhead would be big, since you anyway already have
to have some scripts doing the separation properly, so making those into
a filesystem is probably not too much, I guess.

For a (basic) example, see slide 16 (shameless self-reference):
    https://elinux.org/images/8/8e/2017-10-24_-_ELCE-Buildroot.pdf

(note: the macro to call has changed in master since that presentation,
but you get the idea.)

In the end, I believe this is more robust and more generic than this
extra option to the intermediate tarball, which is purely an internal
detail. We may tomorrow decide on another solution for the internal
state in the future...

My 2 cents...

Regards,
Yann E. MORIN.

> I added some protection mechanisms to stop the build if a stale /var/lib
> backup is found but quite frankly the whole thing stinks.
> 
> The solution with custom extraction arguments, OTOH, is bullet-proof
> because it does not change $(TARGET_DIR), so it is always safe to run
> dirty builds.
> ---
>  docs/manual/customize-rootfs.txt | 10 ++++++++++
>  fs/Config.in                     | 13 +++++++++++++
>  fs/common.mk                     |  3 ++-
>  3 files changed, 25 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt
> index bb6d8da6bf..6dd9365723 100644
> --- a/docs/manual/customize-rootfs.txt
> +++ b/docs/manual/customize-rootfs.txt
> @@ -121,6 +121,16 @@ This method is not recommended because it duplicates the entire
>    skeleton, which prevents taking advantage of the fixes or improvements
>    brought to the default skeleton in later Buildroot releases.
>  
> +Rootfs tarball extraction arguments (+BR2_ROOTFS_COMMON_UNTAR_ARGS+)::
> ++
> +Buildroot creates a .tar filesystem image and re-extracts it in a private
> + directory to create each format-specific image. Use this option to pass
> + extra arguments to tar when the commom root image is extracted.
> ++
> +For example, you can pass "--exclude=\'./var/lib/*\'" to exclude that path from
> + the rootfs image and use a post-fakeroot script (see below) to create a
> + separate filesystem image for '/var/lib'.
> +
>  Post-fakeroot scripts (+BR2_ROOTFS_POST_FAKEROOT_SCRIPT+)::
>  +
>  When aggregating the final images, some parts of the process requires
> diff --git a/fs/Config.in b/fs/Config.in
> index c25b01c3de..74487b1172 100644
> --- a/fs/Config.in
> +++ b/fs/Config.in
> @@ -1,5 +1,18 @@
>  menu "Filesystem images"
>  
> +config BR2_ROOTFS_COMMON_UNTAR_ARGS
> +	string "Rootfs tarball extraction arguments"
> +	help
> +	  Buildroot creates a .tar filesystem image and re-extracts it
> +	  in a private directory to create each format-specific image.
> +	  Use this option to pass extra arguments to tar when the commom
> +	  root image is extracted.
> +
> +	  For example, you can pass "--exclude='./var/lib/*'" to exclude
> +	  that path from the rootfs image and use a post-fakeroot script
> +	  (see BR2_ROOTFS_POST_FAKEROOT_SCRIPT) to create a separate
> +	  filesystem image for /var/lib.
> +
>  source "fs/axfs/Config.in"
>  source "fs/cloop/Config.in"
>  source "fs/cpio/Config.in"
> diff --git a/fs/common.mk b/fs/common.mk
> index abf35418cb..33ad1ab2a9 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -50,9 +50,10 @@ define ROOTFS_COMMON_TAR_CMD
>  endef
>  
>  # Command to extract the common tarball into the per-rootfs target directory
> +ROOTFS_COMMON_UNTAR_ARGS = $(call qstrip,$(BR2_ROOTFS_COMMON_UNTAR_ARGS))
>  define ROOTFS_COMMON_UNTAR_CMD
>  	mkdir -p $(TARGET_DIR)
> -	tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR)
> +	tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) $(ROOTFS_COMMON_UNTAR_ARGS)
>  endef
>  
>  .PHONY: rootfs-common
> -- 
> 2.17.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Carlos Santos June 6, 2018, 12:46 p.m. | #2
> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "DATACOM" <casantos@datacom.com.br>
> Cc: "buildroot" <buildroot@buildroot.org>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>
> Sent: Tuesday, June 5, 2018 2:23:45 PM
> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction

> Carlos, All,
> 
> On 2018-06-02 23:21 -0300, Carlos Santos spake thusly:
>> Since commit 118534fe54b (fs: use a common tarball as base for the other
>> filesystems) Buildroot creates a .tar filesystem image and re-extracts
>> it in a private directory to create each format-specific image. Add an
>> option to pass extra arguments to tar when that commom root image is
>> extracted.
>> 
>> This option is useful when the root filesystem is volatile (e.g. initrd)
>> or read-only but a read-write subtree is still necessary for persistent
>> data modified by programs as they run.
>> 
>> For example, one can pass "--exclude='./var/lib/*'" to exclude that path
>> from the rootfs image and use a post-fakeroot script to make a separate
>> filesystem image for /var/lib.
> 
> Well, I am not too fond of this extra option. But see below...
> 
>> Signed-off-by: Carlos Santos <casantos@datacom.com.br>
>> ---
>> Additional explanation
>> 
>> This change solves a real-life problem of my current project. Before
>> buildroot 2018.05 I had to use a trickier approach:
>> 
>> 1. Add a post-fakeroot script to
>>    1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib.
>>    1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR).
>>    1.3. Create an empty $(TARGET_DIR)/var/lib.
>> 2. Add a post-image script to
>>    2.1. Copy the rootfs and /var/lib images to the disk image.
>>    2.2. Remove the empty $(TARGET_DIR)/var/lib.
>>    2.3. Restore $(TARGET_DIR)/var/lib from the backup path.
>> 
>> That solution is fragile: if something goes wrong during the execution
>> of the post-image script it can lead to a cripled $(TARGET_DIR) with an
>> empty var/lib subdir. Rebuilding a package without restoring the backup
>> can easily cause a kaboom and force me to run a lengthy clean build.
> 
> What prevents you from providing your own filesystem mechanism at all?

I'm already doing this with the post-fakeroot and post-image scripts.

> Since you have post-scripts, it means you do have an existing tree
> where they are stored (even probably a git trre, at that).
> 
> So you could use that location as a br2-external, if that's not what
> you already do. And then, you can add a new filesystem type in that
> br2-external tree.
> 
> Jsut make your filesystem depend on rootfs-tar, then you can basically
> do something like:
> 
>    $ cat my-mkfs-tool
>    #!/bin/sh
>    if [ $(id -u) -ne 0 ]; then
>        exec fakeroot "${0}" "${@}"
>    fi
> 
>    mktempdir && cd tempdir # You get the idea
>    tar xf $(IMAGES_DIR)/rootfs.tar
>    do-soemthing
> 
> And then you have a mechanism that cleanly and reliably fits in
> Buildroot's filesystem infra.
> 
> I don't think the overhead would be big, since you anyway already have
> to have some scripts doing the separation properly, so making those into
> a filesystem is probably not too much, I guess.
> 
> For a (basic) example, see slide 16 (shameless self-reference):
>    https://elinux.org/images/8/8e/2017-10-24_-_ELCE-Buildroot.pdf
> 
> (note: the macro to call has changed in master since that presentation,
> but you get the idea.)

The inner-rootfs macro does not seem to be generic enough but I can
give it a try. Notice, however, that it is an internal thing, like
the common tarball. The BR2_ROOTFS_COMMON_UNTAR_ARGS confing, OTOH,
would be documented. If the corresponding infrastructure changes in
the future we can move it to Config.in.legacy, preventing the user
from using an unsupported feature.

> In the end, I believe this is more robust and more generic than this
> extra option to the intermediate tarball, which is purely an internal
> detail. We may tomorrow decide on another solution for the internal
> state in the future...
> 
> My 2 cents...
> 
> Regards,
> Yann E. MORIN.

[...]
Yann E. MORIN June 6, 2018, 6:51 p.m. | #3
Carlos, All,

On 2018-06-06 09:46 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
> > On 2018-06-02 23:21 -0300, Carlos Santos spake thusly:
> >> Since commit 118534fe54b (fs: use a common tarball as base for the other
> >> filesystems) Buildroot creates a .tar filesystem image and re-extracts
> >> it in a private directory to create each format-specific image. Add an
> >> option to pass extra arguments to tar when that commom root image is
> >> extracted.
> >> 
> >> This option is useful when the root filesystem is volatile (e.g. initrd)
> >> or read-only but a read-write subtree is still necessary for persistent
> >> data modified by programs as they run.
[--SNIP--]
> > What prevents you from providing your own filesystem mechanism at all?
[--SNIP--]
> The inner-rootfs macro does not seem to be generic enough but I can
> give it a try. Notice, however, that it is an internal thing, like
> the common tarball.

The inner-rootfs macro is indeed an internal one, and you should use the
macro that is named 'rootfs', like is done in all the filesystems, with:

    $(eval $(rootfs))

This macro is for filesystems what the various generic-, autotools-, and
the other *-package macros are for packages.

Indeed, this is not documented, but it *is* made to be used to implement
filesystems in br2-external trees.

I have started writing the documentation for it. And even I made a
mistake: you need not even extract the rootfs.tar: Buidlroot does it for
you before calling your filesystem generator. So, the documentation is
really needed, because even I, whoo wrote the code for the internal
tarball, forgot about that.

So, basically,  your own custom-fs.mk would contain something based on:

    # This is custom-fs.mk
    define ROOTFS_CUSTOM_FS_CMDS
        $(BR2_EXTERNAL_foo_PATH)/mkfs.custom-fs --root-dir=$(TARGET_DIR)
    endef
    $(eval $(rootfs))

> The BR2_ROOTFS_COMMON_UNTAR_ARGS confing, OTOH,
> would be documented. If the corresponding infrastructure changes in
> the future we can move it to Config.in.legacy, preventing the user
> from using an unsupported feature.

What I don't like in the exlanations you gave for your use-case, is that
you anyway already have to handle the content of /var/lib (or whatever)
in an ad-hoc manner from a fakeroot-script, so the behaviour you had to
split in two: one part in the fakeroot script that you own, and the
other part in the buildroot filesystem infra, which is generic.

Whereas if you write your own filesystem, then you do everything on your
own, which gives you absolute flexibility about what you can do with
the layout.

Yes, there *is* one thing that can be seen as a "waste" of Buildroot's
infra: if your main filesystem is of a type that Buildroot already knows
hos to generate (like a squashfs).

But my position has always been consistent on this topic: the images
that Buildroot generates only ever covers just "basic" situations, using
a single-filesystem layout. Anything that needs to do a multi-filesystem
layout should be done as a new filesystem. Doing it in a new filesystem
is much more flexible than whatever kconfig option we may ever add. And
since we already have this wonderful flexibility, I don't think it makes
sense to add a new option that duplicates only a very limited subset of
that flexibility. That duplication is not good, IMNSHO...

Regards,
Yann E. MORIN.

> > In the end, I believe this is more robust and more generic than this
> > extra option to the intermediate tarball, which is purely an internal
> > detail. We may tomorrow decide on another solution for the internal
> > state in the future...
> > 
> > My 2 cents...
> > 
> > Regards,
> > Yann E. MORIN.
> 
> [...]
> 
> -- 
> Carlos Santos (Casantos) - DATACOM, P&D
> “Marched towards the enemy, spear upright, armed with the certainty
> that only the ignorant can have.” — Epitaph of a volunteer
Arnout Vandecappelle (Essensium/Mind) June 7, 2018, 9:03 p.m. | #4
On 06-06-18 20:51, Yann E. MORIN wrote:
> But my position has always been consistent on this topic: the images
> that Buildroot generates only ever covers just "basic" situations, using
> a single-filesystem layout. Anything that needs to do a multi-filesystem
> layout should be done as a new filesystem. Doing it in a new filesystem
> is much more flexible than whatever kconfig option we may ever add. And
> since we already have this wonderful flexibility, I don't think it makes
> sense to add a new option that duplicates only a very limited subset of
> that flexibility. That duplication is not good, IMNSHO...

 There is (in my even less humble opinion) one way that we can solve this
generically: by adding a genimage filesystem. genimage is able to create
separate filesystem images for subtrees. so it can cover many use cases in a
generic way.

 There are probably a few gotchas, but I still believe it should be possible.

 Regards,
 Arnout
Carlos Santos June 7, 2018, 11:12 p.m. | #5
> From: "Arnout Vandecappelle" <arnout@mind.be>
> To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br>
> Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org>
> Sent: Thursday, June 7, 2018 6:03:48 PM
> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction

> On 06-06-18 20:51, Yann E. MORIN wrote:
>> But my position has always been consistent on this topic: the images
>> that Buildroot generates only ever covers just "basic" situations, using
>> a single-filesystem layout. Anything that needs to do a multi-filesystem
>> layout should be done as a new filesystem. Doing it in a new filesystem
>> is much more flexible than whatever kconfig option we may ever add. And
>> since we already have this wonderful flexibility, I don't think it makes
>> sense to add a new option that duplicates only a very limited subset of
>> that flexibility. That duplication is not good, IMNSHO...
> 
> There is (in my even less humble opinion) one way that we can solve this
> generically: by adding a genimage filesystem. genimage is able to create
> separate filesystem images for subtrees. so it can cover many use cases in a
> generic way.

I'm already working on a patch to convert inner-rootfs into a generic
inner-filesystem macro. I will submit it for comments soon.

BTW, it took me some time to understand the dual personality of TARGET_DIR
in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
the top Makefile with

  TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))

I understand that this trick avoids changing fs/*/*.mk replacing each
reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
the readability a lot. I'm compelled to restore it to how it was prior
to commit 7e9870ce32d.
Arnout Vandecappelle (Essensium/Mind) June 8, 2018, 7:48 a.m. | #6
On 08-06-18 01:12, Carlos Santos wrote:
>> From: "Arnout Vandecappelle" <arnout@mind.be>
>> To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br>
>> Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org>
>> Sent: Thursday, June 7, 2018 6:03:48 PM
>> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction
> 
>> On 06-06-18 20:51, Yann E. MORIN wrote:
>>> But my position has always been consistent on this topic: the images
>>> that Buildroot generates only ever covers just "basic" situations, using
>>> a single-filesystem layout. Anything that needs to do a multi-filesystem
>>> layout should be done as a new filesystem. Doing it in a new filesystem
>>> is much more flexible than whatever kconfig option we may ever add. And
>>> since we already have this wonderful flexibility, I don't think it makes
>>> sense to add a new option that duplicates only a very limited subset of
>>> that flexibility. That duplication is not good, IMNSHO...
>>
>> There is (in my even less humble opinion) one way that we can solve this
>> generically: by adding a genimage filesystem. genimage is able to create
>> separate filesystem images for subtrees. so it can cover many use cases in a
>> generic way.
> 
> I'm already working on a patch to convert inner-rootfs into a generic
> inner-filesystem macro. I will submit it for comments soon.
> 
> BTW, it took me some time to understand the dual personality of TARGET_DIR
> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
> the top Makefile with
> 
>   TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
> 
> I understand that this trick avoids changing fs/*/*.mk replacing each
> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
> the readability a lot. I'm compelled to restore it to how it was prior
> to commit 7e9870ce32d.

 The same (or a similar) trick will be applied to the normal packages for
per-package host/staging/target (which is ultimately needed for top-level
parallel build). If we want to avoid it there, we would have to change 785
package.mk files, and also many package definitions in BR2_EXTERNALs that we
don't control...

 That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But
doing so *will* require legacy handling.

 Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs
definitions, document that that is the one to use, and wait a year or two before
deprecating TARGET_DIR entirely.

 Yann?

 Regards,
 Arnout
Yann E. MORIN June 8, 2018, 5:19 p.m. | #7
Arnout, All,

On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly:
> On 06-06-18 20:51, Yann E. MORIN wrote:
> > But my position has always been consistent on this topic: the images
> > that Buildroot generates only ever covers just "basic" situations, using
> > a single-filesystem layout. Anything that needs to do a multi-filesystem
> > layout should be done as a new filesystem. Doing it in a new filesystem
> > is much more flexible than whatever kconfig option we may ever add. And
> > since we already have this wonderful flexibility, I don't think it makes
> > sense to add a new option that duplicates only a very limited subset of
> > that flexibility. That duplication is not good, IMNSHO...
> 
>  There is (in my even less humble opinion) one way that we can solve this
> generically: by adding a genimage filesystem. genimage is able to create
> separate filesystem images for subtrees. so it can cover many use cases in a
> generic way.
> 
>  There are probably a few gotchas, but I still believe it should be possible.

Like tweaking the /etc/fstab accordingly, which genimage does not do on
its own?

So this "genimage fs" would not be so complex, but not so obvious either,
to write, as we'd need to have some additional script that does the fstab
tweaking... Probably that info would have to be scrapped out of the
genimage.cfg file.

Regards,
Yann E. MORIN.
Yann E. MORIN June 8, 2018, 5:26 p.m. | #8
Carlos, All,

On 2018-06-07 20:12 -0300, Carlos Santos spake thusly:
> > From: "Arnout Vandecappelle" <arnout@mind.be>
> > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br>
> > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org>
> > Sent: Thursday, June 7, 2018 6:03:48 PM
> > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction
> 
> > On 06-06-18 20:51, Yann E. MORIN wrote:
> >> But my position has always been consistent on this topic: the images
> >> that Buildroot generates only ever covers just "basic" situations, using
> >> a single-filesystem layout. Anything that needs to do a multi-filesystem
> >> layout should be done as a new filesystem. Doing it in a new filesystem
> >> is much more flexible than whatever kconfig option we may ever add. And
> >> since we already have this wonderful flexibility, I don't think it makes
> >> sense to add a new option that duplicates only a very limited subset of
> >> that flexibility. That duplication is not good, IMNSHO...
> > 
> > There is (in my even less humble opinion) one way that we can solve this
> > generically: by adding a genimage filesystem. genimage is able to create
> > separate filesystem images for subtrees. so it can cover many use cases in a
> > generic way.
> 
> I'm already working on a patch to convert inner-rootfs into a generic
> inner-filesystem macro. I will submit it for comments soon.

Why do you need to do so? You can use the 'rootfs' macro, as I already
explained, and for which I have already sent a patch to add it to the
manual:

    https://patchwork.ozlabs.org/patch/926425/

> BTW, it took me some time to understand the dual personality of TARGET_DIR
> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
> the top Makefile with
> 
>   TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
> 
> I understand that this trick avoids changing fs/*/*.mk replacing each
> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
> the readability a lot. I'm compelled to restore it to how it was prior
> to commit 7e9870ce32d.

But if you revert that, then TARGET_DIR points to the original target/
directory, which does *not* contain the completely-finalised content.

Please see the commits around that one for the full picture:

    git log --oneline 14d43aea0a..543107d390

And see the cover-letter that explains the motivations behind all those
changes:

    http://lists.busybox.net/pipermail/buildroot/2018-March/215450.html

Regards,
Yann E. MORIN.
Yann E. MORIN June 8, 2018, 5:34 p.m. | #9
Arnout, All,

On 2018-06-08 09:48 +0200, Arnout Vandecappelle spake thusly:
> On 08-06-18 01:12, Carlos Santos wrote:
[--SNIP--]
> > BTW, it took me some time to understand the dual personality of TARGET_DIR
> > in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
> > the top Makefile with
> > 
> >   TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
> > 
> > I understand that this trick avoids changing fs/*/*.mk replacing each
> > reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
> > the readability a lot. I'm compelled to restore it to how it was prior
> > to commit 7e9870ce32d.
> 
>  The same (or a similar) trick will be applied to the normal packages for
> per-package host/staging/target (which is ultimately needed for top-level
> parallel build). If we want to avoid it there, we would have to change 785
> package.mk files, and also many package definitions in BR2_EXTERNALs that we
> don't control...
> 
>  That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But
> doing so *will* require legacy handling.
> 
>  Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs
> definitions, document that that is the one to use, and wait a year or two before
> deprecating TARGET_DIR entirely.
> 
>  Yann?

Well, It is really cumbersome to have to prefix that directory with the
package name or the rootfs name... TARGET_DIR is just plainly easy to
use.

Especially since we would break a long-established variable name.

And it is not the only variable that changes its content based on the
current package: $(@D) and $(@) et al. Yes, they are make special
variables, but so?

In the end, I still fail to see the problem. But I am usual pretty
stubborn, so... ;-)

Regards,
Yann E. MORIN.
Arnout Vandecappelle (Essensium/Mind) June 8, 2018, 7:58 p.m. | #10
On 08-06-18 19:34, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2018-06-08 09:48 +0200, Arnout Vandecappelle spake thusly:
>> On 08-06-18 01:12, Carlos Santos wrote:
> [--SNIP--]
>>> BTW, it took me some time to understand the dual personality of TARGET_DIR
>>> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
>>> the top Makefile with
>>>
>>>   TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
>>>
>>> I understand that this trick avoids changing fs/*/*.mk replacing each
>>> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
>>> the readability a lot. I'm compelled to restore it to how it was prior
>>> to commit 7e9870ce32d.
>>
>>  The same (or a similar) trick will be applied to the normal packages for
>> per-package host/staging/target (which is ultimately needed for top-level
>> parallel build). If we want to avoid it there, we would have to change 785
>> package.mk files, and also many package definitions in BR2_EXTERNALs that we
>> don't control...
>>
>>  That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But
>> doing so *will* require legacy handling.
>>
>>  Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs
>> definitions, document that that is the one to use, and wait a year or two before
>> deprecating TARGET_DIR entirely.
>>
>>  Yann?
> 
> Well, It is really cumbersome to have to prefix that directory with the
> package name or the rootfs name... TARGET_DIR is just plainly easy to
> use.d

 I don't agree with that - we use FOO_SOME_VARIABLE all over the place in
package foo, so using FOO_TARGET_DIR feels quite natural to me.

> Especially since we would break a long-established variable name.

 That's the worst reason ever :-) Except that indeed there should be a (long)
transition period where the old name is still supported, but we don't use it
internally.


> And it is not the only variable that changes its content based on the
> current package: $(@D) and $(@) et al. Yes, they are make special
> variables, but so?

 That makes a huge difference - people know those variables.

 And anyway, I wouldn't mind to use $(FOO_BUILD_DIR) instead of $(@D). I think
that is easier for people to understand. $(@D) just happens to work because the
stamp files are created in the build directory, which IMO is an implementation
detail that shouldn't leak into the packages.


 Regards,
 Arnout


> In the end, I still fail to see the problem. But I am usual pretty
> stubborn, so... ;-)
> 
> Regards,
> Yann E. MORIN.
>
Arnout Vandecappelle (Essensium/Mind) June 8, 2018, 7:59 p.m. | #11
On 08-06-18 19:19, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly:
>> On 06-06-18 20:51, Yann E. MORIN wrote:
>>> But my position has always been consistent on this topic: the images
>>> that Buildroot generates only ever covers just "basic" situations, using
>>> a single-filesystem layout. Anything that needs to do a multi-filesystem
>>> layout should be done as a new filesystem. Doing it in a new filesystem
>>> is much more flexible than whatever kconfig option we may ever add. And
>>> since we already have this wonderful flexibility, I don't think it makes
>>> sense to add a new option that duplicates only a very limited subset of
>>> that flexibility. That duplication is not good, IMNSHO...
>>
>>  There is (in my even less humble opinion) one way that we can solve this
>> generically: by adding a genimage filesystem. genimage is able to create
>> separate filesystem images for subtrees. so it can cover many use cases in a
>> generic way.
>>
>>  There are probably a few gotchas, but I still believe it should be possible.
> 
> Like tweaking the /etc/fstab accordingly, which genimage does not do on
> its own?

 No, we don't want to tweak fstab. There is no way to know if you want things to
be mounted in that particular way, and whether it should be mounted
automatically, and if some flags are needed, and so on.


 Regards,
 Arnout


> So this "genimage fs" would not be so complex, but not so obvious either,
> to write, as we'd need to have some additional script that does the fstab
> tweaking... Probably that info would have to be scrapped out of the
> genimage.cfg file.
> 
> Regards,
> Yann E. MORIN.
>
Yann E. MORIN June 8, 2018, 9:06 p.m. | #12
Arnout, Carlos, All,

On 2018-06-08 21:59 +0200, Arnout Vandecappelle spake thusly:
> On 08-06-18 19:19, Yann E. MORIN wrote:
> > On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly:
> >> On 06-06-18 20:51, Yann E. MORIN wrote:
> >>> But my position has always been consistent on this topic: the images
> >>> that Buildroot generates only ever covers just "basic" situations, using
> >>> a single-filesystem layout. Anything that needs to do a multi-filesystem
> >>> layout should be done as a new filesystem. Doing it in a new filesystem
> >>> is much more flexible than whatever kconfig option we may ever add. And
> >>> since we already have this wonderful flexibility, I don't think it makes
> >>> sense to add a new option that duplicates only a very limited subset of
> >>> that flexibility. That duplication is not good, IMNSHO...
> >>
> >>  There is (in my even less humble opinion) one way that we can solve this
> >> generically: by adding a genimage filesystem. genimage is able to create
> >> separate filesystem images for subtrees. so it can cover many use cases in a
> >> generic way.
> >>
> >>  There are probably a few gotchas, but I still believe it should be possible.
> > 
> > Like tweaking the /etc/fstab accordingly, which genimage does not do on
> > its own?
> 
>  No, we don't want to tweak fstab. There is no way to know if you want things to
> be mounted in that particular way, and whether it should be mounted
> automatically, and if some flags are needed, and so on.

Right, fstab was just an example of site-specific tweaks to be done.
Others may want to use startup scripts or systemd units to do the
mounts, or to format a partition and rxtract a factory-defaults tarball
in that filesystem, or get that from The Cloud, or whatever.

Waht I mean is that splitting the filesystem is easy. What is not easy
is that each part thus split can be in different formats, located in so
different locations, retrieved and accessed in so many different ways,
that we can't possibly imagine, and much less cover in a generic way.

The only way we can allow people to do what they want, is by providing
them with a per-filesysem target/ directory already prepared that they
can arrange at will.

And as an example of a split-var filesytem, see the attached br2-external
tree. It ultimagtely creates $(BINARIES_DIR)/rootfs.meh, a squashfs
image of /) and $(BINARIES_DIR)/rootfs.meh-var.tar, a tarball of /var.
The only missing pieve is the runtime startup scripts that will format
the on-board partition, extact /var.tar in there, and mount it.

Regards,
Yann E. MORIN.
Carlos Santos June 9, 2018, 12:58 a.m. | #13
> From: "Arnout Vandecappelle" <arnout@mind.be>
> To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br>
> Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org>
> Sent: Thursday, June 7, 2018 6:03:48 PM
> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction

> On 06-06-18 20:51, Yann E. MORIN wrote:
>> But my position has always been consistent on this topic: the images
>> that Buildroot generates only ever covers just "basic" situations, using
>> a single-filesystem layout. Anything that needs to do a multi-filesystem
>> layout should be done as a new filesystem. Doing it in a new filesystem
>> is much more flexible than whatever kconfig option we may ever add. And
>> since we already have this wonderful flexibility, I don't think it makes
>> sense to add a new option that duplicates only a very limited subset of
>> that flexibility. That duplication is not good, IMNSHO...
> 
> There is (in my even less humble opinion) one way that we can solve this
> generically: by adding a genimage filesystem. genimage is able to create
> separate filesystem images for subtrees. so it can cover many use cases in a
> generic way.
> 
> There are probably a few gotchas, but I still believe it should be possible.

Genimage is unable to deal with GPT partitioning, which is much better
to use with UEFI BIOS. I dropped it in favour of a custom script that
creates an empty disk image file using sfdisk to partition it using a
file containing the partitioning scheme. Then it reads the start offset
an size of each partition and dumps the corresponding FS image there.
Carlos Santos June 9, 2018, 1:06 a.m. | #14
> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "DATACOM" <casantos@datacom.com.br>
> Cc: "Arnout Vandecappelle" <arnout@mind.be>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot"
> <buildroot@buildroot.org>
> Sent: Friday, June 8, 2018 2:26:51 PM
> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction

> Carlos, All,
> 
> On 2018-06-07 20:12 -0300, Carlos Santos spake thusly:
>> > From: "Arnout Vandecappelle" <arnout@mind.be>
>> > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br>
>> > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot"
>> > <buildroot@buildroot.org>
>> > Sent: Thursday, June 7, 2018 6:03:48 PM
>> > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball
>> > extraction
>> 
>> > On 06-06-18 20:51, Yann E. MORIN wrote:
>> >> But my position has always been consistent on this topic: the images
>> >> that Buildroot generates only ever covers just "basic" situations, using
>> >> a single-filesystem layout. Anything that needs to do a multi-filesystem
>> >> layout should be done as a new filesystem. Doing it in a new filesystem
>> >> is much more flexible than whatever kconfig option we may ever add. And
>> >> since we already have this wonderful flexibility, I don't think it makes
>> >> sense to add a new option that duplicates only a very limited subset of
>> >> that flexibility. That duplication is not good, IMNSHO...
>> > 
>> > There is (in my even less humble opinion) one way that we can solve this
>> > generically: by adding a genimage filesystem. genimage is able to create
>> > separate filesystem images for subtrees. so it can cover many use cases in a
>> > generic way.
>> 
>> I'm already working on a patch to convert inner-rootfs into a generic
>> inner-filesystem macro. I will submit it for comments soon.
> 
> Why do you need to do so? You can use the 'rootfs' macro, as I already
> explained, and for which I have already sent a patch to add it to the
> manual:
> 
>    https://patchwork.ozlabs.org/patch/926425/

I've seen your patch and will review it as soon as possible.

>> BTW, it took me some time to understand the dual personality of TARGET_DIR
>> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in
>> the top Makefile with
>> 
>>   TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
>> 
>> I understand that this trick avoids changing fs/*/*.mk replacing each
>> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
>> the readability a lot. I'm compelled to restore it to how it was prior
>> to commit 7e9870ce32d.
> 
> But if you revert that, then TARGET_DIR points to the original target/
> directory, which does *not* contain the completely-finalised content.

I pass TARGET_DIR=$(ROOTFS_COMMON_TARGET_DIR) in the environment to
each post-fakeroot script to mimic the previous behavior, since the
scripts may take it from the environment variable instead of from the
first argument. Anyway, this chaneg is not critical and can be postponed
tu reduce the fuzz.
 
> Please see the commits around that one for the full picture:
> 
>    git log --oneline 14d43aea0a..543107d390
> 
> And see the cover-letter that explains the motivations behind all those
> changes:
> 
>    http://lists.busybox.net/pipermail/buildroot/2018-March/215450.html
Yann E. MORIN June 9, 2018, 7:31 a.m. | #15
Carlos, All,

On 2018-06-08 22:06 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
> > Why do you need to do so? You can use the 'rootfs' macro, as I already
> > explained, and for which I have already sent a patch to add it to the
> > manual:
> >    https://patchwork.ozlabs.org/patch/926425/
> I've seen your patch and will review it as soon as possible.

OK, great. Thanks! :-)

[--SNIP--]
> >> I understand that this trick avoids changing fs/*/*.mk replacing each
> >> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces
> >> the readability a lot. I'm compelled to restore it to how it was prior
> >> to commit 7e9870ce32d.
> > 
> > But if you revert that, then TARGET_DIR points to the original target/
> > directory, which does *not* contain the completely-finalised content.
> 
> I pass TARGET_DIR=$(ROOTFS_COMMON_TARGET_DIR) in the environment to

No, don't point to the common one, because that is not parallel-safe!
The whole change was made so that two filesystems could be build in
parallel. Ad=nd since some filesystems want to muck with the layout
(e.g. iso9660), that is not compatible with two filesystems building
simultaneously from the same directory.

At the very least, make TARGET_DIR=$(ROOTFS_$(ROOTFS)_TARGET_DIR) as
Arnout hinted previously.

Regards,
Yann E. MORIN.
Carlos Santos June 10, 2018, 11:48 p.m. | #16
> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "Arnout Vandecappelle" <arnout@mind.be>
> Cc: "DATACOM" <casantos@datacom.com.br>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot"
> <buildroot@buildroot.org>
> Sent: Friday, June 8, 2018 6:06:41 PM
> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction

> Arnout, Carlos, All,
[...]
> And as an example of a split-var filesytem, see the attached br2-external
> tree. It ultimagtely creates $(BINARIES_DIR)/rootfs.meh, a squashfs
> image of /) and $(BINARIES_DIR)/rootfs.meh-var.tar, a tarball of /var.
> The only missing pieve is the runtime startup scripts that will format
> the on-board partition, extact /var.tar in there, and mount it.

Thanks for your example. I transformed it on an example of what I'm
attempting to achieve. Please use the contents of the attached file as
a br2_external and use the defconfig contained there; then run

    $ make rootfs-cpio rootfs-var-{lib,log}-ext4

It will create three image files files under $(BINARIES_DIR): rootfs.cpio,
rootfs.var-lib.ext4 and rootfs.var-log-ext4.

If you apply on buildroot the patch contained there the names will be
rootfs.cpio, rootfs-var-lib.ext4 and rootfs-var-log.ext4, which looks
better, IMO.

Extracting rootfs.common.tar for rootfs-var-log-ext4 is a waste of time
and disk space, since it creates an empty filesystem. Perhaps we could
add a BR2_TARGET_ROOTFS_<FOO>_START_EMPTY knob for filesystems that
populate $(TARGET_DIR) by themselves.

The example does not install a /etc/fstab with entries for /var/lib and
/var/log but of it would be necessary in a real system, of course.

Patch

diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt
index bb6d8da6bf..6dd9365723 100644
--- a/docs/manual/customize-rootfs.txt
+++ b/docs/manual/customize-rootfs.txt
@@ -121,6 +121,16 @@  This method is not recommended because it duplicates the entire
   skeleton, which prevents taking advantage of the fixes or improvements
   brought to the default skeleton in later Buildroot releases.
 
+Rootfs tarball extraction arguments (+BR2_ROOTFS_COMMON_UNTAR_ARGS+)::
++
+Buildroot creates a .tar filesystem image and re-extracts it in a private
+ directory to create each format-specific image. Use this option to pass
+ extra arguments to tar when the commom root image is extracted.
++
+For example, you can pass "--exclude=\'./var/lib/*\'" to exclude that path from
+ the rootfs image and use a post-fakeroot script (see below) to create a
+ separate filesystem image for '/var/lib'.
+
 Post-fakeroot scripts (+BR2_ROOTFS_POST_FAKEROOT_SCRIPT+)::
 +
 When aggregating the final images, some parts of the process requires
diff --git a/fs/Config.in b/fs/Config.in
index c25b01c3de..74487b1172 100644
--- a/fs/Config.in
+++ b/fs/Config.in
@@ -1,5 +1,18 @@ 
 menu "Filesystem images"
 
+config BR2_ROOTFS_COMMON_UNTAR_ARGS
+	string "Rootfs tarball extraction arguments"
+	help
+	  Buildroot creates a .tar filesystem image and re-extracts it
+	  in a private directory to create each format-specific image.
+	  Use this option to pass extra arguments to tar when the commom
+	  root image is extracted.
+
+	  For example, you can pass "--exclude='./var/lib/*'" to exclude
+	  that path from the rootfs image and use a post-fakeroot script
+	  (see BR2_ROOTFS_POST_FAKEROOT_SCRIPT) to create a separate
+	  filesystem image for /var/lib.
+
 source "fs/axfs/Config.in"
 source "fs/cloop/Config.in"
 source "fs/cpio/Config.in"
diff --git a/fs/common.mk b/fs/common.mk
index abf35418cb..33ad1ab2a9 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -50,9 +50,10 @@  define ROOTFS_COMMON_TAR_CMD
 endef
 
 # Command to extract the common tarball into the per-rootfs target directory
+ROOTFS_COMMON_UNTAR_ARGS = $(call qstrip,$(BR2_ROOTFS_COMMON_UNTAR_ARGS))
 define ROOTFS_COMMON_UNTAR_CMD
 	mkdir -p $(TARGET_DIR)
-	tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR)
+	tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) $(ROOTFS_COMMON_UNTAR_ARGS)
 endef
 
 .PHONY: rootfs-common