| Message ID | 20180603022145.14222-1-casantos@datacom.com.br |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
| Series | fs: allow extra arguments to common tarball extraction | expand |
Carlos, All, On 2018-06-02 23:21 -0300, Carlos Santos spake thusly: > Since commit 118534fe54b (fs: use a common tarball as base for the other > filesystems) Buildroot creates a .tar filesystem image and re-extracts > it in a private directory to create each format-specific image. Add an > option to pass extra arguments to tar when that commom root image is > extracted. > > This option is useful when the root filesystem is volatile (e.g. initrd) > or read-only but a read-write subtree is still necessary for persistent > data modified by programs as they run. > > For example, one can pass "--exclude='./var/lib/*'" to exclude that path > from the rootfs image and use a post-fakeroot script to make a separate > filesystem image for /var/lib. Well, I am not too fond of this extra option. But see below... > Signed-off-by: Carlos Santos <casantos@datacom.com.br> > --- > Additional explanation > > This change solves a real-life problem of my current project. Before > buildroot 2018.05 I had to use a trickier approach: > > 1. Add a post-fakeroot script to > 1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib. > 1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR). > 1.3. Create an empty $(TARGET_DIR)/var/lib. > 2. Add a post-image script to > 2.1. Copy the rootfs and /var/lib images to the disk image. > 2.2. Remove the empty $(TARGET_DIR)/var/lib. > 2.3. Restore $(TARGET_DIR)/var/lib from the backup path. > > That solution is fragile: if something goes wrong during the execution > of the post-image script it can lead to a cripled $(TARGET_DIR) with an > empty var/lib subdir. Rebuilding a package without restoring the backup > can easily cause a kaboom and force me to run a lengthy clean build. What prevents you from providing your own filesystem mechanism at all? Since you have post-scripts, it means you do have an existing tree where they are stored (even probably a git trre, at that). So you could use that location as a br2-external, if that's not what you already do. And then, you can add a new filesystem type in that br2-external tree. Jsut make your filesystem depend on rootfs-tar, then you can basically do something like: $ cat my-mkfs-tool #!/bin/sh if [ $(id -u) -ne 0 ]; then exec fakeroot "${0}" "${@}" fi mktempdir && cd tempdir # You get the idea tar xf $(IMAGES_DIR)/rootfs.tar do-soemthing And then you have a mechanism that cleanly and reliably fits in Buildroot's filesystem infra. I don't think the overhead would be big, since you anyway already have to have some scripts doing the separation properly, so making those into a filesystem is probably not too much, I guess. For a (basic) example, see slide 16 (shameless self-reference): https://elinux.org/images/8/8e/2017-10-24_-_ELCE-Buildroot.pdf (note: the macro to call has changed in master since that presentation, but you get the idea.) In the end, I believe this is more robust and more generic than this extra option to the intermediate tarball, which is purely an internal detail. We may tomorrow decide on another solution for the internal state in the future... My 2 cents... Regards, Yann E. MORIN. > I added some protection mechanisms to stop the build if a stale /var/lib > backup is found but quite frankly the whole thing stinks. > > The solution with custom extraction arguments, OTOH, is bullet-proof > because it does not change $(TARGET_DIR), so it is always safe to run > dirty builds. > --- > docs/manual/customize-rootfs.txt | 10 ++++++++++ > fs/Config.in | 13 +++++++++++++ > fs/common.mk | 3 ++- > 3 files changed, 25 insertions(+), 1 deletion(-) > > diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt > index bb6d8da6bf..6dd9365723 100644 > --- a/docs/manual/customize-rootfs.txt > +++ b/docs/manual/customize-rootfs.txt > @@ -121,6 +121,16 @@ This method is not recommended because it duplicates the entire > skeleton, which prevents taking advantage of the fixes or improvements > brought to the default skeleton in later Buildroot releases. > > +Rootfs tarball extraction arguments (+BR2_ROOTFS_COMMON_UNTAR_ARGS+):: > ++ > +Buildroot creates a .tar filesystem image and re-extracts it in a private > + directory to create each format-specific image. Use this option to pass > + extra arguments to tar when the commom root image is extracted. > ++ > +For example, you can pass "--exclude=\'./var/lib/*\'" to exclude that path from > + the rootfs image and use a post-fakeroot script (see below) to create a > + separate filesystem image for '/var/lib'. > + > Post-fakeroot scripts (+BR2_ROOTFS_POST_FAKEROOT_SCRIPT+):: > + > When aggregating the final images, some parts of the process requires > diff --git a/fs/Config.in b/fs/Config.in > index c25b01c3de..74487b1172 100644 > --- a/fs/Config.in > +++ b/fs/Config.in > @@ -1,5 +1,18 @@ > menu "Filesystem images" > > +config BR2_ROOTFS_COMMON_UNTAR_ARGS > + string "Rootfs tarball extraction arguments" > + help > + Buildroot creates a .tar filesystem image and re-extracts it > + in a private directory to create each format-specific image. > + Use this option to pass extra arguments to tar when the commom > + root image is extracted. > + > + For example, you can pass "--exclude='./var/lib/*'" to exclude > + that path from the rootfs image and use a post-fakeroot script > + (see BR2_ROOTFS_POST_FAKEROOT_SCRIPT) to create a separate > + filesystem image for /var/lib. > + > source "fs/axfs/Config.in" > source "fs/cloop/Config.in" > source "fs/cpio/Config.in" > diff --git a/fs/common.mk b/fs/common.mk > index abf35418cb..33ad1ab2a9 100644 > --- a/fs/common.mk > +++ b/fs/common.mk > @@ -50,9 +50,10 @@ define ROOTFS_COMMON_TAR_CMD > endef > > # Command to extract the common tarball into the per-rootfs target directory > +ROOTFS_COMMON_UNTAR_ARGS = $(call qstrip,$(BR2_ROOTFS_COMMON_UNTAR_ARGS)) > define ROOTFS_COMMON_UNTAR_CMD > mkdir -p $(TARGET_DIR) > - tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) > + tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) $(ROOTFS_COMMON_UNTAR_ARGS) > endef > > .PHONY: rootfs-common > -- > 2.17.0 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
> From: "Yann Morin" <yann.morin.1998@free.fr> > To: "DATACOM" <casantos@datacom.com.br> > Cc: "buildroot" <buildroot@buildroot.org>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com> > Sent: Tuesday, June 5, 2018 2:23:45 PM > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > Carlos, All, > > On 2018-06-02 23:21 -0300, Carlos Santos spake thusly: >> Since commit 118534fe54b (fs: use a common tarball as base for the other >> filesystems) Buildroot creates a .tar filesystem image and re-extracts >> it in a private directory to create each format-specific image. Add an >> option to pass extra arguments to tar when that commom root image is >> extracted. >> >> This option is useful when the root filesystem is volatile (e.g. initrd) >> or read-only but a read-write subtree is still necessary for persistent >> data modified by programs as they run. >> >> For example, one can pass "--exclude='./var/lib/*'" to exclude that path >> from the rootfs image and use a post-fakeroot script to make a separate >> filesystem image for /var/lib. > > Well, I am not too fond of this extra option. But see below... > >> Signed-off-by: Carlos Santos <casantos@datacom.com.br> >> --- >> Additional explanation >> >> This change solves a real-life problem of my current project. Before >> buildroot 2018.05 I had to use a trickier approach: >> >> 1. Add a post-fakeroot script to >> 1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib. >> 1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR). >> 1.3. Create an empty $(TARGET_DIR)/var/lib. >> 2. Add a post-image script to >> 2.1. Copy the rootfs and /var/lib images to the disk image. >> 2.2. Remove the empty $(TARGET_DIR)/var/lib. >> 2.3. Restore $(TARGET_DIR)/var/lib from the backup path. >> >> That solution is fragile: if something goes wrong during the execution >> of the post-image script it can lead to a cripled $(TARGET_DIR) with an >> empty var/lib subdir. Rebuilding a package without restoring the backup >> can easily cause a kaboom and force me to run a lengthy clean build. > > What prevents you from providing your own filesystem mechanism at all? I'm already doing this with the post-fakeroot and post-image scripts. > Since you have post-scripts, it means you do have an existing tree > where they are stored (even probably a git trre, at that). > > So you could use that location as a br2-external, if that's not what > you already do. And then, you can add a new filesystem type in that > br2-external tree. > > Jsut make your filesystem depend on rootfs-tar, then you can basically > do something like: > > $ cat my-mkfs-tool > #!/bin/sh > if [ $(id -u) -ne 0 ]; then > exec fakeroot "${0}" "${@}" > fi > > mktempdir && cd tempdir # You get the idea > tar xf $(IMAGES_DIR)/rootfs.tar > do-soemthing > > And then you have a mechanism that cleanly and reliably fits in > Buildroot's filesystem infra. > > I don't think the overhead would be big, since you anyway already have > to have some scripts doing the separation properly, so making those into > a filesystem is probably not too much, I guess. > > For a (basic) example, see slide 16 (shameless self-reference): > https://elinux.org/images/8/8e/2017-10-24_-_ELCE-Buildroot.pdf > > (note: the macro to call has changed in master since that presentation, > but you get the idea.) The inner-rootfs macro does not seem to be generic enough but I can give it a try. Notice, however, that it is an internal thing, like the common tarball. The BR2_ROOTFS_COMMON_UNTAR_ARGS confing, OTOH, would be documented. If the corresponding infrastructure changes in the future we can move it to Config.in.legacy, preventing the user from using an unsupported feature. > In the end, I believe this is more robust and more generic than this > extra option to the intermediate tarball, which is purely an internal > detail. We may tomorrow decide on another solution for the internal > state in the future... > > My 2 cents... > > Regards, > Yann E. MORIN. [...]
Carlos, All, On 2018-06-06 09:46 -0300, Carlos Santos spake thusly: > > From: "Yann Morin" <yann.morin.1998@free.fr> > > On 2018-06-02 23:21 -0300, Carlos Santos spake thusly: > >> Since commit 118534fe54b (fs: use a common tarball as base for the other > >> filesystems) Buildroot creates a .tar filesystem image and re-extracts > >> it in a private directory to create each format-specific image. Add an > >> option to pass extra arguments to tar when that commom root image is > >> extracted. > >> > >> This option is useful when the root filesystem is volatile (e.g. initrd) > >> or read-only but a read-write subtree is still necessary for persistent > >> data modified by programs as they run. [--SNIP--] > > What prevents you from providing your own filesystem mechanism at all? [--SNIP--] > The inner-rootfs macro does not seem to be generic enough but I can > give it a try. Notice, however, that it is an internal thing, like > the common tarball. The inner-rootfs macro is indeed an internal one, and you should use the macro that is named 'rootfs', like is done in all the filesystems, with: $(eval $(rootfs)) This macro is for filesystems what the various generic-, autotools-, and the other *-package macros are for packages. Indeed, this is not documented, but it *is* made to be used to implement filesystems in br2-external trees. I have started writing the documentation for it. And even I made a mistake: you need not even extract the rootfs.tar: Buidlroot does it for you before calling your filesystem generator. So, the documentation is really needed, because even I, whoo wrote the code for the internal tarball, forgot about that. So, basically, your own custom-fs.mk would contain something based on: # This is custom-fs.mk define ROOTFS_CUSTOM_FS_CMDS $(BR2_EXTERNAL_foo_PATH)/mkfs.custom-fs --root-dir=$(TARGET_DIR) endef $(eval $(rootfs)) > The BR2_ROOTFS_COMMON_UNTAR_ARGS confing, OTOH, > would be documented. If the corresponding infrastructure changes in > the future we can move it to Config.in.legacy, preventing the user > from using an unsupported feature. What I don't like in the exlanations you gave for your use-case, is that you anyway already have to handle the content of /var/lib (or whatever) in an ad-hoc manner from a fakeroot-script, so the behaviour you had to split in two: one part in the fakeroot script that you own, and the other part in the buildroot filesystem infra, which is generic. Whereas if you write your own filesystem, then you do everything on your own, which gives you absolute flexibility about what you can do with the layout. Yes, there *is* one thing that can be seen as a "waste" of Buildroot's infra: if your main filesystem is of a type that Buildroot already knows hos to generate (like a squashfs). But my position has always been consistent on this topic: the images that Buildroot generates only ever covers just "basic" situations, using a single-filesystem layout. Anything that needs to do a multi-filesystem layout should be done as a new filesystem. Doing it in a new filesystem is much more flexible than whatever kconfig option we may ever add. And since we already have this wonderful flexibility, I don't think it makes sense to add a new option that duplicates only a very limited subset of that flexibility. That duplication is not good, IMNSHO... Regards, Yann E. MORIN. > > In the end, I believe this is more robust and more generic than this > > extra option to the intermediate tarball, which is purely an internal > > detail. We may tomorrow decide on another solution for the internal > > state in the future... > > > > My 2 cents... > > > > Regards, > > Yann E. MORIN. > > [...] > > -- > Carlos Santos (Casantos) - DATACOM, P&D > “Marched towards the enemy, spear upright, armed with the certainty > that only the ignorant can have.” — Epitaph of a volunteer
On 06-06-18 20:51, Yann E. MORIN wrote: > But my position has always been consistent on this topic: the images > that Buildroot generates only ever covers just "basic" situations, using > a single-filesystem layout. Anything that needs to do a multi-filesystem > layout should be done as a new filesystem. Doing it in a new filesystem > is much more flexible than whatever kconfig option we may ever add. And > since we already have this wonderful flexibility, I don't think it makes > sense to add a new option that duplicates only a very limited subset of > that flexibility. That duplication is not good, IMNSHO... There is (in my even less humble opinion) one way that we can solve this generically: by adding a genimage filesystem. genimage is able to create separate filesystem images for subtrees. so it can cover many use cases in a generic way. There are probably a few gotchas, but I still believe it should be possible. Regards, Arnout
> From: "Arnout Vandecappelle" <arnout@mind.be> > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br> > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org> > Sent: Thursday, June 7, 2018 6:03:48 PM > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > On 06-06-18 20:51, Yann E. MORIN wrote: >> But my position has always been consistent on this topic: the images >> that Buildroot generates only ever covers just "basic" situations, using >> a single-filesystem layout. Anything that needs to do a multi-filesystem >> layout should be done as a new filesystem. Doing it in a new filesystem >> is much more flexible than whatever kconfig option we may ever add. And >> since we already have this wonderful flexibility, I don't think it makes >> sense to add a new option that duplicates only a very limited subset of >> that flexibility. That duplication is not good, IMNSHO... > > There is (in my even less humble opinion) one way that we can solve this > generically: by adding a genimage filesystem. genimage is able to create > separate filesystem images for subtrees. so it can cover many use cases in a > generic way. I'm already working on a patch to convert inner-rootfs into a generic inner-filesystem macro. I will submit it for comments soon. BTW, it took me some time to understand the dual personality of TARGET_DIR in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in the top Makefile with TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) I understand that this trick avoids changing fs/*/*.mk replacing each reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces the readability a lot. I'm compelled to restore it to how it was prior to commit 7e9870ce32d.
On 08-06-18 01:12, Carlos Santos wrote: >> From: "Arnout Vandecappelle" <arnout@mind.be> >> To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br> >> Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org> >> Sent: Thursday, June 7, 2018 6:03:48 PM >> Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > >> On 06-06-18 20:51, Yann E. MORIN wrote: >>> But my position has always been consistent on this topic: the images >>> that Buildroot generates only ever covers just "basic" situations, using >>> a single-filesystem layout. Anything that needs to do a multi-filesystem >>> layout should be done as a new filesystem. Doing it in a new filesystem >>> is much more flexible than whatever kconfig option we may ever add. And >>> since we already have this wonderful flexibility, I don't think it makes >>> sense to add a new option that duplicates only a very limited subset of >>> that flexibility. That duplication is not good, IMNSHO... >> >> There is (in my even less humble opinion) one way that we can solve this >> generically: by adding a genimage filesystem. genimage is able to create >> separate filesystem images for subtrees. so it can cover many use cases in a >> generic way. > > I'm already working on a patch to convert inner-rootfs into a generic > inner-filesystem macro. I will submit it for comments soon. > > BTW, it took me some time to understand the dual personality of TARGET_DIR > in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in > the top Makefile with > > TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) > > I understand that this trick avoids changing fs/*/*.mk replacing each > reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces > the readability a lot. I'm compelled to restore it to how it was prior > to commit 7e9870ce32d. The same (or a similar) trick will be applied to the normal packages for per-package host/staging/target (which is ultimately needed for top-level parallel build). If we want to avoid it there, we would have to change 785 package.mk files, and also many package definitions in BR2_EXTERNALs that we don't control... That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But doing so *will* require legacy handling. Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs definitions, document that that is the one to use, and wait a year or two before deprecating TARGET_DIR entirely. Yann? Regards, Arnout
Arnout, All, On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly: > On 06-06-18 20:51, Yann E. MORIN wrote: > > But my position has always been consistent on this topic: the images > > that Buildroot generates only ever covers just "basic" situations, using > > a single-filesystem layout. Anything that needs to do a multi-filesystem > > layout should be done as a new filesystem. Doing it in a new filesystem > > is much more flexible than whatever kconfig option we may ever add. And > > since we already have this wonderful flexibility, I don't think it makes > > sense to add a new option that duplicates only a very limited subset of > > that flexibility. That duplication is not good, IMNSHO... > > There is (in my even less humble opinion) one way that we can solve this > generically: by adding a genimage filesystem. genimage is able to create > separate filesystem images for subtrees. so it can cover many use cases in a > generic way. > > There are probably a few gotchas, but I still believe it should be possible. Like tweaking the /etc/fstab accordingly, which genimage does not do on its own? So this "genimage fs" would not be so complex, but not so obvious either, to write, as we'd need to have some additional script that does the fstab tweaking... Probably that info would have to be scrapped out of the genimage.cfg file. Regards, Yann E. MORIN.
Carlos, All, On 2018-06-07 20:12 -0300, Carlos Santos spake thusly: > > From: "Arnout Vandecappelle" <arnout@mind.be> > > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br> > > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org> > > Sent: Thursday, June 7, 2018 6:03:48 PM > > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > > > On 06-06-18 20:51, Yann E. MORIN wrote: > >> But my position has always been consistent on this topic: the images > >> that Buildroot generates only ever covers just "basic" situations, using > >> a single-filesystem layout. Anything that needs to do a multi-filesystem > >> layout should be done as a new filesystem. Doing it in a new filesystem > >> is much more flexible than whatever kconfig option we may ever add. And > >> since we already have this wonderful flexibility, I don't think it makes > >> sense to add a new option that duplicates only a very limited subset of > >> that flexibility. That duplication is not good, IMNSHO... > > > > There is (in my even less humble opinion) one way that we can solve this > > generically: by adding a genimage filesystem. genimage is able to create > > separate filesystem images for subtrees. so it can cover many use cases in a > > generic way. > > I'm already working on a patch to convert inner-rootfs into a generic > inner-filesystem macro. I will submit it for comments soon. Why do you need to do so? You can use the 'rootfs' macro, as I already explained, and for which I have already sent a patch to add it to the manual: https://patchwork.ozlabs.org/patch/926425/ > BTW, it took me some time to understand the dual personality of TARGET_DIR > in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in > the top Makefile with > > TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) > > I understand that this trick avoids changing fs/*/*.mk replacing each > reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces > the readability a lot. I'm compelled to restore it to how it was prior > to commit 7e9870ce32d. But if you revert that, then TARGET_DIR points to the original target/ directory, which does *not* contain the completely-finalised content. Please see the commits around that one for the full picture: git log --oneline 14d43aea0a..543107d390 And see the cover-letter that explains the motivations behind all those changes: http://lists.busybox.net/pipermail/buildroot/2018-March/215450.html Regards, Yann E. MORIN.
Arnout, All, On 2018-06-08 09:48 +0200, Arnout Vandecappelle spake thusly: > On 08-06-18 01:12, Carlos Santos wrote: [--SNIP--] > > BTW, it took me some time to understand the dual personality of TARGET_DIR > > in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in > > the top Makefile with > > > > TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) > > > > I understand that this trick avoids changing fs/*/*.mk replacing each > > reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces > > the readability a lot. I'm compelled to restore it to how it was prior > > to commit 7e9870ce32d. > > The same (or a similar) trick will be applied to the normal packages for > per-package host/staging/target (which is ultimately needed for top-level > parallel build). If we want to avoid it there, we would have to change 785 > package.mk files, and also many package definitions in BR2_EXTERNALs that we > don't control... > > That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But > doing so *will* require legacy handling. > > Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs > definitions, document that that is the one to use, and wait a year or two before > deprecating TARGET_DIR entirely. > > Yann? Well, It is really cumbersome to have to prefix that directory with the package name or the rootfs name... TARGET_DIR is just plainly easy to use. Especially since we would break a long-established variable name. And it is not the only variable that changes its content based on the current package: $(@D) and $(@) et al. Yes, they are make special variables, but so? In the end, I still fail to see the problem. But I am usual pretty stubborn, so... ;-) Regards, Yann E. MORIN.
On 08-06-18 19:34, Yann E. MORIN wrote: > Arnout, All, > > On 2018-06-08 09:48 +0200, Arnout Vandecappelle spake thusly: >> On 08-06-18 01:12, Carlos Santos wrote: > [--SNIP--] >>> BTW, it took me some time to understand the dual personality of TARGET_DIR >>> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in >>> the top Makefile with >>> >>> TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) >>> >>> I understand that this trick avoids changing fs/*/*.mk replacing each >>> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces >>> the readability a lot. I'm compelled to restore it to how it was prior >>> to commit 7e9870ce32d. >> >> The same (or a similar) trick will be applied to the normal packages for >> per-package host/staging/target (which is ultimately needed for top-level >> parallel build). If we want to avoid it there, we would have to change 785 >> package.mk files, and also many package definitions in BR2_EXTERNALs that we >> don't control... >> >> That said, I would also prefer to use explicit $(FOO)_TARGET_DIR variables. But >> doing so *will* require legacy handling. >> >> Maybe we should indeed use explicit ROOTFS_$(ROOTFS)_TARGET_DIR in our rootfs >> definitions, document that that is the one to use, and wait a year or two before >> deprecating TARGET_DIR entirely. >> >> Yann? > > Well, It is really cumbersome to have to prefix that directory with the > package name or the rootfs name... TARGET_DIR is just plainly easy to > use.d I don't agree with that - we use FOO_SOME_VARIABLE all over the place in package foo, so using FOO_TARGET_DIR feels quite natural to me. > Especially since we would break a long-established variable name. That's the worst reason ever :-) Except that indeed there should be a (long) transition period where the old name is still supported, but we don't use it internally. > And it is not the only variable that changes its content based on the > current package: $(@D) and $(@) et al. Yes, they are make special > variables, but so? That makes a huge difference - people know those variables. And anyway, I wouldn't mind to use $(FOO_BUILD_DIR) instead of $(@D). I think that is easier for people to understand. $(@D) just happens to work because the stamp files are created in the build directory, which IMO is an implementation detail that shouldn't leak into the packages. Regards, Arnout > In the end, I still fail to see the problem. But I am usual pretty > stubborn, so... ;-) > > Regards, > Yann E. MORIN. >
On 08-06-18 19:19, Yann E. MORIN wrote: > Arnout, All, > > On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly: >> On 06-06-18 20:51, Yann E. MORIN wrote: >>> But my position has always been consistent on this topic: the images >>> that Buildroot generates only ever covers just "basic" situations, using >>> a single-filesystem layout. Anything that needs to do a multi-filesystem >>> layout should be done as a new filesystem. Doing it in a new filesystem >>> is much more flexible than whatever kconfig option we may ever add. And >>> since we already have this wonderful flexibility, I don't think it makes >>> sense to add a new option that duplicates only a very limited subset of >>> that flexibility. That duplication is not good, IMNSHO... >> >> There is (in my even less humble opinion) one way that we can solve this >> generically: by adding a genimage filesystem. genimage is able to create >> separate filesystem images for subtrees. so it can cover many use cases in a >> generic way. >> >> There are probably a few gotchas, but I still believe it should be possible. > > Like tweaking the /etc/fstab accordingly, which genimage does not do on > its own? No, we don't want to tweak fstab. There is no way to know if you want things to be mounted in that particular way, and whether it should be mounted automatically, and if some flags are needed, and so on. Regards, Arnout > So this "genimage fs" would not be so complex, but not so obvious either, > to write, as we'd need to have some additional script that does the fstab > tweaking... Probably that info would have to be scrapped out of the > genimage.cfg file. > > Regards, > Yann E. MORIN. >
Arnout, Carlos, All, On 2018-06-08 21:59 +0200, Arnout Vandecappelle spake thusly: > On 08-06-18 19:19, Yann E. MORIN wrote: > > On 2018-06-07 23:03 +0200, Arnout Vandecappelle spake thusly: > >> On 06-06-18 20:51, Yann E. MORIN wrote: > >>> But my position has always been consistent on this topic: the images > >>> that Buildroot generates only ever covers just "basic" situations, using > >>> a single-filesystem layout. Anything that needs to do a multi-filesystem > >>> layout should be done as a new filesystem. Doing it in a new filesystem > >>> is much more flexible than whatever kconfig option we may ever add. And > >>> since we already have this wonderful flexibility, I don't think it makes > >>> sense to add a new option that duplicates only a very limited subset of > >>> that flexibility. That duplication is not good, IMNSHO... > >> > >> There is (in my even less humble opinion) one way that we can solve this > >> generically: by adding a genimage filesystem. genimage is able to create > >> separate filesystem images for subtrees. so it can cover many use cases in a > >> generic way. > >> > >> There are probably a few gotchas, but I still believe it should be possible. > > > > Like tweaking the /etc/fstab accordingly, which genimage does not do on > > its own? > > No, we don't want to tweak fstab. There is no way to know if you want things to > be mounted in that particular way, and whether it should be mounted > automatically, and if some flags are needed, and so on. Right, fstab was just an example of site-specific tweaks to be done. Others may want to use startup scripts or systemd units to do the mounts, or to format a partition and rxtract a factory-defaults tarball in that filesystem, or get that from The Cloud, or whatever. Waht I mean is that splitting the filesystem is easy. What is not easy is that each part thus split can be in different formats, located in so different locations, retrieved and accessed in so many different ways, that we can't possibly imagine, and much less cover in a generic way. The only way we can allow people to do what they want, is by providing them with a per-filesysem target/ directory already prepared that they can arrange at will. And as an example of a split-var filesytem, see the attached br2-external tree. It ultimagtely creates $(BINARIES_DIR)/rootfs.meh, a squashfs image of /) and $(BINARIES_DIR)/rootfs.meh-var.tar, a tarball of /var. The only missing pieve is the runtime startup scripts that will format the on-board partition, extact /var.tar in there, and mount it. Regards, Yann E. MORIN.
> From: "Arnout Vandecappelle" <arnout@mind.be> > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br> > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" <buildroot@buildroot.org> > Sent: Thursday, June 7, 2018 6:03:48 PM > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > On 06-06-18 20:51, Yann E. MORIN wrote: >> But my position has always been consistent on this topic: the images >> that Buildroot generates only ever covers just "basic" situations, using >> a single-filesystem layout. Anything that needs to do a multi-filesystem >> layout should be done as a new filesystem. Doing it in a new filesystem >> is much more flexible than whatever kconfig option we may ever add. And >> since we already have this wonderful flexibility, I don't think it makes >> sense to add a new option that duplicates only a very limited subset of >> that flexibility. That duplication is not good, IMNSHO... > > There is (in my even less humble opinion) one way that we can solve this > generically: by adding a genimage filesystem. genimage is able to create > separate filesystem images for subtrees. so it can cover many use cases in a > generic way. > > There are probably a few gotchas, but I still believe it should be possible. Genimage is unable to deal with GPT partitioning, which is much better to use with UEFI BIOS. I dropped it in favour of a custom script that creates an empty disk image file using sfdisk to partition it using a file containing the partitioning scheme. Then it reads the start offset an size of each partition and dumps the corresponding FS image there.
> From: "Yann Morin" <yann.morin.1998@free.fr> > To: "DATACOM" <casantos@datacom.com.br> > Cc: "Arnout Vandecappelle" <arnout@mind.be>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" > <buildroot@buildroot.org> > Sent: Friday, June 8, 2018 2:26:51 PM > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > Carlos, All, > > On 2018-06-07 20:12 -0300, Carlos Santos spake thusly: >> > From: "Arnout Vandecappelle" <arnout@mind.be> >> > To: "Yann Morin" <yann.morin.1998@free.fr>, "DATACOM" <casantos@datacom.com.br> >> > Cc: "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" >> > <buildroot@buildroot.org> >> > Sent: Thursday, June 7, 2018 6:03:48 PM >> > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball >> > extraction >> >> > On 06-06-18 20:51, Yann E. MORIN wrote: >> >> But my position has always been consistent on this topic: the images >> >> that Buildroot generates only ever covers just "basic" situations, using >> >> a single-filesystem layout. Anything that needs to do a multi-filesystem >> >> layout should be done as a new filesystem. Doing it in a new filesystem >> >> is much more flexible than whatever kconfig option we may ever add. And >> >> since we already have this wonderful flexibility, I don't think it makes >> >> sense to add a new option that duplicates only a very limited subset of >> >> that flexibility. That duplication is not good, IMNSHO... >> > >> > There is (in my even less humble opinion) one way that we can solve this >> > generically: by adding a genimage filesystem. genimage is able to create >> > separate filesystem images for subtrees. so it can cover many use cases in a >> > generic way. >> >> I'm already working on a patch to convert inner-rootfs into a generic >> inner-filesystem macro. I will submit it for comments soon. > > Why do you need to do so? You can use the 'rootfs' macro, as I already > explained, and for which I have already sent a patch to add it to the > manual: > > https://patchwork.ozlabs.org/patch/926425/ I've seen your patch and will review it as soon as possible. >> BTW, it took me some time to understand the dual personality of TARGET_DIR >> in fs/common.mk and the role of BASE_TARGET_DIR. Then I found the line in >> the top Makefile with >> >> TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) >> >> I understand that this trick avoids changing fs/*/*.mk replacing each >> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces >> the readability a lot. I'm compelled to restore it to how it was prior >> to commit 7e9870ce32d. > > But if you revert that, then TARGET_DIR points to the original target/ > directory, which does *not* contain the completely-finalised content. I pass TARGET_DIR=$(ROOTFS_COMMON_TARGET_DIR) in the environment to each post-fakeroot script to mimic the previous behavior, since the scripts may take it from the environment variable instead of from the first argument. Anyway, this chaneg is not critical and can be postponed tu reduce the fuzz. > Please see the commits around that one for the full picture: > > git log --oneline 14d43aea0a..543107d390 > > And see the cover-letter that explains the motivations behind all those > changes: > > http://lists.busybox.net/pipermail/buildroot/2018-March/215450.html
Carlos, All, On 2018-06-08 22:06 -0300, Carlos Santos spake thusly: > > From: "Yann Morin" <yann.morin.1998@free.fr> > > Why do you need to do so? You can use the 'rootfs' macro, as I already > > explained, and for which I have already sent a patch to add it to the > > manual: > > https://patchwork.ozlabs.org/patch/926425/ > I've seen your patch and will review it as soon as possible. OK, great. Thanks! :-) [--SNIP--] > >> I understand that this trick avoids changing fs/*/*.mk replacing each > >> reference to TARGET_DIR by a ROOTFS_<FOO>_TARGET_DIR but it reduces > >> the readability a lot. I'm compelled to restore it to how it was prior > >> to commit 7e9870ce32d. > > > > But if you revert that, then TARGET_DIR points to the original target/ > > directory, which does *not* contain the completely-finalised content. > > I pass TARGET_DIR=$(ROOTFS_COMMON_TARGET_DIR) in the environment to No, don't point to the common one, because that is not parallel-safe! The whole change was made so that two filesystems could be build in parallel. Ad=nd since some filesystems want to muck with the layout (e.g. iso9660), that is not compatible with two filesystems building simultaneously from the same directory. At the very least, make TARGET_DIR=$(ROOTFS_$(ROOTFS)_TARGET_DIR) as Arnout hinted previously. Regards, Yann E. MORIN.
> From: "Yann Morin" <yann.morin.1998@free.fr> > To: "Arnout Vandecappelle" <arnout@mind.be> > Cc: "DATACOM" <casantos@datacom.com.br>, "Thomas De Schampheleire" <thomas.de_schampheleire@nokia.com>, "buildroot" > <buildroot@buildroot.org> > Sent: Friday, June 8, 2018 6:06:41 PM > Subject: Re: [Buildroot] [PATCH] fs: allow extra arguments to common tarball extraction > Arnout, Carlos, All, [...] > And as an example of a split-var filesytem, see the attached br2-external > tree. It ultimagtely creates $(BINARIES_DIR)/rootfs.meh, a squashfs > image of /) and $(BINARIES_DIR)/rootfs.meh-var.tar, a tarball of /var. > The only missing pieve is the runtime startup scripts that will format > the on-board partition, extact /var.tar in there, and mount it. Thanks for your example. I transformed it on an example of what I'm attempting to achieve. Please use the contents of the attached file as a br2_external and use the defconfig contained there; then run $ make rootfs-cpio rootfs-var-{lib,log}-ext4 It will create three image files files under $(BINARIES_DIR): rootfs.cpio, rootfs.var-lib.ext4 and rootfs.var-log-ext4. If you apply on buildroot the patch contained there the names will be rootfs.cpio, rootfs-var-lib.ext4 and rootfs-var-log.ext4, which looks better, IMO. Extracting rootfs.common.tar for rootfs-var-log-ext4 is a waste of time and disk space, since it creates an empty filesystem. Perhaps we could add a BR2_TARGET_ROOTFS_<FOO>_START_EMPTY knob for filesystems that populate $(TARGET_DIR) by themselves. The example does not install a /etc/fstab with entries for /var/lib and /var/log but of it would be necessary in a real system, of course.
>>>>> "Carlos" == Carlos Santos <casantos@datacom.com.br> writes: Hi, >> There is (in my even less humble opinion) one way that we can solve this >> generically: by adding a genimage filesystem. genimage is able to create >> separate filesystem images for subtrees. so it can cover many use cases in a >> generic way. >> >> There are probably a few gotchas, but I still believe it should be possible. > Genimage is unable to deal with GPT partitioning, which is much better > to use with UEFI BIOS. I dropped it in favour of a custom script that > creates an empty disk image file using sfdisk to partition it using a > file containing the partitioning scheme. Then it reads the start offset > an size of each partition and dumps the corresponding FS image there. It would really be nice if genimage was extended to also handle GPT partitioning. Did anyone discuss this with upstream?
diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt index bb6d8da6bf..6dd9365723 100644 --- a/docs/manual/customize-rootfs.txt +++ b/docs/manual/customize-rootfs.txt @@ -121,6 +121,16 @@ This method is not recommended because it duplicates the entire skeleton, which prevents taking advantage of the fixes or improvements brought to the default skeleton in later Buildroot releases. +Rootfs tarball extraction arguments (+BR2_ROOTFS_COMMON_UNTAR_ARGS+):: ++ +Buildroot creates a .tar filesystem image and re-extracts it in a private + directory to create each format-specific image. Use this option to pass + extra arguments to tar when the commom root image is extracted. ++ +For example, you can pass "--exclude=\'./var/lib/*\'" to exclude that path from + the rootfs image and use a post-fakeroot script (see below) to create a + separate filesystem image for '/var/lib'. + Post-fakeroot scripts (+BR2_ROOTFS_POST_FAKEROOT_SCRIPT+):: + When aggregating the final images, some parts of the process requires diff --git a/fs/Config.in b/fs/Config.in index c25b01c3de..74487b1172 100644 --- a/fs/Config.in +++ b/fs/Config.in @@ -1,5 +1,18 @@ menu "Filesystem images" +config BR2_ROOTFS_COMMON_UNTAR_ARGS + string "Rootfs tarball extraction arguments" + help + Buildroot creates a .tar filesystem image and re-extracts it + in a private directory to create each format-specific image. + Use this option to pass extra arguments to tar when the commom + root image is extracted. + + For example, you can pass "--exclude='./var/lib/*'" to exclude + that path from the rootfs image and use a post-fakeroot script + (see BR2_ROOTFS_POST_FAKEROOT_SCRIPT) to create a separate + filesystem image for /var/lib. + source "fs/axfs/Config.in" source "fs/cloop/Config.in" source "fs/cpio/Config.in" diff --git a/fs/common.mk b/fs/common.mk index abf35418cb..33ad1ab2a9 100644 --- a/fs/common.mk +++ b/fs/common.mk @@ -50,9 +50,10 @@ define ROOTFS_COMMON_TAR_CMD endef # Command to extract the common tarball into the per-rootfs target directory +ROOTFS_COMMON_UNTAR_ARGS = $(call qstrip,$(BR2_ROOTFS_COMMON_UNTAR_ARGS)) define ROOTFS_COMMON_UNTAR_CMD mkdir -p $(TARGET_DIR) - tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) + tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) $(ROOTFS_COMMON_UNTAR_ARGS) endef .PHONY: rootfs-common
Since commit 118534fe54b (fs: use a common tarball as base for the other filesystems) Buildroot creates a .tar filesystem image and re-extracts it in a private directory to create each format-specific image. Add an option to pass extra arguments to tar when that commom root image is extracted. This option is useful when the root filesystem is volatile (e.g. initrd) or read-only but a read-write subtree is still necessary for persistent data modified by programs as they run. For example, one can pass "--exclude='./var/lib/*'" to exclude that path from the rootfs image and use a post-fakeroot script to make a separate filesystem image for /var/lib. Signed-off-by: Carlos Santos <casantos@datacom.com.br> --- Additional explanation This change solves a real-life problem of my current project. Before buildroot 2018.05 I had to use a trickier approach: 1. Add a post-fakeroot script to 1.1. Create a filesystem image (ext4) from $(TARGET_DIR)/var/lib. 1.2. Move $(TARGET_DIR)/var/lib to a backup path under $(BUILD_DIR). 1.3. Create an empty $(TARGET_DIR)/var/lib. 2. Add a post-image script to 2.1. Copy the rootfs and /var/lib images to the disk image. 2.2. Remove the empty $(TARGET_DIR)/var/lib. 2.3. Restore $(TARGET_DIR)/var/lib from the backup path. That solution is fragile: if something goes wrong during the execution of the post-image script it can lead to a cripled $(TARGET_DIR) with an empty var/lib subdir. Rebuilding a package without restoring the backup can easily cause a kaboom and force me to run a lengthy clean build. I added some protection mechanisms to stop the build if a stale /var/lib backup is found but quite frankly the whole thing stinks. The solution with custom extraction arguments, OTOH, is bullet-proof because it does not change $(TARGET_DIR), so it is always safe to run dirty builds. --- docs/manual/customize-rootfs.txt | 10 ++++++++++ fs/Config.in | 13 +++++++++++++ fs/common.mk | 3 ++- 3 files changed, 25 insertions(+), 1 deletion(-)