| Message ID | 20180609094647.21460-1-yann.morin.1998@free.fr |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | docs/manual: document filesystems | expand |
> From: "Yann Morin" <yann.morin.1998@free.fr> > To: "buildroot" <buildroot@buildroot.org> > Cc: "Yann Morin" <yann.morin.1998@free.fr>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "DATACOM" > <casantos@datacom.com.br>, "Thomas Petazzoni" <thomas.petazzoni@bootlin.com> > Sent: Saturday, June 9, 2018 6:46:47 AM > Subject: [PATCH] docs/manual: document filesystems [...] > > diff --git a/docs/manual/adding-filesystems.txt > b/docs/manual/adding-filesystems.txt > new file mode 100644 > index 0000000000..4731526c0f > --- /dev/null > +++ b/docs/manual/adding-filesystems.txt > @@ -0,0 +1,119 @@ > +// -*- mode:doc; -*- > +// vim: set syntax=asciidoc: > + > +=== Infrastructure for filesystems Shouldn't this be a separate chapter instead of a section of packages? Filesystems are different from packages, regardless the syntactical similarities in the configurations and recipe files. See also the comment below about the hooks section. > + > +As for packages, Buildroot provides an infrastructure to build > +filesystems. > + > +The filesystem infrastructure follows a carefully-crafted sequence: > + > +1. Assemble an intermediate, internal tarball from the +target/+ > + directory: > + * create users and groups, > + * set the correct rights on files, > + * create device nodes, > + * create the intermediate tarball; > +2. Then for each filesystem: > + * extract the intermediate tarball to a filesystem-specific > + +$(TARGET_DIR)+, > + * execute the filesystem pre-gen hooks (see below), > + * optionally, touch everything to a specific date, for reproducibility, > + * execute the filesystem commands, > + * execute the filesystem post-gen hooks (see below), > + * remove the filesystem-specific +$(TARGET_DIR)+; > +3. Remove the intermediate tarball. > + > +[[filesystem-tutorial]] > +==== +filesystem+ tutorial > + > +Let's look at an example filesytem. First and foremost, like for a > +package, create a directory named from your filesystem, e.g. +foo+. > + > +Then, create a +Config.in+ file describing the menuconfig entries > +for the filesystem. Be sure to have that +Config.in+ included from > ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree). > +For example: > + > +---- > +01: config BR2_TARGET_ROOTFS_FOO > +02: bool "foo root filesystem" > +03: help > +04: Build a foo root filesystem. > +05: > +06: if BR2_TARGET_ROOTFS_FOO > +07: > +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR > +09: bool "bar format" > +10: help > +11: Enable the bar format. > +12: > +13: endif > +---- > + > +Then, in the same directory, create a +foo.mk+ file describing how > +to actually assemble the filesystem image from the content of > ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the > +Makefile code (automatic for built-in filesystems, not for filesystems > +in a *br2-external* tree.) For example: > + > +---- > +01: > ################################################################################ > +02: # > +03: # foo filesystem > +04: # > +05: > ################################################################################ > +06: > +07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils > +08: > +09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y) > +10: ROOTFS_FOO_OPTS = --format=bar > +11: else > +12: ROOTFS_FOO_OPTS = --format=foo > +13: endif > +14: > +15: define ROOTFS_FOO_CMD > +16: $(HOST_DIR)/bin/mkfs.foo \ > +17: --root-dir=$(TARGET_DIR) \ > +18: --option-foo --option-bar \ > +19: $(ROOTFS_FOO_OPTS) \ > +20: --output-file=$(@) > +21: endef > +22: > +23: $(eval $(rootfs)) > +---- > + > +On line 7, we specify the dependencies on packages needed to build the > +filesystem. Usually, those dependencies are one or more host-packages > +that provide the filesystem generator. > + > +On lines 9 to 13, we define a conditional set of options, depending on > +the user's choice. > + > +On lines 15 to 21, we define the actual command that will generate the > +filesystem image. > + > +Finally, on line 23, we invoke the +rootfs+ macro that generates all the > +Makefile rules that actually allow the filesystem to be built. > + > +[[filesystem-reference]] > +==== +filesystem+ reference > + > +If one compares a package with a filesystem, one may notice that the > +variables for a filesystem start with the prefix +ROOTFS_+, when the > +variables for packages have no prefix. > + > +There is a single macro that a filesystem may set: > + > +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually > + generate the filesystem iamge from the content of the +$(TARGET_DIR)+ > + directory. > + > +Additionally, like for packages, there are a set of hooks; see > +xref:hooks[] for details. There are two fundamental differences between packages and filesystem images: Filesystem images are always rebuilt and installed while packages are built/installed once and rebuilt/re-installed only on explicit calls to "make pkg-name-{reconfigure,rebuild,reinstall}". Packages are supposed to install, remove or modify files in the global $(TARGET_DIR) hierarchy. Genarating filesystem images do not modify the contents of the global $(TARGET_DIR) and user-provided rootfs extensions must not do this. [Side note: we could add an optional "paranoid mode" config which would trigger a checksum generation/checking right before/after the creation of the rootfs images.] > +And here is a list of variables that may be used (but may *not* be set): "must" not be set. > + > +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. > + > +* +$(@)+ is the filename where to store the filesystem image into. > diff --git a/docs/manual/adding-packages-hooks.txt > b/docs/manual/adding-packages-hooks.txt > index 0ce79f8907..34d5801169 100644 > --- a/docs/manual/adding-packages-hooks.txt > +++ b/docs/manual/adding-packages-hooks.txt Consider moving the hooks section to a separate chapter if you put the filesystem text in a separate chapter too. [...]
Carlos, All, Thanks for the review! :-) On 2018-06-09 19:20 -0300, Carlos Santos spake thusly: > > From: "Yann Morin" <yann.morin.1998@free.fr> [--SNIP--] > > diff --git a/docs/manual/adding-filesystems.txt > > b/docs/manual/adding-filesystems.txt > > new file mode 100644 > > index 0000000000..4731526c0f > > --- /dev/null > > +++ b/docs/manual/adding-filesystems.txt > > @@ -0,0 +1,119 @@ > > +// -*- mode:doc; -*- > > +// vim: set syntax=asciidoc: > > + > > +=== Infrastructure for filesystems > > Shouldn't this be a separate chapter instead of a section of packages? > Filesystems are different from packages, regardless the syntactical > similarities in the configurations and recipe files. See also the > comment below about the hooks section. I also wondered about that. But I was a bit wary of creating another chapter just for filesystems. And notice that in the package chapter, we also talk about the 'asciidoc-document' infra, which is already not a package infra. I'd like input from others. Thomas, Peter, Arnout? [--SNIP--] > There are two fundamental differences between packages and filesystem > images: > > Filesystem images are always rebuilt and installed while packages are > built/installed once and rebuilt/re-installed only on explicit calls > to "make pkg-name-{reconfigure,rebuild,reinstall}". ACK, good point. I'll add this to the first section, that explains the infra. > Packages are supposed to install, remove or modify files in the global > $(TARGET_DIR) hierarchy. Well, in the first part of this patch, I added a description of the sequence, making it obvious (or so I hoped) that indeed filesystems were acting on a transient copy: 2. Then for each filesystem: * extract the intermediate tarball to a filesystem-specific $(TARGET_DIR), [...] * remove the filesystem-specific $(TARGET_DIR); > Genarating filesystem images do not modify the > contents of the global $(TARGET_DIR) and user-provided rootfs extensions > must not do this. Why do you insist on modifying the global target/ directory from your filesystem? If you do that, and build two or more filesystems [*], they might each try to modify target/ is incompatible ways, and this is not even safe, concurrency-wise (e.g. ony may try to archive the content of /var while another is doing a rm -rf on it.) The filesystem infra has been changed to provide each filesystem a copy of the target/ directory, so that each filesystem can do their own tweaks without impacting the others, without the need to be idempotent, and allowing filesystems to be run in parallel. I have still not seen a *compelling* reason to let filesystems muck in the original target/ directory. [*] Buildroot terminology, like BR2_TARGET_ROOTFS_SQUASHFS=y and BR2_TARGET_ROOTFS_ISO9660=y at the same time. > [Side note: we could add an optional "paranoid mode" config which would > trigger a checksum generation/checking right before/after the creation > of the rootfs images.] Not needed, since there is no reason to let them do. > > +And here is a list of variables that may be used (but may *not* be set): > "must" not be set. https://www.merriam-webster.com/dictionary/may 4. : SHALL, MUST — used in law where the sense, purpose, or policy requires this interpretation (yes, I tend to use a pedant or/and archaic english. ;-]) > > +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. > > + > > +* +$(@)+ is the filename where to store the filesystem image into. > > diff --git a/docs/manual/adding-packages-hooks.txt > > b/docs/manual/adding-packages-hooks.txt > > index 0ce79f8907..34d5801169 100644 > > --- a/docs/manual/adding-packages-hooks.txt > > +++ b/docs/manual/adding-packages-hooks.txt > > Consider moving the hooks section to a separate chapter if you put the > filesystem text in a separate chapter too. If others agreee that the filesystem infra requires its own chapter, then I will, yes. Thanks again for the review! :-) Regards, Yann E. MORIN.
Carlos, Arnout, All, On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly: > On 2018-06-09 19:20 -0300, Carlos Santos spake thusly: [--SNIP--] > > Packages are supposed to install, remove or modify files in the global > > $(TARGET_DIR) hierarchy. I forgot to say that this will no longer be true when we (eventually) introduce the top-level parallel build, because, as Arnout already said, each package will install in its own, private copy of TARGET_DIR, not only that, but also its private STAGING_DIR and HOST_DIR as well. One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR, in favour of a package-prefix FOO_TARGET_DIR (amd similarly for filesystems). However I still don't think that is a good idea, because that would allow packages to easily find and write into another package's TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder (but not impossible, agreed). Ditto filesystems. So I believe we still want to keep a context-specific TARGET_DIR. It is easier to use, it is more stable, we can update it at will, and it makes it harder for packages and filesystems to do weird stuff. Regards, Yann E. MORIN.
Hello, On Sun, 10 Jun 2018 10:12:17 +0200, Yann E. MORIN wrote: > I also wondered about that. > > But I was a bit wary of creating another chapter just for filesystems. > > And notice that in the package chapter, we also talk about the > 'asciidoc-document' infra, which is already not a package infra. > > I'd like input from others. Thomas, Peter, Arnout? The section about packages is already very long, so I wouldn't mind seeing other sections being created outside of it. Thomas
> From: "Yann Morin" <yann.morin.1998@free.fr> > To: "DATACOM" <casantos@datacom.com.br> > Cc: "buildroot" <buildroot@buildroot.org>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "Thomas Petazzoni" > <thomas.petazzoni@bootlin.com>, "Peter Korsgaard" <peter@korsgaard.com>, "Arnout Vandecappelle" <arnout@mind.be> > Sent: Sunday, June 10, 2018 5:12:17 AM > Subject: Re: [PATCH] docs/manual: document filesystems > Carlos, All, > > Thanks for the review! :-) > [...] >> Genarating filesystem images do not modify the >> contents of the global $(TARGET_DIR) and user-provided rootfs extensions >> must not do this. > > Why do you insist on modifying the global target/ directory from your > filesystem? If you do that, and build two or more filesystems [*], they > might each try to modify target/ is incompatible ways, and this is not > even safe, concurrency-wise (e.g. ony may try to archive the content of > /var while another is doing a rm -rf on it.) I never intended to modify the global $(TARGET_DIR).
Carlos, All, On 2018-06-10 10:38 -0300, Carlos Santos spake thusly: > > From: "Yann Morin" <yann.morin.1998@free.fr> > >> Genarating filesystem images do not modify the > >> contents of the global $(TARGET_DIR) and user-provided rootfs extensions > >> must not do this. [--SNIP--] > > Why do you insist on modifying the global target/ directory from your > > filesystem? If you do that, and build two or more filesystems [*], they > > might each try to modify target/ is incompatible ways, and this is not > > even safe, concurrency-wise (e.g. ony may try to archive the content of > > /var while another is doing a rm -rf on it.) > > I never intended to modify the global $(TARGET_DIR). Sorry, I mis-interpreted your comment above, as "user-provided rootfs [] must not do this" to which I appended in my head "but that is exactly what I want to do in my case." I'm sorry, I should be more careful in my interpretations in the future. So, I'll take that as a hint to further improve the section that explains how filesystems are generated, to more carefully describe and explain the sequence. Thanks again for the feedback! :-) Regards, Yann E. MORIN.
On 10-06-18 10:35, Yann E. MORIN wrote: > Carlos, Arnout, All, > > On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly: >> On 2018-06-09 19:20 -0300, Carlos Santos spake thusly: > [--SNIP--] >>> Packages are supposed to install, remove or modify files in the global >>> $(TARGET_DIR) hierarchy. > > I forgot to say that this will no longer be true when we (eventually) > introduce the top-level parallel build, because, as Arnout already said, > each package will install in its own, private copy of TARGET_DIR, not > only that, but also its private STAGING_DIR and HOST_DIR as well. > > One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR, > in favour of a package-prefix FOO_TARGET_DIR (amd similarly for > filesystems). > > However I still don't think that is a good idea, because that would > allow packages to easily find and write into another package's > TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder > (but not impossible, agreed). How does the definition of TARGET_DIR have any influence on how fs/foo/foo.mk uses ROOTFS_BAR_TARGET_DIR? > Ditto filesystems. > > So I believe we still want to keep a context-specific TARGET_DIR. It is > easier to use, TARGET_DIR is only marginally easier to use than FOO_TARGET_DIR... > it is more stable, ??? You actually may have a point here, but I don't understand it. > we can update it at will We can also update the definition of FOO_TARGET_DIR at will. , and it makes it harder for packages and filesystems to do weird stuff. No it doesn't. Regards, Arnout
diff --git a/docs/manual/adding-filesystems.txt b/docs/manual/adding-filesystems.txt new file mode 100644 index 0000000000..4731526c0f --- /dev/null +++ b/docs/manual/adding-filesystems.txt @@ -0,0 +1,119 @@ +// -*- mode:doc; -*- +// vim: set syntax=asciidoc: + +=== Infrastructure for filesystems + +As for packages, Buildroot provides an infrastructure to build +filesystems. + +The filesystem infrastructure follows a carefully-crafted sequence: + +1. Assemble an intermediate, internal tarball from the +target/+ + directory: + * create users and groups, + * set the correct rights on files, + * create device nodes, + * create the intermediate tarball; +2. Then for each filesystem: + * extract the intermediate tarball to a filesystem-specific + +$(TARGET_DIR)+, + * execute the filesystem pre-gen hooks (see below), + * optionally, touch everything to a specific date, for reproducibility, + * execute the filesystem commands, + * execute the filesystem post-gen hooks (see below), + * remove the filesystem-specific +$(TARGET_DIR)+; +3. Remove the intermediate tarball. + +[[filesystem-tutorial]] +==== +filesystem+ tutorial + +Let's look at an example filesytem. First and foremost, like for a +package, create a directory named from your filesystem, e.g. +foo+. + +Then, create a +Config.in+ file describing the menuconfig entries +for the filesystem. Be sure to have that +Config.in+ included from ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree). +For example: + +---- +01: config BR2_TARGET_ROOTFS_FOO +02: bool "foo root filesystem" +03: help +04: Build a foo root filesystem. +05: +06: if BR2_TARGET_ROOTFS_FOO +07: +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR +09: bool "bar format" +10: help +11: Enable the bar format. +12: +13: endif +---- + +Then, in the same directory, create a +foo.mk+ file describing how +to actually assemble the filesystem image from the content of ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the +Makefile code (automatic for built-in filesystems, not for filesystems +in a *br2-external* tree.) For example: + +---- +01: ################################################################################ +02: # +03: # foo filesystem +04: # +05: ################################################################################ +06: +07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils +08: +09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y) +10: ROOTFS_FOO_OPTS = --format=bar +11: else +12: ROOTFS_FOO_OPTS = --format=foo +13: endif +14: +15: define ROOTFS_FOO_CMD +16: $(HOST_DIR)/bin/mkfs.foo \ +17: --root-dir=$(TARGET_DIR) \ +18: --option-foo --option-bar \ +19: $(ROOTFS_FOO_OPTS) \ +20: --output-file=$(@) +21: endef +22: +23: $(eval $(rootfs)) +---- + +On line 7, we specify the dependencies on packages needed to build the +filesystem. Usually, those dependencies are one or more host-packages +that provide the filesystem generator. + +On lines 9 to 13, we define a conditional set of options, depending on +the user's choice. + +On lines 15 to 21, we define the actual command that will generate the +filesystem image. + +Finally, on line 23, we invoke the +rootfs+ macro that generates all the +Makefile rules that actually allow the filesystem to be built. + +[[filesystem-reference]] +==== +filesystem+ reference + +If one compares a package with a filesystem, one may notice that the +variables for a filesystem start with the prefix +ROOTFS_+, when the +variables for packages have no prefix. + +There is a single macro that a filesystem may set: + +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually + generate the filesystem iamge from the content of the +$(TARGET_DIR)+ + directory. + +Additionally, like for packages, there are a set of hooks; see +xref:hooks[] for details. + +And here is a list of variables that may be used (but may *not* be set): + +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. + +* +$(@)+ is the filename where to store the filesystem image into. diff --git a/docs/manual/adding-packages-hooks.txt b/docs/manual/adding-packages-hooks.txt index 0ce79f8907..34d5801169 100644 --- a/docs/manual/adding-packages-hooks.txt +++ b/docs/manual/adding-packages-hooks.txt @@ -11,7 +11,7 @@ Most hooks aren't really useful for generic packages, since the +.mk+ file already has full control over the actions performed in each step of the package construction. -The following hook points are available: +The following hook points are available for packages: * +LIBFOO_PRE_DOWNLOAD_HOOKS+ * +LIBFOO_POST_DOWNLOAD_HOOKS+ @@ -46,6 +46,11 @@ The following hook points are available: * +LIBFOO_PRE_LEGAL_INFO_HOOKS+ * +LIBFOO_POST_LEGAL_INFO_HOOKS+ +The following hook points are available for filesystems: + +* +ROOTFS_FOO_PRE_GEN_HOOKS+ +* +ROOTFS_FOO_POST_GEN_HOOKS+ + These variables are 'lists' of variable names containing actions to be performed at this hook point. This allows several hooks to be registered at a given hook point. Here is an example: diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt index 4a4a17e879..83c93f3dde 100644 --- a/docs/manual/adding-packages.txt +++ b/docs/manual/adding-packages.txt @@ -47,6 +47,8 @@ include::adding-packages-asciidoc.txt[] include::adding-packages-linux-kernel-spec-infra.txt[] +include::adding-filesystems.txt[] + include::adding-packages-hooks.txt[] include::adding-packages-gettext.txt[]
We do expect that users be able to implement new filesystems and submit them, or have local, custom and private filesystems in their br2-external trees. So, add documentation for filesystems, based on the format we have for packages. Reported-by: Carlos Santos <casantos@datacom.com.br> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Cc: Carlos Santos <casantos@datacom.com.br> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- Changes v1 -> v2: - credit Carlos for prompting this addition to the manual. :-) - fix the name of the _CMD macro (no trailing 'S') --- docs/manual/adding-filesystems.txt | 119 ++++++++++++++++++++++++++++++++++ docs/manual/adding-packages-hooks.txt | 7 +- docs/manual/adding-packages.txt | 2 + 3 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 docs/manual/adding-filesystems.txt