docs/manual: document filesystems

Message ID 20180609094647.21460-1-yann.morin.1998@free.fr
State Changes Requested
Headers show
Series
  • docs/manual: document filesystems
Related show

Commit Message

Yann E. MORIN June 9, 2018, 9:46 a.m.
We do expect that users be able to implement new filesystems and
submit them, or have local, custom and private filesystems in
their br2-external trees.

So, add documentation for filesystems, based on the format we have
for packages.

Reported-by: Carlos Santos <casantos@datacom.com.br>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Carlos Santos <casantos@datacom.com.br>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
Changes v1 -> v2:
  - credit Carlos for prompting this addition to the manual. :-)
  - fix the name of the _CMD macro (no trailing 'S')
---
 docs/manual/adding-filesystems.txt    | 119 ++++++++++++++++++++++++++++++++++
 docs/manual/adding-packages-hooks.txt |   7 +-
 docs/manual/adding-packages.txt       |   2 +
 3 files changed, 127 insertions(+), 1 deletion(-)
 create mode 100644 docs/manual/adding-filesystems.txt

Comments

Carlos Santos June 9, 2018, 10:20 p.m. | #1
> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "buildroot" <buildroot@buildroot.org>
> Cc: "Yann Morin" <yann.morin.1998@free.fr>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "DATACOM"
> <casantos@datacom.com.br>, "Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
> Sent: Saturday, June 9, 2018 6:46:47 AM
> Subject: [PATCH] docs/manual: document filesystems

[...]
> 
> diff --git a/docs/manual/adding-filesystems.txt
> b/docs/manual/adding-filesystems.txt
> new file mode 100644
> index 0000000000..4731526c0f
> --- /dev/null
> +++ b/docs/manual/adding-filesystems.txt
> @@ -0,0 +1,119 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +=== Infrastructure for filesystems

Shouldn't this be a separate chapter instead of a section of packages?
Filesystems are different from packages, regardless the syntactical
similarities in the configurations and recipe files. See also the
comment below about the hooks section.

> +
> +As for packages, Buildroot provides an infrastructure to build
> +filesystems.
> +
> +The filesystem infrastructure follows a carefully-crafted sequence:
> +
> +1. Assemble an intermediate, internal tarball from the +target/+
> +   directory:
> +  * create users and groups,
> +  * set the correct rights on files,
> +  * create device nodes,
> +  * create the intermediate tarball;
> +2. Then for each filesystem:
> +  * extract the intermediate tarball to a filesystem-specific
> +    +$(TARGET_DIR)+,
> +  * execute the filesystem pre-gen hooks (see below),
> +  * optionally, touch everything to a specific date, for reproducibility,
> +  * execute the filesystem commands,
> +  * execute the filesystem post-gen hooks (see below),
> +  * remove the filesystem-specific +$(TARGET_DIR)+;
> +3. Remove the intermediate tarball.
> +
> +[[filesystem-tutorial]]
> +==== +filesystem+ tutorial
> +
> +Let's look at an example filesytem. First and foremost, like for a
> +package, create a directory named from your filesystem, e.g. +foo+.
> +
> +Then, create a +Config.in+ file describing the menuconfig entries
> +for the filesystem. Be sure to have that +Config.in+ included from
> ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree).
> +For example:
> +
> +----
> +01: config BR2_TARGET_ROOTFS_FOO
> +02: 	bool "foo root filesystem"
> +03: 	help
> +04: 	  Build a foo root filesystem.
> +05:
> +06: if BR2_TARGET_ROOTFS_FOO
> +07:
> +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR
> +09: 	bool "bar format"
> +10: 	help
> +11: 	  Enable the bar format.
> +12:
> +13: endif
> +----
> +
> +Then, in the same directory, create a +foo.mk+ file describing how
> +to actually assemble the filesystem image from the content of
> ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the
> +Makefile code (automatic for built-in filesystems, not for filesystems
> +in a *br2-external* tree.) For example:
> +
> +----
> +01:
> ################################################################################
> +02: #
> +03: # foo filesystem
> +04: #
> +05:
> ################################################################################
> +06:
> +07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils
> +08:
> +09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
> +10: ROOTFS_FOO_OPTS = --format=bar
> +11: else
> +12: ROOTFS_FOO_OPTS = --format=foo
> +13: endif
> +14:
> +15: define ROOTFS_FOO_CMD
> +16: 	$(HOST_DIR)/bin/mkfs.foo \
> +17: 		--root-dir=$(TARGET_DIR) \
> +18: 		--option-foo --option-bar \
> +19: 		$(ROOTFS_FOO_OPTS) \
> +20: 		--output-file=$(@)
> +21: endef
> +22:
> +23: $(eval $(rootfs))
> +----
> +
> +On line 7, we specify the dependencies on packages needed to build the
> +filesystem. Usually, those dependencies are one or more host-packages
> +that provide the filesystem generator.
> +
> +On lines 9 to 13, we define a conditional set of options, depending on
> +the user's choice.
> +
> +On lines 15 to 21, we define the actual command that will generate the
> +filesystem image.
> +
> +Finally, on line 23, we invoke the +rootfs+ macro that generates all the
> +Makefile rules that actually allow the filesystem to be built.
> +
> +[[filesystem-reference]]
> +==== +filesystem+ reference
> +
> +If one compares a package with a filesystem, one may notice that the
> +variables for a filesystem start with the prefix +ROOTFS_+, when the
> +variables for packages have no prefix.
> +
> +There is a single macro that a filesystem may set:
> +
> +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually
> +  generate the filesystem iamge from the content of the +$(TARGET_DIR)+
> +  directory.
> +
> +Additionally, like for packages, there are a set of hooks; see
> +xref:hooks[] for details.

There are two fundamental differences between packages and filesystem
images:

Filesystem images are always rebuilt and installed while packages are
built/installed once and rebuilt/re-installed only on explicit calls
to "make pkg-name-{reconfigure,rebuild,reinstall}".

Packages are supposed to install, remove or modify files in the global
$(TARGET_DIR) hierarchy. Genarating filesystem images do not modify the
contents of the global $(TARGET_DIR) and user-provided rootfs extensions
must not do this.

[Side note: we could add an optional "paranoid mode" config which would
trigger a checksum generation/checking right before/after the creation
of the rootfs images.] 

> +And here is a list of variables that may be used (but may *not* be set):

"must" not be set.

> +
> +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
> +
> +* +$(@)+ is the filename where to store the filesystem image into.
> diff --git a/docs/manual/adding-packages-hooks.txt
> b/docs/manual/adding-packages-hooks.txt
> index 0ce79f8907..34d5801169 100644
> --- a/docs/manual/adding-packages-hooks.txt
> +++ b/docs/manual/adding-packages-hooks.txt

Consider moving the hooks section to a separate chapter if you put the
filesystem text in a separate chapter too.

[...]
Yann E. MORIN June 10, 2018, 8:12 a.m. | #2
Carlos, All,

Thanks for the review! :-)

On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
[--SNIP--]
> > diff --git a/docs/manual/adding-filesystems.txt
> > b/docs/manual/adding-filesystems.txt
> > new file mode 100644
> > index 0000000000..4731526c0f
> > --- /dev/null
> > +++ b/docs/manual/adding-filesystems.txt
> > @@ -0,0 +1,119 @@
> > +// -*- mode:doc; -*-
> > +// vim: set syntax=asciidoc:
> > +
> > +=== Infrastructure for filesystems
> 
> Shouldn't this be a separate chapter instead of a section of packages?
> Filesystems are different from packages, regardless the syntactical
> similarities in the configurations and recipe files. See also the
> comment below about the hooks section.

I also wondered about that.

But I was a bit wary of creating another chapter just for filesystems.

And notice that in the package chapter, we also talk about the
'asciidoc-document' infra, which is already not a package infra.

I'd like input from others. Thomas, Peter, Arnout?

[--SNIP--]
> There are two fundamental differences between packages and filesystem
> images:
> 
> Filesystem images are always rebuilt and installed while packages are
> built/installed once and rebuilt/re-installed only on explicit calls
> to "make pkg-name-{reconfigure,rebuild,reinstall}".

ACK, good point. I'll add this to the first section, that explains the
infra.

> Packages are supposed to install, remove or modify files in the global
> $(TARGET_DIR) hierarchy.

Well, in the first part of this patch, I added a description of the
sequence, making it obvious (or so I hoped) that indeed filesystems were
acting on a transient copy:

    2.  Then for each filesystem:
      * extract the intermediate tarball to a filesystem-specific $(TARGET_DIR),
      [...]
      * remove the filesystem-specific $(TARGET_DIR);

> Genarating filesystem images do not modify the
> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
> must not do this.

Why do you insist on modifying the global target/ directory from your
filesystem? If you do that, and build two or more filesystems [*], they
might each try to modify target/ is incompatible ways, and this is not
even safe, concurrency-wise (e.g. ony may try to archive the content of
/var while another is doing a rm -rf on it.)

The filesystem infra has been changed to provide each filesystem a copy
of the target/ directory, so that each filesystem can do their own
tweaks without impacting the others, without the need to be idempotent,
and allowing filesystems to be run in parallel.

I have still not seen a *compelling* reason to let filesystems muck in
the original target/ directory.

[*] Buildroot terminology, like BR2_TARGET_ROOTFS_SQUASHFS=y and
BR2_TARGET_ROOTFS_ISO9660=y at the same time.

> [Side note: we could add an optional "paranoid mode" config which would
> trigger a checksum generation/checking right before/after the creation
> of the rootfs images.] 

Not needed, since there is no reason to let them do.

> > +And here is a list of variables that may be used (but may *not* be set):
> "must" not be set.

https://www.merriam-webster.com/dictionary/may

4. : SHALL, MUST — used in law where the sense, purpose, or policy
     requires this interpretation

(yes, I tend to use a pedant or/and archaic english. ;-])

> > +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
> > +
> > +* +$(@)+ is the filename where to store the filesystem image into.
> > diff --git a/docs/manual/adding-packages-hooks.txt
> > b/docs/manual/adding-packages-hooks.txt
> > index 0ce79f8907..34d5801169 100644
> > --- a/docs/manual/adding-packages-hooks.txt
> > +++ b/docs/manual/adding-packages-hooks.txt
> 
> Consider moving the hooks section to a separate chapter if you put the
> filesystem text in a separate chapter too.

If others agreee that the filesystem infra requires its own chapter,
then I will, yes.

Thanks again for the review! :-)

Regards,
Yann E. MORIN.
Yann E. MORIN June 10, 2018, 8:35 a.m. | #3
Carlos, Arnout, All,

On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
> On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
[--SNIP--]
> > Packages are supposed to install, remove or modify files in the global
> > $(TARGET_DIR) hierarchy.

I forgot to say that this will no longer be true when we (eventually)
introduce the top-level parallel build, because, as Arnout already said,
each package will install in its own, private copy of TARGET_DIR, not
only that, but also its private STAGING_DIR and HOST_DIR as well.

One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR,
in favour of a package-prefix FOO_TARGET_DIR (amd similarly for
filesystems).

However I still don't think that is a good idea, because that would
allow packages to easily find and write into another package's
TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder
(but not impossible, agreed).

Ditto filesystems.

So I believe we still want to keep a context-specific TARGET_DIR. It is
easier to use, it is more stable, we can update it at will, and it makes
it harder for packages and filesystems to do weird stuff.

Regards,
Yann E. MORIN.
Thomas Petazzoni June 10, 2018, 10:53 a.m. | #4
Hello,

On Sun, 10 Jun 2018 10:12:17 +0200, Yann E. MORIN wrote:

> I also wondered about that.
> 
> But I was a bit wary of creating another chapter just for filesystems.
> 
> And notice that in the package chapter, we also talk about the
> 'asciidoc-document' infra, which is already not a package infra.
> 
> I'd like input from others. Thomas, Peter, Arnout?

The section about packages is already very long, so I wouldn't mind
seeing other sections being created outside of it.

Thomas
Carlos Santos June 10, 2018, 1:38 p.m. | #5
> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "DATACOM" <casantos@datacom.com.br>
> Cc: "buildroot" <buildroot@buildroot.org>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "Thomas Petazzoni"
> <thomas.petazzoni@bootlin.com>, "Peter Korsgaard" <peter@korsgaard.com>, "Arnout Vandecappelle" <arnout@mind.be>
> Sent: Sunday, June 10, 2018 5:12:17 AM
> Subject: Re: [PATCH] docs/manual: document filesystems

> Carlos, All,
> 
> Thanks for the review! :-)
> 

[...]

>> Genarating filesystem images do not modify the
>> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
>> must not do this.
> 
> Why do you insist on modifying the global target/ directory from your
> filesystem? If you do that, and build two or more filesystems [*], they
> might each try to modify target/ is incompatible ways, and this is not
> even safe, concurrency-wise (e.g. ony may try to archive the content of
> /var while another is doing a rm -rf on it.)

I never intended to modify the global $(TARGET_DIR).
Yann E. MORIN June 10, 2018, 3:07 p.m. | #6
Carlos, All,

On 2018-06-10 10:38 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
> >> Genarating filesystem images do not modify the
> >> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
> >> must not do this.
[--SNIP--]
> > Why do you insist on modifying the global target/ directory from your
> > filesystem? If you do that, and build two or more filesystems [*], they
> > might each try to modify target/ is incompatible ways, and this is not
> > even safe, concurrency-wise (e.g. ony may try to archive the content of
> > /var while another is doing a rm -rf on it.)
> 
> I never intended to modify the global $(TARGET_DIR).

Sorry, I mis-interpreted your comment above, as "user-provided rootfs []
must not do this" to which I appended in my head "but that is exactly
what I want to do in my case."

I'm sorry, I should be more careful in my interpretations in the future.

So, I'll take that as a hint to further improve the section that
explains how filesystems are generated, to more carefully describe and
explain the sequence.

Thanks again for the feedback! :-)

Regards,
Yann E. MORIN.
Arnout Vandecappelle June 10, 2018, 9:26 p.m. | #7
On 10-06-18 10:35, Yann E. MORIN wrote:
> Carlos, Arnout, All,
> 
> On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
>> On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
> [--SNIP--]
>>> Packages are supposed to install, remove or modify files in the global
>>> $(TARGET_DIR) hierarchy.
> 
> I forgot to say that this will no longer be true when we (eventually)
> introduce the top-level parallel build, because, as Arnout already said,
> each package will install in its own, private copy of TARGET_DIR, not
> only that, but also its private STAGING_DIR and HOST_DIR as well.
> 
> One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR,
> in favour of a package-prefix FOO_TARGET_DIR (amd similarly for
> filesystems).
> 
> However I still don't think that is a good idea, because that would
> allow packages to easily find and write into another package's
> TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder
> (but not impossible, agreed).

 How does the definition of TARGET_DIR have any influence on how fs/foo/foo.mk
uses ROOTFS_BAR_TARGET_DIR?


> Ditto filesystems.
> 
> So I believe we still want to keep a context-specific TARGET_DIR. It is
> easier to use,

 TARGET_DIR is only marginally easier to use than FOO_TARGET_DIR...


> it is more stable,

 ???

 You actually may have a point here, but I don't understand it.

> we can update it at will

 We can also update the definition of FOO_TARGET_DIR at will.


, and it makes it harder for packages and filesystems to do weird stuff.

 No it doesn't.


 Regards,
 Arnout

Patch

diff --git a/docs/manual/adding-filesystems.txt b/docs/manual/adding-filesystems.txt
new file mode 100644
index 0000000000..4731526c0f
--- /dev/null
+++ b/docs/manual/adding-filesystems.txt
@@ -0,0 +1,119 @@ 
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+=== Infrastructure for filesystems
+
+As for packages, Buildroot provides an infrastructure to build
+filesystems.
+
+The filesystem infrastructure follows a carefully-crafted sequence:
+
+1. Assemble an intermediate, internal tarball from the +target/+
+   directory:
+  * create users and groups,
+  * set the correct rights on files,
+  * create device nodes,
+  * create the intermediate tarball;
+2. Then for each filesystem:
+  * extract the intermediate tarball to a filesystem-specific
+    +$(TARGET_DIR)+,
+  * execute the filesystem pre-gen hooks (see below),
+  * optionally, touch everything to a specific date, for reproducibility,
+  * execute the filesystem commands,
+  * execute the filesystem post-gen hooks (see below),
+  * remove the filesystem-specific +$(TARGET_DIR)+;
+3. Remove the intermediate tarball.
+
+[[filesystem-tutorial]]
+==== +filesystem+ tutorial
+
+Let's look at an example filesytem. First and foremost, like for a
+package, create a directory named from your filesystem, e.g. +foo+.
+
+Then, create a +Config.in+ file describing the menuconfig entries
+for the filesystem. Be sure to have that +Config.in+ included from
++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree).
+For example:
+
+----
+01: config BR2_TARGET_ROOTFS_FOO
+02: 	bool "foo root filesystem"
+03: 	help
+04: 	  Build a foo root filesystem.
+05: 
+06: if BR2_TARGET_ROOTFS_FOO
+07: 
+08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR
+09: 	bool "bar format"
+10: 	help
+11: 	  Enable the bar format.
+12: 
+13: endif
+----
+
+Then, in the same directory, create a +foo.mk+ file describing how
+to actually assemble the filesystem image from the content of
++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the
+Makefile code (automatic for built-in filesystems, not for filesystems
+in a *br2-external* tree.) For example:
+
+----
+01: ################################################################################
+02: #
+03: # foo filesystem
+04: #
+05: ################################################################################
+06: 
+07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils
+08: 
+09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
+10: ROOTFS_FOO_OPTS = --format=bar
+11: else
+12: ROOTFS_FOO_OPTS = --format=foo
+13: endif
+14: 
+15: define ROOTFS_FOO_CMD
+16: 	$(HOST_DIR)/bin/mkfs.foo \
+17: 		--root-dir=$(TARGET_DIR) \
+18: 		--option-foo --option-bar \
+19: 		$(ROOTFS_FOO_OPTS) \
+20: 		--output-file=$(@)
+21: endef
+22: 
+23: $(eval $(rootfs))
+----
+
+On line 7, we specify the dependencies on packages needed to build the
+filesystem. Usually, those dependencies are one or more host-packages
+that provide the filesystem generator.
+
+On lines 9 to 13, we define a conditional set of options, depending on
+the user's choice.
+
+On lines 15 to 21, we define the actual command that will generate the
+filesystem image.
+
+Finally, on line 23, we invoke the +rootfs+ macro that generates all the
+Makefile rules that actually allow the filesystem to be built.
+
+[[filesystem-reference]]
+==== +filesystem+ reference
+
+If one compares a package with a filesystem, one may notice that the
+variables for a filesystem start with the prefix +ROOTFS_+, when the
+variables for packages have no prefix.
+
+There is a single macro that a filesystem may set:
+
+* +ROOTFS_FOO_CMD+ contains the commands to execute to actually
+  generate the filesystem iamge from the content of the +$(TARGET_DIR)+
+  directory.
+
+Additionally, like for packages, there are a set of hooks; see
+xref:hooks[] for details.
+
+And here is a list of variables that may be used (but may *not* be set):
+
+* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
+
+* +$(@)+ is the filename where to store the filesystem image into.
diff --git a/docs/manual/adding-packages-hooks.txt b/docs/manual/adding-packages-hooks.txt
index 0ce79f8907..34d5801169 100644
--- a/docs/manual/adding-packages-hooks.txt
+++ b/docs/manual/adding-packages-hooks.txt
@@ -11,7 +11,7 @@  Most hooks aren't really useful for generic packages, since the +.mk+
 file already has full control over the actions performed in each step
 of the package construction.
 
-The following hook points are available:
+The following hook points are available for packages:
 
 * +LIBFOO_PRE_DOWNLOAD_HOOKS+
 * +LIBFOO_POST_DOWNLOAD_HOOKS+
@@ -46,6 +46,11 @@  The following hook points are available:
 * +LIBFOO_PRE_LEGAL_INFO_HOOKS+
 * +LIBFOO_POST_LEGAL_INFO_HOOKS+
 
+The following hook points are available for filesystems:
+
+* +ROOTFS_FOO_PRE_GEN_HOOKS+
+* +ROOTFS_FOO_POST_GEN_HOOKS+
+
 These variables are 'lists' of variable names containing actions to be
 performed at this hook point. This allows several hooks to be
 registered at a given hook point. Here is an example:
diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt
index 4a4a17e879..83c93f3dde 100644
--- a/docs/manual/adding-packages.txt
+++ b/docs/manual/adding-packages.txt
@@ -47,6 +47,8 @@  include::adding-packages-asciidoc.txt[]
 
 include::adding-packages-linux-kernel-spec-infra.txt[]
 
+include::adding-filesystems.txt[]
+
 include::adding-packages-hooks.txt[]
 
 include::adding-packages-gettext.txt[]