[2/5] package/rpi-firmware: add option to install firmware files in target/boot/

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

The firmware files must reside in a FAT filesystem, in the first partition
of the SDcard, so it makes sense to install them in target/ (eg. for a
post-image script to generate the different partitions from the rootfs.tar).

Add an option (off by default) to install the bootloader files into
$(TARGET_DIR)/boot .

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
Note: this will come very handy when we eventually have our generic
multi-image generation infra, to come in a following patch.
---
 package/rpi-firmware/Config.in       |  9 +++++++++
 package/rpi-firmware/rpi-firmware.mk | 16 +++++++++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)

Dear Yann E. MORIN,

On Fri, 22 Nov 2013 23:50:55 +0100, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> The firmware files must reside in a FAT filesystem, in the first partition
> of the SDcard, so it makes sense to install them in target/ (eg. for a
> post-image script to generate the different partitions from the rootfs.tar).

I don't understand the reasoning here. These files must be at the root
of a specific FAT filesystem, so I fail to see why installing them
inside the root filesystem is of any help to achieve that.

Yes, I've seen that it's probably related to the proposed logic to
generate a bundled image in PATCH 5/5, but for the moment, my feeling
is that if that logic requires *all* files to be present in the root
filesystem to be later installed in a separate FAT/boot filesystem,
then the logic isn't really appropriate, and should be improved, no?

I believe that the existing choice of installing things in
$(BINARIES_DIR)/rpi-firmware/ was a much better choice. That's a choice
I've kept in the series on Grub/Grub2/Gummiboot, by creating a
$(BINARIES_DIR)/efi/ directory.

Best regards,

Thomas

Thomas, All,

On 2013-11-28 21:11 +0100, Thomas Petazzoni spake thusly:
> On Fri, 22 Nov 2013 23:50:55 +0100, Yann E. MORIN wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > 
> > The firmware files must reside in a FAT filesystem, in the first partition
> > of the SDcard, so it makes sense to install them in target/ (eg. for a
> > post-image script to generate the different partitions from the rootfs.tar).
> 
> I don't understand the reasoning here. These files must be at the root
> of a specific FAT filesystem, so I fail to see why installing them
> inside the root filesystem is of any help to achieve that.
> 
> Yes, I've seen that it's probably related to the proposed logic to
> generate a bundled image in PATCH 5/5, but for the moment, my feeling
> is that if that logic requires *all* files to be present in the root
> filesystem to be later installed in a separate FAT/boot filesystem,
> then the logic isn't really appropriate, and should be improved, no?

The basic idea is to not depend on board-specific layout of the boot
files in the genimages infra.

Notice how the boot files for the Raspberry Pi are located in
images/rpi-firmware? This is not a 'generic' location, and we would need
a way for a 'board' to pass a 'description' to genimages of what should
be copied to the partition.

Since we do not have board description in Buildroot (and have gone quite
our way to remove it in the past), I tried not too add more. Of course,
the 'partition layout description' is board-specific, but its syntax is
not board-specific; the one we bundle is just an example, to serve as a
basis for the real file for a real project.

I designed genimages so that it does *not* deal with adding/removing
stuff to/from the filesystem, but just dispatch the directory layout
into different partitions. I find it odd to tinker with the content of
the filesystems at that point (but maybe I am biased, since genimages
used to be a post-image hook, but now is a real fs target).

> I believe that the existing choice of installing things in
> $(BINARIES_DIR)/rpi-firmware/ was a much better choice. That's a choice
> I've kept in the series on Grub/Grub2/Gummiboot, by creating a
> $(BINARIES_DIR)/efi/ directory.

Yes, but 'efi' is generic, it is not named after the target system,
while 'rpi-firmware' is very specific. Maybe we could harmonise the name
for that directory (eg. 'boot-files') and use that name to install all
bootloader files, in which case that'd be probably OK.

( BTW, the current grub package does copy its files into
$(TARGET_DIR)/boot/grub and I like it ;-) )

Except that bootloaders may install different sets of files, and we can't
find a common name for those files. Eg. rpi-firmware installs 5 different
files (bootcode.bin, start.elf, fixup.dat, config.txt, and cmdline.txt),
while grub installs at least three files (stage1, stage2, and any
fs-specific stage1_5, not counting the splashcreens).

Yet, we might be able to use that, if the bootloaders also install a
list of files to copy to the FS.

But then it means 'rootfs.tar' (or any other fs images) does not contain
the same as the 'genimages' images. I think this is bad, and I think all
images should have the same FS content.

Regards,
Yann E. MORIN.

> > 
> > I don't understand the reasoning here. These files must be at the
> > root
> > of a specific FAT filesystem, so I fail to see why installing them
> > inside the root filesystem is of any help to achieve that.
> > 
> > Yes, I've seen that it's probably related to the proposed logic to
> > generate a bundled image in PATCH 5/5, but for the moment, my
> > feeling
> > is that if that logic requires *all* files to be present in the
> > root
> > filesystem to be later installed in a separate FAT/boot filesystem,
> > then the logic isn't really appropriate, and should be improved,
> > no?

If I may chip-in, I really like the logic of the proposed image builder...

Basically, you build the root filesystem the way it will look when all
sub-filesystems are mounted, you describe how they are mounted and 
the system builds the partitions accordingly.

I know that what I described is not exactly the logic behind the proposed
patch, but it is a logic I like a lot because it is very easy to understand
and explain. it could also allow us to automatically generate some fstab
lines at some point, which could be nice.

Jeremy, All,

On Friday 29 November 2013 09:00:19 Jeremy Rosen wrote:
> If I may chip-in, I really like the logic of the proposed image builder...
> 
> Basically, you build the root filesystem the way it will look when all
> sub-filesystems are mounted, you describe how they are mounted and 
> the system builds the partitions accordingly.

Yes, that's exactly what I feel, too. When the image creation kicks in, the
filesystem(s) content is complete. I see 'genimages' like I look at genext2fs:
simply an image generator.

> I know that what I described is not exactly the logic behind the proposed
> patch, but it is a logic I like a lot because it is very easy to understand
> and explain. it could also allow us to automatically generate some fstab
> lines at some point, which could be nice.

And 'genimages' already does add the filesystems to the fstab. ;-)

Regards,
Yann E. MORIN.

Dear Jeremy Rosen,

On Fri, 29 Nov 2013 09:00:19 +0100 (CET), Jeremy Rosen wrote:

> Basically, you build the root filesystem the way it will look when all
> sub-filesystems are mounted, you describe how they are mounted and 
> the system builds the partitions accordingly.

I don't necessarily agree here, because this strategy makes the
confusion between two things:

 * The contents of the /boot directory in the root filesystem.

 * The contents of the boot partition needed for some platforms (OMAP,
   RPi, UEFI-based x86 platforms, etc.)

Those are two different things, and Yann's proposal merges them. As an
example, provided your bootloader has ext3 support and you're on an
OMAP platform, you may want to have something like:

 * The /boot directory in the root filesystem contains the kernel image
   (uImage, zImage)

 * The boot partition contains just the bootloader images (MLO,
   u-boot.img).

There is no point to have the kernel image in the boot partition, and
there is no point in having the bootloader images in the /boot
directory of the root filesystem.

Best regards,

Thomas

Thomas, All,

On 2013-11-29 09:27 +0100, Thomas Petazzoni spake thusly:
> Dear Jeremy Rosen,
> 
> On Fri, 29 Nov 2013 09:00:19 +0100 (CET), Jeremy Rosen wrote:
> 
> > Basically, you build the root filesystem the way it will look when all
> > sub-filesystems are mounted, you describe how they are mounted and 
> > the system builds the partitions accordingly.
> 
> I don't necessarily agree here, because this strategy makes the
> confusion between two things:
> 
>  * The contents of the /boot directory in the root filesystem.
> 
>  * The contents of the boot partition needed for some platforms (OMAP,
>    RPi, UEFI-based x86 platforms, etc.)
> 
> Those are two different things, and Yann's proposal merges them. As an
> example, provided your bootloader has ext3 support and you're on an
> OMAP platform, you may want to have something like:
> 
>  * The /boot directory in the root filesystem contains the kernel image
>    (uImage, zImage)
> 
>  * The boot partition contains just the bootloader images (MLO,
>    u-boot.img).
> 
> There is no point to have the kernel image in the boot partition, and
> there is no point in having the bootloader images in the /boot
> directory of the root filesystem.

OK, now I can follow what you mean (although one may argue whether the
kernel is part of those 'boot' files, since it does not make sense to
have it in the root file system).

However, this means that each 'platform' (eg. the RPi, your special
OMAP board...) will each have to provide a manifest of things to install
in the boot partition, and 'genimages' will have to interpret this
manifest to populate the boot partition.

I won't say I like this, even if I see your point. And from a more
global perspective, the boot partition might also be something else than
a filesystem.

So OK, I'll see how to manage this. Here is a quick proposal:

1) genimages should be able to create board-specific partitions, eg.:
   (a) a filesystem mounted on, and filled from /boot
   (b) an unmounted filesystem filled with an arbitrary set of files
   (c) a binary blob

  - For (a), we can already do that, it's like any other filesystem.
  - For (b), the submitted genimages can't do that, but the one in
    buildroot.config can, so I just have to reinstate my 'fs_filler'
    hook.
  - For (c), this has to be implemented. I suggest we postpone this
    until the basic genimages is upstream, and we have an actual
    use-case for it (no need to over-engineer the sutff, YAGNI...)

Of course, there are the cases where we may need any conbinations of one
or more of:
  - a boot blob (eg. MBR, UEFI; obviously only one!)
  - a binary blob partition (eg. a U-Boot image)
  - a mounted /boot filesystem
  - a unmounted filesystem

2) bootloaders for case (a) will have to:
   (a) install their files in $(TARGET_DIR)/boot

genimages should be prepared to generate a partition with a filesystem
filed from the content of /boot, and add an entry for it in fstab.
(Done! :-) )

We currently (as far as I know) have no such bootloader; not even
rpi-firmware fits in this category.

3) bootloaders for case (b) will have to:
   (a) install their files, and a manifest of those files, in a known
       location
   (b) that location should be platform-agnostic, so that genimages will
       find it; $(BINARIES_DIR)/boot-files/ looks like a good candidate.

genimages should be prepared to generate a partition with a filesystem
filled by copying files listed in a manifest file.
(Relatively easy, I think.)

The kind of bootloaders I know that fit in this category are
rpi-firmware and grub. There may have others.

4) bootloaders for case (c) will have to:
   (a) install the binary blob in a known location;
       $(BINARIES_DIR)/boot-files/ looks like a good candidate.

genimages should be prepared to generate a partition by simply dumping
the content of a file.

The bootloaders that fit in this category are U-Boot, RedBoot... Note
that this does not imply an MBR-type of partitioning.

Any comment on the above?

And thanks for the guidance and enlightment! :-)

Regards,
Yann E. MORIN.

On 29/11/13 20:01, Yann E. MORIN wrote:
> 3) bootloaders for case (b) will have to:
>     (a) install their files, and a manifest of those files, in a known
>         location
>     (b) that location should be platform-agnostic, so that genimages will
>         find it; $(BINARIES_DIR)/boot-files/  looks like a good candidate.
>
> genimages should be prepared to generate a partition with a filesystem
> filled by copying files listed in a manifest file.
> (Relatively easy, I think.)

  Yeah, it's just adding a flag that tells genimages that fs_root is 
relative to the images directory instead of the target directory.

  I think it is safe to assume that the bootloaders of this type will put 
their output files in a directory, rather than flat in the images 
directory. So a manifest is not really needed. Although... If you want to 
create a partition with SPL + U-Boot + uImage, you 'll have to specify 
these three explicitly.

  Regards,
  Arnout

>
> The kind of bootloaders I know that fit in this category are
> rpi-firmware and grub. There may have others.

Arnout, All,

On 2013-12-01 01:59 +0100, Arnout Vandecappelle spake thusly:
> On 29/11/13 20:01, Yann E. MORIN wrote:
> >3) bootloaders for case (b) will have to:
> >    (a) install their files, and a manifest of those files, in a known
> >        location
> >    (b) that location should be platform-agnostic, so that genimages will
> >        find it; $(BINARIES_DIR)/boot-files/  looks like a good candidate.
> >
> >genimages should be prepared to generate a partition with a filesystem
> >filled by copying files listed in a manifest file.
> >(Relatively easy, I think.)
> 
>  Yeah, it's just adding a flag that tells genimages that fs_root is relative
> to the images directory instead of the target directory.

Then, what about those targets that require the kernel to be there, too?

For the Raspberry Pi, we must have those files in the boot partition:
  - the boot loader files: bootcode.bin, start.elf, fixup.dat
  - the bootloader config files: config.txt, cmdline.txt
  - the kernel file: zImage

For Thomas' board, we would not need the kernel file in there.

So, genimages need to be able to generate the partition cntent not only
from files in $(BINARIES_DIR)/boot-files, but also from $(BIANRIES_DIR)
itself.

>  I think it is safe to assume that the bootloaders of this type will put
> their output files in a directory, rather than flat in the images directory.
> So a manifest is not really needed. Although... If you want to create a
> partition with SPL + U-Boot + uImage, you 'll have to specify these three
> explicitly.

Indeed. But we do not want to hard-code anything in genimages, and we
want to keep the partition table layout description as simple as
possible.

I'll see what I can come up with.

Regards,
Yann E. MORIN.