Patchwork [1/5] Add a post-image script mechanism

login
register
mail settings
Submitter Thomas Petazzoni
Date Jan. 16, 2013, 7:09 p.m.
Message ID <1358363393-29977-2-git-send-email-thomas.petazzoni@free-electrons.com>
Download mbox | patch
Permalink /patch/212989/
State Changes Requested
Headers show

Comments

Thomas Petazzoni - Jan. 16, 2013, 7:09 p.m.
Just like we have a post-build script mechanism that gets executed
after the build of all packages but before the creation of the
filesystem images, let's introduce a post-image script mechanism, that
gets executed once all filesystem images have been generated.

This can for example be used to call a tool building a firmware image
from different images generated by Buildroot, or automatically extract
the tarball root filesystem image into some location exported by NFS,
or any other custom action.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile         |    9 +++++++++
 system/Config.in |   18 ++++++++++++++++++
 2 files changed, 27 insertions(+)
Stefan Fröberg - Jan. 16, 2013, 7:22 p.m.
16.1.2013 21:09, Thomas Petazzoni kirjoitti:
> Just like we have a post-build script mechanism that gets executed
> after the build of all packages but before the creation of the
> filesystem images, let's introduce a post-image script mechanism, that
> gets executed once all filesystem images have been generated.
>
> This can for example be used to call a tool building a firmware image
> from different images generated by Buildroot, or automatically extract
> the tarball root filesystem image into some location exported by NFS,
> or any other custom action.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile         |    9 +++++++++
>  system/Config.in |   18 ++++++++++++++++++
>  2 files changed, 27 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..88b5a85 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -353,6 +353,8 @@ endif
>  
>  include fs/common.mk
>  
> +TARGETS+=target-post-image
> +
>  TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
>  TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
>  TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
> @@ -548,6 +550,13 @@ target-generatelocales: host-localedef
>  	done
>  endif
>  
> +target-post-image:
> +ifneq ($(BR2_ROOTFS_POST_IMAGE_SCRIPT),)
> +	@$(call MESSAGE,"Executing post-image script\(s\)")
> +	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> +		$(s) $(BINARIES_DIR)$(sep))
> +endif
> +
>  toolchain-eclipse-register:
>  	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
>  
> diff --git a/system/Config.in b/system/Config.in
> index 69863c4..1ffe8a0 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -265,4 +265,22 @@ config BR2_ROOTFS_POST_BUILD_SCRIPT
>  	  only argument. Make sure the exit code of those scripts are 0,
>  	  otherwise make will stop after calling them.
>  
> +config BR2_ROOTFS_POST_IMAGE_SCRIPT
> +	string "Custom script to run after creating filesystem images"
> +	default ""
> +	help
> +	  Specify a space-separated list of scripts to be run after
> +	  the build has finished and after Buildroot has packed the
> +	  files into selected filesystem images.
> +
> +	  This can for example be used to call a tool building a
> +	  firmware image from different images generated by Buildroot,
> +	  or automatically extract the tarball root filesystem image
> +	  into some location exported by NFS, or any other custom
> +	  action.
> +
> +	  These scripts are called with the images directory name as
> +	  first and only argument. The script is executed from the
> +	  main Buildroot source directory as the current directory.
> +
>  endmenu

Nice.  I know exactly what Im going to use this mechanism for.

Packing target binaries with UPX (http://upx.sourceforge.net/) at the
end of image creation.

Every byte counts :-)

Regards
Stefan
Yann E. MORIN - Jan. 16, 2013, 8:03 p.m.
Thomas, All,

On Wednesday 16 January 2013 Thomas Petazzoni wrote:
> Just like we have a post-build script mechanism that gets executed
> after the build of all packages but before the creation of the
> filesystem images, let's introduce a post-image script mechanism, that
> gets executed once all filesystem images have been generated.
[--SNIP--]
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..88b5a85 100644
> --- a/Makefile
> +++ b/Makefile
[--SNIP--]
> @@ -548,6 +550,13 @@ target-generatelocales: host-localedef
>  	done
>  endif
>  
> +target-post-image:
> +ifneq ($(BR2_ROOTFS_POST_IMAGE_SCRIPT),)
> +	@$(call MESSAGE,"Executing post-image script\(s\)")
> +	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> +		$(s) $(BINARIES_DIR)$(sep))

No need to pass $(BINARIES_DIR) as it now is in the environment.

Would it make sense to run this under fakeroot?

Also, I wonder if we could instead call these scripts for each image we
generate, so it will already be run under fakeroot.

Not sure how to pass the image type to the script, though.
Maybe, something like:
  post-image.sh <image-file> <image-type> <compression>

with image-type ∈ {tar,ext2,squashfs...}
and  compression ∈ {none,gz,bz2,lzma,xz...}

And the script would typically contain:

  #!/bin/sh
  img_type="${1}"
  img_file="${2}"
  img_comp="${3}"
  case "${img_type}" in
    tar)
      handle_tar "${img_file}" "#{img_comp}"
      ;;
    squashfs)
      handle_squash "${img_file}"
      ;;
    *)
      # Ignore other types
     ;;
  esac

At least, I find it more interesting.

Regards,
Yann E. MORIN.
Thomas Petazzoni - Jan. 16, 2013, 9:41 p.m.
Dear Yann E. MORIN,

On Wed, 16 Jan 2013 21:03:43 +0100, Yann E. MORIN wrote:

> No need to pass $(BINARIES_DIR) as it now is in the environment.

I'd prefer to pass $(BINARIES_DIR) as argument, in order to keep a
symmetry with what we do for the post-build script.

> Would it make sense to run this under fakeroot?

Why?

> Also, I wonder if we could instead call these scripts for each image
> we generate, so it will already be run under fakeroot.

No, I don't think it's a good idea. For example, in my use-case, it was
used to create a firmware image that bundles multiple images generated
by Buildroot. So getting called on a per-image basis doesn't make sense.

> Not sure how to pass the image type to the script, though.
> Maybe, something like:
>   post-image.sh <image-file> <image-type> <compression>
> 
> with image-type ∈ {tar,ext2,squashfs...}
> and  compression ∈ {none,gz,bz2,lzma,xz...}
> 
> And the script would typically contain:
> 
>   #!/bin/sh
>   img_type="${1}"
>   img_file="${2}"
>   img_comp="${3}"
>   case "${img_type}" in
>     tar)
>       handle_tar "${img_file}" "#{img_comp}"
>       ;;
>     squashfs)
>       handle_squash "${img_file}"
>       ;;
>     *)
>       # Ignore other types
>      ;;
>   esac
> 
> At least, I find it more interesting.

Unfortunately, not me. Calling a script once all images are generated
is *simple*, the script is free to do whatever it wants. Iterate over
all images, iterate over certain images only, etc. There is really no
need to over-engineer this thing.

I really don't see the point of putting more complexity into this.
Let's have something nicely symmetric between the post-build script
feature and the post-image script feature.

Best regards,

Thomas
Yann E. MORIN - Jan. 16, 2013, 10:06 p.m.
Thomas, All,

On Wednesday 16 January 2013 Thomas Petazzoni wrote:
> On Wed, 16 Jan 2013 21:03:43 +0100, Yann E. MORIN wrote:
> > No need to pass $(BINARIES_DIR) as it now is in the environment.
> I'd prefer to pass $(BINARIES_DIR) as argument, in order to keep a
> symmetry with what we do for the post-build script.
> 
> > Would it make sense to run this under fakeroot?
> Why?

For example, I need to create multiple partitions for my device, so I
need one file system image per partition:
  - vfat on /boot
  - squashfs on /
  - ext2 on /home

So I would :
  - configure Buildroot to build only a tarball
  - configure Buildroot to run my post-image script
  - extract the tarball
  - generate each file system images

But the untar would loose the device nodes and the files UIDs and GIDs,
forcing me do run fakeroot inside the script, saving/loading the fakeroot
DB, for every commands in my script.

Having Buildroot do the fakeroot would ensure this consistency.

> > Also, I wonder if we could instead call these scripts for each image
> > we generate, so it will already be run under fakeroot.
> 
> No, I don't think it's a good idea. For example, in my use-case, it was
> used to create a firmware image that bundles multiple images generated
> by Buildroot. So getting called on a per-image basis doesn't make sense.

OK, so it makes sense to run the scripts only once at the end. Agreed.

[--SNIP--]
> I really don't see the point of putting more complexity into this.
> Let's have something nicely symmetric between the post-build script
> feature and the post-image script feature.

Right, agreed.

Regards,
Yann E. MORIN.
Jeremy Rosen - Jan. 17, 2013, 8:14 a.m.
I think we have two different use cases here...


* we need a hook within fakeroot to make multiple images when each image is a different partition

* we need a hook post-image to bundle multiple images into a single binary files

that's a lot of hook, but i'm not sure how to not have all of them... 

    Regards

    Jérémy Rosen

fight key loggers : write some perl using vim
Yann E. MORIN - Jan. 25, 2013, 6:48 p.m.
On Wednesday 16 January 2013 Thomas Petazzoni wrote:
> Just like we have a post-build script mechanism that gets executed
> after the build of all packages but before the creation of the
> filesystem images, let's introduce a post-image script mechanism, that
> gets executed once all filesystem images have been generated.
> 
> This can for example be used to call a tool building a firmware image
> from different images generated by Buildroot, or automatically extract
> the tarball root filesystem image into some location exported by NFS,
> or any other custom action.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

That is really usefull! Thank you! :-)

Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

> ---
>  Makefile         |    9 +++++++++
>  system/Config.in |   18 ++++++++++++++++++
>  2 files changed, 27 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..88b5a85 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -353,6 +353,8 @@ endif
>  
>  include fs/common.mk
>  
> +TARGETS+=target-post-image
> +
>  TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
>  TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
>  TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
> @@ -548,6 +550,13 @@ target-generatelocales: host-localedef
>  	done
>  endif
>  
> +target-post-image:
> +ifneq ($(BR2_ROOTFS_POST_IMAGE_SCRIPT),)
> +	@$(call MESSAGE,"Executing post-image script\(s\)")
> +	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> +		$(s) $(BINARIES_DIR)$(sep))
> +endif
> +
>  toolchain-eclipse-register:
>  	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
>  
> diff --git a/system/Config.in b/system/Config.in
> index 69863c4..1ffe8a0 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -265,4 +265,22 @@ config BR2_ROOTFS_POST_BUILD_SCRIPT
>  	  only argument. Make sure the exit code of those scripts are 0,
>  	  otherwise make will stop after calling them.
>  
> +config BR2_ROOTFS_POST_IMAGE_SCRIPT
> +	string "Custom script to run after creating filesystem images"
> +	default ""
> +	help
> +	  Specify a space-separated list of scripts to be run after
> +	  the build has finished and after Buildroot has packed the
> +	  files into selected filesystem images.
> +
> +	  This can for example be used to call a tool building a
> +	  firmware image from different images generated by Buildroot,
> +	  or automatically extract the tarball root filesystem image
> +	  into some location exported by NFS, or any other custom
> +	  action.
> +
> +	  These scripts are called with the images directory name as
> +	  first and only argument. The script is executed from the
> +	  main Buildroot source directory as the current directory.
> +
>  endmenu
> -- 
> 1.7.9.5
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
>
Luca Ceresoli - Jan. 29, 2013, 4:27 p.m.
Thomas Petazzoni wrote:
> Just like we have a post-build script mechanism that gets executed
> after the build of all packages but before the creation of the
> filesystem images, let's introduce a post-image script mechanism, that
> gets executed once all filesystem images have been generated.
>
> This can for example be used to call a tool building a firmware image
> from different images generated by Buildroot, or automatically extract
> the tarball root filesystem image into some location exported by NFS,
> or any other custom action.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Acked-by: Luca Ceresoli <luca@lucaceresoli.net>

Patch

diff --git a/Makefile b/Makefile
index 6f8ed0e..88b5a85 100644
--- a/Makefile
+++ b/Makefile
@@ -353,6 +353,8 @@  endif
 
 include fs/common.mk
 
+TARGETS+=target-post-image
+
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -548,6 +550,13 @@  target-generatelocales: host-localedef
 	done
 endif
 
+target-post-image:
+ifneq ($(BR2_ROOTFS_POST_IMAGE_SCRIPT),)
+	@$(call MESSAGE,"Executing post-image script\(s\)")
+	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
+		$(s) $(BINARIES_DIR)$(sep))
+endif
+
 toolchain-eclipse-register:
 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
 
diff --git a/system/Config.in b/system/Config.in
index 69863c4..1ffe8a0 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -265,4 +265,22 @@  config BR2_ROOTFS_POST_BUILD_SCRIPT
 	  only argument. Make sure the exit code of those scripts are 0,
 	  otherwise make will stop after calling them.
 
+config BR2_ROOTFS_POST_IMAGE_SCRIPT
+	string "Custom script to run after creating filesystem images"
+	default ""
+	help
+	  Specify a space-separated list of scripts to be run after
+	  the build has finished and after Buildroot has packed the
+	  files into selected filesystem images.
+
+	  This can for example be used to call a tool building a
+	  firmware image from different images generated by Buildroot,
+	  or automatically extract the tarball root filesystem image
+	  into some location exported by NFS, or any other custom
+	  action.
+
+	  These scripts are called with the images directory name as
+	  first and only argument. The script is executed from the
+	  main Buildroot source directory as the current directory.
+
 endmenu