diff mbox series

[RFC] base-files: sysupgrade: always setup overlay when creating backup

Message ID 20240213213129.18484-1-zajec5@gmail.com
State RFC
Delegated to: Rafał Miłecki
Headers show
Series [RFC] base-files: sysupgrade: always setup overlay when creating backup | expand

Commit Message

Rafał Miłecki Feb. 13, 2024, 9:31 p.m. UTC 
From: Rafał Miłecki <rafal@milecki.pl>

Setting overlay while creating backup allows including extra files in
archive without actually writing them to flash. Right now this feature
is limited to /etc/backup/ directory and is used only for including
installed_packages.txt.

Extend this solution to make it more generic:
1. Always mount overlay while creating backup
2. Overlay whole / to don't limit it to /etc/backup/

This allows including any additional files in backups and adding more
sysupgrade features.

Cc: Luiz Angelo Daros de Luca <luizluca@gmail.com>
Cc: Christian Marangi <ansuelsmth@gmail.com>
Cc: Jo-Philipp Wich <jo@mein.io>
Cc: Jonas Gorski <jonas.gorski@gmail.com>
Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
This will allow me to include /etc/uci-defaults/ scripts in backups so
we can e.g. have script disabling previously disabled services.

 package/base-files/files/sbin/sysupgrade | 37 ++++++++++++++----------
 1 file changed, 21 insertions(+), 16 deletions(-)

Comments

Eric Feb. 16, 2024, 3:48 p.m. UTC | #1 
The sender domain has a DMARC Reject/Quarantine policy which disallows
sending mailing list messages using the original "From" header.

To mitigate this problem, the original message has been wrapped
automatically by the mailing list software.
On Tuesday, February 13th, 2024 at 13:31, Rafał Miłecki <zajec5@gmail.com> wrote:
> From: Rafał Miłecki rafal@milecki.pl
> 
> Setting overlay while creating backup allows including extra files in
> archive without actually writing them to flash. Right now this feature
> is limited to /etc/backup/ directory and is used only for including
> installed_packages.txt.
> 
> Extend this solution to make it more generic:
> 1. Always mount overlay while creating backup
> 2. Overlay whole / to don't limit it to /etc/backup/
> 
> This allows including any additional files in backups and adding more
> sysupgrade features.
... 
> ---
> This will allow me to include /etc/uci-defaults/ scripts in backups so
> we can e.g. have script disabling previously disabled services.

There is a pending PR[1] against the auc package that looks for
    /rom/etc/uci-defaults/99-asu-defaults
and if found sends its contents in a request to the ASU build server
for inclusion in uci-defaults.  Am I correct in assuming that this
proposal would allow an alternative mechanism for accomplishing the
same thing, thus making the PR (and, in fact, the 'defaults' mechanism
of the ASU build server) redundant?

Also, what are the implications for uci-defaults on ext4?  Do we need
something that creates a first-boot-backup for sysupgrade to use later?

[1] - https://github.com/openwrt/packages/pull/22144

Eric
Luiz Angelo Daros de Luca Feb. 16, 2024, 4:42 p.m. UTC | #2 
> From: Rafał Miłecki <rafal@milecki.pl>
>
> Setting overlay while creating backup allows including extra files in
> archive without actually writing them to flash. Right now this feature
> is limited to /etc/backup/ directory and is used only for including
> installed_packages.txt.
>
> Extend this solution to make it more generic:
> 1. Always mount overlay while creating backup
> 2. Overlay whole / to don't limit it to /etc/backup/
>
> This allows including any additional files in backups and adding more
> sysupgrade features.

It is a nice feature. Thanks.

>
> Cc: Luiz Angelo Daros de Luca <luizluca@gmail.com>
> Cc: Christian Marangi <ansuelsmth@gmail.com>
> Cc: Jo-Philipp Wich <jo@mein.io>
> Cc: Jonas Gorski <jonas.gorski@gmail.com>
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
> This will allow me to include /etc/uci-defaults/ scripts in backups so
> we can e.g. have script disabling previously disabled services.
>
>  package/base-files/files/sbin/sysupgrade | 37 ++++++++++++++----------
>  1 file changed, 21 insertions(+), 16 deletions(-)
>
> diff --git a/package/base-files/files/sbin/sysupgrade b/package/base-files/files/sbin/sysupgrade
> index 1fcd44da2a..1e09f65e07 100755
> --- a/package/base-files/files/sbin/sysupgrade
> +++ b/package/base-files/files/sbin/sysupgrade
> @@ -237,7 +237,8 @@ include /lib/upgrade
>  create_backup_archive() {
>         local conf_tar="$1"
>
> -       local umount_etcbackup_dir=0
> +       local overlay
> +       local dir
>
>         [ "$(rootfs_type)" = "tmpfs" ] && {
>                 echo "Cannot save config while running from ramdisk." >&2
> @@ -248,17 +249,20 @@ create_backup_archive() {
>         run_hooks "$CONFFILES" $sysupgrade_init_conffiles
>         ask_bool 0 "Edit config file list" && vi "$CONFFILES"
>
> +       # Mount root directory with temporary overlay on top of it.
> +       # This allows including extra (temporary) files in backup archive
> +       # without messing actual rootfs.
> +       overlay="$(mktemp -d -t overlay.XXXXXX)"
> +       mkdir -p "$overlay/upper" "$overlay/work"
> +       dir="$(mktemp -d -t backup.XXXXXX)"

You could avoid two mktemp creating a single unique parent directory.
It makes the cleanup simpler.

> +       if ! mount -t overlay overlay -o lowerdir=/,upperdir="$overlay/upper",workdir="$overlay/work" "$dir"; then
> +               echo "Cannot mount backup overlay to $dir." >&2
> +               ask_bool 0 "Abort" && exit
> +       fi
> +
>         if [ "$SAVE_INSTALLED_PKGS" -eq 1 ]; then
>                 echo "${INSTALLED_PACKAGES}" >> "$CONFFILES"
> -               mkdir -p "$ETCBACKUP_DIR"
> -               # Avoid touching filesystem on each backup
> -               RAMFS="$(mktemp -d -t sysupgrade.XXXXXX)"
> -               mkdir -p "$RAMFS/upper" "$RAMFS/work"
> -               mount -t overlay overlay -o lowerdir=$ETCBACKUP_DIR,upperdir=$RAMFS/upper,workdir=$RAMFS/work $ETCBACKUP_DIR &&
> -                       umount_etcbackup_dir=1 || {
> -                               echo "Cannot mount '$ETCBACKUP_DIR' as tmpfs to avoid touching disk while saving the list of installed packages." >&2
> -                               ask_bool 0 "Abort" && exit
> -                       }
> +               mkdir -p "$dir/$ETCBACKUP_DIR"
>
>                 # Format: pkg-name<TAB>{rom,overlay,unkown}
>                 # rom is used for pkgs in /rom, even if updated later
> @@ -266,22 +270,23 @@ create_backup_archive() {
>                         \( -exec test -f /rom/{} \; -exec echo {} rom \; \) -o \
>                         \( -exec test -f /overlay/upper/{} \; -exec echo {} overlay \; \) -o \
>                         \( -exec echo {} unknown \; \) \
> -                       \) | sed -e 's,.*/,,;s/\.control /\t/' > ${INSTALLED_PACKAGES}
> +                       \) | sed -e 's,.*/,,;s/\.control /\t/' > "$dir/${INSTALLED_PACKAGES}"
>         fi
>
>         v "Saving config files..."
> +       sed -i 's/^\///' "$CONFFILES" # Drop leading slashes
>         [ "$VERBOSE" -gt 1 ] && TAR_V="v" || TAR_V=""
> -       tar c${TAR_V}zf "$conf_tar" -T "$CONFFILES" 2>/dev/null
> +       tar c${TAR_V}zf "$conf_tar" -C "${dir:-/}" -T "$CONFFILES"

These were already part of an PR that fixed a couple of sysupgrade issues:
https://github.com/openwrt/openwrt/pull/11022 (since Oct 2022 but
still relevant), including here.

>         if [ "$?" -ne 0 ]; then
>                 echo "Failed to create the configuration backup."
>                 rm -f "$conf_tar"
>                 exit 1

Like this one. You need to unmount and delete everything before
bailing out. Check the PR.
Maybe we should merge it and rebase your change over it. Or you can
merge it into this patch.

>         fi
>
> -       [ "$umount_etcbackup_dir" -eq 1 ] && {
> -               umount "$ETCBACKUP_DIR"
> -               rm -rf "$RAMFS"
> -       }
> +       umount "$dir"
> +       rm -rf "$dir"
> +       rm -rf "$overlay"
> +
>         rm -f "$CONFFILES"
>  }
>
> --
> 2.35.3
>
diff mbox series

Patch

diff --git a/package/base-files/files/sbin/sysupgrade b/package/base-files/files/sbin/sysupgrade
index 1fcd44da2a..1e09f65e07 100755
--- a/package/base-files/files/sbin/sysupgrade
+++ b/package/base-files/files/sbin/sysupgrade
@@ -237,7 +237,8 @@  include /lib/upgrade
 create_backup_archive() {
 	local conf_tar="$1"
 
-	local umount_etcbackup_dir=0
+	local overlay
+	local dir
 
 	[ "$(rootfs_type)" = "tmpfs" ] && {
 		echo "Cannot save config while running from ramdisk." >&2
@@ -248,17 +249,20 @@  create_backup_archive() {
 	run_hooks "$CONFFILES" $sysupgrade_init_conffiles
 	ask_bool 0 "Edit config file list" && vi "$CONFFILES"
 
+	# Mount root directory with temporary overlay on top of it.
+	# This allows including extra (temporary) files in backup archive
+	# without messing actual rootfs.
+	overlay="$(mktemp -d -t overlay.XXXXXX)"
+	mkdir -p "$overlay/upper" "$overlay/work"
+	dir="$(mktemp -d -t backup.XXXXXX)"
+	if ! mount -t overlay overlay -o lowerdir=/,upperdir="$overlay/upper",workdir="$overlay/work" "$dir"; then
+		echo "Cannot mount backup overlay to $dir." >&2
+		ask_bool 0 "Abort" && exit
+	fi
+
 	if [ "$SAVE_INSTALLED_PKGS" -eq 1 ]; then
 		echo "${INSTALLED_PACKAGES}" >> "$CONFFILES"
-		mkdir -p "$ETCBACKUP_DIR"
-		# Avoid touching filesystem on each backup
-		RAMFS="$(mktemp -d -t sysupgrade.XXXXXX)"
-		mkdir -p "$RAMFS/upper" "$RAMFS/work"
-		mount -t overlay overlay -o lowerdir=$ETCBACKUP_DIR,upperdir=$RAMFS/upper,workdir=$RAMFS/work $ETCBACKUP_DIR &&
-			umount_etcbackup_dir=1 || {
-				echo "Cannot mount '$ETCBACKUP_DIR' as tmpfs to avoid touching disk while saving the list of installed packages." >&2
-				ask_bool 0 "Abort" && exit
-			}
+		mkdir -p "$dir/$ETCBACKUP_DIR"
 
 		# Format: pkg-name<TAB>{rom,overlay,unkown}
 		# rom is used for pkgs in /rom, even if updated later
@@ -266,22 +270,23 @@  create_backup_archive() {
 			\( -exec test -f /rom/{} \; -exec echo {} rom \; \) -o \
 			\( -exec test -f /overlay/upper/{} \; -exec echo {} overlay \; \) -o \
 			\( -exec echo {} unknown \; \) \
-			\) | sed -e 's,.*/,,;s/\.control /\t/' > ${INSTALLED_PACKAGES}
+			\) | sed -e 's,.*/,,;s/\.control /\t/' > "$dir/${INSTALLED_PACKAGES}"
 	fi
 
 	v "Saving config files..."
+	sed -i 's/^\///' "$CONFFILES" # Drop leading slashes
 	[ "$VERBOSE" -gt 1 ] && TAR_V="v" || TAR_V=""
-	tar c${TAR_V}zf "$conf_tar" -T "$CONFFILES" 2>/dev/null
+	tar c${TAR_V}zf "$conf_tar" -C "${dir:-/}" -T "$CONFFILES"
 	if [ "$?" -ne 0 ]; then
 		echo "Failed to create the configuration backup."
 		rm -f "$conf_tar"
 		exit 1
 	fi
 
-	[ "$umount_etcbackup_dir" -eq 1 ] && {
-		umount "$ETCBACKUP_DIR"
-		rm -rf "$RAMFS"
-	}
+	umount "$dir"
+	rm -rf "$dir"
+	rm -rf "$overlay"
+
 	rm -f "$CONFFILES"
 }