[OpenWrt-Devel] brcm2708: add squashfs rootfs image

Message ID 20180327174218.8574-1-chunkeey@gmail.com
State Accepted
Delegated to: John Crispin
Headers show
Series
  • [OpenWrt-Devel] brcm2708: add squashfs rootfs image
Related show

Commit Message

Christian Lamparter March 27, 2018, 5:42 p.m.
This patch adds a image with squashfs as the root filesystem.
A rootfs_data partition will be generated on the first boot
and placed inside the rootfs partition (just after the squashfs
image).

advantages:
 - it is possible to migrate from an existing -ext4
   installation and back via sysupgrade.
 - existing partition layout will not be lost.
 - slightly smaller image size.
 - support for attendedsysupgrade

disadvantages:
 - needs f2fs + tools as well. This is because fs-tools decides on the
   blocksize of the sdcard. So either f2fs or ext4 can get choosen as
   the rootfs_data filesystem (depends on the size of the root partition).
 - rootfs_data is placed into the rootfs partition. This makes
   it difficult for tools that expect a /dev/mmc0pX device.
   It also makes it difficult for data recovery tools since they
   might not expect to find a embedded partition or will be
   confused.

For people with existing build configurations: make sure to include mkf2fs
and f2fsck package into the image... Otherwise the new -squashfs image will
boot of a ram-overlay and won't keep the configurations after a reboot.

Cc: Álvaro Fernández Rojas <noltari@gmail.com>
Cc: Paul Spooren <spooren@informatik.uni-leipzig.de>
Cc: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
---
Hello Álvaro,

The squashfs image was requested by Paul:
<https://github.com/aparcar/attendedsysupgrade-server/issues/88#issuecomment-365036459>
I'm not sure why, but he hardcoded his tool to only work with "squashfs".
<https://github.com/aparcar/attendedsysupgrade-server/blob/c8684b12d7a0b95222eec8aea20608d2595ce0ed/worker/worker.py#L209>

I've tested this with a raspberrypi model b. But I'm confident the
other targets rpis will work as well. What's your take on this?

Regards,
Christian
---
 target/linux/brcm2708/Makefile           | 5 +++--
 target/linux/brcm2708/bcm2708/config-4.9 | 6 +++++-
 target/linux/brcm2708/bcm2709/config-4.9 | 6 +++++-
 target/linux/brcm2708/bcm2710/config-4.9 | 6 +++++-
 target/linux/brcm2708/image/Makefile     | 1 -
 target/linux/brcm2708/image/cmdline.txt  | 2 +-
 6 files changed, 19 insertions(+), 7 deletions(-)

Comments

Daniel Golle March 29, 2018, 12:22 a.m. | #1
On Tue, Mar 27, 2018 at 07:42:18PM +0200, Christian Lamparter wrote:
> This patch adds a image with squashfs as the root filesystem.
> A rootfs_data partition will be generated on the first boot
> and placed inside the rootfs partition (just after the squashfs
> image).
> 
> advantages:
>  - it is possible to migrate from an existing -ext4
>    installation and back via sysupgrade.
>  - existing partition layout will not be lost.
>  - slightly smaller image size.
>  - support for attendedsysupgrade
> 
> disadvantages:
>  - needs f2fs + tools as well. This is because fs-tools decides on the
>    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
>    the rootfs_data filesystem (depends on the size of the root partition).
>  - rootfs_data is placed into the rootfs partition. This makes
>    it difficult for tools that expect a /dev/mmc0pX device.
>    It also makes it difficult for data recovery tools since they
>    might not expect to find a embedded partition or will be
>    confused.
> 
> For people with existing build configurations: make sure to include mkf2fs
> and f2fsck package into the image... Otherwise the new -squashfs image will
> boot of a ram-overlay and won't keep the configurations after a reboot.
> 
> Cc: Álvaro Fernández Rojas <noltari@gmail.com>
> Cc: Paul Spooren <spooren@informatik.uni-leipzig.de>
> Cc: Daniel Golle <daniel@makrotopia.org>
> Signed-off-by: Christian Lamparter <chunkeey@gmail.com>

Acked-by: Daniel Golle <daniel@makrotopia.org>
Daniel Golle April 12, 2018, 1:30 p.m. | #2
On Thu, Mar 29, 2018 at 02:22:48AM +0200, Daniel Golle wrote:
> On Tue, Mar 27, 2018 at 07:42:18PM +0200, Christian Lamparter wrote:
> > This patch adds a image with squashfs as the root filesystem.
> > A rootfs_data partition will be generated on the first boot
> > and placed inside the rootfs partition (just after the squashfs
> > image).
> > 
> > advantages:
> >  - it is possible to migrate from an existing -ext4
> >    installation and back via sysupgrade.
> >  - existing partition layout will not be lost.
> >  - slightly smaller image size.
> >  - support for attendedsysupgrade
> > 
> > disadvantages:
> >  - needs f2fs + tools as well. This is because fs-tools decides on the
> >    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
> >    the rootfs_data filesystem (depends on the size of the root partition).
> >  - rootfs_data is placed into the rootfs partition. This makes
> >    it difficult for tools that expect a /dev/mmc0pX device.
> >    It also makes it difficult for data recovery tools since they
> >    might not expect to find a embedded partition or will be
> >    confused.
> > 
> > For people with existing build configurations: make sure to include mkf2fs
> > and f2fsck package into the image... Otherwise the new -squashfs image will
> > boot of a ram-overlay and won't keep the configurations after a reboot.
> > 
> > Cc: Álvaro Fernández Rojas <noltari@gmail.com>
> > Cc: Paul Spooren <spooren@informatik.uni-leipzig.de>
> > Cc: Daniel Golle <daniel@makrotopia.org>
> > Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
> 
> Acked-by: Daniel Golle <daniel@makrotopia.org>

Álvaro, please review the patch or at least give us a sign of life.


Cheers


Daniel
Stijn Tintel April 24, 2018, 11:29 a.m. | #3
On 27-03-18 20:42, Christian Lamparter wrote:
> This patch adds a image with squashfs as the root filesystem.
> A rootfs_data partition will be generated on the first boot
> and placed inside the rootfs partition (just after the squashfs
> image).
>
> advantages:
>  - it is possible to migrate from an existing -ext4
>    installation and back via sysupgrade.
>  - existing partition layout will not be lost.
>  - slightly smaller image size.
>  - support for attendedsysupgrade
>
> disadvantages:
>  - needs f2fs + tools as well. This is because fs-tools decides on the
>    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
>    the rootfs_data filesystem (depends on the size of the root partition).
>  - rootfs_data is placed into the rootfs partition. This makes
>    it difficult for tools that expect a /dev/mmc0pX device.
>    It also makes it difficult for data recovery tools since they
>    might not expect to find a embedded partition or will be
>    confused.
>
> For people with existing build configurations: make sure to include mkf2fs
> and f2fsck package into the image... Otherwise the new -squashfs image will
> boot of a ram-overlay and won't keep the configurations after a reboot.
>
Tried to sysupgrade one of my RPi0W from ext4 to squashfs, and it
appeared to work at first, until I rebooted it again. Apparently the
overlay is completely missing, the SD card only has /boot (FAT32) and /
(squashfs). Can you please look into this? If you can get it resolved I
will merge the patch.

Stijn
Christian Lamparter April 24, 2018, 2:59 p.m. | #4
On Dienstag, 24. April 2018 13:29:17 CEST Stijn Tintel wrote:
> On 27-03-18 20:42, Christian Lamparter wrote:
> > This patch adds a image with squashfs as the root filesystem.
> > A rootfs_data partition will be generated on the first boot
> > and placed inside the rootfs partition (just after the squashfs
> > image).
> >
> > advantages:
> >  - it is possible to migrate from an existing -ext4
> >    installation and back via sysupgrade.
> >  - existing partition layout will not be lost.
> >  - slightly smaller image size.
> >  - support for attendedsysupgrade
> >
> > disadvantages:
> >  - needs f2fs + tools as well. This is because fs-tools decides on the
> >    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
> >    the rootfs_data filesystem (depends on the size of the root partition).
> >  - rootfs_data is placed into the rootfs partition. This makes
> >    it difficult for tools that expect a /dev/mmc0pX device.
> >    It also makes it difficult for data recovery tools since they
> >    might not expect to find a embedded partition or will be
> >    confused.
> >
> > For people with existing build configurations: make sure to include mkf2fs
> > and f2fsck package into the image... Otherwise the new -squashfs image will
> > boot of a ram-overlay and won't keep the configurations after a reboot.
> >
> Tried to sysupgrade one of my RPi0W from ext4 to squashfs, and it
> appeared to work at first, until I rebooted it again. Apparently the
> overlay is completely missing, the SD card only has /boot (FAT32) and /
> (squashfs). Can you please look into this?
I think know the problem. The commit-message mentions it and the symptoms
you are describing would fit. But just in case: can you please provide the
image's diffconfig (or full-config ... and I guess a bootlog wouldn't
hurt either - but it would be optional) to confirm?

(I also tested it again on an uptodate "OpenWrt SNAPSHOT, r6759+2-9e848f901c"
with my rpi-b - which should be in the same family/subtarget as your RPi0W so
I can try your generated images as well.)

> If you can get it resolved I will merge the patch.
The original issue [0] has been addressed by this patch [1]. And indeed
the luci-app will now successfully generate a image (currently, only for the 
rpi-3 though!):
"openwrt-dbbb09d2658d5c2-brcm2708-bcm2710-rpi-3-ext4-sdcard.img.gz"
and install it.

this patch is therefore no longer necessary since it the existing ext4
images work as well. :) ... Plus the ext4 images don't confuse tools and
brains as much.

Regards,
Christian

[0] <https://github.com/aparcar/attendedsysupgrade-server/issues/88>
[1] <https://github.com/aparcar/attendedsysupgrade-server/commit/d477a87db41ae1e44325acf22fd4cb37d269f8b9#diff-b7368b46572bf04e88e8fcd94d072a1d>
Stijn Tintel April 24, 2018, 4:17 p.m. | #5
On 24-04-18 17:59, Christian Lamparter wrote:
> On Dienstag, 24. April 2018 13:29:17 CEST Stijn Tintel wrote:
>> On 27-03-18 20:42, Christian Lamparter wrote:
>>> This patch adds a image with squashfs as the root filesystem.
>>> A rootfs_data partition will be generated on the first boot
>>> and placed inside the rootfs partition (just after the squashfs
>>> image).
>>>
>>> advantages:
>>>  - it is possible to migrate from an existing -ext4
>>>    installation and back via sysupgrade.
>>>  - existing partition layout will not be lost.
>>>  - slightly smaller image size.
>>>  - support for attendedsysupgrade
>>>
>>> disadvantages:
>>>  - needs f2fs + tools as well. This is because fs-tools decides on the
>>>    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
>>>    the rootfs_data filesystem (depends on the size of the root partition).
>>>  - rootfs_data is placed into the rootfs partition. This makes
>>>    it difficult for tools that expect a /dev/mmc0pX device.
>>>    It also makes it difficult for data recovery tools since they
>>>    might not expect to find a embedded partition or will be
>>>    confused.
>>>
>>> For people with existing build configurations: make sure to include mkf2fs
>>> and f2fsck package into the image... Otherwise the new -squashfs image will
>>> boot of a ram-overlay and won't keep the configurations after a reboot.
>>>
>> Tried to sysupgrade one of my RPi0W from ext4 to squashfs, and it
>> appeared to work at first, until I rebooted it again. Apparently the
>> overlay is completely missing, the SD card only has /boot (FAT32) and /
>> (squashfs). Can you please look into this?
> I think know the problem. The commit-message mentions it and the symptoms
> you are describing would fit. But just in case: can you please provide the
> image's diffconfig (or full-config ... and I guess a bootlog wouldn't
> hurt either - but it would be optional) to confirm?
Please find diffconfig attached. The machine was running headless so I
don't have a bootlog.
>
> (I also tested it again on an uptodate "OpenWrt SNAPSHOT, r6759+2-9e848f901c"
> with my rpi-b - which should be in the same family/subtarget as your RPi0W so
> I can try your generated images as well.)
>
>> If you can get it resolved I will merge the patch.
> The original issue [0] has been addressed by this patch [1]. And indeed
> the luci-app will now successfully generate a image (currently, only for the 
> rpi-3 though!):
> "openwrt-dbbb09d2658d5c2-brcm2708-bcm2710-rpi-3-ext4-sdcard.img.gz"
> and install it.
>
> this patch is therefore no longer necessary since it the existing ext4
> images work as well. :) ... Plus the ext4 images don't confuse tools and
> brains as much.
I still find it useful, I've had way too many corrupted ext4 filesystems
on my RPis, and I'm hoping squashfs + f2fs would be more reliable than
ext4 on SD cards...

Thanks,
Stijn
CONFIG_TARGET_brcm2708=y
CONFIG_TARGET_brcm2708_bcm2708=y
CONFIG_TARGET_brcm2708_bcm2708_DEVICE_rpi=y
CONFIG_BRCM2708_SD_BOOT_PARTSIZE=256
CONFIG_BUILD_PATENTED=y
CONFIG_DEBUG=y
CONFIG_DROPBEAR_ECC=y
# CONFIG_KERNEL_CC_STACKPROTECTOR_REGULAR is not set
CONFIG_KERNEL_CC_STACKPROTECTOR_STRONG=y
CONFIG_LIBCURL_COOKIES=y
CONFIG_LIBCURL_FILE=y
CONFIG_LIBCURL_FTP=y
CONFIG_LIBCURL_HTTP=y
CONFIG_LIBCURL_NGHTTP2=y
CONFIG_LIBCURL_NO_SMB="!"
CONFIG_LIBCURL_OPENSSL=y
CONFIG_LIBCURL_PROXY=y
CONFIG_LIBIIO_LOCAL_BACKEND=y
CONFIG_LIBIIO_NETWORK_BACKEND=y
CONFIG_LIBIIO_XML_BACKEND=y
CONFIG_LLDPD_WITH_CDP=y
CONFIG_LLDPD_WITH_CUSTOM=y
CONFIG_LLDPD_WITH_DOT1=y
CONFIG_LLDPD_WITH_DOT3=y
CONFIG_LLDPD_WITH_EDP=y
CONFIG_LLDPD_WITH_FDP=y
CONFIG_LLDPD_WITH_LLDPMED=y
CONFIG_LLDPD_WITH_PRIVSEP=y
CONFIG_LLDPD_WITH_SONMP=y
CONFIG_MINIUPNPD_IGDv2=y
CONFIG_OPENSSL_OPTIMIZE_SPEED=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_EC=y
CONFIG_OPENSSL_WITH_NPN=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_PACKAGE_BRCM80211_DEBUG=y
CONFIG_PACKAGE_avahi-dbus-daemon=y
CONFIG_PACKAGE_bfgminer=y
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_ca-certificates=y
CONFIG_PACKAGE_cgi-io=y
CONFIG_PACKAGE_dbus=y
# CONFIG_PACKAGE_dnsmasq is not set
CONFIG_PACKAGE_fontconfig=y
CONFIG_PACKAGE_gdb=y
CONFIG_PACKAGE_gdbserver=y
CONFIG_PACKAGE_i2c-tools=y
CONFIG_PACKAGE_iio-utils=y
CONFIG_PACKAGE_iiod=y
CONFIG_PACKAGE_ip-full=y
CONFIG_PACKAGE_iperf=y
CONFIG_PACKAGE_iperf3=y
CONFIG_PACKAGE_ipset=y
CONFIG_PACKAGE_iptables-mod-ipsec=y
CONFIG_PACKAGE_iputils-ping=y
CONFIG_PACKAGE_iputils-ping6=y
# CONFIG_PACKAGE_iwinfo is not set
CONFIG_PACKAGE_jansson=y
CONFIG_PACKAGE_kmod-bluetooth=y
CONFIG_PACKAGE_kmod-crypto-aead=y
CONFIG_PACKAGE_kmod-crypto-authenc=y
CONFIG_PACKAGE_kmod-crypto-cbc=y
CONFIG_PACKAGE_kmod-crypto-cmac=y
CONFIG_PACKAGE_kmod-crypto-deflate=y
CONFIG_PACKAGE_kmod-crypto-des=y
CONFIG_PACKAGE_kmod-crypto-ecb=y
CONFIG_PACKAGE_kmod-crypto-ecdh=y
CONFIG_PACKAGE_kmod-crypto-echainiv=y
CONFIG_PACKAGE_kmod-crypto-hash=y
CONFIG_PACKAGE_kmod-crypto-hmac=y
CONFIG_PACKAGE_kmod-crypto-iv=y
CONFIG_PACKAGE_kmod-crypto-kpp=y
CONFIG_PACKAGE_kmod-crypto-manager=y
CONFIG_PACKAGE_kmod-crypto-md4=y
CONFIG_PACKAGE_kmod-crypto-md5=y
CONFIG_PACKAGE_kmod-crypto-null=y
CONFIG_PACKAGE_kmod-crypto-pcompress=y
CONFIG_PACKAGE_kmod-crypto-rng=y
CONFIG_PACKAGE_kmod-crypto-sha1=y
CONFIG_PACKAGE_kmod-crypto-sha256=y
CONFIG_PACKAGE_kmod-crypto-wq=y
CONFIG_PACKAGE_kmod-dma-buf=y
CONFIG_PACKAGE_kmod-drm=y
CONFIG_PACKAGE_kmod-drm-vc4=y
CONFIG_PACKAGE_kmod-fs-cifs=y
CONFIG_PACKAGE_kmod-i2c-bcm2708=y
CONFIG_PACKAGE_kmod-i2c-bcm2835=y
CONFIG_PACKAGE_kmod-i2c-core=y
CONFIG_PACKAGE_kmod-iio-bmp280=y
CONFIG_PACKAGE_kmod-iio-bmp280-i2c=y
CONFIG_PACKAGE_kmod-iio-bmp280-spi=y
CONFIG_PACKAGE_kmod-iio-core=y
CONFIG_PACKAGE_kmod-iio-dht11=y
CONFIG_PACKAGE_kmod-ip-vti=y
CONFIG_PACKAGE_kmod-ip6-tunnel=y
CONFIG_PACKAGE_kmod-ip6-vti=y
CONFIG_PACKAGE_kmod-ipsec=y
CONFIG_PACKAGE_kmod-ipsec4=y
CONFIG_PACKAGE_kmod-ipsec6=y
CONFIG_PACKAGE_kmod-ipt-ipsec=y
CONFIG_PACKAGE_kmod-ipt-ipset=y
CONFIG_PACKAGE_kmod-iptunnel=y
CONFIG_PACKAGE_kmod-iptunnel4=y
CONFIG_PACKAGE_kmod-iptunnel6=y
CONFIG_PACKAGE_kmod-lib-crc16=y
CONFIG_PACKAGE_kmod-lib-lzo=y
CONFIG_PACKAGE_kmod-lib-zlib-deflate=y
CONFIG_PACKAGE_kmod-lib-zlib-inflate=y
CONFIG_PACKAGE_kmod-nfnetlink=y
CONFIG_PACKAGE_kmod-regmap=y
CONFIG_PACKAGE_kmod-sched-core=y
CONFIG_PACKAGE_kmod-spi-bcm2835=y
CONFIG_PACKAGE_kmod-spi-bcm2835-aux=y
CONFIG_PACKAGE_kmod-spi-bitbang=y
CONFIG_PACKAGE_kmod-udptunnel4=y
CONFIG_PACKAGE_kmod-udptunnel6=y
CONFIG_PACKAGE_kmod-usb-acm=y
CONFIG_PACKAGE_kmod-usb-serial=y
CONFIG_PACKAGE_kmod-usb-serial-ftdi=y
CONFIG_PACKAGE_kmod-wireguard=y
CONFIG_PACKAGE_libavahi-client=y
CONFIG_PACKAGE_libavahi-dbus-support=y
CONFIG_PACKAGE_libbase58=y
CONFIG_PACKAGE_libbz2=y
CONFIG_PACKAGE_libcares=y
CONFIG_PACKAGE_libcurl=y
CONFIG_PACKAGE_libdaemon=y
CONFIG_PACKAGE_libdbus=y
CONFIG_PACKAGE_libevent2=y
CONFIG_PACKAGE_libexpat=y
# CONFIG_PACKAGE_libf2fs is not set
CONFIG_PACKAGE_libfreetype=y
CONFIG_PACKAGE_libiio=y
CONFIG_PACKAGE_libipset=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_libmbedtls=y
CONFIG_PACKAGE_libmnl=y
CONFIG_PACKAGE_libmosquitto-ssl=y
CONFIG_PACKAGE_libncurses=y
CONFIG_PACKAGE_libnetsnmp=y
CONFIG_PACKAGE_libnghttp2=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_PACKAGE_libpcap=y
CONFIG_PACKAGE_libpng=y
CONFIG_PACKAGE_libreadline=y
CONFIG_PACKAGE_librpc=y
CONFIG_PACKAGE_libsensors=y
# CONFIG_PACKAGE_libsmartcols is not set
CONFIG_PACKAGE_libsysfs=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libuci-lua=y
CONFIG_PACKAGE_libusb-1.0=y
CONFIG_PACKAGE_libustream-openssl=y
CONFIG_PACKAGE_libxml2=y
CONFIG_PACKAGE_lldpd=y
CONFIG_PACKAGE_lsof=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-json=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-rpc=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
# CONFIG_PACKAGE_mkf2fs is not set
CONFIG_PACKAGE_mtr=y
# CONFIG_PACKAGE_odhcp6c is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
CONFIG_PACKAGE_openssl-util=y
# CONFIG_PACKAGE_partx-utils is not set
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_snmpd=y
CONFIG_PACKAGE_ss=y
CONFIG_PACKAGE_strace=y
CONFIG_PACKAGE_sysfsutils=y
CONFIG_PACKAGE_tc=y
CONFIG_PACKAGE_tcpdump=y
CONFIG_PACKAGE_terminfo=y
CONFIG_PACKAGE_uclibcxx=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
CONFIG_PACKAGE_usbutils=y
CONFIG_PACKAGE_wpad=y
# CONFIG_PACKAGE_wpad-mini is not set
CONFIG_PACKAGE_zlib=y
CONFIG_PKG_CC_STACKPROTECTOR_STRONG=y
# CONFIG_PKG_FORTIFY_SOURCE_1 is not set
CONFIG_PKG_FORTIFY_SOURCE_2=y
CONFIG_TARGET_ROOTFS_PARTSIZE=512
CONFIG_WPA_SUPPLICANT_INTERNAL=y
CONFIG_ZLIB_OPTIMIZE_SPEED=y
Stijn Tintel April 24, 2018, 5:56 p.m. | #6
On 24-04-18 19:17, Stijn Tintel wrote:
> On 24-04-18 17:59, Christian Lamparter wrote:
>> On Dienstag, 24. April 2018 13:29:17 CEST Stijn Tintel wrote:
>>> On 27-03-18 20:42, Christian Lamparter wrote:
>>>> This patch adds a image with squashfs as the root filesystem.
>>>> A rootfs_data partition will be generated on the first boot
>>>> and placed inside the rootfs partition (just after the squashfs
>>>> image).
>>>>
>>>> advantages:
>>>>  - it is possible to migrate from an existing -ext4
>>>>    installation and back via sysupgrade.
>>>>  - existing partition layout will not be lost.
>>>>  - slightly smaller image size.
>>>>  - support for attendedsysupgrade
>>>>
>>>> disadvantages:
>>>>  - needs f2fs + tools as well. This is because fs-tools decides on the
>>>>    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
>>>>    the rootfs_data filesystem (depends on the size of the root partition).
>>>>  - rootfs_data is placed into the rootfs partition. This makes
>>>>    it difficult for tools that expect a /dev/mmc0pX device.
>>>>    It also makes it difficult for data recovery tools since they
>>>>    might not expect to find a embedded partition or will be
>>>>    confused.
>>>>
>>>> For people with existing build configurations: make sure to include mkf2fs
>>>> and f2fsck package into the image... Otherwise the new -squashfs image will
>>>> boot of a ram-overlay and won't keep the configurations after a reboot.
>>>>
>>> Tried to sysupgrade one of my RPi0W from ext4 to squashfs, and it
>>> appeared to work at first, until I rebooted it again. Apparently the
>>> overlay is completely missing, the SD card only has /boot (FAT32) and /
>>> (squashfs). Can you please look into this?
>> I think know the problem. The commit-message mentions it and the symptoms
>> you are describing would fit. But just in case: can you please provide the
>> image's diffconfig (or full-config ... and I guess a bootlog wouldn't
>> hurt either - but it would be optional) to confirm?
> Please find diffconfig attached. The machine was running headless so I
> don't have a bootlog.
>> (I also tested it again on an uptodate "OpenWrt SNAPSHOT, r6759+2-9e848f901c"
>> with my rpi-b - which should be in the same family/subtarget as your RPi0W so
>> I can try your generated images as well.)
>>
>>> If you can get it resolved I will merge the patch.
>> The original issue [0] has been addressed by this patch [1]. And indeed
>> the luci-app will now successfully generate a image (currently, only for the 
>> rpi-3 though!):
>> "openwrt-dbbb09d2658d5c2-brcm2708-bcm2710-rpi-3-ext4-sdcard.img.gz"
>> and install it.
>>
>> this patch is therefore no longer necessary since it the existing ext4
>> images work as well. :) ... Plus the ext4 images don't confuse tools and
>> brains as much.
> I still find it useful, I've had way too many corrupted ext4 filesystems
> on my RPis, and I'm hoping squashfs + f2fs would be more reliable than
> ext4 on SD cards...
Read the commit message again, and I see what you mean. Must have
misread CONFIG_DEFAULT_mkf2fs=y as CONFIG_PACKAGE_mkf2fs=y. Too bad make
defconfig doesn't enable DEFAULT_PACKAGES. Sorry for the noise, I'll do
some more testing and accept this patch.

Thanks,
Stijn
Christian Lamparter April 24, 2018, 6:07 p.m. | #7
On Dienstag, 24. April 2018 19:56:38 CEST Stijn Tintel wrote:
> On 24-04-18 19:17, Stijn Tintel wrote:
> > On 24-04-18 17:59, Christian Lamparter wrote:
> >> On Dienstag, 24. April 2018 13:29:17 CEST Stijn Tintel wrote:
> >>> On 27-03-18 20:42, Christian Lamparter wrote:
> >>>> This patch adds a image with squashfs as the root filesystem.
> >>>> A rootfs_data partition will be generated on the first boot
> >>>> and placed inside the rootfs partition (just after the squashfs
> >>>> image).
> >>>>
> >>>> advantages:
> >>>>  - it is possible to migrate from an existing -ext4
> >>>>    installation and back via sysupgrade.
> >>>>  - existing partition layout will not be lost.
> >>>>  - slightly smaller image size.
> >>>>  - support for attendedsysupgrade
> >>>>
> >>>> disadvantages:
> >>>>  - needs f2fs + tools as well. This is because fs-tools decides on the
> >>>>    blocksize of the sdcard. So either f2fs or ext4 can get choosen as
> >>>>    the rootfs_data filesystem (depends on the size of the root partition).
> >>>>  - rootfs_data is placed into the rootfs partition. This makes
> >>>>    it difficult for tools that expect a /dev/mmc0pX device.
> >>>>    It also makes it difficult for data recovery tools since they
> >>>>    might not expect to find a embedded partition or will be
> >>>>    confused.
> >>>>
> >>>> For people with existing build configurations: make sure to include mkf2fs
> >>>> and f2fsck package into the image... Otherwise the new -squashfs image will
> >>>> boot of a ram-overlay and won't keep the configurations after a reboot.
> >>>>
> >>> Tried to sysupgrade one of my RPi0W from ext4 to squashfs, and it
> >>> appeared to work at first, until I rebooted it again. Apparently the
> >>> overlay is completely missing, the SD card only has /boot (FAT32) and /
> >>> (squashfs). Can you please look into this?
> >> I think know the problem. The commit-message mentions it and the symptoms
> >> you are describing would fit. But just in case: can you please provide the
> >> image's diffconfig (or full-config ... and I guess a bootlog wouldn't
> >> hurt either - but it would be optional) to confirm?
> > Please find diffconfig attached. The machine was running headless so I
> > don't have a bootlog.
> >> (I also tested it again on an uptodate "OpenWrt SNAPSHOT, r6759+2-9e848f901c"
> >> with my rpi-b - which should be in the same family/subtarget as your RPi0W so
> >> I can try your generated images as well.)
> >>
> >>> If you can get it resolved I will merge the patch.
> >> The original issue [0] has been addressed by this patch [1]. And indeed
> >> the luci-app will now successfully generate a image (currently, only for the 
> >> rpi-3 though!):
> >> "openwrt-dbbb09d2658d5c2-brcm2708-bcm2710-rpi-3-ext4-sdcard.img.gz"
> >> and install it.
> >>
> >> this patch is therefore no longer necessary since it the existing ext4
> >> images work as well. :) ... Plus the ext4 images don't confuse tools and
> >> brains as much.
> > I still find it useful, I've had way too many corrupted ext4 filesystems
> > on my RPis, and I'm hoping squashfs + f2fs would be more reliable than
> > ext4 on SD cards...
> Read the commit message again, and I see what you mean. Must have
> misread CONFIG_DEFAULT_mkf2fs=y as CONFIG_PACKAGE_mkf2fs=y. Too bad make
> defconfig doesn't enable DEFAULT_PACKAGES. Sorry for the noise, I'll do
> some more testing and accept this patch.
Well, if you want to go through with it, you could always add more 
"WARNING - DANGER - ATTENTION" stickers and point out that 
CONFIG_DEFAULT_mkf2fs=y != CONFIG_PACKAGE_mkf2fs=y . So people will
really watch out.

Regards,
Christian
Daniel Golle May 28, 2018, 2:44 p.m. | #8
Hi!

On Tue, Apr 24, 2018 at 04:59:34PM +0200, Christian Lamparter wrote:
> ...
> this patch is therefore no longer necessary since it the existing ext4
> images work as well. :) ... Plus the ext4 images don't confuse tools and
> brains as much.

While this assumption might hold true for some people (apparently all
of them Rasbpi users), it is currently an exception from OpenWrt's
point of view (probably because most users flash their devices by
removing the microSD and writing it via another device rather than
on-target system upgrades, no even speaking about remote and/or
over-the-air upgrades...).

Imho it would still be very nice to at least have the option for
overlayfs-based images on all targets because otherwise we kill some
of the features OpenWrt-users have learned to rely on, such as
the robustness of preinit, availability of recovery in case of
misconfiguration, factory-reset/firstboot, snapshots, ...
While this might not be so much of an issue for people using the
rasbpi as a devboard, it does make things more difficult for people
building end-user-ready devices based on that hardware.


Cheers


Daniel

Patch

diff --git a/target/linux/brcm2708/Makefile b/target/linux/brcm2708/Makefile
index 244513dfc6..870029c4c8 100644
--- a/target/linux/brcm2708/Makefile
+++ b/target/linux/brcm2708/Makefile
@@ -10,7 +10,7 @@  include $(TOPDIR)/rules.mk
 ARCH:=arm
 BOARD:=brcm2708
 BOARDNAME:=Broadcom BCM27xx
-FEATURES:=ext4 audio usb usbgadget display gpio fpu
+FEATURES:=ext4 audio usb usbgadget display gpio fpu squashfs
 MAINTAINER:=Álvaro Fernández Rojas <noltari@gmail.com>
 SUBTARGETS:=bcm2708 bcm2709 bcm2710
 
@@ -27,7 +27,8 @@  DEFAULT_PACKAGES += \
 	kmod-usb-hid \
 	kmod-sound-core kmod-sound-arm-bcm2835 \
 	kmod-fs-vfat kmod-nls-cp437 kmod-nls-iso8859-1 \
-	brcmfmac-firmware-43430-sdio kmod-brcmfmac wpad-mini
+	brcmfmac-firmware-43430-sdio kmod-brcmfmac wpad-mini \
+	partx-utils mkf2fs e2fsprogs
 
 KERNELNAME:=Image dtbs
 
diff --git a/target/linux/brcm2708/bcm2708/config-4.9 b/target/linux/brcm2708/bcm2708/config-4.9
index dccf85fac3..116dd257dd 100644
--- a/target/linux/brcm2708/bcm2708/config-4.9
+++ b/target/linux/brcm2708/bcm2708/config-4.9
@@ -141,6 +141,11 @@  CONFIG_ENABLE_MUST_CHECK=y
 CONFIG_EXT4_FS=y
 CONFIG_EXT4_FS_POSIX_ACL=y
 CONFIG_EXT4_FS_SECURITY=y
+# CONFIG_F2FS_CHECK_FS is not set
+CONFIG_F2FS_FS=y
+# CONFIG_F2FS_FS_SECURITY is not set
+CONFIG_F2FS_FS_XATTR=y
+CONFIG_F2FS_STAT_FS=y
 CONFIG_FB=y
 CONFIG_FB_BCM2708=y
 CONFIG_FB_CFB_COPYAREA=y
@@ -342,7 +347,6 @@  CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
 CONFIG_SERIAL_OF_PLATFORM=y
 CONFIG_SG_POOL=y
 CONFIG_SPARSE_IRQ=y
-# CONFIG_SQUASHFS is not set
 CONFIG_SRCU=y
 # CONFIG_STRIP_ASM_SYMS is not set
 CONFIG_SUSPEND=y
diff --git a/target/linux/brcm2708/bcm2709/config-4.9 b/target/linux/brcm2708/bcm2709/config-4.9
index 71884a5fa1..2571a4dc0a 100644
--- a/target/linux/brcm2708/bcm2709/config-4.9
+++ b/target/linux/brcm2708/bcm2709/config-4.9
@@ -155,6 +155,11 @@  CONFIG_ENABLE_MUST_CHECK=y
 CONFIG_EXT4_FS=y
 CONFIG_EXT4_FS_POSIX_ACL=y
 CONFIG_EXT4_FS_SECURITY=y
+# CONFIG_F2FS_CHECK_FS is not set
+CONFIG_F2FS_FS=y
+# CONFIG_F2FS_FS_SECURITY is not set
+CONFIG_F2FS_FS_XATTR=y
+CONFIG_F2FS_STAT_FS=y
 CONFIG_FB=y
 CONFIG_FB_BCM2708=y
 CONFIG_FB_CFB_COPYAREA=y
@@ -372,7 +377,6 @@  CONFIG_SG_POOL=y
 CONFIG_SMP=y
 CONFIG_SMP_ON_UP=y
 CONFIG_SPARSE_IRQ=y
-# CONFIG_SQUASHFS is not set
 CONFIG_SRCU=y
 # CONFIG_STRIP_ASM_SYMS is not set
 CONFIG_SUSPEND=y
diff --git a/target/linux/brcm2708/bcm2710/config-4.9 b/target/linux/brcm2708/bcm2710/config-4.9
index d402f4b573..87e0289347 100644
--- a/target/linux/brcm2708/bcm2710/config-4.9
+++ b/target/linux/brcm2708/bcm2710/config-4.9
@@ -162,6 +162,11 @@  CONFIG_ENABLE_MUST_CHECK=y
 CONFIG_EXT4_FS=y
 CONFIG_EXT4_FS_POSIX_ACL=y
 CONFIG_EXT4_FS_SECURITY=y
+# CONFIG_F2FS_CHECK_FS is not set
+CONFIG_F2FS_FS=y
+# CONFIG_F2FS_FS_SECURITY is not set
+CONFIG_F2FS_FS_XATTR=y
+CONFIG_F2FS_STAT_FS=y
 CONFIG_FB=y
 CONFIG_FB_BCM2708=y
 CONFIG_FB_CFB_COPYAREA=y
@@ -393,7 +398,6 @@  CONFIG_SPARSEMEM_MANUAL=y
 CONFIG_SPARSEMEM_VMEMMAP=y
 CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
 CONFIG_SPARSE_IRQ=y
-# CONFIG_SQUASHFS is not set
 CONFIG_SRCU=y
 # CONFIG_STRIP_ASM_SYMS is not set
 CONFIG_SUSPEND=y
diff --git a/target/linux/brcm2708/image/Makefile b/target/linux/brcm2708/image/Makefile
index 9ea9a07a9d..28dfa70628 100644
--- a/target/linux/brcm2708/image/Makefile
+++ b/target/linux/brcm2708/image/Makefile
@@ -47,7 +47,6 @@  endef
 
 ### Devices ###
 define Device/Default
-  FILESYSTEMS := ext4
   KERNEL := kernel-bin | kernel-img
   KERNEL_IMG := kernel.img
   IMAGES := sdcard.img.gz
diff --git a/target/linux/brcm2708/image/cmdline.txt b/target/linux/brcm2708/image/cmdline.txt
index 24f4f0c276..c1099afb00 100644
--- a/target/linux/brcm2708/image/cmdline.txt
+++ b/target/linux/brcm2708/image/cmdline.txt
@@ -1 +1 @@ 
-dwc_otg.lpm_enable=0 console=serial0,115200 kgdboc=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
+dwc_otg.lpm_enable=0 console=serial0,115200 kgdboc=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=squashfs,ext4 rootwait