[OpenWrt-Devel,v2] build: add mkrasimage

The current make-ras.sh image generation script for the ZyXEL NBG6617
has portability issues with bash. Because of this, factory images are
currently not built correctly by the OpenWRT buildbots.

This commit replaces the make-ras.sh by C-written mkrasimage. The old
script is still kept but can be deleted in the future.

The new mkrasimage is also compatible with other ZyXEL devices using
the ras image-format. This is not tested with a OpenWRT build but
it correctly builds the header for ZyXEL factory images for all devices
it is added to.

Signed-off-by: David Bauer <mail@david-bauer.net>
---

v2 changes:
 - Rework image-generation code
 - Add factory image for NBG6616
 - Add factory image for NBG6817

 include/image-commands.mk             |  18 +-
 scripts/make-ras.sh                   | 196 -----------
 target/linux/ar71xx/image/generic.mk  |   6 +-
 target/linux/ipq40xx/image/Makefile   |   2 +-
 target/linux/ipq806x/image/Makefile   |   6 +-
 tools/firmware-utils/Makefile         |   1 +
 tools/firmware-utils/src/mkrasimage.c | 474 ++++++++++++++++++++++++++
 7 files changed, 495 insertions(+), 208 deletions(-)
 delete mode 100755 scripts/make-ras.sh
 create mode 100644 tools/firmware-utils/src/mkrasimage.c

David Bauer <mail@david-bauer.net> wrote:
> The current make-ras.sh image generation script for the ZyXEL
> NBG6617 has portability issues with bash. Because of this,
> factory images are currently not built correctly by the OpenWRT
> buildbots.
> 
> This commit replaces the make-ras.sh by C-written mkrasimage.
> The old script is still kept but can be deleted in the future.

You need to update your commit message, this is no longer the
case.

> 
> The new mkrasimage is also compatible with other ZyXEL devices
> using the ras image-format. This is not tested with a OpenWRT
> build but it correctly builds the header for ZyXEL factory
> images for all devices it is added to.
> 
> Signed-off-by: David Bauer <mail@david-bauer.net>
> ---
> 
> v2 changes:
>  - Rework image-generation code
>  - Add factory image for NBG6616
>  - Add factory image for NBG6817
> 
>  include/image-commands.mk             |  18 +-
>  scripts/make-ras.sh                   | 196 -----------
>  target/linux/ar71xx/image/generic.mk  |   6 +-
>  target/linux/ipq40xx/image/Makefile   |   2 +-
>  target/linux/ipq806x/image/Makefile   |   6 +-
>  tools/firmware-utils/Makefile         |   1 +
>  tools/firmware-utils/src/mkrasimage.c | 474 ++++++++++++++++++++++++++
>  7 files changed, 495 insertions(+), 208 deletions(-)
>  delete mode 100755 scripts/make-ras.sh
>  create mode 100644 tools/firmware-utils/src/mkrasimage.c
> 
> diff --git a/include/image-commands.mk
> b/include/image-commands.mk index 3cc5dc21e1..61ba49de51 100644
> --- a/include/image-commands.mk
> +++ b/include/image-commands.mk
> @@ -49,17 +49,17 @@ define Build/eva-image
>  	mv $@.new $@
>  endef
>  
> -define Build/make-ras
> +define Build/zyxel-ras-image
>  	let \
>  		newsize="$(subst k,* 1024,$(RAS_ROOTFS_SIZE))"; \
> -		$(TOPDIR)/scripts/make-ras.sh \
> -			--board $(RAS_BOARD) \
> -			--version $(RAS_VERSION) \
> -			--kernel $(call param_get_default,kernel,$(1),$(IMAGE_KERNEL)) \
> -			--rootfs $@ \
> -			--rootfssize $$newsize \
> -			$@.new
> -	@mv $@.new $@
> +		$(STAGING_DIR_HOST)/bin/mkrasimage \
> +			-b $(RAS_BOARD) \
> +			-v $(RAS_VERSION) \
> +			-r $@ \
> +			-s $$newsize \
> +			-o $@.new \
> +			$(if $(findstring seperate-kernel,$(word 1,$(1))),-k $(IMAGE_KERNEL)) \
> +		&& mv $@.new $@
>  endef
>  
>  define Build/mkbuffaloimg
> diff --git a/scripts/make-ras.sh b/scripts/make-ras.sh deleted
> file mode 100755 index ccddaa0016..0000000000
> --- a/scripts/make-ras.sh
> +++ /dev/null
> @@ -1,196 +0,0 @@
> -#!/usr/bin/env bash
> -#
> -# --- ZyXEL header format ---
> -# Original Version by Benjamin Berg <benjamin@sipsolutions.net>
> -#
> -# The firmware image prefixed with a header (which is written into the MTD device).
> -# The header is one erase block (~64KiB) in size, but the checksum only convers the
> -# first 2KiB. Padding is 0xff. All integers are in big-endian.
> -#
> -# The checksum is always a 16-Bit System V checksum (sum -s) stored in a 32-Bit integer.
> -#
> -#   4 bytes:  checksum of the rootfs image
> -#   4 bytes:  length of the contained rootfs image file (big endian)
> -#  32 bytes:  Firmware Version string (NUL terminated, 0xff padded)
> -#   4 bytes:  checksum over the header partition (big endian - see below)
> -#  32 bytes:  Model (e.g. "NBG6617", NUL termiated, 0xff padded)
> -#   4 bytes:  checksum of the kernel partition
> -#   4 bytes:  length of the contained kernel image file (big endian)
> -#      rest: 0xff padding
> -#
> -# The checksums are calculated by adding up all bytes and if a 16bit
> -# overflow occurs, one is added and the sum is masked to 16 bit:
> -#   csum = csum + databyte; if (csum > 0xffff) { csum += 1; csum &= 0xffff };
> -# Should the file have an odd number of bytes then the byte len-0x800 is
> -# used additionally.
> -#
> -# The checksum for the header is calculated over the first 2048 bytes with
> -# the rootfs image checksum as the placeholder during calculation.
> -#
> -# The header is padded with 0xff to the erase block size of the device.
> -#
> -board=""
> -version=""
> -kernel=""
> -rootfs=""
> -outfile=""
> -err=""
> -
> -while [ "$1" ]; do
> -	case "$1" in
> -	"--board")
> -		board="$2"
> -		shift
> -		shift
> -		continue
> -		;;
> -	"--version")
> -		version="$2"
> -		shift
> -		shift
> -		continue
> -		;;
> -	"--kernel")
> -		kernel="$2"
> -		shift
> -		shift
> -		continue
> -		;;
> -	"--rootfs")
> -		rootfs="$2"
> -		shift
> -		shift
> -		continue
> -		;;
> -	"--rootfssize")
> -		rootfssize="$2"
> -		shift
> -		shift
> -		continue
> -		;;
> -	*)
> -		if [ ! "$outfile" ]; then
> -			outfile=$1
> -			shift
> -			continue
> -		fi
> -		;;
> -	esac
> -done
> -
> -if [ ! -n "$board" -o ! -n "$version" -o ! -r "$kernel" -o !
> -r "$rootfs" -o ! "$rootfssize" -o ! "$outfile" ]; then
> -	echo "syntax: $0 [--board ras-boardname] [--version ras-version] [--kernel kernelimage] [--rootfs rootfs] out"
> -	exit 1
> -fi
> -
> -rootfs_len=$(wc -c < "$rootfs")
> -
> -if [ "$rootfs_len" -lt "$rootfssize" ]; then
> -	dd if=$rootfs of=$rootfs.new bs=$rootfssize conv=sync
> -	mv $rootfs.new $rootfs
> -fi
> -
> -if [ ${#version} -ge 28 ]; then
> -	echo "version: '$version' is too long"
> -	exit 1
> -fi
> -
> -tmpdir="$( mktemp -d 2> /dev/null )"
> -if [ -z "$tmpdir" ]; then
> -	# try OSX signature
> -	tmpdir="$( mktemp -t 'ubitmp' -d )"
> -fi
> -
> -if [ -z "$tmpdir" ]; then
> -	exit 1
> -fi
> -
> -to_be() {
> -	local val="$1"
> -	local size="$2"
> -
> -	case "$size" in
> -	4)
> -		echo $(( "$val" >> 24 | ("$val" & 0xff0000) >> 8 | ("$val" & 0xff00) << 8 | ("$val" & 0xff) << 24 ))
> -		;;
> -	2)
> -		echo $(( "$val" >> 8 | ("$val" & 0xff) << 8))
> -		;;
> -	esac
> -}
> -
> -checksum_file() {
> -	local file=$1
> -
> -	# ZyXEL seems to use System V sum mode... Now this is classy, who would have thought?!
> -	echo $(sum -s ${file} | cut -f1 -d" ")
> -}
> -
> -append_bin() {
> -	local val=$1
> -	local size=$2
> -	local file=$3
> -
> -	while [ "$size" -ne 0 ]; do
> -		printf \\$(printf %o $(("$val" & 0xff)))  >> "$file"
> -		val=$(($val >> 8))
> -		let size-=1
> -	done
> -	return
> -}
> -
> -tf=${tmpdir}/out
> -pad=$(printf '%0.1s' $(printf "\xff"){1..64})
> -
> -rootfs_header_file="$tmpdir/rootfs_header"
> -rootfs_chksum=$(to_be $(checksum_file ${rootfs}) 4)
> -rootfs_len=$(to_be $(wc -c < "$rootfs") 4)
> -
> -versionpadlen=$(( 32 - ( ${#version} + 1) ))
> -
> -# 4 bytes:  checksum of the rootfs image
> -append_bin "$rootfs_chksum" 4 "$rootfs_header_file"
> -# 4 bytes:  length of the contained rootfs image file (big endian)
> -append_bin "$rootfs_len" 4 "$rootfs_header_file"
> -# 32 bytes:  Firmware Version string (NUL terminated, 0xff padded)
> -printf "%s\x00%.*s" "$version" "$versionpadlen" "$pad" >>
> "$rootfs_header_file"
> -
> -kernel_header_file="$tmpdir/kernel_header"
> -kernel_chksum=$(to_be $(checksum_file ${kernel}) 4)
> -kernel_len=$(to_be $(wc -c < "$kernel") 4)
> -
> -# 4 bytes:  checksum of the kernel image
> -append_bin "$kernel_chksum" 4 "$kernel_header_file"
> -# 4 bytes:  length of the contained kernel image file (big endian)
> -append_bin "$kernel_len" 4 "$kernel_header_file"
> -
> -board_header_file="$tmpdir/board_header"
> -board_file="$tmpdir/board"
> -boardpadlen=$(( 64 - ( ${#board} + 1) ))
> -# 32 bytes:  Model (e.g. "NBG6617", NUL termiated, 0xff padded)
> -printf "%s\x00%.*s" "$board" "$boardpadlen" "$pad" >
> "$board_file" -cat "$kernel_header_file" >> "$board_file"
> -printf "%.12s" "$pad" >> "$board_file"
> -#      rest: 0xff padding
> -for i in {1..511}; do
> -	printf "%s%s" "$pad" "$pad" >> "$board_file"
> -done
> -
> -tmp_board_file="$tmpdir/tmp_board_file"
> -cat "$rootfs_header_file" > "$tmp_board_file"
> -
> -# The checksum for the header is calculated over the first 2048 bytes with
> -# the rootfs image checksum as the placeholder during calculation.
> -append_bin "$rootfs_chksum" 4 "$tmp_board_file"
> -cat "$board_file" >> "$tmp_board_file"
> -
> -truncate -s 2048 $tmp_board_file
> -board_chksum=$(to_be $(checksum_file ${tmp_board_file}) 4)
> -
> -# 4 bytes:  checksum over the header partition (big endian)
> -append_bin "$board_chksum" 4 "$board_header_file"
> -cat "$board_file" >> "$board_header_file"
> -
> -cat "$rootfs_header_file" "$board_header_file" "$rootfs"
> "$kernel" > "$outfile"
> -
> -rm -rf "$tmpdir"
> diff --git a/target/linux/ar71xx/image/generic.mk
> b/target/linux/ar71xx/image/generic.mk index
> 3b320814c5..55963c5768 100644
> --- a/target/linux/ar71xx/image/generic.mk
> +++ b/target/linux/ar71xx/image/generic.mk
> @@ -1063,8 +1063,12 @@ define Device/NBG6616
>    IMAGE_SIZE := 15323k
>    MTDPARTS := spi0.0:192k(u-boot)ro,64k(env)ro,64k(RFdata)ro,384k(zyxel_rfsd),384k(romd),64k(header),2048k(kernel),13184k(rootfs),15232k@0x120000(firmware)
>    CMDLINE += mem=128M
> -  IMAGES := sysupgrade.bin
> +  RAS_BOARD := NBG6616
> +  RAS_ROOTFS_SIZE := 14464k
> +  RAS_VERSION := "$(VERSION_DIST) $(REVISION)"
> +  IMAGES := factory.bin sysupgrade.bin
>    KERNEL := kernel-bin | patch-cmdline | lzma | uImage lzma | jffs2 boot/vmlinux.lzma.uImage
> +  IMAGE/factory.bin := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-rootfs | pad-rootfs | pad-to 64k | check-size $$$$(IMAGE_SIZE) | zyxel-ras-image
>    IMAGE/sysupgrade.bin := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-rootfs | pad-rootfs | check-size $$$$(IMAGE_SIZE)
>    # We cannot currently build a factory image. It is the sysupgrade image
>    # prefixed with a header (which is actually written into the MTD device).
> diff --git a/target/linux/ipq40xx/image/Makefile
> b/target/linux/ipq40xx/image/Makefile index
> d1ee1004fd..5a08312d2c 100644
> --- a/target/linux/ipq40xx/image/Makefile
> +++ b/target/linux/ipq40xx/image/Makefile
> @@ -221,7 +221,7 @@ define Device/zyxel_nbg6617
>  #	at least as large as the one of the initial firmware image (not the current
>  #	one on the device). This only applies to the Web-UI, the bootlaoder ignores
>  #	this minimum-size. However, the larger image can be flashed both ways.
> -	IMAGE/factory.bin := append-rootfs | pad-rootfs | check-size $$$$(ROOTFS_SIZE) | make-ras
> +	IMAGE/factory.bin := append-rootfs | pad-rootfs | pad-to 64k | check-size $$$$(ROOTFS_SIZE) | zyxel-ras-image seperate-kernel
>  	IMAGE/sysupgrade.bin/squashfs := append-rootfs | pad-rootfs | check-size $$$$(ROOTFS_SIZE) | sysupgrade-tar rootfs=$$$$@ | append-metadata
>  	DEVICE_PACKAGES := ipq-wifi-zyxel_nbg6617 uboot-envtools
>  endef
> diff --git a/target/linux/ipq806x/image/Makefile
> b/target/linux/ipq806x/image/Makefile index
> 2902af3231..cbb03272fb 100644
> --- a/target/linux/ipq806x/image/Makefile
> +++ b/target/linux/ipq806x/image/Makefile
> @@ -67,7 +67,8 @@ define Device/ZyXELImage
>  	KERNEL_SUFFIX := -uImage
>  	KERNEL = kernel-bin | append-dtb | uImage none | pad-to $${KERNEL_SIZE}
>  	KERNEL_NAME := zImage
> -	IMAGES := sysupgrade.bin mmcblk0p5-rootfs.bin mmcblk0p4-kernel.bin
> +	IMAGES := factory.bin sysupgrade.bin mmcblk0p5-rootfs.bin mmcblk0p4-kernel.bin
> +	IMAGE/factory.bin := append-rootfs | pad-rootfs | pad-to $$$$(BLOCKSIZE) | zyxel-ras-image seperate-kernel
>  	IMAGE/sysupgrade.bin/squashfs := append-rootfs | pad-to $$$${BLOCKSIZE} | sysupgrade-tar rootfs=$$$$@ | append-metadata
>  	IMAGE/mmcblk0p5-rootfs.bin := append-rootfs | pad-rootfs | pad-to $$$${BLOCKSIZE}
>  	IMAGE/mmcblk0p4-kernel.bin := append-kernel
> @@ -245,6 +246,9 @@ define Device/zyxel_nbg6817
>  	KERNEL_SIZE := 4096k
>  	BLOCKSIZE := 64k
>  	BOARD_NAME := nbg6817
> +	RAS_BOARD := NBG6817
> +	RAS_ROOTFS_SIZE := 20934k
> +	RAS_VERSION := "$(VERSION_DIST) $(REVISION)"
>  	SUPPORTED_DEVICES += nbg6817
>  	DEVICE_TITLE := ZyXEL NBG6817
>  	DEVICE_PACKAGES := ath10k-firmware-qca9984 e2fsprogs kmod-fs-ext4 losetup
> diff --git a/tools/firmware-utils/Makefile
> b/tools/firmware-utils/Makefile index 436a43794c..00917c3417
> 100644
> --- a/tools/firmware-utils/Makefile
> +++ b/tools/firmware-utils/Makefile
> @@ -70,6 +70,7 @@ define Host/Compile
>  	$(call cc,fix-u-media-header cyg_crc32,-Wall)
>  	$(call cc,hcsmakeimage bcmalgo)
>  	$(call cc,mkporayfw, -Wall)
> +	$(call cc,mkrasimage, --std=gnu99)
>  	$(call cc,mkhilinkfw, -lcrypto)
>  	$(call cc,mkdcs932, -Wall)
>  	$(call cc,mkheader_gemtek,-lz)
> diff --git a/tools/firmware-utils/src/mkrasimage.c
> b/tools/firmware-utils/src/mkrasimage.c new file mode 100644
> index 0000000000..f0f8a54371
> --- /dev/null
> +++ b/tools/firmware-utils/src/mkrasimage.c
> @@ -0,0 +1,474 @@
> +/*
> + * --- ZyXEL header format ---
> + * Original Version by Benjamin Berg <benjamin@sipsolutions.net>
> + * C implementation based on generation-script by Christian Lamparter <chunkeey@gmail.com>
> + *
> + * The firmware image prefixed with a header (which is written into the MTD device).
> + * The header is one erase block (~64KiB) in size, but the checksum only convers the
> + * first 2KiB. Padding is 0xff. All integers are in big-endian.
> + *
> + * The checksum is always a 16-Bit System V checksum (sum -s) stored in a 32-Bit integer.
> + *
> + *   4 bytes:  checksum of the rootfs image
> + *   4 bytes:  length of the contained rootfs image file (big endian)
> + *  32 bytes:  Firmware Version string (NUL terminated, 0xff padded)
> + *   4 bytes:  checksum over the header partition (big endian - see below)
> + *  64 bytes:  Model (e.g. "NBG6617", NUL termiated, 0xff padded)
> + *   4 bytes:  checksum of the kernel partition
> + *   4 bytes:  length of the contained kernel image file (big endian)
> + *      rest:  0xff padding (To erase block size)
> + *
> + * The kernel partition checksum and length is not used for every device.
> + * If it's notused, pad those 8 bytes with 0xFF.
> + *
> + * The checksums are calculated by adding up all bytes and if a 16bit
> + * overflow occurs, one is added and the sum is masked to 16 bit:
> + *   csum = csum + databyte; if (csum > 0xffff) { csum += 1; csum &= 0xffff };
> + * Should the file have an odd number of bytes then the byte len-0x800 is
> + * used additionally.
> + *
> + * The checksum for the header is calculated over the first 2048 bytes with
> + * the rootfs image checksum as the placeholder during calculation.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published
> + * by the Free Software Foundation.
> + *
> + */
> +#include <fcntl.h>
> +#include <getopt.h>
> +#include <libgen.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +#include <sys/mman.h>
> +#include <sys/stat.h>
> +
> +#define VERSION_STRING_LEN 31
> +#define ROOTFS_HEADER_LEN 40
> +
> +#define KERNEL_HEADER_LEN 8
> +
> +#define BOARD_NAME_LEN 64
> +#define BOARD_HEADER_LEN 68
> +
> +#define HEADER_PARTITION_CALC_LENGTH 2048
> +#define HEADER_PARTITION_LENGTH 0x10000
> +
> +struct file_info {
> +    char *name;    /* name of the file */
> +    char *data;    /* file content */
> +    size_t size;   /* length of the file */
> +};
> +
> +static char *progname;
> +
> +static char *board_name = 0;
> +static char *version_name = 0;
> +static unsigned int rootfs_size = 0;
> +
> +static struct file_info kernel = { NULL, NULL, 0 };
> +static struct file_info rootfs = { NULL, NULL, 0 };
> +static struct file_info rootfs_out = { NULL, NULL, 0 };
> +static struct file_info out = { NULL, NULL, 0 };
> +
> +#define ERR(fmt, ...) do { \
> +    fprintf(stderr, "[%s] *** error: " fmt "\n", \
> +            progname, ## __VA_ARGS__ ); \
> +} while (0)
> +
> +void map_file(struct file_info *finfo)
> +{
> +    struct stat file_stat = {0};
> +    int fd;
> +
> +    fd = open(finfo->name, O_RDONLY, (mode_t)0600);
> +    if (fd == -1) {
> +        ERR("Error while opening file %s.", finfo->name);
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    if (fstat(fd, &file_stat) == -1) {
> +        ERR("Error getting file size for %s.", finfo->name);
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    finfo->size = file_stat.st_size;
> +    finfo->data = mmap(0, finfo->size, PROT_READ, MAP_SHARED, fd, 0);
> +
> +    if (finfo->data == MAP_FAILED) {
> +        ERR("Error mapping file %s.", finfo->name);
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    close(fd);
> +}
> +
> +void unmap_file(struct file_info *finfo)
> +{
> +    if(munmap(finfo->data, finfo->size) == -1) {
> +        ERR("Error unmapping file %s.", finfo->name);
> +        exit(EXIT_FAILURE);
> +    }
> +}
> +
> +void write_file(struct file_info *finfo)
> +{
> +    FILE *fout = fopen(finfo->name, "w");
> +
> +    fwrite(finfo->data, finfo->size, 1, fout);
> +
> +    if (ferror(fout)) {
> +        ERR("Wanted to write, but something went wrong.");
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    fclose(fout);
> +}
> +
> +void usage(int status)
> +{
> +    FILE *stream = (status != EXIT_SUCCESS) ? stderr : stdout;
> +
> +    fprintf(stream, "Usage: %s [OPTIONS...]\n", progname);
> +    fprintf(stream,
> +            "\n"
> +            "Options:\n"
> +            "  -k <kernel>     path for kernel image\n"
> +            "  -r <rootfs>     path for rootfs image\n"
> +            "  -s <rfssize>    size of output rootfs\n"
> +            "  -v <version>    version string\n"
> +            "  -b <boardname>  name of board to generate image for\n"
> +            "  -o <out_name>   name of output image\n"
> +            "  -h              show this screen\n"
> +    );
> +
> +    exit(status);
> +}
> +
> +static unsigned int get_big_endian(const unsigned int in)
> +{
> +    int x = 1;
> +    unsigned int out;
> +
> +    /*
> +     * checksum is stored as big-endian in header, so we need to convert it
> +     * in case we are on a litte-endian system.
> +     */
> +    if (*((char *) &x) == 1) {
> +        /* Little endian, need to convert */
> +        out = ((in << 8) & 0xFF00FF00 ) | ((in >> 8) & 0x00FF00FF );
> +        out = (out << 16) | (out >> 16);
> +    }
> +    return out;
> +}
> +
> +static int sysv_chksm(const unsigned char *data, int size)
> +{
> +    int r;
> +    int checksum;
> +    unsigned int s = 0; /* The sum of all the input bytes, modulo (UINT_MAX + 1).  */
> +
> +
> +    for (int i = 0; i < size; i++) {
> +        s += data[i];
> +    }
> +
> +    r = (s & 0xffff) + ((s & 0xffffffff) >> 16);
> +    checksum = (r & 0xffff) + (r >> 16);
> +
> +    return checksum;
> +}
> +
> +static int zyxel_chksm(const unsigned char *data, int size)
> +{
> +     return get_big_endian(sysv_chksm(data, size));
> +}
> +
> +char *generate_rootfs_header(struct file_info filesystem, char
> *version)
> +{
> +    size_t version_string_length;
> +    unsigned int chksm, size;
> +    char *rootfs_header;
> +    size_t ptr = 0;
> +
> +    rootfs_header = malloc(ROOTFS_HEADER_LEN);
> +    if (!rootfs_header) {
> +        ERR("Couldn't allocate memory for rootfs header!");
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    /* Prepare padding for firmware-version string here */
> +    memset(rootfs_header, 0xff, ROOTFS_HEADER_LEN);
> +
> +    chksm = zyxel_chksm((const unsigned char *)filesystem.data, filesystem.size);
> +    size = get_big_endian(filesystem.size);
> +
> +    /* 4 bytes:  checksum of the rootfs image */
> +    memcpy(rootfs_header + ptr, &chksm, 4);
> +    ptr += 4;
> +
> +    /* 4 bytes:  length of the contained rootfs image file (big endian) */
> +    memcpy(rootfs_header + ptr, &size, 4);
> +    ptr += 4;
> +
> +    /* 32 bytes:  Firmware Version string (NUL terminated, 0xff padded) */
> +    version_string_length = strlen(version) <= VERSION_STRING_LEN ? strlen(version) : VERSION_STRING_LEN;
> +    memcpy(rootfs_header + ptr, version, version_string_length);
> +    ptr += version_string_length;
> +    /* Add null-terminator */
> +    rootfs_header[ptr] = 0x0;
> +
> +    return rootfs_header;
> +}
> +
> +char *generate_kernel_header(struct file_info kernel)
> +{
> +    unsigned int chksm, size;
> +    char *kernel_header;
> +    size_t ptr = 0;
> +
> +    kernel_header = malloc(KERNEL_HEADER_LEN);
> +    if (!kernel_header) {
> +        ERR("Couldn't allocate memory for kernel header!");
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    chksm = zyxel_chksm((const unsigned char *)kernel.data, kernel.size);
> +    size = get_big_endian(kernel.size);
> +
> +    /* 4 bytes:  checksum of the kernel image */
> +    memcpy(kernel_header + ptr, &chksm, 4);
> +    ptr += 4;
> +
> +    /* 4 bytes:  length of the contained kernel image file (big endian) */
> +    memcpy(kernel_header + ptr, &size, 4);
> +
> +    return kernel_header;
> +}
> +
> +unsigned int generate_board_header_checksum(char *kernel_hdr,
> char *rootfs_hdr, char *boardname)
> +{
> +    char *board_hdr_tmp;
> +    unsigned int sum;
> +    size_t ptr = 0;
> +
> +    /*
> +     * The checksum of the board header is calculated over the first 2048 bytes of
> +     * the header partition with the rootfs checksum used as a placeholder for then
> +     * board checksum we calculate in this step. The checksum gained from this step
> +     * is then used for the final board header partition.
> +     */
> +
> +    board_hdr_tmp = malloc(HEADER_PARTITION_CALC_LENGTH);
> +    if (!board_hdr_tmp) {
> +        ERR("Couldn't allocate memory for temporary board header!");
> +        exit(EXIT_FAILURE);
> +    }
> +    memset(board_hdr_tmp, 0xff, HEADER_PARTITION_CALC_LENGTH);
> +
> +    /* 40 bytes:  RootFS header */
> +    memcpy(board_hdr_tmp, rootfs_hdr, ROOTFS_HEADER_LEN);
> +    ptr += ROOTFS_HEADER_LEN;
> +
> +    /* 4 bytes:  RootFS checksum (BE) as placeholder for board-header checksum */
> +    memcpy(board_hdr_tmp + ptr, rootfs_hdr, 4);
> +    ptr += 4;
> +
> +    /* 32 bytes:  Model (e.g. "NBG6617", NUL termiated, 0xff padded) */
> +    memcpy(board_hdr_tmp + ptr, boardname, strlen(boardname));
> +    ptr += strlen(boardname);
> +    /* Add null-terminator */
> +    board_hdr_tmp[ptr] = 0x0;
> +    ptr = ROOTFS_HEADER_LEN + 4 + BOARD_NAME_LEN;
> +
> +    /* 8 bytes:  Kernel header */
> +    if (kernel_hdr)
> +        memcpy(board_hdr_tmp + ptr, kernel_hdr, 8);
> +
> +    /* Calculate the checksum over the first 2048 bytes */
> +    sum = zyxel_chksm((const unsigned char *)board_hdr_tmp, HEADER_PARTITION_CALC_LENGTH);
> +    free(board_hdr_tmp);
> +    return sum;
> +}
> +
> +char *generate_board_header(char *kernel_hdr, char
> *rootfs_hdr, char *boardname)
> +{
> +    unsigned int board_checksum;
> +    char *board_hdr;
> +
> +    board_hdr = malloc(BOARD_HEADER_LEN);
> +    if (!board_hdr) {
> +        ERR("Couldn't allocate memory for board header!");
> +        exit(EXIT_FAILURE);
> +    }
> +    memset(board_hdr, 0xff, BOARD_HEADER_LEN);
> +
> +    /* 4 bytes:  checksum over the header partition (big endian) */
> +    board_checksum = generate_board_header_checksum(kernel_hdr, rootfs_hdr, boardname);
> +    memcpy(board_hdr, &board_checksum, 4);
> +
> +    /* 32 bytes:  Model (e.g. "NBG6617", NUL termiated, 0xff padded) */
> +    memcpy(board_hdr + 4, boardname, strlen(boardname));
> +    board_hdr[4 + strlen(boardname)] = 0x0;
> +
> +    return board_hdr;
> +}
> +
> +int build_image()
> +{
> +    char *rootfs_header = NULL;
> +    char *kernel_header = NULL;
> +    char *board_header = NULL;
> +
> +    size_t ptr;
> +
> +    /* Load files */
> +    if (kernel.name)
> +        map_file(&kernel);
> +    map_file(&rootfs);
> +
> +    /*
> +     * Allocate memory and copy input rootfs for temporary output rootfs.
> +     * This is important as we have to generate the rootfs checksum over the
> +     * entire rootfs partition. As we might have to pad the partition to allow
> +     * for flashing via ZyXEL's Web-GUI, we prepare the rootfs partition for the
> +     * output image here (and also use it for calculating the rootfs checksum).
> +     *
> +     * The roofs padding has to be done with 0x00.
> +     */
> +    rootfs_out.data = calloc(rootfs_out.size, sizeof(char));
> +    memcpy(rootfs_out.data, rootfs.data, rootfs.size);
> +
> +    /* Prepare headers */
> +    rootfs_header = generate_rootfs_header(rootfs_out, version_name);
> +    if (kernel.name)
> +        kernel_header = generate_kernel_header(kernel);
> +    board_header = generate_board_header(kernel_header, rootfs_header, board_name);
> +
> +    /* Prepare output file */
> +    out.size = HEADER_PARTITION_LENGTH + rootfs_out.size;
> +    if (kernel.name)
> +        out.size += kernel.size;
> +    out.data = malloc(out.size);
> +    memset(out.data, 0xFF, out.size);
> +
> +    /* Build output image */
> +    memcpy(out.data, rootfs_header, ROOTFS_HEADER_LEN);
> +    memcpy(out.data + ROOTFS_HEADER_LEN, board_header, BOARD_HEADER_LEN);
> +    if (kernel.name)
> +        memcpy(out.data + ROOTFS_HEADER_LEN + BOARD_HEADER_LEN, kernel_header, KERNEL_HEADER_LEN);
> +    ptr = HEADER_PARTITION_LENGTH;
> +    memcpy(out.data + ptr, rootfs_out.data, rootfs_out.size);
> +    ptr += rootfs_out.size;
> +    if (kernel.name)
> +        memcpy(out.data + ptr, kernel.data, kernel.size);
> +
> +    /* Write back output image */
> +    write_file(&out);
> +
> +    /* Free allocated memory */
> +    if (kernel.name)
> +        unmap_file(&kernel);
> +    unmap_file(&rootfs);
> +    free(out.data);
> +    free(rootfs_out.data);
> +
> +    free(rootfs_header);
> +    if (kernel.name)
> +        free(kernel_header);
> +    free(board_header);
> +
> +    return 0;
> +}
> +
> +int check_options()
> +{
> +    if (!rootfs.name) {
> +        ERR("No rootfs filename supplied");
> +        return -2;
> +    }
> +
> +    if (!out.name) {
> +        ERR("No output filename supplied");
> +        return -3;
> +    }
> +
> +    if (!board_name) {
> +        ERR("No board-name supplied");
> +        return -4;
> +    }
> +
> +    if (!version_name) {
> +        ERR("No version supplied");
> +        return -5;
> +    }
> +
> +    if (rootfs_size <= 0) {
> +        ERR("Invalid rootfs size supplied");
> +        return -6;
> +    }
> +
> +    if (strlen(board_name) > 31) {
> +        ERR("Board name is to long");
> +        return -7;
> +    }
> +    return 0;
> +}
> +
> +int main(int argc, char *argv[])
> +{
> +    int ret;
> +    progname = basename(argv[0]);
> +    while (1) {
> +        int c;
> +
> +        c = getopt(argc, argv, "b:k:o:r:s:v:h");
> +        if (c == -1)
> +            break;
> +
> +        switch (c) {
> +            case 'b':
> +                board_name = optarg;
> +                break;
> +            case 'h':
> +                usage(EXIT_SUCCESS);
> +                break;
> +            case 'k':
> +                kernel.name = optarg;
> +                break;
> +            case 'o':
> +                out.name = optarg;
> +                break;
> +            case 'r':
> +                rootfs.name = optarg;
> +                break;
> +            case 's':
> +                sscanf(optarg, "%u", &rootfs_size);
> +                break;
> +            case 'v':
> +                version_name = optarg;
> +                break;
> +            default:
> +                usage(EXIT_FAILURE);
> +                break;
> +        }
> +    }
> +
> +    ret = check_options();
> +    if (ret)
> +        usage(EXIT_FAILURE);
> +
> +    /* As ZyXEL Web-GUI only accept images with a rootfs equal or larger than the first firmware shipped
> +     * for the device, we need to pad rootfs partition to this size. To perform further calculations, we
> +     * decide the size of this part here. In case the rootfs we want to integrate in our image is larger,
> +     * take it's size, otherwise the supplied size.
> +     *
> +     * Be careful! We rely on assertion of correct size to be performed beforehand. It is unknown if images
> +     * with a to large rootfs are accepted or not.
> +     */
> +    rootfs_out.size = rootfs_size < rootfs.size ? rootfs.size : rootfs_size;
> +    return build_image();
> +}

Hi

On 2018-08-20, David Bauer wrote:
> The current make-ras.sh image generation script for the ZyXEL NBG6617
> has portability issues with bash. Because of this, factory images are
> currently not built correctly by the OpenWRT buildbots.
> 
> This commit replaces the make-ras.sh by C-written mkrasimage. The old
> script is still kept but can be deleted in the future.
> 
> The new mkrasimage is also compatible with other ZyXEL devices using
> the ras image-format. This is not tested with a OpenWRT build but
> it correctly builds the header for ZyXEL factory images for all devices
> it is added to.
> 
> Signed-off-by: David Bauer <mail@david-bauer.net>

with the caveat explained below:
Tested-by: Stefan Lippers-Hollmann <s.l-h@gmx.de>

> v2 changes:
>  - Rework image-generation code
>  - Add factory image for NBG6616
>  - Add factory image for NBG6817

Thanks a lot for looking into this!
I've given this a test on my ZyXEL NBG6817 and it's basically working, 
ZyXEL just decided to prevent downgrades from the firmware upgrade
functionality of their webinterface, which means it parses the 
RAS_VERSION and rejects the currently chosen version number.
Flashing this image from a tftpd avoids the strict version 
requirements and can flash the image as-is (press WPS key while 
power-on, have the factory image renamed to "ras.bin" available from a 
tftpd at 192.168.1.99).

[...]
> diff --git a/target/linux/ipq806x/image/Makefile b/target/linux/ipq806x/image/Makefile
> index 2902af3231..cbb03272fb 100644
> --- a/target/linux/ipq806x/image/Makefile
> +++ b/target/linux/ipq806x/image/Makefile
[...]
> @@ -245,6 +246,9 @@ define Device/zyxel_nbg6817
>  	KERNEL_SIZE := 4096k
>  	BLOCKSIZE := 64k
>  	BOARD_NAME := nbg6817
> +	RAS_BOARD := NBG6817
> +	RAS_ROOTFS_SIZE := 20934k
> +	RAS_VERSION := "$(VERSION_DIST) $(REVISION)"
[...]

For now, I've supplied a valid/ future firmware version for testing

RAS_VERSION := "V1.00(ABCS.9)C0"

and it flashes nicely from the webinterface (until ZyXEL actually 
releases this version, which is next in line). I'll have a look if I
can track down the version check and supply something that's both a
bit closer to OpenWrt and more future safe, but I'd suggest to go 
with "V1.00(ABCS.9)C0" for now, as that has been confirmed to work
for the next couple of weeks.

Flashed OpenWrt master image via web-interface, with a fake ZyXEL version
number:

# hexdump -C /dev/mmcblk0p6 | head -n6
00000000  00 00 7a 9f 01 47 18 00  56 31 2e 30 30 28 41 42  |..z..G..V1.00(AB|
00000010  43 53 2e 39 29 43 30 00  ff ff ff ff ff ff ff ff  |CS.9)C0.........|
00000020  ff ff ff ff ff ff ff ff  00 00 d5 73 4e 42 47 36  |...........sNBG6|
00000030  38 31 37 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |817.............|
00000040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*

Flashed via tftpd, using OpenWrt's preliminary version info:

# hexdump -C /dev/mmcblk0p3 | head -n6
00000000  00 00 7a 9f 01 47 18 00  4f 70 65 6e 57 72 74 20  |..z..G..OpenWrt |
00000010  72 37 38 38 33 2d 66 63  39 63 62 66 33 62 63 30  |r7883-fc9cbf3bc0|
00000020  00 ff ff ff ff ff ff ff  00 00 d0 e0 4e 42 47 36  |............NBG6|
00000030  38 31 37 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |817.............|
00000040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*

Regards, thank you very much!
	Stefan Lippers-Hollmann

Hi

On 2018-08-21, Stefan Lippers-Hollmann wrote:
> On 2018-08-20, David Bauer wrote:
[...]
> For now, I've supplied a valid/ future firmware version for testing
> 
> RAS_VERSION := "V1.00(ABCS.9)C0"

While perhaps not ideal yet, this is accepted by the OEM webinterface 
and should be safe to use for the time being.

RAS_VERSION := "V1.99(OWRT.7890)C0"

# hexdump -C /dev/mmcblk0p6 
00000000  00 00 b8 49 01 47 18 00  56 31 2e 39 39 28 4f 57  |...I.G..V1.99(OW|
00000010  52 54 2e 37 38 39 30 29  43 30 00 ff ff ff ff ff  |RT.7890)C0......|
00000020  ff ff ff ff ff ff ff ff  00 00 d3 3f 4e 42 47 36  |...........?NBG6|
00000030  38 31 37 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |817.............|
00000040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00000060  ff ff ff ff ff ff ff ff  ff ff ff ff 00 00 98 e5  |................|
00000070  00 40 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |.@..............|
00000080  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00010000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000

Regards
	Stefan Lippers-Hollmann

On 2018-08-21, Stefan Lippers-Hollmann wrote:
> Hi
> 
> On 2018-08-21, Stefan Lippers-Hollmann wrote:
> > On 2018-08-20, David Bauer wrote:  
> [...]
> > For now, I've supplied a valid/ future firmware version for testing
> > 
> > RAS_VERSION := "V1.00(ABCS.9)C0"  
> 
> While perhaps not ideal yet, this is accepted by the OEM webinterface 
> and should be safe to use for the time being.
> 
> RAS_VERSION := "V1.99(OWRT.7890)C0"

Sorry, this was meant to read:

RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed s/^r//))C0"

# hexdump -C /dev/mmcblk0p3
00000000  00 00 c4 38 01 47 18 00  56 31 2e 39 39 28 4f 57  |...8.G..V1.99(OW|
00000010  52 54 2e 37 38 39 30 2d  34 30 65 62 39 62 64 61  |RT.7890-40eb9bda|
00000020  34 34 29 43 30 00 ff ff  00 00 cb 60 4e 42 47 36  |44)C0......`NBG6|
00000030  38 31 37 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |817.............|
00000040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00000060  ff ff ff ff ff ff ff ff  ff ff ff ff 00 00 98 e5  |................|
00000070  00 40 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |.@..............|
00000080  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00010000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000

Regards
	Stefan Lippers-Hollmann

Hi,

> RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed s/^r//))C0"

make that  RAS_VERSION := "V1.99(OWRT.$(patsubst r%,%,$(REVISION)))C0"
to avoid spawning a shell + sed process.

~ Jo

Hi

On 2018-08-21, Jo-Philipp Wich wrote:
> Hi,
> 
> > RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed s/^r//))C0"  

While this passes the version check, the appended git hash does break 
later on in the flashing process - so at the moment only the numerical
revision (of DISTRIB_REVISION='r7890-40eb9bda44') seems to be safe:

RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed -e s/^r// -e s/\\-.*//))C0"

# hexdump -C /dev/mmcblk0p6 
00000000  00 00 b4 bc 01 47 18 00  56 31 2e 39 39 28 4f 57  |.....G..V1.99(OW|
00000010  52 54 2e 37 38 39 30 29  43 30 00 ff ff ff ff ff  |RT.7890)C0......|
00000020  ff ff ff ff ff ff ff ff  00 00 d4 1d 4e 42 47 36  |............NBG6|
00000030  38 31 37 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |817.............|
00000040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00000060  ff ff ff ff ff ff ff ff  ff ff ff ff 00 00 98 e5  |................|
00000070  00 40 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |.@..............|
00000080  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00010000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000

> make that  RAS_VERSION := "V1.99(OWRT.$(patsubst r%,%,$(REVISION)))C0"
> to avoid spawning a shell + sed process.

I'll have a look at improving this tonight, thanks.

Regards
	Stefan Lippers-Hollmann

Hi,

On 8/21/18 8:31 AM, Stefan Lippers-Hollmann wrote:
> While this passes the version check, the appended git hash does break 
> later on in the flashing process - so at the moment only the numerical
> revision (of DISTRIB_REVISION='r7890-40eb9bda44') seems to be safe:
> 
> RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed -e s/^r// -e s/\\-.*//))C0"

Does the version string contained in the RAS-header get exposed to a
user in any way? If not, i would go for using a static (high enough so
it probably never get's used by ZyXEL) version string and call it a day
before we are over-complicating things here.

The NBG6617 btw. does not have such a validation (Downgrades using older
ZyXEL-firmware are also possible using the Web-UI)

Best wishes
David

Hi

On 2018-08-21, David Bauer wrote:
> On 8/21/18 8:31 AM, Stefan Lippers-Hollmann wrote:
> > While this passes the version check, the appended git hash does break 
> > later on in the flashing process - so at the moment only the numerical
> > revision (of DISTRIB_REVISION='r7890-40eb9bda44') seems to be safe:
> > 
> > RAS_VERSION := "V1.99(OWRT.$$(shell echo $(REVISION) | sed -e s/^r// -e s/\\-.*//))C0"  
> 
> Does the version string contained in the RAS-header get exposed to a
> user in any way? If not, i would go for using a static (high enough so
> it probably never get's used by ZyXEL) version string and call it a day
> before we are over-complicating things here.
[...]

The OEM firmware doesn't expose any information about the other 
partition set, OpenWrt could make use of it via 
luci-app-advanced-reboot  (but hasn't been doing that so far, as
OpenWrt installs and sysupgrades don't update the header partition
so far).

But hardcoding RAS_VERSION := "V1.99(OWRT.9999)C0 should be fine for 
the forseeable future as well.

Regards
	Stefan Lippers-Hollmann