RFC: New testing script with remote test capabilities and region awareness

From: David Hendricks <dhendrix@chromium.org>

Hi folks,
As part of an effort underway to get chromium.org's flashrom sync'd with
upstream (for real), I have written an entirely new test script to help get
thru regression testing. It's still rather fugly and will likely go thru
several more revisions, but is currently usable for all major use cases I
could think of and would benefit from some early feedback.

I wrote up a document with usage examples here: https://goo.gl/3jNoL7 .
Long-term the plan is to migrate this to the flashrom wiki.

There were a few "must haves" for this script in the near-term:
- Region awareness. The script understands how to use layout files and
flashmaps (for Chromebooks). Descriptor awareness is on the TODO list for
Intel platforms.
- Can control flashrom on a remote host.
- Can utilize an external programmer in addition to the native programmer
on the target machine. This allows us to verify changes made to programmer
code since one programmer can be considered to be trusted while the other
can have code that was modified.

Longer-term goals include performance testing and flash endurance testing.
We might also add write-protect testing, though there's an open question of
whether that would be better in another script or as a whitebox test (more
on that later...).

Here's the patch as of revision #14 on chromium.org's gerrit server:
https://chromium-review.googlesource.com/#/c/353912/

As in the gerrit message, Signed-off-by: David Hendricks <
dhendrix@chromium.org>

From 353aa26a174d9beb1ec886f7f752473f94023ed8 Mon Sep 17 00:00:00 2001
From: David Hendricks <dhendrix@chromium.org>
Date: Sun, 19 Jun 2016 12:53:22 -0700
Subject: [PATCH] WIP: New test script for flashrom

** work in progress **

Shiny new testing capabilities for Flashrom:
- Region awareness
- Remote testing
- Primary and secondary programmer support
- Autotest-friendly

Long-term goals:
- Performance measurement
- Endurance testing

BUG=chromium:621715
BRANCH=none
TEST=this *is* the test

Change-Id: Ic643cf2edaf7d8772c63ce8119363d882d1b116d
Signed-off-by: David Hendricks <dhendrix@chromium.org>
---
 tests/tests_v2/cmd.sh     | 109 ++++++
 tests/tests_v2/test_v2.sh | 909
++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1018 insertions(+)
 create mode 100644 tests/tests_v2/cmd.sh
 create mode 100644 tests/tests_v2/test_v2.sh

+ test_fail "Failed to verify write of $filename to $region_name"
+ fi
+
+ if [ -n "$SECONDARY_OPTS" ]; then
+ scmd $LOCAL "$OLD_FLASHROM $SECONDARY_OPTS $opts -v -i
${region_name}:${TMPDIR}/${filename}"
+ if [ $? -ne 0 ]; then
+ test_fail "Failed to verify write of $filename to $region_name using
secondary programmer"
+ fi
+ fi
+
+ printf "\tWrote $filename to $region_name region successfully.\n"
+ done
+
+ return $EXIT_SUCCESS
+}
+
+# Do a consistency check for sanity before any other test.
+if [ $SKIP_CONSISTENCY_CHECK -eq 0 ]; then
+ double_read_test 0 $CHIP_SIZE
+fi
+
+if [ $REGION_MODE -eq $REGION_MODE_CLOBBER ]; then
+ random_file="${TMPDIR}/random_${CHIP_SIZE_KB}K.bin"
+ cmp_file="${TMPDIR}/cmp.bin"
+
+ scmd $DO_REMOTE "dd if=/dev/urandom of=$random_file bs=1k
count=${CHIP_SIZE_KB}"
+ scmd $DO_REMOTE "$NEW_FLASHROM $PRIMARY_OPTS -w $random_file"
+ scmd $DO_REMOTE "$OLD_FLASHROM $PRIMARY_OPTS -r $cmp_file"
+ scmd $DO_REMOTE "cmp $random_file $cmp_file"
+ if [ $? -ne 0 ]; then
+ write_backup_image
+ test_fail "Failed to clobber entire ROM."
+ fi
+ scmd $DO_REMOTE "rm -f $cmp_file $random_file"
+
+ simple_partial_write_test "0" "$CHIP_SIZE"
+elif [ $REGION_MODE -eq $REGION_MODE_DESCRIPTOR ]; then
+ # This depends on descriptor regions being translated into internal
+ # layout representation automatically so we can target them using -i.
+ echo "TODO: Descriptor mode"
+ exit $EXIT_FAILURE
+elif [ $REGION_MODE -eq $REGION_MODE_FLASHMAP ]; then
+ region_partial_write_test "$FLASHMAP_REGION" 0
+ if [ $? -ne 0 ]; then
+ echo "Flashmap mode test failed"
+ RC=$EXIT_FAILURE
+ fi
+elif [ $REGION_MODE -eq $REGION_MODE_LAYOUT ]; then
+ rw_layout=""
+ addr=""
+ end=""
+ size=""
+
+ # Look for a region named "RW" with any amount of leading whitespace
+ # and no trailing whitespace or characters.
+ rw_layout=$(grep "\s${LAYOUT_REGION}$" $LAYOUT_FILE | head -n 1)
+ if [ -z "$rw_layout" ]; then
+ printf "No region matching \"${LAYOUT_REGION}\" found layout file
\"%s\"\n" "$LAYOUT_FILE"
+ exit $EXIT_FAILURE
+ fi
+
+ addr="0x$(echo "$rw_layout" | cut -d ' ' -f -1 | awk -F ':' '{ print $1
}')"
+ end="0x$(echo "$rw_layout" | cut -d ' ' -f -1 | awk -F ':' '{ print $2
}')"
+ size="$(($end - $addr + 1))"
+
+ printf "RW region address: ${addr}, size: %u KiB\n" $(($size / $K))
+ simple_partial_write_test "$addr" "$size"
+ if [ $? -ne 0 ]; then
+ echo "Flashmap mode test failed"
+ RC=$EXIT_FAILURE
+ fi
+fi
+
+# restore and cleanup
+write_backup_image
+do_cleanup
+
+if [ $RC -eq 0 ]; then
+ echo "Test status: PASS"
+else
+ echo "Test status: FAIL"
+fi
+
+exit $RC

RFC: New testing script with remote test capabilities and region awareness

Commit Message

Comments

Patch