Message ID | 1476992420-3744-1-git-send-email-trini@konsulko.com |
---|---|
State | Accepted |
Delegated to: | Tom Rini |
Headers | show |
On 10/20/2016 01:40 PM, Tom Rini wrote: > - Add a PPA for a more recent qemu (required for PowerPC to work) > - Add tests to run test.py for various QEMU platforms. This relies on > swarren's uboot-test-hooks repository to provide the abstractions. Overall, the series, Acked-by: Stephen Warren <swarren@nvidia.com> A few comments on this particular patch below: > diff --git a/.travis.yml b/.travis.yml > addons: > apt: > + sources: > + - sourceline: 'ppa:gns3/qemu' Have you co-ordinated with "qns3" to ensure that PPA is always going to be available, and host binaries for Ubuntuy 16.04 (some PPAs eventually disappear as the owner moves on to other things, or change the set of distro releases they support at the owner's whim). > + # Clone uboot-test-hooks > + - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks Does it make sense to rely on my copy of the repo being around all the time? I have no intention of removing it, but when I created it I'd assumed others would create their own clones for their own infra-structure. I wonder if all the dependencies for U-Boot's testing should be hosted on U-Boot infra-structure? That all said, this should be fine; just something to ponder. > + - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` It may be useful to add the following too: - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` That way, if we ever need per-board test configuration files, the path to access them will already be available. > + - env: > + - CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-" > + script: > + - ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build; > + ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build; > + ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build; Does it make sense to make that 3 separate scripts? I assume that would (a) allow the individual status of the 3 test.py invocations to be seen separately (b) might allow/cause travis-ci to run them in parallel; I'm not sure if this Travis config file is a sequential list of commands, or a list of potentially parallel actions.
On Fri, Oct 21, 2016 at 12:24:07PM -0600, Stephen Warren wrote: > On 10/20/2016 01:40 PM, Tom Rini wrote: > >- Add a PPA for a more recent qemu (required for PowerPC to work) > >- Add tests to run test.py for various QEMU platforms. This relies on > > swarren's uboot-test-hooks repository to provide the abstractions. > > Overall, the series, > Acked-by: Stephen Warren <swarren@nvidia.com> > > A few comments on this particular patch below: > > >diff --git a/.travis.yml b/.travis.yml > > > addons: > > apt: > >+ sources: > >+ - sourceline: 'ppa:gns3/qemu' > > Have you co-ordinated with "qns3" to ensure that PPA is always going > to be available, and host binaries for Ubuntuy 16.04 (some PPAs > eventually disappear as the owner moves on to other things, or > change the set of distro releases they support at the owner's whim). I don't like this part either. I couldn't find an "official" PPA to provide a newer qemu nor does it appear to be in the official backports channel either. My current plan is that if something happens to the PPA I'll have to host one myself. > >+ # Clone uboot-test-hooks > >+ - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks > > Does it make sense to rely on my copy of the repo being around all > the time? I have no intention of removing it, but when I created it > I'd assumed others would create their own clones for their own > infra-structure. I wonder if all the dependencies for U-Boot's > testing should be hosted on U-Boot infra-structure? A good point. The way I've been thinking of it is that we also need a repo with as many and varied configs as possible, which has become yours :). That said, we could move a copy of this over to the u-boot account on github if you'd like. > That all said, this should be fine; just something to ponder. > > >+ - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` > > It may be useful to add the following too: > > - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` > > That way, if we ever need per-board test configuration files, the > path to access them will already be available. A good point. I think we could get the network tests going, maybe with a little re-work to dynamically hash the file from the host side? But that's an aside. > >+ - env: > >+ - CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-" > >+ script: > >+ - ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build; > >+ ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build; > >+ ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build; > > Does it make sense to make that 3 separate scripts? I assume that > would (a) allow the individual status of the 3 test.py invocations > to be seen separately (b) might allow/cause travis-ci to run them in > parallel; I'm not sure if this Travis config file is a sequential > list of commands, or a list of potentially parallel actions. So, the way travis-ci works is that you can (and we do) split tasks up to run in parallel. The hunk above is a series to run in sequence on the same host. The downside to moving this to parallel is the amount of duplication you need in each stanza. We could fix that (at least in the case of non-buildman-fetched toolchains) if test.py used buildman to build, as that will determine CROSS_COMPILE correctly from the default .buildman file we create earlier on. Thanks!
On Thu, Oct 20, 2016 at 03:40:20PM -0400, Tom Rini wrote: > - Add a PPA for a more recent qemu (required for PowerPC to work) > - Add tests to run test.py for various QEMU platforms. This relies on > swarren's uboot-test-hooks repository to provide the abstractions. > > Signed-off-by: Tom Rini <trini@konsulko.com> After adding in Stephen's suggested edit, applied to u-boot/master, thanks!
On 10/22/2016 06:56 AM, Tom Rini wrote: > On Fri, Oct 21, 2016 at 12:24:07PM -0600, Stephen Warren wrote: >> On 10/20/2016 01:40 PM, Tom Rini wrote: >>> - Add a PPA for a more recent qemu (required for PowerPC to work) >>> - Add tests to run test.py for various QEMU platforms. This relies on >>> swarren's uboot-test-hooks repository to provide the abstractions. >>> + # Clone uboot-test-hooks >>> + - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks >> >> Does it make sense to rely on my copy of the repo being around all >> the time? I have no intention of removing it, but when I created it >> I'd assumed others would create their own clones for their own >> infra-structure. I wonder if all the dependencies for U-Boot's >> testing should be hosted on U-Boot infra-structure? > > A good point. The way I've been thinking of it is that we also need a > repo with as many and varied configs as possible, which has become yours > :). That said, we could move a copy of this over to the u-boot account > on github if you'd like. I think it's fine for now. >> That all said, this should be fine; just something to ponder. >> >>> + - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` >> >> It may be useful to add the following too: >> >> - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` >> >> That way, if we ever need per-board test configuration files, the >> path to access them will already be available. > > A good point. I think we could get the network tests going, maybe with > a little re-work to dynamically hash the file from the host side? But > that's an aside. Yes, assuming qemu networking works, I expect that would be possible. Yes, I think it'd be pretty easy to dynamically create a test download file and calculate the CRC from the per-board Python config files. >>> + - env: >>> + - CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-" >>> + script: >>> + - ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build; >>> + ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build; >>> + ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build; >> >> Does it make sense to make that 3 separate scripts? I assume that >> would (a) allow the individual status of the 3 test.py invocations >> to be seen separately (b) might allow/cause travis-ci to run them in >> parallel; I'm not sure if this Travis config file is a sequential >> list of commands, or a list of potentially parallel actions. > > So, the way travis-ci works is that you can (and we do) split tasks up > to run in parallel. The hunk above is a series to run in sequence on > the same host. The downside to moving this to parallel is the amount of > duplication you need in each stanza. The duplication doesn't seem like a terribly big deal given it will allow finer-grained status on the main per-build page. > We could fix that (at least in the > case of non-buildman-fetched toolchains) if test.py used buildman to > build, as that will determine CROSS_COMPILE correctly from the default > .buildman file we create earlier on. Thanks! I'll look into whether that's possible. BTW, does/can Travis save the test-log.html from the test.py invocations? That would make debugging any failures a little easier.
On Mon, Oct 24, 2016 at 04:21:36PM -0600, Stephen Warren wrote: [snip] > BTW, does/can Travis save the test-log.html from the test.py > invocations? That would make debugging any failures a little easier. I'm far from a travis "expert". But the real nice thing about it is that you can hook up and test via github. It took maybe 2 minutes to get my u-boot clone going in travis-ci so I could test things out, and that's something I want to emphasize in -rc3 is how easy it is to get a lot more coverage with minimal effort. So since a script clause is just a bit of bash commands basically, some sort of quick wrapper or maybe just pass in -s to test.py since another nice thing is that the web view will collapse output from each step?
diff --git a/.travis.yml b/.travis.yml index 25b0023d631b..1aff5d607d7e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,6 +10,8 @@ language: c addons: apt: + sources: + - sourceline: 'ppa:gns3/qemu' packages: - cppcheck - sloccount @@ -19,6 +21,10 @@ addons: - libsdl1.2-dev - python - python-virtualenv + - qemu-system-arm + - qemu-system-mips + - qemu-system-ppc + - qemu-system-x86 - gcc-powerpc-linux-gnu - gcc-arm-linux-gnueabihf - iasl @@ -27,6 +33,9 @@ install: # install latest device tree compiler - git clone --depth=1 git://git.kernel.org/pub/scm/utils/dtc/dtc.git /tmp/dtc - make -j4 -C /tmp/dtc + # Clone uboot-test-hooks + - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks + - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` # prepare buildman environment - export BUILDMAN_ROOT="root:" - echo -e "[toolchain]\n${BUILDMAN_ROOT} /usr" > ~/.buildman @@ -38,7 +47,7 @@ install: env: global: - - PATH=/tmp/dtc:$PATH + - PATH=/tmp/dtc:/tmp/uboot-test-hooks/bin:$PATH - BUILD_DIR=build - HOSTCC="cc" - HOSTCXX="c++" @@ -167,5 +176,29 @@ matrix: # test/py - script: - ./test/py/test.py --bd sandbox --build + - env: + - CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-" + script: + - ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build; + ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build; + ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build; + - env: + - TOOLCHAIN="mips" + CROSS_COMPILE="${HOME}/.buildman-toolchains/gcc-4.9.0-nolibc/mips-linux/bin/mips-linux-" + script: + - ./test/py/test.py --bd qemu_mips --build -k 'not sleep'; + ./test/py/test.py --bd qemu_mipsel --build -k 'not sleep'; + ./test/py/test.py --bd qemu_mips64 --build -k 'not sleep'; + ./test/py/test.py --bd qemu_mips64el --build -k 'not sleep'; + - env: + - CROSS_COMPILE="/usr/bin/powerpc-linux-gnu-" + script: + - ./test/py/test.py --bd qemu-ppce500 --build -k 'not sleep' + - env: + - TOOLCHAIN="x86_64" + BUILD_ROM=yes + CROSS_COMPILE="${HOME}/.buildman-toolchains/gcc-4.9.0-nolibc/x86_64-linux/bin/x86_64-linux-" + script: + - ./test/py/test.py --bd qemu-x86 --build -k 'not sleep' # TODO make it perfect ;-r
- Add a PPA for a more recent qemu (required for PowerPC to work) - Add tests to run test.py for various QEMU platforms. This relies on swarren's uboot-test-hooks repository to provide the abstractions. Signed-off-by: Tom Rini <trini@konsulko.com> --- Ooops, Stephen has merged my request to add the conf files now, so really reference his repository and not mine. .travis.yml | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-)