| Message ID | 20180525141910.6384-6-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | fix buildings of tests/tcg | expand |
(Cc'ing Samuel who replied in a previous thread) Hi Alex, On 05/25/2018 11:19 AM, Alex Bennée wrote: > We can't use our normal Debian based compilers as Alpha isn't an > officially supported architecture. However it is available as a port > and fortunately cross compilers for all these targets are included in > Debian Sid, the perpetual rolling/unstable/testing version of Debian. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> These tags are only for the Alpha part, maybe I wasn't clear enough in my previous review: https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04443.html > --- > v5 > - drop the unused debian-cross-build docker image > - add note to debian-sid docker image > - echo reference to notes on build failure > - include EXTRA_RUNS for mmap tests > --- > tests/docker/Makefile.include | 1 + > .../dockerfiles/debian-alpha-cross.docker | 12 +++++++ > tests/docker/dockerfiles/debian-sid.docker | 32 +++++++++++++++++++ > tests/tcg/alpha/Makefile.include | 2 ++ > tests/tcg/alpha/Makefile.target | 6 ++++ > 5 files changed, 53 insertions(+) > create mode 100644 tests/docker/dockerfiles/debian-alpha-cross.docker > create mode 100644 tests/docker/dockerfiles/debian-sid.docker > create mode 100644 tests/tcg/alpha/Makefile.include > create mode 100644 tests/tcg/alpha/Makefile.target > > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include > index b56386f49e..5dc49c6a54 100644 > --- a/tests/docker/Makefile.include > +++ b/tests/docker/Makefile.include > @@ -62,6 +62,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 > docker-image-debian-s390x-cross: docker-image-debian9 > docker-image-debian-win32-cross: docker-image-debian8-mxe > docker-image-debian-win64-cross: docker-image-debian8-mxe > +docker-image-debian-alpha-cross: docker-image-debian-sid > docker-image-travis: NOUSER=1 > > # Expand all the pre-requistes for each docker image and test combination > diff --git a/tests/docker/dockerfiles/debian-alpha-cross.docker b/tests/docker/dockerfiles/debian-alpha-cross.docker > new file mode 100644 > index 0000000000..29a25d0dfd > --- /dev/null > +++ b/tests/docker/dockerfiles/debian-alpha-cross.docker > @@ -0,0 +1,12 @@ > +# > +# Docker cross-compiler target > +# > +# This docker target builds on the debian sid base image which > +# contains cross compilers for Debian "ports" targets. > +# > +FROM qemu:debian-sid > + > +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ > + apt-get install -y --no-install-recommends \ > + gcc-alpha-linux-gnu \ > + libc6.1-dev-alpha-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } > diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker > new file mode 100644 > index 0000000000..9a3d168705 > --- /dev/null > +++ b/tests/docker/dockerfiles/debian-sid.docker > @@ -0,0 +1,32 @@ > +# > +# Debian Sid Base > +# > +# A number of our guests exist as ports only. We can either use the > +# ports repo or get everything from Sid. However Sid is a rolling > +# distro which may be broken at any particular time. If you are > +# unlucky and try and build your images while gcc is in the process of > +# being uploaded this can fail. Your only recourse is to try again in > +# a few hours when the repos have re-synced. Once built however you > +# won't be affected by repo changes unless the docker recipies are > +# updated and trigger a re-build. OK, the comment is explaining well. > +# > + > +FROM debian:sid-slim > + > +# Duplicate deb line as deb-src > +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list > + > +# Install common build utilities > +RUN apt update > +RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata > +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ > + apt install -y --no-install-recommends \ > + bison \ > + build-essential \ > + ca-certificates \ > + flex \ > + git \ > + pkg-config \ > + psmisc \ > + python \ > + texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } Output is: I have a very low success rate at building Sid-based images. Recently trying your debian-powerpc-cross: [...] Need to get 21.8 MB of archives. After this operation, 72.0 MB of additional disk space will be used. Get:1 http://deb.debian.org/debian sid/main amd64 binutils-powerpc-linux-gnu amd64 2.30-15 [2411 kB] Err:2 http://deb.debian.org/debian sid/main amd64 gcc-7-powerpc-linux-gnu-base amd64 7.3.0-15cross1 404 Not Found Err:3 http://deb.debian.org/debian sid/main amd64 cpp-7-powerpc-linux-gnu amd64 7.3.0-15cross1 404 Not Found Get:4 http://deb.debian.org/debian sid/main amd64 cpp-powerpc-linux-gnu amd64 4:7.3.0-3 [16.0 kB] Get:5 http://deb.debian.org/debian sid/main amd64 gcc-7-cross-base-ports all 7.3.0-15cross1 [182 kB] Get:6 http://deb.debian.org/debian sid/main amd64 gcc-8-cross-base-ports all 8-20180402-1cross2 [179 kB] Get:7 http://deb.debian.org/debian sid/main amd64 libc6-powerpc-cross all 2.27-3cross2 [1373 kB] [...] E: Failed to fetch http://deb.debian.org/debian/pool/main/g/gcc-7-cross-ports/gcc-7-powerpc-linux-gnu_7.3.0-15cross1_amd64.deb 404 Not Found E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Failed to build - see debian-sid.docker notes The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends gcc-powerpc-linux-gnu libc6-dev-powerpc-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }' returned a non-zero code: 1 Traceback (most recent call last): File "./tests/docker/docker.py", line 456, in <module> sys.exit(main()) File "./tests/docker/docker.py", line 453, in main return args.cmdobj.run(args, argv) File "./tests/docker/docker.py", line 326, in run extra_files_cksum=cksum) File "./tests/docker/docker.py", line 209, in build_image quiet=quiet) File "./tests/docker/docker.py", line 147, in _do_check return subprocess.check_call(self._command + cmd, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'qemu:debian-powerpc-cross', '-f', '/tmp/docker_buildQI1jpf/tmpf8x3A7.docker', '/tmp/docker_buildQI1jpf']' returned non-zero exit status 1 make: *** [tests/docker/Makefile.include:39: docker-image-debian-powerpc-cross] Error 1 ~/source/qemu ((0340bb063f...))$ Due to the ugly output, I didn't notice the "see debian-sid.docker notes" at first. I'm not sure using apt repository snapshot resolve everything. Assuming I'm located in an area where Debian mirrors are very unreliable, and most of QEMU developers are in area with strong connectivity, I won't object more on this patch. My network issues shouldn't be a blocker for the community. I'll see if I can arrange Dockerfiles to optionally use snapshots. So finally, you are welcome to keep my R-b and T-b tags :) Regards, Phil. > diff --git a/tests/tcg/alpha/Makefile.include b/tests/tcg/alpha/Makefile.include > new file mode 100644 > index 0000000000..c7dc48eadb > --- /dev/null > +++ b/tests/tcg/alpha/Makefile.include > @@ -0,0 +1,2 @@ > +DOCKER_IMAGE=debian-alpha-cross > +DOCKER_CROSS_COMPILER=alpha-linux-gnu-gcc > diff --git a/tests/tcg/alpha/Makefile.target b/tests/tcg/alpha/Makefile.target > new file mode 100644 > index 0000000000..bedf1b6444 > --- /dev/null > +++ b/tests/tcg/alpha/Makefile.target > @@ -0,0 +1,6 @@ > +# -*- Mode: makefile -*- > +# > +# Alpha specific tweaks > + > +# On Alpha Linux only supports 8k pages > +EXTRA_RUNS+=run-test-mmap-8192 >
Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > (Cc'ing Samuel who replied in a previous thread) > > Hi Alex, > > On 05/25/2018 11:19 AM, Alex Bennée wrote: >> We can't use our normal Debian based compilers as Alpha isn't an >> officially supported architecture. However it is available as a port >> and fortunately cross compilers for all these targets are included in >> Debian Sid, the perpetual rolling/unstable/testing version of Debian. >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > > These tags are only for the Alpha part, maybe I wasn't clear enough in > my previous review: > https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04443.html > >> --- >> v5 >> - drop the unused debian-cross-build docker image >> - add note to debian-sid docker image >> - echo reference to notes on build failure >> - include EXTRA_RUNS for mmap tests >> --- >> tests/docker/Makefile.include | 1 + >> .../dockerfiles/debian-alpha-cross.docker | 12 +++++++ >> tests/docker/dockerfiles/debian-sid.docker | 32 +++++++++++++++++++ >> tests/tcg/alpha/Makefile.include | 2 ++ >> tests/tcg/alpha/Makefile.target | 6 ++++ >> 5 files changed, 53 insertions(+) >> create mode 100644 tests/docker/dockerfiles/debian-alpha-cross.docker >> create mode 100644 tests/docker/dockerfiles/debian-sid.docker >> create mode 100644 tests/tcg/alpha/Makefile.include >> create mode 100644 tests/tcg/alpha/Makefile.target >> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >> index b56386f49e..5dc49c6a54 100644 >> --- a/tests/docker/Makefile.include >> +++ b/tests/docker/Makefile.include >> @@ -62,6 +62,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >> docker-image-debian-s390x-cross: docker-image-debian9 >> docker-image-debian-win32-cross: docker-image-debian8-mxe >> docker-image-debian-win64-cross: docker-image-debian8-mxe >> +docker-image-debian-alpha-cross: docker-image-debian-sid >> docker-image-travis: NOUSER=1 >> >> # Expand all the pre-requistes for each docker image and test combination >> diff --git a/tests/docker/dockerfiles/debian-alpha-cross.docker b/tests/docker/dockerfiles/debian-alpha-cross.docker >> new file mode 100644 >> index 0000000000..29a25d0dfd >> --- /dev/null >> +++ b/tests/docker/dockerfiles/debian-alpha-cross.docker >> @@ -0,0 +1,12 @@ >> +# >> +# Docker cross-compiler target >> +# >> +# This docker target builds on the debian sid base image which >> +# contains cross compilers for Debian "ports" targets. >> +# >> +FROM qemu:debian-sid >> + >> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ >> + apt-get install -y --no-install-recommends \ >> + gcc-alpha-linux-gnu \ >> + libc6.1-dev-alpha-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } >> diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker >> new file mode 100644 >> index 0000000000..9a3d168705 >> --- /dev/null >> +++ b/tests/docker/dockerfiles/debian-sid.docker >> @@ -0,0 +1,32 @@ >> +# >> +# Debian Sid Base >> +# >> +# A number of our guests exist as ports only. We can either use the >> +# ports repo or get everything from Sid. However Sid is a rolling >> +# distro which may be broken at any particular time. If you are >> +# unlucky and try and build your images while gcc is in the process of >> +# being uploaded this can fail. Your only recourse is to try again in >> +# a few hours when the repos have re-synced. Once built however you >> +# won't be affected by repo changes unless the docker recipies are >> +# updated and trigger a re-build. > > OK, the comment is explaining well. > >> +# >> + >> +FROM debian:sid-slim >> + >> +# Duplicate deb line as deb-src >> +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list >> + >> +# Install common build utilities >> +RUN apt update >> +RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata >> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ >> + apt install -y --no-install-recommends \ >> + bison \ >> + build-essential \ >> + ca-certificates \ >> + flex \ >> + git \ >> + pkg-config \ >> + psmisc \ >> + python \ >> + texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } > > Output is: > > I have a very low success rate at building Sid-based images. > Recently trying your debian-powerpc-cross: > > [...] > Need to get 21.8 MB of archives. > After this operation, 72.0 MB of additional disk space will be used. > Get:1 http://deb.debian.org/debian sid/main amd64 > binutils-powerpc-linux-gnu amd64 2.30-15 [2411 kB] > Err:2 http://deb.debian.org/debian sid/main amd64 > gcc-7-powerpc-linux-gnu-base amd64 7.3.0-15cross1 > 404 Not Found > Err:3 http://deb.debian.org/debian sid/main amd64 > cpp-7-powerpc-linux-gnu amd64 7.3.0-15cross1 > 404 Not Found > Get:4 http://deb.debian.org/debian sid/main amd64 cpp-powerpc-linux-gnu > amd64 4:7.3.0-3 [16.0 kB] > Get:5 http://deb.debian.org/debian sid/main amd64 gcc-7-cross-base-ports > all 7.3.0-15cross1 [182 kB] > Get:6 http://deb.debian.org/debian sid/main amd64 gcc-8-cross-base-ports > all 8-20180402-1cross2 [179 kB] > Get:7 http://deb.debian.org/debian sid/main amd64 libc6-powerpc-cross > all 2.27-3cross2 [1373 kB] > [...] > E: Failed to fetch > http://deb.debian.org/debian/pool/main/g/gcc-7-cross-ports/gcc-7-powerpc-linux-gnu_7.3.0-15cross1_amd64.deb > 404 Not Found > E: Unable to fetch some archives, maybe run apt-get update or try with > --fix-missing? > Failed to build - see debian-sid.docker notes > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive eatmydata > apt-get install -y --no-install-recommends gcc-powerpc-linux-gnu > libc6-dev-powerpc-cross || { echo "Failed to build - see > debian-sid.docker notes"; exit 1; }' returned a non-zero code: 1 > Traceback (most recent call last): > File "./tests/docker/docker.py", line 456, in <module> > sys.exit(main()) > File "./tests/docker/docker.py", line 453, in main > return args.cmdobj.run(args, argv) > File "./tests/docker/docker.py", line 326, in run > extra_files_cksum=cksum) > File "./tests/docker/docker.py", line 209, in build_image > quiet=quiet) > File "./tests/docker/docker.py", line 147, in _do_check > return subprocess.check_call(self._command + cmd, **kwargs) > File "/usr/lib/python2.7/subprocess.py", line 190, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['docker', 'build', '-t', > 'qemu:debian-powerpc-cross', '-f', > '/tmp/docker_buildQI1jpf/tmpf8x3A7.docker', '/tmp/docker_buildQI1jpf']' > returned non-zero exit status 1 > make: *** [tests/docker/Makefile.include:39: > docker-image-debian-powerpc-cross] Error 1 > ~/source/qemu ((0340bb063f...))$ > > Due to the ugly output, I didn't notice the "see debian-sid.docker > notes" at first. What about running with NOCACHE=1, I suspect this is drift between when your debian-sid image was created and the follow-up. Really we could do with just forcing debian-sid to re-build whenever we build a sub-image. > > I'm not sure using apt repository snapshot resolve everything. > > Assuming I'm located in an area where Debian mirrors are very > unreliable, and most of QEMU developers are in area with strong > connectivity, I won't object more on this patch. My network issues > shouldn't be a blocker for the community. > > I'll see if I can arrange Dockerfiles to optionally use snapshots. > > So finally, you are welcome to keep my R-b and T-b tags :) > > Regards, > > Phil. > >> diff --git a/tests/tcg/alpha/Makefile.include b/tests/tcg/alpha/Makefile.include >> new file mode 100644 >> index 0000000000..c7dc48eadb >> --- /dev/null >> +++ b/tests/tcg/alpha/Makefile.include >> @@ -0,0 +1,2 @@ >> +DOCKER_IMAGE=debian-alpha-cross >> +DOCKER_CROSS_COMPILER=alpha-linux-gnu-gcc >> diff --git a/tests/tcg/alpha/Makefile.target b/tests/tcg/alpha/Makefile.target >> new file mode 100644 >> index 0000000000..bedf1b6444 >> --- /dev/null >> +++ b/tests/tcg/alpha/Makefile.target >> @@ -0,0 +1,6 @@ >> +# -*- Mode: makefile -*- >> +# >> +# Alpha specific tweaks >> + >> +# On Alpha Linux only supports 8k pages >> +EXTRA_RUNS+=run-test-mmap-8192 >> -- Alex Bennée
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index b56386f49e..5dc49c6a54 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -62,6 +62,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 docker-image-debian-s390x-cross: docker-image-debian9 docker-image-debian-win32-cross: docker-image-debian8-mxe docker-image-debian-win64-cross: docker-image-debian8-mxe +docker-image-debian-alpha-cross: docker-image-debian-sid docker-image-travis: NOUSER=1 # Expand all the pre-requistes for each docker image and test combination diff --git a/tests/docker/dockerfiles/debian-alpha-cross.docker b/tests/docker/dockerfiles/debian-alpha-cross.docker new file mode 100644 index 0000000000..29a25d0dfd --- /dev/null +++ b/tests/docker/dockerfiles/debian-alpha-cross.docker @@ -0,0 +1,12 @@ +# +# Docker cross-compiler target +# +# This docker target builds on the debian sid base image which +# contains cross compilers for Debian "ports" targets. +# +FROM qemu:debian-sid + +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ + apt-get install -y --no-install-recommends \ + gcc-alpha-linux-gnu \ + libc6.1-dev-alpha-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker new file mode 100644 index 0000000000..9a3d168705 --- /dev/null +++ b/tests/docker/dockerfiles/debian-sid.docker @@ -0,0 +1,32 @@ +# +# Debian Sid Base +# +# A number of our guests exist as ports only. We can either use the +# ports repo or get everything from Sid. However Sid is a rolling +# distro which may be broken at any particular time. If you are +# unlucky and try and build your images while gcc is in the process of +# being uploaded this can fail. Your only recourse is to try again in +# a few hours when the repos have re-synced. Once built however you +# won't be affected by repo changes unless the docker recipies are +# updated and trigger a re-build. +# + +FROM debian:sid-slim + +# Duplicate deb line as deb-src +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list + +# Install common build utilities +RUN apt update +RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ + apt install -y --no-install-recommends \ + bison \ + build-essential \ + ca-certificates \ + flex \ + git \ + pkg-config \ + psmisc \ + python \ + texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; } diff --git a/tests/tcg/alpha/Makefile.include b/tests/tcg/alpha/Makefile.include new file mode 100644 index 0000000000..c7dc48eadb --- /dev/null +++ b/tests/tcg/alpha/Makefile.include @@ -0,0 +1,2 @@ +DOCKER_IMAGE=debian-alpha-cross +DOCKER_CROSS_COMPILER=alpha-linux-gnu-gcc diff --git a/tests/tcg/alpha/Makefile.target b/tests/tcg/alpha/Makefile.target new file mode 100644 index 0000000000..bedf1b6444 --- /dev/null +++ b/tests/tcg/alpha/Makefile.target @@ -0,0 +1,6 @@ +# -*- Mode: makefile -*- +# +# Alpha specific tweaks + +# On Alpha Linux only supports 8k pages +EXTRA_RUNS+=run-test-mmap-8192