diff mbox series

[1/1] Dockerfile: compile QEMU for Nokia N900 emulation

Message ID 20200713171046.230013-1-xypron.glpk@gmx.de
State New
Delegated to: Tom Rini
Headers show
Series [1/1] Dockerfile: compile QEMU for Nokia N900 emulation | expand

Commit Message

Heinrich Schuchardt July 13, 2020, 5:10 p.m. UTC
For the Nokia N900 emulation we need a special QEMU version. Up to now we
compile it in the Gitlab runner. By compiling it in Docker we can reduce
the Gitlab job duration.

To avoid naming conflicts the QEMU binary is stored as
/opt/qemu/bin/qemu-system-n900.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 Dockerfile | 10 ++++++++++
 1 file changed, 10 insertions(+)

--
2.27.0

Comments

Tom Rini July 13, 2020, 5:20 p.m. UTC | #1
On Mon, Jul 13, 2020 at 07:10:46PM +0200, Heinrich Schuchardt wrote:

> For the Nokia N900 emulation we need a special QEMU version. Up to now we
> compile it in the Gitlab runner. By compiling it in Docker we can reduce
> the Gitlab job duration.
> 
> To avoid naming conflicts the QEMU binary is stored as
> /opt/qemu/bin/qemu-system-n900.
> 
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  Dockerfile | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/Dockerfile b/Dockerfile
> index 209e008..bc3cdee 100644
> --- a/Dockerfile
> +++ b/Dockerfile
> @@ -171,6 +171,16 @@ RUN git clone git://git.qemu.org/qemu.git /tmp/qemu && \
>  	make -j$(nproc) all install && \
>  	rm -rf /tmp/qemu
> 
> +# Compile QEMU for Nokia N900 emulation
> +# Last working commit is 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1
> +RUN git clone https://git.linaro.org/qemu/qemu-linaro.git /tmp/qemu && \
> +	cd /tmp/qemu && \
> +	git checkout 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1 && \
> +	./configure --enable-system --target-list=arm-softmmu --disable-sdl --disable-gtk --disable-curses --audio-drv-list= --audio-card-list= --disable-werror --disable-xen --disable-xen-pci-passthrough --disable-brlapi --disable-vnc --disable-curl --disable-slirp --disable-kvm --disable-user --disable-linux-user --disable-bsd-user --disable-guest-base --disable-uuid --disable-vde --disable-linux-aio --disable-cap-ng --disable-attr --disable-blobs --disable-docs --disable-spice --disable-libiscsi --disable-smartcard-nss --disable-usb-redir --disable-guest-agent --disable-seccomp --disable-glusterfs --disable-nptl --disable-fdt && \
> +	make -j$(nproc) && \
> +	cp arm-softmmu/qemu-system-arm /opt/qemu/bin/qemu-system-n900 && \
> +	rm -rf /tmp/qemu
> +
>  # Create our user/group
>  RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
>  RUN useradd -m -U uboot

Azure and GitLab will pick this up, if we modify the job to link the
binary in.  Travis will not, of course.  But this feels like a bit of a
micro-optimization.  The test job is 3-4 minutes currently.  If we want
to reduce that, I think switching the test script to use $(nproc) rather
than 4 for all make jobs would be a bit better.  Keeping all 3 CI
scripts as close as possible is a goal we have to have I believe to
reduce the likelyhood of one CI failing when another passes and it being
the CI at fault rather than exposing a race condition in our code.
Heinrich Schuchardt July 13, 2020, 9:27 p.m. UTC | #2
On 13.07.20 19:20, Tom Rini wrote:
> On Mon, Jul 13, 2020 at 07:10:46PM +0200, Heinrich Schuchardt wrote:
>
>> For the Nokia N900 emulation we need a special QEMU version. Up to now we
>> compile it in the Gitlab runner. By compiling it in Docker we can reduce
>> the Gitlab job duration.
>>
>> To avoid naming conflicts the QEMU binary is stored as
>> /opt/qemu/bin/qemu-system-n900.
>>
>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
>> ---
>>  Dockerfile | 10 ++++++++++
>>  1 file changed, 10 insertions(+)
>>
>> diff --git a/Dockerfile b/Dockerfile
>> index 209e008..bc3cdee 100644
>> --- a/Dockerfile
>> +++ b/Dockerfile
>> @@ -171,6 +171,16 @@ RUN git clone git://git.qemu.org/qemu.git /tmp/qemu && \
>>  	make -j$(nproc) all install && \
>>  	rm -rf /tmp/qemu
>>
>> +# Compile QEMU for Nokia N900 emulation
>> +# Last working commit is 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1
>> +RUN git clone https://git.linaro.org/qemu/qemu-linaro.git /tmp/qemu && \
>> +	cd /tmp/qemu && \
>> +	git checkout 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1 && \
>> +	./configure --enable-system --target-list=arm-softmmu --disable-sdl --disable-gtk --disable-curses --audio-drv-list= --audio-card-list= --disable-werror --disable-xen --disable-xen-pci-passthrough --disable-brlapi --disable-vnc --disable-curl --disable-slirp --disable-kvm --disable-user --disable-linux-user --disable-bsd-user --disable-guest-base --disable-uuid --disable-vde --disable-linux-aio --disable-cap-ng --disable-attr --disable-blobs --disable-docs --disable-spice --disable-libiscsi --disable-smartcard-nss --disable-usb-redir --disable-guest-agent --disable-seccomp --disable-glusterfs --disable-nptl --disable-fdt && \
>> +	make -j$(nproc) && \
>> +	cp arm-softmmu/qemu-system-arm /opt/qemu/bin/qemu-system-n900 && \
>> +	rm -rf /tmp/qemu
>> +
>>  # Create our user/group
>>  RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
>>  RUN useradd -m -U uboot
>
> Azure and GitLab will pick this up, if we modify the job to link the
> binary in.  Travis will not, of course.  But this feels like a bit of a
> micro-optimization.  The test job is 3-4 minutes currently.  If we want

Yes, time I am waiting for the tests that interest me. And sometimes it
fails because the source repository is not available.

> to reduce that, I think switching the test script to use $(nproc) rather
> than 4 for all make jobs would be a bit better.  Keeping all 3 CI

This depends on the number of cores you have on Travis CI. Is this more
than one?

> scripts as close as possible is a goal we have to have I believe to
> reduce the likelyhood of one CI failing when another passes and it being
> the CI at fault rather than exposing a race condition in our code.
>

https://docs.travis-ci.com/user/docker/a
says that you can use Docker images in Travis CI too.

https://docs.travis-ci.com/user/build-stages/share-docker-image/
shows how to trigger tests inside a Docker image.

I think the Docker route is the only way to minimize differences between
the build systems.

Best regards

Heinrich
Tom Rini July 13, 2020, 10:56 p.m. UTC | #3
On Mon, Jul 13, 2020 at 11:27:40PM +0200, Heinrich Schuchardt wrote:
> On 13.07.20 19:20, Tom Rini wrote:
> > On Mon, Jul 13, 2020 at 07:10:46PM +0200, Heinrich Schuchardt wrote:
> >
> >> For the Nokia N900 emulation we need a special QEMU version. Up to now we
> >> compile it in the Gitlab runner. By compiling it in Docker we can reduce
> >> the Gitlab job duration.
> >>
> >> To avoid naming conflicts the QEMU binary is stored as
> >> /opt/qemu/bin/qemu-system-n900.
> >>
> >> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> >> ---
> >>  Dockerfile | 10 ++++++++++
> >>  1 file changed, 10 insertions(+)
> >>
> >> diff --git a/Dockerfile b/Dockerfile
> >> index 209e008..bc3cdee 100644
> >> --- a/Dockerfile
> >> +++ b/Dockerfile
> >> @@ -171,6 +171,16 @@ RUN git clone git://git.qemu.org/qemu.git /tmp/qemu && \
> >>  	make -j$(nproc) all install && \
> >>  	rm -rf /tmp/qemu
> >>
> >> +# Compile QEMU for Nokia N900 emulation
> >> +# Last working commit is 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1
> >> +RUN git clone https://git.linaro.org/qemu/qemu-linaro.git /tmp/qemu && \
> >> +	cd /tmp/qemu && \
> >> +	git checkout 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1 && \
> >> +	./configure --enable-system --target-list=arm-softmmu --disable-sdl --disable-gtk --disable-curses --audio-drv-list= --audio-card-list= --disable-werror --disable-xen --disable-xen-pci-passthrough --disable-brlapi --disable-vnc --disable-curl --disable-slirp --disable-kvm --disable-user --disable-linux-user --disable-bsd-user --disable-guest-base --disable-uuid --disable-vde --disable-linux-aio --disable-cap-ng --disable-attr --disable-blobs --disable-docs --disable-spice --disable-libiscsi --disable-smartcard-nss --disable-usb-redir --disable-guest-agent --disable-seccomp --disable-glusterfs --disable-nptl --disable-fdt && \
> >> +	make -j$(nproc) && \
> >> +	cp arm-softmmu/qemu-system-arm /opt/qemu/bin/qemu-system-n900 && \
> >> +	rm -rf /tmp/qemu
> >> +
> >>  # Create our user/group
> >>  RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
> >>  RUN useradd -m -U uboot
> >
> > Azure and GitLab will pick this up, if we modify the job to link the
> > binary in.  Travis will not, of course.  But this feels like a bit of a
> > micro-optimization.  The test job is 3-4 minutes currently.  If we want
> 
> Yes, time I am waiting for the tests that interest me. And sometimes it
> fails because the source repository is not available.

Right.  But I assume you're doing unit testing locally and not via CI.
I would be quite open to moving this test to the rest of the QEMU based
group, in GitLab as that might result in the overall run completing
quicker.

> > to reduce that, I think switching the test script to use $(nproc) rather
> > than 4 for all make jobs would be a bit better.  Keeping all 3 CI
> 
> This depends on the number of cores you have on Travis CI. Is this more
> than one?

I think so, yes.

> > scripts as close as possible is a goal we have to have I believe to
> > reduce the likelyhood of one CI failing when another passes and it being
> > the CI at fault rather than exposing a race condition in our code.
> >
> 
> https://docs.travis-ci.com/user/docker/a
> says that you can use Docker images in Travis CI too.
> 
> https://docs.travis-ci.com/user/build-stages/share-docker-image/
> shows how to trigger tests inside a Docker image.
> 
> I think the Docker route is the only way to minimize differences between
> the build systems.

I was under the impression that it would result in a fetch of the Docker
image every time and thus probably take much longer overall and possibly
require further splitting up of the Travis job.

Honestly and off-topic for this particular thread, at this point I'd
almost rather just drop Travis support and as much as I like the control
we have in GitLab the fact that every runner has occasional bouts of
random bad luck (DNS failures or jobs get stuck) means I wouldn't get
overly upset if we just used Azure as it's still quicker than Travis,
only a little slower than GitLab when it's fast and can be triggered by
anyone.  It would however require custodians to setup yet another
account somewhere and that is a fairly big drawback (as I do expect
custodians to CI their tree before sending a PR and in turn that
blocking on the general Azure queue would be a big bottleneck I fear).
diff mbox series

Patch

diff --git a/Dockerfile b/Dockerfile
index 209e008..bc3cdee 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -171,6 +171,16 @@  RUN git clone git://git.qemu.org/qemu.git /tmp/qemu && \
 	make -j$(nproc) all install && \
 	rm -rf /tmp/qemu

+# Compile QEMU for Nokia N900 emulation
+# Last working commit is 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1
+RUN git clone https://git.linaro.org/qemu/qemu-linaro.git /tmp/qemu && \
+	cd /tmp/qemu && \
+	git checkout 8f8d8e0796efe1a6f34cdd83fb798f3c41217ec1 && \
+	./configure --enable-system --target-list=arm-softmmu --disable-sdl --disable-gtk --disable-curses --audio-drv-list= --audio-card-list= --disable-werror --disable-xen --disable-xen-pci-passthrough --disable-brlapi --disable-vnc --disable-curl --disable-slirp --disable-kvm --disable-user --disable-linux-user --disable-bsd-user --disable-guest-base --disable-uuid --disable-vde --disable-linux-aio --disable-cap-ng --disable-attr --disable-blobs --disable-docs --disable-spice --disable-libiscsi --disable-smartcard-nss --disable-usb-redir --disable-guest-agent --disable-seccomp --disable-glusterfs --disable-nptl --disable-fdt && \
+	make -j$(nproc) && \
+	cp arm-softmmu/qemu-system-arm /opt/qemu/bin/qemu-system-n900 && \
+	rm -rf /tmp/qemu
+
 # Create our user/group
 RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
 RUN useradd -m -U uboot