diff mbox

[RFC,v4,3/4] docker: Support "QEMU_CHROOT" in dockerfiles

Message ID 1468207242-5015-4-git-send-email-famz@redhat.com
State New
Headers show

Commit Message

Fam Zheng July 11, 2016, 3:20 a.m. UTC 
This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
indicate that the test execution should be done in a chroot in the
container.

Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
testing scripts.

The SYS_ADMIN is a required capability for mount, add it to the
docker run command line.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include |  1 +
 tests/docker/run              | 12 ++++++++++++
 2 files changed, 13 insertions(+)

Comments

Paolo Bonzini July 11, 2016, 9:20 a.m. UTC | #1 
On 11/07/2016 05:20, Fam Zheng wrote:
> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
> 
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
> 
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
> 
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>  
>  BASE="$(dirname $(realpath $0))"
>  
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp

Can you ask docker to do these bind mounts instead?

Thanks,

Paolo

> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +
>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH
>
Alex Bennée July 11, 2016, 10:08 a.m. UTC | #2 
Paolo Bonzini <pbonzini@redhat.com> writes:

> On 11/07/2016 05:20, Fam Zheng wrote:
>> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
>> indicate that the test execution should be done in a chroot in the
>> container.
>>
>> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
>> testing scripts.
>>
>> The SYS_ADMIN is a required capability for mount, add it to the
>> docker run command line.
>>
>> Signed-off-by: Fam Zheng <famz@redhat.com>
>> ---
>>  tests/docker/Makefile.include |  1 +
>>  tests/docker/run              | 12 ++++++++++++
>>  2 files changed, 13 insertions(+)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index c5546ee..e9821ba 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>>  		$(call quiet-command,\
>>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>>  				-t \
>> +				--cap-add SYS_ADMIN \
>>  				$(if $(DEBUG),-i,--net=none) \
>>  				-e TARGET_LIST=$(TARGET_LIST) \
>>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
>> diff --git a/tests/docker/run b/tests/docker/run
>> index 38ce789..4e80cc3 100755
>> --- a/tests/docker/run
>> +++ b/tests/docker/run
>> @@ -19,6 +19,18 @@ fi
>>
>>  BASE="$(dirname $(realpath $0))"
>>
>> +# cp files into the chroot and execute there
>> +if test -n "$QEMU_CHROOT"; then
>> +    mkdir -p $QEMU_CHROOT/$BASE
>> +    cp $BASE/* $QEMU_CHROOT/$BASE
>> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
>> +    for bp in dev sys proc; do
>> +        mount --bind /$bp $QEMU_CHROOT/$bp
>
> Can you ask docker to do these bind mounts instead?

AFAICT docker's various mount directives are all focused on tasks like mounting data
volumes from the host into the container.

It's a bit of a shame having to do this as the original approach was to
use docker to avoid having fancy bind mounts on my hosts system. We are
now getting to inception levels of nesting here. But the benefit is not
requiring the host having the pre-requisites to bootstrap the system.

That said looking at the debootstrap requirements I've seen instructions
that start with download the deb, ar extract and then run the script by
hand so maybe this is over complicating things?

Is it possible to bootstrap a Fedora rootfs with a similar script?

>
> Thanks,
>
> Paolo
>
>> +    done
>> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
>> +    exit 0
>> +fi
>> +
>>  # Prepare the environment
>>  . /etc/profile || true
>>  export PATH=/usr/lib/ccache:$PATH
>>


--
Alex Bennée
Paolo Bonzini July 11, 2016, 10:37 a.m. UTC | #3 
On 11/07/2016 12:08, Alex Bennée wrote:
> 
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> On 11/07/2016 05:20, Fam Zheng wrote:
>>> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
>>> indicate that the test execution should be done in a chroot in the
>>> container.
>>>
>>> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
>>> testing scripts.
>>>
>>> The SYS_ADMIN is a required capability for mount, add it to the
>>> docker run command line.
>>>
>>> Signed-off-by: Fam Zheng <famz@redhat.com>
>>> ---
>>>  tests/docker/Makefile.include |  1 +
>>>  tests/docker/run              | 12 ++++++++++++
>>>  2 files changed, 13 insertions(+)
>>>
>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>>> index c5546ee..e9821ba 100644
>>> --- a/tests/docker/Makefile.include
>>> +++ b/tests/docker/Makefile.include
>>> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>>>  		$(call quiet-command,\
>>>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>>>  				-t \
>>> +				--cap-add SYS_ADMIN \
>>>  				$(if $(DEBUG),-i,--net=none) \
>>>  				-e TARGET_LIST=$(TARGET_LIST) \
>>>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
>>> diff --git a/tests/docker/run b/tests/docker/run
>>> index 38ce789..4e80cc3 100755
>>> --- a/tests/docker/run
>>> +++ b/tests/docker/run
>>> @@ -19,6 +19,18 @@ fi
>>>
>>>  BASE="$(dirname $(realpath $0))"
>>>
>>> +# cp files into the chroot and execute there
>>> +if test -n "$QEMU_CHROOT"; then
>>> +    mkdir -p $QEMU_CHROOT/$BASE
>>> +    cp $BASE/* $QEMU_CHROOT/$BASE
>>> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
>>> +    for bp in dev sys proc; do
>>> +        mount --bind /$bp $QEMU_CHROOT/$bp
>>
>> Can you ask docker to do these bind mounts instead?
> 
> AFAICT docker's various mount directives are all focused on tasks like mounting data
> volumes from the host into the container.
> 
> It's a bit of a shame having to do this as the original approach was to
> use docker to avoid having fancy bind mounts on my hosts system. We are
> now getting to inception levels of nesting here. But the benefit is not
> requiring the host having the pre-requisites to bootstrap the system.
> 
> That said looking at the debootstrap requirements I've seen instructions
> that start with download the deb, ar extract and then run the script by
> hand so maybe this is over complicating things?
> 
> Is it possible to bootstrap a Fedora rootfs with a similar script?

In theory "yum" is all that you need to bootstrap a Fedora rootfs.

Paolo
Alex Bennée July 11, 2016, 11:31 a.m. UTC | #4 
Fam Zheng <famz@redhat.com> writes:

> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
>
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
>
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>
>  BASE="$(dirname $(realpath $0))"
>
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp

So this misses out /dev/pts which is a subdir of dev which leads to
messages like:

debconf: delaying package configuration, since apt-utils is not
installed
Fetched 103 kB in 0s (133 kB/s)
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such
file or directory)


> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +
>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH


--
Alex Bennée
Alex Bennée July 11, 2016, 12:17 p.m. UTC | #5 
Fam Zheng <famz@redhat.com> writes:

> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
>
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
>
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>
>  BASE="$(dirname $(realpath $0))"
>
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp
> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +

Running:

make docker-test TEST="test-quick" IMAGES="debootstrap-arm" V=1 J=9

Hmm another failure:

Image is up to date.
/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py run  -t --cap-add SYS_ADMIN --net=none -e TARGET_LIST= -e EXTRA_CONFIGURE_OPTS= -e V=1 -e J=9 -e DEBUG= -e CCACHE_DIR=/var/tmp/ccache -v $(realpath docker-src.2016-07-11-13.16.09.23334):/var/tmp/qemu:z,ro -v $HOME/.cache/qemu-docker-ccache:/var/tmp/ccache:z qemu:debootstrap-arm /var/tmp/qemu/run test-clang;
+++ realpath /var/tmp/qemu/run
++ dirname /var/tmp/qemu/run
+ BASE=/var/tmp/qemu
+ test -n /debootstrap-arm
+ mkdir -p /debootstrap-arm//var/tmp/qemu
+ cp /var/tmp/qemu/dtc.tgz /var/tmp/qemu/pixman.tgz /var/tmp/qemu/qemu.tgz /var/tmp/qemu/run /debootstrap-arm//var/tmp/qemu
+ QEMU_CHROOT_SAVE=/debootstrap-arm
+ for bp in dev sys proc
+ mount --bind /dev /debootstrap-arm/dev
mount: mount /dev on /debootstrap-arm/dev failed: Permission denied
/home/alex/lsrc/qemu/qemu.git/tests/docker/Makefile.include:102: recipe for target 'docker-run-test-clang@debootstrap-arm' failed
make: *** [docker-run-test-clang@debootstrap-arm] Error 32
You have new mail in /var/mail/alex

>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH


--
Alex Bennée
diff mbox

Patch

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c5546ee..e9821ba 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -107,6 +107,7 @@  docker-run-%: docker-qemu-src
 		$(call quiet-command,\
 			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
 				-t \
+				--cap-add SYS_ADMIN \
 				$(if $(DEBUG),-i,--net=none) \
 				-e TARGET_LIST=$(TARGET_LIST) \
 				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
diff --git a/tests/docker/run b/tests/docker/run
index 38ce789..4e80cc3 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -19,6 +19,18 @@  fi
 
 BASE="$(dirname $(realpath $0))"
 
+# cp files into the chroot and execute there
+if test -n "$QEMU_CHROOT"; then
+    mkdir -p $QEMU_CHROOT/$BASE
+    cp $BASE/* $QEMU_CHROOT/$BASE
+    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
+    for bp in dev sys proc; do
+        mount --bind /$bp $QEMU_CHROOT/$bp
+    done
+    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
+    exit 0
+fi
+
 # Prepare the environment
 . /etc/profile || true
 export PATH=/usr/lib/ccache:$PATH