diff mbox series

UBUNTU: SAUCE: ubuntu_fan_smoke_test: workaround docker 'dependency' failure

Message ID 20200921161623.GA32698@xps-13
State New
Headers show
Series UBUNTU: SAUCE: ubuntu_fan_smoke_test: workaround docker 'dependency' failure | expand

Commit Message

Andrea Righi Sept. 21, 2020, 4:16 p.m. UTC
BugLink: https://bugs.launchpad.net/bugs/1896511

ubuntu_fan_smoke_test is failing with the following error:

111. 	09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
112. 	09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1)
113. 	09/16 10:14:02 ERROR| test:0414| Exception escaping from test:
114. 	Traceback (most recent call last):
115. 	File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
116. 	_call_test_function(self.execute, *p_args, **p_dargs)
117. 	File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
118. 	return func(*args, **dargs)
119. 	File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
120. 	postprocess_profiled_run, args, dargs)
121. 	File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
122. 	self.run_once(*args, **dargs)
123. 	File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once
124. 	self.results = utils.system_output(cmd, retain_output=True)
125. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
126. 	verbose=verbose, args=args).stdout
127. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
128. 	"Command returned non-zero exit status")
129. 	CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status
130. 	* Command:
131. 	./ubuntu_fan_smoke_test.sh 10.246.0.0/16
132. 	Exit status: 1

And this message is showing up in the log:

systemd: Job docker.service/start failed with result 'dependency'.

Apparently it seems to be a docker issue, as reported here [1]. Fix by
applying the resolutions steps suggested in this article.

[1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine

Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
---
 ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++
 1 file changed, 6 insertions(+)

Comments

Sean Feole Sept. 21, 2020, 4:27 p.m. UTC | #1
+1 LGTM and tested in the clouds.


On 9/21/20 12:17 PM, Andrea Righi wrote:
> Forgot to add [autotest-client-tests] in the subject. Sorry.
>
> -Andrea
>
> On Mon, Sep 21, 2020 at 06:16:25PM +0200, Andrea Righi wrote:
>> BugLink: https://bugs.launchpad.net/bugs/1896511
>>
>> ubuntu_fan_smoke_test is failing with the following error:
>>
>> 111. 	09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
>> 112. 	09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1)
>> 113. 	09/16 10:14:02 ERROR| test:0414| Exception escaping from test:
>> 114. 	Traceback (most recent call last):
>> 115. 	File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
>> 116. 	_call_test_function(self.execute, *p_args, **p_dargs)
>> 117. 	File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
>> 118. 	return func(*args, **dargs)
>> 119. 	File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
>> 120. 	postprocess_profiled_run, args, dargs)
>> 121. 	File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
>> 122. 	self.run_once(*args, **dargs)
>> 123. 	File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once
>> 124. 	self.results = utils.system_output(cmd, retain_output=True)
>> 125. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
>> 126. 	verbose=verbose, args=args).stdout
>> 127. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
>> 128. 	"Command returned non-zero exit status")
>> 129. 	CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status
>> 130. 	* Command:
>> 131. 	./ubuntu_fan_smoke_test.sh 10.246.0.0/16
>> 132. 	Exit status: 1
>>
>> And this message is showing up in the log:
>>
>> systemd: Job docker.service/start failed with result 'dependency'.
>>
>> Apparently it seems to be a docker issue, as reported here [1]. Fix by
>> applying the resolutions steps suggested in this article.
>>
>> [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine
>>
>> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
>> ---
>>   ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
>> index 74aa5d87..095a922f 100755
>> --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
>> +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
>> @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then
>>               ;;
>>       esac
>>   
>> +    mkdir -p /etc/systemd/system/containerd.service.d
>> +    touch /etc/systemd/system/containerd.service.d/override.conf
>> +    echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf
>> +    systemctl daemon-reload
>> +    systemctl restart docker.service
>> +
>>       echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \
>>            > /etc/systemd/system/docker.service.d/http-proxy.conf
>>       systemctl daemon-reload
>> -- 
>> 2.27.0
>>
Stefan Bader Sept. 22, 2020, 8:41 a.m. UTC | #2
On 21.09.20 18:16, Andrea Righi wrote:
> BugLink: https://bugs.launchpad.net/bugs/1896511
> 
> ubuntu_fan_smoke_test is failing with the following error:
> 
> 111. 	09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
> 112. 	09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1)
> 113. 	09/16 10:14:02 ERROR| test:0414| Exception escaping from test:
> 114. 	Traceback (most recent call last):
> 115. 	File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
> 116. 	_call_test_function(self.execute, *p_args, **p_dargs)
> 117. 	File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
> 118. 	return func(*args, **dargs)
> 119. 	File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
> 120. 	postprocess_profiled_run, args, dargs)
> 121. 	File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
> 122. 	self.run_once(*args, **dargs)
> 123. 	File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once
> 124. 	self.results = utils.system_output(cmd, retain_output=True)
> 125. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
> 126. 	verbose=verbose, args=args).stdout
> 127. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
> 128. 	"Command returned non-zero exit status")
> 129. 	CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status
> 130. 	* Command:
> 131. 	./ubuntu_fan_smoke_test.sh 10.246.0.0/16
> 132. 	Exit status: 1
> 
> And this message is showing up in the log:
> 
> systemd: Job docker.service/start failed with result 'dependency'.
> 
> Apparently it seems to be a docker issue, as reported here [1]. Fix by
> applying the resolutions steps suggested in this article.
> 
> [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine
> 
> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
> ---
>  ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> index 74aa5d87..095a922f 100755
> --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then
>              ;;
>      esac
>  
> +    mkdir -p /etc/systemd/system/containerd.service.d
> +    touch /etc/systemd/system/containerd.service.d/override.conf
> +    echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf

That would work until someone already created that config. And with docker that
might happen sooner than later. So I tend to think this should be a little more
bullet-proof. Or did you ensure that adding multiple "Service" stanzas does
work? The touch and append approach seems odd (why not just "echo >".

-Stefan
> +    systemctl daemon-reload
> +    systemctl restart docker.service
> +
>      echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \
>           > /etc/systemd/system/docker.service.d/http-proxy.conf
>      systemctl daemon-reload
>
Andrea Righi Sept. 22, 2020, 8:59 a.m. UTC | #3
On Tue, Sep 22, 2020 at 10:41:04AM +0200, Stefan Bader wrote:
> On 21.09.20 18:16, Andrea Righi wrote:
> > BugLink: https://bugs.launchpad.net/bugs/1896511
> > 
> > ubuntu_fan_smoke_test is failing with the following error:
> > 
> > 111. 	09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
> > 112. 	09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1)
> > 113. 	09/16 10:14:02 ERROR| test:0414| Exception escaping from test:
> > 114. 	Traceback (most recent call last):
> > 115. 	File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
> > 116. 	_call_test_function(self.execute, *p_args, **p_dargs)
> > 117. 	File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
> > 118. 	return func(*args, **dargs)
> > 119. 	File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
> > 120. 	postprocess_profiled_run, args, dargs)
> > 121. 	File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
> > 122. 	self.run_once(*args, **dargs)
> > 123. 	File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once
> > 124. 	self.results = utils.system_output(cmd, retain_output=True)
> > 125. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
> > 126. 	verbose=verbose, args=args).stdout
> > 127. 	File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
> > 128. 	"Command returned non-zero exit status")
> > 129. 	CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status
> > 130. 	* Command:
> > 131. 	./ubuntu_fan_smoke_test.sh 10.246.0.0/16
> > 132. 	Exit status: 1
> > 
> > And this message is showing up in the log:
> > 
> > systemd: Job docker.service/start failed with result 'dependency'.
> > 
> > Apparently it seems to be a docker issue, as reported here [1]. Fix by
> > applying the resolutions steps suggested in this article.
> > 
> > [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine
> > 
> > Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
> > ---
> >  ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> > index 74aa5d87..095a922f 100755
> > --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> > +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
> > @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then
> >              ;;
> >      esac
> >  
> > +    mkdir -p /etc/systemd/system/containerd.service.d
> > +    touch /etc/systemd/system/containerd.service.d/override.conf
> > +    echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf
> 
> That would work until someone already created that config. And with docker that
> might happen sooner than later. So I tend to think this should be a little more
> bullet-proof. Or did you ensure that adding multiple "Service" stanzas does
> work? The touch and append approach seems odd (why not just "echo >".

I agree, a better approach would be to check if this file already exists
and if it already contains that ExecStartPre definition. When docker
will be fixed I think we will not require this workaround anymore and I
assume the fix will be inside docker itself, this one really looks like
a temporary workaround.

But more in general, I think it's weird that we're hitting this problem
now, the original bug in docker is quite old:
https://github.com/docker/for-linux/issues/475

I'll investigate a bit more on this.

-Andrea
diff mbox series

Patch

diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
index 74aa5d87..095a922f 100755
--- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
+++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh
@@ -60,6 +60,12 @@  if [ -n "$http_proxy" ]; then
             ;;
     esac
 
+    mkdir -p /etc/systemd/system/containerd.service.d
+    touch /etc/systemd/system/containerd.service.d/override.conf
+    echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf
+    systemctl daemon-reload
+    systemctl restart docker.service
+
     echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \
          > /etc/systemd/system/docker.service.d/http-proxy.conf
     systemctl daemon-reload