Message ID | 20200921161623.GA32698@xps-13 |
---|---|
State | New |
Headers | show |
Series | UBUNTU: SAUCE: ubuntu_fan_smoke_test: workaround docker 'dependency' failure | expand |
+1 LGTM and tested in the clouds. On 9/21/20 12:17 PM, Andrea Righi wrote: > Forgot to add [autotest-client-tests] in the subject. Sorry. > > -Andrea > > On Mon, Sep 21, 2020 at 06:16:25PM +0200, Andrea Righi wrote: >> BugLink: https://bugs.launchpad.net/bugs/1896511 >> >> ubuntu_fan_smoke_test is failing with the following error: >> >> 111. 09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? >> 112. 09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1) >> 113. 09/16 10:14:02 ERROR| test:0414| Exception escaping from test: >> 114. Traceback (most recent call last): >> 115. File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec >> 116. _call_test_function(self.execute, *p_args, **p_dargs) >> 117. File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function >> 118. return func(*args, **dargs) >> 119. File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute >> 120. postprocess_profiled_run, args, dargs) >> 121. File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once >> 122. self.run_once(*args, **dargs) >> 123. File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once >> 124. self.results = utils.system_output(cmd, retain_output=True) >> 125. File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output >> 126. verbose=verbose, args=args).stdout >> 127. File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run >> 128. "Command returned non-zero exit status") >> 129. CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status >> 130. * Command: >> 131. ./ubuntu_fan_smoke_test.sh 10.246.0.0/16 >> 132. Exit status: 1 >> >> And this message is showing up in the log: >> >> systemd: Job docker.service/start failed with result 'dependency'. >> >> Apparently it seems to be a docker issue, as reported here [1]. Fix by >> applying the resolutions steps suggested in this article. >> >> [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine >> >> Signed-off-by: Andrea Righi <andrea.righi@canonical.com> >> --- >> ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh >> index 74aa5d87..095a922f 100755 >> --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh >> +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh >> @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then >> ;; >> esac >> >> + mkdir -p /etc/systemd/system/containerd.service.d >> + touch /etc/systemd/system/containerd.service.d/override.conf >> + echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf >> + systemctl daemon-reload >> + systemctl restart docker.service >> + >> echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \ >> > /etc/systemd/system/docker.service.d/http-proxy.conf >> systemctl daemon-reload >> -- >> 2.27.0 >>
On 21.09.20 18:16, Andrea Righi wrote: > BugLink: https://bugs.launchpad.net/bugs/1896511 > > ubuntu_fan_smoke_test is failing with the following error: > > 111. 09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? > 112. 09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1) > 113. 09/16 10:14:02 ERROR| test:0414| Exception escaping from test: > 114. Traceback (most recent call last): > 115. File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec > 116. _call_test_function(self.execute, *p_args, **p_dargs) > 117. File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function > 118. return func(*args, **dargs) > 119. File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute > 120. postprocess_profiled_run, args, dargs) > 121. File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once > 122. self.run_once(*args, **dargs) > 123. File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once > 124. self.results = utils.system_output(cmd, retain_output=True) > 125. File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output > 126. verbose=verbose, args=args).stdout > 127. File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run > 128. "Command returned non-zero exit status") > 129. CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status > 130. * Command: > 131. ./ubuntu_fan_smoke_test.sh 10.246.0.0/16 > 132. Exit status: 1 > > And this message is showing up in the log: > > systemd: Job docker.service/start failed with result 'dependency'. > > Apparently it seems to be a docker issue, as reported here [1]. Fix by > applying the resolutions steps suggested in this article. > > [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine > > Signed-off-by: Andrea Righi <andrea.righi@canonical.com> > --- > ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > index 74aa5d87..095a922f 100755 > --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then > ;; > esac > > + mkdir -p /etc/systemd/system/containerd.service.d > + touch /etc/systemd/system/containerd.service.d/override.conf > + echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf That would work until someone already created that config. And with docker that might happen sooner than later. So I tend to think this should be a little more bullet-proof. Or did you ensure that adding multiple "Service" stanzas does work? The touch and append approach seems odd (why not just "echo >". -Stefan > + systemctl daemon-reload > + systemctl restart docker.service > + > echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \ > > /etc/systemd/system/docker.service.d/http-proxy.conf > systemctl daemon-reload >
On Tue, Sep 22, 2020 at 10:41:04AM +0200, Stefan Bader wrote: > On 21.09.20 18:16, Andrea Righi wrote: > > BugLink: https://bugs.launchpad.net/bugs/1896511 > > > > ubuntu_fan_smoke_test is failing with the following error: > > > > 111. 09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? > > 112. 09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1) > > 113. 09/16 10:14:02 ERROR| test:0414| Exception escaping from test: > > 114. Traceback (most recent call last): > > 115. File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec > > 116. _call_test_function(self.execute, *p_args, **p_dargs) > > 117. File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function > > 118. return func(*args, **dargs) > > 119. File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute > > 120. postprocess_profiled_run, args, dargs) > > 121. File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once > > 122. self.run_once(*args, **dargs) > > 123. File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once > > 124. self.results = utils.system_output(cmd, retain_output=True) > > 125. File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output > > 126. verbose=verbose, args=args).stdout > > 127. File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run > > 128. "Command returned non-zero exit status") > > 129. CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status > > 130. * Command: > > 131. ./ubuntu_fan_smoke_test.sh 10.246.0.0/16 > > 132. Exit status: 1 > > > > And this message is showing up in the log: > > > > systemd: Job docker.service/start failed with result 'dependency'. > > > > Apparently it seems to be a docker issue, as reported here [1]. Fix by > > applying the resolutions steps suggested in this article. > > > > [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine > > > > Signed-off-by: Andrea Righi <andrea.righi@canonical.com> > > --- > > ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > > index 74aa5d87..095a922f 100755 > > --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > > +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh > > @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then > > ;; > > esac > > > > + mkdir -p /etc/systemd/system/containerd.service.d > > + touch /etc/systemd/system/containerd.service.d/override.conf > > + echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf > > That would work until someone already created that config. And with docker that > might happen sooner than later. So I tend to think this should be a little more > bullet-proof. Or did you ensure that adding multiple "Service" stanzas does > work? The touch and append approach seems odd (why not just "echo >". I agree, a better approach would be to check if this file already exists and if it already contains that ExecStartPre definition. When docker will be fixed I think we will not require this workaround anymore and I assume the fix will be inside docker itself, this one really looks like a temporary workaround. But more in general, I think it's weird that we're hitting this problem now, the original bug in docker is quite old: https://github.com/docker/for-linux/issues/475 I'll investigate a bit more on this. -Andrea
diff --git a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh index 74aa5d87..095a922f 100755 --- a/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh +++ b/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh @@ -60,6 +60,12 @@ if [ -n "$http_proxy" ]; then ;; esac + mkdir -p /etc/systemd/system/containerd.service.d + touch /etc/systemd/system/containerd.service.d/override.conf + echo -e "[Service]\nExecStartPre=" >> /etc/systemd/system/containerd.service.d/override.conf + systemctl daemon-reload + systemctl restart docker.service + echo $EOPTS "[Service]\nEnvironment=\"HTTP_PROXY=$http_proxy\"\n" \ > /etc/systemd/system/docker.service.d/http-proxy.conf systemctl daemon-reload
BugLink: https://bugs.launchpad.net/bugs/1896511 ubuntu_fan_smoke_test is failing with the following error: 111. 09/16 10:14:02 ERROR| utils:0153| [stderr] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 112. 09/16 10:14:02 DEBUG| utils:0153| [stdout] docker pull ubuntu: FAILED (docker pull returned 1) 113. 09/16 10:14:02 ERROR| test:0414| Exception escaping from test: 114. Traceback (most recent call last): 115. File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec 116. _call_test_function(self.execute, *p_args, **p_dargs) 117. File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function 118. return func(*args, **dargs) 119. File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute 120. postprocess_profiled_run, args, dargs) 121. File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once 122. self.run_once(*args, **dargs) 123. File "/home/ubuntu/autotest/client/tests/ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.py", line 53, in run_once 124. self.results = utils.system_output(cmd, retain_output=True) 125. File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output 126. verbose=verbose, args=args).stdout 127. File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run 128. "Command returned non-zero exit status") 129. CmdError: Command <./ubuntu_fan_smoke_test.sh 10.246.0.0/16> failed, rc=1, Command returned non-zero exit status 130. * Command: 131. ./ubuntu_fan_smoke_test.sh 10.246.0.0/16 132. Exit status: 1 And this message is showing up in the log: systemd: Job docker.service/start failed with result 'dependency'. Apparently it seems to be a docker issue, as reported here [1]. Fix by applying the resolutions steps suggested in this article. [1] https://support.plesk.com/hc/en-us/articles/360012448554-Unable-to-start-Docker-service-Dependency-failed-for-Docker-Application-Container-Engine Signed-off-by: Andrea Righi <andrea.righi@canonical.com> --- ubuntu_fan_smoke_test/ubuntu_fan_smoke_test.sh | 6 ++++++ 1 file changed, 6 insertions(+)