| Message ID | 20181009041826.19462-4-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Bootstrap Python venv and acceptance/functional tests | expand |
Hi Cleber, On 09/10/2018 06:18, Cleber Rosa wrote: > This enables the execution of the acceptance tests on Travis. Did you test this? =) > > Because the Travis environment is based on Ubuntu Trusty, it requires > the python3-pip. > > Note: while another supposedely required component on newer versions > (such as on Bionic) split the Python 3 installation further on the > python3-venv package. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > .travis.yml | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/.travis.yml b/.travis.yml > index 95be6ec59f..db1a31ea51 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -36,6 +36,7 @@ addons: > - liburcu-dev > - libusb-1.0-0-dev > - libvte-2.90-dev > + - python3-pip > - sparse > - uuid-dev > - gcovr > @@ -117,6 +118,11 @@ matrix: > - env: CONFIG="--target-list=x86_64-softmmu" > python: > - "3.6" > + # Acceptance (Functional) tests > + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" > + TEST_CMD="make check-acceptance" > + python: > + - "3.6" > # Using newer GCC with sanitizers > - addons: > apt: > Using the following patch: -- >8 -- diff --git a/.travis.yml b/.travis.yml index 95be6ec59f..87e0c9a13f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -118,4 +118,15 @@ matrix: python: - "3.6" + # Acceptance (Functional) tests + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" + TEST_CMD="make check-acceptance" + # sudo rm /usr/local/bin/pip* + python: + - "3.6" + addons: + apt: + packages: + - python3-pip + - python3.4-venv # Using newer GCC with sanitizers - addons: --- I got some improvements until: VENV /home/travis/build/philmd/qemu/tests/venv MKDIR /home/travis/build/philmd/qemu/tests/results PIP /home/travis/build/philmd/qemu/tests/venv-requirements.txt Exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install requirement.install(install_options, global_options, *args, **kwargs) File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install self.move_wheel_files(self.source_dir, root=root) File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in move_wheel_files pycompile=self.pycompile, File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in move_wheel_files clobber(source, lib_dir, True) File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber os.makedirs(destsubdir) File "/usr/lib/python3.4/os.py", line 237, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.4/dist-packages/avocado' See: https://travis-ci.org/philmd/qemu/jobs/439138706
Philippe Mathieu-Daudé <philmd@redhat.com> writes: > Hi Cleber, > > On 09/10/2018 06:18, Cleber Rosa wrote: >> This enables the execution of the acceptance tests on Travis. > > Did you test this? =) > >> >> Because the Travis environment is based on Ubuntu Trusty, it requires >> the python3-pip. >> >> Note: while another supposedely required component on newer versions >> (such as on Bionic) split the Python 3 installation further on the >> python3-venv package. >> >> Signed-off-by: Cleber Rosa <crosa@redhat.com> >> --- >> .travis.yml | 6 ++++++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/.travis.yml b/.travis.yml >> index 95be6ec59f..db1a31ea51 100644 >> --- a/.travis.yml >> +++ b/.travis.yml >> @@ -36,6 +36,7 @@ addons: >> - liburcu-dev >> - libusb-1.0-0-dev >> - libvte-2.90-dev >> + - python3-pip >> - sparse >> - uuid-dev >> - gcovr >> @@ -117,6 +118,11 @@ matrix: >> - env: CONFIG="--target-list=x86_64-softmmu" >> python: >> - "3.6" >> + # Acceptance (Functional) tests >> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >> + TEST_CMD="make check-acceptance" >> + python: >> + - "3.6" >> # Using newer GCC with sanitizers >> - addons: >> apt: >> > > Using the following patch: > > -- >8 -- > diff --git a/.travis.yml b/.travis.yml > index 95be6ec59f..87e0c9a13f 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -118,4 +118,15 @@ matrix: > python: > - "3.6" > + # Acceptance (Functional) tests > + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" > + TEST_CMD="make check-acceptance" > + # sudo rm /usr/local/bin/pip* That snuck in ^ > + python: > + - "3.6" > + addons: > + apt: > + packages: > + - python3-pip > + - python3.4-venv > # Using newer GCC with sanitizers > - addons: > --- > > I got some improvements until: > > VENV /home/travis/build/philmd/qemu/tests/venv > MKDIR /home/travis/build/philmd/qemu/tests/results > PIP /home/travis/build/philmd/qemu/tests/venv-requirements.txt > Exception: > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in > main > status = self.run(options, args) > File "/usr/lib/python3/dist-packages/pip/commands/install.py", line > 283, in run > requirement_set.install(install_options, global_options, > root=options.root_path) > File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install > requirement.install(install_options, global_options, *args, **kwargs) > File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install > self.move_wheel_files(self.source_dir, root=root) > File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in > move_wheel_files > pycompile=self.pycompile, > File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in > move_wheel_files > clobber(source, lib_dir, True) > File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber > os.makedirs(destsubdir) > File "/usr/lib/python3.4/os.py", line 237, in makedirs > mkdir(name, mode) > PermissionError: [Errno 13] Permission denied: > '/usr/local/lib/python3.4/dist-packages/avocado' > > See: https://travis-ci.org/philmd/qemu/jobs/439138706 Don't we need to do a pip install or is avocado already included? -- Alex Bennée
On 09/10/2018 16:15, Alex Bennée wrote: > > Philippe Mathieu-Daudé <philmd@redhat.com> writes: > >> Hi Cleber, >> >> On 09/10/2018 06:18, Cleber Rosa wrote: >>> This enables the execution of the acceptance tests on Travis. >> >> Did you test this? =) >> >>> >>> Because the Travis environment is based on Ubuntu Trusty, it requires >>> the python3-pip. >>> >>> Note: while another supposedely required component on newer versions >>> (such as on Bionic) split the Python 3 installation further on the >>> python3-venv package. >>> >>> Signed-off-by: Cleber Rosa <crosa@redhat.com> >>> --- >>> .travis.yml | 6 ++++++ >>> 1 file changed, 6 insertions(+) >>> >>> diff --git a/.travis.yml b/.travis.yml >>> index 95be6ec59f..db1a31ea51 100644 >>> --- a/.travis.yml >>> +++ b/.travis.yml >>> @@ -36,6 +36,7 @@ addons: >>> - liburcu-dev >>> - libusb-1.0-0-dev >>> - libvte-2.90-dev >>> + - python3-pip >>> - sparse >>> - uuid-dev >>> - gcovr >>> @@ -117,6 +118,11 @@ matrix: >>> - env: CONFIG="--target-list=x86_64-softmmu" >>> python: >>> - "3.6" >>> + # Acceptance (Functional) tests >>> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >>> + TEST_CMD="make check-acceptance" >>> + python: >>> + - "3.6" >>> # Using newer GCC with sanitizers >>> - addons: >>> apt: >>> >> >> Using the following patch: >> >> -- >8 -- >> diff --git a/.travis.yml b/.travis.yml >> index 95be6ec59f..87e0c9a13f 100644 >> --- a/.travis.yml >> +++ b/.travis.yml >> @@ -118,4 +118,15 @@ matrix: >> python: >> - "3.6" >> + # Acceptance (Functional) tests >> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >> + TEST_CMD="make check-acceptance" >> + # sudo rm /usr/local/bin/pip* > > That snuck in ^ =) Travis-ci image comes with this version which is Python2. >> + python: >> + - "3.6" >> + addons: >> + apt: >> + packages: >> + - python3-pip >> + - python3.4-venv >> # Using newer GCC with sanitizers >> - addons: >> --- >> >> I got some improvements until: >> >> VENV /home/travis/build/philmd/qemu/tests/venv >> MKDIR /home/travis/build/philmd/qemu/tests/results >> PIP /home/travis/build/philmd/qemu/tests/venv-requirements.txt >> Exception: >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in >> main >> status = self.run(options, args) >> File "/usr/lib/python3/dist-packages/pip/commands/install.py", line >> 283, in run >> requirement_set.install(install_options, global_options, >> root=options.root_path) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install >> requirement.install(install_options, global_options, *args, **kwargs) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install >> self.move_wheel_files(self.source_dir, root=root) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in >> move_wheel_files >> pycompile=self.pycompile, >> File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in >> move_wheel_files >> clobber(source, lib_dir, True) >> File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber >> os.makedirs(destsubdir) >> File "/usr/lib/python3.4/os.py", line 237, in makedirs >> mkdir(name, mode) >> PermissionError: [Errno 13] Permission denied: >> '/usr/local/lib/python3.4/dist-packages/avocado' >> >> See: https://travis-ci.org/philmd/qemu/jobs/439138706 > > Don't we need to do a pip install or is avocado already included? Yes I think we do. This is why I asked Cleber on patch #1 of this series why he choose to use "venv ... --system-site-packages".
On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote: > Hi Cleber, > > On 09/10/2018 06:18, Cleber Rosa wrote: >> This enables the execution of the acceptance tests on Travis. > > Did you test this? =) > I did have some jobs on Travis that looked promising. But yeah, later I found some issues. :/ >> >> Because the Travis environment is based on Ubuntu Trusty, it requires >> the python3-pip. >> >> Note: while another supposedely required component on newer versions >> (such as on Bionic) split the Python 3 installation further on the >> python3-venv package. >> >> Signed-off-by: Cleber Rosa <crosa@redhat.com> >> --- >> .travis.yml | 6 ++++++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/.travis.yml b/.travis.yml >> index 95be6ec59f..db1a31ea51 100644 >> --- a/.travis.yml >> +++ b/.travis.yml >> @@ -36,6 +36,7 @@ addons: >> - liburcu-dev >> - libusb-1.0-0-dev >> - libvte-2.90-dev >> + - python3-pip >> - sparse >> - uuid-dev >> - gcovr >> @@ -117,6 +118,11 @@ matrix: >> - env: CONFIG="--target-list=x86_64-softmmu" >> python: >> - "3.6" >> + # Acceptance (Functional) tests >> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >> + TEST_CMD="make check-acceptance" >> + python: >> + - "3.6" >> # Using newer GCC with sanitizers >> - addons: >> apt: >> > > Using the following patch: > > -- >8 -- > diff --git a/.travis.yml b/.travis.yml > index 95be6ec59f..87e0c9a13f 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -118,4 +118,15 @@ matrix: > python: > - "3.6" > + # Acceptance (Functional) tests > + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" > + TEST_CMD="make check-acceptance" > + # sudo rm /usr/local/bin/pip* > + python: > + - "3.6" > + addons: > + apt: > + packages: > + - python3-pip > + - python3.4-venv This patch revealed to me that, even though we're asking Travis for a Python 3.6 stack, we're manually pointing to the primary Python (3.4) installation. This is in itself a source of problems. Read on... > # Using newer GCC with sanitizers > - addons: > --- > > I got some improvements until: > > VENV /home/travis/build/philmd/qemu/tests/venv > MKDIR /home/travis/build/philmd/qemu/tests/results > PIP /home/travis/build/philmd/qemu/tests/venv-requirements.txt > Exception: > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in > main > status = self.run(options, args) > File "/usr/lib/python3/dist-packages/pip/commands/install.py", line > 283, in run > requirement_set.install(install_options, global_options, > root=options.root_path) > File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install > requirement.install(install_options, global_options, *args, **kwargs) > File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install > self.move_wheel_files(self.source_dir, root=root) > File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in > move_wheel_files > pycompile=self.pycompile, > File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in > move_wheel_files > clobber(source, lib_dir, True) > File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber > os.makedirs(destsubdir) > File "/usr/lib/python3.4/os.py", line 237, in makedirs > mkdir(name, mode) > PermissionError: [Errno 13] Permission denied: > '/usr/local/lib/python3.4/dist-packages/avocado' > This looks to me like it's caused by the use of the system wide Python 3.4 installation, as mentioned above. That, and possibly the "--system-wide-packages" flag to the venv. So, in order to understand if, in those environments, we should let "configure" pick up the right Python, I did the following experiment: $ python3 -m venv /tmp/py3-build $ . /tmp/py3-build/bin/activate $ which python /tmp/py3-build/bin/python $ /tmp/py3-build/bin/python --version Python 3.6.6 This shows that when a venv is activated, the "right" Python should be available in the $PATH. So, building QEMU without pointint to a specific Python binary (when inside a venv) gives you: $ mkdir -p /tmp/qemu-build $ cd /tmp/qemu-build $ ~/src/qemu/configure --target-list="x86_64-softmmu" $ make print-PYTHON PYTHON=python -B $ which python /tmp/py3-build/bin/python The whole question now is how/where Travis puts the requested Python version (python: - "3.6"). Is it on a venv? Is it on a specific location? Is that reliable enough? I'm going to look for those answers now. - Cleber. > See: https://travis-ci.org/philmd/qemu/jobs/439138706 >
Cleber Rosa <crosa@redhat.com> writes: > On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote: >> Hi Cleber, >> >> On 09/10/2018 06:18, Cleber Rosa wrote: >>> This enables the execution of the acceptance tests on Travis. >> >> Did you test this? =) >> > > I did have some jobs on Travis that looked promising. But yeah, later I > found some issues. :/ > >>> >>> Because the Travis environment is based on Ubuntu Trusty, it requires >>> the python3-pip. >>> >>> Note: while another supposedely required component on newer versions >>> (such as on Bionic) split the Python 3 installation further on the >>> python3-venv package. >>> >>> Signed-off-by: Cleber Rosa <crosa@redhat.com> >>> --- >>> .travis.yml | 6 ++++++ >>> 1 file changed, 6 insertions(+) >>> >>> diff --git a/.travis.yml b/.travis.yml >>> index 95be6ec59f..db1a31ea51 100644 >>> --- a/.travis.yml >>> +++ b/.travis.yml >>> @@ -36,6 +36,7 @@ addons: >>> - liburcu-dev >>> - libusb-1.0-0-dev >>> - libvte-2.90-dev >>> + - python3-pip >>> - sparse >>> - uuid-dev >>> - gcovr >>> @@ -117,6 +118,11 @@ matrix: >>> - env: CONFIG="--target-list=x86_64-softmmu" >>> python: >>> - "3.6" >>> + # Acceptance (Functional) tests >>> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >>> + TEST_CMD="make check-acceptance" >>> + python: >>> + - "3.6" >>> # Using newer GCC with sanitizers >>> - addons: >>> apt: >>> >> >> Using the following patch: >> >> -- >8 -- >> diff --git a/.travis.yml b/.travis.yml >> index 95be6ec59f..87e0c9a13f 100644 >> --- a/.travis.yml >> +++ b/.travis.yml >> @@ -118,4 +118,15 @@ matrix: >> python: >> - "3.6" >> + # Acceptance (Functional) tests >> + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" >> + TEST_CMD="make check-acceptance" >> + # sudo rm /usr/local/bin/pip* >> + python: >> + - "3.6" >> + addons: >> + apt: >> + packages: >> + - python3-pip >> + - python3.4-venv > > This patch revealed to me that, even though we're asking Travis for a > Python 3.6 stack, we're manually pointing to the primary Python (3.4) > installation. This is in itself a source of problems. > > Read on... > >> # Using newer GCC with sanitizers >> - addons: >> --- >> >> I got some improvements until: >> >> VENV /home/travis/build/philmd/qemu/tests/venv >> MKDIR /home/travis/build/philmd/qemu/tests/results >> PIP /home/travis/build/philmd/qemu/tests/venv-requirements.txt >> Exception: >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in >> main >> status = self.run(options, args) >> File "/usr/lib/python3/dist-packages/pip/commands/install.py", line >> 283, in run >> requirement_set.install(install_options, global_options, >> root=options.root_path) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install >> requirement.install(install_options, global_options, *args, **kwargs) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install >> self.move_wheel_files(self.source_dir, root=root) >> File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in >> move_wheel_files >> pycompile=self.pycompile, >> File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in >> move_wheel_files >> clobber(source, lib_dir, True) >> File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber >> os.makedirs(destsubdir) >> File "/usr/lib/python3.4/os.py", line 237, in makedirs >> mkdir(name, mode) >> PermissionError: [Errno 13] Permission denied: >> '/usr/local/lib/python3.4/dist-packages/avocado' >> > > This looks to me like it's caused by the use of the system wide Python > 3.4 installation, as mentioned above. That, and possibly the > "--system-wide-packages" flag to the venv. > > So, in order to understand if, in those environments, we should let > "configure" pick up the right Python, I did the following experiment: > > $ python3 -m venv /tmp/py3-build > $ . /tmp/py3-build/bin/activate > $ which python > /tmp/py3-build/bin/python > $ /tmp/py3-build/bin/python --version > Python 3.6.6 > > This shows that when a venv is activated, the "right" Python should be > available in the $PATH. So, building QEMU without pointint to a > specific Python binary (when inside a venv) gives you: > > $ mkdir -p /tmp/qemu-build > $ cd /tmp/qemu-build > $ ~/src/qemu/configure --target-list="x86_64-softmmu" > $ make print-PYTHON > PYTHON=python -B > $ which python > /tmp/py3-build/bin/python > > The whole question now is how/where Travis puts the requested Python > version (python: - "3.6"). Is it on a venv? Is it on a specific > location? Is that reliable enough? > > I'm going to look for those answers now. You can experiment with the Travis environment on your own system using: make docker-test-build@travis DEBUG=1 > > - Cleber. > >> See: https://travis-ci.org/philmd/qemu/jobs/439138706 >> -- Alex Bennée
diff --git a/.travis.yml b/.travis.yml index 95be6ec59f..db1a31ea51 100644 --- a/.travis.yml +++ b/.travis.yml @@ -36,6 +36,7 @@ addons: - liburcu-dev - libusb-1.0-0-dev - libvte-2.90-dev + - python3-pip - sparse - uuid-dev - gcovr @@ -117,6 +118,11 @@ matrix: - env: CONFIG="--target-list=x86_64-softmmu" python: - "3.6" + # Acceptance (Functional) tests + - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" + TEST_CMD="make check-acceptance" + python: + - "3.6" # Using newer GCC with sanitizers - addons: apt:
This enables the execution of the acceptance tests on Travis. Because the Travis environment is based on Ubuntu Trusty, it requires the python3-pip. Note: while another supposedely required component on newer versions (such as on Bionic) split the Python 3 installation further on the python3-venv package. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- .travis.yml | 6 ++++++ 1 file changed, 6 insertions(+)