| Message ID | 20191023032010.27725-7-trini@konsulko.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Tom Rini |
| Headers | show |
| Series | Moving test/py to Python 3 | expand |
On 10/22/19 9:20 PM, Tom Rini wrote: > - Modern pytest is more visible in telling us about parameters that we > had not described, so describe a few more. > - ConfigParser.readfp(...) is now configparser.read_file(...) > - As part of the "strings vs bytes" conversions in Python 3, we use the > default encoding/decoding of utf-8 but in some places tell Python to > replace problematic conversions rather than throw a fatal error. > - Fix a typo noticed while doing the above ("tot he" -> "to the"). > - As suggested by Stephen, re-alphabetize the import list > - Per Heinrich, replace how we write contents in test_fit.py > diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py > @@ -133,7 +134,7 @@ class RunAndLog(object): > self.logfile.write(self, msg) > > try: > - p = subprocess.Popen(cmd, cwd=cwd, > + p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8", The encoding parameter was added in Python 3.6, so this change causes test.py to crash for me (with Python 3.5.2). The following will fix that: > p = subprocess.Popen(cmd, cwd=cwd, > stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) > (stdout, stderr) = p.communicate() > if stdout is not None: > stdout = stdout.decode('utf-8') > if stderr is not None: > stderr = stderr.decode('utf-8') I also noticed that a variety of strings in these patches use "" not '' in at least this patch. When I first wrote test/py, Simon made me go through everything and replace "" with '', so we should probably make new code follow the convention of using ''.
On Wed, Oct 23, 2019 at 12:50:12PM -0600, Stephen Warren wrote: > On 10/22/19 9:20 PM, Tom Rini wrote: > > - Modern pytest is more visible in telling us about parameters that we > > had not described, so describe a few more. > > - ConfigParser.readfp(...) is now configparser.read_file(...) > > - As part of the "strings vs bytes" conversions in Python 3, we use the > > default encoding/decoding of utf-8 but in some places tell Python to > > replace problematic conversions rather than throw a fatal error. > > - Fix a typo noticed while doing the above ("tot he" -> "to the"). > > - As suggested by Stephen, re-alphabetize the import list > > - Per Heinrich, replace how we write contents in test_fit.py > > > diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py > > > @@ -133,7 +134,7 @@ class RunAndLog(object): > > self.logfile.write(self, msg) > > try: > > - p = subprocess.Popen(cmd, cwd=cwd, > > + p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8", > > The encoding parameter was added in Python 3.6, so this change causes > test.py to crash for me (with Python 3.5.2). The following will fix that: Right, Ubuntu 16.04 has Python 3.5.x and later has 3.6 or higher. It's reasonable to ask what the minimum host version is going to be and how much overhead things get the lower we go. Given: > > p = subprocess.Popen(cmd, cwd=cwd, > > stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) > > (stdout, stderr) = p.communicate() > > if stdout is not None: > > stdout = stdout.decode('utf-8') > > if stderr is not None: > > stderr = stderr.decode('utf-8') That's not too bad, so we can go with that. Can you please chime in on the "Minimum Python 3 version?" thread with what I assume is your use-case of needing to use older-than-latest-LTS distributions? > I also noticed that a variety of strings in these patches use "" not '' in > at least this patch. When I first wrote test/py, Simon made me go through > everything and replace "" with '', so we should probably make new code > follow the convention of using ''. OK, I'll get on top of that, thanks.
On 10/23/19 1:01 PM, Tom Rini wrote: > On Wed, Oct 23, 2019 at 12:50:12PM -0600, Stephen Warren wrote: >> On 10/22/19 9:20 PM, Tom Rini wrote: >>> - Modern pytest is more visible in telling us about parameters that we >>> had not described, so describe a few more. >>> - ConfigParser.readfp(...) is now configparser.read_file(...) >>> - As part of the "strings vs bytes" conversions in Python 3, we use the >>> default encoding/decoding of utf-8 but in some places tell Python to >>> replace problematic conversions rather than throw a fatal error. >>> - Fix a typo noticed while doing the above ("tot he" -> "to the"). >>> - As suggested by Stephen, re-alphabetize the import list >>> - Per Heinrich, replace how we write contents in test_fit.py >> >>> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py >> >>> @@ -133,7 +134,7 @@ class RunAndLog(object): >>> self.logfile.write(self, msg) >>> try: >>> - p = subprocess.Popen(cmd, cwd=cwd, >>> + p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8", >> >> The encoding parameter was added in Python 3.6, so this change causes >> test.py to crash for me (with Python 3.5.2). The following will fix that: > > Right, Ubuntu 16.04 has Python 3.5.x and later has 3.6 or higher. It's > reasonable to ask what the minimum host version is going to be and how > much overhead things get the lower we go. Given: > >>> p = subprocess.Popen(cmd, cwd=cwd, >>> stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) >>> (stdout, stderr) = p.communicate() >>> if stdout is not None: >>> stdout = stdout.decode('utf-8') >>> if stderr is not None: >>> stderr = stderr.decode('utf-8') > > That's not too bad, so we can go with that. Can you please chime in on > the "Minimum Python 3 version?" thread with what I assume is your > use-case of needing to use older-than-latest-LTS distributions? A very quick survey of distros that I can quickly find package lists for: RHEL 6 - 2 only RHEL 7 - 3.6 Ubuntu 16.04 - 3.5 Ubuntu 18.04 - 3.6 Debian oldtsable - 3.5 Debian stable and all later - 3.7 So, Ubuntu 16.04 is perhaps the only distro that still contains Python 3.5. That said, 16.04 is still fully supported for 18 more months, so it feels quite reasonable to support it. I have no choice but to use it for now.
On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote: > > - Modern pytest is more visible in telling us about parameters that we > had not described, so describe a few more. > - ConfigParser.readfp(...) is now configparser.read_file(...) > - As part of the "strings vs bytes" conversions in Python 3, we use the > default encoding/decoding of utf-8 but in some places tell Python to > replace problematic conversions rather than throw a fatal error. > - Fix a typo noticed while doing the above ("tot he" -> "to the"). > - As suggested by Stephen, re-alphabetize the import list > - Per Heinrich, replace how we write contents in test_fit.py > > Signed-off-by: Tom Rini <trini@konsulko.com> > --- > test/py/conftest.py | 9 ++++----- > test/py/multiplexed_log.py | 9 +++++---- > test/py/pytest.ini | 3 +++ > test/py/tests/test_fit.py | 4 ++-- > test/py/u_boot_spawn.py | 4 ++-- > 5 files changed, 16 insertions(+), 13 deletions(-) Reviewed-by: Simon Glass <sjg@chromium.org> Tested on sandbox: Tested-by: Simon Glass <sjg@chromium.org> (I agree about supporting the older Python 3, BTW)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 5c19af1d5034..bffee6b8a3a2 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -13,17 +13,16 @@ # - Implementing custom pytest markers. import atexit +import configparser import errno +import io import os import os.path import pytest -from _pytest.runner import runtestprotocol import re -import io +from _pytest.runner import runtestprotocol import sys -import configparser - # Globals: The HTML log file, and the connection to the U-Boot console. log = None console = None @@ -168,7 +167,7 @@ def pytest_configure(config): ini_str = '[root]\n' + f.read() ini_sio = io.StringIO(ini_str) parser = configparser.RawConfigParser() - parser.readfp(ini_sio) + parser.read_file(ini_sio) ubconfig.buildconfig.update(parser.items('root')) ubconfig.test_py_dir = test_py_dir diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py index de0aacc659b8..5ea4c055e072 100644 --- a/test/py/multiplexed_log.py +++ b/test/py/multiplexed_log.py @@ -51,7 +51,7 @@ class LogfileStream(object): """Write data to the log stream. Args: - data: The data to write tot he file. + data: The data to write to the file. implicit: Boolean indicating whether data actually appeared in the stream, or was implicitly generated. A valid use-case is to repeat a shell prompt at the start of each separate log @@ -64,7 +64,8 @@ class LogfileStream(object): self.logfile.write(self, data, implicit) if self.chained_file: - self.chained_file.write(data) + # Chained file is console, convert things a little + self.chained_file.write((data.encode("ascii", "replace")).decode()) def flush(self): """Flush the log stream, to ensure correct log interleaving. @@ -133,7 +134,7 @@ class RunAndLog(object): self.logfile.write(self, msg) try: - p = subprocess.Popen(cmd, cwd=cwd, + p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8", stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) (stdout, stderr) = p.communicate() output = '' @@ -215,7 +216,7 @@ class Logfile(object): Nothing. """ - self.f = open(fn, 'wt') + self.f = open(fn, 'wt', encoding="utf-8") self.last_stream = None self.blocks = [] self.cur_evt = 1 diff --git a/test/py/pytest.ini b/test/py/pytest.ini index 7e400682bf25..e93d010f1fa2 100644 --- a/test/py/pytest.ini +++ b/test/py/pytest.ini @@ -8,3 +8,6 @@ markers = boardspec: U-Boot: Describes the set of boards a test can/can't run on. buildconfigspec: U-Boot: Describes Kconfig/config-header constraints. + notbuildconfigspec: U-Boot: Describes required disabled Kconfig options. + requiredtool: U-Boot: Required host tools for a test. + slow: U-Boot: Specific test will run slowly. diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py index 4922b9dcc664..356d9a20f299 100755 --- a/test/py/tests/test_fit.py +++ b/test/py/tests/test_fit.py @@ -153,7 +153,7 @@ def test_fit(u_boot_console): src = make_fname('u-boot.dts') dtb = make_fname('u-boot.dtb') with open(src, 'w') as fd: - print(base_fdt, file=fd) + fd.write(base_fdt) util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb]) return dtb @@ -186,7 +186,7 @@ def test_fit(u_boot_console): its = make_its(params) util.run_and_log(cons, [mkimage, '-f', its, fit]) with open(make_fname('u-boot.dts'), 'w') as fd: - print(base_fdt, file=fd) + fd.write(base_fdt) return fit def make_kernel(filename, text): diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py index b011a3e3da25..7fedb18d2790 100644 --- a/test/py/u_boot_spawn.py +++ b/test/py/u_boot_spawn.py @@ -113,7 +113,7 @@ class Spawn(object): Nothing. """ - os.write(self.fd, data) + os.write(self.fd, data.encode(errors="replace")) def expect(self, patterns): """Wait for the sub-process to emit specific data. @@ -171,7 +171,7 @@ class Spawn(object): events = self.poll.poll(poll_maxwait) if not events: raise Timeout() - c = os.read(self.fd, 1024) + c = os.read(self.fd, 1024).decode(errors="replace") if not c: raise EOFError() if self.logfile_read:
- Modern pytest is more visible in telling us about parameters that we had not described, so describe a few more. - ConfigParser.readfp(...) is now configparser.read_file(...) - As part of the "strings vs bytes" conversions in Python 3, we use the default encoding/decoding of utf-8 but in some places tell Python to replace problematic conversions rather than throw a fatal error. - Fix a typo noticed while doing the above ("tot he" -> "to the"). - As suggested by Stephen, re-alphabetize the import list - Per Heinrich, replace how we write contents in test_fit.py Signed-off-by: Tom Rini <trini@konsulko.com> --- test/py/conftest.py | 9 ++++----- test/py/multiplexed_log.py | 9 +++++---- test/py/pytest.ini | 3 +++ test/py/tests/test_fit.py | 4 ++-- test/py/u_boot_spawn.py | 4 ++-- 5 files changed, 16 insertions(+), 13 deletions(-)