| Message ID | 20240228223736.2376826-1-yann.morin.1998@free.fr |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | support/scripts/cve: fix running on older ijson versions | expand |
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for > the JSON files) had to change the decompressor from gz to xz, as the new > location is using xz compression. > That commit mentioned that it was spawning an external xz process to do > the decompression, on the pretence that "there is no xz decompressor in > Python stdlib." > ijson started to accept bytes() (and str()) only with version 3.1, and > using a subprocess means we are now passing bytes() to ijson, which it > is not expecting as input on such older versions, casuing build failures > such as: > [...] > File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer > if type(f.read(0)) == bytetype: > AttributeError: 'bytes' object has no attribute 'read' > Ubuntu 20.04, on which the pkg-stats run to generate the daily report, > only has ijson 2.3. More recent distros have more recent versions of > ijson, like Fedora 39 that has 3.2.3, recent enough to supoprt being fed > bytes(). > However, the reasonining in 22b69455526f is wrong: there *is* the lzma > module, at least since python 3.3, that is, aeons ago, which is able to > read xz-compressed files; it also has an API similar to the gzip module, > and can provide a file-like object that exposes the decompressed data. > So, do just that: provide an lzma-wrapped file-like object to ijson, so > that we can eventually recover our daily reports that everything is > broken! :-] > Note that this construct still works on recent versions! > Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> > --- > support/scripts/cve.py | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > index 1a3c307e12..7167ecbc6a 100755 > --- a/support/scripts/cve.py > +++ b/support/scripts/cve.py > @@ -21,8 +21,8 @@ import datetime > import os > import requests # URL checking > import distutils.version > +import lzma > import time > -import subprocess > import sys > import operator > @@ -134,8 +134,7 @@ class CVE: > for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1): > filename = CVE.download_nvd_year(nvd_dir, year) > try: > - uncompressed = subprocess.check_output(["xz", "-d", "-c", filename]) > - content = ijson.items(uncompressed, 'cve_items.item') > + content = ijson.items(lzma.LZMAFile(filename), 'cve_items.item') Are you sure this provides str()? xz GPL-2.0 python3 Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import lzma >>> lzma.LZMAFile('GPL-2.0.xz').read(100) b'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G' Whereas lzma.open() accepts a 'rt' mode: >>> lzma.open('GPL-2.0.xz', mode='rt').read(100) 'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G'
diff --git a/support/scripts/cve.py b/support/scripts/cve.py index 1a3c307e12..7167ecbc6a 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -21,8 +21,8 @@ import datetime import os import requests # URL checking import distutils.version +import lzma import time -import subprocess import sys import operator @@ -134,8 +134,7 @@ class CVE: for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1): filename = CVE.download_nvd_year(nvd_dir, year) try: - uncompressed = subprocess.check_output(["xz", "-d", "-c", filename]) - content = ijson.items(uncompressed, 'cve_items.item') + content = ijson.items(lzma.LZMAFile(filename), 'cve_items.item') except: # noqa: E722 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename) raise
Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for the JSON files) had to change the decompressor from gz to xz, as the new location is using xz compression. That commit mentioned that it was spawning an external xz process to do the decompression, on the pretence that "there is no xz decompressor in Python stdlib." ijson started to accept bytes() (and str()) only with version 3.1, and using a subprocess means we are now passing bytes() to ijson, which it is not expecting as input on such older versions, casuing build failures such as: [...] File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer if type(f.read(0)) == bytetype: AttributeError: 'bytes' object has no attribute 'read' Ubuntu 20.04, on which the pkg-stats run to generate the daily report, only has ijson 2.3. More recent distros have more recent versions of ijson, like Fedora 39 that has 3.2.3, recent enough to supoprt being fed bytes(). However, the reasonining in 22b69455526f is wrong: there *is* the lzma module, at least since python 3.3, that is, aeons ago, which is able to read xz-compressed files; it also has an API similar to the gzip module, and can provide a file-like object that exposes the decompressed data. So, do just that: provide an lzma-wrapped file-like object to ijson, so that we can eventually recover our daily reports that everything is broken! :-] Note that this construct still works on recent versions! Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> --- support/scripts/cve.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)