| Message ID | 20170531134002.5360-1-daniel.schwierzeck@gmail.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Simon Glass |
| Headers | show |
Hi Daniel, On 31 May 2017 at 07:40, Daniel Schwierzeck <daniel.schwierzeck@gmail.com> wrote: > This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814. > > This fixes following exception: > > Exception in thread Thread-7: > Traceback (most recent call last): > File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/u-boot/tools/buildman/builderthread.py", line 475, in run > self.RunJob(job) > File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob > self._WriteResult(result, job.keep_outputs) > File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult > print >>fd, dump_result.stdout, > UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128) > > Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com> > > --- > > tools/buildman/builderthread.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py > index acaf5007f5..0a460878f3 100644 > --- a/tools/buildman/builderthread.py > +++ b/tools/buildman/builderthread.py > @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread): > objdump = self.builder.GetObjdumpFile(result.commit_upto, > result.brd.target, fname) > with open(objdump, 'w') as fd: > - print >>fd, dump_result.stdout, > + print >>fd, dump_result.stdout.encode('latin-1', 'ignore'), I am worried that this will produce gibberish in the output. Do you know what specific characters are causing problems? Can we encode to utf-8? BTW I recently sent a patman series to try to resolve unicode issues in patman. > for line in dump_result.stdout.splitlines(): > fields = line.split() > if len(fields) > 5 and fields[1] == '.rodata': > -- > 2.13.0 > Regards, Simon
Am 01.06.2017 um 15:21 schrieb Simon Glass: > Hi Daniel, > > On 31 May 2017 at 07:40, Daniel Schwierzeck > <daniel.schwierzeck@gmail.com> wrote: >> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814. >> >> This fixes following exception: >> >> Exception in thread Thread-7: >> Traceback (most recent call last): >> File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner >> self.run() >> File "/u-boot/tools/buildman/builderthread.py", line 475, in run >> self.RunJob(job) >> File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob >> self._WriteResult(result, job.keep_outputs) >> File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult >> print >>fd, dump_result.stdout, >> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128) >> >> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com> >> >> --- >> >> tools/buildman/builderthread.py | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py >> index acaf5007f5..0a460878f3 100644 >> --- a/tools/buildman/builderthread.py >> +++ b/tools/buildman/builderthread.py >> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread): >> objdump = self.builder.GetObjdumpFile(result.commit_upto, >> result.brd.target, fname) >> with open(objdump, 'w') as fd: >> - print >>fd, dump_result.stdout, >> + print >>fd, dump_result.stdout.encode('latin-1', 'ignore'), > > I am worried that this will produce gibberish in the output. Do you > know what specific characters are causing problems? Can we encode to > utf-8? BTW I recently sent a patman series to try to resolve unicode > issues in patman. > objdump uses localized output. If I use LC_ALL=C, the exception doesn't occur. My system's default language is German. Thus the offending character is the German 'ö': ... Sektionen: Idx Name Größe VMA LMA Datei-Off Ausr. 0 .text 00032a94 9f000000 9f000000 000000c0 2**4 CONTENTS, ALLOC, LOAD, CODE 1 .rodata 00008e50 9f032aa0 9f032aa0 00032b60 2**4 CONTENTS, ALLOC, LOAD, READONLY, DATA ... Actually I'd chosen UTF-8 too, but I followed Tom's patch which used latin-1 ;) So we should always encode the output of tools such as binutils to UTF-8, shouldn't we? I can send an updated patch.
On Fri, Jun 02, 2017 at 11:18:27AM +0200, Daniel Schwierzeck wrote: > > > Am 01.06.2017 um 15:21 schrieb Simon Glass: > > Hi Daniel, > > > > On 31 May 2017 at 07:40, Daniel Schwierzeck > > <daniel.schwierzeck@gmail.com> wrote: > >> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814. > >> > >> This fixes following exception: > >> > >> Exception in thread Thread-7: > >> Traceback (most recent call last): > >> File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner > >> self.run() > >> File "/u-boot/tools/buildman/builderthread.py", line 475, in run > >> self.RunJob(job) > >> File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob > >> self._WriteResult(result, job.keep_outputs) > >> File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult > >> print >>fd, dump_result.stdout, > >> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128) > >> > >> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com> > >> > >> --- > >> > >> tools/buildman/builderthread.py | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py > >> index acaf5007f5..0a460878f3 100644 > >> --- a/tools/buildman/builderthread.py > >> +++ b/tools/buildman/builderthread.py > >> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread): > >> objdump = self.builder.GetObjdumpFile(result.commit_upto, > >> result.brd.target, fname) > >> with open(objdump, 'w') as fd: > >> - print >>fd, dump_result.stdout, > >> + print >>fd, dump_result.stdout.encode('latin-1', 'ignore'), > > > > I am worried that this will produce gibberish in the output. Do you > > know what specific characters are causing problems? Can we encode to > > utf-8? BTW I recently sent a patman series to try to resolve unicode > > issues in patman. > > > > objdump uses localized output. If I use LC_ALL=C, the exception doesn't > occur. My system's default language is German. Thus the offending > character is the German 'ö': > > ... > Sektionen: > Idx Name Größe VMA LMA Datei-Off Ausr. > 0 .text 00032a94 9f000000 9f000000 000000c0 2**4 > CONTENTS, ALLOC, LOAD, CODE > 1 .rodata 00008e50 9f032aa0 9f032aa0 00032b60 2**4 > CONTENTS, ALLOC, LOAD, READONLY, DATA > ... > > Actually I'd chosen UTF-8 too, but I followed Tom's patch which used > latin-1 ;) > > So we should always encode the output of tools such as binutils to > UTF-8, shouldn't we? I can send an updated patch. I'd be fine with UTF-8 everywhere too, I just borrowed an existing example that was latin-1 :)
Hi Daniel, On 2 June 2017 at 07:21, Tom Rini <trini@konsulko.com> wrote: > On Fri, Jun 02, 2017 at 11:18:27AM +0200, Daniel Schwierzeck wrote: >> >> >> Am 01.06.2017 um 15:21 schrieb Simon Glass: >> > Hi Daniel, >> > >> > On 31 May 2017 at 07:40, Daniel Schwierzeck >> > <daniel.schwierzeck@gmail.com> wrote: >> >> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814. >> >> >> >> This fixes following exception: >> >> >> >> Exception in thread Thread-7: >> >> Traceback (most recent call last): >> >> File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner >> >> self.run() >> >> File "/u-boot/tools/buildman/builderthread.py", line 475, in run >> >> self.RunJob(job) >> >> File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob >> >> self._WriteResult(result, job.keep_outputs) >> >> File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult >> >> print >>fd, dump_result.stdout, >> >> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128) >> >> >> >> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com> >> >> >> >> --- >> >> >> >> tools/buildman/builderthread.py | 2 +- >> >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> >> >> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py >> >> index acaf5007f5..0a460878f3 100644 >> >> --- a/tools/buildman/builderthread.py >> >> +++ b/tools/buildman/builderthread.py >> >> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread): >> >> objdump = self.builder.GetObjdumpFile(result.commit_upto, >> >> result.brd.target, fname) >> >> with open(objdump, 'w') as fd: >> >> - print >>fd, dump_result.stdout, >> >> + print >>fd, dump_result.stdout.encode('latin-1', 'ignore'), >> > >> > I am worried that this will produce gibberish in the output. Do you >> > know what specific characters are causing problems? Can we encode to >> > utf-8? BTW I recently sent a patman series to try to resolve unicode >> > issues in patman. >> > >> >> objdump uses localized output. If I use LC_ALL=C, the exception doesn't >> occur. My system's default language is German. Thus the offending >> character is the German 'ö': >> >> ... >> Sektionen: >> Idx Name Größe VMA LMA Datei-Off Ausr. >> 0 .text 00032a94 9f000000 9f000000 000000c0 2**4 >> CONTENTS, ALLOC, LOAD, CODE >> 1 .rodata 00008e50 9f032aa0 9f032aa0 00032b60 2**4 >> CONTENTS, ALLOC, LOAD, READONLY, DATA >> ... >> >> Actually I'd chosen UTF-8 too, but I followed Tom's patch which used >> latin-1 ;) >> >> So we should always encode the output of tools such as binutils to >> UTF-8, shouldn't we? I can send an updated patch. > > I'd be fine with UTF-8 everywhere too, I just borrowed an existing > example that was latin-1 :) Are you planning to send an updated patch? Regards, Simon
Am 07.06.2017 um 13:55 schrieb Simon Glass: ... >>> >>> Actually I'd chosen UTF-8 too, but I followed Tom's patch which used >>> latin-1 ;) >>> >>> So we should always encode the output of tools such as binutils to >>> UTF-8, shouldn't we? I can send an updated patch. >> >> I'd be fine with UTF-8 everywhere too, I just borrowed an existing >> example that was latin-1 :) > > Are you planning to send an updated patch? > sure, I forgot to send it ;)
diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index acaf5007f5..0a460878f3 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread): objdump = self.builder.GetObjdumpFile(result.commit_upto, result.brd.target, fname) with open(objdump, 'w') as fd: - print >>fd, dump_result.stdout, + print >>fd, dump_result.stdout.encode('latin-1', 'ignore'), for line in dump_result.stdout.splitlines(): fields = line.split() if len(fields) > 5 and fields[1] == '.rodata':
This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814. This fixes following exception: Exception in thread Thread-7: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/u-boot/tools/buildman/builderthread.py", line 475, in run self.RunJob(job) File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob self._WriteResult(result, job.keep_outputs) File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult print >>fd, dump_result.stdout, UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128) Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com> --- tools/buildman/builderthread.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)