| Message ID | 1292536325-12881-1-git-send-email-weil@mail.berlios.de |
|---|---|
| State | Superseded |
| Headers | show |
On 12/16/2010 10:52 PM, Stefan Weil wrote: > QEMU source code with CRLF line endings > which is quite common on windows hosts > fails with current make_device_config.sh. > > The awk script gets the name of the included > file with \r, so instead of pci.mak it will > search for pci.mak\r which of course does > not work. > > Fix this by removing any \r at end of line. Why isn't cygwin (or whatever you're using) removing it? Paolo
Am 16.12.2010 um 23:42 schrieb Paolo Bonzini: > On 12/16/2010 10:52 PM, Stefan Weil wrote: >> QEMU source code with CRLF line endings >> which is quite common on windows hosts >> fails with current make_device_config.sh. >> >> The awk script gets the name of the included >> file with \r, so instead of pci.mak it will >> search for pci.mak\r which of course does >> not work. >> >> Fix this by removing any \r at end of line. > > Why isn't cygwin (or whatever you're using) removing it? I believe Cygwin had an option to choose the line ending style once during setup. By default it uses Unix-style line endings. If someone edits source files in some Windows editor, the above issue seems likely. My awk foo is insufficient to ack the patch but if it doesn't break Unices it looks reasonable to me. Andreas
On 12/17/2010 02:34 PM, Andreas Färber wrote: >>> >>> Fix this by removing any \r at end of line. >> >> Why isn't cygwin (or whatever you're using) removing it? > > I believe Cygwin had an option to choose the line ending style once > during setup. By default it uses Unix-style line endings. > If someone edits source files in some Windows editor, the above issue > seems likely. Hmm, MSYS is more problematic: the manual says "In the MSYS environment under Windows, `gawk' automatically uses binary mode for reading and writing files. Thus there is no need to use the `BINMODE' variable. This can cause problems with other Unix-like components that have been ported to Windows that expect `gawk' to do automatic translationof `"\r\n"', since it won't.". > My awk foo is insufficient to ack the patch but if it doesn't break > Unices it looks reasonable to me. I'd be worried a bit about Solaris and other proprietary OSes with prehistoric Unix utilities. Perhaps using \012 instead of \r is better. Paolo
Am 17.12.2010 14:44, schrieb Paolo Bonzini: > On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>> >>>> Fix this by removing any \r at end of line. >>> >>> Why isn't cygwin (or whatever you're using) removing it? >> >> I believe Cygwin had an option to choose the line ending style once >> during setup. By default it uses Unix-style line endings. >> If someone edits source files in some Windows editor, the above issue >> seems likely. > > Hmm, MSYS is more problematic: the manual says "In the MSYS > environment under Windows, `gawk' automatically uses binary mode for > reading and writing files. Thus there is no need to use the `BINMODE' > variable. This can cause problems with other Unix-like components that > have been ported to Windows that expect `gawk' to do automatic > translationof `"\r\n"', since it won't.". I had the \r problem when using msys. There seem to be several factors: * git configuration (add CR / don't add CR) * msys mount options (binmode / textmode) * (g)awk (read binary) > >> My awk foo is insufficient to ack the patch but if it doesn't break >> Unices it looks reasonable to me. > > I'd be worried a bit about Solaris and other proprietary OSes with > prehistoric Unix utilities. Perhaps using \012 instead of \r is better. > > Paolo I currently have no possibility to run tests with other awk implementations, so I cannot tell whether \012 is needed at all (\r is not a new invention). Do all awk versions support regular expressions with characters in octal encoding?. Perhaps we can wait several days for feedback from users with an old awk and apply the patch if there is none. Regards, Stefan
Am 17.12.2010 um 20:01 schrieb Stefan Weil: > Am 17.12.2010 14:44, schrieb Paolo Bonzini: >> On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>>> >>>>> Fix this by removing any \r at end of line. >>>> >>>> Why isn't cygwin (or whatever you're using) removing it? >>> >>> I believe Cygwin had an option to choose the line ending style once >>> during setup. By default it uses Unix-style line endings. >>> If someone edits source files in some Windows editor, the above >>> issue >>> seems likely. >> >> Hmm, MSYS is more problematic: the manual says "In the MSYS >> environment under Windows, `gawk' automatically uses binary mode >> for reading and writing files. Thus there is no need to use the >> `BINMODE' variable. This can cause problems with other Unix-like >> components that have been ported to Windows that expect `gawk' to >> do automatic translationof `"\r\n"', since it won't.". > > I had the \r problem when using msys. There seem to be several > factors: > > * git configuration (add CR / don't add CR) > * msys mount options (binmode / textmode) > * (g)awk (read binary) > >> >>> My awk foo is insufficient to ack the patch but if it doesn't break >>> Unices it looks reasonable to me. >> >> I'd be worried a bit about Solaris and other proprietary OSes with >> prehistoric Unix utilities. Perhaps using \012 instead of \r is >> better. >> >> Paolo > > I currently have no possibility to run tests with other awk > implementations, > so I cannot tell whether \012 is needed at all (\r is not a new > invention). > Do all awk versions support regular expressions with characters in > octal encoding?. > > Perhaps we can wait several days for feedback from users with an old > awk > and apply the patch if there is none. I have an open bug ticket wrt configure being broken on Solaris 10 due to tracing shell incompatibilities. If you provide a small self-contained example I can check though. Andreas
Am 17.12.2010 20:33, schrieb Andreas Färber: > Am 17.12.2010 um 20:01 schrieb Stefan Weil: > >> Am 17.12.2010 14:44, schrieb Paolo Bonzini: >>> On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>>>> >>>>>> Fix this by removing any \r at end of line. >>>>> >>>>> Why isn't cygwin (or whatever you're using) removing it? >>>> >>>> I believe Cygwin had an option to choose the line ending style once >>>> during setup. By default it uses Unix-style line endings. >>>> If someone edits source files in some Windows editor, the above issue >>>> seems likely. >>> >>> Hmm, MSYS is more problematic: the manual says "In the MSYS >>> environment under Windows, `gawk' automatically uses binary mode for >>> reading and writing files. Thus there is no need to use the >>> `BINMODE' variable. This can cause problems with other Unix-like >>> components that have been ported to Windows that expect `gawk' to do >>> automatic translationof `"\r\n"', since it won't.". >> >> I had the \r problem when using msys. There seem to be several factors: >> >> * git configuration (add CR / don't add CR) >> * msys mount options (binmode / textmode) >> * (g)awk (read binary) >> >>> >>>> My awk foo is insufficient to ack the patch but if it doesn't break >>>> Unices it looks reasonable to me. >>> >>> I'd be worried a bit about Solaris and other proprietary OSes with >>> prehistoric Unix utilities. Perhaps using \012 instead of \r is >>> better. >>> >>> Paolo >> >> I currently have no possibility to run tests with other awk >> implementations, >> so I cannot tell whether \012 is needed at all (\r is not a new >> invention). >> Do all awk versions support regular expressions with characters in >> octal encoding?. >> >> Perhaps we can wait several days for feedback from users with an old awk >> and apply the patch if there is none. > > I have an open bug ticket wrt configure being broken on Solaris 10 due > to tracing shell incompatibilities. > > If you provide a small self-contained example I can check though. > > Andreas Example (needs bash's echo -e): # create line with crlf ending: echo -e 'include xy\r' >file # returns xy\r: awk '/^include / {ORS=" "; print $2}' file | od # should return xy: awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od Regards, Stefan
On 12/17/2010 08:01 PM, Stefan Weil wrote: > I currently have no possibility to run tests with other awk > implementations, > so I cannot tell whether \012 is needed at all (\r is not a new invention). > Do all awk versions support regular expressions with characters in > octal encoding?. The awk manual reports \b, \f and \r as unportable, and says that \a, \v, \x have been introduced after those. \x is not in POSIX. That leaves \a, \n, \t, \v and \NNN as portable. Paolo
Am 17.12.2010 um 20:54 schrieb Stefan Weil: > Am 17.12.2010 20:33, schrieb Andreas Färber: >> Am 17.12.2010 um 20:01 schrieb Stefan Weil: >> >>> Am 17.12.2010 14:44, schrieb Paolo Bonzini: >>>> On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>>>>> >>>>>>> Fix this by removing any \r at end of line. >>>>>> >>>>>> Why isn't cygwin (or whatever you're using) removing it? >>>>> >>>>> I believe Cygwin had an option to choose the line ending style >>>>> once >>>>> during setup. By default it uses Unix-style line endings. >>>>> If someone edits source files in some Windows editor, the above >>>>> issue >>>>> seems likely. >>>> >>>> Hmm, MSYS is more problematic: the manual says "In the MSYS >>>> environment under Windows, `gawk' automatically uses binary mode >>>> for reading and writing files. Thus there is no need to use the >>>> `BINMODE' variable. This can cause problems with other Unix-like >>>> components that have been ported to Windows that expect `gawk' to >>>> do automatic translationof `"\r\n"', since it won't.". >>> >>> I had the \r problem when using msys. There seem to be several >>> factors: >>> >>> * git configuration (add CR / don't add CR) >>> * msys mount options (binmode / textmode) >>> * (g)awk (read binary) >>> >>>> >>>>> My awk foo is insufficient to ack the patch but if it doesn't >>>>> break >>>>> Unices it looks reasonable to me. >>>> >>>> I'd be worried a bit about Solaris and other proprietary OSes >>>> with prehistoric Unix utilities. Perhaps using \012 instead of >>>> \r is better. >>>> >>>> Paolo >>> >>> I currently have no possibility to run tests with other awk >>> implementations, >>> so I cannot tell whether \012 is needed at all (\r is not a new >>> invention). >>> Do all awk versions support regular expressions with characters in >>> octal encoding?. >>> >>> Perhaps we can wait several days for feedback from users with an >>> old awk >>> and apply the patch if there is none. >> >> I have an open bug ticket wrt configure being broken on Solaris 10 >> due to tracing shell incompatibilities. >> >> If you provide a small self-contained example I can check though. >> >> Andreas > > > Example (needs bash's echo -e): > > # create line with crlf ending: > echo -e 'include xy\r' >file > > # returns xy\r: > awk '/^include / {ORS=" "; print $2}' file | od 0000000 074570 020015 0000004 > # should return xy: > awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od awk: syntax error near line 1 awk: illegal statement near line 1 0000000 Andreas
On 12/17/2010 10:00 PM, Andreas Färber wrote: > Am 17.12.2010 um 20:54 schrieb Stefan Weil: > >> Am 17.12.2010 20:33, schrieb Andreas Färber: >>> Am 17.12.2010 um 20:01 schrieb Stefan Weil: >>> >>>> Am 17.12.2010 14:44, schrieb Paolo Bonzini: >>>>> On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>>>>>> >>>>>>>> Fix this by removing any \r at end of line. >>>>>>> >>>>>>> Why isn't cygwin (or whatever you're using) removing it? >>>>>> >>>>>> I believe Cygwin had an option to choose the line ending style once >>>>>> during setup. By default it uses Unix-style line endings. >>>>>> If someone edits source files in some Windows editor, the above issue >>>>>> seems likely. >>>>> >>>>> Hmm, MSYS is more problematic: the manual says "In the MSYS >>>>> environment under Windows, `gawk' automatically uses binary mode >>>>> for reading and writing files. Thus there is no need to use the >>>>> `BINMODE' variable. This can cause problems with other Unix-like >>>>> components that have been ported to Windows that expect `gawk' to >>>>> do automatic translationof `"\r\n"', since it won't.". >>>> >>>> I had the \r problem when using msys. There seem to be several factors: >>>> >>>> * git configuration (add CR / don't add CR) >>>> * msys mount options (binmode / textmode) >>>> * (g)awk (read binary) >>>> >>>>> >>>>>> My awk foo is insufficient to ack the patch but if it doesn't break >>>>>> Unices it looks reasonable to me. >>>>> >>>>> I'd be worried a bit about Solaris and other proprietary OSes with >>>>> prehistoric Unix utilities. Perhaps using \012 instead of \r is >>>>> better. >>>>> >>>>> Paolo >>>> >>>> I currently have no possibility to run tests with other awk >>>> implementations, >>>> so I cannot tell whether \012 is needed at all (\r is not a new >>>> invention). >>>> Do all awk versions support regular expressions with characters in >>>> octal encoding?. >>>> >>>> Perhaps we can wait several days for feedback from users with an old >>>> awk >>>> and apply the patch if there is none. >>> >>> I have an open bug ticket wrt configure being broken on Solaris 10 >>> due to tracing shell incompatibilities. >>> >>> If you provide a small self-contained example I can check though. >>> >>> Andreas >> >> >> Example (needs bash's echo -e): >> >> # create line with crlf ending: >> echo -e 'include xy\r' >file >> >> # returns xy\r: >> awk '/^include / {ORS=" "; print $2}' file | od > > 0000000 074570 020015 > 0000004 > >> # should return xy: >> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od > > awk: syntax error near line 1 > awk: illegal statement near line 1 > 0000000 Can you try \015 instead of \r? Paolo
Am 17.12.2010 um 22:25 schrieb Paolo Bonzini: > On 12/17/2010 10:00 PM, Andreas Färber wrote: >> Am 17.12.2010 um 20:54 schrieb Stefan Weil: >> >>> Am 17.12.2010 20:33, schrieb Andreas Färber: >>>> Am 17.12.2010 um 20:01 schrieb Stefan Weil: >>>> >>>>> Am 17.12.2010 14:44, schrieb Paolo Bonzini: >>>>>> On 12/17/2010 02:34 PM, Andreas Färber wrote: >>>>>>>>> >>>>>>>>> Fix this by removing any \r at end of line. >>>>>>>> >>>>>>>> Why isn't cygwin (or whatever you're using) removing it? >>>>>>> >>>>>>> I believe Cygwin had an option to choose the line ending style >>>>>>> once >>>>>>> during setup. By default it uses Unix-style line endings. >>>>>>> If someone edits source files in some Windows editor, the >>>>>>> above issue >>>>>>> seems likely. >>>>>> >>>>>> Hmm, MSYS is more problematic: the manual says "In the MSYS >>>>>> environment under Windows, `gawk' automatically uses binary mode >>>>>> for reading and writing files. Thus there is no need to use the >>>>>> `BINMODE' variable. This can cause problems with other Unix-like >>>>>> components that have been ported to Windows that expect `gawk' to >>>>>> do automatic translationof `"\r\n"', since it won't.". >>>>> >>>>> I had the \r problem when using msys. There seem to be several >>>>> factors: >>>>> >>>>> * git configuration (add CR / don't add CR) >>>>> * msys mount options (binmode / textmode) >>>>> * (g)awk (read binary) >>>>> >>>>>> >>>>>>> My awk foo is insufficient to ack the patch but if it doesn't >>>>>>> break >>>>>>> Unices it looks reasonable to me. >>>>>> >>>>>> I'd be worried a bit about Solaris and other proprietary OSes >>>>>> with >>>>>> prehistoric Unix utilities. Perhaps using \012 instead of \r is >>>>>> better. >>>>>> >>>>>> Paolo >>>>> >>>>> I currently have no possibility to run tests with other awk >>>>> implementations, >>>>> so I cannot tell whether \012 is needed at all (\r is not a new >>>>> invention). >>>>> Do all awk versions support regular expressions with characters in >>>>> octal encoding?. >>>>> >>>>> Perhaps we can wait several days for feedback from users with an >>>>> old >>>>> awk >>>>> and apply the patch if there is none. >>>> >>>> I have an open bug ticket wrt configure being broken on Solaris 10 >>>> due to tracing shell incompatibilities. >>>> >>>> If you provide a small self-contained example I can check though. >>>> >>>> Andreas >>> >>> >>> Example (needs bash's echo -e): >>> >>> # create line with crlf ending: >>> echo -e 'include xy\r' >file >>> >>> # returns xy\r: >>> awk '/^include / {ORS=" "; print $2}' file | od >> >> 0000000 074570 020015 >> 0000004 >> >>> # should return xy: >>> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od >> >> awk: syntax error near line 1 >> awk: illegal statement near line 1 >> 0000000 > > Can you try \015 instead of \r? Same. Andreas
On 12/17/2010 11:17 PM, Andreas Färber wrote: >>>> Example (needs bash's echo -e): >>>> >>>> # create line with crlf ending: >>>> echo -e 'include xy\r' >file >>>> >>>> # returns xy\r: >>>> awk '/^include / {ORS=" "; print $2}' file | od >>> >>> 0000000 074570 020015 >>> 0000004 >>> >>>> # should return xy: >>>> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od >>> >>> awk: syntax error near line 1 >>> awk: illegal statement near line 1 >>> 0000000 >> >> Can you try \015 instead of \r? > > Same. Then I guess Stefan should use tr -d '\015' < file | awk '/^include / {ORS=" "; print $2}' or something like that. Paolo
Am 18.12.2010 00:24, schrieb Paolo Bonzini: > On 12/17/2010 11:17 PM, Andreas Färber wrote: >>>>> Example (needs bash's echo -e): >>>>> >>>>> # create line with crlf ending: >>>>> echo -e 'include xy\r' >file >>>>> >>>>> # returns xy\r: >>>>> awk '/^include / {ORS=" "; print $2}' file | od >>>> >>>> 0000000 074570 020015 >>>> 0000004 >>>> >>>>> # should return xy: >>>>> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od >>>> >>>> awk: syntax error near line 1 >>>> awk: illegal statement near line 1 >>>> 0000000 >>> >>> Can you try \015 instead of \r? >> >> Same. > > Then I guess Stefan should use > > tr -d '\015' < file | awk '/^include / {ORS=" "; print $2}' > > or something like that. > > Paolo Andreas, please try this variant of Paolo's suggestion: tr -d '\r' file | awk '/^include / {ORS=" "; print $2}' | od -c I found some SunOS man pages online: tr should work with '\r', and awk does not support the sub function (therefore you always got a syntax error). If the new code works, I'll send a modified patch. Thanks, Stefan
Am 18.12.2010 um 11:19 schrieb Stefan Weil: > Am 18.12.2010 00:24, schrieb Paolo Bonzini: >> On 12/17/2010 11:17 PM, Andreas Färber wrote: >>>>>> Example (needs bash's echo -e): >>>>>> >>>>>> # create line with crlf ending: >>>>>> echo -e 'include xy\r' >file >>>>>> >>>>>> # returns xy\r: >>>>>> awk '/^include / {ORS=" "; print $2}' file | od >>>>> >>>>> 0000000 074570 020015 >>>>> 0000004 >>>>> >>>>>> # should return xy: >>>>>> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file >>>>>> | od >>>>> >>>>> awk: syntax error near line 1 >>>>> awk: illegal statement near line 1 >>>>> 0000000 >>>> >>>> Can you try \015 instead of \r? >>> >>> Same. >> >> Then I guess Stefan should use >> >> tr -d '\015' < file | awk '/^include / {ORS=" "; print $2}' >> >> or something like that. >> >> Paolo > > > Andreas, please try this variant of Paolo's suggestion: > > tr -d '\r' file | awk '/^include / {ORS=" "; print $2}' | od -c Like that, the tr hangs. The following works better: bash-3.00$ tr -d '\r' < file | awk '/^include / {ORS=" "; print $2}' | od -c 0000000 x y 0000003 Andreas
Am 18.12.2010 13:02, schrieb Andreas Färber: > Am 18.12.2010 um 11:19 schrieb Stefan Weil: > >> Am 18.12.2010 00:24, schrieb Paolo Bonzini: >>> On 12/17/2010 11:17 PM, Andreas Färber wrote: >>>>>>> Example (needs bash's echo -e): >>>>>>> >>>>>>> # create line with crlf ending: >>>>>>> echo -e 'include xy\r' >file >>>>>>> >>>>>>> # returns xy\r: >>>>>>> awk '/^include / {ORS=" "; print $2}' file | od >>>>>> >>>>>> 0000000 074570 020015 >>>>>> 0000004 >>>>>> >>>>>>> # should return xy: >>>>>>> awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print $2}' file | od >>>>>> >>>>>> awk: syntax error near line 1 >>>>>> awk: illegal statement near line 1 >>>>>> 0000000 >>>>> >>>>> Can you try \015 instead of \r? >>>> >>>> Same. >>> >>> Then I guess Stefan should use >>> >>> tr -d '\015' < file | awk '/^include / {ORS=" "; print $2}' >>> >>> or something like that. >>> >>> Paolo >> >> >> Andreas, please try this variant of Paolo's suggestion: >> >> tr -d '\r' file | awk '/^include / {ORS=" "; print $2}' | od -c > > Like that, the tr hangs. The following works better: > > bash-3.00$ tr -d '\r' < file | awk '/^include / {ORS=" "; print $2}' | > od -c > 0000000 x y > 0000003 > > Andreas Mea culpa. Of course tr expects its data from stdin - the '<' got lost when I wrote the mail. Thanks for the testing. Stefan
diff --git a/make_device_config.sh b/make_device_config.sh index 8abadfe..f0ce10a 100644 --- a/make_device_config.sh +++ b/make_device_config.sh @@ -18,7 +18,7 @@ process_includes () { f=$src while [ -n "$f" ] ; do - f=`awk '/^include / {ORS=" " ; print "'$src_dir'/" $2}' $f` + f=`awk '/^include / {ORS=" "; sub(/\r$/, "", $2); print "'$src_dir'/" $2}' $f` [ $? = 0 ] || exit 1 all_includes="$all_includes $f" done
QEMU source code with CRLF line endings which is quite common on windows hosts fails with current make_device_config.sh. The awk script gets the name of the included file with \r, so instead of pci.mak it will search for pci.mak\r which of course does not work. Fix this by removing any \r at end of line. Signed-off-by: Stefan Weil <weil@mail.berlios.de> --- make_device_config.sh | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)