diff mbox series

[RFC,02/28] cli: Add LIL shell

Message ID 20210701061611.957918-3-seanga2@gmail.com
State RFC
Delegated to: Tom Rini
Headers show
Series cli: Add a new shell | expand

Commit Message

Sean Anderson July 1, 2021, 6:15 a.m. UTC
This is the LIL programming language [1] as originally written by Kostas
Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
variant. Many syntax features are very similar to shell:

	=> echo Hello from $stdout
	Hello from serial,vidconsole

LIL supports functions, scoped variables, and command
substitution. For a short example, running the following script

	func fact {n} {
		if {$n} {
			expr {$n * [fact [expr {$n - 1}]]}
		} {
			return 1
		}
	}
	echo [fact 10]
	echo [fact 20]

would result in the output

	fact
	3628800
	2432902008176640000

For more examples, please refer to the "test: Add tests for LIL" later in
this series.

I have made several mostly-mechanical modifications to the source before
importing it to reduce patches to functional changes. If you are interested
in what has been changed before this commit, refer to my lil git
repository [2]. In addition, some functions and features have been removed
from this patch as I have better determined what is necessary for U-Boot
and what is not. If there is sufficient interest, I can add those changes
to the repository as well.

[1] http://runtimeterror.com/tech/lil/
[2] https://github.com/Forty-Bot/lil/commits/master

Signed-off-by: Sean Anderson <seanga2@gmail.com>
---

 MAINTAINERS       |    6 +
 cmd/Kconfig       |   11 +-
 common/Makefile   |    1 +
 common/cli_lil.c  | 2991 +++++++++++++++++++++++++++++++++++++++++++++
 include/cli_lil.h |  111 ++
 5 files changed, 3119 insertions(+), 1 deletion(-)
 create mode 100644 common/cli_lil.c
 create mode 100644 include/cli_lil.h

Comments

Wolfgang Denk July 2, 2021, 11:03 a.m. UTC | #1
Dear Sean,

In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
> This is the LIL programming language [1] as originally written by Kostas
> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
> variant. Many syntax features are very similar to shell:

Do you have a list of the exact differencec between LIL and a
standard shell?

I wonder, if we deviate from standard shell syntax anyway, we could
also have a look at lua, for example?

Best regards,

Wolfgang Denk
Sean Anderson July 2, 2021, 1:33 p.m. UTC | #2
On 7/2/21 7:03 AM, Wolfgang Denk wrote:
> Dear Sean,
> 
> In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
>> This is the LIL programming language [1] as originally written by Kostas
>> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
>> variant. Many syntax features are very similar to shell:
> 
> Do you have a list of the exact differencec between LIL and a
> standard shell?

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

> I wonder, if we deviate from standard shell syntax anyway, we could
> also have a look at lua, for example?

I also looked at lua (see the cover letter), but I rejected it based on
size constraints (eLua was around the size of U-Boot itself).

Because of how often the shell is used to debug things, I wanted the
candidate I picked to have similar syntax to the existing shell. For
example,

	load mmc ${mmcdev}:${mmcpart} $loadaddr $image

is a valid command in both Hush and LIL. Compare with lua, which might
express the above as

	load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)

which I think is a much larger deviation from existing syntax.

--Sean
Sean Anderson July 3, 2021, 2:12 a.m. UTC | #3
On 7/2/21 9:33 AM, Sean Anderson wrote:
> On 7/2/21 7:03 AM, Wolfgang Denk wrote:
>> Dear Sean,
>>
>> In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
>>> This is the LIL programming language [1] as originally written by Kostas
>>> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
>>> variant. Many syntax features are very similar to shell:
>>
>> Do you have a list of the exact differencec between LIL and a
>> standard shell?
> 
> For a partial list, see
> 
> [1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I
don't have an exhaustive list of differences, but here are some similar
things expressed in both languages:

sh				tcl

foo=bar				set foo bar
echo $foo			echo $foo

if [ 1 -gt 2 ]; then		if {1 > 2} {
	echo a				echo a
else				} {
	echo b				echo b
fi				}

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do	foreach file [glob *.c] {
	echo $file			echo $file
done				}

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}

				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}

Hopefully this gives you a bit of a feel for the basic differences.

--Sean

> 
>> I wonder, if we deviate from standard shell syntax anyway, we could
>> also have a look at lua, for example?
> 
> I also looked at lua (see the cover letter), but I rejected it based on
> size constraints (eLua was around the size of U-Boot itself).
> 
> Because of how often the shell is used to debug things, I wanted the
> candidate I picked to have similar syntax to the existing shell. For
> example,
> 
>      load mmc ${mmcdev}:${mmcpart} $loadaddr $image
> 
> is a valid command in both Hush and LIL. Compare with lua, which might
> express the above as
> 
>      load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)
> 
> which I think is a much larger deviation from existing syntax.
> 
> --Sean
Wolfgang Denk July 3, 2021, 7:23 p.m. UTC | #4
Dear Sean,

In message <c8e20c0c-bcf0-4692-6a8b-0c4ea970b656@gmail.com> you wrote:
> > 
> > Do you have a list of the exact differencec between LIL and a
> > standard shell?
>
> For a partial list, see
>
> [1] https://github.com/Forty-Bot/lil/commits/master

Hm, this list of commits is not exactly helpful, I'm afraid.

Where _exactly_ should I look?

> > I wonder, if we deviate from standard shell syntax anyway, we could
> > also have a look at lua, for example?
>
> I also looked at lua (see the cover letter), but I rejected it based on
> size constraints (eLua was around the size of U-Boot itself).

I have to admit that I never tried myself to build a lua based
system, optimizing for minimal size - but I'm surprised by your
results.

> Because of how often the shell is used to debug things, I wanted the
> candidate I picked to have similar syntax to the existing shell. For
> example,
>
> 	load mmc ${mmcdev}:${mmcpart} $loadaddr $image
>
> is a valid command in both Hush and LIL. Compare with lua, which might
> express the above as

Yes, I'm aware of this.  But then, it would add another level of
powerful scripting capabilities - and as it was already suggested
before, _replacing_ the standard shell is only one way to use lua -
another would be to use lua as a command that can be started from
the shell when needed - assuming you want to pay the price in terms
of size.

Best regards,

Wolfgang Denk
Wolfgang Denk July 3, 2021, 7:33 p.m. UTC | #5
Dear Sean,

In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>
> > For a partial list, see
> > 
> > [1] https://github.com/Forty-Bot/lil/commits/master
>
> Whoops, looks like I completely misread what you were asking here. I
> don't have an exhaustive list of differences, but here are some similar
> things expressed in both languages:
>
> sh				tcl
>
> foo=bar				set foo bar
> echo $foo			echo $foo
>
> if [ 1 -gt 2 ]; then		if {1 > 2} {
> 	echo a				echo a
> else				} {
> 	echo b				echo b
> fi				}
>
> foo() {				proc foo {first second} {
> 	echo $1 $2			echo $first $second
> }				}
>
> for file in $(ls *.c); do	foreach file [glob *.c] {
> 	echo $file			echo $file
> done				}
>
> fact() {
> 	if [ $1 -eq 0 ]; then
> 		echo 1
> 	else
> 		echo $(($1 * $(fact $(($1 - 1)))))
> 	fi
> }
>
> 				proc fact {n} {
> 					if {$n} {
> 						expr {$n * [fact [expr {$n - 1}]]}
> 					} {
> 						return 1
> 					}
> 				}
>
> Hopefully this gives you a bit of a feel for the basic differences.

Well, I know TCL, and there has been a zillion of reasons to move
_from_ that language many, many years ago.  Intoducing this as new
"shell" language in U-Boot does not look attractive to me.
Actually, it gives me the creeps.

Best regards,

Wolfgang Denk
Simon Glass July 5, 2021, 3:29 p.m. UTC | #6
Hi,

On Sat, 3 Jul 2021 at 13:33, Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Sean,
>
> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> >
> > > For a partial list, see
> > >
> > > [1] https://github.com/Forty-Bot/lil/commits/master
> >
> > Whoops, looks like I completely misread what you were asking here. I
> > don't have an exhaustive list of differences, but here are some similar
> > things expressed in both languages:
> >
> > sh                            tcl
> >
> > foo=bar                               set foo bar
> > echo $foo                     echo $foo
> >
> > if [ 1 -gt 2 ]; then          if {1 > 2} {
> >       echo a                          echo a
> > else                          } {
> >       echo b                          echo b
> > fi                            }
> >
> > foo() {                               proc foo {first second} {
> >       echo $1 $2                      echo $first $second
> > }                             }
> >
> > for file in $(ls *.c); do     foreach file [glob *.c] {
> >       echo $file                      echo $file
> > done                          }
> >
> > fact() {
> >       if [ $1 -eq 0 ]; then
> >               echo 1
> >       else
> >               echo $(($1 * $(fact $(($1 - 1)))))
> >       fi
> > }
> >
> >                               proc fact {n} {
> >                                       if {$n} {
> >                                               expr {$n * [fact [expr {$n - 1}]]}
> >                                       } {
> >                                               return 1
> >                                       }
> >                               }
> >
> > Hopefully this gives you a bit of a feel for the basic differences.
>
> Well, I know TCL, and there has been a zillion of reasons to move
> _from_ that language many, many years ago.  Intoducing this as new
> "shell" language in U-Boot does not look attractive to me.
> Actually, it gives me the creeps.

I was no fan of TCL either but this does seem like a reasonable idea to me.

Obviously whoever wrote it was very concerned about compiler
performance as there seem to be no comments?

Reviewed-by: Simon Glass <sjg@chromium.org>

Regards,
Simon
Tom Rini July 5, 2021, 7:10 p.m. UTC | #7
On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
> Dear Sean,
> 
> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> >
> > > For a partial list, see
> > > 
> > > [1] https://github.com/Forty-Bot/lil/commits/master
> >
> > Whoops, looks like I completely misread what you were asking here. I
> > don't have an exhaustive list of differences, but here are some similar
> > things expressed in both languages:
> >
> > sh				tcl
> >
> > foo=bar				set foo bar
> > echo $foo			echo $foo
> >
> > if [ 1 -gt 2 ]; then		if {1 > 2} {
> > 	echo a				echo a
> > else				} {
> > 	echo b				echo b
> > fi				}
> >
> > foo() {				proc foo {first second} {
> > 	echo $1 $2			echo $first $second
> > }				}
> >
> > for file in $(ls *.c); do	foreach file [glob *.c] {
> > 	echo $file			echo $file
> > done				}
> >
> > fact() {
> > 	if [ $1 -eq 0 ]; then
> > 		echo 1
> > 	else
> > 		echo $(($1 * $(fact $(($1 - 1)))))
> > 	fi
> > }
> >
> > 				proc fact {n} {
> > 					if {$n} {
> > 						expr {$n * [fact [expr {$n - 1}]]}
> > 					} {
> > 						return 1
> > 					}
> > 				}
> >
> > Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of
U-Boot?  That's important too.

> Well, I know TCL, and there has been a zillion of reasons to move
> _from_ that language many, many years ago.  Intoducing this as new
> "shell" language in U-Boot does not look attractive to me.
> Actually, it gives me the creeps.

This is I think the hard question.  A draw of the current shell is that
it it looks and acts like bash/sh/etc, for at least basic operations.
That's something that's comfortable to a large audience.  That has
disadvantages when people want to start doing something complex.  Sean
has shown that several times and he's not the only one.  LIL being
tcl'ish is not.

Something that has "sh" syntax but also clear to the user errors when
trying to do something not supported would also be interesting to see.
It seems like a lot of the frustration from users with our shell is that
it's not clear where the line between "this is an sh-like shell" and
"no, not like that" is.
Sean Anderson July 5, 2021, 7:47 p.m. UTC | #8
On 7/5/21 3:10 PM, Tom Rini wrote:
> On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
>> Dear Sean,
>>
>> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>>>
>>>> For a partial list, see
>>>>
>>>> [1] https://github.com/Forty-Bot/lil/commits/master
>>>
>>> Whoops, looks like I completely misread what you were asking here. I
>>> don't have an exhaustive list of differences, but here are some similar
>>> things expressed in both languages:
>>>
>>> sh				tcl
>>>
>>> foo=bar				set foo bar
>>> echo $foo			echo $foo
>>>
>>> if [ 1 -gt 2 ]; then		if {1 > 2} {
>>> 	echo a				echo a
>>> else				} {
>>> 	echo b				echo b
>>> fi				}

The left side is possible with something like

if itest 1 -gt 2; then # etc.

>>>
>>> foo() {				proc foo {first second} {
>>> 	echo $1 $2			echo $first $second
>>> }				}

This is not possible. We only have eval (run) as of today. I view adding
functions as one of the most important usability improvements we can
make.

>>>
>>> for file in $(ls *.c); do	foreach file [glob *.c] {
>>> 	echo $file			echo $file
>>> done				}

This is possible only if you already have a list of files. For example,
one could do

part list mmc 0 -bootable parts
for p in $parts; do #etc

but the part command is one of the only ones which produces output in
the correct format. If you want to (e.g.) dynamically construct a list
you will have a much harder time.

>>> fact() {
>>> 	if [ $1 -eq 0 ]; then
>>> 		echo 1
>>> 	else
>>> 		echo $(($1 * $(fact $(($1 - 1)))))
>>> 	fi
>>> }

This is technically possible with run and setexpr, but fairly cumbersome
to do.

>>>
>>> 				proc fact {n} {
>>> 					if {$n} {
>>> 						expr {$n * [fact [expr {$n - 1}]]}
>>> 					} {
>>> 						return 1
>>> 					}
>>> 				}
>>>
>>> Hopefully this gives you a bit of a feel for the basic differences.
> 
> Which of these things, from each column, can you do in the context of
> U-Boot?  That's important too.

See above.

--Sean
Tom Rini July 5, 2021, 7:53 p.m. UTC | #9
On Mon, Jul 05, 2021 at 03:47:47PM -0400, Sean Anderson wrote:
> On 7/5/21 3:10 PM, Tom Rini wrote:
> > On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
> > > Dear Sean,
> > > 
> > > In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> > > > 
> > > > > For a partial list, see
> > > > > 
> > > > > [1] https://github.com/Forty-Bot/lil/commits/master
> > > > 
> > > > Whoops, looks like I completely misread what you were asking here. I
> > > > don't have an exhaustive list of differences, but here are some similar
> > > > things expressed in both languages:
> > > > 
> > > > sh				tcl
> > > > 
> > > > foo=bar				set foo bar
> > > > echo $foo			echo $foo
> > > > 
> > > > if [ 1 -gt 2 ]; then		if {1 > 2} {
> > > > 	echo a				echo a
> > > > else				} {
> > > > 	echo b				echo b
> > > > fi				}
> 
> The left side is possible with something like
> 
> if itest 1 -gt 2; then # etc.
> 
> > > > 
> > > > foo() {				proc foo {first second} {
> > > > 	echo $1 $2			echo $first $second
> > > > }				}
> 
> This is not possible. We only have eval (run) as of today. I view adding
> functions as one of the most important usability improvements we can
> make.
> 
> > > > 
> > > > for file in $(ls *.c); do	foreach file [glob *.c] {
> > > > 	echo $file			echo $file
> > > > done				}
> 
> This is possible only if you already have a list of files. For example,
> one could do
> 
> part list mmc 0 -bootable parts
> for p in $parts; do #etc
> 
> but the part command is one of the only ones which produces output in
> the correct format. If you want to (e.g.) dynamically construct a list
> you will have a much harder time.
> 
> > > > fact() {
> > > > 	if [ $1 -eq 0 ]; then
> > > > 		echo 1
> > > > 	else
> > > > 		echo $(($1 * $(fact $(($1 - 1)))))
> > > > 	fi
> > > > }
> 
> This is technically possible with run and setexpr, but fairly cumbersome
> to do.
> 
> > > > 
> > > > 				proc fact {n} {
> > > > 					if {$n} {
> > > > 						expr {$n * [fact [expr {$n - 1}]]}
> > > > 					} {
> > > > 						return 1
> > > > 					}
> > > > 				}
> > > > 
> > > > Hopefully this gives you a bit of a feel for the basic differences.
> > 
> > Which of these things, from each column, can you do in the context of
> > U-Boot?  That's important too.
> 
> See above.

And for clarity, on the LIL side, with a few things like needing to
bring in the list of files somehow, all of those would work in U-Boot?
Sean Anderson July 5, 2021, 7:55 p.m. UTC | #10
On 7/5/21 3:53 PM, Tom Rini wrote:
> On Mon, Jul 05, 2021 at 03:47:47PM -0400, Sean Anderson wrote:
>> On 7/5/21 3:10 PM, Tom Rini wrote:
>>> On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
>>>> Dear Sean,
>>>>
>>>> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>>>>>
>>>>>> For a partial list, see
>>>>>>
>>>>>> [1] https://github.com/Forty-Bot/lil/commits/master
>>>>>
>>>>> Whoops, looks like I completely misread what you were asking here. I
>>>>> don't have an exhaustive list of differences, but here are some similar
>>>>> things expressed in both languages:
>>>>>
>>>>> sh				tcl
>>>>>
>>>>> foo=bar				set foo bar
>>>>> echo $foo			echo $foo
>>>>>
>>>>> if [ 1 -gt 2 ]; then		if {1 > 2} {
>>>>> 	echo a				echo a
>>>>> else				} {
>>>>> 	echo b				echo b
>>>>> fi				}
>>
>> The left side is possible with something like
>>
>> if itest 1 -gt 2; then # etc.
>>
>>>>>
>>>>> foo() {				proc foo {first second} {
>>>>> 	echo $1 $2			echo $first $second
>>>>> }				}
>>
>> This is not possible. We only have eval (run) as of today. I view adding
>> functions as one of the most important usability improvements we can
>> make.
>>
>>>>>
>>>>> for file in $(ls *.c); do	foreach file [glob *.c] {
>>>>> 	echo $file			echo $file
>>>>> done				}
>>
>> This is possible only if you already have a list of files. For example,
>> one could do
>>
>> part list mmc 0 -bootable parts
>> for p in $parts; do #etc
>>
>> but the part command is one of the only ones which produces output in
>> the correct format. If you want to (e.g.) dynamically construct a list
>> you will have a much harder time.
>>
>>>>> fact() {
>>>>> 	if [ $1 -eq 0 ]; then
>>>>> 		echo 1
>>>>> 	else
>>>>> 		echo $(($1 * $(fact $(($1 - 1)))))
>>>>> 	fi
>>>>> }
>>
>> This is technically possible with run and setexpr, but fairly cumbersome
>> to do.
>>
>>>>>
>>>>> 				proc fact {n} {
>>>>> 					if {$n} {
>>>>> 						expr {$n * [fact [expr {$n - 1}]]}
>>>>> 					} {
>>>>> 						return 1
>>>>> 					}
>>>>> 				}
>>>>>
>>>>> Hopefully this gives you a bit of a feel for the basic differences.
>>>
>>> Which of these things, from each column, can you do in the context of
>>> U-Boot?  That's important too.
>>
>> See above.
> 
> And for clarity, on the LIL side, with a few things like needing to
> bring in the list of files somehow, all of those would work in U-Boot?
> 

Correct. The only unimplemented function is "glob".

--Sean
Wolfgang Denk July 6, 2021, 7:44 a.m. UTC | #11
Dear Tom,

In message <20210705191058.GB9516@bill-the-cat> you wrote:
> 
> > > foo=bar				set foo bar
> > > echo $foo			echo $foo
> > >
> > > if [ 1 -gt 2 ]; then		if {1 > 2} {
> > > 	echo a				echo a
> > > else				} {
> > > 	echo b				echo b
> > > fi				}
> > >
> > > foo() {				proc foo {first second} {
> > > 	echo $1 $2			echo $first $second
> > > }				}
> > >
> > > for file in $(ls *.c); do	foreach file [glob *.c] {
> > > 	echo $file			echo $file
> > > done				}
> > >
> > > fact() {
> > > 	if [ $1 -eq 0 ]; then
> > > 		echo 1
> > > 	else
> > > 		echo $(($1 * $(fact $(($1 - 1)))))
> > > 	fi
> > > }
> > >
> > > 				proc fact {n} {
> > > 					if {$n} {
> > > 						expr {$n * [fact [expr {$n - 1}]]}
> > > 					} {
> > > 						return 1
> > > 					}
> > > 				}
> > >
> > > Hopefully this gives you a bit of a feel for the basic differences.
> 
> Which of these things, from each column, can you do in the context of
> U-Boot?  That's important too.

Well, with a current version of hush we can do:

-> foo=bar
-> echo $foo
bar

-> if [ 1 -gt 2 ]; then
>   echo a
> else
>   echo b
> fi
b

-> foo() {
>   echo $1 $2
> }
-> foo bar baz
bar baz

-> for file in $(ls *.c); do
> echo $file
> done
ls: cannot access '*.c': No such file or directory

-> fact() {
>   if [ $1 -eq 0 ]; then
>           echo 1
>   else
>           echo $(($1 * $(fact $(($1 - 1)))))
>   fi
> }
-> fact 4
24


Oh, in the contect of U-Boot?  Well, there are of course
limitations, but not because of the shell, but because of the fact
that we have no concept of files, for example.

But another command interpreter will not fix this.

> This is I think the hard question.  A draw of the current shell is that
> it it looks and acts like bash/sh/etc, for at least basic operations.
> That's something that's comfortable to a large audience.  That has
> disadvantages when people want to start doing something complex.  Sean
> has shown that several times and he's not the only one.  LIL being
> tcl'ish is not.

Tcl is a horror of a language for anything that is above trivial
level.

Do you really think that replacing standard shell syntax with Tcl is
"something that's comfortable to a large audience"?  I seriously
doubt that.

> Something that has "sh" syntax but also clear to the user errors when
> trying to do something not supported would also be interesting to see.
> It seems like a lot of the frustration from users with our shell is that
> it's not clear where the line between "this is an sh-like shell" and
> "no, not like that" is.

Did you run some tests on the version of hush as comes with recent
busybox releases?  Which of our user's requirements does it fail to
meet?


Best regards,

Wolfgang Denk
Wolfgang Denk July 6, 2021, 7:46 a.m. UTC | #12
Dear Sean,

In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
>
> >>> foo() {				proc foo {first second} {
> >>> 	echo $1 $2			echo $first $second
> >>> }				}
>
> This is not possible. We only have eval (run) as of today. I view adding
> functions as one of the most important usability improvements we can
> make.

Again:  this is not an issue with hush as is, but only with our
resource-limited port of a nearly 20 year old version of it.

Updating to a current version would fix this, in an almost 100%
backward compatible way.

Best regards,

Wolfgang Denk
Michael Nazzareno Trimarchi July 6, 2021, 7:52 a.m. UTC | #13
On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Sean,
>
> In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> >
> > >>> foo() {                           proc foo {first second} {
> > >>>   echo $1 $2                      echo $first $second
> > >>> }                         }
> >
> > This is not possible. We only have eval (run) as of today. I view adding
> > functions as one of the most important usability improvements we can
> > make.
>
> Again:  this is not an issue with hush as is, but only with our
> resource-limited port of a nearly 20 year old version of it.
>
> Updating to a current version would fix this, in an almost 100%
> backward compatible way.

Agree on this. There is another email about the LIL and how is right now
maintained and if the solutions that we have already in place can be just
updated, I don't find any valid reason to change it but update
seems more straightforward.

Michael

>
> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> If you don't have time to do it right, when will you have time to  do
> it over?                                                - John Wooden
Tom Rini July 6, 2021, 2:54 p.m. UTC | #14
On Tue, Jul 06, 2021 at 09:46:43AM +0200, Wolfgang Denk wrote:
> Dear Sean,
> 
> In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> >
> > >>> foo() {				proc foo {first second} {
> > >>> 	echo $1 $2			echo $first $second
> > >>> }				}
> >
> > This is not possible. We only have eval (run) as of today. I view adding
> > functions as one of the most important usability improvements we can
> > make.
> 
> Again:  this is not an issue with hush as is, but only with our
> resource-limited port of a nearly 20 year old version of it.
> 
> Updating to a current version would fix this, in an almost 100%
> backward compatible way.

Let us cut to the chase then.  Who is going to port a modern version of
hush over to U-Boot, and maintain it?  If we fork and forget again,
we'll be in a bad place once again in 2-3 years.
Simon Glass July 6, 2021, 2:57 p.m. UTC | #15
Hi Michael,

On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
<michael@amarulasolutions.com> wrote:
>
> On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> >
> > Dear Sean,
> >
> > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > >
> > > >>> foo() {                           proc foo {first second} {
> > > >>>   echo $1 $2                      echo $first $second
> > > >>> }                         }
> > >
> > > This is not possible. We only have eval (run) as of today. I view adding
> > > functions as one of the most important usability improvements we can
> > > make.
> >
> > Again:  this is not an issue with hush as is, but only with our
> > resource-limited port of a nearly 20 year old version of it.
> >
> > Updating to a current version would fix this, in an almost 100%
> > backward compatible way.
>
> Agree on this. There is another email about the LIL and how is right now
> maintained and if the solutions that we have already in place can be just
> updated, I don't find any valid reason to change it but update
> seems more straightforward.

Would you be interested in taking this on? It has been talked about
for some years...

Regards,
Simon
Tom Rini July 6, 2021, 3:43 p.m. UTC | #16
On Tue, Jul 06, 2021 at 09:44:20AM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210705191058.GB9516@bill-the-cat> you wrote:
> > 
> > > > foo=bar				set foo bar
> > > > echo $foo			echo $foo
> > > >
> > > > if [ 1 -gt 2 ]; then		if {1 > 2} {
> > > > 	echo a				echo a
> > > > else				} {
> > > > 	echo b				echo b
> > > > fi				}
> > > >
> > > > foo() {				proc foo {first second} {
> > > > 	echo $1 $2			echo $first $second
> > > > }				}
> > > >
> > > > for file in $(ls *.c); do	foreach file [glob *.c] {
> > > > 	echo $file			echo $file
> > > > done				}
> > > >
> > > > fact() {
> > > > 	if [ $1 -eq 0 ]; then
> > > > 		echo 1
> > > > 	else
> > > > 		echo $(($1 * $(fact $(($1 - 1)))))
> > > > 	fi
> > > > }
> > > >
> > > > 				proc fact {n} {
> > > > 					if {$n} {
> > > > 						expr {$n * [fact [expr {$n - 1}]]}
> > > > 					} {
> > > > 						return 1
> > > > 					}
> > > > 				}
> > > >
> > > > Hopefully this gives you a bit of a feel for the basic differences.
> > 
> > Which of these things, from each column, can you do in the context of
> > U-Boot?  That's important too.
> 
> Well, with a current version of hush we can do:
> 
> -> foo=bar
> -> echo $foo
> bar
> 
> -> if [ 1 -gt 2 ]; then
> >   echo a
> > else
> >   echo b
> > fi
> b
> 
> -> foo() {
> >   echo $1 $2
> > }
> -> foo bar baz
> bar baz
> 
> -> for file in $(ls *.c); do
> > echo $file
> > done
> ls: cannot access '*.c': No such file or directory
> 
> -> fact() {
> >   if [ $1 -eq 0 ]; then
> >           echo 1
> >   else
> >           echo $(($1 * $(fact $(($1 - 1)))))
> >   fi
> > }
> -> fact 4
> 24
> 
> 
> Oh, in the contect of U-Boot?  Well, there are of course
> limitations, but not because of the shell, but because of the fact
> that we have no concept of files, for example.
> 
> But another command interpreter will not fix this.

Yes, clearly the file based examples won't work either way, as-is.  I
was asking for what things can be done today with the implementations we
have now.

I'm pretty confident that exactly zero people have written complex
U-Boot scripts and then been happy about the experience.

> > This is I think the hard question.  A draw of the current shell is that
> > it it looks and acts like bash/sh/etc, for at least basic operations.
> > That's something that's comfortable to a large audience.  That has
> > disadvantages when people want to start doing something complex.  Sean
> > has shown that several times and he's not the only one.  LIL being
> > tcl'ish is not.
> 
> Tcl is a horror of a language for anything that is above trivial
> level.

TCL has its fans.  csh has it's fans.  The question isn't what's the
best desktop shell or general scripting language, but what's the most
useful in our environment an use cases.

> Do you really think that replacing standard shell syntax with Tcl is
> "something that's comfortable to a large audience"?  I seriously
> doubt that.

I don't know if it's right either.  But drawing on my comment just now
and above about complex boot scripts, I also don't know if "it's sh but
quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
TCL?  I don't know that, let me hit stackoverflow and do a little
reading" as would be the common experience.  Especially if we document
up-front what the quirks we have are.

> > Something that has "sh" syntax but also clear to the user errors when
> > trying to do something not supported would also be interesting to see.
> > It seems like a lot of the frustration from users with our shell is that
> > it's not clear where the line between "this is an sh-like shell" and
> > "no, not like that" is.
> 
> Did you run some tests on the version of hush as comes with recent
> busybox releases?  Which of our user's requirements does it fail to
> meet?

It fails to meet the requirement of having been ported to U-Boot.  I'd
joke that perhaps you can bootefi busybox, but a quick search says
that particular trick looks to still just be for python
(https://lwn.net/Articles/641244/).
Tom Rini July 6, 2021, 3:48 p.m. UTC | #17
On Tue, Jul 06, 2021 at 08:57:23AM -0600, Simon Glass wrote:
> Hi Michael,
> 
> On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
> <michael@amarulasolutions.com> wrote:
> >
> > On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> > >
> > > Dear Sean,
> > >
> > > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > > >
> > > > >>> foo() {                           proc foo {first second} {
> > > > >>>   echo $1 $2                      echo $first $second
> > > > >>> }                         }
> > > >
> > > > This is not possible. We only have eval (run) as of today. I view adding
> > > > functions as one of the most important usability improvements we can
> > > > make.
> > >
> > > Again:  this is not an issue with hush as is, but only with our
> > > resource-limited port of a nearly 20 year old version of it.
> > >
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Agree on this. There is another email about the LIL and how is right now
> > maintained and if the solutions that we have already in place can be just
> > updated, I don't find any valid reason to change it but update
> > seems more straightforward.
> 
> Would you be interested in taking this on? It has been talked about
> for some years...

To repeat myself, someone posting the patches to update (or more likely,
replace as a new Kconfig option and I'd be fine with a "pick old or new"
as a starting point) is the main gating factor on using a new version of
hush.
Kostas Michalopoulos July 6, 2021, 4:09 p.m. UTC | #18
On 7/6/2021 6:43 PM, Tom Rini wrote:
> I don't know if it's right either.  But drawing on my comment just now
> and above about complex boot scripts, I also don't know if "it's sh but
> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> TCL?  I don't know that, let me hit stackoverflow and do a little
> reading" as would be the common experience.  Especially if we document
> up-front what the quirks we have are.

I think the same would happen with Tcl and LIL. LIL might look similar 
to Tcl (and it is inspired by it), but it doesn't have the same syntax. 
A big difference comes from how variables are parsed with LIL allowing 
more variables symbols than Tcl without requiring escaping which affects 
some uses like expr, so e.g. in Tcl this

     set a 10 ; set b 20 ; expr $a+$b

will give back 30 however in LIL will give back 20. This is because Tcl 
parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b", 
evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b" 
as 20 and then just runs expr with the single argument "20".

Kostas
Wolfgang Denk July 7, 2021, 8:15 a.m. UTC | #19
Dear Tom,

In message <20210706145420.GQ9516@bill-the-cat> you wrote:
> 
> > Updating to a current version would fix this, in an almost 100%
> > backward compatible way.
>
> Let us cut to the chase then.  Who is going to port a modern version of
> hush over to U-Boot, and maintain it?  If we fork and forget again,
> we'll be in a bad place once again in 2-3 years.

Would we really be better off if we switch to some exotic piece of
code instead (I was not able to locate any user base, nor other
developers), which has been reported to have poor or no error
handling, and comes with an incompatible command line interface?

There is a zillion of shell scripts in the field, from non-trivial
boot sequences to complex download-and-upgrade scripts. You can't
really even think of breaking compatibility on such a level.

Best regards,

Wolfgang Denk
Wolfgang Denk July 7, 2021, 8:15 a.m. UTC | #20
Dear Tom,

In message <20210706154346.GT9516@bill-the-cat> you wrote:
> 
> I'm pretty confident that exactly zero people have written complex
> U-Boot scripts and then been happy about the experience.

I have seen many U-Boot scripts which were pretty complex, but
working absolutely reliably.

> TCL has its fans.  csh has it's fans.  The question isn't what's the
> best desktop shell or general scripting language, but what's the most
> useful in our environment an use cases.

Maybe you should try and do a poll of our user base which CLI they
_want_?  I doubt there will be any significant percentage voting for
Tcl.

I know of a large number of systems which offer a shell interface on
their command line, and those who don't usually use completely
proprietary code.  I know of very few examples where Tcl is being
used.


> I don't know if it's right either.  But drawing on my comment just now
> and above about complex boot scripts, I also don't know if "it's sh but
> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> TCL?  I don't know that, let me hit stackoverflow and do a little
> reading" as would be the common experience.  Especially if we document
> up-front what the quirks we have are.


Point taken.  But if you think this to an end, the result is: lets
write some documentation and explain the limitations of a shell in
U-Boot environment, and document the warts and bugs of this (or an
updated) version of hush.  This should make more users happy than
completely new and incompatible stuff.


Frankly, I believe when you run into problems with hush in U-Boot
(even the current version) you should lean back and think about what
you are doing.

U-Boot is a boot loader, and while it is powerful enough to do
complex things, this is not necessarily the most clever approach.
15 years ago, I've written complex update scripts for U-Boot. This
was easy enough to do, and worked perfectly.  But there are so many
limitations in a boot loader environment.  We don't do this any
more.  Instead, we use an OS suitable for such tasks (Linux with
SWUpdate).


And talking about problems and limitations in U-Boot... Is the CLI
really our biggest concern right now?  None of our users (customers)
has asked for a better command interpreter - the question we hear
are more like: "When will you support IPv6?", "NFS does not work
with recent Linux distros, will this be fixed?", "Can I download
over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
U-Boot for security-critical environments?", etc.

Best regards,

Wolfgang Denk
Michael Nazzareno Trimarchi July 7, 2021, 8:22 a.m. UTC | #21
Hi

On Tue, Jul 6, 2021 at 4:57 PM Simon Glass <sjg@chromium.org> wrote:
>
> Hi Michael,
>
> On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
> <michael@amarulasolutions.com> wrote:
> >
> > On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> > >
> > > Dear Sean,
> > >
> > > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > > >
> > > > >>> foo() {                           proc foo {first second} {
> > > > >>>   echo $1 $2                      echo $first $second
> > > > >>> }                         }
> > > >
> > > > This is not possible. We only have eval (run) as of today. I view adding
> > > > functions as one of the most important usability improvements we can
> > > > make.
> > >
> > > Again:  this is not an issue with hush as is, but only with our
> > > resource-limited port of a nearly 20 year old version of it.
> > >
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Agree on this. There is another email about the LIL and how is right now
> > maintained and if the solutions that we have already in place can be just
> > updated, I don't find any valid reason to change it but update
> > seems more straightforward.
>
> Would you be interested in taking this on? It has been talked about
> for some years...
>
> Regards,
> Simon

I will ask to someone in the company to do as task. Francis can you do it?

Michael
Sean Anderson July 7, 2021, 1:32 p.m. UTC | #22
On 7/6/21 12:09 PM, Kostas Michalopoulos wrote:
> On 7/6/2021 6:43 PM, Tom Rini wrote:
>> I don't know if it's right either.  But drawing on my comment just now
>> and above about complex boot scripts, I also don't know if "it's sh but
>> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
>> TCL?  I don't know that, let me hit stackoverflow and do a little
>> reading" as would be the common experience.  Especially if we document
>> up-front what the quirks we have are.
> 
> I think the same would happen with Tcl and LIL. LIL might look similar
> to Tcl (and it is inspired by it), but it doesn't have the same
> syntax. A big difference comes from how variables are parsed with LIL
> allowing more variables symbols than Tcl without requiring escaping
> which affects some uses like expr, so e.g. in Tcl this
> 
>      set a 10 ; set b 20 ; expr $a+$b
> 
> will give back 30 however in LIL will give back 20. This is because
> Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+"
> "$b", evaluates "$a+" as an empty string (there isn't any "a+"
> variable), "$b" as 20 and then just runs expr with the single argument
> "20".

IMO both of these are nuts. I think the best thing to do here is
to raise an error about a missing variable "a+" to force the user to
rewrite this as

	set a 10 ; set b 20 ; expr ${a}+$b

removing the ambiguity. Although I would generally like to follow TCL's
lead, there are several places where TCL does crazy things (e.g.
comments) which I have purposely not replicated.

--Sean
Sean Anderson July 7, 2021, 1:46 p.m. UTC | #23
On 7/7/21 4:15 AM, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210706154346.GT9516@bill-the-cat> you wrote:
>>
>> I'm pretty confident that exactly zero people have written complex
>> U-Boot scripts and then been happy about the experience.
> 
> I have seen many U-Boot scripts which were pretty complex, but
> working absolutely reliably.
> 
>> TCL has its fans.  csh has it's fans.  The question isn't what's the
>> best desktop shell or general scripting language, but what's the most
>> useful in our environment an use cases.
> 
> Maybe you should try and do a poll of our user base which CLI they
> _want_?  I doubt there will be any significant percentage voting for
> Tcl.
>
> I know of a large number of systems which offer a shell interface on
> their command line, and those who don't usually use completely
> proprietary code.  I know of very few examples where Tcl is being
> used.

Off the top of my head, most of the tooling for FPGAs uses TCL for
scripting. OpenOCD uses it too.

>> I don't know if it's right either.  But drawing on my comment just now
>> and above about complex boot scripts, I also don't know if "it's sh but
>> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
>> TCL?  I don't know that, let me hit stackoverflow and do a little
>> reading" as would be the common experience.  Especially if we document
>> up-front what the quirks we have are.
> 
> 
> Point taken.  But if you think this to an end, the result is: lets
> write some documentation and explain the limitations of a shell in
> U-Boot environment, and document the warts and bugs of this (or an
> updated) version of hush.  This should make more users happy than
> completely new and incompatible stuff.
> 
> 
> Frankly, I believe when you run into problems with hush in U-Boot
> (even the current version) you should lean back and think about what
> you are doing.
> 
> U-Boot is a boot loader, and while it is powerful enough to do
> complex things, this is not necessarily the most clever approach.
> 15 years ago, I've written complex update scripts for U-Boot. This
> was easy enough to do, and worked perfectly.  But there are so many
> limitations in a boot loader environment.  We don't do this any
> more.  Instead, we use an OS suitable for such tasks (Linux with
> SWUpdate).
> 
> 
> And talking about problems and limitations in U-Boot... Is the CLI
> really our biggest concern right now?  None of our users (customers)
> has asked for a better command interpreter - the question we hear
> are more like: "When will you support IPv6?", "NFS does not work
> with recent Linux distros, will this be fixed?", "Can I download
> over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
> U-Boot for security-critical environments?", etc.

I wanted a better shell, so I worked on it.

--Sean
Tom Rini July 7, 2021, 1:51 p.m. UTC | #24
On Wed, Jul 07, 2021 at 09:46:20AM -0400, Sean Anderson wrote:
> 
> On 7/7/21 4:15 AM, Wolfgang Denk wrote:
> > Dear Tom,
> > 
> > In message <20210706154346.GT9516@bill-the-cat> you wrote:
> > > 
> > > I'm pretty confident that exactly zero people have written complex
> > > U-Boot scripts and then been happy about the experience.
> > 
> > I have seen many U-Boot scripts which were pretty complex, but
> > working absolutely reliably.
> > 
> > > TCL has its fans.  csh has it's fans.  The question isn't what's the
> > > best desktop shell or general scripting language, but what's the most
> > > useful in our environment an use cases.
> > 
> > Maybe you should try and do a poll of our user base which CLI they
> > _want_?  I doubt there will be any significant percentage voting for
> > Tcl.
> > 
> > I know of a large number of systems which offer a shell interface on
> > their command line, and those who don't usually use completely
> > proprietary code.  I know of very few examples where Tcl is being
> > used.
> 
> Off the top of my head, most of the tooling for FPGAs uses TCL for
> scripting. OpenOCD uses it too.
> 
> > > I don't know if it's right either.  But drawing on my comment just now
> > > and above about complex boot scripts, I also don't know if "it's sh but
> > > quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> > > TCL?  I don't know that, let me hit stackoverflow and do a little
> > > reading" as would be the common experience.  Especially if we document
> > > up-front what the quirks we have are.
> > 
> > 
> > Point taken.  But if you think this to an end, the result is: lets
> > write some documentation and explain the limitations of a shell in
> > U-Boot environment, and document the warts and bugs of this (or an
> > updated) version of hush.  This should make more users happy than
> > completely new and incompatible stuff.
> > 
> > 
> > Frankly, I believe when you run into problems with hush in U-Boot
> > (even the current version) you should lean back and think about what
> > you are doing.
> > 
> > U-Boot is a boot loader, and while it is powerful enough to do
> > complex things, this is not necessarily the most clever approach.
> > 15 years ago, I've written complex update scripts for U-Boot. This
> > was easy enough to do, and worked perfectly.  But there are so many
> > limitations in a boot loader environment.  We don't do this any
> > more.  Instead, we use an OS suitable for such tasks (Linux with
> > SWUpdate).
> > 
> > 
> > And talking about problems and limitations in U-Boot... Is the CLI
> > really our biggest concern right now?  None of our users (customers)
> > has asked for a better command interpreter - the question we hear
> > are more like: "When will you support IPv6?", "NFS does not work
> > with recent Linux distros, will this be fixed?", "Can I download
> > over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
> > U-Boot for security-critical environments?", etc.
> 
> I wanted a better shell, so I worked on it.

This here is also an important point, and why I'm commenting on the
series.  A developer sees a problem, and works on the problem.  I know I
don't comment on as much stuff as I should, but for wide reaching
patches, I really really try to.
Tom Rini July 7, 2021, 1:58 p.m. UTC | #25
On Wed, Jul 07, 2021 at 10:15:32AM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210706145420.GQ9516@bill-the-cat> you wrote:
> > 
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Let us cut to the chase then.  Who is going to port a modern version of
> > hush over to U-Boot, and maintain it?  If we fork and forget again,
> > we'll be in a bad place once again in 2-3 years.
> 
> Would we really be better off if we switch to some exotic piece of
> code instead (I was not able to locate any user base, nor other
> developers), which has been reported to have poor or no error
> handling, and comes with an incompatible command line interface?
> 
> There is a zillion of shell scripts in the field, from non-trivial
> boot sequences to complex download-and-upgrade scripts. You can't
> really even think of breaking compatibility on such a level.

As I've said a few times in this thread, this not being an sh-style
interpreter is a strike against it.  And if we're going to insist on a
bug-for-bug upgrade to our hush (so that all of the hugely complex
existing scripts keep working) we might as well not upgrade.  Frankly I
suspect that down the line IF a new cli interpreter comes in to U-Boot
we will have to keep the old one around as a "use this instead" option
for another long number of years, so that if there are any systems with
non-trivial scripts but upgrade U-Boot and don't / won't / can't
re-validate their entire sequence, they can just use the old cli.
Tom Rini July 7, 2021, 1:58 p.m. UTC | #26
On Wed, Jul 07, 2021 at 10:15:34AM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210706154346.GT9516@bill-the-cat> you wrote:
> > 
> > I'm pretty confident that exactly zero people have written complex
> > U-Boot scripts and then been happy about the experience.
> 
> I have seen many U-Boot scripts which were pretty complex, but
> working absolutely reliably.

As have I.  We have them in-tree, even.  My point is not "you cannot
write these".  My point is "you come away very disappointed in the
interpreter" after writing these.  It's certainly not bash/zsh.  But
it's also not POSIX-sh.  It's not modern hush.  It's unique, slightly
different from everything else people normally have easy access to and
those quirks aren't documented really.

> > TCL has its fans.  csh has it's fans.  The question isn't what's the
> > best desktop shell or general scripting language, but what's the most
> > useful in our environment an use cases.
> 
> Maybe you should try and do a poll of our user base which CLI they
> _want_?  I doubt there will be any significant percentage voting for
> Tcl.
> 
> I know of a large number of systems which offer a shell interface on
> their command line, and those who don't usually use completely
> proprietary code.  I know of very few examples where Tcl is being
> used.

A good point, yes.

> > I don't know if it's right either.  But drawing on my comment just now
> > and above about complex boot scripts, I also don't know if "it's sh but
> > quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> > TCL?  I don't know that, let me hit stackoverflow and do a little
> > reading" as would be the common experience.  Especially if we document
> > up-front what the quirks we have are.
> 
> Point taken.  But if you think this to an end, the result is: lets
> write some documentation and explain the limitations of a shell in
> U-Boot environment, and document the warts and bugs of this (or an
> updated) version of hush.  This should make more users happy than
> completely new and incompatible stuff.

That would be much appreciated.  And to reiterate, our current hush
isn't going away soon, or likely entirely within the decade.  So this
would be something long term useful to have written.

> Frankly, I believe when you run into problems with hush in U-Boot
> (even the current version) you should lean back and think about what
> you are doing.

Agreed.

> U-Boot is a boot loader, and while it is powerful enough to do
> complex things, this is not necessarily the most clever approach.
> 15 years ago, I've written complex update scripts for U-Boot. This
> was easy enough to do, and worked perfectly.  But there are so many
> limitations in a boot loader environment.  We don't do this any
> more.  Instead, we use an OS suitable for such tasks (Linux with
> SWUpdate).

Yes.  Although there are some complex things that we can't push up
higher (see distro_bootcmd) and update systems still tend to need
something from us (or they do it in grub instead).
Wolfgang Denk July 7, 2021, 2:10 p.m. UTC | #27
Dear Tom,

In message <20210707135839.GW9516@bill-the-cat> you wrote:
> 
> As I've said a few times in this thread, this not being an sh-style
> interpreter is a strike against it.  And if we're going to insist on a
> bug-for-bug upgrade to our hush (so that all of the hugely complex
> existing scripts keep working) we might as well not upgrade.  Frankly I
> suspect that down the line IF a new cli interpreter comes in to U-Boot
> we will have to keep the old one around as a "use this instead" option
> for another long number of years, so that if there are any systems with
> non-trivial scripts but upgrade U-Boot and don't / won't / can't
> re-validate their entire sequence, they can just use the old cli.

Do you actually have an example where code working on our ancient
port of hush would fail on the current upstream version?

Best regards,

Wolfgang Denk
Tom Rini July 7, 2021, 2:14 p.m. UTC | #28
On Wed, Jul 07, 2021 at 04:10:43PM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210707135839.GW9516@bill-the-cat> you wrote:
> > 
> > As I've said a few times in this thread, this not being an sh-style
> > interpreter is a strike against it.  And if we're going to insist on a
> > bug-for-bug upgrade to our hush (so that all of the hugely complex
> > existing scripts keep working) we might as well not upgrade.  Frankly I
> > suspect that down the line IF a new cli interpreter comes in to U-Boot
> > we will have to keep the old one around as a "use this instead" option
> > for another long number of years, so that if there are any systems with
> > non-trivial scripts but upgrade U-Boot and don't / won't / can't
> > re-validate their entire sequence, they can just use the old cli.
> 
> Do you actually have an example where code working on our ancient
> port of hush would fail on the current upstream version?

Have you validated one of those exceedingly complex boot scripts with a
modern hush (and some fakery for u-boot commands) ?  No.  I'm just
saying I expect there to be enough risk-adverse groups that just
dropping our old hush entirely might not be possible right away.  Of
course, if all of the current in-tree complex cases Just Work, that
might be a good argument against needing to keep such levels of
backwards compatibility.
Wolfgang Denk July 7, 2021, 2:23 p.m. UTC | #29
Dear Tom,

In message <20210707141418.GZ9516@bill-the-cat> you wrote:
> 
> Have you validated one of those exceedingly complex boot scripts with a
> modern hush (and some fakery for u-boot commands) ?  No.

No, I havent.

I also don't claim that I know all the warts and issues with our old
hus, but for the ones I am aware the woraround usually splitting
complex or nested expressions into a sequence of simpler steps.  I
cannot remember any cases where the resulting code should be
incompatible to a shell without that specific bug.

And yes, I am aware of the problems with the distro_bootcmd stuff.
Thisis exactly what I had in mind when I wrote: if you run into such
situations you should lean back and reflect a bit.  I can undrstand
the intentions of all this stuff, but the implementation is a
horrible mess.

> I'm just
> saying I expect there to be enough risk-adverse groups that just
> dropping our old hush entirely might not be possible right away.  Of
> course, if all of the current in-tree complex cases Just Work, that
> might be a good argument against needing to keep such levels of
> backwards compatibility.

There is only one way to test this.


Best regards,

Wolfgang Denk
Marek Behún July 7, 2021, 2:48 p.m. UTC | #30
Dear Tom, Sean, Wolfgang and others,

here are some of my opinions for this discussion

- I agree with Wolfgang that there are far better options than
  a Tcl-like shell, if we want to add another language

- I also think that instead of adding another language, it is more
  preferable to improve the existing one. Adding a new language will
  cause more problems in the future:
  - I think it can end up with OS distributions needing to write
    boot scripts in both languages, because they can't be sure which
    will be compiled into U-Boot
  - we will certainly end up with more bugs
  - userbase will fragment between the two languages

- I think we can start improving the current U-Boot's shell in ways
  that are incompatible with upstream Hush.

  The idea back then, as I understand it, was to minimize man-hours
  invested into the CLI code, and so an existing shell was incorporated
  (with many #ifdef guards). But U-Boot has since evolved so much that
  it is very probable it would be more economic to simply fork from
  upsteam Hush, remove all the #ifdefs and start developing features we
  want in U-Boot. Is upstream Hush even maintained properly?
  What is the upstream repository? Is it
  https://github.com/sheumann/hush?

- even if we decide to stay with upstream Hush and just upgrade
  U-Boot's Hush to upstream (since it supports functions, arithmetic
  with $((...)), command substitution with $(...), these are all nice
  features), it is IMO still better than adding a new language

- one of the points Sean mentioned with LIL is that when compiled, it's
  size does not exceed the size of U-Boot's Hush.

  If we were to add new features into U-Boot's Hush, the code size would
  certainly increase.

  I think we should implement these new features, and instead of adding
  a new language, we should work on minimizing the code size /
  resulting U-Boot image size. This is where U-Boot will gain most not
  only with it's CLI, but also everywhere else. Regarding this,
  - we already have LTO
  - Simon worked on dtoc so that devicetrees can be compiled into C code
  - we can start playing with compression
    - either we can compress the whole image for machines with enough
      RAM but small place for U-Boot (Nokia N900 for example has only
      256 KiB space for U-Boot)
    - or we can try to invent a way to decompress code when it is
      needed, for machines with small RAM

Marek
Sean Anderson July 8, 2021, 4:56 a.m. UTC | #31
On 7/6/21 3:44 AM, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20210705191058.GB9516@bill-the-cat> you wrote:
>> This is I think the hard question.  A draw of the current shell is that
>> it it looks and acts like bash/sh/etc, for at least basic operations.
>> That's something that's comfortable to a large audience.  That has
>> disadvantages when people want to start doing something complex.  Sean
>> has shown that several times and he's not the only one.  LIL being
>> tcl'ish is not.
> 
> Tcl is a horror of a language for anything that is above trivial
> level.

Can you please elaborate on this? You've made this claim several times,
but haven't explained your reasoning for it.

--Sean
Michael Nazzareno Trimarchi July 8, 2021, 5:19 a.m. UTC | #32
Hi

On Wed, Jul 7, 2021 at 4:48 PM Marek Behun <marek.behun@nic.cz> wrote:
>
> Dear Tom, Sean, Wolfgang and others,
>
> here are some of my opinions for this discussion
>
> - I agree with Wolfgang that there are far better options than
>   a Tcl-like shell, if we want to add another language
>
> - I also think that instead of adding another language, it is more
>   preferable to improve the existing one. Adding a new language will
>   cause more problems in the future:
>   - I think it can end up with OS distributions needing to write
>     boot scripts in both languages, because they can't be sure which
>     will be compiled into U-Boot
>   - we will certainly end up with more bugs
>   - userbase will fragment between the two languages
>
> - I think we can start improving the current U-Boot's shell in ways
>   that are incompatible with upstream Hush.
>
>   The idea back then, as I understand it, was to minimize man-hours
>   invested into the CLI code, and so an existing shell was incorporated
>   (with many #ifdef guards). But U-Boot has since evolved so much that
>   it is very probable it would be more economic to simply fork from
>   upsteam Hush, remove all the #ifdefs and start developing features we
>   want in U-Boot. Is upstream Hush even maintained properly?
>   What is the upstream repository? Is it
>   https://github.com/sheumann/hush?
>

I think that hush is the one that is now in the busybox. I could spent
ten minutes this morning and this is my short list:

- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there

I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially
solve some of the problems described in the thread

* Sean *: You have spent more on this, can you please complete it.

Out of that. Do we have some script shell unit test in uboot?

Michael

> - even if we decide to stay with upstream Hush and just upgrade
>   U-Boot's Hush to upstream (since it supports functions, arithmetic
>   with $((...)), command substitution with $(...), these are all nice
>   features), it is IMO still better than adding a new language
>
> - one of the points Sean mentioned with LIL is that when compiled, it's
>   size does not exceed the size of U-Boot's Hush.
>
>   If we were to add new features into U-Boot's Hush, the code size would
>   certainly increase.
>
>   I think we should implement these new features, and instead of adding
>   a new language, we should work on minimizing the code size /
>   resulting U-Boot image size. This is where U-Boot will gain most not
>   only with it's CLI, but also everywhere else. Regarding this,
>   - we already have LTO
>   - Simon worked on dtoc so that devicetrees can be compiled into C code
>   - we can start playing with compression
>     - either we can compress the whole image for machines with enough
>       RAM but small place for U-Boot (Nokia N900 for example has only
>       256 KiB space for U-Boot)
>     - or we can try to invent a way to decompress code when it is
>       needed, for machines with small RAM
>
> Marek
Tom Rini July 8, 2021, 3:33 p.m. UTC | #33
On Thu, Jul 08, 2021 at 07:19:05AM +0200, Michael Nazzareno Trimarchi wrote:
> Hi
> 
> On Wed, Jul 7, 2021 at 4:48 PM Marek Behun <marek.behun@nic.cz> wrote:
> >
> > Dear Tom, Sean, Wolfgang and others,
> >
> > here are some of my opinions for this discussion
> >
> > - I agree with Wolfgang that there are far better options than
> >   a Tcl-like shell, if we want to add another language
> >
> > - I also think that instead of adding another language, it is more
> >   preferable to improve the existing one. Adding a new language will
> >   cause more problems in the future:
> >   - I think it can end up with OS distributions needing to write
> >     boot scripts in both languages, because they can't be sure which
> >     will be compiled into U-Boot
> >   - we will certainly end up with more bugs
> >   - userbase will fragment between the two languages
> >
> > - I think we can start improving the current U-Boot's shell in ways
> >   that are incompatible with upstream Hush.
> >
> >   The idea back then, as I understand it, was to minimize man-hours
> >   invested into the CLI code, and so an existing shell was incorporated
> >   (with many #ifdef guards). But U-Boot has since evolved so much that
> >   it is very probable it would be more economic to simply fork from
> >   upsteam Hush, remove all the #ifdefs and start developing features we
> >   want in U-Boot. Is upstream Hush even maintained properly?
> >   What is the upstream repository? Is it
> >   https://github.com/sheumann/hush?
> >
> 
> I think that hush is the one that is now in the busybox. I could spent
> ten minutes this morning and this is my short list:
> 
> - we have several define that allow it to enabled e/o disable a lot of features
> - we are talking about 11K lines compared to 3K (including comment)
> - we have 25-30 configuration option on hush on busybox
> - in u-boot code some of the problem was solved some time ago
> - as describe is 68Kb, I think this consider all the option enables
> - the code is different from what we have and what is there
> 
> I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially
> solve some of the problems described in the thread
> 
> * Sean *: You have spent more on this, can you please complete it.
> 
> Out of that. Do we have some script shell unit test in uboot?

To the last point, yes, we have some, but always need more, tests for
how hush behaves as part of pytest.
Wolfgang Denk July 8, 2021, 5 p.m. UTC | #34
Dear Sean,

In message <43880bf0-baa6-0cb2-80fe-0fe50d43b42f@gmail.com> you wrote:
>
> > Tcl is a horror of a language for anything that is above trivial
> > level.
>
> Can you please elaborate on this? You've made this claim several times,
> but haven't explained your reasoning for it.

A long, long time ago I I was looking for an efficient way to do
regression testing for embedded systems, mainly U-Boot and Linux.  I
found GNU gnats to bee too difficult to handle und started writing
my own test system (you can still find this ancient stuff here [1])
based on expect/Tcl.  At that time, I could not find any other
solution, so I had to accept Tcl as programming language for this
tool.   And at the beginning it usually worked just fine.  Mostly.
Coming from a C programming environment I always found Tcl ...
strange.  For example it's scoping rules (see [2] for an example).
Things got worse when the system grew and we had to deal with
strings or parameters which might contail quotes or other special
characters.  This required fancy quoting, which depended of the
calling depth - if there was only on specific level. you might find
a solution after some (cumbersome) experimenting.  But if the same
string was used in several places in the code, and different levels,
you really lost.  At a certain point all our developers refused to
write new test cases for the system, because it always was the same
nightmare of fighting unforeseeable language problems.

That was when we abandoned DUTS, and some time later Heiko came up
with an initial implementation of tbot, soon to be joined by
Harald to make a production quality test system out of it [4].


IMO, Tcl has a large number of situations where it's actual
behaviours is not what you expect from it - it is good enough for
simple things, but any more complex processing of arbitrary text
data quickly drives you insane - things like quotes or braces and
other special characters.


[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] https://wiki.tcl-lang.org/page/Commonly+encountered+problems+in+Tcl
[3] https://wiki.tcl-lang.org/page/Why+can+I+not+place+unmatched+braces+in+Tcl+comments
[4] https://tbot.tools/

Best regards,

Wolfgang Denk
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index 2d267aeff2..0184de5f93 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -766,6 +766,12 @@  S:	Maintained
 T:	git https://source.denx.de/u-boot/custodians/u-boot-i2c.git
 F:	drivers/i2c/
 
+LIL SHELL
+M:	Sean Anderson <seanga2@gmail.com>
+S:	Maintained
+F:	common/cli_lil.c
+F:	include/cli_lil.h
+
 LOGGING
 M:	Simon Glass <sjg@chromium.org>
 S:	Maintained
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 9e8b69258f..8bccc572af 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -23,6 +23,15 @@  config HUSH_PARSER
 	  If disabled, you get the old, much simpler behaviour with a somewhat
 	  smaller memory footprint.
 
+config LIL
+	bool "Use LIL shell"
+	depends on CMDLINE
+	help
+	  This options enables the "Little Interpreted Language" (LIL) shell as
+	  command line interpreter, thus enabling powerful command line syntax
+	  like `proc name {args} {body}' functions or `echo [some command]`
+	  command substitution ("tcl scripts").
+
 config CMDLINE_EDITING
 	bool "Enable command line editing"
 	depends on CMDLINE
@@ -1379,7 +1388,7 @@  config CMD_ECHO
 
 config CMD_ITEST
 	bool "itest"
-	default y
+	default y if !LIL
 	help
 	  Return true/false on integer compare.
 
diff --git a/common/Makefile b/common/Makefile
index 829ea5fb42..dce04b305e 100644
--- a/common/Makefile
+++ b/common/Makefile
@@ -10,6 +10,7 @@  obj-y += main.o
 obj-y += exports.o
 obj-$(CONFIG_HASH) += hash.o
 obj-$(CONFIG_HUSH_PARSER) += cli_hush.o
+obj-$(CONFIG_LIL) += cli_lil.o
 obj-$(CONFIG_AUTOBOOT) += autoboot.o
 
 # This option is not just y/n - it can have a numeric value
diff --git a/common/cli_lil.c b/common/cli_lil.c
new file mode 100644
index 0000000000..c8c6986fe1
--- /dev/null
+++ b/common/cli_lil.c
@@ -0,0 +1,2991 @@ 
+// SPDX-License-Identifier: GPL-2.0+ AND Zlib
+/*
+ * LIL - Little Interpreted Language
+ * Copyright (C) 2021 Sean Anderson <seanga2@gmail.com>
+ * Copyright (C) 2010-2013 Kostas Michalopoulos <badsector@runtimelegend.com>
+ *
+ * This file originated from the LIL project, licensed under Zlib. All
+ * modifications are licensed under GPL-2.0+
+ */
+
+#include <common.h>
+#include <cli_lil.h>
+#include <ctype.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+/* Enable pools for reusing values, lists and environments. This will use more memory and
+ * will rely on the runtime/OS to free the pools once the program ends, but will cause
+ * considerably less memory fragmentation and improve the script execution performance. */
+/*#define LIL_ENABLE_POOLS*/
+
+/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack
+ * overflows and is also useful when running through an automated fuzzer like AFL */
+/*#define LIL_ENABLE_RECLIMIT 10000*/
+
+#define ERROR_NOERROR 0
+#define ERROR_DEFAULT 1
+#define ERROR_FIXHEAD 2
+
+#define CALLBACKS 8
+#define HASHMAP_CELLS 256
+#define HASHMAP_CELLMASK 0xFF
+
+struct hashentry {
+	char *k;
+	void *v;
+};
+
+struct hashcell {
+	struct hashentry *e;
+	size_t c;
+};
+
+struct hashmap {
+	struct hashcell cell[HASHMAP_CELLS];
+};
+
+struct lil_value {
+	size_t l;
+#ifdef LIL_ENABLE_POOLS
+	size_t c;
+#endif
+	char *d;
+};
+
+struct lil_var {
+	char *n;
+	struct lil_env *env;
+	struct lil_value *v;
+};
+
+struct lil_env {
+	struct lil_env *parent;
+	struct lil_func *func;
+	struct lil_var **var;
+	size_t vars;
+	struct hashmap varmap;
+	struct lil_value *retval;
+	int retval_set;
+	int breakrun;
+};
+
+struct lil_list {
+	struct lil_value **v;
+	size_t c;
+	size_t cap;
+};
+
+struct lil_func {
+	char *name;
+	struct lil_value *code;
+	struct lil_list *argnames;
+	lil_func_proc_t proc;
+};
+
+struct lil {
+	const char *code; /* need save on parse */
+	const char *rootcode;
+	size_t clen; /* need save on parse */
+	size_t head; /* need save on parse */
+	int ignoreeol;
+	struct lil_func **cmd;
+	size_t cmds;
+	struct hashmap cmdmap;
+	struct lil_env *env;
+	struct lil_env *rootenv;
+	struct lil_env *downenv;
+	struct lil_value *empty;
+	int error;
+	size_t err_head;
+	char *err_msg;
+	lil_callback_proc_t callback[CALLBACKS];
+	size_t parse_depth;
+};
+
+struct expreval {
+	const char *code;
+	size_t len, head;
+	ssize_t ival;
+	int error;
+};
+
+static struct lil_value *next_word(struct lil *lil);
+static void register_stdcmds(struct lil *lil);
+
+#ifdef LIL_ENABLE_POOLS
+static struct lil_value **pool;
+static int poolsize, poolcap;
+static struct lil_list **listpool;
+static size_t listpoolsize, listpoolcap;
+static struct lil_env **envpool;
+static size_t envpoolsize, envpoolcap;
+#endif
+
+static char *strclone(const char *s)
+{
+	size_t len = strlen(s) + 1;
+	char *ns = malloc(len);
+
+	if (!ns)
+		return NULL;
+	memcpy(ns, s, len);
+	return ns;
+}
+
+static unsigned long hm_hash(const char *key)
+{
+	unsigned long hash = 5381;
+	int c;
+
+	while ((c = *key++))
+		hash = ((hash << 5) + hash) + c;
+	return hash;
+}
+
+static void hm_init(struct hashmap *hm)
+{
+	memset(hm, 0, sizeof(struct hashmap));
+}
+
+static void hm_destroy(struct hashmap *hm)
+{
+	size_t i, j;
+
+	for (i = 0; i < HASHMAP_CELLS; i++) {
+		for (j = 0; j < hm->cell[i].c; j++)
+			free(hm->cell[i].e[j].k);
+		free(hm->cell[i].e);
+	}
+}
+
+static void hm_put(struct hashmap *hm, const char *key, void *value)
+{
+	struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+	size_t i;
+
+	for (i = 0; i < cell->c; i++) {
+		if (!strcmp(key, cell->e[i].k)) {
+			cell->e[i].v = value;
+			return;
+		}
+	}
+
+	cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
+	cell->e[cell->c].k = strclone(key);
+	cell->e[cell->c].v = value;
+	cell->c++;
+}
+
+static void *hm_get(struct hashmap *hm, const char *key)
+{
+	struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+	size_t i;
+
+	for (i = 0; i < cell->c; i++)
+		if (!strcmp(key, cell->e[i].k))
+			return cell->e[i].v;
+	return NULL;
+}
+
+static int hm_has(struct hashmap *hm, const char *key)
+{
+	struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+	size_t i;
+
+	for (i = 0; i < cell->c; i++)
+		if (!strcmp(key, cell->e[i].k))
+			return 1;
+	return 0;
+}
+
+#ifdef LIL_ENABLE_POOLS
+static struct lil_value *alloc_from_pool(void)
+{
+	if (poolsize > 0) {
+		poolsize--;
+		return pool[poolsize];
+	} else {
+		struct lil_value *val = calloc(1, sizeof(struct lil_value));
+
+		return val;
+	}
+}
+
+static void release_to_pool(struct lil_value *val)
+{
+	if (poolsize == poolcap) {
+		poolcap = poolcap ? (poolcap + poolcap / 2) : 64;
+		pool = realloc(pool, sizeof(struct lil_value *) * poolcap);
+	}
+	pool[poolsize++] = val;
+}
+
+static void ensure_capacity(struct lil_value *val, size_t cap)
+{
+	if (val->c < cap) {
+		val->c = cap + 128;
+		val->d = realloc(val->d, val->c);
+	}
+}
+#endif
+
+static struct lil_value *alloc_value_len(const char *str, size_t len)
+{
+#ifdef LIL_ENABLE_POOLS
+	struct lil_value *val = alloc_from_pool();
+#else
+	struct lil_value *val = calloc(1, sizeof(struct lil_value));
+#endif
+
+	if (!val)
+		return NULL;
+	if (str) {
+		val->l = len;
+#ifdef LIL_ENABLE_POOLS
+		ensure_capacity(val, len + 1);
+#else
+		val->d = malloc(len + 1);
+		if (!val->d) {
+			free(val);
+			return NULL;
+		}
+#endif
+		memcpy(val->d, str, len);
+		val->d[len] = 0;
+	} else {
+		val->l = 0;
+#ifdef LIL_ENABLE_POOLS
+		ensure_capacity(val, 1);
+		val->d[0] = '\0';
+#else
+		val->d = NULL;
+#endif
+	}
+	return val;
+}
+
+static struct lil_value *alloc_value(const char *str)
+{
+	return alloc_value_len(str, str ? strlen(str) : 0);
+}
+
+struct lil_value *lil_clone_value(struct lil_value *src)
+{
+	struct lil_value *val;
+
+	if (!src)
+		return NULL;
+#ifdef LIL_ENABLE_POOLS
+	val = alloc_from_pool();
+#else
+	val = calloc(1, sizeof(struct lil_value));
+#endif
+	if (!val)
+		return NULL;
+
+	val->l = src->l;
+	if (src->l) {
+#ifdef LIL_ENABLE_POOLS
+		ensure_capacity(val, val->l + 1);
+#else
+		val->d = malloc(val->l + 1);
+		if (!val->d) {
+			free(val);
+			return NULL;
+		}
+#endif
+		memcpy(val->d, src->d, val->l + 1);
+	} else {
+#ifdef LIL_ENABLE_POOLS
+		ensure_capacity(val, 1);
+		val->d[0] = '\0';
+#else
+		val->d = NULL;
+#endif
+	}
+	return val;
+}
+
+int lil_append_char(struct lil_value *val, char ch)
+{
+#ifdef LIL_ENABLE_POOLS
+	ensure_capacity(val, val->l + 2);
+	val->d[val->l++] = ch;
+	val->d[val->l] = '\0';
+#else
+	char *new = realloc(val->d, val->l + 2);
+
+	if (!new)
+		return 0;
+
+	new[val->l++] = ch;
+	new[val->l] = 0;
+	val->d = new;
+#endif
+	return 1;
+}
+
+int lil_append_string_len(struct lil_value *val, const char *s, size_t len)
+{
+#ifndef LIL_ENABLE_POOLS
+	char *new;
+#endif
+
+	if (!s || !s[0])
+		return 1;
+
+#ifdef LIL_ENABLE_POOLS
+	ensure_capacity(val, val->l + len + 1);
+	memcpy(val->d + val->l, s, len + 1);
+#else
+	new = realloc(val->d, val->l + len + 1);
+	if (!new)
+		return 0;
+
+	memcpy(new + val->l, s, len + 1);
+	val->d = new;
+#endif
+	val->l += len;
+	return 1;
+}
+
+int lil_append_string(struct lil_value *val, const char *s)
+{
+	return lil_append_string_len(val, s, strlen(s));
+}
+
+int lil_append_val(struct lil_value *val, struct lil_value *v)
+{
+#ifndef LIL_ENABLE_POOLS
+	char *new;
+#endif
+
+	if (!v || !v->l)
+		return 1;
+
+#ifdef LIL_ENABLE_POOLS
+	ensure_capacity(val, val->l + v->l + 1);
+	memcpy(val->d + val->l, v->d, v->l + 1);
+#else
+	new = realloc(val->d, val->l + v->l + 1);
+	if (!new)
+		return 0;
+
+	memcpy(new + val->l, v->d, v->l + 1);
+	val->d = new;
+#endif
+	val->l += v->l;
+	return 1;
+}
+
+void lil_free_value(struct lil_value *val)
+{
+	if (!val)
+		return;
+
+#ifdef LIL_ENABLE_POOLS
+	release_to_pool(val);
+#else
+	free(val->d);
+	free(val);
+#endif
+}
+
+struct lil_list *lil_alloc_list(void)
+{
+	struct lil_list *list;
+
+#ifdef LIL_ENABLE_POOLS
+	if (listpoolsize > 0)
+		return listpool[--listpoolsize];
+#endif
+	list = calloc(1, sizeof(struct lil_list));
+	list->v = NULL;
+	return list;
+}
+
+void lil_free_list(struct lil_list *list)
+{
+	size_t i;
+
+	if (!list)
+		return;
+
+	for (i = 0; i < list->c; i++)
+		lil_free_value(list->v[i]);
+
+#ifdef LIL_ENABLE_POOLS
+	list->c = 0;
+	if (listpoolsize == listpoolcap) {
+		listpoolcap =
+			listpoolcap ? (listpoolcap + listpoolcap / 2) : 32;
+		listpool = realloc(listpool,
+				   sizeof(struct lil_list *) * listpoolcap);
+	}
+	listpool[listpoolsize++] = list;
+#else
+	free(list->v);
+	free(list);
+#endif
+}
+
+void lil_list_append(struct lil_list *list, struct lil_value *val)
+{
+	if (list->c == list->cap) {
+		size_t cap = list->cap ? (list->cap + list->cap / 2) : 32;
+		struct lil_value **nv =
+			realloc(list->v, sizeof(struct lil_value *) * cap);
+
+		if (!nv)
+			return;
+
+		list->cap = cap;
+		list->v = nv;
+	}
+	list->v[list->c++] = val;
+}
+
+size_t lil_list_size(struct lil_list *list)
+{
+	return list->c;
+}
+
+struct lil_value *lil_list_get(struct lil_list *list, size_t index)
+{
+	return index >= list->c ? NULL : list->v[index];
+}
+
+static int needs_escape(const char *str)
+{
+	size_t i;
+
+	if (!str || !str[0])
+		return 1;
+
+	for (i = 0; str[i]; i++)
+		if (ispunct(str[i]) || isspace(str[i]))
+			return 1;
+
+	return 0;
+}
+
+struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)
+{
+	struct lil_value *val = alloc_value(NULL);
+	size_t i, j;
+
+	for (i = 0; i < list->c; i++) {
+		int escape =
+			do_escape ? needs_escape(lil_to_string(list->v[i])) : 0;
+
+		if (i)
+			lil_append_char(val, ' ');
+
+		if (escape) {
+			lil_append_char(val, '{');
+			for (j = 0; j < list->v[i]->l; j++) {
+				if (list->v[i]->d[j] == '{')
+					lil_append_string(val, "}\"\\o\"{");
+				else if (list->v[i]->d[j] == '}')
+					lil_append_string(val, "}\"\\c\"{");
+				else
+					lil_append_char(val, list->v[i]->d[j]);
+			}
+			lil_append_char(val, '}');
+		} else {
+			lil_append_val(val, list->v[i]);
+		}
+	}
+	return val;
+}
+
+struct lil_env *lil_alloc_env(struct lil_env *parent)
+{
+	struct lil_env *env;
+
+#ifdef LIL_ENABLE_POOLS
+	if (envpoolsize > 0) {
+		size_t i, j;
+
+		env = envpool[--envpoolsize];
+		env->parent = parent;
+		env->func = NULL;
+		env->var = NULL;
+		env->vars = 0;
+		env->retval = NULL;
+		env->retval_set = 0;
+		env->breakrun = 0;
+		for (i = 0; i < HASHMAP_CELLS; i++) {
+			for (j = 0; j < env->varmap.cell[i].c; j++)
+				free(env->varmap.cell[i].e[j].k);
+			env->varmap.cell[i].c = 0;
+		}
+		return env;
+	}
+#endif
+	env = calloc(1, sizeof(struct lil_env));
+	env->parent = parent;
+	return env;
+}
+
+void lil_free_env(struct lil_env *env)
+{
+	size_t i;
+
+	if (!env)
+		return;
+
+	lil_free_value(env->retval);
+#ifdef LIL_ENABLE_POOLS
+	for (i = 0; i < env->vars; i++) {
+		free(env->var[i]->n);
+		lil_free_value(env->var[i]->v);
+		free(env->var[i]);
+	}
+	free(env->var);
+
+	if (envpoolsize == envpoolcap) {
+		envpoolcap = envpoolcap ? (envpoolcap + envpoolcap / 2) : 64;
+		envpool =
+			realloc(envpool, sizeof(struct lil_env *) * envpoolcap);
+	}
+	envpool[envpoolsize++] = env;
+#else
+	hm_destroy(&env->varmap);
+	for (i = 0; i < env->vars; i++) {
+		free(env->var[i]->n);
+		lil_free_value(env->var[i]->v);
+		free(env->var[i]);
+	}
+	free(env->var);
+	free(env);
+#endif
+}
+
+static struct lil_var *lil_find_local_var(struct lil *lil, struct lil_env *env,
+					  const char *name)
+{
+	return hm_get(&env->varmap, name);
+}
+
+static struct lil_var *lil_find_var(struct lil *lil, struct lil_env *env,
+				    const char *name)
+{
+	struct lil_var *r = lil_find_local_var(lil, env, name);
+
+	if (r)
+		return r;
+
+	if (env == lil->rootenv)
+		return NULL;
+
+	return lil_find_var(lil, lil->rootenv, name);
+}
+
+static struct lil_func *lil_find_cmd(struct lil *lil, const char *name)
+{
+	return hm_get(&lil->cmdmap, name);
+}
+
+static struct lil_func *add_func(struct lil *lil, const char *name)
+{
+	struct lil_func *cmd;
+	struct lil_func **ncmd;
+
+	cmd = lil_find_cmd(lil, name);
+	if (cmd) {
+		if (cmd->argnames)
+			lil_free_list(cmd->argnames);
+		lil_free_value(cmd->code);
+		cmd->argnames = NULL;
+		cmd->code = NULL;
+		cmd->proc = NULL;
+		return cmd;
+	}
+
+	cmd = calloc(1, sizeof(struct lil_func));
+	cmd->name = strclone(name);
+
+	ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
+	if (!ncmd) {
+		free(cmd);
+		return NULL;
+	}
+
+	lil->cmd = ncmd;
+	ncmd[lil->cmds++] = cmd;
+	hm_put(&lil->cmdmap, name, cmd);
+	return cmd;
+}
+
+static void del_func(struct lil *lil, struct lil_func *cmd)
+{
+	size_t i, index = lil->cmds;
+
+	for (i = 0; i < lil->cmds; i++) {
+		if (lil->cmd[i] == cmd) {
+			index = i;
+			break;
+		}
+	}
+	if (index == lil->cmds)
+		return;
+
+	hm_put(&lil->cmdmap, cmd->name, 0);
+	if (cmd->argnames)
+		lil_free_list(cmd->argnames);
+
+	lil_free_value(cmd->code);
+	free(cmd->name);
+	free(cmd);
+	lil->cmds--;
+	for (i = index; i < lil->cmds; i++)
+		lil->cmd[i] = lil->cmd[i + 1];
+}
+
+int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
+{
+	struct lil_func *cmd = add_func(lil, name);
+
+	if (!cmd)
+		return 0;
+	cmd->proc = proc;
+	return 1;
+}
+
+struct lil_var *lil_set_var(struct lil *lil, const char *name,
+			    struct lil_value *val, int local)
+{
+	struct lil_var **nvar;
+	struct lil_env *env =
+		local == LIL_SETVAR_GLOBAL ? lil->rootenv : lil->env;
+	int freeval = 0;
+
+	if (!name[0])
+		return NULL;
+
+	if (local != LIL_SETVAR_LOCAL_NEW) {
+		struct lil_var *var = lil_find_var(lil, env, name);
+
+		if (local == LIL_SETVAR_LOCAL_ONLY && var &&
+		    var->env == lil->rootenv && var->env != env)
+			var = NULL;
+
+		if (((!var && env == lil->rootenv) ||
+		     (var && var->env == lil->rootenv)) &&
+		    lil->callback[LIL_CALLBACK_SETVAR]) {
+			lil_setvar_callback_proc_t proc =
+				(lil_setvar_callback_proc_t)
+					lil->callback[LIL_CALLBACK_SETVAR];
+			struct lil_value *newval = val;
+			int r = proc(lil, name, &newval);
+
+			if (r < 0) {
+				return NULL;
+			} else if (r > 0) {
+				val = newval;
+				freeval = 1;
+			}
+		}
+
+		if (var) {
+			lil_free_value(var->v);
+			var->v = freeval ? val : lil_clone_value(val);
+			return var;
+		}
+	}
+
+	nvar = realloc(env->var, sizeof(struct lil_var *) * (env->vars + 1));
+	if (!nvar) {
+		/* TODO: report memory error */
+		return NULL;
+	}
+
+	env->var = nvar;
+	nvar[env->vars] = calloc(1, sizeof(struct lil_var));
+	nvar[env->vars]->n = strclone(name);
+	nvar[env->vars]->env = env;
+	nvar[env->vars]->v = freeval ? val : lil_clone_value(val);
+	hm_put(&env->varmap, name, nvar[env->vars]);
+	return nvar[env->vars++];
+}
+
+struct lil_value *lil_get_var(struct lil *lil, const char *name)
+{
+	return lil_get_var_or(lil, name, lil->empty);
+}
+
+struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
+				 struct lil_value *defvalue)
+{
+	struct lil_var *var = lil_find_var(lil, lil->env, name);
+	struct lil_value *retval = var ? var->v : defvalue;
+
+	if (lil->callback[LIL_CALLBACK_GETVAR] &&
+	    (!var || var->env == lil->rootenv)) {
+		lil_getvar_callback_proc_t proc =
+			(lil_getvar_callback_proc_t)
+				lil->callback[LIL_CALLBACK_GETVAR];
+		struct lil_value *newretval = retval;
+
+		if (proc(lil, name, &newretval))
+			retval = newretval;
+	}
+	return retval;
+}
+
+struct lil_env *lil_push_env(struct lil *lil)
+{
+	struct lil_env *env = lil_alloc_env(lil->env);
+
+	lil->env = env;
+	return env;
+}
+
+void lil_pop_env(struct lil *lil)
+{
+	if (lil->env->parent) {
+		struct lil_env *next = lil->env->parent;
+
+		lil_free_env(lil->env);
+		lil->env = next;
+	}
+}
+
+struct lil *lil_new(void)
+{
+	struct lil *lil = calloc(1, sizeof(struct lil));
+
+	lil->rootenv = lil->env = lil_alloc_env(NULL);
+	lil->empty = alloc_value(NULL);
+	hm_init(&lil->cmdmap);
+	register_stdcmds(lil);
+	return lil;
+}
+
+static int islilspecial(char ch)
+{
+	return ch == '$' || ch == '{' || ch == '}' || ch == '[' || ch == ']' ||
+	       ch == '"' || ch == '\'' || ch == ';';
+}
+
+static int eolchar(char ch)
+{
+	return ch == '\n' || ch == '\r' || ch == ';';
+}
+
+static int ateol(struct lil *lil)
+{
+	return !(lil->ignoreeol) && eolchar(lil->code[lil->head]);
+}
+
+static void lil_skip_spaces(struct lil *lil)
+{
+	while (lil->head < lil->clen) {
+		if (lil->code[lil->head] == '#') {
+			if (lil->code[lil->head + 1] == '#' &&
+			    lil->code[lil->head + 2] != '#') {
+				lil->head += 2;
+				while (lil->head < lil->clen) {
+					if ((lil->code[lil->head] == '#') &&
+					    (lil->code[lil->head + 1] == '#') &&
+					    (lil->code[lil->head + 2] != '#')) {
+						lil->head += 2;
+						break;
+					}
+					lil->head++;
+				}
+			} else {
+				while (lil->head < lil->clen &&
+				       !eolchar(lil->code[lil->head]))
+					lil->head++;
+			}
+		} else if (lil->code[lil->head] == '\\' &&
+			   eolchar(lil->code[lil->head + 1])) {
+			lil->head++;
+			while (lil->head < lil->clen &&
+			       eolchar(lil->code[lil->head]))
+				lil->head++;
+		} else if (eolchar(lil->code[lil->head])) {
+			if (lil->ignoreeol)
+				lil->head++;
+			else
+				break;
+		} else if (isspace(lil->code[lil->head])) {
+			lil->head++;
+		} else {
+			break;
+		}
+	}
+}
+
+static struct lil_value *get_bracketpart(struct lil *lil)
+{
+	size_t cnt = 1;
+	struct lil_value *val, *cmd = alloc_value(NULL);
+	int save_eol = lil->ignoreeol;
+
+	lil->ignoreeol = 0;
+	lil->head++;
+	while (lil->head < lil->clen) {
+		if (lil->code[lil->head] == '[') {
+			lil->head++;
+			cnt++;
+			lil_append_char(cmd, '[');
+		} else if (lil->code[lil->head] == ']') {
+			lil->head++;
+			if (--cnt == 0)
+				break;
+			else
+				lil_append_char(cmd, ']');
+		} else {
+			lil_append_char(cmd, lil->code[lil->head++]);
+		}
+	}
+
+	val = lil_parse_value(lil, cmd, 0);
+	lil_free_value(cmd);
+	lil->ignoreeol = save_eol;
+	return val;
+}
+
+static struct lil_value *get_dollarpart(struct lil *lil)
+{
+	struct lil_value *val, *name, *tmp;
+
+	lil->head++;
+	name = next_word(lil);
+	tmp = alloc_value("set ");
+	lil_append_val(tmp, name);
+	lil_free_value(name);
+
+	val = lil_parse_value(lil, tmp, 0);
+	lil_free_value(tmp);
+	return val;
+}
+
+static struct lil_value *next_word(struct lil *lil)
+{
+	struct lil_value *val;
+	size_t start;
+
+	lil_skip_spaces(lil);
+	if (lil->code[lil->head] == '$') {
+		val = get_dollarpart(lil);
+	} else if (lil->code[lil->head] == '{') {
+		size_t cnt = 1;
+
+		lil->head++;
+		val = alloc_value(NULL);
+		while (lil->head < lil->clen) {
+			if (lil->code[lil->head] == '{') {
+				lil->head++;
+				cnt++;
+				lil_append_char(val, '{');
+			} else if (lil->code[lil->head] == '}') {
+				lil->head++;
+				if (--cnt == 0)
+					break;
+				else
+					lil_append_char(val, '}');
+			} else {
+				lil_append_char(val, lil->code[lil->head++]);
+			}
+		}
+	} else if (lil->code[lil->head] == '[') {
+		val = get_bracketpart(lil);
+	} else if (lil->code[lil->head] == '"' ||
+		   lil->code[lil->head] == '\'') {
+		char sc = lil->code[lil->head++];
+
+		val = alloc_value(NULL);
+		while (lil->head < lil->clen) {
+			if (lil->code[lil->head] == '[' ||
+			    lil->code[lil->head] == '$') {
+				struct lil_value *tmp =
+					lil->code[lil->head] == '$' ?
+						      get_dollarpart(lil) :
+						      get_bracketpart(lil);
+
+				lil_append_val(val, tmp);
+				lil_free_value(tmp);
+				lil->head--; /* avoid skipping the char below */
+			} else if (lil->code[lil->head] == '\\') {
+				lil->head++;
+				switch (lil->code[lil->head]) {
+				case 'b':
+					lil_append_char(val, '\b');
+					break;
+				case 't':
+					lil_append_char(val, '\t');
+					break;
+				case 'n':
+					lil_append_char(val, '\n');
+					break;
+				case 'v':
+					lil_append_char(val, '\v');
+					break;
+				case 'f':
+					lil_append_char(val, '\f');
+					break;
+				case 'r':
+					lil_append_char(val, '\r');
+					break;
+				case '0':
+					lil_append_char(val, 0);
+					break;
+				case 'a':
+					lil_append_char(val, '\a');
+					break;
+				case 'c':
+					lil_append_char(val, '}');
+					break;
+				case 'o':
+					lil_append_char(val, '{');
+					break;
+				default:
+					lil_append_char(val,
+							lil->code[lil->head]);
+					break;
+				}
+			} else if (lil->code[lil->head] == sc) {
+				lil->head++;
+				break;
+			} else {
+				lil_append_char(val, lil->code[lil->head]);
+			}
+			lil->head++;
+		}
+	} else {
+		start = lil->head;
+		while (lil->head < lil->clen &&
+		       !isspace(lil->code[lil->head]) &&
+		       !islilspecial(lil->code[lil->head]))
+			lil->head++;
+		val = alloc_value_len(lil->code + start, lil->head - start);
+	}
+	return val ? val : alloc_value(NULL);
+}
+
+static struct lil_list *substitute(struct lil *lil)
+{
+	struct lil_list *words = lil_alloc_list();
+
+	lil_skip_spaces(lil);
+	while (lil->head < lil->clen && !ateol(lil) && !lil->error) {
+		struct lil_value *w = alloc_value(NULL);
+
+		do {
+			size_t head = lil->head;
+			struct lil_value *wp = next_word(lil);
+
+			if (head ==
+			    lil->head) { /* something wrong, the parser can't proceed */
+				lil_free_value(w);
+				lil_free_value(wp);
+				lil_free_list(words);
+				return NULL;
+			}
+
+			lil_append_val(w, wp);
+			lil_free_value(wp);
+		} while (lil->head < lil->clen &&
+			 !eolchar(lil->code[lil->head]) &&
+			 !isspace(lil->code[lil->head]) && !lil->error);
+		lil_skip_spaces(lil);
+
+		lil_list_append(words, w);
+	}
+
+	return words;
+}
+
+struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code)
+{
+	const char *save_code = lil->code;
+	size_t save_clen = lil->clen;
+	size_t save_head = lil->head;
+	int save_igeol = lil->ignoreeol;
+	struct lil_list *words;
+
+	lil->code = lil_to_string(code);
+	lil->clen = code->l;
+	lil->head = 0;
+	lil->ignoreeol = 1;
+
+	words = substitute(lil);
+	if (!words)
+		words = lil_alloc_list();
+
+	lil->code = save_code;
+	lil->clen = save_clen;
+	lil->head = save_head;
+	lil->ignoreeol = save_igeol;
+	return words;
+}
+
+struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code)
+{
+	struct lil_list *words = lil_subst_to_list(lil, code);
+	struct lil_value *val;
+
+	val = lil_list_to_value(words, 0);
+	lil_free_list(words);
+	return val;
+}
+
+static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
+				 struct lil_list *words)
+{
+	struct lil_value *r;
+
+	if (cmd->proc) {
+		size_t shead = lil->head;
+
+		r = cmd->proc(lil, words->c - 1, words->v + 1);
+		if (lil->error == ERROR_FIXHEAD) {
+			lil->error = ERROR_DEFAULT;
+			lil->err_head = shead;
+		}
+	} else {
+		lil_push_env(lil);
+		lil->env->func = cmd;
+
+		if (cmd->argnames->c == 1 &&
+		    !strcmp(lil_to_string(cmd->argnames->v[0]), "args")) {
+			struct lil_value *args = lil_list_to_value(words, 1);
+
+			lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
+			lil_free_value(args);
+		} else {
+			size_t i;
+
+			for (i = 0; i < cmd->argnames->c; i++) {
+				struct lil_value *val;
+
+				if (i < words->c - 1)
+					val = words->v[i + 1];
+				else
+					val = lil->empty;
+
+				lil_set_var(lil,
+					    lil_to_string(cmd->argnames->v[i]),
+					    val, LIL_SETVAR_LOCAL_NEW);
+			}
+		}
+		r = lil_parse_value(lil, cmd->code, 1);
+
+		lil_pop_env(lil);
+	}
+
+	return r;
+}
+
+struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
+			    int funclevel)
+{
+	const char *save_code = lil->code;
+	size_t save_clen = lil->clen;
+	size_t save_head = lil->head;
+	struct lil_value *val = NULL;
+	struct lil_list *words = NULL;
+
+	if (!save_code)
+		lil->rootcode = code;
+	lil->code = code;
+	lil->clen = codelen ? codelen : strlen(code);
+	lil->head = 0;
+
+	lil_skip_spaces(lil);
+	lil->parse_depth++;
+#ifdef LIL_ENABLE_RECLIMIT
+	if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
+		lil_set_error(lil, "Too many recursive calls");
+		goto cleanup;
+	}
+#endif
+
+	if (lil->parse_depth == 1)
+		lil->error = 0;
+
+	if (funclevel)
+		lil->env->breakrun = 0;
+
+	while (lil->head < lil->clen && !lil->error) {
+		if (words)
+			lil_free_list(words);
+
+		if (val)
+			lil_free_value(val);
+		val = NULL;
+
+		words = substitute(lil);
+		if (!words || lil->error)
+			goto cleanup;
+
+		if (words->c) {
+			struct lil_func *cmd =
+				lil_find_cmd(lil, lil_to_string(words->v[0]));
+
+			if (!cmd) {
+				if (words->v[0]->l) {
+					char msg[64];
+
+					snprintf(msg, sizeof(msg),
+						 "unknown function %s",
+						 words->v[0]->d);
+					lil_set_error_at(lil, lil->head, msg);
+					goto cleanup;
+				}
+			} else {
+				val = run_cmd(lil, cmd, words);
+			}
+
+			if (lil->env->breakrun)
+				goto cleanup;
+		}
+
+		lil_skip_spaces(lil);
+		while (ateol(lil))
+			lil->head++;
+		lil_skip_spaces(lil);
+	}
+
+cleanup:
+	if (lil->error && lil->callback[LIL_CALLBACK_ERROR] &&
+	    lil->parse_depth == 1) {
+		lil_error_callback_proc_t proc =
+			(lil_error_callback_proc_t)
+				lil->callback[LIL_CALLBACK_ERROR];
+
+		proc(lil, lil->err_head, lil->err_msg);
+	}
+
+	if (words)
+		lil_free_list(words);
+	lil->code = save_code;
+	lil->clen = save_clen;
+	lil->head = save_head;
+
+	if (funclevel && lil->env->retval_set) {
+		if (val)
+			lil_free_value(val);
+		val = lil->env->retval;
+		lil->env->retval = NULL;
+		lil->env->retval_set = 0;
+		lil->env->breakrun = 0;
+	}
+
+	lil->parse_depth--;
+	return val ? val : alloc_value(NULL);
+}
+
+struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
+				  int funclevel)
+{
+	if (!val || !val->d || !val->l)
+		return alloc_value(NULL);
+
+	return lil_parse(lil, val->d, val->l, funclevel);
+}
+
+void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc)
+{
+	if (cb < 0 || cb > CALLBACKS)
+		return;
+
+	lil->callback[cb] = proc;
+}
+
+void lil_set_error(struct lil *lil, const char *msg)
+{
+	if (lil->error)
+		return;
+
+	free(lil->err_msg);
+	lil->error = ERROR_FIXHEAD;
+	lil->err_head = 0;
+	lil->err_msg = strclone(msg ? msg : "");
+}
+
+void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
+{
+	if (lil->error)
+		return;
+
+	free(lil->err_msg);
+	lil->error = ERROR_DEFAULT;
+	lil->err_head = pos;
+	lil->err_msg = strclone(msg ? msg : "");
+}
+
+int lil_error(struct lil *lil, const char **msg, size_t *pos)
+{
+	if (!lil->error)
+		return 0;
+
+	*msg = lil->err_msg;
+	*pos = lil->err_head;
+	lil->error = ERROR_NOERROR;
+
+	return 1;
+}
+
+#define EERR_NO_ERROR 0
+#define EERR_SYNTAX_ERROR 1
+#define EERR_DIVISION_BY_ZERO 3
+#define EERR_INVALID_EXPRESSION 4
+
+static void ee_expr(struct expreval *ee);
+
+static int ee_invalidpunct(int ch)
+{
+	return ispunct(ch) && ch != '!' && ch != '~' && ch != '(' &&
+	       ch != ')' && ch != '-' && ch != '+';
+}
+
+static void ee_skip_spaces(struct expreval *ee)
+{
+	while (ee->head < ee->len && isspace(ee->code[ee->head]))
+		ee->head++;
+}
+
+static void ee_numeric_element(struct expreval *ee)
+{
+	ee_skip_spaces(ee);
+	ee->ival = 0;
+	while (ee->head < ee->len) {
+		if (!isdigit(ee->code[ee->head]))
+			break;
+
+		ee->ival = ee->ival * 10 + (ee->code[ee->head] - '0');
+		ee->head++;
+	}
+}
+
+static void ee_element(struct expreval *ee)
+{
+	if (isdigit(ee->code[ee->head])) {
+		ee_numeric_element(ee);
+		return;
+	}
+
+	/*
+	 * for anything else that might creep in (usually from strings), we set
+	 * the value to 1 so that strings evaluate as "true" when used in
+	 * conditional expressions
+	 */
+	ee->ival = 1;
+	ee->error = EERR_INVALID_EXPRESSION; /* special flag, will be cleared */
+}
+
+static void ee_paren(struct expreval *ee)
+{
+	ee_skip_spaces(ee);
+	if (ee->code[ee->head] == '(') {
+		ee->head++;
+		ee_expr(ee);
+		ee_skip_spaces(ee);
+
+		if (ee->code[ee->head] == ')')
+			ee->head++;
+		else
+			ee->error = EERR_SYNTAX_ERROR;
+	} else {
+		ee_element(ee);
+	}
+}
+
+static void ee_unary(struct expreval *ee)
+{
+	ee_skip_spaces(ee);
+	if (ee->head < ee->len && !ee->error &&
+	    (ee->code[ee->head] == '-' || ee->code[ee->head] == '+' ||
+	     ee->code[ee->head] == '~' || ee->code[ee->head] == '!')) {
+		char op = ee->code[ee->head++];
+
+		ee_unary(ee);
+		if (ee->error)
+			return;
+
+		switch (op) {
+		case '-':
+			ee->ival = -ee->ival;
+			break;
+		case '+':
+			/* ignore it, doesn't change a thing */
+			break;
+		case '~':
+			ee->ival = ~ee->ival;
+			break;
+		case '!':
+			ee->ival = !ee->ival;
+			break;
+		}
+	} else {
+		ee_paren(ee);
+	}
+}
+
+static void ee_muldiv(struct expreval *ee)
+{
+	ee_unary(ee);
+	if (ee->error)
+		return;
+
+	ee_skip_spaces(ee);
+	while (ee->head < ee->len && !ee->error &&
+	       !ee_invalidpunct(ee->code[ee->head + 1]) &&
+	       (ee->code[ee->head] == '*' || ee->code[ee->head] == '/' ||
+		ee->code[ee->head] == '\\' || ee->code[ee->head] == '%')) {
+		ssize_t oival = ee->ival;
+
+		switch (ee->code[ee->head]) {
+		case '*':
+			ee->head++;
+			ee_unary(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = ee->ival * oival;
+			break;
+		case '%':
+			ee->head++;
+			ee_unary(ee);
+			if (ee->error)
+				return;
+
+			if (ee->ival == 0)
+				ee->error = EERR_DIVISION_BY_ZERO;
+			else
+				ee->ival = oival % ee->ival;
+			break;
+		case '/':
+			ee->head++;
+			ee_unary(ee);
+			if (ee->error)
+				return;
+
+			if (ee->ival == 0)
+				ee->error = EERR_DIVISION_BY_ZERO;
+			else
+				ee->ival = oival / ee->ival;
+			break;
+		case '\\':
+			ee->head++;
+			ee_unary(ee);
+			if (ee->error)
+				return;
+
+			if (ee->ival == 0)
+				ee->error = EERR_DIVISION_BY_ZERO;
+			else
+				ee->ival = oival / ee->ival;
+			break;
+		}
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_addsub(struct expreval *ee)
+{
+	ee_muldiv(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       !ee_invalidpunct(ee->code[ee->head + 1]) &&
+	       (ee->code[ee->head] == '+' || ee->code[ee->head] == '-')) {
+		ssize_t oival = ee->ival;
+
+		switch (ee->code[ee->head]) {
+		case '+':
+			ee->head++;
+			ee_muldiv(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = ee->ival + oival;
+			break;
+		case '-':
+			ee->head++;
+			ee_muldiv(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = oival - ee->ival;
+			break;
+		}
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_shift(struct expreval *ee)
+{
+	ee_addsub(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       ((ee->code[ee->head] == '<' && ee->code[ee->head + 1] == '<') ||
+		(ee->code[ee->head] == '>' && ee->code[ee->head + 1] == '>'))) {
+		ssize_t oival = ee->ival;
+
+		ee->head++;
+		switch (ee->code[ee->head]) {
+		case '<':
+			ee->head++;
+			ee_addsub(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = oival << ee->ival;
+			break;
+		case '>':
+			ee->head++;
+			ee_addsub(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = oival >> ee->ival;
+			break;
+		}
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_compare(struct expreval *ee)
+{
+	ee_shift(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       ((ee->code[ee->head] == '<' &&
+		 !ee_invalidpunct(ee->code[ee->head + 1])) ||
+		(ee->code[ee->head] == '>' &&
+		 !ee_invalidpunct(ee->code[ee->head + 1])) ||
+		(ee->code[ee->head] == '<' && ee->code[ee->head + 1] == '=') ||
+		(ee->code[ee->head] == '>' && ee->code[ee->head + 1] == '='))) {
+		ssize_t oival = ee->ival;
+		int op = 4;
+
+		if (ee->code[ee->head] == '<' &&
+		    !ee_invalidpunct(ee->code[ee->head + 1]))
+			op = 1;
+		else if (ee->code[ee->head] == '>' &&
+			 !ee_invalidpunct(ee->code[ee->head + 1]))
+			op = 2;
+		else if (ee->code[ee->head] == '<' &&
+			 ee->code[ee->head + 1] == '=')
+			op = 3;
+
+		ee->head += op > 2 ? 2 : 1;
+
+		switch (op) {
+		case 1:
+			ee_shift(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival < ee->ival) ? 1 : 0;
+			break;
+		case 2:
+			ee_shift(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival > ee->ival) ? 1 : 0;
+			break;
+		case 3:
+			ee_shift(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival <= ee->ival) ? 1 : 0;
+			break;
+		case 4:
+			ee_shift(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival >= ee->ival) ? 1 : 0;
+			break;
+		}
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_equals(struct expreval *ee)
+{
+	ee_compare(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       ((ee->code[ee->head] == '=' && ee->code[ee->head + 1] == '=') ||
+		(ee->code[ee->head] == '!' && ee->code[ee->head + 1] == '='))) {
+		ssize_t oival = ee->ival;
+		int op = ee->code[ee->head] == '=' ? 1 : 2;
+
+		ee->head += 2;
+
+		switch (op) {
+		case 1:
+			ee_compare(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival == ee->ival) ? 1 : 0;
+			break;
+		case 2:
+			ee_compare(ee);
+			if (ee->error)
+				return;
+
+			ee->ival = (oival != ee->ival) ? 1 : 0;
+			break;
+		}
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_bitand(struct expreval *ee)
+{
+	ee_equals(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       (ee->code[ee->head] == '&' &&
+		!ee_invalidpunct(ee->code[ee->head + 1]))) {
+		ssize_t oival = ee->ival;
+		ee->head++;
+
+		ee_equals(ee);
+		if (ee->error)
+			return;
+
+		ee->ival = oival & ee->ival;
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_bitor(struct expreval *ee)
+{
+	ee_bitand(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       (ee->code[ee->head] == '|' &&
+		!ee_invalidpunct(ee->code[ee->head + 1]))) {
+		ssize_t oival = ee->ival;
+
+		ee->head++;
+
+		ee_bitand(ee);
+		if (ee->error)
+			return;
+
+		ee->ival = oival | ee->ival;
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_logand(struct expreval *ee)
+{
+	ee_bitor(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       (ee->code[ee->head] == '&' && ee->code[ee->head + 1] == '&')) {
+		ssize_t oival = ee->ival;
+
+		ee->head += 2;
+
+		ee_bitor(ee);
+		if (ee->error)
+			return;
+
+		ee->ival = (oival && ee->ival) ? 1 : 0;
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_logor(struct expreval *ee)
+{
+	ee_logand(ee);
+	ee_skip_spaces(ee);
+
+	while (ee->head < ee->len && !ee->error &&
+	       (ee->code[ee->head] == '|' && ee->code[ee->head + 1] == '|')) {
+		ssize_t oival = ee->ival;
+
+		ee->head += 2;
+
+		ee_logand(ee);
+		if (ee->error)
+			return;
+
+		ee->ival = (oival || ee->ival) ? 1 : 0;
+
+		ee_skip_spaces(ee);
+	}
+}
+
+static void ee_expr(struct expreval *ee)
+{
+	ee_logor(ee);
+	if (ee->error == EERR_INVALID_EXPRESSION) {
+		/*
+		 * invalid expression doesn't really matter, it is only used to
+		 * stop the expression parsing.
+		 */
+		ee->error = EERR_NO_ERROR;
+		ee->ival = 1;
+	}
+}
+
+struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
+{
+	struct expreval ee;
+
+	code = lil_subst_to_value(lil, code);
+	if (lil->error)
+		return NULL;
+
+	ee.code = lil_to_string(code);
+	if (!ee.code[0]) {
+		/*
+		 * an empty expression equals to 0 so that it can be used as a
+		 * false value in conditionals
+		 */
+		lil_free_value(code);
+		return lil_alloc_integer(0);
+	}
+
+	ee.head = 0;
+	ee.len = code->l;
+	ee.ival = 0;
+	ee.error = 0;
+
+	ee_expr(&ee);
+	lil_free_value(code);
+	if (ee.error) {
+		switch (ee.error) {
+		case EERR_DIVISION_BY_ZERO:
+			lil_set_error(lil, "division by zero in expression");
+			break;
+		case EERR_SYNTAX_ERROR:
+			lil_set_error(lil, "expression syntax error");
+			break;
+		}
+		return NULL;
+	}
+	return lil_alloc_integer(ee.ival);
+}
+
+struct lil_value *lil_unused_name(struct lil *lil, const char *part)
+{
+	char *name = malloc(strlen(part) + 64);
+	struct lil_value *val;
+	size_t i;
+
+	for (i = 0; i < (size_t)-1; i++) {
+		sprintf(name, "!!un!%s!%09u!nu!!", part, (unsigned int)i);
+		if (lil_find_cmd(lil, name))
+			continue;
+
+		if (lil_find_var(lil, lil->env, name))
+			continue;
+
+		val = lil_alloc_string(name);
+		free(name);
+		return val;
+	}
+	return NULL;
+}
+
+struct lil_value *lil_arg(struct lil_value **argv, size_t index)
+{
+	return argv ? argv[index] : NULL;
+}
+
+const char *lil_to_string(struct lil_value *val)
+{
+	return (val && val->l) ? val->d : "";
+}
+
+ssize_t lil_to_integer(struct lil_value *val)
+{
+	return simple_strtol(lil_to_string(val), NULL, 0);
+}
+
+int lil_to_boolean(struct lil_value *val)
+{
+	const char *s = lil_to_string(val);
+	size_t i, dots = 0;
+
+	if (!s[0])
+		return 0;
+
+	for (i = 0; s[i]; i++) {
+		if (s[i] != '0' && s[i] != '.')
+			return 1;
+
+		if (s[i] == '.') {
+			if (dots)
+				return 1;
+
+			dots = 1;
+		}
+	}
+
+	return 0;
+}
+
+struct lil_value *lil_alloc_string(const char *str)
+{
+	return alloc_value(str);
+}
+
+struct lil_value *lil_alloc_string_len(const char *str, size_t len)
+{
+	return alloc_value_len(str, len);
+}
+
+struct lil_value *lil_alloc_integer(ssize_t num)
+{
+	char buff[128];
+
+	sprintf(buff, "%zd", num);
+	return alloc_value(buff);
+}
+
+void lil_free(struct lil *lil)
+{
+	size_t i;
+
+	if (!lil)
+		return;
+
+	free(lil->err_msg);
+	lil_free_value(lil->empty);
+	while (lil->env) {
+		struct lil_env *next = lil->env->parent;
+
+		lil_free_env(lil->env);
+		lil->env = next;
+	}
+
+	for (i = 0; i < lil->cmds; i++) {
+		if (lil->cmd[i]->argnames)
+			lil_free_list(lil->cmd[i]->argnames);
+
+		lil_free_value(lil->cmd[i]->code);
+		free(lil->cmd[i]->name);
+		free(lil->cmd[i]);
+	}
+
+	hm_destroy(&lil->cmdmap);
+	free(lil->cmd);
+	free(lil);
+}
+
+static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
+				     struct lil_value **argv)
+{
+	struct lil_func *func;
+	const char *type;
+	size_t i;
+	struct lil_value *r;
+
+	if (!argc)
+		return NULL;
+
+	type = lil_to_string(argv[0]);
+	if (!strcmp(type, "version"))
+		return lil_alloc_string(LIL_VERSION_STRING);
+
+	if (!strcmp(type, "args")) {
+		if (argc < 2)
+			return NULL;
+		func = lil_find_cmd(lil, lil_to_string(argv[1]));
+		if (!func || !func->argnames)
+			return NULL;
+		return lil_list_to_value(func->argnames, 1);
+	}
+
+	if (!strcmp(type, "body")) {
+		if (argc < 2)
+			return NULL;
+
+		func = lil_find_cmd(lil, lil_to_string(argv[1]));
+		if (!func || func->proc)
+			return NULL;
+
+		return lil_clone_value(func->code);
+	}
+
+	if (!strcmp(type, "func-count"))
+		return lil_alloc_integer(lil->cmds);
+
+	if (!strcmp(type, "funcs")) {
+		struct lil_list *funcs = lil_alloc_list();
+
+		for (i = 0; i < lil->cmds; i++)
+			lil_list_append(funcs,
+					lil_alloc_string(lil->cmd[i]->name));
+
+		r = lil_list_to_value(funcs, 1);
+		lil_free_list(funcs);
+		return r;
+	}
+
+	if (!strcmp(type, "vars")) {
+		struct lil_list *vars = lil_alloc_list();
+		struct lil_env *env = lil->env;
+
+		while (env) {
+			for (i = 0; i < env->vars; i++) {
+				struct lil_value *var =
+					lil_alloc_string(env->var[i]->n);
+
+				lil_list_append(vars, var);
+			}
+			env = env->parent;
+		}
+
+		r = lil_list_to_value(vars, 1);
+		lil_free_list(vars);
+		return r;
+	}
+
+	if (!strcmp(type, "globals")) {
+		struct lil_list *vars = lil_alloc_list();
+
+		for (i = 0; i < lil->rootenv->vars; i++) {
+			struct lil_value *var =
+				lil_alloc_string(lil->rootenv->var[i]->n);
+
+			lil_list_append(vars, var);
+		}
+
+		r = lil_list_to_value(vars, 1);
+		lil_free_list(vars);
+		return r;
+	}
+
+	if (!strcmp(type, "has-func")) {
+		const char *target;
+
+		if (argc == 1)
+			return NULL;
+
+		target = lil_to_string(argv[1]);
+		return hm_has(&lil->cmdmap, target) ? lil_alloc_string("1") :
+							    NULL;
+	}
+
+	if (!strcmp(type, "has-var")) {
+		const char *target;
+		struct lil_env *env = lil->env;
+
+		if (argc == 1)
+			return NULL;
+
+		target = lil_to_string(argv[1]);
+		while (env) {
+			if (hm_has(&env->varmap, target))
+				return lil_alloc_string("1");
+			env = env->parent;
+		}
+		return NULL;
+	}
+
+	if (!strcmp(type, "has-global")) {
+		const char *target;
+
+		if (argc == 1)
+			return NULL;
+
+		target = lil_to_string(argv[1]);
+		for (i = 0; i < lil->rootenv->vars; i++)
+			if (!strcmp(target, lil->rootenv->var[i]->n))
+				return lil_alloc_string("1");
+		return NULL;
+	}
+
+	if (!strcmp(type, "error"))
+		return lil->err_msg ? lil_alloc_string(lil->err_msg) : NULL;
+
+	if (!strcmp(type, "this")) {
+		struct lil_env *env = lil->env;
+
+		while (env != lil->rootenv && !env->func)
+			env = env->parent;
+
+		if (env == lil->rootenv)
+			return lil_alloc_string(lil->rootcode);
+
+		return env->func ? env->func->code : NULL;
+	}
+
+	if (!strcmp(type, "name")) {
+		struct lil_env *env = lil->env;
+
+		while (env != lil->rootenv && !env->func)
+			env = env->parent;
+
+		if (env == lil->rootenv)
+			return NULL;
+
+		return env->func ? lil_alloc_string(env->func->name) : NULL;
+	}
+
+	return NULL;
+}
+
+static struct lil_value *fnc_func(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	struct lil_value *name;
+	struct lil_func *cmd;
+	struct lil_list *fargs;
+
+	if (argc < 1)
+		return NULL;
+
+	if (argc >= 3) {
+		name = lil_clone_value(argv[0]);
+		fargs = lil_subst_to_list(lil, argv[1]);
+		cmd = add_func(lil, lil_to_string(argv[0]));
+		cmd->argnames = fargs;
+		cmd->code = lil_clone_value(argv[2]);
+	} else {
+		name = lil_unused_name(lil, "anonymous-function");
+		if (argc < 2) {
+			struct lil_value *tmp = lil_alloc_string("args");
+
+			fargs = lil_subst_to_list(lil, tmp);
+			lil_free_value(tmp);
+			cmd = add_func(lil, lil_to_string(name));
+			cmd->argnames = fargs;
+			cmd->code = lil_clone_value(argv[0]);
+		} else {
+			fargs = lil_subst_to_list(lil, argv[0]);
+			cmd = add_func(lil, lil_to_string(name));
+			cmd->argnames = fargs;
+			cmd->code = lil_clone_value(argv[1]);
+		}
+	}
+
+	return name;
+}
+
+static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	struct lil_value *r;
+	struct lil_func *func;
+	const char *oldname;
+	const char *newname;
+
+	if (argc < 2)
+		return NULL;
+
+	oldname = lil_to_string(argv[0]);
+	newname = lil_to_string(argv[1]);
+	func = lil_find_cmd(lil, oldname);
+	if (!func) {
+		char *msg = malloc(24 + strlen(oldname));
+
+		sprintf(msg, "unknown function '%s'", oldname);
+		lil_set_error_at(lil, lil->head, msg);
+		free(msg);
+		return NULL;
+	}
+
+	r = lil_alloc_string(func->name);
+	if (newname[0]) {
+		hm_put(&lil->cmdmap, oldname, 0);
+		hm_put(&lil->cmdmap, newname, func);
+		free(func->name);
+		func->name = strclone(newname);
+	} else {
+		del_func(lil, func);
+	}
+
+	return r;
+}
+
+static struct lil_value *fnc_unusedname(struct lil *lil, size_t argc,
+					struct lil_value **argv)
+{
+	return lil_unused_name(lil, argc > 0 ? lil_to_string(argv[0]) :
+						     "unusedname");
+}
+
+static struct lil_value *fnc_quote(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_value *r;
+	size_t i;
+
+	if (argc < 1)
+		return NULL;
+
+	r = alloc_value(NULL);
+	for (i = 0; i < argc; i++) {
+		if (i)
+			lil_append_char(r, ' ');
+		lil_append_val(r, argv[i]);
+	}
+
+	return r;
+}
+
+static struct lil_value *fnc_set(struct lil *lil, size_t argc,
+				 struct lil_value **argv)
+{
+	size_t i = 0;
+	struct lil_var *var = NULL;
+	int access = LIL_SETVAR_LOCAL;
+
+	if (!argc)
+		return NULL;
+
+	if (!strcmp(lil_to_string(argv[0]), "global")) {
+		i = 1;
+		access = LIL_SETVAR_GLOBAL;
+	}
+
+	while (i < argc) {
+		if (argc == i + 1) {
+			struct lil_value *val =
+				lil_get_var(lil, lil_to_string(argv[i]));
+
+			return lil_clone_value(val);
+		}
+
+		var = lil_set_var(lil, lil_to_string(argv[i]), argv[i + 1],
+				  access);
+		i += 2;
+	}
+
+	return var ? lil_clone_value(var->v) : NULL;
+}
+
+static struct lil_value *fnc_local(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	size_t i;
+
+	for (i = 0; i < argc; i++) {
+		const char *varname = lil_to_string(argv[i]);
+
+		if (!lil_find_local_var(lil, lil->env, varname))
+			lil_set_var(lil, varname, lil->empty,
+				    LIL_SETVAR_LOCAL_NEW);
+	}
+
+	return NULL;
+}
+
+static struct lil_value *fnc_eval(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	if (argc == 1)
+		return lil_parse_value(lil, argv[0], 0);
+
+	if (argc > 1) {
+		struct lil_value *val = alloc_value(NULL), *r;
+		size_t i;
+
+		for (i = 0; i < argc; i++) {
+			if (i)
+				lil_append_char(val, ' ');
+			lil_append_val(val, argv[i]);
+		}
+
+		r = lil_parse_value(lil, val, 0);
+		lil_free_value(val);
+		return r;
+	}
+
+	return NULL;
+}
+
+static struct lil_value *fnc_topeval(struct lil *lil, size_t argc,
+				     struct lil_value **argv)
+{
+	struct lil_env *thisenv = lil->env;
+	struct lil_env *thisdownenv = lil->downenv;
+	struct lil_value *r;
+
+	lil->env = lil->rootenv;
+	lil->downenv = thisenv;
+
+	r = fnc_eval(lil, argc, argv);
+	lil->downenv = thisdownenv;
+	lil->env = thisenv;
+	return r;
+}
+
+static struct lil_value *fnc_upeval(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	struct lil_env *thisenv = lil->env;
+	struct lil_env *thisdownenv = lil->downenv;
+	struct lil_value *r;
+
+	if (lil->rootenv == thisenv)
+		return fnc_eval(lil, argc, argv);
+
+	lil->env = thisenv->parent;
+	lil->downenv = thisenv;
+
+	r = fnc_eval(lil, argc, argv);
+	lil->env = thisenv;
+	lil->downenv = thisdownenv;
+	return r;
+}
+
+static struct lil_value *fnc_downeval(struct lil *lil, size_t argc,
+				      struct lil_value **argv)
+{
+	struct lil_value *r;
+	struct lil_env *upenv = lil->env;
+	struct lil_env *downenv = lil->downenv;
+
+	if (!downenv)
+		return fnc_eval(lil, argc, argv);
+
+	lil->downenv = NULL;
+	lil->env = downenv;
+
+	r = fnc_eval(lil, argc, argv);
+	lil->downenv = downenv;
+	lil->env = upenv;
+	return r;
+}
+
+static struct lil_value *fnc_count(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_list *list;
+	char buff[64];
+
+	if (!argc)
+		return alloc_value("0");
+
+	list = lil_subst_to_list(lil, argv[0]);
+	sprintf(buff, "%u", (unsigned int)list->c);
+	lil_free_list(list);
+	return alloc_value(buff);
+}
+
+static struct lil_value *fnc_index(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_list *list;
+	size_t index;
+	struct lil_value *r;
+
+	if (argc < 2)
+		return NULL;
+
+	list = lil_subst_to_list(lil, argv[0]);
+	index = (size_t)lil_to_integer(argv[1]);
+	if (index >= list->c)
+		r = NULL;
+	else
+		r = lil_clone_value(list->v[index]);
+	lil_free_list(list);
+	return r;
+}
+
+static struct lil_value *fnc_indexof(struct lil *lil, size_t argc,
+				     struct lil_value **argv)
+{
+	struct lil_list *list;
+	size_t index;
+	struct lil_value *r = NULL;
+
+	if (argc < 2)
+		return NULL;
+
+	list = lil_subst_to_list(lil, argv[0]);
+	for (index = 0; index < list->c; index++) {
+		if (!strcmp(lil_to_string(list->v[index]),
+			    lil_to_string(argv[1]))) {
+			r = lil_alloc_integer(index);
+			break;
+		}
+	}
+
+	lil_free_list(list);
+	return r;
+}
+
+static struct lil_value *fnc_append(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	struct lil_list *list;
+	struct lil_value *r;
+	size_t i, base = 1;
+	int access = LIL_SETVAR_LOCAL;
+	const char *varname;
+
+	if (argc < 2)
+		return NULL;
+
+	varname = lil_to_string(argv[0]);
+	if (!strcmp(varname, "global")) {
+		if (argc < 3)
+			return NULL;
+
+		varname = lil_to_string(argv[1]);
+		base = 2;
+		access = LIL_SETVAR_GLOBAL;
+	}
+
+	list = lil_subst_to_list(lil, lil_get_var(lil, varname));
+	for (i = base; i < argc; i++)
+		lil_list_append(list, lil_clone_value(argv[i]));
+
+	r = lil_list_to_value(list, 1);
+	lil_free_list(list);
+	lil_set_var(lil, varname, r, access);
+	return r;
+}
+
+static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_list *list, *slice;
+	size_t i;
+	ssize_t from, to;
+	struct lil_value *r;
+
+	if (argc < 1)
+		return NULL;
+	if (argc < 2)
+		return lil_clone_value(argv[0]);
+
+	from = lil_to_integer(argv[1]);
+	if (from < 0)
+		from = 0;
+
+	list = lil_subst_to_list(lil, argv[0]);
+	to = argc > 2 ? lil_to_integer(argv[2]) : (ssize_t)list->c;
+	if (to > (ssize_t)list->c)
+		to = list->c;
+	else if (to < from)
+		to = from;
+
+	slice = lil_alloc_list();
+	for (i = (size_t)from; i < (size_t)to; i++)
+		lil_list_append(slice, lil_clone_value(list->v[i]));
+	lil_free_list(list);
+
+	r = lil_list_to_value(slice, 1);
+	lil_free_list(slice);
+	return r;
+}
+
+static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	struct lil_list *list, *filtered;
+	size_t i;
+	struct lil_value *r;
+	const char *varname = "x";
+	int base = 0;
+
+	if (argc < 1)
+		return NULL;
+
+	if (argc < 2)
+		return lil_clone_value(argv[0]);
+
+	if (argc > 2) {
+		base = 1;
+		varname = lil_to_string(argv[0]);
+	}
+
+	list = lil_subst_to_list(lil, argv[base]);
+	filtered = lil_alloc_list();
+	for (i = 0; i < list->c && !lil->env->breakrun; i++) {
+		lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+		r = lil_eval_expr(lil, argv[base + 1]);
+		if (lil_to_boolean(r))
+			lil_list_append(filtered, lil_clone_value(list->v[i]));
+		lil_free_value(r);
+	}
+	lil_free_list(list);
+
+	r = lil_list_to_value(filtered, 1);
+	lil_free_list(filtered);
+	return r;
+}
+
+static struct lil_value *fnc_list(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	struct lil_list *list = lil_alloc_list();
+	struct lil_value *r;
+	size_t i;
+
+	for (i = 0; i < argc; i++)
+		lil_list_append(list, lil_clone_value(argv[i]));
+
+	r = lil_list_to_value(list, 1);
+	lil_free_list(list);
+	return r;
+}
+
+static struct lil_value *fnc_subst(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	if (argc < 1)
+		return NULL;
+
+	return lil_subst_to_value(lil, argv[0]);
+}
+
+static struct lil_value *fnc_concat(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	struct lil_list *list;
+	struct lil_value *r, *tmp;
+	size_t i;
+
+	if (argc < 1)
+		return NULL;
+
+	r = lil_alloc_string("");
+	for (i = 0; i < argc; i++) {
+		list = lil_subst_to_list(lil, argv[i]);
+		tmp = lil_list_to_value(list, 1);
+		lil_free_list(list);
+		lil_append_val(r, tmp);
+		lil_free_value(tmp);
+	}
+	return r;
+}
+
+static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
+				     struct lil_value **argv)
+{
+	struct lil_list *list, *rlist;
+	struct lil_value *r;
+	size_t i, listidx = 0, codeidx = 1;
+	const char *varname = "i";
+
+	if (argc < 2)
+		return NULL;
+
+	if (argc >= 3) {
+		varname = lil_to_string(argv[0]);
+		listidx = 1;
+		codeidx = 2;
+	}
+
+	rlist = lil_alloc_list();
+	list = lil_subst_to_list(lil, argv[listidx]);
+	for (i = 0; i < list->c; i++) {
+		struct lil_value *rv;
+
+		lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+		rv = lil_parse_value(lil, argv[codeidx], 0);
+		if (rv->l)
+			lil_list_append(rlist, rv);
+		else
+			lil_free_value(rv);
+
+		if (lil->env->breakrun || lil->error)
+			break;
+	}
+
+	r = lil_list_to_value(rlist, 1);
+	lil_free_list(list);
+	lil_free_list(rlist);
+	return r;
+}
+
+static struct lil_value *fnc_return(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	lil->env->breakrun = 1;
+	lil_free_value(lil->env->retval);
+	lil->env->retval = argc < 1 ? NULL : lil_clone_value(argv[0]);
+	lil->env->retval_set = 1;
+	return argc < 1 ? NULL : lil_clone_value(argv[0]);
+}
+
+static struct lil_value *fnc_result(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	if (argc > 0) {
+		lil_free_value(lil->env->retval);
+		lil->env->retval = lil_clone_value(argv[0]);
+		lil->env->retval_set = 1;
+	}
+
+	return lil->env->retval_set ? lil_clone_value(lil->env->retval) : NULL;
+}
+
+static struct lil_value *fnc_expr(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	if (argc == 1)
+		return lil_eval_expr(lil, argv[0]);
+
+	if (argc > 1) {
+		struct lil_value *val = alloc_value(NULL), *r;
+		size_t i;
+
+		for (i = 0; i < argc; i++) {
+			if (i)
+				lil_append_char(val, ' ');
+			lil_append_val(val, argv[i]);
+		}
+
+		r = lil_eval_expr(lil, val);
+		lil_free_value(val);
+		return r;
+	}
+
+	return NULL;
+}
+
+static struct lil_value *real_inc(struct lil *lil, const char *varname,
+				  ssize_t v)
+{
+	struct lil_value *pv = lil_get_var(lil, varname);
+
+	pv = lil_alloc_integer(lil_to_integer(pv) + v);
+	lil_set_var(lil, varname, pv, LIL_SETVAR_LOCAL);
+	return pv;
+}
+
+static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
+				 struct lil_value **argv)
+{
+	if (argc < 1)
+		return NULL;
+
+	return real_inc(lil, lil_to_string(argv[0]),
+			argc > 1 ? lil_to_integer(argv[1]) : 1);
+}
+
+static struct lil_value *fnc_dec(struct lil *lil, size_t argc,
+				 struct lil_value **argv)
+{
+	if (argc < 1)
+		return NULL;
+
+	return real_inc(lil, lil_to_string(argv[0]),
+			-(argc > 1 ? lil_to_integer(argv[1]) : 1));
+}
+
+static struct lil_value *fnc_if(struct lil *lil, size_t argc,
+				struct lil_value **argv)
+{
+	struct lil_value *val, *r = NULL;
+	int base = 0, not = 0, v;
+
+	if (argc < 1)
+		return NULL;
+
+	if (!strcmp(lil_to_string(argv[0]), "not"))
+		base = not = 1;
+
+	if (argc < (size_t)base + 2)
+		return NULL;
+
+	val = lil_eval_expr(lil, argv[base]);
+	if (!val || lil->error)
+		return NULL;
+
+	v = lil_to_boolean(val);
+	if (not)
+		v = !v;
+
+	if (v)
+		r = lil_parse_value(lil, argv[base + 1], 0);
+	else if (argc > (size_t)base + 2)
+		r = lil_parse_value(lil, argv[base + 2], 0);
+
+	lil_free_value(val);
+	return r;
+}
+
+static struct lil_value *fnc_while(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_value *val, *r = NULL;
+	int base = 0, not = 0, v;
+
+	if (argc < 1)
+		return NULL;
+
+	if (!strcmp(lil_to_string(argv[0]), "not"))
+		base = not = 1;
+
+	if (argc < (size_t)base + 2)
+		return NULL;
+
+	while (!lil->error && !lil->env->breakrun) {
+		val = lil_eval_expr(lil, argv[base]);
+		if (!val || lil->error)
+			return NULL;
+
+		v = lil_to_boolean(val);
+		if (not)
+			v = !v;
+
+		if (!v) {
+			lil_free_value(val);
+			break;
+		}
+
+		if (r)
+			lil_free_value(r);
+		r = lil_parse_value(lil, argv[base + 1], 0);
+		lil_free_value(val);
+	}
+
+	return r;
+}
+
+static struct lil_value *fnc_for(struct lil *lil, size_t argc,
+				 struct lil_value **argv)
+{
+	struct lil_value *val, *r = NULL;
+
+	if (argc < 4)
+		return NULL;
+
+	lil_free_value(lil_parse_value(lil, argv[0], 0));
+	while (!lil->error && !lil->env->breakrun) {
+		val = lil_eval_expr(lil, argv[1]);
+		if (!val || lil->error)
+			return NULL;
+
+		if (!lil_to_boolean(val)) {
+			lil_free_value(val);
+			break;
+		}
+
+		if (r)
+			lil_free_value(r);
+		r = lil_parse_value(lil, argv[3], 0);
+		lil_free_value(val);
+		lil_free_value(lil_parse_value(lil, argv[2], 0));
+	}
+
+	return r;
+}
+
+static struct lil_value *fnc_char(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	char s[2];
+
+	if (!argc)
+		return NULL;
+
+	s[0] = (char)lil_to_integer(argv[0]);
+	s[1] = 0;
+	return lil_alloc_string(s);
+}
+
+static struct lil_value *fnc_charat(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	size_t index;
+	char chstr[2];
+	const char *str;
+
+	if (argc < 2)
+		return NULL;
+
+	str = lil_to_string(argv[0]);
+	index = (size_t)lil_to_integer(argv[1]);
+	if (index >= strlen(str))
+		return NULL;
+
+	chstr[0] = str[index];
+	chstr[1] = 0;
+	return lil_alloc_string(chstr);
+}
+
+static struct lil_value *fnc_codeat(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	size_t index;
+	const char *str;
+
+	if (argc < 2)
+		return NULL;
+
+	str = lil_to_string(argv[0]);
+	index = (size_t)lil_to_integer(argv[1]);
+	if (index >= strlen(str))
+		return NULL;
+
+	return lil_alloc_integer(str[index]);
+}
+
+static struct lil_value *fnc_substr(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	const char *str;
+	struct lil_value *r;
+	size_t start, end, i, slen;
+
+	if (argc < 2)
+		return NULL;
+
+	str = lil_to_string(argv[0]);
+	if (!str[0])
+		return NULL;
+
+	slen = strlen(str);
+	start = simple_strtol(lil_to_string(argv[1]), NULL, 0);
+	end = argc > 2 ? simple_strtol(lil_to_string(argv[2]), NULL, 0) : slen;
+	if (end > slen)
+		end = slen;
+
+	if (start >= end)
+		return NULL;
+
+	r = lil_alloc_string("");
+	for (i = start; i < end; i++)
+		lil_append_char(r, str[i]);
+	return r;
+}
+
+static struct lil_value *fnc_strpos(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	const char *hay;
+	const char *str;
+	size_t min = 0;
+
+	if (argc < 2)
+		return lil_alloc_integer(-1);
+
+	hay = lil_to_string(argv[0]);
+	if (argc > 2) {
+		min = simple_strtol(lil_to_string(argv[2]), NULL, 0);
+		if (min >= strlen(hay))
+			return lil_alloc_integer(-1);
+	}
+
+	str = strstr(hay + min, lil_to_string(argv[1]));
+	if (!str)
+		return lil_alloc_integer(-1);
+
+	return lil_alloc_integer(str - hay);
+}
+
+static struct lil_value *fnc_length(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	size_t i, total = 0;
+
+	for (i = 0; i < argc; i++) {
+		if (i)
+			total++;
+		total += strlen(lil_to_string(argv[i]));
+	}
+
+	return lil_alloc_integer((ssize_t)total);
+}
+
+static struct lil_value *real_trim(const char *str, const char *chars, int left,
+				   int right)
+{
+	int base = 0;
+	struct lil_value *r = NULL;
+
+	if (left) {
+		while (str[base] && strchr(chars, str[base]))
+			base++;
+		if (!right)
+			r = lil_alloc_string(str[base] ? str + base : NULL);
+	}
+
+	if (right) {
+		size_t len;
+		char *s;
+
+		s = strclone(str + base);
+		len = strlen(s);
+		while (len && strchr(chars, s[len - 1]))
+			len--;
+
+		s[len] = 0;
+		r = lil_alloc_string(s);
+		free(s);
+	}
+
+	return r;
+}
+
+static struct lil_value *fnc_trim(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	const char *chars;
+
+	if (!argc)
+		return NULL;
+
+	if (argc < 2)
+		chars = " \f\n\r\t\v";
+	else
+		chars = lil_to_string(argv[1]);
+
+	return real_trim(lil_to_string(argv[0]), chars, 1, 1);
+}
+
+static struct lil_value *fnc_ltrim(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	const char *chars;
+
+	if (!argc)
+		return NULL;
+
+	if (argc < 2)
+		chars = " \f\n\r\t\v";
+	else
+		chars = lil_to_string(argv[1]);
+
+	return real_trim(lil_to_string(argv[0]), chars, 1, 0);
+}
+
+static struct lil_value *fnc_rtrim(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	const char *chars;
+
+	if (!argc)
+		return NULL;
+
+	if (argc < 2)
+		chars = " \f\n\r\t\v";
+	else
+		chars = lil_to_string(argv[1]);
+
+	return real_trim(lil_to_string(argv[0]), chars, 0, 1);
+}
+
+static struct lil_value *fnc_strcmp(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	if (argc < 2)
+		return NULL;
+
+	return lil_alloc_integer(strcmp(lil_to_string(argv[0]),
+				 lil_to_string(argv[1])));
+}
+
+static struct lil_value *fnc_streq(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	if (argc < 2)
+		return NULL;
+
+	return lil_alloc_integer(strcmp(lil_to_string(argv[0]),
+				 lil_to_string(argv[1])) ? 0 : 1);
+}
+
+static struct lil_value *fnc_repstr(struct lil *lil, size_t argc,
+				    struct lil_value **argv)
+{
+	const char *from;
+	const char *to;
+	char *src;
+	const char *sub;
+	size_t idx;
+	size_t fromlen;
+	size_t tolen;
+	size_t srclen;
+	struct lil_value *r;
+
+	if (argc < 1)
+		return NULL;
+
+	if (argc < 3)
+		return lil_clone_value(argv[0]);
+
+	from = lil_to_string(argv[1]);
+	to = lil_to_string(argv[2]);
+	if (!from[0])
+		return NULL;
+
+	src = strclone(lil_to_string(argv[0]));
+	srclen = strlen(src);
+	fromlen = strlen(from);
+	tolen = strlen(to);
+	while ((sub = strstr(src, from))) {
+		char *newsrc = malloc(srclen - fromlen + tolen + 1);
+
+		idx = sub - src;
+		if (idx)
+			memcpy(newsrc, src, idx);
+
+		memcpy(newsrc + idx, to, tolen);
+		memcpy(newsrc + idx + tolen, src + idx + fromlen,
+		       srclen - idx - fromlen);
+		srclen = srclen - fromlen + tolen;
+		free(src);
+		src = newsrc;
+		src[srclen] = 0;
+	}
+
+	r = lil_alloc_string(src);
+	free(src);
+	return r;
+}
+
+static struct lil_value *fnc_split(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	struct lil_list *list;
+	const char *sep = " ";
+	size_t i;
+	struct lil_value *val;
+	const char *str;
+
+	if (argc == 0)
+		return NULL;
+	if (argc > 1) {
+		sep = lil_to_string(argv[1]);
+		if (!sep || !sep[0])
+			return lil_clone_value(argv[0]);
+	}
+
+	val = lil_alloc_string("");
+	str = lil_to_string(argv[0]);
+	list = lil_alloc_list();
+	for (i = 0; str[i]; i++) {
+		if (strchr(sep, str[i])) {
+			lil_list_append(list, val);
+			val = lil_alloc_string("");
+		} else {
+			lil_append_char(val, str[i]);
+		}
+	}
+
+	lil_list_append(list, val);
+	val = lil_list_to_value(list, 1);
+	lil_free_list(list);
+	return val;
+}
+
+static struct lil_value *fnc_try(struct lil *lil, size_t argc,
+				 struct lil_value **argv)
+{
+	struct lil_value *r;
+
+	if (argc < 1)
+		return NULL;
+
+	if (lil->error)
+		return NULL;
+
+	r = lil_parse_value(lil, argv[0], 0);
+	if (lil->error) {
+		lil->error = ERROR_NOERROR;
+		lil_free_value(r);
+
+		if (argc > 1)
+			r = lil_parse_value(lil, argv[1], 0);
+		else
+			r = 0;
+	}
+	return r;
+}
+
+static struct lil_value *fnc_error(struct lil *lil, size_t argc,
+				   struct lil_value **argv)
+{
+	lil_set_error(lil, argc > 0 ? lil_to_string(argv[0]) : NULL);
+	return NULL;
+}
+
+static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
+{
+	struct lil_list *list;
+	size_t i;
+
+	if (argc < 2)
+		return NULL;
+
+	list = lil_subst_to_list(lil, argv[0]);
+	for (i = 1; i < argc; i++)
+		lil_set_var(lil, lil_to_string(argv[i]),
+			    lil_list_get(list, i - 1), LIL_SETVAR_LOCAL);
+	lil_free_list(list);
+	return NULL;
+}
+
+static void register_stdcmds(struct lil *lil)
+{
+	lil_register(lil, "append", fnc_append);
+	lil_register(lil, "char", fnc_char);
+	lil_register(lil, "charat", fnc_charat);
+	lil_register(lil, "codeat", fnc_codeat);
+	lil_register(lil, "concat", fnc_concat);
+	lil_register(lil, "count", fnc_count);
+	lil_register(lil, "dec", fnc_dec);
+	lil_register(lil, "downeval", fnc_downeval);
+	lil_register(lil, "error", fnc_error);
+	lil_register(lil, "eval", fnc_eval);
+	lil_register(lil, "expr", fnc_expr);
+	lil_register(lil, "filter", fnc_filter);
+	lil_register(lil, "for", fnc_for);
+	lil_register(lil, "foreach", fnc_foreach);
+	lil_register(lil, "func", fnc_func);
+	lil_register(lil, "if", fnc_if);
+	lil_register(lil, "inc", fnc_inc);
+	lil_register(lil, "index", fnc_index);
+	lil_register(lil, "indexof", fnc_indexof);
+	lil_register(lil, "length", fnc_length);
+	lil_register(lil, "list", fnc_list);
+	lil_register(lil, "lmap", fnc_lmap);
+	lil_register(lil, "local", fnc_local);
+	lil_register(lil, "ltrim", fnc_ltrim);
+	lil_register(lil, "quote", fnc_quote);
+	lil_register(lil, "reflect", fnc_reflect);
+	lil_register(lil, "rename", fnc_rename);
+	lil_register(lil, "repstr", fnc_repstr);
+	lil_register(lil, "result", fnc_result);
+	lil_register(lil, "return", fnc_return);
+	lil_register(lil, "rtrim", fnc_rtrim);
+	lil_register(lil, "set", fnc_set);
+	lil_register(lil, "slice", fnc_slice);
+	lil_register(lil, "split", fnc_split);
+	lil_register(lil, "strcmp", fnc_strcmp);
+	lil_register(lil, "streq", fnc_streq);
+	lil_register(lil, "strpos", fnc_strpos);
+	lil_register(lil, "subst", fnc_subst);
+	lil_register(lil, "substr", fnc_substr);
+	lil_register(lil, "topeval", fnc_topeval);
+	lil_register(lil, "trim", fnc_trim);
+	lil_register(lil, "try", fnc_try);
+	lil_register(lil, "unusedname", fnc_unusedname);
+	lil_register(lil, "upeval", fnc_upeval);
+	lil_register(lil, "while", fnc_while);
+}
diff --git a/include/cli_lil.h b/include/cli_lil.h
new file mode 100644
index 0000000000..c72977ea5c
--- /dev/null
+++ b/include/cli_lil.h
@@ -0,0 +1,111 @@ 
+/* SPDX-License-Identifier: GPL-2.0+ AND Zlib */
+/*
+ * LIL - Little Interpreted Language
+ * Copyright (C) 2021 Sean Anderson <seanga2@gmail.com>
+ * Copyright (C) 2010-2013 Kostas Michalopoulos <badsector@runtimelegend.com>
+ *
+ * This file originated from the LIL project, licensed under Zlib. All
+ * modifications are licensed under GPL-2.0+
+ */
+
+#ifndef __LIL_H_INCLUDED__
+#define __LIL_H_INCLUDED__
+
+#define LIL_VERSION_STRING "0.1"
+
+#define LIL_SETVAR_GLOBAL 0
+#define LIL_SETVAR_LOCAL 1
+#define LIL_SETVAR_LOCAL_NEW 2
+#define LIL_SETVAR_LOCAL_ONLY 3
+
+#define LIL_CALLBACK_EXIT 0
+#define LIL_CALLBACK_WRITE 1
+#define LIL_CALLBACK_READ 2
+#define LIL_CALLBACK_STORE 3
+#define LIL_CALLBACK_SOURCE 4
+#define LIL_CALLBACK_ERROR 5
+#define LIL_CALLBACK_SETVAR 6
+#define LIL_CALLBACK_GETVAR 7
+
+#include <stdint.h>
+#include <inttypes.h>
+
+struct lil_value;
+struct lil_func;
+struct lil_var;
+struct lil_env;
+struct lil_list;
+struct lil;
+typedef struct lil_value *(*lil_func_proc_t)(struct lil *lil, size_t argc,
+					     struct lil_value **argv);
+typedef void (*lil_exit_callback_proc_t)(struct lil *lil,
+					 struct lil_value *arg);
+typedef void (*lil_write_callback_proc_t)(struct lil *lil, const char *msg);
+typedef char *(*lil_read_callback_proc_t)(struct lil *lil, const char *name);
+typedef char *(*lil_source_callback_proc_t)(struct lil *lil, const char *name);
+typedef void (*lil_store_callback_proc_t)(struct lil *lil, const char *name,
+					  const char *data);
+typedef void (*lil_error_callback_proc_t)(struct lil *lil, size_t pos,
+					  const char *msg);
+typedef int (*lil_setvar_callback_proc_t)(struct lil *lil, const char *name,
+					  struct lil_value **value);
+typedef int (*lil_getvar_callback_proc_t)(struct lil *lil, const char *name,
+					  struct lil_value **value);
+typedef void (*lil_callback_proc_t)(void);
+
+struct lil *lil_new(void);
+void lil_free(struct lil *lil);
+
+int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
+
+struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
+			    int funclevel);
+struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
+				  int funclevel);
+
+void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc);
+
+void lil_set_error(struct lil *lil, const char *msg);
+void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
+int lil_error(struct lil *lil, const char **msg, size_t *pos);
+
+const char *lil_to_string(struct lil_value *val);
+ssize_t lil_to_integer(struct lil_value *val);
+int lil_to_boolean(struct lil_value *val);
+
+struct lil_value *lil_alloc_string(const char *str);
+struct lil_value *lil_alloc_integer(ssize_t num);
+void lil_free_value(struct lil_value *val);
+
+struct lil_value *lil_clone_value(struct lil_value *src);
+int lil_append_char(struct lil_value *val, char ch);
+int lil_append_string(struct lil_value *val, const char *s);
+int lil_append_val(struct lil_value *val, struct lil_value *v);
+
+struct lil_list *lil_alloc_list(void);
+void lil_free_list(struct lil_list *list);
+void lil_list_append(struct lil_list *list, struct lil_value *val);
+size_t lil_list_size(struct lil_list *list);
+struct lil_value *lil_list_get(struct lil_list *list, size_t index);
+struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
+
+struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
+struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
+
+struct lil_env *lil_alloc_env(struct lil_env *parent);
+void lil_free_env(struct lil_env *env);
+struct lil_env *lil_push_env(struct lil *lil);
+void lil_pop_env(struct lil *lil);
+
+struct lil_var *lil_set_var(struct lil *lil, const char *name,
+			    struct lil_value *val, int local);
+struct lil_value *lil_get_var(struct lil *lil, const char *name);
+struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
+				 struct lil_value *defvalue);
+
+struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code);
+struct lil_value *lil_unused_name(struct lil *lil, const char *part);
+
+struct lil_value *lil_arg(struct lil_value **argv, size_t index);
+
+#endif