Patchwork [2/2] Monitor: Make output buffer dynamic

login
register
mail settings
Submitter Wayne Xia
Date March 27, 2013, 6:45 a.m.
Message ID <515295A1.8030705@linux.vnet.ibm.com>
Download mbox | patch
Permalink /patch/231600/
State New
Headers show

Comments

Wayne Xia - March 27, 2013, 6:45 a.m.
Hi, Luiz
  Personally I hope reduce the dynamic allocated buffer which brings
fragments and unexpected memory grow. Instead, how about sacrifice
some time to wait output complete, since monitor is not time critical?
in this case static buffer's size can decide how many work can be
postponded. Following is my suggestion:

 }

> Commit f628926bb423fa8a7e0b114511400ea9df38b76a changed monitor_flush()
> to retry on qemu_chr_fe_write() errors. However, the Monitor's output
> buffer can keep growing while the retry is not issued and this can
> cause the buffer to overflow.
> 
> To reproduce this issue, just start qemu and type on the Monitor:
> 
> (qemu) ?
> 
> This will cause the assertion to trig.
> 
> To fix this problem this commit makes the Monitor buffer dynamic,
> which means that it can grow as much as needed.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>   monitor.c | 42 +++++++++++++++++++++++++-----------------
>   1 file changed, 25 insertions(+), 17 deletions(-)
>
Luiz Capitulino - March 27, 2013, 12:27 p.m.
On Wed, 27 Mar 2013 14:45:53 +0800
Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:

> Hi, Luiz
>   Personally I hope reduce the dynamic allocated buffer which brings
> fragments and unexpected memory grow. Instead, how about sacrifice
> some time to wait output complete, since monitor is not time critical?
> in this case static buffer's size can decide how many work can be
> postponded. Following is my suggestion:

I'd be fine with it if you find a good way to postpone work, but sleeping
on random timers is not a good way.

Also, QEMU allocates fragments of memory in so many places that I doubt
this is an valid argument. You're right that long QMP output can cause
unexpected memory growth, but I expected this to be really rare and if
this really bother us, we can ask monitor_puts() to flush at, say, every
4096 bytes.

> 
> --- a/monitor.c
> +++ b/monitor.c
> @@ -293,17 +293,28 @@ static void monitor_puts(Monitor *mon, const char
> *str)
>  {
>      char c;
> 
> +    /* if mux do not put in any thing to buffer */
> +    if (mon->mux_out) {
> +        return;
> +    }
> +
>      for(;;) {
> -        assert(mon->outbuf_index < sizeof(mon->outbuf) - 1);
> +        if (mon->outbuf_index >= sizeof(mon->outbuf) - 1) {
> +            /* when buffer is full, flush it and retry. If buffer is
> bigger, more
> +               work can be postponed. */
> +            monitor_flush(mon);
> +            usleep(1);
> +            continue;
> +        }
>          c = *str++;
>          if (c == '\0')
>              break;
>          if (c == '\n')
>              mon->outbuf[mon->outbuf_index++] = '\r';
>          mon->outbuf[mon->outbuf_index++] = c;
> -        if (mon->outbuf_index >= (sizeof(mon->outbuf) - 1)
> -            || c == '\n')
> +        if (c == '\n') {
>              monitor_flush(mon);
> +        }
>      }
>  }
> 
> > Commit f628926bb423fa8a7e0b114511400ea9df38b76a changed monitor_flush()
> > to retry on qemu_chr_fe_write() errors. However, the Monitor's output
> > buffer can keep growing while the retry is not issued and this can
> > cause the buffer to overflow.
> > 
> > To reproduce this issue, just start qemu and type on the Monitor:
> > 
> > (qemu) ?
> > 
> > This will cause the assertion to trig.
> > 
> > To fix this problem this commit makes the Monitor buffer dynamic,
> > which means that it can grow as much as needed.
> > 
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > ---
> >   monitor.c | 42 +++++++++++++++++++++++++-----------------
> >   1 file changed, 25 insertions(+), 17 deletions(-)
> > 
>
Wayne Xia - March 28, 2013, 3:21 a.m.
于 2013-3-27 20:27, Luiz Capitulino 写道:
> On Wed, 27 Mar 2013 14:45:53 +0800
> Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
>
>> Hi, Luiz
>>    Personally I hope reduce the dynamic allocated buffer which brings
>> fragments and unexpected memory grow. Instead, how about sacrifice
>> some time to wait output complete, since monitor is not time critical?
>> in this case static buffer's size can decide how many work can be
>> postponded. Following is my suggestion:
>
> I'd be fine with it if you find a good way to postpone work, but sleeping
> on random timers is not a good way.
>
> Also, QEMU allocates fragments of memory in so many places that I doubt
> this is an valid argument. You're right that long QMP output can cause
> unexpected memory growth, but I expected this to be really rare and if
> this really bother us, we can ask monitor_puts() to flush at, say, every
> 4096 bytes.
>
   If the point was async call, you are right it should not sleep. My
thoughts is reducing dynamic allocation when possible. Two alternative
are: make static buffer larger as 4096 bytes and drop the remain if
buffer is full, or dynamic increase the buffer size with a fixed
number when buffer overflow avoiding dynamic allocation every time.
It is a bit overkill, so I am also OK with your solution.

>>
>> --- a/monitor.c
>> +++ b/monitor.c
>> @@ -293,17 +293,28 @@ static void monitor_puts(Monitor *mon, const char
>> *str)
>>   {
>>       char c;
>>
>> +    /* if mux do not put in any thing to buffer */
>> +    if (mon->mux_out) {
>> +        return;
>> +    }
>> +
>>       for(;;) {
>> -        assert(mon->outbuf_index < sizeof(mon->outbuf) - 1);
>> +        if (mon->outbuf_index >= sizeof(mon->outbuf) - 1) {
>> +            /* when buffer is full, flush it and retry. If buffer is
>> bigger, more
>> +               work can be postponed. */
>> +            monitor_flush(mon);
>> +            usleep(1);
>> +            continue;
>> +        }
>>           c = *str++;
>>           if (c == '\0')
>>               break;
>>           if (c == '\n')
>>               mon->outbuf[mon->outbuf_index++] = '\r';
>>           mon->outbuf[mon->outbuf_index++] = c;
>> -        if (mon->outbuf_index >= (sizeof(mon->outbuf) - 1)
>> -            || c == '\n')
>> +        if (c == '\n') {
>>               monitor_flush(mon);
>> +        }
>>       }
>>   }
>>
>>> Commit f628926bb423fa8a7e0b114511400ea9df38b76a changed monitor_flush()
>>> to retry on qemu_chr_fe_write() errors. However, the Monitor's output
>>> buffer can keep growing while the retry is not issued and this can
>>> cause the buffer to overflow.
>>>
>>> To reproduce this issue, just start qemu and type on the Monitor:
>>>
>>> (qemu) ?
>>>
>>> This will cause the assertion to trig.
>>>
>>> To fix this problem this commit makes the Monitor buffer dynamic,
>>> which means that it can grow as much as needed.
>>>
>>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>>> ---
>>>    monitor.c | 42 +++++++++++++++++++++++++-----------------
>>>    1 file changed, 25 insertions(+), 17 deletions(-)
>>>
>>
>
>
Gerd Hoffmann - April 2, 2013, 9:18 a.m.
On 03/27/13 07:45, Wenchao Xia wrote:
> Hi, Luiz
>   Personally I hope reduce the dynamic allocated buffer which brings
> fragments and unexpected memory grow.

It's a tradeoff.  We can reduce the dynamic allocation, by simply
reusing the qstring instead of allocation a new one (after
complete/partial write).  At the cost of wasting some memory.

Given that qstrings grow expotentially I think the dynamic allocation
overhead isn't that bad.

> Instead, how about sacrifice
> some time to wait output complete, since monitor is not time critical?

The watch added recently will fire when there is space for writing.
Right now it will write out the next chunk from the buffer.

In theory we could change the monitor code to continue generating the
response when there is more output space, so we don't have to buffer it
in the first place.  But I think the complexity simply isn't worth it
for the memory savings we can gain.

cheers,
  Gerd

Patch

--- a/monitor.c
+++ b/monitor.c
@@ -293,17 +293,28 @@  static void monitor_puts(Monitor *mon, const char
*str)
 {
     char c;

+    /* if mux do not put in any thing to buffer */
+    if (mon->mux_out) {
+        return;
+    }
+
     for(;;) {
-        assert(mon->outbuf_index < sizeof(mon->outbuf) - 1);
+        if (mon->outbuf_index >= sizeof(mon->outbuf) - 1) {
+            /* when buffer is full, flush it and retry. If buffer is
bigger, more
+               work can be postponed. */
+            monitor_flush(mon);
+            usleep(1);
+            continue;
+        }
         c = *str++;
         if (c == '\0')
             break;
         if (c == '\n')
             mon->outbuf[mon->outbuf_index++] = '\r';
         mon->outbuf[mon->outbuf_index++] = c;
-        if (mon->outbuf_index >= (sizeof(mon->outbuf) - 1)
-            || c == '\n')
+        if (c == '\n') {
             monitor_flush(mon);
+        }
     }