[RFC,3/3] docs: document virtio-balloon stats

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 docs/virtio-balloon-stats.txt | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 docs/virtio-balloon-stats.txt

On Tue, Dec 04, 2012 at 01:04:48PM -0200, Luiz Capitulino wrote:
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  docs/virtio-balloon-stats.txt | 73 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 73 insertions(+)
>  create mode 100644 docs/virtio-balloon-stats.txt
> 
> diff --git a/docs/virtio-balloon-stats.txt b/docs/virtio-balloon-stats.txt
> new file mode 100644
> index 0000000..7e7ddc4
> --- /dev/null
> +++ b/docs/virtio-balloon-stats.txt
> @@ -0,0 +1,73 @@
> +virtio balloon memory statistics
> +================================
> +
> +The virtio balloon driver supports guest memory statistics reporting. These
> +statistics are available to QEMU users as QOM (QEMU Obejct Model) device
> +properties via a polling mechanism.
> +
> +Basically, clients have to enable polling. Then they can query the available
> +statistics.
> +
> +There are two control properties and six memory statistics from the guest.
> +
> +The control properties are:
> +
> + o stats-polling-interval: a value greater than zero enables polling
> +   in the specified interval (in seconds). When value equals zero,
> +   polling is disabled. If polling is already enabled and a value
> +   greater than zero is written, the polling interval time is changed
> +
> + o stats-last-update: last stats update timestamp, in seconds
> +
> +The memory statistics are:
> +
> + o stat-swap-in
> + o stat-swap-out
> + o stat-major-faults
> + o stat-minor-faults
> + o stat-free-memory
> + o stat-total-memory
> +
> +All values are in bytes. A value of -1 means that the statistic isn't
> +available right now.
> +
> +Here are a few examples. The virtio-balloon device is assumed to be in the
> +'/machine/peripheral-anon/device[1]' QOM path.
> +
> +Enable polling with 2 seconds interval:
> +
> +{ "execute": "qom-set",
> +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> +			 "property": "stats-polling-interval", "value": 2 } }
> +
> +{ "return": {} }
> +
> +Change polling to 10 seconds:
> +
> +{ "execute": "qom-set",
> +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> +			 "property": "stats-polling-interval", "value": 10 } }
> +
> +{ "return": {} }
> +
> +Get last update timestamp and free memory stat:
> +
> +{ "execute": "qom-get",
> +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> +  "property": "stats-last-update" } }
> +
> +{ "return": 1354629634 }
> +
> +{ "execute": "qom-get",
> +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> +  "property": "stat-free-memory" } }
> +
> +{ "return": 845115392 }
> +
> +Disable polling:
> +
> +{ "execute": "qom-set",
> +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> +			 "property": "stats-polling-interval", "value": 0 } }
> +
> +{ "return": {} }


What sort of performance implications are there for enabling polling of
virtio stats. Is it the kind of thing that it is reasonable to just
enable for all VMs on a 10 second interval, so we'll always have stats
available without having to have thought to enable them ahead of time ?

eg, the use case I'm wondering is that someone comes along and just
runs   'virsh memstats $DOMAIN' and wants to see the latest data
right away. 

I'm not suggesting that libvirt would be actually asking QEMU for the
stats every 10 seconds. Only that libvirt tells QEMU to collect them.
Then libvirt can just ask for them whenver someone wants them.

Daniel

On Thu, 6 Dec 2012 13:24:11 +0000
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Tue, Dec 04, 2012 at 01:04:48PM -0200, Luiz Capitulino wrote:
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > ---
> >  docs/virtio-balloon-stats.txt | 73 +++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 73 insertions(+)
> >  create mode 100644 docs/virtio-balloon-stats.txt
> > 
> > diff --git a/docs/virtio-balloon-stats.txt b/docs/virtio-balloon-stats.txt
> > new file mode 100644
> > index 0000000..7e7ddc4
> > --- /dev/null
> > +++ b/docs/virtio-balloon-stats.txt
> > @@ -0,0 +1,73 @@
> > +virtio balloon memory statistics
> > +================================
> > +
> > +The virtio balloon driver supports guest memory statistics reporting. These
> > +statistics are available to QEMU users as QOM (QEMU Obejct Model) device
> > +properties via a polling mechanism.
> > +
> > +Basically, clients have to enable polling. Then they can query the available
> > +statistics.
> > +
> > +There are two control properties and six memory statistics from the guest.
> > +
> > +The control properties are:
> > +
> > + o stats-polling-interval: a value greater than zero enables polling
> > +   in the specified interval (in seconds). When value equals zero,
> > +   polling is disabled. If polling is already enabled and a value
> > +   greater than zero is written, the polling interval time is changed
> > +
> > + o stats-last-update: last stats update timestamp, in seconds
> > +
> > +The memory statistics are:
> > +
> > + o stat-swap-in
> > + o stat-swap-out
> > + o stat-major-faults
> > + o stat-minor-faults
> > + o stat-free-memory
> > + o stat-total-memory
> > +
> > +All values are in bytes. A value of -1 means that the statistic isn't
> > +available right now.
> > +
> > +Here are a few examples. The virtio-balloon device is assumed to be in the
> > +'/machine/peripheral-anon/device[1]' QOM path.
> > +
> > +Enable polling with 2 seconds interval:
> > +
> > +{ "execute": "qom-set",
> > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > +			 "property": "stats-polling-interval", "value": 2 } }
> > +
> > +{ "return": {} }
> > +
> > +Change polling to 10 seconds:
> > +
> > +{ "execute": "qom-set",
> > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > +			 "property": "stats-polling-interval", "value": 10 } }
> > +
> > +{ "return": {} }
> > +
> > +Get last update timestamp and free memory stat:
> > +
> > +{ "execute": "qom-get",
> > +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > +  "property": "stats-last-update" } }
> > +
> > +{ "return": 1354629634 }
> > +
> > +{ "execute": "qom-get",
> > +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > +  "property": "stat-free-memory" } }
> > +
> > +{ "return": 845115392 }
> > +
> > +Disable polling:
> > +
> > +{ "execute": "qom-set",
> > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > +			 "property": "stats-polling-interval", "value": 0 } }
> > +
> > +{ "return": {} }
> 
> 
> What sort of performance implications are there for enabling polling of
> virtio stats. Is it the kind of thing that it is reasonable to just
> enable for all VMs on a 10 second interval, so we'll always have stats
> available without having to have thought to enable them ahead of time ?

I can't think of any performance implications. Would be nice to get a
second opinion from the CC'ed people though.

> eg, the use case I'm wondering is that someone comes along and just
> runs   'virsh memstats $DOMAIN' and wants to see the latest data
> right away. 
> 
> I'm not suggesting that libvirt would be actually asking QEMU for the
> stats every 10 seconds. Only that libvirt tells QEMU to collect them.
> Then libvirt can just ask for them whenver someone wants them.

Note that once you enable polling, the balloon driver will immediately make
a request to the guest, that is, it won't wait the specified time interval to
send the first request.

So, the first call to virsh memstats could start polling and also poll for it
(although you do need to be prepared for the case where the guest doesn't
respond).

Also, you could consider adding the time interval in libvirt's API and
virsh memstats.

On Thu, Dec 06, 2012 at 01:31:09PM -0200, Luiz Capitulino wrote:
> On Thu, 6 Dec 2012 13:24:11 +0000
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > On Tue, Dec 04, 2012 at 01:04:48PM -0200, Luiz Capitulino wrote:
> > > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > > ---
> > >  docs/virtio-balloon-stats.txt | 73 +++++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 73 insertions(+)
> > >  create mode 100644 docs/virtio-balloon-stats.txt
> > > 
> > > diff --git a/docs/virtio-balloon-stats.txt b/docs/virtio-balloon-stats.txt
> > > new file mode 100644
> > > index 0000000..7e7ddc4
> > > --- /dev/null
> > > +++ b/docs/virtio-balloon-stats.txt
> > > @@ -0,0 +1,73 @@
> > > +virtio balloon memory statistics
> > > +================================
> > > +
> > > +The virtio balloon driver supports guest memory statistics reporting. These
> > > +statistics are available to QEMU users as QOM (QEMU Obejct Model) device
> > > +properties via a polling mechanism.
> > > +
> > > +Basically, clients have to enable polling. Then they can query the available
> > > +statistics.
> > > +
> > > +There are two control properties and six memory statistics from the guest.
> > > +
> > > +The control properties are:
> > > +
> > > + o stats-polling-interval: a value greater than zero enables polling
> > > +   in the specified interval (in seconds). When value equals zero,
> > > +   polling is disabled. If polling is already enabled and a value
> > > +   greater than zero is written, the polling interval time is changed
> > > +
> > > + o stats-last-update: last stats update timestamp, in seconds
> > > +
> > > +The memory statistics are:
> > > +
> > > + o stat-swap-in
> > > + o stat-swap-out
> > > + o stat-major-faults
> > > + o stat-minor-faults
> > > + o stat-free-memory
> > > + o stat-total-memory
> > > +
> > > +All values are in bytes. A value of -1 means that the statistic isn't
> > > +available right now.
> > > +
> > > +Here are a few examples. The virtio-balloon device is assumed to be in the
> > > +'/machine/peripheral-anon/device[1]' QOM path.
> > > +
> > > +Enable polling with 2 seconds interval:
> > > +
> > > +{ "execute": "qom-set",
> > > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > > +			 "property": "stats-polling-interval", "value": 2 } }
> > > +
> > > +{ "return": {} }
> > > +
> > > +Change polling to 10 seconds:
> > > +
> > > +{ "execute": "qom-set",
> > > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > > +			 "property": "stats-polling-interval", "value": 10 } }
> > > +
> > > +{ "return": {} }
> > > +
> > > +Get last update timestamp and free memory stat:
> > > +
> > > +{ "execute": "qom-get",
> > > +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > > +  "property": "stats-last-update" } }
> > > +
> > > +{ "return": 1354629634 }
> > > +
> > > +{ "execute": "qom-get",
> > > +  "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > > +  "property": "stat-free-memory" } }
> > > +
> > > +{ "return": 845115392 }
> > > +
> > > +Disable polling:
> > > +
> > > +{ "execute": "qom-set",
> > > +             "arguments": { "path": "/machine/peripheral-anon/device[1]",
> > > +			 "property": "stats-polling-interval", "value": 0 } }
> > > +
> > > +{ "return": {} }
> > 
> > 
> > What sort of performance implications are there for enabling polling of
> > virtio stats. Is it the kind of thing that it is reasonable to just
> > enable for all VMs on a 10 second interval, so we'll always have stats
> > available without having to have thought to enable them ahead of time ?
> 
> I can't think of any performance implications. Would be nice to get a
> second opinion from the CC'ed people though.

Pushing/popping/processing one vq entry every 10 seconds should be
virtually unnoticeable given that virtio-net/blk do this much more frequently
with much more processing overhead per entry on a relatively idle guest. So
performance-wise, I don't think it's an issue. As to whether or not it
*should* be enabled by default I'm not so sure, but if it actually simplifies
things on that end I'd say it's worth it if the alternatives are
cumbersome.

> 
> > eg, the use case I'm wondering is that someone comes along and just
> > runs   'virsh memstats $DOMAIN' and wants to see the latest data
> > right away. 
> > 
> > I'm not suggesting that libvirt would be actually asking QEMU for the
> > stats every 10 seconds. Only that libvirt tells QEMU to collect them.
> > Then libvirt can just ask for them whenver someone wants them.
> 
> Note that once you enable polling, the balloon driver will immediately make
> a request to the guest, that is, it won't wait the specified time interval to
> send the first request.
> 
> So, the first call to virsh memstats could start polling and also poll for it
> (although you do need to be prepared for the case where the guest doesn't
> respond).
> 
> Also, you could consider adding the time interval in libvirt's API and
> virsh memstats.
>

> +The memory statistics are:
> +
> + o stat-swap-in
> + o stat-swap-out
> + o stat-major-faults
> + o stat-minor-faults
> + o stat-free-memory
> + o stat-total-memory

I want to implement an automatic ballooning policy, so I need to
know the amount of free RAM inside the guest.

But it seem that the balloon driver reports free ram without accounting
buffers and cache.

Is it possible to extent those statistic to include buffer/cache values?

On Fri, 7 Dec 2012 05:30:17 +0000
Dietmar Maurer <dietmar@proxmox.com> wrote:

> > +The memory statistics are:
> > +
> > + o stat-swap-in
> > + o stat-swap-out
> > + o stat-major-faults
> > + o stat-minor-faults
> > + o stat-free-memory
> > + o stat-total-memory
> 
> I want to implement an automatic ballooning policy, so I need to
> know the amount of free RAM inside the guest.
> 
> But it seem that the balloon driver reports free ram without accounting
> buffers and cache.
> 
> Is it possible to extent those statistic to include buffer/cache values?

Yes, should be possible but the kernel virtio-balloon driver has to be
extended too (and I suspect this kind of change goes into the kernel first).

> > Is it possible to extent those statistic to include buffer/cache values?
> 
> Yes, should be possible but the kernel virtio-balloon driver has to be
> extended too (and I suspect this kind of change goes into the kernel first).

The current statistics are quite useless without that information, and
I guess it is impossible to such changes upstream within a reasonable time?

So what is the purpose of that 'stat-free-memory'? Can we change that to
return the amount of free RAM including buffers?

On Fri, 7 Dec 2012 15:01:36 +0000
Dietmar Maurer <dietmar@proxmox.com> wrote:

> > > Is it possible to extent those statistic to include buffer/cache values?
> > 
> > Yes, should be possible but the kernel virtio-balloon driver has to be
> > extended too (and I suspect this kind of change goes into the kernel first).
> 
> The current statistics are quite useless without that information, and

Maybe it's not possible to implement the feature you want to implement, but
I do get a ping once a while from people asking for these stats. Some of them
are people running virsh dommemstat which also makes use of that information.

> I guess it is impossible to such changes upstream within a reasonable time?

In the kernel, yes it would take a while.

> So what is the purpose of that 'stat-free-memory'? Can we change that to
> return the amount of free RAM including buffers?

That's kernel work, qemu is merely reporting what the kernel is making
available. But giving that you'll touch the kernel anyway, it's better
to add a new stat.

There are two other alternatives:

 1. add a query-memory-stats command to qemu-ga

     Pros: should be quicker and easier to modify/extend
     Cons: has the implications of running this from the guest user-space

 2. wait for the (kernel-based) auto-ballooning feature, which is on the
    works

     Pros: you do nothing and it's going to be completely automatic
     Cons: this is unlikely to be ready for the short term

>  2. wait for the (kernel-based) auto-ballooning feature, which is on the
>     works

Do you have more information on that? Any links?

>>That's kernel work, qemu is merely reporting what the kernel is making 
>>available. But giving that you'll touch the kernel anyway, it's better 
>>to add a new stat. 
>>
>>There are two other alternatives: 
>>
>>1. add a query-memory-stats command to qemu-ga 
>>
>>Pros: should be quicker and easier to modify/extend 
>>Cons: has the implications of running this from the guest user-space 

Hello,
I would like also extend balloon stats to retrieve guest buffer memory, It can help a lot for doing accurate balloning.

Using qemu-ga is easy for linux (we can read /proc/meminfo), but with windows guest is impossible currently.
(we need to query wmi counter, and executing process is no possible with qga)
Adding the new wmi counter ("System Cache Resident Bytes") is quite easy in windows balloon driver.

Best Regards,

Alexandre Derumier


--- Mail original ----- 

De: "Luiz Capitulino" <lcapitulino@redhat.com> 
À: "Dietmar Maurer" <dietmar@proxmox.com> 
Cc: aliguori@us.ibm.com, agl@us.ibm.com, qemu-devel@nongnu.org, mdroth@linux.vnet.ibm.com 
Envoyé: Vendredi 7 Décembre 2012 19:54:44 
Objet: Re: [Qemu-devel] [RFC 3/3] docs: document virtio-balloon stats 

On Fri, 7 Dec 2012 15:01:36 +0000 
Dietmar Maurer <dietmar@proxmox.com> wrote: 

> > > Is it possible to extent those statistic to include buffer/cache values? 
> > 
> > Yes, should be possible but the kernel virtio-balloon driver has to be 
> > extended too (and I suspect this kind of change goes into the kernel first). 
> 
> The current statistics are quite useless without that information, and 

Maybe it's not possible to implement the feature you want to implement, but 
I do get a ping once a while from people asking for these stats. Some of them 
are people running virsh dommemstat which also makes use of that information. 

> I guess it is impossible to such changes upstream within a reasonable time? 

In the kernel, yes it would take a while. 

> So what is the purpose of that 'stat-free-memory'? Can we change that to 
> return the amount of free RAM including buffers? 

That's kernel work, qemu is merely reporting what the kernel is making 
available. But giving that you'll touch the kernel anyway, it's better 
to add a new stat. 

There are two other alternatives: 

1. add a query-memory-stats command to qemu-ga 

Pros: should be quicker and easier to modify/extend 
Cons: has the implications of running this from the guest user-space 

2. wait for the (kernel-based) auto-ballooning feature, which is on the 
works 

Pros: you do nothing and it's going to be completely automatic 
Cons: this is unlikely to be ready for the short term

On Sat, 8 Dec 2012 06:26:05 +0000
Dietmar Maurer <dietmar@proxmox.com> wrote:

> >  2. wait for the (kernel-based) auto-ballooning feature, which is on the
> >     works
> 
> Do you have more information on that? Any links?

No, no links. What's being worked right now is a notification mechanism for
the kernel to inform user-space about memory pressure:

 http://marc.info/?l=linux-mm&m=135513372205134&w=2

Then we can add that to qemu for auto-inflate and change the virtio-balloon
driver to do auto-deflate.

> > >  2. wait for the (kernel-based) auto-ballooning feature, which is on the
> > >     works
> >
> > Do you have more information on that? Any links?
> 
> No, no links. What's being worked right now is a notification mechanism for
> the kernel to inform user-space about memory pressure:
> 
>  http://marc.info/?l=linux-mm&m=135513372205134&w=2
> 
> Then we can add that to qemu for auto-inflate and change the virtio-balloon
> driver to do auto-deflate.

Many thanks for that information.