| Message ID | 1521645911-13336-1-git-send-email-i.maximets@samsung.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [ovs-dev] netdev-dpdk: Limit rate of DPDK logs. | expand |
Ilya Maximets <i.maximets@samsung.com> writes: > DPDK could produce huge amount of logs. For example, in case of > exhausting of a mempool in vhost-user port, following message will be > printed on each call to 'rte_vhost_dequeue_burst()': > > |ERR|VHOST_DATA: Failed to allocate memory for mbuf. > > These messages are increasing ovs-vswitchd.log size extremely fast > making it unreadable and non-parsable by a common linux utils like > grep, less etc. Moreover continuously growing log could exhaust the > HDD space in a few hours breaking normal operation of the whole system. > > To avoid such issues, DPDK log rate limited to 600 messages per minute. > This value is high, because we still want to see many big logs like > vhost-user configuration sequence or debug messages, if debug enabled > in DPDK. > > Signed-off-by: Ilya Maximets <i.maximets@samsung.com> > --- LGTM. Acked-by: Aaron Conole <aconole@redhat.com>
On 21.03.2018 18:37, Ben Pfaff wrote: > On Wed, Mar 21, 2018 at 06:25:11PM +0300, Ilya Maximets wrote: >> DPDK could produce huge amount of logs. For example, in case of >> exhausting of a mempool in vhost-user port, following message will be >> printed on each call to 'rte_vhost_dequeue_burst()': >> >> |ERR|VHOST_DATA: Failed to allocate memory for mbuf. >> >> These messages are increasing ovs-vswitchd.log size extremely fast >> making it unreadable and non-parsable by a common linux utils like >> grep, less etc. Moreover continuously growing log could exhaust the >> HDD space in a few hours breaking normal operation of the whole system. >> >> To avoid such issues, DPDK log rate limited to 600 messages per minute. >> This value is high, because we still want to see many big logs like >> vhost-user configuration sequence or debug messages, if debug enabled >> in DPDK. >> >> Signed-off-by: Ilya Maximets <i.maximets@samsung.com> > > Would it make sense to rate-limit each log level separately, so that a > flood of DBG messages can't starve out ERR messages? Hmm. Good point. I don't think that we need to have separate rl for each level, but it could be useful to handle "debug" level apart of other logs. I'll send v2 with that change. Best regards, Ilya Maximets.
diff --git a/lib/dpdk.c b/lib/dpdk.c index 3f5a55f..0a353f8 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -272,20 +272,21 @@ static ssize_t dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size) { char *str = xmemdup0(buf, size); + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(600, 600); switch (rte_log_cur_msg_loglevel()) { case RTE_LOG_DEBUG: - VLOG_DBG("%s", str); + VLOG_DBG_RL(&rl, "%s", str); break; case RTE_LOG_INFO: case RTE_LOG_NOTICE: - VLOG_INFO("%s", str); + VLOG_INFO_RL(&rl, "%s", str); break; case RTE_LOG_WARNING: - VLOG_WARN("%s", str); + VLOG_WARN_RL(&rl, "%s", str); break; case RTE_LOG_ERR: - VLOG_ERR("%s", str); + VLOG_ERR_RL(&rl, "%s", str); break; case RTE_LOG_CRIT: case RTE_LOG_ALERT:
DPDK could produce huge amount of logs. For example, in case of exhausting of a mempool in vhost-user port, following message will be printed on each call to 'rte_vhost_dequeue_burst()': |ERR|VHOST_DATA: Failed to allocate memory for mbuf. These messages are increasing ovs-vswitchd.log size extremely fast making it unreadable and non-parsable by a common linux utils like grep, less etc. Moreover continuously growing log could exhaust the HDD space in a few hours breaking normal operation of the whole system. To avoid such issues, DPDK log rate limited to 600 messages per minute. This value is high, because we still want to see many big logs like vhost-user configuration sequence or debug messages, if debug enabled in DPDK. Signed-off-by: Ilya Maximets <i.maximets@samsung.com> --- lib/dpdk.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)