| Message ID | 1516969207-17326-1-git-send-email-jan.scheurich@ericsson.com |
|---|---|
| Headers | show |
| Series | dpif-netdev: Detailed PMD performance metrics and supervision | expand |
Gentle reminder to review this series which unfortunately missed the 2.9 deadline. I checked and the patches still apply on today's master. So far I have received one comment from Billy https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343808.html Thanks, Jan > -----Original Message----- > From: Jan Scheurich > Sent: Friday, 26 January, 2018 13:20 > To: dev@openvswitch.org > Cc: ktraynor@redhat.com; ian.stokes@intel.com; i.maximets@samsung.com; billy.o.mahony@intel.com; Jan Scheurich > <jan.scheurich@ericsson.com> > Subject: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and supervision > > The run-time performance of PMDs is often difficult to understand and > trouble-shoot. The existing PMD statistics counters only provide a coarse > grained average picture. At packet rates of several Mpps sporadic drops of > packet bursts happen at sub-millisecond time scales and are impossible to > capture and analyze with existing tools. > > This patch collects a large number of important PMD performance metrics > per PMD iteration, maintaining histograms and circular histories for > iteration metrics and millisecond averages. To capture sporadic drop > events, the patch set can be configured to monitor iterations for suspicious > metrics and to log the neighborhood of such iterations for off-line analysis. > > The extra cost for the performance metric collection and the supervision has > been measured to be in the order of 1% compared to the base commit in a PVP > setup with L3 pipeline over VXLAN tunnels. For that reason the metrics > collection is disabled by default and can be enabled at run-time through > configuration. > > v7 -> v8: > * Rebased on to master (commit 4e99b70df) > * Implemented comments from Ilya Maximets and Billy O'Mahony. > * Replaced netdev_rxq_length() introduced in v7 by optional out > parameter for the remaining rx queue len in netdev_rxq_recv(). > * Fixed thread synchronization issues in clearing PMD stats: > - Use mutex to control whether to clear from main thread directly > or in PMD at start of next iteration. > - Use mutex to prevent concurrent clearing and printing of metrics. > * Added tx packet and batch stats to pmd-perf-show output. > * Delay warning for suspicious iteration to the iteration in which > we also log the neighborhood to not pollute the logged iteration > stats with logging costs. > * Corrected the exact number of iterations logged before and after a > supicious iteration. > * Introduced options -e and -ne in pmd-perf-log-set to control whether > to *extend* the range of logged iterations when additional supicious > iterations are detected before the scheduled end of logging interval > is reached. > * Exclude logging cycles from the iteration stats to avoid confusing > ghost peaks. > * Performance impact compared to master less than 1% even with > supervision enabled. > > v5 -> v7: > * Rebased on to dpdk_merge (commit e666668) > - New base contains earlier refactoring parts of series. > * Implemented comments from Ilya Maximets and Billy O'Mahony. > * Replaced piggybacking qlen on dp_packet_batch with a new netdev API > netdev_rxq_length(). > * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). > * Fixed bug in reporting datapath stats. > * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes > returns bogus in the upper 16 bits of the uint32_t return value. > > v4 -> v5: > * Rebased to master (commit e9de6c0) > * Implemented comments from Aaron Conole and Darrel Ball > > v3 -> v4: > * Rebased to master (commit 4d0a31b) > - Reverting changes to struct dp_netdev_pmd_thread. > * Make metrics collection configurable. > * Several bugfixes. > > v2 -> v3: > * Rebased to OVS master (commit 3728b3b). > * Non-trivial adaptation to struct dp_netdev_pmd_thread. > - refactored in commit a807c157 (Bhanu). > * No other changes compared to v2. > > v1 -> v2: > * Rebased to OVS master (commit 7468ec788). > * No other changes compared to v1. > > > Jan Scheurich (3): > netdev: Add optional qfill output parameter to rxq_recv() > dpif-netdev: Detailed performance stats for PMDs > dpif-netdev: Detection and logging of suspicious PMD iterations > > NEWS | 5 + > lib/automake.mk | 1 + > lib/dpif-netdev-perf.c | 551 +++++++++++++++++++++++++++++++++++++++++++- > lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- > lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ > lib/dpif-netdev.c | 187 ++++++++++++++- > lib/netdev-bsd.c | 8 +- > lib/netdev-dpdk.c | 25 +- > lib/netdev-dummy.c | 8 +- > lib/netdev-linux.c | 7 +- > lib/netdev-provider.h | 7 +- > lib/netdev.c | 5 +- > lib/netdev.h | 3 +- > manpages.mk | 2 + > vswitchd/ovs-vswitchd.8.in | 27 +-- > vswitchd/vswitch.xml | 12 + > 16 files changed, 1302 insertions(+), 62 deletions(-) > create mode 100644 lib/dpif-netdev-unixctl.man > > -- > 1.9.1
Hi Jan, Everyone is probably reviewed-out :) I'm happy to ack once the +='s are fixed. Regards, Billy. > -----Original Message----- > From: Jan Scheurich [mailto:jan.scheurich@ericsson.com] > Sent: Tuesday, February 13, 2018 4:04 PM > To: dev@openvswitch.org > Cc: ktraynor@redhat.com; Stokes, Ian <ian.stokes@intel.com>; > i.maximets@samsung.com; O Mahony, Billy <billy.o.mahony@intel.com> > Subject: RE: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and > supervision > > Gentle reminder to review this series which unfortunately missed the 2.9 > deadline. > > I checked and the patches still apply on today's master. > So far I have received one comment from Billy > https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343808.html > > Thanks, Jan > > > -----Original Message----- > > From: Jan Scheurich > > Sent: Friday, 26 January, 2018 13:20 > > To: dev@openvswitch.org > > Cc: ktraynor@redhat.com; ian.stokes@intel.com; i.maximets@samsung.com; > > billy.o.mahony@intel.com; Jan Scheurich <jan.scheurich@ericsson.com> > > Subject: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics > > and supervision > > > > The run-time performance of PMDs is often difficult to understand and > > trouble-shoot. The existing PMD statistics counters only provide a > > coarse grained average picture. At packet rates of several Mpps > > sporadic drops of packet bursts happen at sub-millisecond time scales > > and are impossible to capture and analyze with existing tools. > > > > This patch collects a large number of important PMD performance > > metrics per PMD iteration, maintaining histograms and circular > > histories for iteration metrics and millisecond averages. To capture > > sporadic drop events, the patch set can be configured to monitor > > iterations for suspicious metrics and to log the neighborhood of such > iterations for off-line analysis. > > > > The extra cost for the performance metric collection and the > > supervision has been measured to be in the order of 1% compared to the > > base commit in a PVP setup with L3 pipeline over VXLAN tunnels. For > > that reason the metrics collection is disabled by default and can be > > enabled at run-time through configuration. > > > > v7 -> v8: > > * Rebased on to master (commit 4e99b70df) > > * Implemented comments from Ilya Maximets and Billy O'Mahony. > > * Replaced netdev_rxq_length() introduced in v7 by optional out > > parameter for the remaining rx queue len in netdev_rxq_recv(). > > * Fixed thread synchronization issues in clearing PMD stats: > > - Use mutex to control whether to clear from main thread directly > > or in PMD at start of next iteration. > > - Use mutex to prevent concurrent clearing and printing of metrics. > > * Added tx packet and batch stats to pmd-perf-show output. > > * Delay warning for suspicious iteration to the iteration in which > > we also log the neighborhood to not pollute the logged iteration > > stats with logging costs. > > * Corrected the exact number of iterations logged before and after a > > supicious iteration. > > * Introduced options -e and -ne in pmd-perf-log-set to control whether > > to *extend* the range of logged iterations when additional supicious > > iterations are detected before the scheduled end of logging interval > > is reached. > > * Exclude logging cycles from the iteration stats to avoid confusing > > ghost peaks. > > * Performance impact compared to master less than 1% even with > > supervision enabled. > > > > v5 -> v7: > > * Rebased on to dpdk_merge (commit e666668) > > - New base contains earlier refactoring parts of series. > > * Implemented comments from Ilya Maximets and Billy O'Mahony. > > * Replaced piggybacking qlen on dp_packet_batch with a new netdev API > > netdev_rxq_length(). > > * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). > > * Fixed bug in reporting datapath stats. > > * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes > > returns bogus in the upper 16 bits of the uint32_t return value. > > > > v4 -> v5: > > * Rebased to master (commit e9de6c0) > > * Implemented comments from Aaron Conole and Darrel Ball > > > > v3 -> v4: > > * Rebased to master (commit 4d0a31b) > > - Reverting changes to struct dp_netdev_pmd_thread. > > * Make metrics collection configurable. > > * Several bugfixes. > > > > v2 -> v3: > > * Rebased to OVS master (commit 3728b3b). > > * Non-trivial adaptation to struct dp_netdev_pmd_thread. > > - refactored in commit a807c157 (Bhanu). > > * No other changes compared to v2. > > > > v1 -> v2: > > * Rebased to OVS master (commit 7468ec788). > > * No other changes compared to v1. > > > > > > Jan Scheurich (3): > > netdev: Add optional qfill output parameter to rxq_recv() > > dpif-netdev: Detailed performance stats for PMDs > > dpif-netdev: Detection and logging of suspicious PMD iterations > > > > NEWS | 5 + > > lib/automake.mk | 1 + > > lib/dpif-netdev-perf.c | 551 > +++++++++++++++++++++++++++++++++++++++++++- > > lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- > > lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ > > lib/dpif-netdev.c | 187 ++++++++++++++- > > lib/netdev-bsd.c | 8 +- > > lib/netdev-dpdk.c | 25 +- > > lib/netdev-dummy.c | 8 +- > > lib/netdev-linux.c | 7 +- > > lib/netdev-provider.h | 7 +- > > lib/netdev.c | 5 +- > > lib/netdev.h | 3 +- > > manpages.mk | 2 + > > vswitchd/ovs-vswitchd.8.in | 27 +-- > > vswitchd/vswitch.xml | 12 + > > 16 files changed, 1302 insertions(+), 62 deletions(-) create mode > > 100644 lib/dpif-netdev-unixctl.man > > > > -- > > 1.9.1
Hello Jan. I have a question: Have you tested the performance difference between netdev_rxq_length() and rxq_recv() parameter approaches to get the rx queue size? Best regards, Ilya Maximets. On 13.02.2018 19:03, Jan Scheurich wrote: > Gentle reminder to review this series which unfortunately missed the 2.9 deadline. > > I checked and the patches still apply on today's master. > So far I have received one comment from Billy > https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343808.html > > Thanks, Jan > >> -----Original Message----- >> From: Jan Scheurich >> Sent: Friday, 26 January, 2018 13:20 >> To: dev@openvswitch.org >> Cc: ktraynor@redhat.com; ian.stokes@intel.com; i.maximets@samsung.com; billy.o.mahony@intel.com; Jan Scheurich >> <jan.scheurich@ericsson.com> >> Subject: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and supervision >> >> The run-time performance of PMDs is often difficult to understand and >> trouble-shoot. The existing PMD statistics counters only provide a coarse >> grained average picture. At packet rates of several Mpps sporadic drops of >> packet bursts happen at sub-millisecond time scales and are impossible to >> capture and analyze with existing tools. >> >> This patch collects a large number of important PMD performance metrics >> per PMD iteration, maintaining histograms and circular histories for >> iteration metrics and millisecond averages. To capture sporadic drop >> events, the patch set can be configured to monitor iterations for suspicious >> metrics and to log the neighborhood of such iterations for off-line analysis. >> >> The extra cost for the performance metric collection and the supervision has >> been measured to be in the order of 1% compared to the base commit in a PVP >> setup with L3 pipeline over VXLAN tunnels. For that reason the metrics >> collection is disabled by default and can be enabled at run-time through >> configuration. >> >> v7 -> v8: >> * Rebased on to master (commit 4e99b70df) >> * Implemented comments from Ilya Maximets and Billy O'Mahony. >> * Replaced netdev_rxq_length() introduced in v7 by optional out >> parameter for the remaining rx queue len in netdev_rxq_recv(). >> * Fixed thread synchronization issues in clearing PMD stats: >> - Use mutex to control whether to clear from main thread directly >> or in PMD at start of next iteration. >> - Use mutex to prevent concurrent clearing and printing of metrics. >> * Added tx packet and batch stats to pmd-perf-show output. >> * Delay warning for suspicious iteration to the iteration in which >> we also log the neighborhood to not pollute the logged iteration >> stats with logging costs. >> * Corrected the exact number of iterations logged before and after a >> supicious iteration. >> * Introduced options -e and -ne in pmd-perf-log-set to control whether >> to *extend* the range of logged iterations when additional supicious >> iterations are detected before the scheduled end of logging interval >> is reached. >> * Exclude logging cycles from the iteration stats to avoid confusing >> ghost peaks. >> * Performance impact compared to master less than 1% even with >> supervision enabled. >> >> v5 -> v7: >> * Rebased on to dpdk_merge (commit e666668) >> - New base contains earlier refactoring parts of series. >> * Implemented comments from Ilya Maximets and Billy O'Mahony. >> * Replaced piggybacking qlen on dp_packet_batch with a new netdev API >> netdev_rxq_length(). >> * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). >> * Fixed bug in reporting datapath stats. >> * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes >> returns bogus in the upper 16 bits of the uint32_t return value. >> >> v4 -> v5: >> * Rebased to master (commit e9de6c0) >> * Implemented comments from Aaron Conole and Darrel Ball >> >> v3 -> v4: >> * Rebased to master (commit 4d0a31b) >> - Reverting changes to struct dp_netdev_pmd_thread. >> * Make metrics collection configurable. >> * Several bugfixes. >> >> v2 -> v3: >> * Rebased to OVS master (commit 3728b3b). >> * Non-trivial adaptation to struct dp_netdev_pmd_thread. >> - refactored in commit a807c157 (Bhanu). >> * No other changes compared to v2. >> >> v1 -> v2: >> * Rebased to OVS master (commit 7468ec788). >> * No other changes compared to v1. >> >> >> Jan Scheurich (3): >> netdev: Add optional qfill output parameter to rxq_recv() >> dpif-netdev: Detailed performance stats for PMDs >> dpif-netdev: Detection and logging of suspicious PMD iterations >> >> NEWS | 5 + >> lib/automake.mk | 1 + >> lib/dpif-netdev-perf.c | 551 +++++++++++++++++++++++++++++++++++++++++++- >> lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- >> lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ >> lib/dpif-netdev.c | 187 ++++++++++++++- >> lib/netdev-bsd.c | 8 +- >> lib/netdev-dpdk.c | 25 +- >> lib/netdev-dummy.c | 8 +- >> lib/netdev-linux.c | 7 +- >> lib/netdev-provider.h | 7 +- >> lib/netdev.c | 5 +- >> lib/netdev.h | 3 +- >> manpages.mk | 2 + >> vswitchd/ovs-vswitchd.8.in | 27 +-- >> vswitchd/vswitch.xml | 12 + >> 16 files changed, 1302 insertions(+), 62 deletions(-) >> create mode 100644 lib/dpif-netdev-unixctl.man >> >> -- >> 1.9.1 > > > >
Consider it done :-) I'll wait with a v9 until I have all review comments implemented. Thanks, Jan > -----Original Message----- > From: O Mahony, Billy [mailto:billy.o.mahony@intel.com] > Sent: Thursday, 15 February, 2018 15:41 > To: Jan Scheurich <jan.scheurich@ericsson.com>; dev@openvswitch.org > Cc: ktraynor@redhat.com; Stokes, Ian <ian.stokes@intel.com>; i.maximets@samsung.com > Subject: RE: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and supervision > > Hi Jan, > > Everyone is probably reviewed-out :) > > I'm happy to ack once the +='s are fixed. > > Regards, > Billy. > > > > > -----Original Message----- > > From: Jan Scheurich [mailto:jan.scheurich@ericsson.com] > > Sent: Tuesday, February 13, 2018 4:04 PM > > To: dev@openvswitch.org > > Cc: ktraynor@redhat.com; Stokes, Ian <ian.stokes@intel.com>; > > i.maximets@samsung.com; O Mahony, Billy <billy.o.mahony@intel.com> > > Subject: RE: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and > > supervision > > > > Gentle reminder to review this series which unfortunately missed the 2.9 > > deadline. > > > > I checked and the patches still apply on today's master. > > So far I have received one comment from Billy > > https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343808.html > > > > Thanks, Jan > > > > > -----Original Message----- > > > From: Jan Scheurich > > > Sent: Friday, 26 January, 2018 13:20 > > > To: dev@openvswitch.org > > > Cc: ktraynor@redhat.com; ian.stokes@intel.com; i.maximets@samsung.com; > > > billy.o.mahony@intel.com; Jan Scheurich <jan.scheurich@ericsson.com> > > > Subject: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics > > > and supervision > > > > > > The run-time performance of PMDs is often difficult to understand and > > > trouble-shoot. The existing PMD statistics counters only provide a > > > coarse grained average picture. At packet rates of several Mpps > > > sporadic drops of packet bursts happen at sub-millisecond time scales > > > and are impossible to capture and analyze with existing tools. > > > > > > This patch collects a large number of important PMD performance > > > metrics per PMD iteration, maintaining histograms and circular > > > histories for iteration metrics and millisecond averages. To capture > > > sporadic drop events, the patch set can be configured to monitor > > > iterations for suspicious metrics and to log the neighborhood of such > > iterations for off-line analysis. > > > > > > The extra cost for the performance metric collection and the > > > supervision has been measured to be in the order of 1% compared to the > > > base commit in a PVP setup with L3 pipeline over VXLAN tunnels. For > > > that reason the metrics collection is disabled by default and can be > > > enabled at run-time through configuration. > > > > > > v7 -> v8: > > > * Rebased on to master (commit 4e99b70df) > > > * Implemented comments from Ilya Maximets and Billy O'Mahony. > > > * Replaced netdev_rxq_length() introduced in v7 by optional out > > > parameter for the remaining rx queue len in netdev_rxq_recv(). > > > * Fixed thread synchronization issues in clearing PMD stats: > > > - Use mutex to control whether to clear from main thread directly > > > or in PMD at start of next iteration. > > > - Use mutex to prevent concurrent clearing and printing of metrics. > > > * Added tx packet and batch stats to pmd-perf-show output. > > > * Delay warning for suspicious iteration to the iteration in which > > > we also log the neighborhood to not pollute the logged iteration > > > stats with logging costs. > > > * Corrected the exact number of iterations logged before and after a > > > supicious iteration. > > > * Introduced options -e and -ne in pmd-perf-log-set to control whether > > > to *extend* the range of logged iterations when additional supicious > > > iterations are detected before the scheduled end of logging interval > > > is reached. > > > * Exclude logging cycles from the iteration stats to avoid confusing > > > ghost peaks. > > > * Performance impact compared to master less than 1% even with > > > supervision enabled. > > > > > > v5 -> v7: > > > * Rebased on to dpdk_merge (commit e666668) > > > - New base contains earlier refactoring parts of series. > > > * Implemented comments from Ilya Maximets and Billy O'Mahony. > > > * Replaced piggybacking qlen on dp_packet_batch with a new netdev API > > > netdev_rxq_length(). > > > * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). > > > * Fixed bug in reporting datapath stats. > > > * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes > > > returns bogus in the upper 16 bits of the uint32_t return value. > > > > > > v4 -> v5: > > > * Rebased to master (commit e9de6c0) > > > * Implemented comments from Aaron Conole and Darrel Ball > > > > > > v3 -> v4: > > > * Rebased to master (commit 4d0a31b) > > > - Reverting changes to struct dp_netdev_pmd_thread. > > > * Make metrics collection configurable. > > > * Several bugfixes. > > > > > > v2 -> v3: > > > * Rebased to OVS master (commit 3728b3b). > > > * Non-trivial adaptation to struct dp_netdev_pmd_thread. > > > - refactored in commit a807c157 (Bhanu). > > > * No other changes compared to v2. > > > > > > v1 -> v2: > > > * Rebased to OVS master (commit 7468ec788). > > > * No other changes compared to v1. > > > > > > > > > Jan Scheurich (3): > > > netdev: Add optional qfill output parameter to rxq_recv() > > > dpif-netdev: Detailed performance stats for PMDs > > > dpif-netdev: Detection and logging of suspicious PMD iterations > > > > > > NEWS | 5 + > > > lib/automake.mk | 1 + > > > lib/dpif-netdev-perf.c | 551 > > +++++++++++++++++++++++++++++++++++++++++++- > > > lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- > > > lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ > > > lib/dpif-netdev.c | 187 ++++++++++++++- > > > lib/netdev-bsd.c | 8 +- > > > lib/netdev-dpdk.c | 25 +- > > > lib/netdev-dummy.c | 8 +- > > > lib/netdev-linux.c | 7 +- > > > lib/netdev-provider.h | 7 +- > > > lib/netdev.c | 5 +- > > > lib/netdev.h | 3 +- > > > manpages.mk | 2 + > > > vswitchd/ovs-vswitchd.8.in | 27 +-- > > > vswitchd/vswitch.xml | 12 + > > > 16 files changed, 1302 insertions(+), 62 deletions(-) create mode > > > 100644 lib/dpif-netdev-unixctl.man > > > > > > -- > > > 1.9.1
Yes, I tested both variants. The rxq_recv() variant is a bit faster. The main reason I settled on the rxq_recv() option is that it is much simpler and has a potential use case also in Billy's upcoming patch for rxq priority polling, where it might add value to know the remaining packets in the queue. BR, Jan > -----Original Message----- > From: Ilya Maximets [mailto:i.maximets@samsung.com] > Sent: Thursday, 15 February, 2018 15:58 > To: Jan Scheurich <jan.scheurich@ericsson.com>; dev@openvswitch.org > Cc: ktraynor@redhat.com; ian.stokes@intel.com; billy.o.mahony@intel.com > Subject: Re: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and supervision > > Hello Jan. > > I have a question: Have you tested the performance difference between > netdev_rxq_length() and rxq_recv() parameter approaches to get the rx queue size? > > Best regards, Ilya Maximets. > > On 13.02.2018 19:03, Jan Scheurich wrote: > > Gentle reminder to review this series which unfortunately missed the 2.9 deadline. > > > > I checked and the patches still apply on today's master. > > So far I have received one comment from Billy > > https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343808.html > > > > Thanks, Jan > > > >> -----Original Message----- > >> From: Jan Scheurich > >> Sent: Friday, 26 January, 2018 13:20 > >> To: dev@openvswitch.org > >> Cc: ktraynor@redhat.com; ian.stokes@intel.com; i.maximets@samsung.com; billy.o.mahony@intel.com; Jan Scheurich > >> <jan.scheurich@ericsson.com> > >> Subject: [PATCH v8 0/3] dpif-netdev: Detailed PMD performance metrics and supervision > >> > >> The run-time performance of PMDs is often difficult to understand and > >> trouble-shoot. The existing PMD statistics counters only provide a coarse > >> grained average picture. At packet rates of several Mpps sporadic drops of > >> packet bursts happen at sub-millisecond time scales and are impossible to > >> capture and analyze with existing tools. > >> > >> This patch collects a large number of important PMD performance metrics > >> per PMD iteration, maintaining histograms and circular histories for > >> iteration metrics and millisecond averages. To capture sporadic drop > >> events, the patch set can be configured to monitor iterations for suspicious > >> metrics and to log the neighborhood of such iterations for off-line analysis. > >> > >> The extra cost for the performance metric collection and the supervision has > >> been measured to be in the order of 1% compared to the base commit in a PVP > >> setup with L3 pipeline over VXLAN tunnels. For that reason the metrics > >> collection is disabled by default and can be enabled at run-time through > >> configuration. > >> > >> v7 -> v8: > >> * Rebased on to master (commit 4e99b70df) > >> * Implemented comments from Ilya Maximets and Billy O'Mahony. > >> * Replaced netdev_rxq_length() introduced in v7 by optional out > >> parameter for the remaining rx queue len in netdev_rxq_recv(). > >> * Fixed thread synchronization issues in clearing PMD stats: > >> - Use mutex to control whether to clear from main thread directly > >> or in PMD at start of next iteration. > >> - Use mutex to prevent concurrent clearing and printing of metrics. > >> * Added tx packet and batch stats to pmd-perf-show output. > >> * Delay warning for suspicious iteration to the iteration in which > >> we also log the neighborhood to not pollute the logged iteration > >> stats with logging costs. > >> * Corrected the exact number of iterations logged before and after a > >> supicious iteration. > >> * Introduced options -e and -ne in pmd-perf-log-set to control whether > >> to *extend* the range of logged iterations when additional supicious > >> iterations are detected before the scheduled end of logging interval > >> is reached. > >> * Exclude logging cycles from the iteration stats to avoid confusing > >> ghost peaks. > >> * Performance impact compared to master less than 1% even with > >> supervision enabled. > >> > >> v5 -> v7: > >> * Rebased on to dpdk_merge (commit e666668) > >> - New base contains earlier refactoring parts of series. > >> * Implemented comments from Ilya Maximets and Billy O'Mahony. > >> * Replaced piggybacking qlen on dp_packet_batch with a new netdev API > >> netdev_rxq_length(). > >> * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). > >> * Fixed bug in reporting datapath stats. > >> * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes > >> returns bogus in the upper 16 bits of the uint32_t return value. > >> > >> v4 -> v5: > >> * Rebased to master (commit e9de6c0) > >> * Implemented comments from Aaron Conole and Darrel Ball > >> > >> v3 -> v4: > >> * Rebased to master (commit 4d0a31b) > >> - Reverting changes to struct dp_netdev_pmd_thread. > >> * Make metrics collection configurable. > >> * Several bugfixes. > >> > >> v2 -> v3: > >> * Rebased to OVS master (commit 3728b3b). > >> * Non-trivial adaptation to struct dp_netdev_pmd_thread. > >> - refactored in commit a807c157 (Bhanu). > >> * No other changes compared to v2. > >> > >> v1 -> v2: > >> * Rebased to OVS master (commit 7468ec788). > >> * No other changes compared to v1. > >> > >> > >> Jan Scheurich (3): > >> netdev: Add optional qfill output parameter to rxq_recv() > >> dpif-netdev: Detailed performance stats for PMDs > >> dpif-netdev: Detection and logging of suspicious PMD iterations > >> > >> NEWS | 5 + > >> lib/automake.mk | 1 + > >> lib/dpif-netdev-perf.c | 551 +++++++++++++++++++++++++++++++++++++++++++- > >> lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- > >> lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ > >> lib/dpif-netdev.c | 187 ++++++++++++++- > >> lib/netdev-bsd.c | 8 +- > >> lib/netdev-dpdk.c | 25 +- > >> lib/netdev-dummy.c | 8 +- > >> lib/netdev-linux.c | 7 +- > >> lib/netdev-provider.h | 7 +- > >> lib/netdev.c | 5 +- > >> lib/netdev.h | 3 +- > >> manpages.mk | 2 + > >> vswitchd/ovs-vswitchd.8.in | 27 +-- > >> vswitchd/vswitch.xml | 12 + > >> 16 files changed, 1302 insertions(+), 62 deletions(-) > >> create mode 100644 lib/dpif-netdev-unixctl.man > >> > >> -- > >> 1.9.1 > > > > > > > >
The run-time performance of PMDs is often difficult to understand and trouble-shoot. The existing PMD statistics counters only provide a coarse grained average picture. At packet rates of several Mpps sporadic drops of packet bursts happen at sub-millisecond time scales and are impossible to capture and analyze with existing tools. This patch collects a large number of important PMD performance metrics per PMD iteration, maintaining histograms and circular histories for iteration metrics and millisecond averages. To capture sporadic drop events, the patch set can be configured to monitor iterations for suspicious metrics and to log the neighborhood of such iterations for off-line analysis. The extra cost for the performance metric collection and the supervision has been measured to be in the order of 1% compared to the base commit in a PVP setup with L3 pipeline over VXLAN tunnels. For that reason the metrics collection is disabled by default and can be enabled at run-time through configuration. v7 -> v8: * Rebased on to master (commit 4e99b70df) * Implemented comments from Ilya Maximets and Billy O'Mahony. * Replaced netdev_rxq_length() introduced in v7 by optional out parameter for the remaining rx queue len in netdev_rxq_recv(). * Fixed thread synchronization issues in clearing PMD stats: - Use mutex to control whether to clear from main thread directly or in PMD at start of next iteration. - Use mutex to prevent concurrent clearing and printing of metrics. * Added tx packet and batch stats to pmd-perf-show output. * Delay warning for suspicious iteration to the iteration in which we also log the neighborhood to not pollute the logged iteration stats with logging costs. * Corrected the exact number of iterations logged before and after a supicious iteration. * Introduced options -e and -ne in pmd-perf-log-set to control whether to *extend* the range of logged iterations when additional supicious iterations are detected before the scheduled end of logging interval is reached. * Exclude logging cycles from the iteration stats to avoid confusing ghost peaks. * Performance impact compared to master less than 1% even with supervision enabled. v5 -> v7: * Rebased on to dpdk_merge (commit e666668) - New base contains earlier refactoring parts of series. * Implemented comments from Ilya Maximets and Billy O'Mahony. * Replaced piggybacking qlen on dp_packet_batch with a new netdev API netdev_rxq_length(). * Thread-safe clearing of pmd counters in pmd_perf_start_iteration(). * Fixed bug in reporting datapath stats. * Work-around a bug in DPDK rte_vhost_rx_queue_count() which sometimes returns bogus in the upper 16 bits of the uint32_t return value. v4 -> v5: * Rebased to master (commit e9de6c0) * Implemented comments from Aaron Conole and Darrel Ball v3 -> v4: * Rebased to master (commit 4d0a31b) - Reverting changes to struct dp_netdev_pmd_thread. * Make metrics collection configurable. * Several bugfixes. v2 -> v3: * Rebased to OVS master (commit 3728b3b). * Non-trivial adaptation to struct dp_netdev_pmd_thread. - refactored in commit a807c157 (Bhanu). * No other changes compared to v2. v1 -> v2: * Rebased to OVS master (commit 7468ec788). * No other changes compared to v1. Jan Scheurich (3): netdev: Add optional qfill output parameter to rxq_recv() dpif-netdev: Detailed performance stats for PMDs dpif-netdev: Detection and logging of suspicious PMD iterations NEWS | 5 + lib/automake.mk | 1 + lib/dpif-netdev-perf.c | 551 +++++++++++++++++++++++++++++++++++++++++++- lib/dpif-netdev-perf.h | 300 +++++++++++++++++++++++- lib/dpif-netdev-unixctl.man | 216 +++++++++++++++++ lib/dpif-netdev.c | 187 ++++++++++++++- lib/netdev-bsd.c | 8 +- lib/netdev-dpdk.c | 25 +- lib/netdev-dummy.c | 8 +- lib/netdev-linux.c | 7 +- lib/netdev-provider.h | 7 +- lib/netdev.c | 5 +- lib/netdev.h | 3 +- manpages.mk | 2 + vswitchd/ovs-vswitchd.8.in | 27 +-- vswitchd/vswitch.xml | 12 + 16 files changed, 1302 insertions(+), 62 deletions(-) create mode 100644 lib/dpif-netdev-unixctl.man