From patchwork Thu Jan 4 20:02:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Scheurich X-Patchwork-Id: 855879 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zCQNl3XXrz9s7n for ; Fri, 5 Jan 2018 11:22:47 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id BE962D22; Fri, 5 Jan 2018 00:19:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 1EB46D05 for ; Fri, 5 Jan 2018 00:19:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mout.web.de (mout.web.de [212.227.15.3]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 21035573 for ; Fri, 5 Jan 2018 00:19:08 +0000 (UTC) Received: from ubuntu.fritz.box ([89.0.12.95]) by smtp.web.de (mrweb004 [213.165.67.108]) with ESMTPSA (Nemesis) id 0M7n5k-1et3vZ2Xp2-00vMI7; Fri, 05 Jan 2018 01:19:01 +0100 From: Jan Scheurich To: dev@openvswitch.org Date: Thu, 4 Jan 2018 21:02:42 +0100 Message-Id: <1515096166-16257-5-git-send-email-jan.scheurich@ericsson.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1515096166-16257-1-git-send-email-jan.scheurich@ericsson.com> References: <1515096166-16257-1-git-send-email-jan.scheurich@ericsson.com> X-Provags-ID: V03:K0:8UeUDq0c79dnmoETCEDbGkDAtctUa9zm/e8CKuOgimdRy9rPOVT UeUnI9Tx2gCml605UGPPyzL3MEPq/xjXzCzj9xXi9X1i5nA43UVbz3sHG8tpfsz1jp31QTT /woDmhTBUFiizhnyEIaDQDtRqIJhWbi/nzMcn88/99TtbTPgwOgio572CUDHaQcTJ7CEi6u SLR4U8cWE5tVVZqk6RViw== X-UI-Out-Filterresults: notjunk:1; V01:K0:V/LYCbY/k2g=:M8Q5nP5sjiOcMKOHJR0PjX /gNYnxt4YI0YoNCMgDEMw1+jkdfG+rlOWp5iRY2xLG2FlykE0uZ6iSTii8seDaXPXA+CGfwSf xr1WH3Y8T1n6AWsocp2DlsOi3+neVIwgVE++xDApA9dEo2ZyMJU4WDuE7S+AynNR4Sj/90fPW Q2dZWgV0KlPxm2Fn0n6M3pTB7PKbv1HuhEjRTqm1D+p550kGeTAqjbq1aiUVPUyVyFtyPF11S niO4ipbgHtBfz0UdzLS7AHutKA47QcS1YL1xXUqlLjuNZ8pqblP7m4T1/MOS+F1rI/CRa6G4O shm6+JjDCtaTbKTbqIcZcqdmnJgV8Qt1BfWISvRKWN+kufjdYle/RCjV48Rxq6Y2fWf9M5MOK Sfp032hgHoCps/ZB7T3Py5Md2dh8tt2JYEJpl3oHFxpPQmq7eSMlTMOe6slpI1HEPs1ECZtoy mJ+xJbbqJaV/XYu42y7hxDlo/GAq27WWtiDqeg2mDxSiWnbef2S2S+6NA+Zzwi10Bv2PY43bM yFc1Jor7Q4HdbvJ1vgUkf2am4qZ6ttTEJO+KRPf2bL4QY5NhhgSX5mQ+P92DEKxrGgaWa4qdC r0Gg4tbHFov6Otbzg5oNpgYOa1kJwHZGuUQPAs7mBxE8bKIBAzMndKKzlwahCUf1E6TQKIFRy g6Z+KfntR89cnr268ZT6HnfR+5n2dVfV+Df/NibVMph/gFWYqqM6IdY/Mxe0jYDN3kCmC7evi 3UUEJ2SpTLcF7SnIHkXpPxjmJii1ejj8Yr1RQtdFJ5+13n5/kkAvaxgkSWHbYEnybxjirQtNF gHhVF7sqjlY0A7j0DAIodusAlr6VQ== X-Spam-Status: No, score=-1.0 required=5.0 tests=BAYES_00, DATE_IN_PAST_03_06, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, T_RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets Subject: [ovs-dev] [RFC PATCH 4/8] docs: Describe output packet batching in DPDK guide. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Ilya Maximets Added information about output packet batching and a way to configure 'tx-flush-interval'. Signed-off-by: Ilya Maximets Co-authored-by: Jan Scheurich Signed-off-by: Jan Scheurich --- Documentation/intro/install/dpdk.rst | 58 ++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst index 3fecb5c..040e62e 100644 --- a/Documentation/intro/install/dpdk.rst +++ b/Documentation/intro/install/dpdk.rst @@ -568,6 +568,64 @@ not needed i.e. jumbo frames are not needed, it can be forced off by adding chains of descriptors it will make more individual virtio descriptors available for rx to the guest using dpdkvhost ports and this can improve performance. +Output Packet Batching +~~~~~~~~~~~~~~~~~~~~~~ + +To make advantage of batched transmit functions, OVS collects packets in +intermediate queues before sending when processing a batch of received packets. +Even if packets are matched by different flows, OVS uses a single send +operation for all packets destined to the same output port. + +Furthermore, OVS is able to buffer packets in these intermediate queues for a +configurable amount of time to reduce the frequency of send bursts at medium +load levels when the packet receive rate is high, but the receive batch size +still very small. This is particularly beneficial for packets transmitted to +VMs using an interrupt-driven virtio driver, where the interrupt overhead is +significant for the OVS PMD, the host operating system and the guest driver. + +The ``tx-flush-interval`` parameter can be used to specify the time in +microseconds OVS should wait between two send bursts to a given port (default +is ``0``). When the intermediate queue fills up before that time is over, the +buffered packet batch is sent immediately:: + + $ ovs-vsctl set Open_vSwitch . other_config:tx-flush-interval=50 + +This parameter influences both throughput and latency, depending on the traffic +load on the port. In general lower values decrease latency while higher values +may be useful to achieve higher throughput. + +Low traffic (``packet rate < 1 / tx-flush-interval``) should not experience +any significant latency or throughput increase as packets are forwarded +immediately. + +At intermediate load levels +(``1 / tx-flush-interval < packet rate < 32 / tx-flush-interval``) traffic +should experience an average latency increase of up to +``1 / 2 * tx-flush-interval`` and a possible throughput improvement. + +Very high traffic (``packet rate >> 32 / tx-flush-interval``) should experience +the average latency increase equal to ``32 / (2 * packet rate)``. Most send +batches in this case will contain the maximum number of packets (``32``). + +A ``tx-burst-interval`` value of ``50`` microseconds has shown to provide a +good performance increase in a ``PHY-VM-PHY`` scenario on ``x86`` system for +interrupt-driven guests while keeping the latency increase at a reasonable +level: + + https://mail.openvswitch.org/pipermail/ovs-dev/2017-December/341628.html + +.. note:: + Throughput impact of this option significantly depends on the scenario and + the traffic patterns. For example: ``tx-burst-interval`` value of ``50`` + microseconds shows performance degradation in ``PHY-VM-PHY`` with bonded PHY + scenario while testing with ``256 - 1024`` packet flows: + + https://mail.openvswitch.org/pipermail/ovs-dev/2017-December/341700.html + +The average number of packets per output batch can be checked in PMD stats:: + + $ ovs-appctl dpif-netdev/pmd-stats-show + Limitations ------------