[ovs-dev,v5,3/4] netdev-dpdk: Add support for egress packet-per-second policing.

From: Lin Huang <linhuang@ruijie.com.cn>

OvS has supported packet-per-second policer which can be set at ingress
and egress side in kernel datapath. But the userspace datapath doesn't
support for ingress and egress packet-per-second policing now.

So, this patch add support for userspace egress pps policing by using
native ovs token bucket library. Token bucket is accumulated by 'rate'
tokens per millisecond and store maximum tokens at 'burst' bucket size.
One token in the bucket means one packet (1 kpkts * millisecond) which
will drop or pass by policer.

This patch add new configuration option 'kpkts_rate' and 'kpkts_burst'
for egress-policer QoS type which now supports setting packet-per-second
limits in addition to the previously configurable byte rate settings.

Examples:
$ovs-vsctl set port vhost-user0 qos=@newqos --
           --id=@newqos create qos type=egress-policer \
           other-config:cir=123000 other-config:cbs=123000
           other-config:kpkts_rate=123 other-config:kpkts_burst=123

Add some unit tests for egress packet-per-second policing.

Signed-off-by: Lin Huang <linhuang@ruijie.com.cn>
---
 Documentation/topics/dpdk/qos.rst |  15 ++-
 NEWS                              |   3 +
 lib/netdev-dpdk.c                 | 141 ++++++++++++++++++----
 tests/system-dpdk.at              | 194 +++++++++++++++++++++++++++++-
 vswitchd/vswitch.xml              |  10 ++
 5 files changed, 337 insertions(+), 26 deletions(-)

On 6/18/23 15:58, miterv@outlook.com wrote:
> From: Lin Huang <linhuang@ruijie.com.cn>
> 
> OvS has supported packet-per-second policer which can be set at ingress
> and egress side in kernel datapath. But the userspace datapath doesn't
> support for ingress and egress packet-per-second policing now.
> 
> So, this patch add support for userspace egress pps policing by using
> native ovs token bucket library. Token bucket is accumulated by 'rate'
> tokens per millisecond and store maximum tokens at 'burst' bucket size.
> One token in the bucket means one packet (1 kpkts * millisecond) which
> will drop or pass by policer.
> 
> This patch add new configuration option 'kpkts_rate' and 'kpkts_burst'
> for egress-policer QoS type which now supports setting packet-per-second
> limits in addition to the previously configurable byte rate settings.
> 
> Examples:
> $ovs-vsctl set port vhost-user0 qos=@newqos --
>            --id=@newqos create qos type=egress-policer \
>            other-config:cir=123000 other-config:cbs=123000
>            other-config:kpkts_rate=123 other-config:kpkts_burst=123
> 
> Add some unit tests for egress packet-per-second policing.
> 
> Signed-off-by: Lin Huang <linhuang@ruijie.com.cn>
> ---
>  Documentation/topics/dpdk/qos.rst |  15 ++-
>  NEWS                              |   3 +
>  lib/netdev-dpdk.c                 | 141 ++++++++++++++++++----
>  tests/system-dpdk.at              | 194 +++++++++++++++++++++++++++++-
>  vswitchd/vswitch.xml              |  10 ++
>  5 files changed, 337 insertions(+), 26 deletions(-)
> 
> diff --git a/Documentation/topics/dpdk/qos.rst b/Documentation/topics/dpdk/qos.rst
> index a98ec672f..5f0b1469a 100644
> --- a/Documentation/topics/dpdk/qos.rst
> +++ b/Documentation/topics/dpdk/qos.rst
> @@ -36,14 +36,21 @@ QoS (Egress Policing)
>  Single Queue Policer
>  ~~~~~~~~~~~~~~~~~~~~
>  
> -Assuming you have a :doc:`vhost-user port <vhost-user>` transmitting traffic
> -consisting of packets of size 64 bytes, the following command would limit the
> -egress transmission rate of the port to ~1,000,000 packets per second::
> +Assuming you have a :doc:`vhost-user port <vhost-user>` transmitting traffic,
> +the following command would limit the egress transmission rate of the port to
> +~1,000,000 bytes per second:
>  
>      $ ovs-vsctl set port vhost-user0 qos=@newqos -- \
> -        --id=@newqos create qos type=egress-policer other-config:cir=46000000 \
> +        --id=@newqos create qos type=egress-policer other-config:cir=1000000 \
>          other-config:cbs=2048`
>  
> +or, the following command would limit the egress transmission rate of the port
> +to ~1,000,000 packets per second:
> +
> +       ovs-vsctl set port vhost-user0 qos=@newqos -- \
> +          --id=@newqos create qos type=egress-policer \
> +          other-config:kpkts_rate=1000 other-config:kpkts_burst=1000
> +
>  To examine the QoS configuration of the port, run::
>  
>      $ ovs-appctl -t ovs-vswitchd qos/show vhost-user0
> diff --git a/NEWS b/NEWS
> index cfd466663..3c33acbbf 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -36,6 +36,9 @@ Post-v3.1.0
>         process extra privileges when mapping physical interconnect memory.
>     - SRv6 Tunnel Protocol
>       * Added support for userspace datapath (only).
> +   - Userspace datapath:
> +     * Added new configuration options 'kpkts_rate' and 'kpkts_burst' for
> +      'egress-policer' to support packet-per-second policing.
>  
>  
>  v3.1.0 - 16 Feb 2023
> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> index 707f622f3..ee5d557b0 100644
> --- a/lib/netdev-dpdk.c
> +++ b/lib/netdev-dpdk.c
> @@ -19,6 +19,7 @@
>  
>  #include <errno.h>
>  #include <signal.h>
> +#include <stdint.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <unistd.h>
> @@ -59,6 +60,7 @@
>  #include "openvswitch/ofp-parse.h"
>  #include "openvswitch/ofp-print.h"
>  #include "openvswitch/shash.h"
> +#include "openvswitch/token-bucket.h"
>  #include "openvswitch/vlog.h"
>  #include "ovs-numa.h"
>  #include "ovs-rcu.h"
> @@ -91,6 +93,8 @@ static bool per_port_memory = false; /* Status of per port memory support */
>  #define OVS_CACHE_LINE_SIZE CACHE_LINE_SIZE
>  #define OVS_VPORT_DPDK "ovs_dpdk"
>  
> +#define MAX_KPKTS_PARAMETER 4294967U  /* UINT32_MAX / 1000 */
> +
>  /*
>   * need to reserve tons of extra space in the mbufs so we can align the
>   * DMA addresses to 4KB.
> @@ -400,6 +404,11 @@ struct dpdk_tx_queue {
>      );
>  };
>  
> +enum policer_type {
> +    POLICER_BPS   = 1 << 0,   /* Rate value in bytes/sec. */
> +    POLICER_PKTPS = 1 << 1,   /* Rate value in packet/sec. */
> +};
> +
>  struct ingress_policer {
>      struct rte_meter_srtcm_params app_srtcm_params;
>      struct rte_meter_srtcm in_policer;
> @@ -2360,6 +2369,50 @@ srtcm_policer_run_single_packet(struct rte_meter_srtcm *meter,
>      return cnt;
>  }
>  
> +static int
> +pkts_policer_run_single_packet(struct token_bucket *tb, struct rte_mbuf **pkts,
> +                               int pkt_cnt)
> +{
> +    int cnt = 0;
> +    struct rte_mbuf *pkt = NULL;
> +    long long int now = time_msec();
> +
> +    for (int i = 0; i < pkt_cnt; i++) {
> +        pkt = pkts[i];

This value is assigned but never used.  This is causing compiler warnings.
We need to remove the pkt variable from this function and mark pkts argument
with OVS_UNUSED.

> +        /* Handle current packet. */
> +        if (!token_bucket_withdraw(tb, 1, now)) {
> +            /* Count dropped packets. */
> +            cnt++;
> +        }
> +    }
> +
> +    /* Count dropped packets */

This comment is not needed.

> +    return cnt;
> +}
> +
> +static int
> +pkts_policer_profile_config(struct token_bucket *tb,
> +                            uint32_t kpkts_rate, uint32_t kpkts_burst)
> +{
> +    if (kpkts_rate > MAX_KPKTS_PARAMETER ||
> +        kpkts_burst > MAX_KPKTS_PARAMETER) {
> +        return EINVAL;
> +    }
> +
> +    /* Rate in kilo-packets/second, bucket 1000 packets. */
> +    /* msec * kilo-packets/sec = 1 packets. */

Please, combine these comments into a signle block:

   /* Rate in kilo-packets/second, bucket 1000 packets.
    * msec * kilo-packets/sec = 1 packets. */

> +    if (kpkts_rate) {
> +        /* Parameters between (1 ~ MAX_KPKTS_PARAMETER). */
> +        token_bucket_init(tb, kpkts_rate, kpkts_burst * 1000);
> +    } else {
> +        /* Zero means not to police the traffic. */
> +        /* Return a error to not set POLICER_PKTPS valid. */

Same here.

> +        return EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
>  static int
>  ingress_policer_run(struct ingress_policer *policer, struct rte_mbuf **pkts,
>                      int pkt_cnt, bool should_steal)
> @@ -4779,24 +4832,37 @@ netdev_dpdk_queue_dump_done(const struct netdev *netdev OVS_UNUSED,
>  
>  
>  /* egress-policer details */
> +struct egress_policer_params {
> +    struct rte_meter_srtcm_params app_srtcm_params;
> +    uint32_t kpkts_rate;
> +    uint32_t kpkts_burst;
> +};
>  
>  struct egress_policer {
>      struct qos_conf qos_conf;
> -    struct rte_meter_srtcm_params app_srtcm_params;
> +    enum policer_type type;
> +    struct token_bucket egress_tb;
> +    struct egress_policer_params params;
>      struct rte_meter_srtcm egress_meter;
>      struct rte_meter_srtcm_profile egress_prof;
>  };
>  
>  static void
>  egress_policer_details_to_param(const struct smap *details,
> -                                struct rte_meter_srtcm_params *params)
> +                                struct egress_policer_params *params)
>  {
> -    memset(params, 0, sizeof *params);
> -    params->cir = smap_get_ullong(details, "cir", 0);
> -    params->cbs = smap_get_ullong(details, "cbs", 0);
> -    params->ebs = 0;
> +    struct rte_meter_srtcm_params *srtcm_params = &params->app_srtcm_params;
> +
> +    memset(srtcm_params, 0, sizeof *srtcm_params);
> +    srtcm_params->cir = smap_get_ullong(details, "cir", 0);
> +    srtcm_params->cbs = smap_get_ullong(details, "cbs", 0);
> +    srtcm_params->ebs = 0;
> +
> +    params->kpkts_rate  = smap_get_uint(details, "kpkts_rate", 0);
> +    params->kpkts_burst = smap_get_uint(details, "kpkts_burst", 0);
>  }
>  
> +
>  static int
>  egress_policer_qos_construct(const struct smap *details,
>                               struct qos_conf **conf)
> @@ -4804,23 +4870,39 @@ egress_policer_qos_construct(const struct smap *details,
>      struct egress_policer *policer;
>      int err = 0;
>  
> -    policer = xmalloc(sizeof *policer);
> +    policer = xzalloc(sizeof *policer);
>      qos_conf_init(&policer->qos_conf, &egress_policer_ops);
> -    egress_policer_details_to_param(details, &policer->app_srtcm_params);
> +    egress_policer_details_to_param(details, &policer->params);
> +
>      err = rte_meter_srtcm_profile_config(&policer->egress_prof,
> -                                         &policer->app_srtcm_params);
> +                                         &policer->params.app_srtcm_params);
>      if (!err) {
>          err = rte_meter_srtcm_config(&policer->egress_meter,
>                                       &policer->egress_prof);
>      }
> +    if (err) {
> +        VLOG_DBG("Could not create rte meter for egress policer");
> +        err = -err;
> +    } else {
> +        policer->type |= POLICER_BPS;
> +    }
>  
> -    if (!err) {
> -        *conf = &policer->qos_conf;
> +    err = pkts_policer_profile_config(&policer->egress_tb,
> +                                      policer->params.kpkts_rate,
> +                                      policer->params.kpkts_burst);
> +    if (err) {
> +        VLOG_DBG("Could not create token bucket for egress policer");
>      } else {
> -        VLOG_ERR("Could not create rte meter for egress policer");
> +        policer->type |= POLICER_PKTPS;
> +    }
> +
> +    if (!policer->type) {
> +        /* both bps and kpkts contrsruction failed.*/

s/both/Both/

And we should, probbaly, print out an error here.  All the messages
above was moved to debug level, but we need an indication that we
wasn't able to create anything at least.

>          free(policer);
>          *conf = NULL;
> -        err = -err;
> +    } else {
> +        err = 0;
> +        *conf = &policer->qos_conf;
>      }
>  
>      return err;
> @@ -4839,9 +4921,12 @@ egress_policer_qos_get(const struct qos_conf *conf, struct smap *details)
>  {
>      struct egress_policer *policer =
>          CONTAINER_OF(conf, struct egress_policer, qos_conf);
> +    struct egress_policer_params *params = &policer->params;
>  
> -    smap_add_format(details, "cir", "%"PRIu64, policer->app_srtcm_params.cir);
> -    smap_add_format(details, "cbs", "%"PRIu64, policer->app_srtcm_params.cbs);
> +    smap_add_format(details, "cir", "%"PRIu64, params->app_srtcm_params.cir);
> +    smap_add_format(details, "cbs", "%"PRIu64, params->app_srtcm_params.cbs);
> +    smap_add_format(details, "kpkts_rate", "%"PRIu32, params->kpkts_rate);
> +    smap_add_format(details, "kpkts_burst", "%"PRIu32, params->kpkts_burst);
>  
>      return 0;
>  }
> @@ -4852,11 +4937,11 @@ egress_policer_qos_is_equal(const struct qos_conf *conf,
>  {
>      struct egress_policer *policer =
>          CONTAINER_OF(conf, struct egress_policer, qos_conf);
> -    struct rte_meter_srtcm_params params;
> +    struct egress_policer_params params;
>  
>      egress_policer_details_to_param(details, &params);
>  
> -    return !memcmp(&params, &policer->app_srtcm_params, sizeof params);
> +    return !memcmp(&params, &policer->params, sizeof params);
>  }
>  
>  static int
> @@ -4864,14 +4949,28 @@ egress_policer_run(struct qos_conf *conf, struct rte_mbuf **pkts, int pkt_cnt,
>                     bool should_steal)
>  {
>      int cnt = 0;
> +    int bps_drop = 0;
> +    int pps_drop = 0;
>      struct egress_policer *policer =
>          CONTAINER_OF(conf, struct egress_policer, qos_conf);
>  
> -    cnt = srtcm_policer_run_single_packet(&policer->egress_meter,
> -                                          &policer->egress_prof, pkts,
> -                                          pkt_cnt);
> +    if (policer->type & POLICER_BPS) {
> +        bps_drop = srtcm_policer_run_single_packet(&policer->egress_meter,
> +                                                   &policer->egress_prof, pkts,
> +                                                   pkt_cnt);
> +    }
>  
> -    return cnt;
> +    if (policer->type & POLICER_PKTPS) {
> +        pps_drop = pkts_policer_run_single_packet(&policer->egress_tb, pkts,
> +                                                  pkt_cnt);
> +    }
> +
> +    cnt = MAX(bps_drop, pps_drop);
> +    if (should_steal && cnt) {
> +        rte_pktmbuf_free_bulk(&pkts[pkt_cnt - cnt], cnt);
> +    }

The packet freeing should be moved into a previous patch, otherwise we'll
leak packets if only patch #2 is applied.

> +
> +    return pkt_cnt - cnt;
>  }
>  
>  static const struct dpdk_qos_ops egress_policer_ops = {
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 0f58e8574..233f7e493 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -435,6 +435,7 @@ OVS_DPDK_PRE_PHY_SKIP()
>  OVS_DPDK_START()
>  
>  dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
>  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>  AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
>  OVS_WAIT_UNTIL([ovs-vsctl set port phy0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cir=1250000 other-config:cbs=2048])
> @@ -453,7 +454,9 @@ AT_CHECK([grep -E 'QoS not configured on phy0' stdout], [], [stdout])
>  
>  dnl Clean up
>  AT_CHECK([ovs-vsctl del-port br10 phy0], [], [stdout], [stderr])
> -OVS_VSWITCHD_STOP("[SYSTEM_DPDK_ALLOWED_LOGS]")
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@Could not create token bucket for egress policer@d
> +])")
>  AT_CLEANUP
>  dnl --------------------------------------------------------------------------
>  
> @@ -468,6 +471,7 @@ OVS_DPDK_PRE_CHECK()
>  OVS_DPDK_START([--no-pci])
>  
>  dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
>  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>  AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
>  OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cir=1250000 \
> @@ -493,6 +497,7 @@ AT_CHECK([grep -E 'QoS not configured on dpdkvhostuserclient0' stdout], [], [std
>  dnl Clean up
>  AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
>  OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@Could not create token bucket for egress policer@d
>  \@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
>  ])")
>  AT_CLEANUP
> @@ -509,6 +514,7 @@ OVS_DPDK_PRE_CHECK()
>  OVS_DPDK_START([--no-pci])
>  
>  dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
>  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>  AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
>  OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cbs=2048])
> @@ -528,6 +534,7 @@ AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
>  OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
>  \@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
>  \@Could not create rte meter for egress policer@d
> +\@Could not create token bucket for egress policer@d
>  \@Failed to set QoS type egress-policer on port dpdkvhostuserclient0: Invalid argument@d
>  ])")
>  AT_CLEANUP
> @@ -544,6 +551,7 @@ OVS_DPDK_PRE_CHECK()
>  OVS_DPDK_START([--no-pci])
>  
>  dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
>  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>  AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
>  OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cir=1250000])
> @@ -563,6 +571,7 @@ AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
>  OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
>  \@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
>  \@Could not create rte meter for egress policer@d
> +\@Could not create token bucket for egress policer@d
>  \@Failed to set QoS type egress-policer on port dpdkvhostuserclient0: Invalid argument@d
>  ])")
>  AT_CLEANUP
> @@ -570,6 +579,189 @@ dnl --------------------------------------------------------------------------
>  
>  
>  
> +dnl --------------------------------------------------------------------------
> +dnl QoS (kpkts) create delete vport port
> +AT_SETUP([OVS-DPDK - QoS (kpkts) create delete vport port])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_PRE_CHECK()
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
> +OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:kpkts_rate=123 other-config:kpkts_burst=456])
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +sleep 2
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Remove egress policer
> +AT_CHECK([ovs-vsctl destroy QoS dpdkvhostuserclient0 -- clear Port dpdkvhostuserclient0 qos])
> +
> +dnl Check egress policer was removed correctly
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +AT_CHECK([grep -E 'QoS not configured on dpdkvhostuserclient0' stdout], [], [stdout])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
> +\@Failed to set QoS type egress-policer on port dpdkvhostuserclient0@d
> +\@Could not create rte meter for egress policer@d
> +])")
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +
> +
> +dnl --------------------------------------------------------------------------
> +dnl QoS (kpkts) no rate
> +AT_SETUP([OVS-DPDK - QoS (kpkts) no rate])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_PRE_CHECK()
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
> +OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:kpkts_burst=123])
> +sleep 2
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Check egress policer was not created
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +AT_CHECK([grep -E 'QoS not configured on dpdkvhostuserclient0' stdout], [], [stdout])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
> +\@Could not create rte meter for egress policer@d
> +\@Could not create token bucket for egress policer@d
> +\@Failed to set QoS type egress-policer on port dpdkvhostuserclient0@d
> +])")
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +
> +
> +dnl --------------------------------------------------------------------------
> +dnl QoS (kpkts) no burst
> +AT_SETUP([OVS-DPDK - QoS (kpkts) no burst])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_PRE_CHECK()
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
> +OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:kpkts_rate=123])
> +sleep 2
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Check egress policer was created
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +AT_CHECK([grep -E 'kpkts_rate: 123' stdout], [], [stdout])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
> +\@Could not create rte meter for egress policer@d
> +\@Failed to set QoS type egress-policer on port dpdkvhostuserclient0@d
> +])")
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +
> +
> +dnl --------------------------------------------------------------------------
> +dnl QoS (kpkts) max rate
> +AT_SETUP([OVS-DPDK - QoS (kpkts) max rate])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_PRE_CHECK()
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
> +OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:kpkts_rate=42949671])
> +sleep 2
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Check egress policer was not created
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +AT_CHECK([grep -E 'QoS not configured on dpdkvhostuserclient0' stdout], [], [stdout])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
> +\@Could not create rte meter for egress policer@d
> +\@Could not create token bucket for egress policer@d
> +\@Failed to set QoS type egress-policer on port dpdkvhostuserclient0@d
> +])")
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +
> +dnl --------------------------------------------------------------------------
> +dnl QoS (kpkts) max burst
> +AT_SETUP([OVS-DPDK - QoS (kpkts) max burst])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_PRE_CHECK()
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS and add egress policer
> +AT_CHECK([ovs-appctl vlog/set netdev_dpdk:dbg])
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], [stdout], [stderr])
> +OVS_WAIT_UNTIL([ovs-vsctl set port dpdkvhostuserclient0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:kpkts_burst=42949671])
> +sleep 2
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Check egress policer was not created
> +AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show dpdkvhostuserclient0], [], [stdout])
> +AT_CHECK([grep -E 'QoS not configured on dpdkvhostuserclient0' stdout], [], [stdout])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [
> +\@VHOST_CONFIG: ($OVS_RUNDIR/dpdkvhostclient0) failed to connect: No such file or directory@d
> +\@Could not create rte meter for egress policer@d
> +\@Could not create token bucket for egress policer@d
> +\@Failed to set QoS type egress-policer on port dpdkvhostuserclient0@d
> +])")
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
>  
>  dnl --------------------------------------------------------------------------
>  dnl MTU increase phy port
> diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
> index 59c404bbb..b94ed2d34 100644
> --- a/vswitchd/vswitch.xml
> +++ b/vswitchd/vswitch.xml
> @@ -4920,6 +4920,16 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \
>          bytes/tokens of the packet. If there are not enough tokens in the cbs
>          bucket the packet might be dropped.
>        </column>
> +      <column name="other_config" key="kpkts_rate"
> +          type='{"type": "integer", "minInteger": 0, "maxInteger": 4294967}'>
> +        The Packets Per Second (PPS) represents the packet per second rate at
> +        which the token bucket will be updated.
> +      </column>
> +      <column name="other_config" key="kpkts_burst"
> +          type='{"type": "integer", "minInteger": 0, "maxInteger": 4294967}'>
> +        The Packets Per Second Burst Size is measured in count and represents a

s/measured in count/measured in kilo-packets/

> +        token bucket.

s/token bucket/token bucket burst/

> +      </column>
>      </group>
>  
>      <group title="Configuration for linux-sfq">