mbox series

[ovs-dev,v2,0/8] netdev-linux: Use event-driven netlink notifications.

Message ID 20260311060533.52598-1-amorenoz@redhat.com
Headers show
Series netdev-linux: Use event-driven netlink notifications. | expand

Message

Adrián Moreno March 11, 2026, 6:05 a.m. UTC 
This series refactors netdev-linux to make more use use of event-driven
netlink notifications instead of polling for device state update, significantly
improving performance under RTNL lock contention.

## Background

The RTNL mutex is used to serialize rtnetlink requests in the Linux kernel.
It's widespread use in many network configuration paths make it a problem which
the kernel community is well aware of.

When there is a lot of network configuration activity, like when lots of
interfaces are being created or deleted, specially interfaces that interact
with HW resources such as SR-IOV VFs, contention on the RTNL mutex can make
rtnetlink requests be quite slow.

The impact of RTNL contention on OVS's main thread can be high, making the
entire loop take several seconds (even minutes!) to complete, affecting other
periodic tasks such as OVSDB updates, OpenFlow flow programming, etc.

After analyzing what requests were being sent by OVS, it was observed that most
of them came from netdev-linux state checking mechanisms. While some state is
cached (such as the MTU or MAC address), the netdev's flags are not, and they
are checked very often.

On the other hand, Netlink provides a reliable notification mechanism via
multicast groups that allows userspace to receive asynchronouse updates when
device state changes, and OVS already has some infrastructure for that
purpose.

## Approach

The series aims to change two aspects of netdev-linux operations:
A - Cache the netdev's flags
B - Reuse existing rtnetlink event infrastructure in netdev-linux's main
loop run to avoid races.

In order to accomplish A (commit 2) some refactoring is done first (commit 1).
In roder to accomplish B (commit 5) some refactoring is done to enhance existing
rtnetlink notifier infrastructure (commits 3-4).

Finally, there are some extra consolidation and cleanups (commits 6-8).

The following diagram represents the resulting infrastructure:

            +---------------------------------+                                                  
            | bridge.c                        |                                                  
            |                                 |                                                  
            |   to call bridge_reconfigure()  |                                                  
            |     if ifaces changed           |                                                  
            +--------------------+------------+                                                  
                                 |                                                               
                           +-----v-------+                                                       
                           | if-notifier |                                                       
                           +-----+-------+                                                       
                                 | 
         +-----------------------+                                                               
         | +------------------------------------------------------------------------------+      
   +-----+ | route_table                                                                  |      
   |       |                                +-------------------------------------------+ |      
   |       |  +---------------------------+ | for route change detection                | |      
   |       |  |                           | |                                           | |      
   |    +--+--+  for link change detection| |      family: NETLINK_ROUTE                +-+--+   
   |    |  |  |                           | |      mcast: RTNLGRP_IPV{4,6}_{ROUTE,RULE} | |  |   
   |    |  |  +---------------------------+ |      all_ns:  false                       | |  |   
   |    |  |                                +-------------------------------------------+ |  |   
   |    |  +------------------------------------------------------------------------------+  |   
   |    |                                                                                    |   
   |    |    +----------------------------------------------------------------------------+  |   
   |    |    | netdev_linux                                                               |  |   
   |    |    |                                +----------------------------------------+  |  |   
   |    |    | +-----------------------------+|for address change detection            |  |  |   
   |    |    | |   to update netdev          ||                                        |  |  |   
   |    | +--+-+   internal (cached) state   ||   family: NETLINK_ROUTE                |  |  |   
   |    | |  | +-----------------------------+|   mcast: RTNLGRP_IPV4_IFADDR,          |  |  |   
   |    | |  |                                |          RTNLGRP_IPV6_{IFADDR,IFINFO}  |  |  |   
   |    | |  |                                |   all_ns: true                         |  |  |   
   |    | |  |                                +-----------+----------------------------+  |  |   
   |    | |  +--------------------------------------------+-------------------------------+  |   
   |    | |                                               |                                  |   
 +-v----v-v-------------------+                           |                                  |   
 | rtnetlink_notifier.{c,h}   |                           |                                  |   
 |                            |                           |                                  |   
 |    family: NETLINK_ROUTE   |                           |                                  |   
 |    mcast: RTNLGRP_LINK     |                           |                                  |   
 |    all_ns: true            |                           |                                  |   
 +----------------------------+                           |                                  |   
               |                                          |                                  |   
               |                                          |                                  |   
        +------v------------------------------------------v----------------------------------v--+
        |                                                                                       |
        |                            nln     (netlink_notifier.{h,c})                           |
        |                                                                                       |
        +---------------------------------------------------------------------------------------+


## Testing and results

In order to test this series, I have written a small script that chruns
(deletes and recreates) some ovs ports (veths) in a way an SDN would do.
I increased the number of interfaces to churn from 10 to 100

In order to simulate RTNL mutex contention I used delay-kfunc [1] to
introduce latency to 'rtnl_lock'. The following table shows the time it
takes to complete the test:

==============================================================================
N ifaces   RTNL Delay(μs)  Main (s)             Series (s)          Delta (%)
------------------------------------------------------------------------------
10         0               0.275(0.008)         0.234(0.014)        -14.9%
10         50              0.269(0.009)         0.249(0.012)        -7.3%
10         100             0.278(0.011)         0.266(0.007)        -4.6%
10         500             0.423(0.039)         0.395(0.046)        -6.7%
10         1000            0.695(0.060)         0.586(0.045)        -15.6%
10         5000            1.855(0.099)         1.818(0.041)        -2.0%
10         10000           4.361(0.074)         3.106(0.111)        -28.8%
20         0               0.485(0.014)         0.424(0.019)        -12.6%
20         50              0.478(0.018)         0.472(0.015)        -1.3%
20         100             0.504(0.018)         0.493(0.020)        -2.3%
20         500             0.716(0.022)         0.678(0.031)        -5.3%
20         1000            0.994(0.026)         0.926(0.083)        -6.9%
20         5000            3.313(0.133)         2.851(0.039)        -13.9%
20         10000           6.803(0.093)         4.875(0.117)        -28.3%
30         0               0.716(0.024)         0.645(0.033)        -10.0%
30         50              0.723(0.018)         0.692(0.019)        -4.2%
30         100             0.744(0.024)         0.745(0.031)        +0.1%
30         500             0.981(0.031)         0.997(0.034)        +1.6%
30         1000            1.328(0.046)         1.222(0.040)        -8.0%
30         5000            4.838(0.059)         3.865(0.079)        -20.1%
30         10000           9.146(0.110)         6.653(0.110)        -27.3%
40         0               0.974(0.042)         0.864(0.065)        -11.3%
40         50              0.963(0.032)         0.961(0.044)        -0.2%
40         100             0.997(0.040)         1.004(0.043)        +0.7%
40         500             1.397(0.105)         1.359(0.035)        -2.7%
40         1000            1.990(0.107)         1.805(0.096)        -9.3%
40         5000            7.240(1.751)         4.967(0.587)        -31.4%
40         10000           11.657(0.131)        8.289(0.308)        -28.9%
50         0               1.340(0.111)         1.253(0.167)        -6.5%
50         50              1.410(0.196)         1.274(0.059)        -9.7%
50         100             1.411(0.108)         1.329(0.111)        -5.8%
50         500             1.788(0.060)         1.779(0.079)        -0.5%
50         1000            2.656(0.220)         2.446(0.097)        -7.9%
50         5000            11.532(0.132)        8.216(0.094)        -28.8%
50         10000           22.685(1.157)        14.098(0.186)       -37.8%
60         0               1.760(0.249)         1.738(0.333)        -1.3%
60         50              1.945(0.283)         1.851(0.305)        -4.8%
60         100             1.777(0.340)         1.613(0.116)        -9.2%
60         500             2.525(0.184)         2.330(0.125)        -7.7%
60         1000            3.497(0.327)         3.247(0.174)        -7.2%
60         5000            14.390(0.172)        10.093(0.138)       -29.9%
60         10000           27.980(0.545)        17.383(0.211)       -37.9%
80         0               3.977(0.767)         3.632(0.651)        -8.7%
80         50              3.550(0.667)         3.294(0.645)        -7.2%
80         100             3.854(0.679)         3.182(0.763)        -17.4%
80         500             4.571(0.685)         3.998(0.619)        -12.5%
80         1000            6.445(0.490)         4.955(0.281)        -23.1%
80         5000            27.107(0.331)        17.348(0.197)       -36.0%
80         10000           54.738(0.971)        31.525(1.116)       -42.4%
100        0               8.509(2.392)         7.452(0.138)        -12.4%
100        50              7.730(0.552)         7.278(1.877)        -5.8%
100        100             8.084(2.648)         7.342(1.124)        -9.2%
100        500             7.543(0.551)         6.851(0.997)        -9.2%
100        1000            10.784(0.782)        7.990(0.651)        -25.9%
100        5000            36.393(0.626)        25.800(0.363)       -29.1%
100        10000           72.916(2.488)        45.648(1.929)       -37.4%
==============================================================================

Notes about the above results:
- Values are shown as "{mean}({std})".
- I did not perform any kind of tuning or cpu isolation the test server.
- delay-kfunc does not always introduce the exact same delay so there is some
source of variance there as well.
- Beyond 200 interfaces, limitations of the test script itself make the results
rather unreliable.

All in all, a pretty consistent improvement is observed which increases with the
number of interfaces that we churn and with the amount of external RTNL
pressure we add.

## Future work

This is part of a larger effort to improve robustness against RTNL contention.
I plan to work on more optimizations in future series.


[1] https://github.com/xdp-project/bpf-examples/tree/main/delay-kfunc

Adrian Moreno (8):
  netdev_linux: Refactor netdev flag update.
  netdev-linux: Cache netdev flags.
  netlink-notifier: Drain socket on overflow.
  netlink-notifier: Include nsid in callbacks.
  netdev-linux: Use rtnetlink to update state.
  netdev-linux: Consolidate RTM_GETLINK parsing.
  linux-netdev: Check status when reading stats.
  netdev-linux: Consolidate netlink updates.

 lib/if-notifier.c              |   3 +-
 lib/netdev-afxdp.c             |   2 +-
 lib/netdev-linux-private.h     |   5 +-
 lib/netdev-linux.c             | 416 +++++++++++++++------------------
 lib/netdev-linux.h             |   1 +
 lib/netlink-notifier.c         |  15 +-
 lib/netlink-notifier.h         |   9 +-
 lib/netnsid.h                  |   1 +
 lib/route-table.c              |  12 +-
 lib/route-table.h              |   2 +-
 lib/rtnetlink.c                |  40 +++-
 lib/rtnetlink.h                |  56 ++++-
 lib/tnl-ports.c                |   2 +-
 tests/system-interface.at      |   2 +
 tests/system-tap.at            |   5 +-
 tests/system-traffic.at        |   3 +-
 tests/test-lib-route-table.c   |   9 +-
 tests/test-netlink-conntrack.c |   9 +-
 18 files changed, 338 insertions(+), 254 deletions(-)