{"id":2197695,"url":"http://patchwork.ozlabs.org/api/1.0/covers/2197695/?format=json","project":{"id":47,"url":"http://patchwork.ozlabs.org/api/1.0/projects/47/?format=json","name":"Open vSwitch","link_name":"openvswitch","list_id":"ovs-dev.openvswitch.org","list_email":"ovs-dev@openvswitch.org","web_url":"http://openvswitch.org/","scm_url":"git@github.com:openvswitch/ovs.git","webscm_url":"https://github.com/openvswitch/ovs"},"msgid":"<20260218103716.3135692-1-amorenoz@redhat.com>","date":"2026-02-18T10:37:14","name":"[ovs-dev,RFC,v1,0/2] Introduce upcall (live) tracing.","submitter":{"id":77477,"url":"http://patchwork.ozlabs.org/api/1.0/people/77477/?format=json","name":"Adrián Moreno","email":"amorenoz@redhat.com"},"series":[{"id":492538,"url":"http://patchwork.ozlabs.org/api/1.0/series/492538/?format=json","date":"2026-02-18T10:37:14","name":"Introduce upcall (live) tracing.","version":1,"mbox":"http://patchwork.ozlabs.org/series/492538/mbox/"}],"headers":{"Return-Path":"","X-Original-To":["incoming@patchwork.ozlabs.org","dev@openvswitch.org"],"Delivered-To":["patchwork-incoming@legolas.ozlabs.org","ovs-dev@lists.linuxfoundation.org"],"Authentication-Results":["legolas.ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=RwiNw7Ao;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org\n (client-ip=140.211.166.136; helo=smtp3.osuosl.org;\n envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org)","smtp3.osuosl.org;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key)\n header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=RwiNw7Ao","smtp4.osuosl.org; dmarc=pass (p=quarantine dis=none)\n header.from=redhat.com","smtp4.osuosl.org;\n dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com\n header.a=rsa-sha256 header.s=mimecast20190719 header.b=RwiNw7Ao"],"Received":["from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4fGCdC48Ckz1xpY\n\tfor ; Wed, 18 Feb 2026 21:37:34 +1100 (AEDT)","from localhost (localhost [127.0.0.1])\n\tby smtp3.osuosl.org (Postfix) with ESMTP id 4A7CE607FF;\n\tWed, 18 Feb 2026 10:37:31 +0000 (UTC)","from smtp3.osuosl.org ([127.0.0.1])\n by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP\n id 0w__cBkxjt1I; Wed, 18 Feb 2026 10:37:29 +0000 (UTC)","from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56])\n\tby smtp3.osuosl.org (Postfix) with ESMTPS id 4FDD560627;\n\tWed, 18 Feb 2026 10:37:29 +0000 (UTC)","from lf-lists.osuosl.org (localhost [127.0.0.1])\n\tby lists.linuxfoundation.org (Postfix) with ESMTP id 2ACB6C0035;\n\tWed, 18 Feb 2026 10:37:29 +0000 (UTC)","from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137])\n by lists.linuxfoundation.org (Postfix) with ESMTP id 6AE4DC0033\n for ; Wed, 18 Feb 2026 10:37:27 +0000 (UTC)","from localhost (localhost [127.0.0.1])\n by smtp4.osuosl.org (Postfix) with ESMTP id 511DF407A1\n for ; Wed, 18 Feb 2026 10:37:27 +0000 (UTC)","from smtp4.osuosl.org ([127.0.0.1])\n by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP\n id BADJiaImFoGx for ;\n Wed, 18 Feb 2026 10:37:26 +0000 (UTC)","from us-smtp-delivery-124.mimecast.com\n (us-smtp-delivery-124.mimecast.com [170.10.133.124])\n by smtp4.osuosl.org (Postfix) with ESMTPS id 4758740368\n for ; Wed, 18 Feb 2026 10:37:25 +0000 (UTC)","from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-654-WGfekWdCMbGxLaP6SIjrvQ-1; Wed,\n 18 Feb 2026 05:37:21 -0500","from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id B4D96180025A\n for ; Wed, 18 Feb 2026 10:37:20 +0000 (UTC)","from antares.redhat.com (unknown [10.45.224.73])\n by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP\n id A257419560AD; Wed, 18 Feb 2026 10:37:19 +0000 (UTC)"],"X-Virus-Scanned":["amavis at osuosl.org","amavis at osuosl.org"],"X-Comment":"SPF check N/A for local connections - client-ip=140.211.9.56;\n helo=lists.linuxfoundation.org;\n envelope-from=ovs-dev-bounces@openvswitch.org; receiver= ","DKIM-Filter":["OpenDKIM Filter v2.11.0 smtp3.osuosl.org 4FDD560627","OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4758740368"],"Received-SPF":"Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124;\n helo=us-smtp-delivery-124.mimecast.com; envelope-from=amorenoz@redhat.com;\n receiver=","DMARC-Filter":"OpenDMARC Filter v1.4.2 smtp4.osuosl.org 4758740368","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1771411044;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding;\n bh=P3FYi05qEMvKfSeeWjW0cI1mTX8BeHIOaZEYH4wa4I0=;\n b=RwiNw7AosRVoDV64EgnpXK55/bkUondRScpsrsHvYFpmWJBlS8gmoqNlO010AzQ8xGp3k2\n vhCVTcBr3o+83aYV/J6dxIFMfDpLafPU1kFy3LWlOYeNM87cQq7umTxx5CnBl6meiQBCYz\n M6AAHzxEF+xjJlhBHtxj4RVNZTgu64A=","X-MC-Unique":"WGfekWdCMbGxLaP6SIjrvQ-1","X-Mimecast-MFC-AGG-ID":"WGfekWdCMbGxLaP6SIjrvQ_1771411040","To":"dev@openvswitch.org","Date":"Wed, 18 Feb 2026 11:37:14 +0100","Message-ID":"<20260218103716.3135692-1-amorenoz@redhat.com>","MIME-Version":"1.0","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.12","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"3-XSgz5CTH6asfQqG4lbCyPggppraLtRuOHVAGiJ2-A_1771411040","X-Mimecast-Originator":"redhat.com","Subject":"[ovs-dev] [RFC PATCH v1 0/2] Introduce upcall (live) tracing.","X-BeenThere":"ovs-dev@openvswitch.org","X-Mailman-Version":"2.1.30","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","From":"Adrian Moreno via dev ","Reply-To":"Adrian Moreno ","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"ovs-dev-bounces@openvswitch.org","Sender":"\"dev\" "},"content":"ofproto/trace is one of the most useful debugging tools OVS provides.\nHowever, it's \"offline\" nature comes with limitations:\n- Users need to know exactly what their packets look like\n- Runtime information such as conntrack states has to be guessed\n\nThis RFC introduces the idea of upcall (live) tracing. In a nutshell,\nthe idea is:\n- The user activates upcall tracing by specifying an openflow filter\n- When packet is upcalled, OVS checks if it matches the filter and if it\n does, it collects traces from the xlate layer (same traces as\n ofproto/trace emits).\n- If a frozen state is created from this upcall, an ID is stored in it\n so if a subsequent upcall resumes the frozen state, it inherits its\n ID, the \"trace_id\".\n- Traces are stored in some kind of ring-buffer or fixed-size list\n arranged by these tracing IDs.\n- Traces are accessed (printed) by the user after the experiment has\n ended.\n\nThe code in this RFC is in early state but it can be used to play\naround. I'm sending it out early to get some feedback.\nThe following topics are not clear to me at the moment:\n\n- Naming ana layering: I have called the thing \"upcall-tracing\", and\n unixctl commands are \"upcall/trace/{create,list,get,show}\". I tried to\n differentiate from \"ofproto/trace\" but maybe this is not very\n intuitive? A (kind of crazy) idea that I have as a followup is to\n persist the trace_id into the udpif_key so we can also trace\n revalidations associated with an upcall. In such scenario,\n \"upcall/trace\" might fall semantically short.\n\n- ofproto vs dpif: Connected to the previous topic. The tracing\n infrastructure is bound to the \"udpif\", the upcall engine, which is\n part of the datapath (dpif), not the bridge. However, ofproto flows\n are easier to write and more familiar to users so from their PoV,\n specifying a bridge and a ofp filter is nicer. Writing\n \"upcall/trace/create br0 in_port=myport,ip\" and then visualizing a\n trace associated with `system@ovs-system` feels weird.\n\n- Trace ID != Packet ID: The RFC generates a trace_id for each new\n upcall that matches the filter and persistes the trace_id inside the\n frozen_state. If the same packet gets recirculated and upcalled, it\n will (nicely) inherit the same trace_id and be grouped together. We\n see all the recirculation rounds of the same packet (same as with\n ofproto/trace but for real!). BUT, frozen_states are shared. If\n another packet hits it, it will also inherit that ID and be printed\n alongside the first. Is this good? bad? acceptable? Do we need to\n the packet's metadata instead of the frozen_state to persist this\n trace_id?\n\n- Another idea that I considered is binding the tracing to a specific\n port, i.e: \"upcall/trace/create br0 p1 {ofp_filter}\". Although this\n would deviate from the ofproto/trace syntax, it would make it easier\n to avoid an extra flow match on traffic that is not from the\n \"port-under-test\".\n\nOf course any other feedback is hugely appreciated.\n\nThis RFC contains the core feature but lacks and some configurability\nthat might be interesting for the actual patch. Additionally, I would\nlike to measure the performance impact of enabling it in a loaded\nsystem.\n\nExample: a simple scenario with 2 veths pairs connected through OVS\nwith some ct. It ilustrates the fozen state reuse mentioned above.\n\n$ ovs-ofctl dump-flows ktest\n cookie=0x0, duration=2.104s, table=0, n_packets=0, n_bytes=0, priority=100,arp actions=NORMAL\n cookie=0x0, duration=2.104s, table=0, n_packets=0, n_bytes=0, priority=10,ip,in_port=\"p1_l\" actions=ct(table=1,zone=1)\n cookie=0x0, duration=2.104s, table=0, n_packets=0, n_bytes=0, priority=10,ip,in_port=\"p2_l\" actions=ct(table=1,zone=1)\n cookie=0x0, duration=5.096s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL\n cookie=0x0, duration=2.104s, table=0, n_packets=0, n_bytes=0, priority=1 actions=drop\n cookie=0x0, duration=2.104s, table=1, n_packets=0, n_bytes=0, priority=10,ct_state=+new+trk,ip,in_port=\"p1_l\" actions=ct(commit,zone=1),output:\"p2_l\"\n cookie=0x0, duration=2.103s, table=1, n_packets=0, n_bytes=0, priority=10,ct_state=+est+trk,ip,in_port=\"p1_l\" actions=output:\"p2_l\"\n cookie=0x0, duration=2.103s, table=1, n_packets=0, n_bytes=0, priority=10,ct_state=+est+trk,ip,in_port=\"p2_l\" actions=output:\"p1_l\"\n cookie=0x0, duration=2.103s, table=1, n_packets=0, n_bytes=0, priority=1 actions=drop\n\n$ ovs-appctl upcall/trace/show\nsystem@ovs-system: disabled\n$ ovs-appctl upcall/trace/create ktest \"in_port=1,icmp\"\n$ sudo ip netns exec ns1 ping -c 2 172.200.0.3\nPING 172.200.0.3 (172.200.0.3) 56(84) bytes of data.\n64 bytes from 172.200.0.3: icmp_seq=1 ttl=64 time=0.847 ms\n64 bytes from 172.200.0.3: icmp_seq=2 ttl=64 time=0.599 ms\n\n--- 172.200.0.3 ping statistics ---\n2 packets transmitted, 2 received, 0% packet loss, time 1005ms\nrtt min/avg/max/mdev = 0.599/0.723/0.847/0.124 ms\n\n$ ovs-appctl upcall/trace/show\nsystem@ovs-system: enabled, max_traces=64, current=1, filter: priority=0,icmp,in_port=1\n\n$ ovs-appctl upcall/trace/list\nsystem@ovs-system:\ntrace_id=0x1 ts=11:16:25.380 nodes=3 flow=icmp,in_port=1,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:01,dl_dst=0a:58:0a:f4:00:02,nw_src=172.200.0.2,nw_dst=172.200.0.3,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0\n\n$ ovs-appctl upcall/trace/get 0x1\n=== Trace 0x1 ===\n--- recirc_id=0x0 [11:16:25.380] ---\n\nInitial flow: icmp,in_port=1,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:01,dl_dst=0a:58:0a:f4:00:02,nw_src=172.200.0.2,nw_dst=172.200.0.3,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0\n\nbridge(\"ktest\")\n---------------\n 0. ip,in_port=1, priority 10\n ct(table=1,zone=1)\n drop\n -> Sets the packet to an untracked state, and clears all the conntrack fields.\n\n--- recirc_id=0x1 [11:16:25.381] ---\n\nInitial flow: recirc_id=0x1,ct_state=new|trk,ct_zone=1,ct_nw_src=172.200.0.2,ct_nw_dst=172.200.0.3,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,eth,icmp,in_port=1,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:01,dl_dst=0a:58:0a:f4:00:02,nw_src=172.200.0.2,nw_dst=172.200.0.3,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0\n\nbridge(\"ktest\")\n---------------\n thaw\n Resuming from table 1\n 1. ct_state=+new+trk,ip,in_port=1, priority 10\n ct(commit,zone=1)\n drop\n -> Sets the packet to an untracked state, and clears all the conntrack fields.\n output:2\n\n--- recirc_id=0x1 [11:16:26.385] ---\n\nInitial flow: recirc_id=0x1,ct_state=est|trk,ct_zone=1,ct_nw_src=172.200.0.2,ct_nw_dst=172.200.0.3,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,eth,icmp,in_port=1,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:01,dl_dst=0a:58:0a:f4:00:02,nw_src=172.200.0.2,nw_dst=172.200.0.3,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0\n\nbridge(\"ktest\")\n---------------\n thaw\n Resuming from table 1\n 1. ct_state=+est+trk,ip,in_port=1, priority 10\n output:2\n\n\nAdrian Moreno (2):\n ofproto-dpif-upcall: Introduce tracing scheleton.\n ofproto-dpif_upcall: Implement upcall tracing.\n\n ofproto/automake.mk | 2 +\n ofproto/ofproto-dpif-rid.h | 1 +\n ofproto/ofproto-dpif-trace.c | 4 +-\n ofproto/ofproto-dpif-trace.h | 4 +-\n ofproto/ofproto-dpif-upcall-trace.c | 442 ++++++++++++++++++++++++++++\n ofproto/ofproto-dpif-upcall-trace.h | 56 ++++\n ofproto/ofproto-dpif-upcall.c | 211 +++++++++++++\n ofproto/ofproto-dpif-xlate.c | 5 +-\n ofproto/ofproto-dpif-xlate.h | 4 +\n 9 files changed, 725 insertions(+), 4 deletions(-)\n create mode 100644 ofproto/ofproto-dpif-upcall-trace.c\n create mode 100644 ofproto/ofproto-dpif-upcall-trace.h"}