[LEDE-DEV,1/3] examples (test): add example/test that shows tx-queue-ing issue

Message ID 1496833771-32723-1-git-send-email-ardeleanalex@gmail.com
State Deferred
Delegated to: Felix Fietkau
Headers show

Commit Message

Alexandru Ardelean June 7, 2017, 11:09 a.m.
The context where we've seen this issue, is in
our internal tests (which are mostly shell scripts).

When we try to setup 200 interfaces with netifd,
there are about 200+ ubus entries (network.interface.* ).

When we use `ubus list` to iterate through those
entries, we've noticed the `ubus list` hangs indefinitely.
Naturally, the test times out.

After a few wrong assumptions, we've identified 4 issues:
* One addressed here 97ac89f
* One addressed here c09e4f0
* One to be addressed: tx_queue backlog getting filled up
  and messages getting silently discarded
* One to be addressed: the `retmsg` blob buf seems to be
  shared amongst clients, and there's a rare race where
  the `retmsg` seq & peer are inconsistent for a certain client.

The overall problem seems to be a case of
"fast producer, slow consumer".

If the `ubus list` output (from this script/test) is
piped to a file, the problem doesn't show up.
But if it's displayed on the serial console (which is
a slow consumer), the issue is easily reproduce-able
with this lua script.

In our tests (shell scripts) we don't display the `ubus list`
output to the serial console ; maybe the netifd data is
big enough that the tx_queue-ing issue occurs (also due
to socket buffer size).

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
 lua/test_many_ubus_entries.lua | 43 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 lua/test_many_ubus_entries.lua


diff --git a/lua/test_many_ubus_entries.lua b/lua/test_many_ubus_entries.lua
new file mode 100755
index 0000000..2324bf7
--- /dev/null
+++ b/lua/test_many_ubus_entries.lua
@@ -0,0 +1,43 @@ 
+#!/usr/bin/env lua
+require "ubus"
+require "uloop"
+local conn = ubus.connect()
+if not conn then
+	error("Failed to connect to ubus")
+local my_method = {
+	test_0 = {
+		hello = {
+			function(req, msg)
+				conn:reply(req, {message="foo"});
+				print("Call to function 'hello'")
+				for k, v in pairs(msg) do
+					print("key=" .. k .. " value=" .. tostring(v))
+				end
+			end, {id = ubus.INT32, msg = ubus.STRING }
+		},
+	}
+for i=1,2000 do
+	my_method["test_" .. tostring(i)] = my_method["test_0"]
+	function()
+		-- running "ubus list" to make sure ubus contexts are isolated
+		os.execute("ubus list")
+		conn:close()
+		uloop.cancel()
+	end,
+	2000