| Message ID | 20200731044944.30817-1-zajec5@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [V2,uhttpd] ubus: add new RESTful API | expand |
Hi Rafel, this is really great stuff. It would help me to get forward with my wifi controller. Could it be possible to subsribe to multiple sources to limit the connections to ubus? 2 SSIDs with 2.4 ad 5GHz would me 4 concurrent channels if I understand right. Kind regards, André Am 31.07.20 um 06:49 schrieb Rafał Miłecki: > From: Rafał Miłecki <rafal@milecki.pl> > > Initial uhttpd ubus API was fully based on JSON-RPC. That restricted it > from supporting ubus notifications that don't fit its model. > > Notifications require protocol that allows server to send data without > being polled. There are two candidates for that: > 1. Server-sent events > 2. WebSocket > > The later one is overcomplex for this simple task so ideally uhttps ubus > should support text-based server-sent events. It's not possible with > JSON-RPC without violating it. Specification requires server to reply > with Response object. Replying with text/event-stream is not allowed. > > All above led to designing new API that: > 1. Uses GET and POST requests > 2. Makes use of RESTful URLs > 3. Uses JSON-RPC in cleaner form and only for calling ubus methods > > This new API allows: > 1. Listing all ubus objects and their methods using GET <prefix>/list > 2. Listing object methods using GET <prefix>/list/<path> > 3. Listening to object notifications with GET <prefix>/subscribe/<path> > 4. Calling ubus methods using POST <prefix>/call/<path> > > JSON-RPC custom protocol was also simplified to: > 1. Use "method" member for ubus object method name > It was possible thanks to using RESTful URLs. Previously "method" > had to be "list" or "call". > 2. Reply with Error object on ubus method call error > This simplified "result" member format as it doesn't need to contain > ubus result code anymore. > > This patch doesn't break or change the old API. The biggest downside of > the new API is no support for batch requests. It's cost of using RESTful > URLs. It should not matter much as uhttpd supports keep alive. > > Example usages: > > 1. Getting all objects and their methods: > $ curl http://192.168.1.1/ubus/list > { > "dhcp": { > "ipv4leases": { > > }, > "ipv6leases": { > > } > }, > "log": { > "read": { > "lines": "number", > "stream": "boolean", > "oneshot": "boolean" > }, > "write": { > "event": "string" > } > } > } > > 2. Getting object methods: > $ curl http://192.168.1.1/ubus/list/log > { > "read": { > "lines": "number", > "stream": "boolean", > "oneshot": "boolean" > }, > "write": { > "event": "string" > } > } > > 3. Subscribing to notifications: > $ curl http://192.168.1.1/ubus/subscribe/foo > event: status > data: {"count":5} > > 4. Calling ubus object method: > $ curl -d '{ > "jsonrpc": "2.0", > "id": 1, > "method": "login", > "params": {"username": "root", "password": "password" } > }' http://192.168.1.1/ubus/call/session > { > "jsonrpc": "2.0", > "id": 1, > "result": { > "ubus_rpc_session": "01234567890123456789012345678901", > (...) > } > } > > $ curl -H 'Authorization: Bearer 01234567890123456789012345678901' -d '{ > "jsonrpc": "2.0", > "id": 1, > "method": "write", > "params": {"event": "Hello world" } > }' http://192.168.1.1/ubus/call/log > { > "jsonrpc": "2.0", > "id": 1, > "result": null > } > > Signed-off-by: Rafał Miłecki <rafal@milecki.pl> > --- > V2: Use "Authorization" with Bearer for rpcd session id / token > Treat missing session id as UH_UBUS_DEFAULT_SID > Fix "result" format (was: "result":{{"foo":"bar"}}) > --- > main.c | 8 +- > ubus.c | 326 +++++++++++++++++++++++++++++++++++++++++++++++++++---- > uhttpd.h | 5 + > 3 files changed, 318 insertions(+), 21 deletions(-) > > diff --git a/main.c b/main.c > index 26e74ec..73e3d42 100644 > --- a/main.c > +++ b/main.c > @@ -159,6 +159,7 @@ static int usage(const char *name) > " -U file Override ubus socket path\n" > " -a Do not authenticate JSON-RPC requests against UBUS session api\n" > " -X Enable CORS HTTP headers on JSON-RPC api\n" > + " -e Events subscription reconnection time (retry value)\n" > #endif > " -x string URL prefix for CGI handler, default is '/cgi-bin'\n" > " -y alias[=path] URL alias handle\n" > @@ -262,7 +263,7 @@ int main(int argc, char **argv) > init_defaults_pre(); > signal(SIGPIPE, SIG_IGN); > > - while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { > + while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:e:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { > switch(ch) { > #ifdef HAVE_TLS > case 'C': > @@ -490,11 +491,16 @@ int main(int argc, char **argv) > case 'X': > conf.ubus_cors = 1; > break; > + > + case 'e': > + conf.events_retry = atoi(optarg); > + break; > #else > case 'a': > case 'u': > case 'U': > case 'X': > + case 'e': > fprintf(stderr, "uhttpd: UBUS support not compiled, " > "ignoring -%c\n", ch); > break; > diff --git a/ubus.c b/ubus.c > index c22e07a..fd907db 100644 > --- a/ubus.c > +++ b/ubus.c > @@ -73,6 +73,7 @@ struct rpc_data { > > struct list_data { > bool verbose; > + bool add_object; > struct blob_buf *buf; > }; > > @@ -154,14 +155,14 @@ static void uh_ubus_add_cors_headers(struct client *cl) > ustream_printf(cl->us, "Access-Control-Allow-Credentials: true\r\n"); > } > > -static void uh_ubus_send_header(struct client *cl) > +static void uh_ubus_send_header(struct client *cl, int code, const char *summary, const char *content_type) > { > - ops->http_header(cl, 200, "OK"); > + ops->http_header(cl, code, summary); > > if (conf.ubus_cors) > uh_ubus_add_cors_headers(cl); > > - ustream_printf(cl->us, "Content-Type: application/json\r\n"); > + ustream_printf(cl->us, "Content-Type: %s\r\n", content_type); > > if (cl->request.method == UH_HTTP_MSG_OPTIONS) > ustream_printf(cl->us, "Content-Length: 0\r\n"); > @@ -217,12 +218,165 @@ static void uh_ubus_json_rpc_error(struct client *cl, enum rpc_error type) > uh_ubus_send_response(cl); > } > > +static void uh_ubus_error(struct client *cl, int code, const char *message) > +{ > + blobmsg_add_u32(&buf, "code", code); > + blobmsg_add_string(&buf, "message", message); > + uh_ubus_send_response(cl); > +} > + > +static void uh_ubus_posix_error(struct client *cl, int err) > +{ > + uh_ubus_error(cl, -err, strerror(err)); > +} > + > +static void uh_ubus_ubus_error(struct client *cl, int err) > +{ > + uh_ubus_error(cl, err, ubus_strerror(err)); > +} > + > +/* GET requests handling */ > + > +static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *obj, void *priv); > + > +static void uh_ubus_handle_get_list(struct client *cl, const char *path) > +{ > + static struct blob_buf tmp; > + struct list_data data = { .verbose = true, .add_object = !path, .buf = &tmp}; > + struct blob_attr *cur; > + int rem; > + int err; > + > + blob_buf_init(&tmp, 0); > + > + err = ubus_lookup(ctx, path, uh_ubus_list_cb, &data); > + if (err) { > + uh_ubus_send_header(cl, 500, "Ubus Protocol Error", "application/json"); > + uh_ubus_ubus_error(cl, err); > + return; > + } > + > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + blob_for_each_attr(cur, tmp.head, rem) > + blobmsg_add_blob(&buf, cur); > + uh_ubus_send_response(cl); > +} > + > +static int uh_ubus_subscription_notification_cb(struct ubus_context *ctx, > + struct ubus_object *obj, > + struct ubus_request_data *req, > + const char *method, > + struct blob_attr *msg) > +{ > + struct ubus_subscriber *s; > + struct dispatch_ubus *du; > + struct client *cl; > + char *json; > + > + s = container_of(obj, struct ubus_subscriber, obj); > + du = container_of(s, struct dispatch_ubus, sub); > + cl = container_of(du, struct client, dispatch.ubus); > + > + json = blobmsg_format_json(msg, true); > + if (json) { > + ops->chunk_printf(cl, "event: %s\ndata: %s\n\n", method, json); > + free(json); > + } > + > + return 0; > +} > + > +static void uh_ubus_subscription_notification_remove_cb(struct ubus_context *ctx, struct ubus_subscriber *s, uint32_t id) > +{ > + struct dispatch_ubus *du; > + struct client *cl; > + > + du = container_of(s, struct dispatch_ubus, sub); > + cl = container_of(du, struct client, dispatch.ubus); > + > + ops->request_done(cl); > +} > + > +static void uh_ubus_handle_get_subscribe(struct client *cl, const char *sid, const char *path) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + uint32_t id; > + int err; > + > + /* TODO: add ACL support */ > + if (!conf.ubus_noauth) { > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + uh_ubus_posix_error(cl, EACCES); > + return; > + } > + > + du->sub.cb = uh_ubus_subscription_notification_cb; > + du->sub.remove_cb = uh_ubus_subscription_notification_remove_cb; > + > + uh_client_ref(cl); > + > + err = ubus_register_subscriber(ctx, &du->sub); > + if (err) > + goto err_unref; > + > + err = ubus_lookup_id(ctx, path, &id); > + if (err) > + goto err_unregister; > + > + err = ubus_subscribe(ctx, &du->sub, id); > + if (err) > + goto err_unregister; > + > + uh_ubus_send_header(cl, 200, "OK", "text/event-stream"); > + > + if (conf.events_retry) > + ops->chunk_printf(cl, "retry: %d\n", conf.events_retry); > + > + return; > + > +err_unregister: > + ubus_unregister_subscriber(ctx, &du->sub); > +err_unref: > + uh_client_unref(cl); > + if (err) { > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + uh_ubus_ubus_error(cl, err); > + } > +} > + > +static void uh_ubus_handle_get(struct client *cl) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + const char *url = du->url; > + > + url += strlen(conf.ubus_prefix); > + > + if (!strcmp(url, "/list") || !strncmp(url, "/list/", strlen("/list/"))) { > + url += strlen("/list"); > + > + uh_ubus_handle_get_list(cl, *url ? url + 1 : NULL); > + } else if (!strncmp(url, "/subscribe/", strlen("/subscribe/"))) { > + url += strlen("/subscribe"); > + > + uh_ubus_handle_get_subscribe(cl, NULL, url + 1); > + } else { > + ops->http_header(cl, 404, "Not Found"); > + ustream_printf(cl->us, "\r\n"); > + ops->request_done(cl); > + } > +} > + > +/* POST requests handling */ > + > static void > uh_ubus_request_data_cb(struct ubus_request *req, int type, struct blob_attr *msg) > { > struct dispatch_ubus *du = container_of(req, struct dispatch_ubus, req); > + struct blob_attr *cur; > + int len; > > - blobmsg_add_field(&du->buf, BLOBMSG_TYPE_TABLE, "", blob_data(msg), blob_len(msg)); > + blob_for_each_attr(cur, msg, len) > + blobmsg_add_blob(&du->buf, cur); > } > > static void > @@ -235,13 +389,46 @@ uh_ubus_request_cb(struct ubus_request *req, int ret) > int rem; > > uloop_timeout_cancel(&du->timeout); > - uh_ubus_init_json_rpc_response(cl); > - r = blobmsg_open_array(&buf, "result"); > - blobmsg_add_u32(&buf, "", ret); > - blob_for_each_attr(cur, du->buf.head, rem) > - blobmsg_add_blob(&buf, cur); > - blobmsg_close_array(&buf, r); > - uh_ubus_send_response(cl); > + > + /* Legacy format always uses "result" array - even for errors and empty > + * results. */ > + if (du->legacy) { > + void *c; > + > + uh_ubus_init_json_rpc_response(cl); > + r = blobmsg_open_array(&buf, "result"); > + blobmsg_add_u32(&buf, "", ret); > + c = blobmsg_open_table(&buf, NULL); > + blob_for_each_attr(cur, du->buf.head, rem) > + blobmsg_add_blob(&buf, cur); > + blobmsg_close_table(&buf, c); > + blobmsg_close_array(&buf, r); > + uh_ubus_send_response(cl); > + return; > + } > + > + if (ret) { > + void *c; > + > + uh_ubus_init_json_rpc_response(cl); > + c = blobmsg_open_table(&buf, "error"); > + blobmsg_add_u32(&buf, "code", ret); > + blobmsg_add_string(&buf, "message", ubus_strerror(ret)); > + blobmsg_close_table(&buf, c); > + uh_ubus_send_response(cl); > + } else { > + uh_ubus_init_json_rpc_response(cl); > + if (blob_len(du->buf.head)) { > + r = blobmsg_open_table(&buf, "result"); > + blob_for_each_attr(cur, du->buf.head, rem) > + blobmsg_add_blob(&buf, cur); > + blobmsg_close_table(&buf, r); > + } else { > + blobmsg_add_field(&buf, BLOBMSG_TYPE_UNSPEC, "result", NULL, 0); > + } > + uh_ubus_send_response(cl); > + } > + > } > > static void > @@ -282,7 +469,7 @@ static void uh_ubus_request_free(struct client *cl) > > static void uh_ubus_single_error(struct client *cl, enum rpc_error type) > { > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > uh_ubus_json_rpc_error(cl, type); > ops->request_done(cl); > } > @@ -335,7 +522,8 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o > if (!obj->signature) > return; > > - o = blobmsg_open_table(data->buf, obj->path); > + if (data->add_object) > + o = blobmsg_open_table(data->buf, obj->path); > blob_for_each_attr(sig, obj->signature, rem) { > t = blobmsg_open_table(data->buf, blobmsg_name(sig)); > rem2 = blobmsg_data_len(sig); > @@ -366,13 +554,14 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o > } > blobmsg_close_table(data->buf, t); > } > - blobmsg_close_table(data->buf, o); > + if (data->add_object) > + blobmsg_close_table(data->buf, o); > } > > static void uh_ubus_send_list(struct client *cl, struct blob_attr *params) > { > struct blob_attr *cur, *dup; > - struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false }; > + struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false, .add_object = true }; > void *r; > int rem; > > @@ -471,7 +660,7 @@ static void uh_ubus_init_batch(struct client *cl) > struct dispatch_ubus *du = &cl->dispatch.ubus; > > du->array = true; > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > ops->chunk_printf(cl, "["); > } > > @@ -594,7 +783,7 @@ static void uh_ubus_data_done(struct client *cl) > > switch (obj ? json_object_get_type(obj) : json_type_null) { > case json_type_object: > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > return uh_ubus_handle_request_object(cl, obj); > case json_type_array: > uh_ubus_init_batch(cl); > @@ -604,6 +793,96 @@ static void uh_ubus_data_done(struct client *cl) > } > } > > +static void uh_ubus_call(struct client *cl, const char *path, const char *sid) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + struct json_object *obj = du->jsobj; > + struct rpc_data data = {}; > + enum rpc_error err = ERROR_PARSE; > + static struct blob_buf req; > + > + uh_client_ref(cl); > + > + if (!obj || json_object_get_type(obj) != json_type_object) > + goto error; > + > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + > + du->jsobj_cur = obj; > + blob_buf_init(&req, 0); > + if (!blobmsg_add_object(&req, obj)) > + goto error; > + > + if (!parse_json_rpc(&data, req.head)) > + goto error; > + > + du->func = data.method; > + if (ubus_lookup_id(ctx, path, &du->obj)) { > + err = ERROR_OBJECT; > + goto error; > + } > + > + if (!conf.ubus_noauth && !uh_ubus_allowed(sid, path, data.method)) { > + err = ERROR_ACCESS; > + goto error; > + } > + > + uh_ubus_send_request(cl, sid, data.params); > + goto out; > + > +error: > + uh_ubus_json_rpc_error(cl, err); > +out: > + if (data.params) > + free(data.params); > + > + uh_client_unref(cl); > +} > + > +enum ubus_hdr { > + HDR_AUTHORIZATION, > + __HDR_UBUS_MAX > +}; > + > +static void uh_ubus_handle_post(struct client *cl) > +{ > + static const struct blobmsg_policy hdr_policy[__HDR_UBUS_MAX] = { > + [HDR_AUTHORIZATION] = { "authorization", BLOBMSG_TYPE_STRING }, > + }; > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + struct blob_attr *tb[__HDR_UBUS_MAX]; > + const char *url = du->url; > + const char *auth; > + > + if (!strcmp(url, conf.ubus_prefix)) { > + du->legacy = true; > + uh_ubus_data_done(cl); > + return; > + } > + > + blobmsg_parse(hdr_policy, __HDR_UBUS_MAX, tb, blob_data(cl->hdr.head), blob_len(cl->hdr.head)); > + > + auth = UH_UBUS_DEFAULT_SID; > + if (tb[HDR_AUTHORIZATION]) { > + const char *tmp = blobmsg_get_string(tb[HDR_AUTHORIZATION]); > + > + if (!strncasecmp(tmp, "Bearer ", 7)) > + auth = tmp + 7; > + } > + > + url += strlen(conf.ubus_prefix); > + > + if (!strncmp(url, "/call/", strlen("/call/"))) { > + url += strlen("/call/"); > + > + uh_ubus_call(cl, url, auth); > + } else { > + ops->http_header(cl, 404, "Not Found"); > + ustream_printf(cl->us, "\r\n"); > + ops->request_done(cl); > + } > +} > + > static int uh_ubus_data_send(struct client *cl, const char *data, int len) > { > struct dispatch_ubus *du = &cl->dispatch.ubus; > @@ -626,21 +905,28 @@ error: > static void uh_ubus_handle_request(struct client *cl, char *url, struct path_info *pi) > { > struct dispatch *d = &cl->dispatch; > + struct dispatch_ubus *du = &d->ubus; > > blob_buf_init(&buf, 0); > > + du->url = url; > + du->legacy = false; > + > switch (cl->request.method) > { > + case UH_HTTP_MSG_GET: > + uh_ubus_handle_get(cl); > + break; > case UH_HTTP_MSG_POST: > d->data_send = uh_ubus_data_send; > - d->data_done = uh_ubus_data_done; > + d->data_done = uh_ubus_handle_post; > d->close_fds = uh_ubus_close_fds; > d->free = uh_ubus_request_free; > - d->ubus.jstok = json_tokener_new(); > + du->jstok = json_tokener_new(); > break; > > case UH_HTTP_MSG_OPTIONS: > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > ops->request_done(cl); > break; > > diff --git a/uhttpd.h b/uhttpd.h > index f77718e..75dd747 100644 > --- a/uhttpd.h > +++ b/uhttpd.h > @@ -82,6 +82,7 @@ struct config { > int ubus_noauth; > int ubus_cors; > int cgi_prefix_len; > + int events_retry; > struct list_head cgi_alias; > struct list_head lua_prefix; > }; > @@ -209,6 +210,7 @@ struct dispatch_ubus { > struct json_tokener *jstok; > struct json_object *jsobj; > struct json_object *jsobj_cur; > + const char *url; > int post_len; > > uint32_t obj; > @@ -218,6 +220,9 @@ struct dispatch_ubus { > bool req_pending; > bool array; > int array_idx; > + bool legacy; /* Got legacy request => use legacy reply */ > + > + struct ubus_subscriber sub; > }; > #endif > >
On 7/31/20 1:49 AM, Rafał Miłecki wrote: > From: Rafał Miłecki <rafal@milecki.pl> > > Initial uhttpd ubus API was fully based on JSON-RPC. That restricted it > from supporting ubus notifications that don't fit its model. > > Notifications require protocol that allows server to send data without > being polled. There are two candidates for that: > 1. Server-sent events I did a quick and dirty implementation of it some time ago, that works with everything as it is. You might want to check it out. https://bugs.openwrt.org/index.php?do=details&task_id=2248 Still, eager to see this work done! > 2. WebSocket > > The later one is overcomplex for this simple task so ideally uhttps ubus > should support text-based server-sent events. It's not possible with > JSON-RPC without violating it. Specification requires server to reply > with Response object. Replying with text/event-stream is not allowed. > > All above led to designing new API that: > 1. Uses GET and POST requests > 2. Makes use of RESTful URLs > 3. Uses JSON-RPC in cleaner form and only for calling ubus methods > > This new API allows: > 1. Listing all ubus objects and their methods using GET <prefix>/list > 2. Listing object methods using GET <prefix>/list/<path> > 3. Listening to object notifications with GET <prefix>/subscribe/<path> > 4. Calling ubus methods using POST <prefix>/call/<path> > > JSON-RPC custom protocol was also simplified to: > 1. Use "method" member for ubus object method name > It was possible thanks to using RESTful URLs. Previously "method" > had to be "list" or "call". > 2. Reply with Error object on ubus method call error > This simplified "result" member format as it doesn't need to contain > ubus result code anymore. > > This patch doesn't break or change the old API. The biggest downside of > the new API is no support for batch requests. It's cost of using RESTful > URLs. It should not matter much as uhttpd supports keep alive. > > Example usages: > > 1. Getting all objects and their methods: > $ curl http://192.168.1.1/ubus/list > { > "dhcp": { > "ipv4leases": { > > }, > "ipv6leases": { > > } > }, > "log": { > "read": { > "lines": "number", > "stream": "boolean", > "oneshot": "boolean" > }, > "write": { > "event": "string" > } > } > } > > 2. Getting object methods: > $ curl http://192.168.1.1/ubus/list/log > { > "read": { > "lines": "number", > "stream": "boolean", > "oneshot": "boolean" > }, > "write": { > "event": "string" > } > } > > 3. Subscribing to notifications: > $ curl http://192.168.1.1/ubus/subscribe/foo > event: status > data: {"count":5} > > 4. Calling ubus object method: > $ curl -d '{ > "jsonrpc": "2.0", > "id": 1, > "method": "login", > "params": {"username": "root", "password": "password" } > }' http://192.168.1.1/ubus/call/session > { > "jsonrpc": "2.0", > "id": 1, > "result": { > "ubus_rpc_session": "01234567890123456789012345678901", > (...) > } > } > > $ curl -H 'Authorization: Bearer 01234567890123456789012345678901' -d '{ > "jsonrpc": "2.0", > "id": 1, > "method": "write", > "params": {"event": "Hello world" } > }' http://192.168.1.1/ubus/call/log > { > "jsonrpc": "2.0", > "id": 1, > "result": null > } > > Signed-off-by: Rafał Miłecki <rafal@milecki.pl> > --- > V2: Use "Authorization" with Bearer for rpcd session id / token > Treat missing session id as UH_UBUS_DEFAULT_SID > Fix "result" format (was: "result":{{"foo":"bar"}}) > --- > main.c | 8 +- > ubus.c | 326 +++++++++++++++++++++++++++++++++++++++++++++++++++---- > uhttpd.h | 5 + > 3 files changed, 318 insertions(+), 21 deletions(-) > > diff --git a/main.c b/main.c > index 26e74ec..73e3d42 100644 > --- a/main.c > +++ b/main.c > @@ -159,6 +159,7 @@ static int usage(const char *name) > " -U file Override ubus socket path\n" > " -a Do not authenticate JSON-RPC requests against UBUS session api\n" > " -X Enable CORS HTTP headers on JSON-RPC api\n" > + " -e Events subscription reconnection time (retry value)\n" > #endif > " -x string URL prefix for CGI handler, default is '/cgi-bin'\n" > " -y alias[=path] URL alias handle\n" > @@ -262,7 +263,7 @@ int main(int argc, char **argv) > init_defaults_pre(); > signal(SIGPIPE, SIG_IGN); > > - while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { > + while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:e:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { > switch(ch) { > #ifdef HAVE_TLS > case 'C': > @@ -490,11 +491,16 @@ int main(int argc, char **argv) > case 'X': > conf.ubus_cors = 1; > break; > + > + case 'e': > + conf.events_retry = atoi(optarg); > + break; > #else > case 'a': > case 'u': > case 'U': > case 'X': > + case 'e': > fprintf(stderr, "uhttpd: UBUS support not compiled, " > "ignoring -%c\n", ch); > break; > diff --git a/ubus.c b/ubus.c > index c22e07a..fd907db 100644 > --- a/ubus.c > +++ b/ubus.c > @@ -73,6 +73,7 @@ struct rpc_data { > > struct list_data { > bool verbose; > + bool add_object; > struct blob_buf *buf; > }; > > @@ -154,14 +155,14 @@ static void uh_ubus_add_cors_headers(struct client *cl) > ustream_printf(cl->us, "Access-Control-Allow-Credentials: true\r\n"); > } > > -static void uh_ubus_send_header(struct client *cl) > +static void uh_ubus_send_header(struct client *cl, int code, const char *summary, const char *content_type) > { > - ops->http_header(cl, 200, "OK"); > + ops->http_header(cl, code, summary); > > if (conf.ubus_cors) > uh_ubus_add_cors_headers(cl); > > - ustream_printf(cl->us, "Content-Type: application/json\r\n"); > + ustream_printf(cl->us, "Content-Type: %s\r\n", content_type); > > if (cl->request.method == UH_HTTP_MSG_OPTIONS) > ustream_printf(cl->us, "Content-Length: 0\r\n"); > @@ -217,12 +218,165 @@ static void uh_ubus_json_rpc_error(struct client *cl, enum rpc_error type) > uh_ubus_send_response(cl); > } > > +static void uh_ubus_error(struct client *cl, int code, const char *message) > +{ > + blobmsg_add_u32(&buf, "code", code); > + blobmsg_add_string(&buf, "message", message); > + uh_ubus_send_response(cl); > +} > + > +static void uh_ubus_posix_error(struct client *cl, int err) > +{ > + uh_ubus_error(cl, -err, strerror(err)); > +} > + > +static void uh_ubus_ubus_error(struct client *cl, int err) > +{ > + uh_ubus_error(cl, err, ubus_strerror(err)); > +} > + > +/* GET requests handling */ > + > +static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *obj, void *priv); > + > +static void uh_ubus_handle_get_list(struct client *cl, const char *path) > +{ > + static struct blob_buf tmp; > + struct list_data data = { .verbose = true, .add_object = !path, .buf = &tmp}; > + struct blob_attr *cur; > + int rem; > + int err; > + > + blob_buf_init(&tmp, 0); > + > + err = ubus_lookup(ctx, path, uh_ubus_list_cb, &data); > + if (err) { > + uh_ubus_send_header(cl, 500, "Ubus Protocol Error", "application/json"); > + uh_ubus_ubus_error(cl, err); > + return; > + } > + > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + blob_for_each_attr(cur, tmp.head, rem) > + blobmsg_add_blob(&buf, cur); > + uh_ubus_send_response(cl); > +} > + > +static int uh_ubus_subscription_notification_cb(struct ubus_context *ctx, > + struct ubus_object *obj, > + struct ubus_request_data *req, > + const char *method, > + struct blob_attr *msg) > +{ > + struct ubus_subscriber *s; > + struct dispatch_ubus *du; > + struct client *cl; > + char *json; > + > + s = container_of(obj, struct ubus_subscriber, obj); > + du = container_of(s, struct dispatch_ubus, sub); > + cl = container_of(du, struct client, dispatch.ubus); > + > + json = blobmsg_format_json(msg, true); > + if (json) { > + ops->chunk_printf(cl, "event: %s\ndata: %s\n\n", method, json); > + free(json); > + } > + > + return 0; > +} > + > +static void uh_ubus_subscription_notification_remove_cb(struct ubus_context *ctx, struct ubus_subscriber *s, uint32_t id) > +{ > + struct dispatch_ubus *du; > + struct client *cl; > + > + du = container_of(s, struct dispatch_ubus, sub); > + cl = container_of(du, struct client, dispatch.ubus); > + > + ops->request_done(cl); > +} > + > +static void uh_ubus_handle_get_subscribe(struct client *cl, const char *sid, const char *path) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + uint32_t id; > + int err; > + > + /* TODO: add ACL support */ > + if (!conf.ubus_noauth) { > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + uh_ubus_posix_error(cl, EACCES); > + return; > + } > + > + du->sub.cb = uh_ubus_subscription_notification_cb; > + du->sub.remove_cb = uh_ubus_subscription_notification_remove_cb; > + > + uh_client_ref(cl); > + > + err = ubus_register_subscriber(ctx, &du->sub); > + if (err) > + goto err_unref; > + > + err = ubus_lookup_id(ctx, path, &id); > + if (err) > + goto err_unregister; > + > + err = ubus_subscribe(ctx, &du->sub, id); > + if (err) > + goto err_unregister; > + > + uh_ubus_send_header(cl, 200, "OK", "text/event-stream"); > + > + if (conf.events_retry) > + ops->chunk_printf(cl, "retry: %d\n", conf.events_retry); > + > + return; > + > +err_unregister: > + ubus_unregister_subscriber(ctx, &du->sub); > +err_unref: > + uh_client_unref(cl); > + if (err) { > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + uh_ubus_ubus_error(cl, err); > + } > +} > + > +static void uh_ubus_handle_get(struct client *cl) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + const char *url = du->url; > + > + url += strlen(conf.ubus_prefix); > + > + if (!strcmp(url, "/list") || !strncmp(url, "/list/", strlen("/list/"))) { > + url += strlen("/list"); > + > + uh_ubus_handle_get_list(cl, *url ? url + 1 : NULL); > + } else if (!strncmp(url, "/subscribe/", strlen("/subscribe/"))) { > + url += strlen("/subscribe"); > + > + uh_ubus_handle_get_subscribe(cl, NULL, url + 1); > + } else { > + ops->http_header(cl, 404, "Not Found"); > + ustream_printf(cl->us, "\r\n"); > + ops->request_done(cl); > + } > +} > + > +/* POST requests handling */ > + > static void > uh_ubus_request_data_cb(struct ubus_request *req, int type, struct blob_attr *msg) > { > struct dispatch_ubus *du = container_of(req, struct dispatch_ubus, req); > + struct blob_attr *cur; > + int len; > > - blobmsg_add_field(&du->buf, BLOBMSG_TYPE_TABLE, "", blob_data(msg), blob_len(msg)); > + blob_for_each_attr(cur, msg, len) > + blobmsg_add_blob(&du->buf, cur); > } > > static void > @@ -235,13 +389,46 @@ uh_ubus_request_cb(struct ubus_request *req, int ret) > int rem; > > uloop_timeout_cancel(&du->timeout); > - uh_ubus_init_json_rpc_response(cl); > - r = blobmsg_open_array(&buf, "result"); > - blobmsg_add_u32(&buf, "", ret); > - blob_for_each_attr(cur, du->buf.head, rem) > - blobmsg_add_blob(&buf, cur); > - blobmsg_close_array(&buf, r); > - uh_ubus_send_response(cl); > + > + /* Legacy format always uses "result" array - even for errors and empty > + * results. */ > + if (du->legacy) { > + void *c; > + > + uh_ubus_init_json_rpc_response(cl); > + r = blobmsg_open_array(&buf, "result"); > + blobmsg_add_u32(&buf, "", ret); > + c = blobmsg_open_table(&buf, NULL); > + blob_for_each_attr(cur, du->buf.head, rem) > + blobmsg_add_blob(&buf, cur); > + blobmsg_close_table(&buf, c); > + blobmsg_close_array(&buf, r); > + uh_ubus_send_response(cl); > + return; > + } > + > + if (ret) { > + void *c; > + > + uh_ubus_init_json_rpc_response(cl); > + c = blobmsg_open_table(&buf, "error"); > + blobmsg_add_u32(&buf, "code", ret); > + blobmsg_add_string(&buf, "message", ubus_strerror(ret)); > + blobmsg_close_table(&buf, c); > + uh_ubus_send_response(cl); > + } else { > + uh_ubus_init_json_rpc_response(cl); > + if (blob_len(du->buf.head)) { > + r = blobmsg_open_table(&buf, "result"); > + blob_for_each_attr(cur, du->buf.head, rem) > + blobmsg_add_blob(&buf, cur); > + blobmsg_close_table(&buf, r); > + } else { > + blobmsg_add_field(&buf, BLOBMSG_TYPE_UNSPEC, "result", NULL, 0); > + } > + uh_ubus_send_response(cl); > + } > + > } > > static void > @@ -282,7 +469,7 @@ static void uh_ubus_request_free(struct client *cl) > > static void uh_ubus_single_error(struct client *cl, enum rpc_error type) > { > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > uh_ubus_json_rpc_error(cl, type); > ops->request_done(cl); > } > @@ -335,7 +522,8 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o > if (!obj->signature) > return; > > - o = blobmsg_open_table(data->buf, obj->path); > + if (data->add_object) > + o = blobmsg_open_table(data->buf, obj->path); > blob_for_each_attr(sig, obj->signature, rem) { > t = blobmsg_open_table(data->buf, blobmsg_name(sig)); > rem2 = blobmsg_data_len(sig); > @@ -366,13 +554,14 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o > } > blobmsg_close_table(data->buf, t); > } > - blobmsg_close_table(data->buf, o); > + if (data->add_object) > + blobmsg_close_table(data->buf, o); > } > > static void uh_ubus_send_list(struct client *cl, struct blob_attr *params) > { > struct blob_attr *cur, *dup; > - struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false }; > + struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false, .add_object = true }; > void *r; > int rem; > > @@ -471,7 +660,7 @@ static void uh_ubus_init_batch(struct client *cl) > struct dispatch_ubus *du = &cl->dispatch.ubus; > > du->array = true; > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > ops->chunk_printf(cl, "["); > } > > @@ -594,7 +783,7 @@ static void uh_ubus_data_done(struct client *cl) > > switch (obj ? json_object_get_type(obj) : json_type_null) { > case json_type_object: > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > return uh_ubus_handle_request_object(cl, obj); > case json_type_array: > uh_ubus_init_batch(cl); > @@ -604,6 +793,96 @@ static void uh_ubus_data_done(struct client *cl) > } > } > > +static void uh_ubus_call(struct client *cl, const char *path, const char *sid) > +{ > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + struct json_object *obj = du->jsobj; > + struct rpc_data data = {}; > + enum rpc_error err = ERROR_PARSE; > + static struct blob_buf req; > + > + uh_client_ref(cl); > + > + if (!obj || json_object_get_type(obj) != json_type_object) > + goto error; > + > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > + > + du->jsobj_cur = obj; > + blob_buf_init(&req, 0); > + if (!blobmsg_add_object(&req, obj)) > + goto error; > + > + if (!parse_json_rpc(&data, req.head)) > + goto error; > + > + du->func = data.method; > + if (ubus_lookup_id(ctx, path, &du->obj)) { > + err = ERROR_OBJECT; > + goto error; > + } > + > + if (!conf.ubus_noauth && !uh_ubus_allowed(sid, path, data.method)) { > + err = ERROR_ACCESS; > + goto error; > + } > + > + uh_ubus_send_request(cl, sid, data.params); > + goto out; > + > +error: > + uh_ubus_json_rpc_error(cl, err); > +out: > + if (data.params) > + free(data.params); > + > + uh_client_unref(cl); > +} > + > +enum ubus_hdr { > + HDR_AUTHORIZATION, > + __HDR_UBUS_MAX > +}; > + > +static void uh_ubus_handle_post(struct client *cl) > +{ > + static const struct blobmsg_policy hdr_policy[__HDR_UBUS_MAX] = { > + [HDR_AUTHORIZATION] = { "authorization", BLOBMSG_TYPE_STRING }, > + }; > + struct dispatch_ubus *du = &cl->dispatch.ubus; > + struct blob_attr *tb[__HDR_UBUS_MAX]; > + const char *url = du->url; > + const char *auth; > + > + if (!strcmp(url, conf.ubus_prefix)) { > + du->legacy = true; > + uh_ubus_data_done(cl); > + return; > + } > + > + blobmsg_parse(hdr_policy, __HDR_UBUS_MAX, tb, blob_data(cl->hdr.head), blob_len(cl->hdr.head)); > + > + auth = UH_UBUS_DEFAULT_SID; > + if (tb[HDR_AUTHORIZATION]) { > + const char *tmp = blobmsg_get_string(tb[HDR_AUTHORIZATION]); > + > + if (!strncasecmp(tmp, "Bearer ", 7)) > + auth = tmp + 7; > + } > + > + url += strlen(conf.ubus_prefix); > + > + if (!strncmp(url, "/call/", strlen("/call/"))) { > + url += strlen("/call/"); > + > + uh_ubus_call(cl, url, auth); > + } else { > + ops->http_header(cl, 404, "Not Found"); > + ustream_printf(cl->us, "\r\n"); > + ops->request_done(cl); > + } > +} > + > static int uh_ubus_data_send(struct client *cl, const char *data, int len) > { > struct dispatch_ubus *du = &cl->dispatch.ubus; > @@ -626,21 +905,28 @@ error: > static void uh_ubus_handle_request(struct client *cl, char *url, struct path_info *pi) > { > struct dispatch *d = &cl->dispatch; > + struct dispatch_ubus *du = &d->ubus; > > blob_buf_init(&buf, 0); > > + du->url = url; > + du->legacy = false; > + > switch (cl->request.method) > { > + case UH_HTTP_MSG_GET: > + uh_ubus_handle_get(cl); > + break; > case UH_HTTP_MSG_POST: > d->data_send = uh_ubus_data_send; > - d->data_done = uh_ubus_data_done; > + d->data_done = uh_ubus_handle_post; > d->close_fds = uh_ubus_close_fds; > d->free = uh_ubus_request_free; > - d->ubus.jstok = json_tokener_new(); > + du->jstok = json_tokener_new(); > break; > > case UH_HTTP_MSG_OPTIONS: > - uh_ubus_send_header(cl); > + uh_ubus_send_header(cl, 200, "OK", "application/json"); > ops->request_done(cl); > break; > > diff --git a/uhttpd.h b/uhttpd.h > index f77718e..75dd747 100644 > --- a/uhttpd.h > +++ b/uhttpd.h > @@ -82,6 +82,7 @@ struct config { > int ubus_noauth; > int ubus_cors; > int cgi_prefix_len; > + int events_retry; > struct list_head cgi_alias; > struct list_head lua_prefix; > }; > @@ -209,6 +210,7 @@ struct dispatch_ubus { > struct json_tokener *jstok; > struct json_object *jsobj; > struct json_object *jsobj_cur; > + const char *url; > int post_len; > > uint32_t obj; > @@ -218,6 +220,9 @@ struct dispatch_ubus { > bool req_pending; > bool array; > int array_idx; > + bool legacy; /* Got legacy request => use legacy reply */ > + > + struct ubus_subscriber sub; > }; > #endif > >
On 31.07.2020 13:02, Andre Valentin wrote: > this is really great stuff. It would help me to get forward with my wifi controller. > Could it be possible to subsribe to multiple sources to limit the connections to ubus? > 2 SSIDs with 2.4 ad 5GHz would me 4 concurrent channels if I understand right. I'm happy someone finds it useful! If you mean hostapd.* objects, that's right. That would require you to use: /ubus/subscribe/hostapd.wlan0 /ubus/subscribe/hostapd.wlan0-1 /ubus/subscribe/hostapd.wlan1 /ubus/subscribe/hostapd.wlan1-1 For subscribing to multiple objects we would need to: 1. Stick to GET due to the way EventSource works 2. Pick some more generic URL 3. Adjust output format ("event" and "data" fields) So my guess would be something like: $ curl http://192.168.1.1/ubus/subscribe?path=hostapd.wlan0&path=hostapd.wlan1 event: hostapd.wlan1 status data: {"count":5} event: hostapd.wlan0-1 status data: {"count":5} event: hostapd.wlan1 status data: {"count":7} Regarding parsing events stream, event names with spaces seem to be OK: https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream field value can use any scalar value other than line break char. We should use some special character as separator of object name and notification name. It must be something that ubus doesn't use in any of them. Should space be OK? Or should we use some more fancy char? I quickly tested space and it seems to work well in Firefox and Chromium.
On Fri, 31 Jul 2020 at 14:35, Nicolas Pace <nico@libre.ws> wrote: > On 7/31/20 1:49 AM, Rafał Miłecki wrote: > > From: Rafał Miłecki <rafal@milecki.pl> > > > > Initial uhttpd ubus API was fully based on JSON-RPC. That restricted it > > from supporting ubus notifications that don't fit its model. > > > > Notifications require protocol that allows server to send data without > > being polled. There are two candidates for that: > > 1. Server-sent events > > I did a quick and dirty implementation of it some time ago, that works > with everything as it is. > You might want to check it out. > https://bugs.openwrt.org/index.php?do=details&task_id=2248 > > Still, eager to see this work done! My first initial implementation was also PHP based :) It didn't work well with lighttpd though as I had to disable buffering using server.stream-response-body That and looking for something more generic (not PHP based) made me look at uhttpd.
Hi! Am 03.08.20 um 07:49 schrieb Rafał Miłecki: > On 31.07.2020 13:02, Andre Valentin wrote: >> this is really great stuff. It would help me to get forward with my wifi controller. >> Could it be possible to subsribe to multiple sources to limit the connections to ubus? >> 2 SSIDs with 2.4 ad 5GHz would me 4 concurrent channels if I understand right. > > I'm happy someone finds it useful! > > If you mean hostapd.* objects, that's right. That would require you to > use: > /ubus/subscribe/hostapd.wlan0 > /ubus/subscribe/hostapd.wlan0-1 > /ubus/subscribe/hostapd.wlan1 > /ubus/subscribe/hostapd.wlan1-1 > > For subscribing to multiple objects we would need to: > 1. Stick to GET due to the way EventSource works > 2. Pick some more generic URL > 3. Adjust output format ("event" and "data" fields) > > So my guess would be something like: > > $ curl http://192.168.1.1/ubus/subscribe?path=hostapd.wlan0&path=hostapd.wlan1 Good idea! > event: hostapd.wlan1 status > data: {"count":5} > > event: hostapd.wlan0-1 status > data: {"count":5} > > event: hostapd.wlan1 status > data: {"count":7} > > > Regarding parsing events stream, event names with spaces seem to be OK: > https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream > field value can use any scalar value other than line break char. Why do you need the status there, is it part of the standard? > We should use some special character as separator of object name and > notification name. It must be something that ubus doesn't use in any of > them. Should space be OK? Or should we use some more fancy char? I > quickly tested space and it seems to work well in Firefox and Chromium. Oh, I'm nut sure. But I think space is fine. Did you use a special uhttpd version. I couldn't apply your patch to the uhttpd in openwrt master. Kind regards, André
Hi, > Regarding parsing events stream, event names with spaces seem to be OK: > https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream To me it feels quirky to separate the path and the type of the event by space. Personally I'd only report the type as "event:" and move the source path into the JSON data portion, or even omit it entirely. Considering the design of the client side API: eventSource.addEventListener(type, handler); Most use-cases probably want to register a handler for a specific event type, e.g. "status", and not N handlers to handle the different object path variations, so given the subscribe examples in the previous mail: eventSource.addEventListener("status", (ev) => { ... }) instead of eventSource.addEventListener("hostapd.wlan0 status", (ev) => { ... }) eventSource.addEventListener("hostapd.wlan0-1 status", (ev) => { ... }) eventSource.addEventListener("hostapd.wlan1 status", (ev) => { ... }) eventSource.addEventListener("hostapd.wlan1-1 status", (ev) => { ... }) Granted, one could use the `onmessage` event to implement a catch-all handler which is then filtering and dispatching according to the type, but even then string operations like split(), indexOf(), regex matches or similar would be required to match event types, that feels unelegant and not very performant. ~ Jo
On 04.08.2020 09:43, Andre Valentin wrote: > Am 03.08.20 um 07:49 schrieb Rafał Miłecki: >> On 31.07.2020 13:02, Andre Valentin wrote: >>> this is really great stuff. It would help me to get forward with my wifi controller. >>> Could it be possible to subsribe to multiple sources to limit the connections to ubus? >>> 2 SSIDs with 2.4 ad 5GHz would me 4 concurrent channels if I understand right. >> >> I'm happy someone finds it useful! >> >> If you mean hostapd.* objects, that's right. That would require you to >> use: >> /ubus/subscribe/hostapd.wlan0 >> /ubus/subscribe/hostapd.wlan0-1 >> /ubus/subscribe/hostapd.wlan1 >> /ubus/subscribe/hostapd.wlan1-1 >> >> For subscribing to multiple objects we would need to: >> 1. Stick to GET due to the way EventSource works >> 2. Pick some more generic URL >> 3. Adjust output format ("event" and "data" fields) >> >> So my guess would be something like: >> >> $ curl http://192.168.1.1/ubus/subscribe?path=hostapd.wlan0&path=hostapd.wlan1 > > Good idea! > >> event: hostapd.wlan1 status >> data: {"count":5} >> >> event: hostapd.wlan0-1 status >> data: {"count":5} >> >> event: hostapd.wlan1 status >> data: {"count":7} >> >> >> Regarding parsing events stream, event names with spaces seem to be OK: >> https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream >> field value can use any scalar value other than line break char. > > Why do you need the status there, is it part of the standard? That was meant to separate object name from notification name. >> We should use some special character as separator of object name and >> notification name. It must be something that ubus doesn't use in any of >> them. Should space be OK? Or should we use some more fancy char? I >> quickly tested space and it seems to work well in Firefox and Chromium. > > Oh, I'm nut sure. But I think space is fine. > > Did you use a special uhttpd version. I couldn't apply your patch to the uhttpd in openwrt master. There are few more uhttpd pending patches that I sent, see: https://patchwork.ozlabs.org/project/openwrt/list/?series=&submitter=5824&state=*&q=uhttpd&archive=&delegate=
On 04.08.2020 09:57, Jo-Philipp Wich wrote: >> Regarding parsing events stream, event names with spaces seem to be OK: >> https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream > > To me it feels quirky to separate the path and the type of the event by space. > > Personally I'd only report the type as "event:" and move the source path into > the JSON data portion, This seems to break hierarchy. I see correct hierarchy as: 1. Object 2. Notification type 3. Notification data If we put type in "event:" and object path and notification data in "data:" it seems unnatural. > or even omit it entirely. I'm sure one may need to distinct notifications from wlan0 vs. wlan1. That's probably not an option unless I missed something. > Considering the design of the client side API: > > eventSource.addEventListener(type, handler); > > Most use-cases probably want to register a handler for a specific event type, > e.g. "status", and not N handlers to handle the different object path > variations, so given the subscribe examples in the previous mail: > > eventSource.addEventListener("status", (ev) => { ... }) > > instead of > > eventSource.addEventListener("hostapd.wlan0 status", (ev) => { ... }) > eventSource.addEventListener("hostapd.wlan0-1 status", (ev) => { ... }) > eventSource.addEventListener("hostapd.wlan1 status", (ev) => { ... }) > eventSource.addEventListener("hostapd.wlan1-1 status", (ev) => { ... }) I see how it simplifies JavaScript code though so I'm confused. Should we maybe totally drop "event:" and just put everything (object path, event type, data) in "data:"? > Granted, one could use the `onmessage` event to implement a catch-all handler > which is then filtering and dispatching according to the type, but even then > string operations like split(), indexOf(), regex matches or similar would be > required to match event types, that feels unelegant and not very performant. I'm not sure if "message" event fires for *named* events (event: foo).
Hi! Am 04.08.20 um 18:40 schrieb Rafał Miłecki: > On 04.08.2020 09:43, Andre Valentin wrote: >> Am 03.08.20 um 07:49 schrieb Rafał Miłecki: >>> On 31.07.2020 13:02, Andre Valentin wrote: >>>> this is really great stuff. It would help me to get forward with my wifi controller. >>>> Could it be possible to subsribe to multiple sources to limit the connections to ubus? >>>> 2 SSIDs with 2.4 ad 5GHz would me 4 concurrent channels if I understand right. >>> >>> I'm happy someone finds it useful! >>> >>> If you mean hostapd.* objects, that's right. That would require you to >>> use: >>> /ubus/subscribe/hostapd.wlan0 >>> /ubus/subscribe/hostapd.wlan0-1 >>> /ubus/subscribe/hostapd.wlan1 >>> /ubus/subscribe/hostapd.wlan1-1 >>> >>> For subscribing to multiple objects we would need to: >>> 1. Stick to GET due to the way EventSource works >>> 2. Pick some more generic URL >>> 3. Adjust output format ("event" and "data" fields) >>> >>> So my guess would be something like: >>> >>> $ curl http://192.168.1.1/ubus/subscribe?path=hostapd.wlan0&path=hostapd.wlan1 >> >> Good idea! >> >>> event: hostapd.wlan1 status >>> data: {"count":5} >>> >>> event: hostapd.wlan0-1 status >>> data: {"count":5} >>> >>> event: hostapd.wlan1 status >>> data: {"count":7} >>> >>> >>> Regarding parsing events stream, event names with spaces seem to be OK: >>> https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream >>> field value can use any scalar value other than line break char. >> >> Why do you need the status there, is it part of the standard? > > That was meant to separate object name from notification name. > OK, I understand. > >>> We should use some special character as separator of object name and >>> notification name. It must be something that ubus doesn't use in any of >>> them. Should space be OK? Or should we use some more fancy char? I >>> quickly tested space and it seems to work well in Firefox and Chromium. >> >> Oh, I'm nut sure. But I think space is fine. >> >> Did you use a special uhttpd version. I couldn't apply your patch to the uhttpd in openwrt master. > > There are few more uhttpd pending patches that I sent, see: > https://patchwork.ozlabs.org/project/openwrt/list/?series=&submitter=5824&state=*&q=uhttpd&archive=&delegate= Okay, will give it a try again at the weekend. Kind regards, André
On 08.08.2020 01:47, Andre Valentin wrote: > I'm just experimenting a bit with the patch. > There were multiple this: You need to reply to all, this is public stuff potentially affecting all OpenWrt users. > 1) > I habe some mips routers and an ipq806x based router. > After I added that patch, I cannot authenticate on via json-rpc on the ipq806x. > I remove the patch, rebuild. It works again. > This does not seem to happen on the mips routers. I can't think of any explanation for this. Platform or endianess should not affect ubus behaviour. Maybe you just missed something in your rpcd confguration? Acl stuff? I can't say anything without providing more debugging info. > 2) > Authentication over rest-api works fine: > "jsonrpc": "2.0", > "id": 1, > "result": { > "ubus_rpc_session": "3653e64078f1f6ebaf4803e67c18fa2a", > "timeout": 300, Perfect. > But if I try to subscribe I get this error: >> GET /ubus/subscribe/hostapd.wap-knet1 HTTP/1.1 >> Host: ap-av-grwz >> User-Agent: curl/7.64.0 >> Accept: */* >> Authorization: Bearer 3653e64078f1f6ebaf4803e67c18fa2a >> > < HTTP/1.1 200 OK > HTTP/1.1 200 OK > < Connection: Keep-Alive > Connection: Keep-Alive > < Transfer-Encoding: chunked > Transfer-Encoding: chunked > < Keep-Alive: timeout=20 > Keep-Alive: timeout=20 > < Content-Type: application/json > Content-Type: application/json > > < > * Connection #0 to host ap-av-grwz left intact > {"code":-13,"message":"Permission denied"} > > I do not have an idea what's wrong, perhaps you have an idea? Sure, it means you / your session is not allowed to subscribe. See: if (!conf.ubus_noauth && !uh_ubus_allowed(sid, path, data.method)) { err = ERROR_ACCESS; goto error; } Until we add acl.d rule for allowing subscription access it's possible only with ubus_noauth. I'm going to work on proper / new acl.d as soon as I get this patch accepted.
diff --git a/main.c b/main.c index 26e74ec..73e3d42 100644 --- a/main.c +++ b/main.c @@ -159,6 +159,7 @@ static int usage(const char *name) " -U file Override ubus socket path\n" " -a Do not authenticate JSON-RPC requests against UBUS session api\n" " -X Enable CORS HTTP headers on JSON-RPC api\n" + " -e Events subscription reconnection time (retry value)\n" #endif " -x string URL prefix for CGI handler, default is '/cgi-bin'\n" " -y alias[=path] URL alias handle\n" @@ -262,7 +263,7 @@ int main(int argc, char **argv) init_defaults_pre(); signal(SIGPIPE, SIG_IGN); - while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { + while ((ch = getopt(argc, argv, "A:aC:c:Dd:E:e:fh:H:I:i:K:k:L:l:m:N:n:P:p:qRr:Ss:T:t:U:u:Xx:y:")) != -1) { switch(ch) { #ifdef HAVE_TLS case 'C': @@ -490,11 +491,16 @@ int main(int argc, char **argv) case 'X': conf.ubus_cors = 1; break; + + case 'e': + conf.events_retry = atoi(optarg); + break; #else case 'a': case 'u': case 'U': case 'X': + case 'e': fprintf(stderr, "uhttpd: UBUS support not compiled, " "ignoring -%c\n", ch); break; diff --git a/ubus.c b/ubus.c index c22e07a..fd907db 100644 --- a/ubus.c +++ b/ubus.c @@ -73,6 +73,7 @@ struct rpc_data { struct list_data { bool verbose; + bool add_object; struct blob_buf *buf; }; @@ -154,14 +155,14 @@ static void uh_ubus_add_cors_headers(struct client *cl) ustream_printf(cl->us, "Access-Control-Allow-Credentials: true\r\n"); } -static void uh_ubus_send_header(struct client *cl) +static void uh_ubus_send_header(struct client *cl, int code, const char *summary, const char *content_type) { - ops->http_header(cl, 200, "OK"); + ops->http_header(cl, code, summary); if (conf.ubus_cors) uh_ubus_add_cors_headers(cl); - ustream_printf(cl->us, "Content-Type: application/json\r\n"); + ustream_printf(cl->us, "Content-Type: %s\r\n", content_type); if (cl->request.method == UH_HTTP_MSG_OPTIONS) ustream_printf(cl->us, "Content-Length: 0\r\n"); @@ -217,12 +218,165 @@ static void uh_ubus_json_rpc_error(struct client *cl, enum rpc_error type) uh_ubus_send_response(cl); } +static void uh_ubus_error(struct client *cl, int code, const char *message) +{ + blobmsg_add_u32(&buf, "code", code); + blobmsg_add_string(&buf, "message", message); + uh_ubus_send_response(cl); +} + +static void uh_ubus_posix_error(struct client *cl, int err) +{ + uh_ubus_error(cl, -err, strerror(err)); +} + +static void uh_ubus_ubus_error(struct client *cl, int err) +{ + uh_ubus_error(cl, err, ubus_strerror(err)); +} + +/* GET requests handling */ + +static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *obj, void *priv); + +static void uh_ubus_handle_get_list(struct client *cl, const char *path) +{ + static struct blob_buf tmp; + struct list_data data = { .verbose = true, .add_object = !path, .buf = &tmp}; + struct blob_attr *cur; + int rem; + int err; + + blob_buf_init(&tmp, 0); + + err = ubus_lookup(ctx, path, uh_ubus_list_cb, &data); + if (err) { + uh_ubus_send_header(cl, 500, "Ubus Protocol Error", "application/json"); + uh_ubus_ubus_error(cl, err); + return; + } + + uh_ubus_send_header(cl, 200, "OK", "application/json"); + blob_for_each_attr(cur, tmp.head, rem) + blobmsg_add_blob(&buf, cur); + uh_ubus_send_response(cl); +} + +static int uh_ubus_subscription_notification_cb(struct ubus_context *ctx, + struct ubus_object *obj, + struct ubus_request_data *req, + const char *method, + struct blob_attr *msg) +{ + struct ubus_subscriber *s; + struct dispatch_ubus *du; + struct client *cl; + char *json; + + s = container_of(obj, struct ubus_subscriber, obj); + du = container_of(s, struct dispatch_ubus, sub); + cl = container_of(du, struct client, dispatch.ubus); + + json = blobmsg_format_json(msg, true); + if (json) { + ops->chunk_printf(cl, "event: %s\ndata: %s\n\n", method, json); + free(json); + } + + return 0; +} + +static void uh_ubus_subscription_notification_remove_cb(struct ubus_context *ctx, struct ubus_subscriber *s, uint32_t id) +{ + struct dispatch_ubus *du; + struct client *cl; + + du = container_of(s, struct dispatch_ubus, sub); + cl = container_of(du, struct client, dispatch.ubus); + + ops->request_done(cl); +} + +static void uh_ubus_handle_get_subscribe(struct client *cl, const char *sid, const char *path) +{ + struct dispatch_ubus *du = &cl->dispatch.ubus; + uint32_t id; + int err; + + /* TODO: add ACL support */ + if (!conf.ubus_noauth) { + uh_ubus_send_header(cl, 200, "OK", "application/json"); + uh_ubus_posix_error(cl, EACCES); + return; + } + + du->sub.cb = uh_ubus_subscription_notification_cb; + du->sub.remove_cb = uh_ubus_subscription_notification_remove_cb; + + uh_client_ref(cl); + + err = ubus_register_subscriber(ctx, &du->sub); + if (err) + goto err_unref; + + err = ubus_lookup_id(ctx, path, &id); + if (err) + goto err_unregister; + + err = ubus_subscribe(ctx, &du->sub, id); + if (err) + goto err_unregister; + + uh_ubus_send_header(cl, 200, "OK", "text/event-stream"); + + if (conf.events_retry) + ops->chunk_printf(cl, "retry: %d\n", conf.events_retry); + + return; + +err_unregister: + ubus_unregister_subscriber(ctx, &du->sub); +err_unref: + uh_client_unref(cl); + if (err) { + uh_ubus_send_header(cl, 200, "OK", "application/json"); + uh_ubus_ubus_error(cl, err); + } +} + +static void uh_ubus_handle_get(struct client *cl) +{ + struct dispatch_ubus *du = &cl->dispatch.ubus; + const char *url = du->url; + + url += strlen(conf.ubus_prefix); + + if (!strcmp(url, "/list") || !strncmp(url, "/list/", strlen("/list/"))) { + url += strlen("/list"); + + uh_ubus_handle_get_list(cl, *url ? url + 1 : NULL); + } else if (!strncmp(url, "/subscribe/", strlen("/subscribe/"))) { + url += strlen("/subscribe"); + + uh_ubus_handle_get_subscribe(cl, NULL, url + 1); + } else { + ops->http_header(cl, 404, "Not Found"); + ustream_printf(cl->us, "\r\n"); + ops->request_done(cl); + } +} + +/* POST requests handling */ + static void uh_ubus_request_data_cb(struct ubus_request *req, int type, struct blob_attr *msg) { struct dispatch_ubus *du = container_of(req, struct dispatch_ubus, req); + struct blob_attr *cur; + int len; - blobmsg_add_field(&du->buf, BLOBMSG_TYPE_TABLE, "", blob_data(msg), blob_len(msg)); + blob_for_each_attr(cur, msg, len) + blobmsg_add_blob(&du->buf, cur); } static void @@ -235,13 +389,46 @@ uh_ubus_request_cb(struct ubus_request *req, int ret) int rem; uloop_timeout_cancel(&du->timeout); - uh_ubus_init_json_rpc_response(cl); - r = blobmsg_open_array(&buf, "result"); - blobmsg_add_u32(&buf, "", ret); - blob_for_each_attr(cur, du->buf.head, rem) - blobmsg_add_blob(&buf, cur); - blobmsg_close_array(&buf, r); - uh_ubus_send_response(cl); + + /* Legacy format always uses "result" array - even for errors and empty + * results. */ + if (du->legacy) { + void *c; + + uh_ubus_init_json_rpc_response(cl); + r = blobmsg_open_array(&buf, "result"); + blobmsg_add_u32(&buf, "", ret); + c = blobmsg_open_table(&buf, NULL); + blob_for_each_attr(cur, du->buf.head, rem) + blobmsg_add_blob(&buf, cur); + blobmsg_close_table(&buf, c); + blobmsg_close_array(&buf, r); + uh_ubus_send_response(cl); + return; + } + + if (ret) { + void *c; + + uh_ubus_init_json_rpc_response(cl); + c = blobmsg_open_table(&buf, "error"); + blobmsg_add_u32(&buf, "code", ret); + blobmsg_add_string(&buf, "message", ubus_strerror(ret)); + blobmsg_close_table(&buf, c); + uh_ubus_send_response(cl); + } else { + uh_ubus_init_json_rpc_response(cl); + if (blob_len(du->buf.head)) { + r = blobmsg_open_table(&buf, "result"); + blob_for_each_attr(cur, du->buf.head, rem) + blobmsg_add_blob(&buf, cur); + blobmsg_close_table(&buf, r); + } else { + blobmsg_add_field(&buf, BLOBMSG_TYPE_UNSPEC, "result", NULL, 0); + } + uh_ubus_send_response(cl); + } + } static void @@ -282,7 +469,7 @@ static void uh_ubus_request_free(struct client *cl) static void uh_ubus_single_error(struct client *cl, enum rpc_error type) { - uh_ubus_send_header(cl); + uh_ubus_send_header(cl, 200, "OK", "application/json"); uh_ubus_json_rpc_error(cl, type); ops->request_done(cl); } @@ -335,7 +522,8 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o if (!obj->signature) return; - o = blobmsg_open_table(data->buf, obj->path); + if (data->add_object) + o = blobmsg_open_table(data->buf, obj->path); blob_for_each_attr(sig, obj->signature, rem) { t = blobmsg_open_table(data->buf, blobmsg_name(sig)); rem2 = blobmsg_data_len(sig); @@ -366,13 +554,14 @@ static void uh_ubus_list_cb(struct ubus_context *ctx, struct ubus_object_data *o } blobmsg_close_table(data->buf, t); } - blobmsg_close_table(data->buf, o); + if (data->add_object) + blobmsg_close_table(data->buf, o); } static void uh_ubus_send_list(struct client *cl, struct blob_attr *params) { struct blob_attr *cur, *dup; - struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false }; + struct list_data data = { .buf = &cl->dispatch.ubus.buf, .verbose = false, .add_object = true }; void *r; int rem; @@ -471,7 +660,7 @@ static void uh_ubus_init_batch(struct client *cl) struct dispatch_ubus *du = &cl->dispatch.ubus; du->array = true; - uh_ubus_send_header(cl); + uh_ubus_send_header(cl, 200, "OK", "application/json"); ops->chunk_printf(cl, "["); } @@ -594,7 +783,7 @@ static void uh_ubus_data_done(struct client *cl) switch (obj ? json_object_get_type(obj) : json_type_null) { case json_type_object: - uh_ubus_send_header(cl); + uh_ubus_send_header(cl, 200, "OK", "application/json"); return uh_ubus_handle_request_object(cl, obj); case json_type_array: uh_ubus_init_batch(cl); @@ -604,6 +793,96 @@ static void uh_ubus_data_done(struct client *cl) } } +static void uh_ubus_call(struct client *cl, const char *path, const char *sid) +{ + struct dispatch_ubus *du = &cl->dispatch.ubus; + struct json_object *obj = du->jsobj; + struct rpc_data data = {}; + enum rpc_error err = ERROR_PARSE; + static struct blob_buf req; + + uh_client_ref(cl); + + if (!obj || json_object_get_type(obj) != json_type_object) + goto error; + + uh_ubus_send_header(cl, 200, "OK", "application/json"); + + du->jsobj_cur = obj; + blob_buf_init(&req, 0); + if (!blobmsg_add_object(&req, obj)) + goto error; + + if (!parse_json_rpc(&data, req.head)) + goto error; + + du->func = data.method; + if (ubus_lookup_id(ctx, path, &du->obj)) { + err = ERROR_OBJECT; + goto error; + } + + if (!conf.ubus_noauth && !uh_ubus_allowed(sid, path, data.method)) { + err = ERROR_ACCESS; + goto error; + } + + uh_ubus_send_request(cl, sid, data.params); + goto out; + +error: + uh_ubus_json_rpc_error(cl, err); +out: + if (data.params) + free(data.params); + + uh_client_unref(cl); +} + +enum ubus_hdr { + HDR_AUTHORIZATION, + __HDR_UBUS_MAX +}; + +static void uh_ubus_handle_post(struct client *cl) +{ + static const struct blobmsg_policy hdr_policy[__HDR_UBUS_MAX] = { + [HDR_AUTHORIZATION] = { "authorization", BLOBMSG_TYPE_STRING }, + }; + struct dispatch_ubus *du = &cl->dispatch.ubus; + struct blob_attr *tb[__HDR_UBUS_MAX]; + const char *url = du->url; + const char *auth; + + if (!strcmp(url, conf.ubus_prefix)) { + du->legacy = true; + uh_ubus_data_done(cl); + return; + } + + blobmsg_parse(hdr_policy, __HDR_UBUS_MAX, tb, blob_data(cl->hdr.head), blob_len(cl->hdr.head)); + + auth = UH_UBUS_DEFAULT_SID; + if (tb[HDR_AUTHORIZATION]) { + const char *tmp = blobmsg_get_string(tb[HDR_AUTHORIZATION]); + + if (!strncasecmp(tmp, "Bearer ", 7)) + auth = tmp + 7; + } + + url += strlen(conf.ubus_prefix); + + if (!strncmp(url, "/call/", strlen("/call/"))) { + url += strlen("/call/"); + + uh_ubus_call(cl, url, auth); + } else { + ops->http_header(cl, 404, "Not Found"); + ustream_printf(cl->us, "\r\n"); + ops->request_done(cl); + } +} + static int uh_ubus_data_send(struct client *cl, const char *data, int len) { struct dispatch_ubus *du = &cl->dispatch.ubus; @@ -626,21 +905,28 @@ error: static void uh_ubus_handle_request(struct client *cl, char *url, struct path_info *pi) { struct dispatch *d = &cl->dispatch; + struct dispatch_ubus *du = &d->ubus; blob_buf_init(&buf, 0); + du->url = url; + du->legacy = false; + switch (cl->request.method) { + case UH_HTTP_MSG_GET: + uh_ubus_handle_get(cl); + break; case UH_HTTP_MSG_POST: d->data_send = uh_ubus_data_send; - d->data_done = uh_ubus_data_done; + d->data_done = uh_ubus_handle_post; d->close_fds = uh_ubus_close_fds; d->free = uh_ubus_request_free; - d->ubus.jstok = json_tokener_new(); + du->jstok = json_tokener_new(); break; case UH_HTTP_MSG_OPTIONS: - uh_ubus_send_header(cl); + uh_ubus_send_header(cl, 200, "OK", "application/json"); ops->request_done(cl); break; diff --git a/uhttpd.h b/uhttpd.h index f77718e..75dd747 100644 --- a/uhttpd.h +++ b/uhttpd.h @@ -82,6 +82,7 @@ struct config { int ubus_noauth; int ubus_cors; int cgi_prefix_len; + int events_retry; struct list_head cgi_alias; struct list_head lua_prefix; }; @@ -209,6 +210,7 @@ struct dispatch_ubus { struct json_tokener *jstok; struct json_object *jsobj; struct json_object *jsobj_cur; + const char *url; int post_len; uint32_t obj; @@ -218,6 +220,9 @@ struct dispatch_ubus { bool req_pending; bool array; int array_idx; + bool legacy; /* Got legacy request => use legacy reply */ + + struct ubus_subscriber sub; }; #endif