| Message ID | 20220509215923.36255-2-stokito@gmail.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Petr Štetiar |
| Headers | show |
| Series | Make wget useful for API calls i.e. goodbye curl. | expand |
Hi Sergey, a minor nitpick inline below. On 5/9/22 11:59 PM, Sergey Ponomarev wrote: > You can add a custom HTTP header(s) to request: > > wget --header='Authorization: Bearer TOKEN' \ > --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ > https://example.com/ > > Some headers like Authorization or User-Agent may be already set by --password or --user-agent. > We may override them but it's a protection from user itself. > To keep code concise the logic omitted. > > Signed-off-by: Sergey Ponomarev <stokito@gmail.com> > --- > [...] > + case L_HEADER: > + if (!raw_headers) { > + /* Max possible count of headers is the count of args (argc) - 2 > + Since the first arg is program and last is a URL. > + But user may forget the URL and raw_headers is null terminated so allocate argc */ > + raw_headers = calloc(argc, sizeof(char *)); Please handle a possible calloc() error here. Maybe consider introducing an "xalloc()" or similar helper which wraps calloc() and invokes abort() on NULL return. > + } > + raw_headers[raw_headers_count] = optarg; > + raw_headers_count++; > + break; > case L_POST_DATA: > post_data = optarg; > break; > [...]
Hi Sergey, any update on this series? I'd be interested in the --header option. Thanks, Andre On 10/05/2022 11:11, Jo-Philipp Wich wrote: > Hi Sergey, > > a minor nitpick inline below. > > On 5/9/22 11:59 PM, Sergey Ponomarev wrote: >> You can add a custom HTTP header(s) to request: >> >> wget --header='Authorization: Bearer TOKEN' \ >> --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ >> https://example.com/ >> >> Some headers like Authorization or User-Agent may be already set by --password or --user-agent. >> We may override them but it's a protection from user itself. >> To keep code concise the logic omitted. >> >> Signed-off-by: Sergey Ponomarev <stokito@gmail.com> >> --- >> [...] >> + case L_HEADER: >> + if (!raw_headers) { >> + /* Max possible count of headers is the count of args (argc) - 2 >> + Since the first arg is program and last is a URL. >> + But user may forget the URL and raw_headers is null terminated so allocate argc */ >> + raw_headers = calloc(argc, sizeof(char *)); > > Please handle a possible calloc() error here. Maybe consider introducing an > "xalloc()" or similar helper which wraps calloc() and invokes abort() on NULL > return. > >> + } >> + raw_headers[raw_headers_count] = optarg; >> + raw_headers_count++; >> + break; >> case L_POST_DATA: >> post_data = optarg; >> break; >> [...] > > > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Hi Andre and Jo-Philipp, Please add the calloc() error handler yourself because I'm not experienced in C and I haven't time. But for you this may be just 5 minutes. The feature is very important from my point of view. The first version I sent a year(s?) ago but still no progress. If there are any problems please let me know On Wed, 22 Jun 2022 at 08:33, Andre Heider <a.heider@gmail.com> wrote: > > Hi Sergey, > > any update on this series? I'd be interested in the --header option. > > Thanks, > Andre > > On 10/05/2022 11:11, Jo-Philipp Wich wrote: > > Hi Sergey, > > > > a minor nitpick inline below. > > > > On 5/9/22 11:59 PM, Sergey Ponomarev wrote: > >> You can add a custom HTTP header(s) to request: > >> > >> wget --header='Authorization: Bearer TOKEN' \ > >> --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ > >> https://example.com/ > >> > >> Some headers like Authorization or User-Agent may be already set by --password or --user-agent. > >> We may override them but it's a protection from user itself. > >> To keep code concise the logic omitted. > >> > >> Signed-off-by: Sergey Ponomarev <stokito@gmail.com> > >> --- > >> [...] > >> + case L_HEADER: > >> + if (!raw_headers) { > >> + /* Max possible count of headers is the count of args (argc) - 2 > >> + Since the first arg is program and last is a URL. > >> + But user may forget the URL and raw_headers is null terminated so allocate argc */ > >> + raw_headers = calloc(argc, sizeof(char *)); > > > > Please handle a possible calloc() error here. Maybe consider introducing an > > "xalloc()" or similar helper which wraps calloc() and invokes abort() on NULL > > return. > > > >> + } > >> + raw_headers[raw_headers_count] = optarg; > >> + raw_headers_count++; > >> + break; > >> case L_POST_DATA: > >> post_data = optarg; > >> break; > >> [...] > > > > > > _______________________________________________ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel >
On 22/06/2022 12:26, Sergey Ponomarev wrote: > Hi Andre and Jo-Philipp, > > Please add the calloc() error handler yourself because I'm not > experienced in C and I haven't time. But for you this may be just 5 > minutes. Sure, I had a look, and while adding that wrapper isn't a problem, the patchset itself has an issue: --header 'Content-type: foo/bar' doesn't work as intended because 'Content-type: application/x-www-form-urlencoded' is still added too. So both headers are sent, which breaks my use case. I may be looking at that, but I'm time constrained myself too at the moment. Cheers, Andre > The feature is very important from my point of view. The first version > I sent a year(s?) ago but still no progress. > If there are any problems please let me know > > On Wed, 22 Jun 2022 at 08:33, Andre Heider <a.heider@gmail.com> wrote: >> >> Hi Sergey, >> >> any update on this series? I'd be interested in the --header option. >> >> Thanks, >> Andre >> >> On 10/05/2022 11:11, Jo-Philipp Wich wrote: >>> Hi Sergey, >>> >>> a minor nitpick inline below. >>> >>> On 5/9/22 11:59 PM, Sergey Ponomarev wrote: >>>> You can add a custom HTTP header(s) to request: >>>> >>>> wget --header='Authorization: Bearer TOKEN' \ >>>> --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ >>>> https://example.com/ >>>> >>>> Some headers like Authorization or User-Agent may be already set by --password or --user-agent. >>>> We may override them but it's a protection from user itself. >>>> To keep code concise the logic omitted. >>>> >>>> Signed-off-by: Sergey Ponomarev <stokito@gmail.com> >>>> --- >>>> [...] >>>> + case L_HEADER: >>>> + if (!raw_headers) { >>>> + /* Max possible count of headers is the count of args (argc) - 2 >>>> + Since the first arg is program and last is a URL. >>>> + But user may forget the URL and raw_headers is null terminated so allocate argc */ >>>> + raw_headers = calloc(argc, sizeof(char *)); >>> >>> Please handle a possible calloc() error here. Maybe consider introducing an >>> "xalloc()" or similar helper which wraps calloc() and invokes abort() on NULL >>> return. >>> >>>> + } >>>> + raw_headers[raw_headers_count] = optarg; >>>> + raw_headers_count++; >>>> + break; >>>> case L_POST_DATA: >>>> post_data = optarg; >>>> break; >>>> [...] >>> >>> >>> _______________________________________________ >>> openwrt-devel mailing list >>> openwrt-devel@lists.openwrt.org >>> https://lists.openwrt.org/mailman/listinfo/openwrt-devel >> > >
Andre, --header 'Content-type: foo/bar' doesn't work as intended because 'Content-type: application/x-www-form-urlencoded' is still added too. So both headers are sent, which breaks my use case. Yes, you are right and I described the behaviour in a comment. The headers overriding doesn't work in this version: it makes code slightly complicated and slower. But you can achieve your goal by doing --method=POST --post-data="OLOLO" --header 'Content-type: foo/bar' On Thu, 23 Jun 2022 at 09:20, Andre Heider <a.heider@gmail.com> wrote: > > On 22/06/2022 12:26, Sergey Ponomarev wrote: > > Hi Andre and Jo-Philipp, > > > > Please add the calloc() error handler yourself because I'm not > > experienced in C and I haven't time. But for you this may be just 5 > > minutes. > > Sure, I had a look, and while adding that wrapper isn't a problem, the > patchset itself has an issue: > > --header 'Content-type: foo/bar' doesn't work as intended because > 'Content-type: application/x-www-form-urlencoded' is still added too. So > both headers are sent, which breaks my use case. > > I may be looking at that, but I'm time constrained myself too at the moment. > > Cheers, > Andre > > > The feature is very important from my point of view. The first version > > I sent a year(s?) ago but still no progress. > > If there are any problems please let me know > > > > On Wed, 22 Jun 2022 at 08:33, Andre Heider <a.heider@gmail.com> wrote: > >> > >> Hi Sergey, > >> > >> any update on this series? I'd be interested in the --header option. > >> > >> Thanks, > >> Andre > >> > >> On 10/05/2022 11:11, Jo-Philipp Wich wrote: > >>> Hi Sergey, > >>> > >>> a minor nitpick inline below. > >>> > >>> On 5/9/22 11:59 PM, Sergey Ponomarev wrote: > >>>> You can add a custom HTTP header(s) to request: > >>>> > >>>> wget --header='Authorization: Bearer TOKEN' \ > >>>> --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ > >>>> https://example.com/ > >>>> > >>>> Some headers like Authorization or User-Agent may be already set by --password or --user-agent. > >>>> We may override them but it's a protection from user itself. > >>>> To keep code concise the logic omitted. > >>>> > >>>> Signed-off-by: Sergey Ponomarev <stokito@gmail.com> > >>>> --- > >>>> [...] > >>>> + case L_HEADER: > >>>> + if (!raw_headers) { > >>>> + /* Max possible count of headers is the count of args (argc) - 2 > >>>> + Since the first arg is program and last is a URL. > >>>> + But user may forget the URL and raw_headers is null terminated so allocate argc */ > >>>> + raw_headers = calloc(argc, sizeof(char *)); > >>> > >>> Please handle a possible calloc() error here. Maybe consider introducing an > >>> "xalloc()" or similar helper which wraps calloc() and invokes abort() on NULL > >>> return. > >>> > >>>> + } > >>>> + raw_headers[raw_headers_count] = optarg; > >>>> + raw_headers_count++; > >>>> + break; > >>>> case L_POST_DATA: > >>>> post_data = optarg; > >>>> break; > >>>> [...] > >>> > >>> > >>> _______________________________________________ > >>> openwrt-devel mailing list > >>> openwrt-devel@lists.openwrt.org > >>> https://lists.openwrt.org/mailman/listinfo/openwrt-devel > >> > > > > >
diff --git a/tests/cram/test-san_uclient-fetch.t b/tests/cram/test-san_uclient-fetch.t index 435659b..047749a 100644 --- a/tests/cram/test-san_uclient-fetch.t +++ b/tests/cram/test-san_uclient-fetch.t @@ -15,6 +15,7 @@ check uclient-fetch usage: \t--user=<user>\t\t\tHTTP authentication username (esc) \t--password=<password>\t\tHTTP authentication password (esc) \t--user-agent | -U <str>\t\tSet HTTP user agent (esc) + \t--header='Header: value'\t\tAdd HTTP header. Multiple allowed (esc) \t--post-data=STRING\t\tuse the POST method; send STRING as the data (esc) \t--post-file=FILE\t\tuse the POST method; send FILE as the data (esc) \t--spider | -s\t\t\tSpider mode - only check file existence (esc) diff --git a/tests/cram/test_uclient-fetch.t b/tests/cram/test_uclient-fetch.t index e22aa40..cb70271 100644 --- a/tests/cram/test_uclient-fetch.t +++ b/tests/cram/test_uclient-fetch.t @@ -15,6 +15,7 @@ check uclient-fetch usage: \t--user=<user>\t\t\tHTTP authentication username (esc) \t--password=<password>\t\tHTTP authentication password (esc) \t--user-agent | -U <str>\t\tSet HTTP user agent (esc) + \t--header='Header: value'\t\tAdd HTTP header. Multiple allowed (esc) \t--post-data=STRING\t\tuse the POST method; send STRING as the data (esc) \t--post-file=FILE\t\tuse the POST method; send FILE as the data (esc) \t--spider | -s\t\t\tSpider mode - only check file existence (esc) diff --git a/uclient-fetch.c b/uclient-fetch.c index 282092e..4efc917 100644 --- a/uclient-fetch.c +++ b/uclient-fetch.c @@ -44,6 +44,7 @@ static const char *user_agent = "uclient-fetch"; static const char *post_data; static const char *post_file; +static const char **raw_headers = NULL; static struct ustream_ssl_ctx *ssl_ctx; static const struct ustream_ssl_ops *ssl_ops; static int quiet = false; @@ -342,6 +343,7 @@ static int init_request(struct uclient *cl) uclient_http_reset_headers(cl); uclient_http_set_header(cl, "User-Agent", user_agent); + uclient_http_set_raw_headers(cl, raw_headers); if (cur_resume) check_resume_offset(cl); @@ -481,6 +483,7 @@ static int usage(const char *progname) " --continue | -c Continue a partially-downloaded file\n" " --user=<user> HTTP authentication username\n" " --password=<password> HTTP authentication password\n" + " --header='Header: value' Add HTTP header. Multiple allowed\n" " --user-agent | -U <str> Set HTTP user agent\n" " --post-data=STRING use the POST method; send STRING as the data\n" " --post-file=FILE use the POST method; send FILE as the data\n" @@ -542,6 +545,7 @@ enum { L_USER, L_PASSWORD, L_USER_AGENT, + L_HEADER, L_POST_DATA, L_POST_FILE, L_SPIDER, @@ -559,6 +563,7 @@ static const struct option longopts[] = { [L_USER] = { "user", required_argument, NULL, 0 }, [L_PASSWORD] = { "password", required_argument, NULL, 0 }, [L_USER_AGENT] = { "user-agent", required_argument, NULL, 0 }, + [L_HEADER] = { "header", required_argument, NULL, 0 }, [L_POST_DATA] = { "post-data", required_argument, NULL, 0 }, [L_POST_FILE] = { "post-file", required_argument, NULL, 0 }, [L_SPIDER] = { "spider", no_argument, NULL, 0 }, @@ -578,6 +583,7 @@ int main(int argc, char **argv) const char *proxy_url; char *username = NULL; char *password = NULL; + int raw_headers_count = 0; struct uclient *cl; int longopt_idx = 0; bool has_cert = false; @@ -626,6 +632,16 @@ int main(int argc, char **argv) case L_USER_AGENT: user_agent = optarg; break; + case L_HEADER: + if (!raw_headers) { + /* Max possible count of headers is the count of args (argc) - 2 + Since the first arg is program and last is a URL. + But user may forget the URL and raw_headers is null terminated so allocate argc */ + raw_headers = calloc(argc, sizeof(char *)); + } + raw_headers[raw_headers_count] = optarg; + raw_headers_count++; + break; case L_POST_DATA: post_data = optarg; break; diff --git a/uclient-http.c b/uclient-http.c index c2bba6b..3d59de8 100644 --- a/uclient-http.c +++ b/uclient-http.c @@ -96,6 +96,7 @@ struct uclient_http { uint32_t nc; + const char **raw_headers; struct blob_buf headers; struct blob_buf meta; }; @@ -587,6 +588,20 @@ uclient_http_add_auth_header(struct uclient_http *uh) return 0; } +static void +uclient_http_send_raw_headers(const struct uclient_http *uh) { + if (!uh->raw_headers) { + return; + } + const char **raw_headers = uh->raw_headers; + const char *raw_header = *raw_headers; + while (raw_header != NULL) { + ustream_printf(uh->us, "%s\r\n", raw_header); + raw_headers++; + raw_header = *raw_headers; + } +} + static int uclient_http_send_headers(struct uclient_http *uh) { @@ -625,6 +640,7 @@ uclient_http_send_headers(struct uclient_http *uh) if (err) return err; + uclient_http_send_raw_headers(uh); ustream_printf(uh->us, "\r\n"); uh->state = HTTP_STATE_HEADERS_SENT; @@ -1026,6 +1042,21 @@ uclient_http_set_header(struct uclient *cl, const char *name, const char *value) return 0; } +int +uclient_http_set_raw_headers(struct uclient *cl, const char **raw_headers) +{ + struct uclient_http *uh = container_of(cl, struct uclient_http, uc); + + if (cl->backend != &uclient_backend_http) + return -1; + + if (uh->state > HTTP_STATE_INIT) + return -1; + + uh->raw_headers = raw_headers; + return 0; +} + static int uclient_http_send_data(struct uclient *cl, const char *buf, unsigned int len) { diff --git a/uclient.h b/uclient.h index 4f37364..f1977bc 100644 --- a/uclient.h +++ b/uclient.h @@ -121,6 +121,7 @@ extern const struct uclient_backend uclient_backend_http; int uclient_http_reset_headers(struct uclient *cl); int uclient_http_set_header(struct uclient *cl, const char *name, const char *value); +int uclient_http_set_raw_headers(struct uclient *cl, const char **raw_headers); int uclient_http_set_request_type(struct uclient *cl, const char *type); int uclient_http_redirect(struct uclient *cl);
You can add a custom HTTP header(s) to request: wget --header='Authorization: Bearer TOKEN' \ --header='If-Modified-Since: Wed, 9 May 2021 12:16:00 GMT' \ https://example.com/ Some headers like Authorization or User-Agent may be already set by --password or --user-agent. We may override them but it's a protection from user itself. To keep code concise the logic omitted. Signed-off-by: Sergey Ponomarev <stokito@gmail.com> --- tests/cram/test-san_uclient-fetch.t | 1 + tests/cram/test_uclient-fetch.t | 1 + uclient-fetch.c | 16 +++++++++++++++ uclient-http.c | 31 +++++++++++++++++++++++++++++ uclient.h | 1 + 5 files changed, 50 insertions(+)