From patchwork Mon Nov 20 15:54:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Babic X-Patchwork-Id: 1866121 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20230601 header.b=wDdqB3ZN; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::33b; helo=mail-wm1-x33b.google.com; envelope-from=swupdate+bncbaabbwec52vamgqec43cghy@googlegroups.com; receiver=patchwork.ozlabs.org) Received: from mail-wm1-x33b.google.com (mail-wm1-x33b.google.com [IPv6:2a00:1450:4864:20::33b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SYsXY4z6Jz1yST for ; Tue, 21 Nov 2023 02:55:08 +1100 (AEDT) Received: by mail-wm1-x33b.google.com with SMTP id 5b1f17b1804b1-408374a3d6bsf12893335e9.0 for ; Mon, 20 Nov 2023 07:55:08 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700495705; cv=pass; d=google.com; s=arc-20160816; b=dZ3vHA0XtNS5EILuNtzua/vYiNB9vJB0a7L3RoXDbYhgj7oplRoEcny/I0KQeoHqV1 wWdxU+dnvLANLeLTRzwn4lZPw7ReNlenPewkB4gljvI9LkGFQ5vjzPCBMlz9HSDyou/w p7XcOhDWIWD5uThH6sn4eSpVBUqNwsyssUUoIjaxYJgIGmbQgyW5NmFGf+ExOXcu4auy lPVkEtvYYCw1Ye4aXd/o0mQ1ld1c4seOe6jWC8+T+UI3nIIv3YjvYPKPeanMMs1ax+EG tqXDf29D8gPKSy0Vw0qosN/WZpDIOCrNsh18+5yEhlS10vGIgE7DN7023+25/13vC7AJ 5eTA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:ui-outboundreport:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=X81UBqQGJSg7WsQXGjvO4rKw4mRl/BSWyG8YS3MwOUc=; fh=TiEWcqMcKpHk5s7uErzpntoONrNfOXwKpI5P8bIlggk=; b=BTB04kw/rrlkT37UZdIox1fAhdzgm2a1/7rwTNcBdHQIArbFLXlP1IydzMGOsC5Zm2 pknm/yDPFp5s+l/+hf7mdc1dU63BVm8xtdmagnEdPJj7canSUhhB54exN3jAwU/SOGst 82WREK6umbXONju+gvlYT0XoefGl2K3B2VR6mY3HAKOgDQh5+7bcNzTQc/9UHI6BNLTU u49DBZUatRk/t2y8i/1pke8JA+FPOPKVD1lMxaykDTmAVaO4XuJ+/VMoAt50wCufew0H nqwzk/HY3GxLt4F7IDnAgpq2EPjGJ3//M2jaXjKHD8WnOqi9xJEz0hTzH1TzSUaA1gr3 mBZw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 217.72.192.73 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1700495705; x=1701100505; darn=patchwork.ozlabs.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:ui-outboundreport:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:sender:from:to:cc :subject:date:message-id:reply-to; bh=X81UBqQGJSg7WsQXGjvO4rKw4mRl/BSWyG8YS3MwOUc=; b=wDdqB3ZNIGqaO43eSbH9vmu9E4pR+zkGCk+fxbwJhy2Yc448P1HYIn8aVft/Smj7l6 K+1Tcc1N98/v4OSIMT4aMlP8XxhKBB0hxSBDXmq2rXMQccSLkn2gSwVEw63iM77japfz Y43XN+IZ7oWoBZcyfyDViJsw69wid5Mer5H13LpQ3EXOkzt5cpDuCGp5Y4ojdFLB8Rur WBNcHO87yRXj4sWAK03BCJmY2DGuzt1bkWGkNKxZAkka+c70i4ZDoGH7pJv4yPXxmSGM mcn+x8aAyFCEYmjvooOpENEHAxvYixpw40eegcV4aKCO9zAvsJq8j1vAfC1Sd/y3DKKf 2ASQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700495705; x=1701100505; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender :ui-outboundreport:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:x-beenthere:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=X81UBqQGJSg7WsQXGjvO4rKw4mRl/BSWyG8YS3MwOUc=; b=NB5IwwVDD9YoNN7ZdFQI+FC4ZThPlivjj1YsGhdpKC+smxQ5sgV4QoAkjaaBrf0V3J vyb1f0Dc6IjQv1s37XNTTU2vEhTru1ZuVEHcKlvODsScy7fzk96yb5AineFMWnLkt20A Gopu6kgGwuFDh8HSvIP5xwilDro8MpPwyUgJys2bmo5q1rXexaljiaB0rb2EuYljBEfZ zpkrejgR1XYEIqNRm35YZe4UYco1EBfhlwPErUozpJ7WRH6NAMqpHRXdAH+MYoUw8qkT Bxifv8TFKHOn692SwFi5t3nHrpZBIYcWnhWxfLWbUNwCRrZuHGh2ou9tU+KWaPLaqKCw OH/g== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOJu0YzAe5XkQULrAxnIreoHIe7BFkh4ZAd10B4y8QIlllXCEBj4dyRo ypctbTDWE0O4eXfA/jrf81U= X-Google-Smtp-Source: AGHT+IELdxTpQmQNISwLp6z4CGxUDpXRmbRjzX61V7HYPyYbep46nC4gfHdkypu2WbEwAwfpFU3JbA== X-Received: by 2002:a05:600c:1c0a:b0:408:59d4:f3c4 with SMTP id j10-20020a05600c1c0a00b0040859d4f3c4mr7403895wms.10.1700495704687; Mon, 20 Nov 2023 07:55:04 -0800 (PST) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a05:600c:3147:b0:40a:7784:bcd1 with SMTP id h7-20020a05600c314700b0040a7784bcd1ls202595wmo.0.-pod-prod-05-eu; Mon, 20 Nov 2023 07:55:03 -0800 (PST) X-Received: by 2002:adf:cd91:0:b0:32d:a495:a9b7 with SMTP id q17-20020adfcd91000000b0032da495a9b7mr4207282wrj.61.1700495703114; Mon, 20 Nov 2023 07:55:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700495703; cv=none; d=google.com; s=arc-20160816; b=qD/9UkaQL+qlrCQ1BJjE05jPESRdl6VDZJ8B6Gtkpiv+Fq8nl+TfKKjPaMO0sb6VBE 90MbTLQ7TYNoUmBk0OxW0HOuXSebAsDsvR0S2F0LJ9LioPQzgbfqyHBQtS/pDvGqgFjS vQFZWdNOjEE+P7eftU5sP1ENMDXbqISlml0ycZgwyoF1tkzzvXNJrats9EX/ae0pzrlJ svl5RtQ7a4wCkUxOXOSVv7iLyxl6KvONewzVuMH2s/kFu7nB+R61FJbgLJsHSlo9t9IJ SsxQGx9ZGg3NDNQgesVOi+nwdxzB1W8CkktBCzCS/qusrAxlAVJi/6NDYyyYYyC3/kaB qt3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=ui-outboundreport:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from; bh=InRETLdaQ/PX5ezYwJS65Da4eWdxNMsUtRPNDJQWeo0=; fh=TiEWcqMcKpHk5s7uErzpntoONrNfOXwKpI5P8bIlggk=; b=NW7AWJWOC7xzvAFuDN88C/fHlA/r2qW11ogJbwomzWGyGsSMIOT6IM5l/xNp9NaXtn ZCd2ymWaYEBZbQ8o4jN40fSNgAmw6TsSmNO9q1ufeibMMGBy6KM6YM8uGkoQKekk+uq0 H7PTZkZ1AihfuAxBF/vQpNMvX8/G2CIz3E1DTclYaeBhERMZoMBRLwBXKuzri+uWwwpr 2BZaEPMYYaL7GGM0whkFEE0J+0fnb2B4MvrX2OoFFbfqEW8M+rBBOEFUzsXYqQM9P4KI FwBMG1Zx8Ng5ssIu1uw6ibguuva94LPoN0CiFg8yfm47Y2RB2fDDlor4Vli9kSRlF/nI xu7w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 217.72.192.73 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org Received: from mout.kundenserver.de (mout.kundenserver.de. [217.72.192.73]) by gmr-mx.google.com with ESMTPS id p2-20020a5d68c2000000b00332cc5c485csi16799wrw.3.2023.11.20.07.55.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 07:55:03 -0800 (PST) Received-SPF: pass (google.com: domain of stefano.babic@swupdate.org designates 217.72.192.73 as permitted sender) client-ip=217.72.192.73; Received: from paperino.fritz.box ([88.217.136.221]) by mrelayeu.kundenserver.de (mreue109 [213.165.67.113]) with ESMTPSA (Nemesis) id 1MUokB-1qw8Jj2ZKD-00QkYL; Mon, 20 Nov 2023 16:55:02 +0100 From: Stefano Babic To: swupdate@googlegroups.com Cc: Stefano Babic Subject: [swupdate] [PATCH 05/13] channel_curl: add fifo to read callback Date: Mon, 20 Nov 2023 16:54:51 +0100 Message-Id: <20231120155459.45188-6-stefano.babic@swupdate.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231120155459.45188-1-stefano.babic@swupdate.org> References: <20231120155459.45188-1-stefano.babic@swupdate.org> MIME-Version: 1.0 X-Provags-ID: V03:K1:NARbalNmrk8JjgccRQ8/6luMhmVZeSQXteDr+tYWkn0SMDl3geV eAuw4/jJy092Jii1bSiVKo2qIdlOcuhu5A5s1Bvn30eodyZvLkGNAjC8N1OxcuLxu8q2BLe M68pZF7sKWc6igj0K3UI4BA7AEWT0CSTMSWrXTaksOnvHSIKNLoD/NSFYiQ0ZplfFcKqa43 qx8uL7B6SLYpp6aol1ivQ== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:sTVYXWLUzSQ=;Zk7slkQvZ7LWrrJJngkFGgRJvEg k+1dmfhdryrVsGLSNYpm8IUG/5aJsy0Qiu1sMLdgVJrfFHWz1GIcYXGqlCEMlGGd8jU86fmhR KvjxQT4FMyWMqWXeWasYbNZdxtaDvwQqttS4xnqkOZoZb7CuU4FYpwX/5OQ8Sj1EpWvzJd2MD c1JJWnToV3XatSvQraHLgYa5BE+ExmVkyZdDV31J3W0aPPrsV8b1BWj+KclJiTJQecwBYbOFy Kt49l907ngoKFiOS5HEhlLxh0dqOZOjs6F4o4mp1OXXbVKIyB1VTZKZyOUd2GVt8g8J23p/0v zaOU755iuqsSRoL7/CIl35MILPx/erlIclF2bASXxZEiLMpjsyLj1OMOLc2moCz6E5CzWtTVe Wdl280UI/r0nnT/Ld/EytKr5/9I5VM9wZqk99jdLlMmW0Wnvij76YXDxSJax+rtzYExCaTBtq z14eOA0bQVcn3jB4V+KUtSl4jRfnZy1AVIq01+me0V9vWy2r3QiI1q2ld4Ka6P9TCzeJBMqR3 9HP39oe01r64/3iEUrAtb4C/bdyQcqvBdvjCoei4yBY+8P0quSaxPrqgsLpw0DRZ+cFYNP5La D2mOJQ00ruDVq+vXGBQ6kBga9xYNdUfk1nUZ5UG+0Jh8AaKWZv70XWSlZaWjb6RdGHqfk2ntD /DJyF7KnSsASfge6TCyxkhcVPKi0WeEcG0x3xfM8+iWRQGb9tYq18nyq7OFz8P1iYxR91xOPp ftWQZp9ITsQSRwXIgGvxtjDNl/aHY0r7ooviPKsThoyba+1y0fhm8ZFs68HTZZ5aqn0QxZ7gl ad14dMCPKkfPXv+TsvY5Dila+rePUqaFyw+lO2Z+ik0+zvA0Go4+rVf4FnXEhGQntZUwt1DPh Xh9/5weBkqzl9q3+nD2zrB+ptEex1qMCipgo= X-Original-Sender: stefano.babic@swupdate.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 217.72.192.73 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , In case of upload, the data is read from a buffer that must be allocated and filled by the caller (request_body field in channel_data_t). This is not suitable in case a large amount of data should be upload, for example in case a big image should be transferred. Let the callback to read from a fifo that must be created by the caller. The caller writes then into the FIFO, and the callback consumes the data by reading from the FIFO. Signed-off-by: Stefano Babic --- corelib/channel_curl.c | 58 ++++++++++++++++++++++++++++++++---------- include/channel_curl.h | 5 ++++ 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/corelib/channel_curl.c b/corelib/channel_curl.c index 2da49ce3..4a209bcd 100644 --- a/corelib/channel_curl.c +++ b/corelib/channel_curl.c @@ -891,19 +891,41 @@ cleanup: static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *data) { channel_data_t *channel_data = (channel_data_t *)data; - unsigned int bytes; - size_t n; + ssize_t bytes; + size_t n = 0; + int ret; - /* Check data to be sent */ - bytes = strlen(channel_data->request_body) - channel_data->offs; + /* + * Check if data is stored in a buffer or should be read + * form the input pipe + */ + if (channel_data->request_body) { + /* Check data to be sent */ + bytes = strlen(channel_data->request_body) - channel_data->offs; - if (!bytes) - return 0; + if (!bytes) + return 0; + + n = min(bytes, size * nmemb); - n = min(bytes, size * nmemb); + memcpy(ptr, &channel_data->request_body[channel_data->offs], n); + channel_data->offs += n; + } else { + bytes = nmemb * size; + ret = read(channel_data->read_fifo, ptr, bytes); + if (ret < 0) { + if (errno == EAGAIN) { + TRACE("READ EAGAIN"); + bytes = 0; + } else { + ERROR("Cannot read from FIFO"); + return CURL_READFUNC_ABORT; + } + } else + bytes = ret; - memcpy(ptr, &channel_data->request_body[channel_data->offs], n); - channel_data->offs += n; + n = bytes / nmemb; + } return n; } @@ -992,6 +1014,16 @@ static channel_op_res_t parse_reply(channel_data_t *channel_data, output_data_t return CHANNEL_OK; } +static CURLcode channel_set_read_callback(channel_curl_t *handle, channel_data_t *channel_data) +{ + + return curl_easy_setopt(handle, CURLOPT_READFUNCTION, read_callback) || + (channel_data->request_body ? + curl_easy_setopt(handle, CURLOPT_INFILESIZE_LARGE, + (curl_off_t)strlen(channel_data->request_body)) : 0) || + curl_easy_setopt(handle, CURLOPT_READDATA, channel_data); +} + static channel_op_res_t channel_post_method(channel_t *this, void *data, int method) { channel_curl_t *channel_curl = this->priv; @@ -1031,6 +1063,8 @@ static channel_op_res_t channel_post_method(channel_t *this, void *data, int met curl_result |= curl_easy_setopt(channel_curl->handle, CURLOPT_POSTFIELDS, channel_data->request_body); + if (channel_data->read_fifo) + curl_result |= channel_set_read_callback(channel_curl->handle, channel_data); break; case CHANNEL_PUT: @@ -1040,11 +1074,7 @@ static channel_op_res_t channel_post_method(channel_t *this, void *data, int met #else CURLOPT_PUT, #endif - 1L) || - curl_easy_setopt(channel_curl->handle, CURLOPT_READFUNCTION, read_callback) || - curl_easy_setopt(channel_curl->handle, CURLOPT_INFILESIZE_LARGE, - (curl_off_t)strlen(channel_data->request_body)) || - curl_easy_setopt(channel_curl->handle, CURLOPT_READDATA, channel_data); + 1L) || channel_set_read_callback(channel_curl->handle, channel_data); break; } diff --git a/include/channel_curl.h b/include/channel_curl.h index 1eefbb1c..bf7a508f 100644 --- a/include/channel_curl.h +++ b/include/channel_curl.h @@ -68,6 +68,11 @@ typedef struct { bool nofollow; size_t (*dwlwrdata)(char *streamdata, size_t size, size_t nmemb, void *data); + /* + * read_fifo is used as alternative to request_body buffer to push data + * in case of large data. This lets push a stream instead of a buffer. + */ + int read_fifo; size_t (*headers)(char *streamdata, size_t size, size_t nmemb, void *data); struct swupdate_digest *dgst;