From patchwork Mon Nov 20 15:47:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Babic X-Patchwork-Id: 1866112 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20230601 header.b=OjJ9UKxb; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::13b; helo=mail-lf1-x13b.google.com; envelope-from=swupdate+bncbaabbgh75wvamgqelehifry@googlegroups.com; receiver=patchwork.ozlabs.org) Received: from mail-lf1-x13b.google.com (mail-lf1-x13b.google.com [IPv6:2a00:1450:4864:20::13b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SYsMx35T6z1ySX for ; Tue, 21 Nov 2023 02:47:41 +1100 (AEDT) Received: by mail-lf1-x13b.google.com with SMTP id 2adb3069b0e04-50aaa5d2a12sf1465856e87.3 for ; Mon, 20 Nov 2023 07:47:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700495257; cv=pass; d=google.com; s=arc-20160816; b=cDwlEg47hz7QkadA74MHq69Rge1PLPKD3A1TsPVfnSPloc4Ue0Cl9W+oh++F8nl60j p4+QNN5oZj1BwCgT8UKU3OpXdbd7P4BI8DuVC/YUzP6IrLLXHqkJ+o0QDDyWCxMLzR0k IBrQz85imiT+2qAMFUlQQIKDamDPDGBiiMZ5EjZTg14htA0dVLu/XhMqNe4cmZ5VdxWK cwuCpE/oQ7nTs0WizOfMzLAE5wt4LGjZHzUpRuQPBHUEMwjl6Sk73A9GrMBRnE19Cp+t cEqaXSdAz/L1jioTvtMu4+DELMh8ybj8OtyAzLLFaptD7vN2nvNiF+fev4DfvFy1879i ruHw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:ui-outboundreport:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=U+lopzLneah3LhprK9RRff+CdEyBNHnATSJP37Y9GY4=; fh=TiEWcqMcKpHk5s7uErzpntoONrNfOXwKpI5P8bIlggk=; b=PzV9RxuX17kxsLSV68ZtOIS/XAtOl0oIg4NFL18weYn721VEP/ylT08j8wy50MWwC5 TcL94hldE+3pcP0qfnppCU5zDMteBQuJPet+Pjv7o23ig2aDWzArA8ONEtM6vC2QRBXP /c+R+nZ6UJavO5N8FAIjwe/M5wUywjPSSCsiBZBF038mRMTrRsbxCwHTzU00nsiYF6hv k/vYGOEBGY1dm3TG7/chmlTTnUIXCEcnQhiEj11f/H8cZfu8gfGPegom077N1qe/2cWM XhO+e6pcozMrxbK5Ex2jA1i/rTXes6G7t/5m1T9ZOn0dooJF3uALNXwtQLuzIhsnQLSm +PJw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 212.227.17.13 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1700495257; x=1701100057; darn=patchwork.ozlabs.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:ui-outboundreport:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:sender:from:to:cc :subject:date:message-id:reply-to; bh=U+lopzLneah3LhprK9RRff+CdEyBNHnATSJP37Y9GY4=; b=OjJ9UKxbwJs7FI2HAf5gEOXm4NdgC6UdUU+Ucp3jmXyGn53ODpKJgwxCIKdLHBboFm YfVBGEg486cCO6KA2QroAbKnZaGtXp+phclSqhwestlykOGtlcH8+u4xCPXpKk/SdX+1 +hhTDi44vHET6OSQDanoM/2MQOqOcBVujiyjVYBt1tH9Dabk95VLpTD8uUgRcZiwMhed Z9PdypOWDeDuTAr3sCa+fMma8PjwaMKEZB9M2sHvpslQWCgYV+k6ezzW0jUBXtFhcuzZ rOYqBw+tTMMPRI5qjA+qewcH2pu33+fBpCFge7NwIbxHeTTVyZfeAbh/frc2uaTrksaM g46w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700495257; x=1701100057; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender :ui-outboundreport:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:x-beenthere:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=U+lopzLneah3LhprK9RRff+CdEyBNHnATSJP37Y9GY4=; b=IKGAehKPcjdSCIK86lAr5vY1JZzU6Yr8+QWKoT+yotEOSLON2XKgiigvTo2VgvfCxW N8wVnVDqqwf65FXalV71ZPo9ixCZsTq7W8Vs1gP8acj/p1S972the3VrJHU+ccKIe3ik CtrhmNRGI0Njb1uX7vQZSp7YXObcwH/cxdFG2UzyochSxB+egKlo1RqgjL180nwHk8+x 6Wrz2lQeigfae0oSnsre5JoJ56I+dv+rDWSZuu6kl1Gxwqbs+l0obPXR6cw2VfUV48mN zPBaYxxixsTnHRj0pudKmY9JUj4MeUrgvm710lEqR7zzXEuNiy4lfmkBl+bW4wJzD5dt OI1g== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOJu0YymlIH5MyufI+io4WqIWSQQmDd7fHymnKo+TPs1XOBonsoqKs5E dH5rRHKZd4sRPNgND/cRupY= X-Google-Smtp-Source: AGHT+IHhUHmF62ZSz6urbvWqbSBG1z3+Kxl3jabN8LDS69xfjGqmhl6uUdTyRKUb0AVljzFblGDBeA== X-Received: by 2002:ac2:4423:0:b0:50a:77c5:12b5 with SMTP id w3-20020ac24423000000b0050a77c512b5mr5209503lfl.23.1700495256850; Mon, 20 Nov 2023 07:47:36 -0800 (PST) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a05:6512:104c:b0:503:7ac:19f3 with SMTP id c12-20020a056512104c00b0050307ac19f3ls170625lfb.0.-pod-prod-07-eu; Mon, 20 Nov 2023 07:47:35 -0800 (PST) X-Received: by 2002:a19:f80c:0:b0:507:b911:f869 with SMTP id a12-20020a19f80c000000b00507b911f869mr5194873lff.43.1700495254922; Mon, 20 Nov 2023 07:47:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700495254; cv=none; d=google.com; s=arc-20160816; b=L1g3rRRvKVEn4Q/vPeX4hqWJRTjxJVLp5dJNvVVxnTzXPxCyvfIGLezfWVueJmv77X G/x7yyxnUD5leDpY4EltdJ+nJP69zjPuMdTO4vPd2gaCMrSFBEKZ0OlT0sbwOXWuFG2r EODYr+8pMaV/tnfEwFgj2OKdl62Zlf3YOoAO0e2CO8IvkUoskBkz528j1goEjJR3TKur Pbg9xXcRMHNw8wgpzCStUG5Dvg8DSxdFA6LNDTk5lmAoIluOgA8yYfufdotafjng55A1 9HgPbwW1iX6EzWWXK8fY3ZY6w0HsU+0PJ09q+eQ4Lv2RS0gE00eRrJi4e7ii9cdzOZCj xT+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=ui-outboundreport:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from; bh=InRETLdaQ/PX5ezYwJS65Da4eWdxNMsUtRPNDJQWeo0=; fh=TiEWcqMcKpHk5s7uErzpntoONrNfOXwKpI5P8bIlggk=; b=UJBQPgBSQ/M9CQrAZZUGMxy1evhFHQ74FJ3pi5GvABABi/aOk2yptuleI3JljLeDa8 2DM2VwyUOYCUA9LdwyOxp52EH5GZ4AJUS6q4fuNzUPk+OFQSTGWz4cM41ZS8r8s6ryT0 bWdPyRZUrLVHDdimiXGYvFSe7D7aZcBLgObfmLpwg5x4r8fUaxFeRK4kcqyfZBVNRyXh lgKGHSoYyMLChhzV1Sxz/5LbHfN2rJHMWEe6TimQ8lyMGEycimaG/ek13JbTfMOVwDNb Rpe7HGtgIK/GoMBnvW0iSNHtUJG4mWoIIEwn/X3kQSLU+oPMVtz1eYmUPJtQDzFJLVOR 2OCQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 212.227.17.13 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org Received: from mout.kundenserver.de (mout.kundenserver.de. [212.227.17.13]) by gmr-mx.google.com with ESMTPS id g4-20020a0565123b8400b0050aa77fac93si208149lfv.2.2023.11.20.07.47.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 07:47:34 -0800 (PST) Received-SPF: pass (google.com: domain of stefano.babic@swupdate.org designates 212.227.17.13 as permitted sender) client-ip=212.227.17.13; Received: from paperino.fritz.box ([88.217.136.221]) by mrelayeu.kundenserver.de (mreue109 [213.165.67.113]) with ESMTPSA (Nemesis) id 1Myb8N-1rGxuV0pfZ-00z1rt; Mon, 20 Nov 2023 16:47:34 +0100 From: Stefano Babic To: swupdate@googlegroups.com Cc: Stefano Babic Subject: [swupdate] [PATCH 05/13] channel_curl: add fifo to read callback Date: Mon, 20 Nov 2023 16:47:27 +0100 Message-Id: <20231120154731.44988-6-stefano.babic@swupdate.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231120154731.44988-1-stefano.babic@swupdate.org> References: <20231120154731.44988-1-stefano.babic@swupdate.org> MIME-Version: 1.0 X-Provags-ID: V03:K1:Zpw0cxqyDdWxbbAXo2Ftk3aY1JasDehwRiyOsxRsdmoozCgneW3 J0lBsxab/g26jCBCuZomSvAmI72/BF/R3nOum4AKBHLuiVeadbwCgQ17HxsNSuDVT6mWySq xv2mDL9rGLp4XxPDPXy+k7k8TwXMwX9Z0zyODdnG6BlAW0q1dguQKR2vDmvvgV2wIxEbMCg WtxbPBOV14Cc4ve6qleWg== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:eU93lC5LAq4=;DyR92DApLU7v+PwLnyKnBGl6H+j Zb+o+aMs2R6CgzEey/S8uuLUbbahDZ4aOYe9VYRDECaixKBEgY6Mun/txnibmii9zIeVbjQiM VueGcahPiKb0zYt3c0m2h2TeGqkvjBnMC5Y5HRhgJQxCakKV2d/+LodrfeK9JMgoLLtyAxeYK ZQwCTjYrrqDU/nqt9sMpm74/9+3LQLBsLSIeJVr4U8CHg2sfzib7sY0cnNpXmjg5WELvh3ekn nFHir4JTW7B1g/FEwWkU1hUJbTiQBFNG7veL0GdhiM3KQe37KRUSGZDKeawmlimavQ+V5xAAT /JyprAIVti8Z1CZ3ej7+NddDPHyntSPR67PcchJuLopXVhu1U7/ghcF+SMwP9NujtoxEk2VqT V6ufRll1vdLcBayEbuWbDJ9jTUB6v0c6YLkMtGo4By/aYFJvuWfVRDW2FQsyhqWPzSn99utbZ s3mfOzrjqY8HQ/wRih2cC+7RUp9+ZwGNvit1hoXoi4NL5wexgIYvUdKEl7ZpSjvDVkU6eo8k2 SS3zSs6oapn7VJGWBi+Y0WGncscx7sNCSRrJETaAoDaeIAavxMdFc+Y1dzqAT0OJe1hX8QSM2 rR0zF5wSMYpB6Pyhj2ATIPSkaAnKgGxeYeOb3IdWMfkAgvWSCqpXO0cSiSnNx4dRA4+AKHIoC TZn37k0QXhtcEZvvitlwLiKgeJ+bdsnhRLUf2oEneld5vlC3V9ey7jhg34rN4ArQTJqU86xAa 9eGvXj7dOxZbkULk9ohEotMbdWO1owrIFtvyppcfcr11dqllFDzF7UOD4UafS4XqxX/+qYpXU 6hWPhkFaUQFSr9I99fXhGMsfMR/2BYgHRjJM9vyyR/t/pNg59Y6auB+jQIw8NplqCMJYtRnDT R56Tni2BQiEZH4sB0nEiLItnLfJI9/Ah4QW0= X-Original-Sender: stefano.babic@swupdate.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of stefano.babic@swupdate.org designates 212.227.17.13 as permitted sender) smtp.mailfrom=stefano.babic@swupdate.org Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , In case of upload, the data is read from a buffer that must be allocated and filled by the caller (request_body field in channel_data_t). This is not suitable in case a large amount of data should be upload, for example in case a big image should be transferred. Let the callback to read from a fifo that must be created by the caller. The caller writes then into the FIFO, and the callback consumes the data by reading from the FIFO. Signed-off-by: Stefano Babic --- corelib/channel_curl.c | 58 ++++++++++++++++++++++++++++++++---------- include/channel_curl.h | 5 ++++ 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/corelib/channel_curl.c b/corelib/channel_curl.c index 2da49ce3..4a209bcd 100644 --- a/corelib/channel_curl.c +++ b/corelib/channel_curl.c @@ -891,19 +891,41 @@ cleanup: static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *data) { channel_data_t *channel_data = (channel_data_t *)data; - unsigned int bytes; - size_t n; + ssize_t bytes; + size_t n = 0; + int ret; - /* Check data to be sent */ - bytes = strlen(channel_data->request_body) - channel_data->offs; + /* + * Check if data is stored in a buffer or should be read + * form the input pipe + */ + if (channel_data->request_body) { + /* Check data to be sent */ + bytes = strlen(channel_data->request_body) - channel_data->offs; - if (!bytes) - return 0; + if (!bytes) + return 0; + + n = min(bytes, size * nmemb); - n = min(bytes, size * nmemb); + memcpy(ptr, &channel_data->request_body[channel_data->offs], n); + channel_data->offs += n; + } else { + bytes = nmemb * size; + ret = read(channel_data->read_fifo, ptr, bytes); + if (ret < 0) { + if (errno == EAGAIN) { + TRACE("READ EAGAIN"); + bytes = 0; + } else { + ERROR("Cannot read from FIFO"); + return CURL_READFUNC_ABORT; + } + } else + bytes = ret; - memcpy(ptr, &channel_data->request_body[channel_data->offs], n); - channel_data->offs += n; + n = bytes / nmemb; + } return n; } @@ -992,6 +1014,16 @@ static channel_op_res_t parse_reply(channel_data_t *channel_data, output_data_t return CHANNEL_OK; } +static CURLcode channel_set_read_callback(channel_curl_t *handle, channel_data_t *channel_data) +{ + + return curl_easy_setopt(handle, CURLOPT_READFUNCTION, read_callback) || + (channel_data->request_body ? + curl_easy_setopt(handle, CURLOPT_INFILESIZE_LARGE, + (curl_off_t)strlen(channel_data->request_body)) : 0) || + curl_easy_setopt(handle, CURLOPT_READDATA, channel_data); +} + static channel_op_res_t channel_post_method(channel_t *this, void *data, int method) { channel_curl_t *channel_curl = this->priv; @@ -1031,6 +1063,8 @@ static channel_op_res_t channel_post_method(channel_t *this, void *data, int met curl_result |= curl_easy_setopt(channel_curl->handle, CURLOPT_POSTFIELDS, channel_data->request_body); + if (channel_data->read_fifo) + curl_result |= channel_set_read_callback(channel_curl->handle, channel_data); break; case CHANNEL_PUT: @@ -1040,11 +1074,7 @@ static channel_op_res_t channel_post_method(channel_t *this, void *data, int met #else CURLOPT_PUT, #endif - 1L) || - curl_easy_setopt(channel_curl->handle, CURLOPT_READFUNCTION, read_callback) || - curl_easy_setopt(channel_curl->handle, CURLOPT_INFILESIZE_LARGE, - (curl_off_t)strlen(channel_data->request_body)) || - curl_easy_setopt(channel_curl->handle, CURLOPT_READDATA, channel_data); + 1L) || channel_set_read_callback(channel_curl->handle, channel_data); break; } diff --git a/include/channel_curl.h b/include/channel_curl.h index 1eefbb1c..bf7a508f 100644 --- a/include/channel_curl.h +++ b/include/channel_curl.h @@ -68,6 +68,11 @@ typedef struct { bool nofollow; size_t (*dwlwrdata)(char *streamdata, size_t size, size_t nmemb, void *data); + /* + * read_fifo is used as alternative to request_body buffer to push data + * in case of large data. This lets push a stream instead of a buffer. + */ + int read_fifo; size_t (*headers)(char *streamdata, size_t size, size_t nmemb, void *data); struct swupdate_digest *dgst;