From patchwork Mon Dec 14 11:59:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sava Jakovljev X-Patchwork-Id: 1415922 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::139; helo=mail-lf1-x139.google.com; envelope-from=swupdate+bncbaabbv5v3x7akgqerjtxnoi@googlegroups.com; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=teufel.de Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20161025 header.b=sa5GfkOy; dkim-atps=neutral Received: from mail-lf1-x139.google.com (mail-lf1-x139.google.com [IPv6:2a00:1450:4864:20::139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cvgjb0LfKz9sSf for ; Mon, 14 Dec 2020 23:31:12 +1100 (AEDT) Received: by mail-lf1-x139.google.com with SMTP id i23sf7350198lfl.10 for ; Mon, 14 Dec 2020 04:31:12 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1607949067; cv=pass; d=google.com; s=arc-20160816; b=oSM65Ak8fzc70FeIYqDHUJqAV24REpEUS35GNn6EcdKPVnvVUygAIGqUMG/yrHYYFx VnUv6+8Rs8wtyCRphw9tJnGcM9YWfi4QrASQT/O/lG67x4NDTU3pZQkU9MpeNL3SiDqW 1ginWnrYaARQ0URKWJo3X/wSM43DJDcr4Xzf7tWKZTISTXxTmtuI3686DdYetNcCkpoy y/gQflh089nkNGnhCup9hMckmgpredXi90NJHGrkLUOtrZf8pS4HqPsIjaf9wz3d0lJf kOxO3vcnRshWYgJXjRrS8wufkcapX5ajRGuG8VqSs7xz7cAV81Juz68UkaEkG7DkmAD9 jSxg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=pov5tz00tzFr9MpIW/tmkLvrayAQgmw3dDv30Jzu24A=; b=GQx6AMXqwhK+iHckWqEN3LJtxCRxMuoDFc2oDy80tznSKip/1HPDkUr98skQXeZVEq Gu/xc2d8akf5EZ/tMLr6S8p88OuASBjgEnmiP4wqYfg2RuNzruzihtxHaoHnZdOqutuT eB8qfFmBxS/cGnt9KXaVjyujXpiPuW4SXW93xGqcFo60QrvG58rq2HFr69AUmyMlAlre ZsrCQVGGXxNP2NvyBDuFgbBqjuAVyPTh4xG5Qne9lP5nsQH17rQuIl+bz8X33Cu70vSf P5GsLeFIg53ZSsXywlewZvXXUQfCgvMymrdBADY67xjT4HnwVwNKK3IojHeufLoZWH8F NvfQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@teufel.de header.s=hse1 header.b=Ao01eKRU; spf=pass (google.com: domain of sava.jakovljev@teufel.de designates 94.100.136.165 as permitted sender) smtp.mailfrom=sava.jakovljev@teufel.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=pov5tz00tzFr9MpIW/tmkLvrayAQgmw3dDv30Jzu24A=; b=sa5GfkOy/j7UyhoLLvhppyD33BawITEaxMh0Fc3Uwc6wp1ujZd35M3NUE+dClfFpFa C5oTn75wAVZvO8+qwIu6j3TaNYhUAf2+x5hsq/Rm3+ru0n5Z+AXpQaLqBSNo5+PMpcVI AeoLZ22cEMBWjrgU/RqBBIlBvesrpQOhOaJqHB6DdMmlspF5M/IZMPyBSFVVflYLpRWr EId768ocftYg72wv3YuaLc+LAGOrQyb8NXaYHKQHCXa8TwfCiCpAWqU/yWfXHZ0GaIBn 661C/K98Vj3GgLSHqggmn6WSh7qkNn1s+OJ8h44t1Hq1Gx2MgwsRphrPMhPcWPjYXnEf BL2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=pov5tz00tzFr9MpIW/tmkLvrayAQgmw3dDv30Jzu24A=; b=VfBUWyRxnzuWEGW60F8Sa3F24ijf1Sj0QCKyJOyB+m5J3pRgDeTBeBGv/qf8tAktS4 eqqrvvYT0kzN3VRvYfqKqcOYIKXcxhbB1544fUbMcvr7wnBXab/dwlIKi9mko7r4gazo 81a6hCI9VwrRGONWjKfK6YWrRotpBHvlI9fX2PyrT3rxxIrivuZEjVzxHXbmtrZGbuX7 btSRGcG5bwF4RbxQX/Z3mgh86DxkC+ghfMPcyQgL6RugOpHMUyNLh1UkG+rEdaRYHhoj GcrVOnDPx5shekG4Sxu2hP75w6tHXaXSmf9QwDYfKYM4dZXuKHVhw8mzVBQ4trNwJQLM HY8g== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOAM531JSQp82yqj3h8WrVQCIW7gV96UZ0kI9POSRkNRcwzZp6JAkli+ JXLTFcyUP+hbN3S9Rr9Gzf8= X-Google-Smtp-Source: ABdhPJx/RgqVbWTFaJRpmaqWmt/1bEQVeE2Fk3cgARA441jcrdbFR2lw/c1RYzLL6GzSzsYTd9XRQw== X-Received: by 2002:a05:6512:3305:: with SMTP id k5mr9172487lfe.35.1607949065865; Mon, 14 Dec 2020 04:31:05 -0800 (PST) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a2e:3001:: with SMTP id w1ls96060ljw.3.gmail; Mon, 14 Dec 2020 04:30:14 -0800 (PST) X-Received: by 2002:a19:316:: with SMTP id 22mr3563794lfd.650.1607949014221; Mon, 14 Dec 2020 04:30:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1607949014; cv=none; d=google.com; s=arc-20160816; b=IZO2OzLuYtFh1rlf5aAVwJ5UkvvgoxGJyB5muOzxNX0Dph4vdlVWMjOYfFkjWPXwr6 zT86ddG5/TDT1bbpjeXVfOa5RBkc0oKq5WABrBx9BoUJlxW2pkaalLITmF7zekQSw2QQ rdN1y+8MnZNkkltbN0ljGGqn5RFMFDfyBgntWRfeZvdrHgslxQH2hRR3v/1wQFJwxTCP Z4q7PGp3eQosstikETgetx9///jbyz8jz+pUEg9AHHwrQOUPg/vW2bR+2XlUp8rc2MET qX4DyxUN+SizCt6mILoz264NbZllLyNM/BTPBUUO9nij4G8KnRl6kpljediY+OSpew8w sSvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=dkim-signature:content-transfer-encoding:mime-version:message-id :date:subject:cc:to:from; bh=71KZ7o/Jx96clpkanmeznSEiR1HY5/CPQAm7hddaw5E=; b=ahD7LhoHJiV9dfW2TYpJuLhYaJo1iQ7OBuqEHDuujh1tHixOiJ6/axkrtMMiu/Htns jIw+F+QLtHZ0gFF4y4XWRbao7wTL9O7B/Dy8hum2Z+/Al82cV/JoEo44wIY+dLX0or6X dwPZ8qRnehJYgpZUMfa6w5zY6RPGTYxxLh24tcM1JlJEd9syTGqoJzLc7VLPzculXQPL 4cSH7il0ggjw9tif/dldqhyam6BggpfCepj7hRD6AROBLzL5dipiSuXK5Lp6xWHMWDYd WPcJGNmE8gwC2buSHqKnVR+lNe/iqOhmW/m6nzWHmC937ncSjFFSFxXdcfEg7HFaLEvI EVQA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@teufel.de header.s=hse1 header.b=Ao01eKRU; spf=pass (google.com: domain of sava.jakovljev@teufel.de designates 94.100.136.165 as permitted sender) smtp.mailfrom=sava.jakovljev@teufel.de Received: from mx-relay65-hz2.antispameurope.com (mx-relay65-hz2.antispameurope.com. [94.100.136.165]) by gmr-mx.google.com with ESMTPS id j15si565816lfk.12.2020.12.14.04.30.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 14 Dec 2020 04:30:14 -0800 (PST) Received-SPF: pass (google.com: domain of sava.jakovljev@teufel.de designates 94.100.136.165 as permitted sender) client-ip=94.100.136.165; Received: from unknown ([212.91.255.190]) by mx-relay65-hz2.antispameurope.com; Mon, 14 Dec 2020 12:59:47 +0100 From: Sava Jakovljev To: CC: Sava Jakovljev Subject: [swupdate] [PATCH v2] channel_curl: Improve handling of tracking download progress Date: Mon, 14 Dec 2020 12:59:27 +0100 Message-ID: <20201214115927.41039-1-sava.jakovljev@teufel.de> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-Originating-IP: [10.10.25.44] X-ClientProxiedBy: DNS-EX-02.teufel.local (10.10.0.81) To DNS-EX-02.teufel.local (10.10.0.81) X-C2ProcessedOrg: b93e13a0-e8da-4ba4-97b8-f14375b21c41 X-cloud-security-sender: sava.jakovljev@teufel.de X-cloud-security-recipient: swupdate@googlegroups.com X-cloud-security-Virusscan: CLEAN X-cloud-security-disclaimer: This E-Mail was scanned by E-Mailservice on mx-relay65-hz2.antispameurope.com with D31B0C878DB X-cloud-security-connect: unknown[212.91.255.190], TLS=1, IP=212.91.255.190 X-cloud-security: scantime:.5187 X-Original-Sender: sava.jakovljev@teufel.de X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@teufel.de header.s=hse1 header.b=Ao01eKRU; spf=pass (google.com: domain of sava.jakovljev@teufel.de designates 94.100.136.165 as permitted sender) smtp.mailfrom=sava.jakovljev@teufel.de Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , * Compile xferinfo_legay function only for version of libcurl not supporting newer xferinfo option. * Use either XFERINFOFUNCTION or PROGRESSFUNCTION libcurl option depending on libcurl version - use XFERINFOFUNCTION if possible. * Introduce a structure containing information about file being downloaded. * Move setting progress tracking options into separate functions. * Improve callback for tracking download progress - correctly track percentage using valid variable and fix passing address of a temporary variable as callback argument. Signed-off-by: Sava Jakovljev --- When a server doesn't insert Content-Length header, there is no sense in enabling libcurl option XFERINFOFUNCTION/PROGRESSFUNCTION. This change also solves a bug that currently exists, where an address of a local variable 'percent' is passed to libcurl, which in turn gives it as an argument to a callback specified with XFERINFOFUNCTION/PROGRESSFUNCTION. Also, this change will use XFERINFOFUNCTION if libcurl supports it, and PROGRESSFUNCTION otherwise, and not both of them will be set, as the situation is right now. Legacy xferinfo callback function is now compiled only when libcurl does not support XFERINFOFUNCTION. Secondly, download progress tracking is now enabled only for get_file HTTP request and if server has correctly reported total download file size. v2 of this ptach deals with compiler warnings and removes unused fields of introduced download_callback_data_t structure. Also, code is cleaned up. Best regards, Sava Jakovljev corelib/channel_curl.c | 122 +++++++++++++++++++++++++++++++---------- 1 file changed, 93 insertions(+), 29 deletions(-) -- 2.26.2 diff --git a/corelib/channel_curl.c b/corelib/channel_curl.c index a3e785f..843da2a 100644 --- a/corelib/channel_curl.c +++ b/corelib/channel_curl.c @@ -55,6 +55,11 @@ typedef struct { output_data_t *outdata; } write_callback_t; +typedef struct { + curl_off_t total_download_size; + uint8_t percent; +} download_callback_data_t; + /* Prototypes for "internal" functions */ /* Note that they're not `static` so that they're callable from unit tests. */ @@ -414,25 +419,33 @@ static int channel_callback_xferinfo(void *p, curl_off_t dltotal, curl_off_t dln curl_off_t __attribute__((__unused__)) ultotal, curl_off_t __attribute__((__unused__)) ulnow) { - if ((dltotal <= 0) || (dlnow > dltotal)) { + download_callback_data_t *data = (download_callback_data_t*)p; + + if (data->total_download_size != dltotal) return 0; - } - double percent = 100.0 * (dlnow/1024.0) / (dltotal/1024.0); - double *last_percent = (double*)p; - if ((int)*last_percent == (int)percent) { + if ((dltotal <= 0) || (dlnow > dltotal)) return 0; - } - *last_percent = percent; + + uint8_t percent = 100.0 * ((double)dlnow / dltotal) + 0.5; + + if (data->percent == percent) + return 0; + else + data->percent = percent; + + TRACE("Download %d%% (%lu of %lu kB).", percent, dlnow, dltotal); swupdate_download_update(percent, dltotal); return 0; } +#if LIBCURL_VERSION_NUM < 0x072000 static int channel_callback_xferinfo_legacy(void *p, double dltotal, double dlnow, double ultotal, double ulnow) { return channel_callback_xferinfo(p, (curl_off_t)dltotal, (curl_off_t)dlnow, (curl_off_t)ultotal, (curl_off_t)ulnow); } +#endif static size_t channel_callback_headers(char *buffer, size_t size, size_t nitems, void *userdata) { @@ -566,28 +579,6 @@ channel_op_res_t channel_set_options(channel_t *this, channel_data_t *channel_da goto cleanup; } - double percent = -INFINITY; - if ((curl_easy_setopt(channel_curl->handle, CURLOPT_PROGRESSFUNCTION, - channel_callback_xferinfo_legacy) != CURLE_OK) || - (curl_easy_setopt(channel_curl->handle, CURLOPT_PROGRESSDATA, - &percent) != CURLE_OK)) { - result = CHANNEL_EINIT; - goto cleanup; - } -#if LIBCURL_VERSION_NUM >= 0x072000 - if ((curl_easy_setopt(channel_curl->handle, CURLOPT_XFERINFOFUNCTION, - channel_callback_xferinfo) != CURLE_OK) || - (curl_easy_setopt(channel_curl->handle, CURLOPT_XFERINFODATA, - &percent) != CURLE_OK)) { - result = CHANNEL_EINIT; - goto cleanup; - } -#endif - if (curl_easy_setopt(channel_curl->handle, CURLOPT_NOPROGRESS, 0L) != CURLE_OK) { - result = CHANNEL_EINIT; - goto cleanup; - } - if (channel_data->headers) { /* * Setup supply request and receive reply HTTP headers. @@ -731,6 +722,69 @@ cleanup: return result; } +static curl_off_t channel_get_total_download_size(channel_curl_t *this, + const char *url) +{ + assert(this != NULL); + assert(url != NULL); + + curl_off_t size = -1; + if (curl_easy_setopt(this->handle, CURLOPT_URL, url) != CURLE_OK || + curl_easy_setopt(this->handle, CURLOPT_NOBODY, 1L) != CURLE_OK || + curl_easy_perform(this->handle) != CURLE_OK) + goto cleanup; + + if (curl_easy_getinfo(this->handle, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T, + &size) != CURLE_OK) + goto cleanup; + +cleanup: + curl_easy_setopt(this->handle, CURLOPT_NOBODY, 0L); + return size; +} + +static channel_op_res_t channel_enable_download_progress_tracking( + channel_curl_t *this, + const char *url, + download_callback_data_t *download_data) +{ + assert(url != NULL); + assert(download_data != NULL); + + download_data->percent = 0; + + channel_op_res_t result = CHANNEL_OK; + if ((download_data->total_download_size = channel_get_total_download_size( + this, url)) <= 0) { + result = CHANNEL_EINIT; + goto cleanup; + } + +#if LIBCURL_VERSION_NUM >= 0x072000 + if ((curl_easy_setopt(this->handle, CURLOPT_XFERINFOFUNCTION, + channel_callback_xferinfo) != CURLE_OK) || + (curl_easy_setopt(this->handle, CURLOPT_XFERINFODATA, + download_data) != CURLE_OK)) { + result = CHANNEL_EINIT; + goto cleanup; + } +#else + if ((curl_easy_setopt(this->handle, CURLOPT_PROGRESSFUNCTION, + channel_callback_xferinfo_legacy) != CURLE_OK) || + (curl_easy_setopt(this->handle, CURLOPT_PROGRESSDATA, + download_data) != CURLE_OK)) { + result = CHANNEL_EINIT; + goto cleanup; + } +#endif + if (curl_easy_setopt(this->handle, CURLOPT_NOPROGRESS, 0L) != CURLE_OK) { + result = CHANNEL_EINIT; + goto cleanup; + } +cleanup: + return result; +} + static size_t put_read_callback(void *ptr, size_t size, size_t nmemb, void *data) { channel_data_t *channel_data = (channel_data_t *)data; @@ -1030,6 +1084,16 @@ channel_op_res_t channel_get_file(channel_t *this, void *data) goto cleanup_header; } + download_callback_data_t download_data; + if (channel_enable_download_progress_tracking(channel_curl, + channel_data->url, + &download_data) == CHANNEL_EINIT) { + WARN("Failed to get total download size for URL %s.", + channel_data->url); + } else + INFO("Total download size is %lu kB.", + download_data.total_download_size / 1024); + if (curl_easy_setopt(channel_curl->handle, CURLOPT_CUSTOMREQUEST, "GET") != CURLE_OK) { ERROR("Set GET channel method option failed.");