From patchwork Fri Oct 30 09:12:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Babic X-Patchwork-Id: 1390883 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::23a; helo=mail-lj1-x23a.google.com; envelope-from=swupdate+bncbcxploxj6ikrbefs576akgqe5mypxmy@googlegroups.com; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=denx.de Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20161025 header.b=BzBZZcyL; dkim-atps=neutral Received: from mail-lj1-x23a.google.com (mail-lj1-x23a.google.com [IPv6:2a00:1450:4864:20::23a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CMxRT3sgCz9sRK for ; Fri, 30 Oct 2020 20:12:53 +1100 (AEDT) Received: by mail-lj1-x23a.google.com with SMTP id r19sf2409899ljj.9 for ; Fri, 30 Oct 2020 02:12:53 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1604049169; cv=pass; d=google.com; s=arc-20160816; b=AF7xTuAl1qNP5xgCjsL/EbmTnaXpHiRZIUHCVoA9AMp/Jx8WhNBlqRfHjoGnzS32Qu b5ogsR0rGRteFY/Ke5EIgVxP7tF0Ega8U09cX/j7J4hcZkI99rhL7b8QJlSkVrEf/KDE Vf1lrCQAhf/7pjq4glSDGLixok8tFCWYgOY+i7Nv97rPI1+dMmuNnCq26xirAHvM5JKh dRA3jNqMbI9gCegb9WUbYS29oG4+69D5aoaV1kuJwcQmheIm2/c9mhwD1p4FWPYuRxn+ KzOvV9t18sx8jIgPJQxgQeUiPkw0qTG8qYFqHl9IC54j+yHNXUzPceOZRfjB3dhv6pSl Z+zg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=Tpj7a+LX/oWLM549E4I9+OPEpOvJNZn4S44ZMm0Kz2Q=; b=tQNxsZKg3X/k2/3ePDXwCLpJxCzs03I7N7Njjq5WhB6ojNxaJT+Og2x5OXn4aWRSD1 3X/dWxA4P+2he1lrXo78wXb0z4JSp11W8wLrpa7liLFMbg91pkQZQToEZff2dUG0j6gP qPQwApccadNw4UewQs5LQt/ew2L/EhFp2U9t7vGk599FlA9GYTdSHNvszMUwwjz9PYe0 Rfv+EK28ECYllsAOied19/B+kn0bQKCuhfLCt0R0RXzaWrChm1zWApkhKVr5hSa7udF3 j1zTV3fleC1XUKokE6OBtT5Ku0V+CNy5XzrVdihzq5xUyrxol4WhCXryxyQ7uyorMCO4 T1Cw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Tpj7a+LX/oWLM549E4I9+OPEpOvJNZn4S44ZMm0Kz2Q=; b=BzBZZcyLpmfIJWIgqvVf8o8b/Lfci60AKFrJOjSbzCJOab2dThJ9t4iLUKhXMYdsLI SF4IW2CTJ9UKaXSay76CsnZnhDgL8FXOZ75XsRmta9ua4ndHAdqCQnDk2NDdJDWbBaB/ ma52mQrQtBWs5ZKolRAkmW1iKPOWZx7pGcnFTRUaixSSHQQFPP2eg4mhipRRlXtwP20+ IOcYs11HOFiU22Fo8Ptt91YHpPDLKgLRLdTm0IzFMtie80xTvJdFSBZOftavJ7EE+ial 6RpH1nWZLZGfEcRlowqVGo320LPgZhWTam3O+RK2chngXE95tol9uO1pn7Y+mew+5/zS YonA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Tpj7a+LX/oWLM549E4I9+OPEpOvJNZn4S44ZMm0Kz2Q=; b=Drlm9l4hiYERmkPJeN08XbbXrm4MSz9UBUmXDw6lBDByOlg89fq759EzUoORV32G9L P2AXxT+6cXWL9CBywZm95YojLEl9QthKTtps/3nJ98dXlUqP8DpA82yReibBj2ZSGMyU S0TjiMo5MIe+BuuodAQAxnhr0XIwhRNIgEmrqtDJdljK9BQ5HtIBpZVOxrhUyhXdswzJ 9QH/HP+h89uCkDeOF7LIDDl3Z26+F//4uP8ROE+bRckDEgaTCI+orpEjTjyk6mRbMVP8 iUtvF70Gz0NQ2Et8lzGio1SMlodOIfVgnqrsBWc0wPnDymafyeNKjcKuZjLwg+fItrB/ CJ3g== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOAM530qIm/xiibvC4LivfOWBgl6qg7jnyZ+CgsCidVYYcDvnJLWfB4D VEYUdYq2d8VTRfIdoADJoYM= X-Google-Smtp-Source: ABdhPJytYjMzKxi87tsesjs2MTeI+k5S32U5DIF5Yu+sxe0qqbHh2/eYlnTnZvXUFIFhTrEuRxTB0A== X-Received: by 2002:a05:651c:141:: with SMTP id c1mr647307ljd.28.1604049169435; Fri, 30 Oct 2020 02:12:49 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a19:84c1:: with SMTP id g184ls287386lfd.3.gmail; Fri, 30 Oct 2020 02:12:48 -0700 (PDT) X-Received: by 2002:a19:8087:: with SMTP id b129mr543215lfd.471.1604049168177; Fri, 30 Oct 2020 02:12:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1604049168; cv=none; d=google.com; s=arc-20160816; b=Qoq62mLNIpwQ1GO+CwUqIZF9ToC2ZCal4JHmjfYuwDEzKyXdwqHUTrlsdhVtMX8Yyx NuN15Z9bKsiqdbhiGDS3EcIrdBFJDIEZ7R3wa10TnIEibu99c6fECR25c0ojc3hOe/kT X6CaEHsMVoI9pt+vFXXn/Q8rdZ+41210obivPCEoIcrEXEaJ5o2OevgVLYu+Nhy71HSA GS1plBMsikL98EgyRjQ631kwe2AkfLSHo1g843XyVkUKdjNdZhsm9ZtvSCmj4pmPrkLC h09WFEZ+SeaU49KzakljlrLyNbKRGDMo+m4+8NVk6phyZ9jSYT2OstlXpee7zmoLrtb8 Z6jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=NdZuPAIxoMTtRXrFb6EHtM4JTwJpImIlKQMgYWPZeUg=; b=Xhetxc5DTANi2c68LdVXunZUrc3mDIhk4TcFq79DYZIQEo5jrbn3SDG0b0sd3YdYzW SYYeedRWgtAYQgaOboJ4H34nq8SV5IPnM9sqYpe0K7OT53/UnlhhGU5h9DjShEHmvoGq OsuKWpn/xK7eCbmm1nMTPLMzl3qLPhYl/AtGZKiHEIAb8hVAr345G4XMkjMWICmGHHrE CnHbOCHVrlT3hDZtBN9E+X3I/upSE5t2ptWY6dNHvbNGHtJKUHTaelvFvJLFcu7eQB70 TsugSbwuDRM9HpW3AlqAvR3a1pqv0eO1qUjvQkEkFnSz2jW8dBeXpD0VjFwZ7yo3xkT1 6diQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.10]) by gmr-mx.google.com with ESMTPS id y17si138415lfh.4.2020.10.30.02.12.47 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Oct 2020 02:12:48 -0700 (PDT) Received-SPF: neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of sbabic@denx.de) client-ip=212.18.0.10; Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4CMxRM49F6z1rxYR; Fri, 30 Oct 2020 10:12:47 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4CMxRM3yZDz1qsWm; Fri, 30 Oct 2020 10:12:47 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id 7VX5ks-YYuog; Fri, 30 Oct 2020 10:12:46 +0100 (CET) Received: from babic.homelinux.org (host-88-217-136-221.customer.m-online.net [88.217.136.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Fri, 30 Oct 2020 10:12:46 +0100 (CET) Received: from localhost (mail.babic.homelinux.org [127.0.0.1]) by babic.homelinux.org (Postfix) with ESMTP id 6E72945402B2; Fri, 30 Oct 2020 10:12:46 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at babic.homelinux.org Received: from babic.homelinux.org ([127.0.0.1]) by localhost (mail.babic.homelinux.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l-Ktga4ZBn4u; Fri, 30 Oct 2020 10:12:44 +0100 (CET) Received: from paperino.arri.de (paperino.fritz.box [192.168.178.64]) by babic.homelinux.org (Postfix) with ESMTP id 5DCCC45408A2; Fri, 30 Oct 2020 10:12:41 +0100 (CET) From: Stefano Babic To: swupdate@googlegroups.com Cc: Stefano Babic Subject: [swupdate] [PATCH 2/4] Add support for cached file to curl Date: Fri, 30 Oct 2020 10:12:37 +0100 Message-Id: <20201030091239.2802869-3-sbabic@denx.de> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201030091239.2802869-1-sbabic@denx.de> References: <20201030091239.2802869-1-sbabic@denx.de> MIME-Version: 1.0 X-Original-Sender: sbabic@denx.de X-Original-Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , This is to allow a resume after SWUpdate was interrupted and it is restarted, for example after a power-cut. If the SWU is also saved with the --output parameter, it should be possible to download just the rest of data instead of restaring from the beginning. The data in the cached file will be read first, and then a RESUME to the server is sent to go on with the rest of data. Signed-off-by: Stefano Babic --- corelib/channel_curl.c | 65 ++++++++++++++++++++++++++++++++++++++++++ include/channel_curl.h | 1 + 2 files changed, 66 insertions(+) diff --git a/corelib/channel_curl.c b/corelib/channel_curl.c index c95b6b3..a91cf5f 100644 --- a/corelib/channel_curl.c +++ b/corelib/channel_curl.c @@ -7,6 +7,9 @@ #include #include +#include +#include +#include #include #include #include @@ -193,6 +196,46 @@ size_t channel_callback_ipc(void *streamdata, size_t size, size_t nmemb, return size * nmemb; } +#define BUFF_SIZE 16384 +static unsigned long long int resume_cache_file(const char *fname, + write_callback_t *data) +{ + int fdsw; + char *buf; + ssize_t cnt; + unsigned long long processed = 0; + + if (!fname || !strlen(fname)) + return 0; + fdsw = open(fname, O_RDONLY); + if (fdsw < 0) + return 0; /* ignore, load from network */ + buf = calloc(1, BUFF_SIZE); + if (!buf) { + ERROR("Channel get operation failed with OOM"); + close(fdsw); + return 0; + } + + while ((cnt = read(fdsw, buf, BUFF_SIZE)) > 0) { + if (!channel_callback_ipc(buf, cnt, 1, data)) + break; + processed += cnt; + } + + close(fdsw); + free(buf); + + /* + * Cache file is used just once: after it is read, it is + * dropped automatically to avoid to reuse it again + * for next update + */ + unlink(fname); + + return processed; +} + size_t channel_callback_membuffer(void *streamdata, size_t size, size_t nmemb, write_callback_t *data) { @@ -1031,6 +1074,28 @@ channel_op_res_t channel_get_file(channel_t *this, void *data) unsigned long long int total_bytes_downloaded = 0; unsigned char try_count = 0; CURLcode curlrc = CURLE_OK; + + if (channel_data->cached_file) { + + total_bytes_downloaded = resume_cache_file(channel_data->cached_file, + &wrdata); + if (total_bytes_downloaded > 0) { + TRACE("Resume from cache file %s, restored %lld bytes", + channel_data->cached_file, + total_bytes_downloaded); + /* + * Simulate that a partial download was already done, + * and tune parameters if retries is not set + */ + channel_data->retries++; + try_count++; + } + } + + /* + * If there is a cache file, read data from cache first + * and load from URL the remaining data + */ do { if (try_count > 0) { if (channel_data->retries == 0) { diff --git a/include/channel_curl.h b/include/channel_curl.h index c3418a5..26b925c 100644 --- a/include/channel_curl.h +++ b/include/channel_curl.h @@ -38,6 +38,7 @@ typedef enum { typedef struct { char *url; + char *cached_file; char *auth; char *request_body; char *iface;