| Message ID | 20200824230032.492951-1-baptiste@bitsofnetworks.org |
|---|---|
| Headers | show |
| Series | Purge packages from cache when they have incorrect checksum | expand |
Hi, Any feedback on this patch series? It fixes this annoying bug in the imagebuilder: https://bugs.openwrt.org/index.php?do=details&task_id=2690 Since the changes are somewhat significant, I would like some exposure in master for a few weeks/months before it can be backported to 18.06 and 19.07. Thanks, Baptiste On 25-08-20, Baptiste Jonglez wrote: > From: Baptiste Jonglez <git@bitsofnetworks.org> > > The motivation of this patch series is to fix FS#2690. Because packages > are continuously rebuilt, the ImageBuilder ends up with old packages in > its cache, and fails because checksums don't match with the new package > index from the download server. > > The approach to solve this problem is the following. Before using a package > from the cache, verify its size and checksum against the package index, and > delete the package from the cache if they don't match. The install process > will then proceed to download the "fixed" package as usual. > > The main patch is "download: purge cached packages that have incorrect > checksum". The other ones are either cleanup or refactoring to prepare > for the main change. > > Baptiste Jonglez (5): > download: remove compatibility with old cache naming scheme > libopkg: factor out checksum and size verification > download: factor out the logic for building cache filenames > download: purge cached packages that have incorrect checksum > opkg_verify_integrity: better logging and error conditions > > libopkg/opkg_download.c | 123 ++++++++++++++++++++++++++++++++-------- > libopkg/opkg_download.h | 1 + > libopkg/opkg_install.c | 76 +++---------------------- > 3 files changed, 108 insertions(+), 92 deletions(-)
Hi, Any news on this patch series? It should definitely be considered before the 20.XX branching because it fixes FS#2690 (bug related to imagebuilder), and I would like to backport it to 19.07 at some point. Thanks, Baptiste On 25-08-20, Baptiste Jonglez wrote: > From: Baptiste Jonglez <git@bitsofnetworks.org> > > The motivation of this patch series is to fix FS#2690. Because packages > are continuously rebuilt, the ImageBuilder ends up with old packages in > its cache, and fails because checksums don't match with the new package > index from the download server. > > The approach to solve this problem is the following. Before using a package > from the cache, verify its size and checksum against the package index, and > delete the package from the cache if they don't match. The install process > will then proceed to download the "fixed" package as usual. > > The main patch is "download: purge cached packages that have incorrect > checksum". The other ones are either cleanup or refactoring to prepare > for the main change. > > Baptiste Jonglez (5): > download: remove compatibility with old cache naming scheme > libopkg: factor out checksum and size verification > download: factor out the logic for building cache filenames > download: purge cached packages that have incorrect checksum > opkg_verify_integrity: better logging and error conditions > > libopkg/opkg_download.c | 123 ++++++++++++++++++++++++++++++++-------- > libopkg/opkg_download.h | 1 + > libopkg/opkg_install.c | 76 +++---------------------- > 3 files changed, 108 insertions(+), 92 deletions(-) > > -- > 2.27.0 > > > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
On Fri Nov 20, 2020 at 12:43 AM HST, Baptiste Jonglez wrote: > Hi, > > Any news on this patch series? It should definitely be considered > before the 20.XX branching because it fixes FS#2690 (bug related to > imagebuilder), and I would like to backport it to 19.07 at some point. I CI built this and tested the ImageBuilder as follows: * Run once and create firmware images * Run `echo "foo" >> dl/base-files*.ipk * Run again The `opkg` detects correctly a checksum missmatch and downloads a fresh copy again. Excellent! Below a snipped log --- 8< --- Installing kernel (5.4.77-1-211d098873122e25225ddfe3f75dbf13) to root... Configuring kernel. Installing base-files (237-r14984-d369993898) to root... Removing base-files from cache because it has incorrect checksum. Downloading http://downloads.openwrt.org/snapshots/targets/ath79/generic/packages/base-files_237-r14984-d369993898_mips_24kc.ipk Installing libubox20191228 (2020-08-06-9e52171d-1) to root... Copying /tmp/f1/openwrt-imagebuilder-ath79-generic.Linux-x86_64/dl/libubox20191228_2020-08-06-9e52171d-1_mips_24kc.ipk. Installing libuci20130104 (2020-10-06-52bbc99f-3) to root... --- >8 --- I couldn't come up with a simple way to change the size while keeping the same checksum (hoping overall there is no such way anyway), so that's untested. However, as this patch "recycles" the previously working size check I have no doubt it keeps working. Size for mvebu/cortexa9/arm_cortex-a9_vfpv3-d16 increased a tiny bit, but that seems totally worth it. Comparing package sizes... Change Local Remote Package +98 66602 66504 opkg Therefore, Acked-by: Paul Spooren <mail@aparcar.org> > > Thanks, > Baptiste > > On 25-08-20, Baptiste Jonglez wrote: > > From: Baptiste Jonglez <git@bitsofnetworks.org> > > > > The motivation of this patch series is to fix FS#2690. Because packages > > are continuously rebuilt, the ImageBuilder ends up with old packages in > > its cache, and fails because checksums don't match with the new package > > index from the download server. > > > > The approach to solve this problem is the following. Before using a package > > from the cache, verify its size and checksum against the package index, and > > delete the package from the cache if they don't match. The install process > > will then proceed to download the "fixed" package as usual. > > > > The main patch is "download: purge cached packages that have incorrect > > checksum". The other ones are either cleanup or refactoring to prepare > > for the main change. > > > > Baptiste Jonglez (5): > > download: remove compatibility with old cache naming scheme > > libopkg: factor out checksum and size verification > > download: factor out the logic for building cache filenames > > download: purge cached packages that have incorrect checksum > > opkg_verify_integrity: better logging and error conditions > > > > libopkg/opkg_download.c | 123 ++++++++++++++++++++++++++++++++-------- > > libopkg/opkg_download.h | 1 + > > libopkg/opkg_install.c | 76 +++---------------------- > > 3 files changed, 108 insertions(+), 92 deletions(-) > > > > -- > > 2.27.0 > > > > > > _______________________________________________ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
From: Baptiste Jonglez <git@bitsofnetworks.org> The motivation of this patch series is to fix FS#2690. Because packages are continuously rebuilt, the ImageBuilder ends up with old packages in its cache, and fails because checksums don't match with the new package index from the download server. The approach to solve this problem is the following. Before using a package from the cache, verify its size and checksum against the package index, and delete the package from the cache if they don't match. The install process will then proceed to download the "fixed" package as usual. The main patch is "download: purge cached packages that have incorrect checksum". The other ones are either cleanup or refactoring to prepare for the main change. Baptiste Jonglez (5): download: remove compatibility with old cache naming scheme libopkg: factor out checksum and size verification download: factor out the logic for building cache filenames download: purge cached packages that have incorrect checksum opkg_verify_integrity: better logging and error conditions libopkg/opkg_download.c | 123 ++++++++++++++++++++++++++++++++-------- libopkg/opkg_download.h | 1 + libopkg/opkg_install.c | 76 +++---------------------- 3 files changed, 108 insertions(+), 92 deletions(-)