| Message ID | 20181104231909.26596-1-daniel.santos@pobox.com |
|---|---|
| State | RFC |
| Delegated to: | John Crispin |
| Headers | show |
| Series | [OpenWrt-Devel,RFC] build: re-enable parallelism for mksquashfs | expand |
On 2018-11-05 00:19, Daniel Santos wrote: > This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due > to a threading bug back in 2009. The specifics of the bug are not given > in the commit message and squashfs-tools has had several updates to it's > parallelism since this time. There are currently no open issues related > to parallelism in their issue tracker: > https://github.com/plougher/squashfs-tools/issues > > It now "works for me" with 16 threads, and while this is a terrible test > for a race condition I still propose we remove this work-around > unless and until we have specific knowledge of a current bug. > > Signed-off-by: Daniel Santos <daniel.santos@pobox.com> Are the images still reproducible after that change? If I remember correctly, threading would break that. - Felix
On 11/07/2018 01:52 PM, Felix Fietkau wrote: > On 2018-11-05 00:19, Daniel Santos wrote: >> This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >> to a threading bug back in 2009. The specifics of the bug are not given >> in the commit message and squashfs-tools has had several updates to it's >> parallelism since this time. There are currently no open issues related >> to parallelism in their issue tracker: >> https://github.com/plougher/squashfs-tools/issues >> >> It now "works for me" with 16 threads, and while this is a terrible test >> for a race condition I still propose we remove this work-around >> unless and until we have specific knowledge of a current bug. >> >> Signed-off-by: Daniel Santos <daniel.santos@pobox.com> > Are the images still reproducible after that change? > If I remember correctly, threading would break that. > > - Felix > Hello. I'm not sure what you mean by the images being reproducible. I've been building with TARGET_ROOTFS_SQUASHFS=y and TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm presuming they've fixed it, but I didn't see specific mention of this bug in their github issue tracker. They may have discussed it elsewhere prior to porting to github. Daniel
Hi,
> I'm not sure what you mean by the images being reproducible.
It means that different mksquashfs runs with different amounts of CPUs
on different hosts should yield bit-identical images having the same
checksum given the same input.
~ Jo
On 2018-11-08 02:20, Daniel Santos wrote: > On 11/07/2018 01:52 PM, Felix Fietkau wrote: >> On 2018-11-05 00:19, Daniel Santos wrote: >>> This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >>> to a threading bug back in 2009. The specifics of the bug are not given >>> in the commit message and squashfs-tools has had several updates to it's >>> parallelism since this time. There are currently no open issues related >>> to parallelism in their issue tracker: >>> https://github.com/plougher/squashfs-tools/issues >>> >>> It now "works for me" with 16 threads, and while this is a terrible test >>> for a race condition I still propose we remove this work-around >>> unless and until we have specific knowledge of a current bug. >>> >>> Signed-off-by: Daniel Santos <daniel.santos@pobox.com> >> Are the images still reproducible after that change? >> If I remember correctly, threading would break that. > Hello. I'm not sure what you mean by the images being reproducible. > I've been building with TARGET_ROOTFS_SQUASHFS=y and > TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm > presuming they've fixed it, but I didn't see specific mention of this > bug in their github issue tracker. They may have discussed it elsewhere > prior to porting to github. Reproducible as in https://reproducible-builds.org/ You're supposed to be able to generate the binary-identical rootfs image on different machines, as long as you're using the same version and the same configuration. From my understanding, using multiple threads for the squashfs build can make the order in which data appears in the filesystem image non-deterministic. - Felix
Op do, 8 nov 2018 om 2:20 , schreef Daniel Santos <daniel.santos@pobox.com>: > On 11/07/2018 01:52 PM, Felix Fietkau wrote: >> On 2018-11-05 00:19, Daniel Santos wrote: >>> This was disabled by commit >>> dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >>> to a threading bug back in 2009. The specifics of the bug are not >>> given >>> in the commit message and squashfs-tools has had several updates >>> to it's >>> parallelism since this time. There are currently no open issues >>> related >>> to parallelism in their issue tracker: >>> https://github.com/plougher/squashfs-tools/issues >>> >>> It now "works for me" with 16 threads, and while this is a >>> terrible test >>> for a race condition I still propose we remove this work-around >>> unless and until we have specific knowledge of a current bug. >>> >>> Signed-off-by: Daniel Santos <daniel.santos@pobox.com> >> Are the images still reproducible after that change? >> If I remember correctly, threading would break that. >> >> - Felix >> > > Hello. I'm not sure what you mean by the images being reproducible. Reproducible means you can compile twice (or more) and it will result in identical (bit by bit, same checksum) binaries every time. Multithreading apparently broke that, as Felix stated. While reproducibility might not be an issue for you personally, I believe it's one of the project's goals - it's a way to ensure the builds that are publicly available have not been tampered with. Stijn > [...] > > Daniel > > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Ahh, thank you both for the clarification. On 11/08/2018 01:35 AM, Felix Fietkau wrote: > On 2018-11-08 02:20, Daniel Santos wrote: >> On 11/07/2018 01:52 PM, Felix Fietkau wrote: >>> On 2018-11-05 00:19, Daniel Santos wrote: >>>> This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >>>> to a threading bug back in 2009. The specifics of the bug are not given >>>> in the commit message and squashfs-tools has had several updates to it's >>>> parallelism since this time. There are currently no open issues related >>>> to parallelism in their issue tracker: >>>> https://github.com/plougher/squashfs-tools/issues >>>> >>>> It now "works for me" with 16 threads, and while this is a terrible test >>>> for a race condition I still propose we remove this work-around >>>> unless and until we have specific knowledge of a current bug. >>>> >>>> Signed-off-by: Daniel Santos <daniel.santos@pobox.com> >>> Are the images still reproducible after that change? >>> If I remember correctly, threading would break that. >> Hello. I'm not sure what you mean by the images being reproducible. >> I've been building with TARGET_ROOTFS_SQUASHFS=y and >> TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm >> presuming they've fixed it, but I didn't see specific mention of this >> bug in their github issue tracker. They may have discussed it elsewhere >> prior to porting to github. > Reproducible as in https://reproducible-builds.org/ > You're supposed to be able to generate the binary-identical rootfs image > on different machines, as long as you're using the same version and the > same configuration. > >From my understanding, using multiple threads for the squashfs build can > make the order in which data appears in the filesystem image > non-deterministic. > > - Felix I see. I have definitely not tested that. I would assume they could vary from one build to another based upon my understanding of how multi-threaded compression is done in general. How about a patch that allows multi-threaded compression via a .config option? Thanks, Daniel
The sender domain has a DMARC Reject/Quarantine policy which disallows sending mailing list messages using the original "From" header. To mitigate this problem, the original message has been wrapped automatically by the mailing list software. Hi, is there a plan to develop a web interface in Luci for handling SMS and USSID exchanges ? Why not integrate as backend the SmsTool3 http://smstools3.kekekasvi.com/ ? I have compiled and tested it, it works pretty well As a front end, I made myself a quick and dirty php interface (attached) but I beleive a good developper can do a much better job in Lua (this woudl also avoid the need of php) ? Waiting for your feedbacks Thank you JM
diff --git a/include/image.mk b/include/image.mk index f2a85f6feb..dcf1e3572a 100644 --- a/include/image.mk +++ b/include/image.mk @@ -203,7 +203,6 @@ define Image/mkfs/squashfs $(STAGING_DIR_HOST)/bin/mksquashfs4 $(call mkfs_target_dir,$(1)) $@ \ -nopad -noappend -root-owned \ -comp $(SQUASHFSCOMP) $(SQUASHFSOPT) \ - -processors 1 \ $(if $(SOURCE_DATE_EPOCH),-fixed-time $(SOURCE_DATE_EPOCH)) endef
This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due to a threading bug back in 2009. The specifics of the bug are not given in the commit message and squashfs-tools has had several updates to it's parallelism since this time. There are currently no open issues related to parallelism in their issue tracker: https://github.com/plougher/squashfs-tools/issues It now "works for me" with 16 threads, and while this is a terrible test for a race condition I still propose we remove this work-around unless and until we have specific knowledge of a current bug. Signed-off-by: Daniel Santos <daniel.santos@pobox.com> --- include/image.mk | 1 - 1 file changed, 1 deletion(-)