| Message ID | 47881381073118@web4j.yandex.ru |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
Konstantin, Sounds interesting, I'll integrate this into a next version of the patch. Perhaps it make sense just to replace favor_lzo and favor_lz4hc into something generic like favor_COMPR1_COMPR2 where COMPR2 gets chosen if it is more than favor-percent better than COMPR1. Even though today you're probably use lzo/zlib to optimise for size at a reasonable cost of some speed while you're probably use lzo/lz4hc if you want to optimise for speed at a reasonable cost of size, but that probably makes it more extensible in the future. Does this sound reasonable ? my 2 cents E. On Sun, Oct 6, 2013 at 5:25 PM, Konstantin Tokarev <annulen@yandex.ru> wrote: > > > 05.10.2013, 22:24, "Konstantin Tokarev" <annulen@yandex.ru>: >> 05.10.2013, 18:48, "Elie De Brauwer" <eliedebrauwer@gmail.com>: >> >>> All, >>> >>> After some time to do some testing this is a second version of the >>> same patch. Now lz4hc gets actually used. However at this moment in >>> time only improvement in terms of cpu time is seen, but lz4hc in this >>> context seems to behave worse than LZO, on my local system: >>> >>> -rw-r--r-- 1 edb edb 28M Oct 5 16:22 imag.lz4hc >>> -rw-r--r-- 1 root root 25M Oct 5 16:22 imag.lzo >>> -rw-r--r-- 1 root root 43M Oct 5 16:11 imag.none >>> -rw-r--r-- 1 root root 22M Oct 5 16:30 imag.zlib >>> >>> The only thing which comes immediately to mind is that due to the size >>> of the chunks (in my case large block being compressed is 4k) lz4hc >>> isn't starting to pay off .. >> >> Looks like it might be useful to have mixed lz4hc/lzo mode - lz4hc is faster, >> lzo provides better ratio, and both don't use additional memory for decompression. >> I did no speed benchmarks yet though. >> >> It also might be that lz4 has tuning parameters which may be useful for rootfs >> compression in case when most of files are not modified most of the times, i.e. >> trade compression time to space efficiency while keeping decompression speed. > > In attached patch I've replaced "favor lz4hc" mode with lz4hc/lzo mix, depending > purely on which algorithm gives smaller size for the block. In my case it allowed > to get slightly smaller image size than lzo (27564 blocks were compressed with lzo > and 1054 - with lz4hc). > > -- > Regards, > Konstantin
07.10.2013, 10:34, "Elie De Brauwer" <eliedebrauwer@gmail.com>: > Konstantin, > > Sounds interesting, I'll integrate this into a next version of the > patch. Perhaps it make sense just to replace favor_lzo and favor_lz4hc > into something generic like favor_COMPR1_COMPR2 where COMPR2 gets > chosen if it is more than favor-percent better than COMPR1. Even > though today you're probably use lzo/zlib to optimise for size at a > reasonable cost of some speed while you're probably use lzo/lz4hc if > you want to optimise for speed at a reasonable cost of size, but that > probably makes it more extensible in the future. Does this sound > reasonable ? Yeah, that would be useful.
From e113595e0bd845bbd3346806c7f10a17fb09927c Mon Sep 17 00:00:00 2001 From: Konstantin Tokarev <ktokarev@smartlabs.tv> Date: Sun, 6 Oct 2013 19:24:27 +0400 Subject: [PATCH] LZ4HC/LZO hybrid compression --- mkfs.ubifs/compr.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 51 insertions(+), 1 deletions(-) diff --git a/mkfs.ubifs/compr.c b/mkfs.ubifs/compr.c index 0900653..8f6e06e 100644 --- a/mkfs.ubifs/compr.c +++ b/mkfs.ubifs/compr.c @@ -182,6 +182,54 @@ select_zlib: return 0; } +static int lz4hc_lzo_compress(void *in_buf, size_t in_len, void *out_buf, + size_t *out_len, int *type) +{ + int lz4hc_ret, lzo_ret; + size_t lz4hc_len, lzo_len; + + lz4hc_len = lzo_len = *out_len; + lzo_ret = lzo_compress(in_buf, in_len, zlib_buf, &lzo_len); + lz4hc_ret = lz4hc_compress(in_buf, in_len, out_buf, &lz4hc_len); + + if (lz4hc_ret && lzo_ret) + /* Both compressors failed */ + return -1; + + if (!lz4hc_ret && !lzo_ret) { + double percent; + + /* Both compressors succeeded */ + if (lz4hc_len <= lzo_len ) + goto select_lz4hc; + + /*percent = (double)lzo_len / (double)lz4hc_len; + percent *= 100; + if (percent > 100 - c->favor_percent) + goto select_lz4hc;*/ + goto select_lzo; + } + + if (lz4hc_ret) + /* Only lzo compressor succeeded */ + goto select_lzo; + + /* Only lz4hc compressor succeeded */ + +select_lz4hc: + printf("lz4hc\n"); + *out_len = lz4hc_len; + *type = MKFS_UBIFS_COMPR_LZ4HC; + return 0; + +select_lzo: + printf("lzo\n"); + *out_len = lzo_len; + *type = MKFS_UBIFS_COMPR_LZO; + memcpy(out_buf, zlib_buf, lzo_len); + return 0; +} + int compress_data(void *in_buf, size_t in_len, void *out_buf, size_t *out_len, int type) { @@ -192,8 +240,10 @@ int compress_data(void *in_buf, size_t in_len, void *out_buf, size_t *out_len, return MKFS_UBIFS_COMPR_NONE; } - if (c->favor_lzo || c->favor_lz4hc) + if (c->favor_lzo) ret = favor_alt_compress(in_buf, in_len, out_buf, out_len, &type, c->favor_lzo); + else if (c->favor_lz4hc) + ret = lz4hc_lzo_compress(in_buf, in_len, out_buf, out_len, &type); else { switch (type) { case MKFS_UBIFS_COMPR_LZ4HC: -- 1.7.2.5