Message ID | CAFwEtmo=tXCckkQFn2VDoNV4tto7MBr0Tj3xVqLroxfJzZ=Fkw@mail.gmail.com |
---|---|
State | Awaiting Upstream |
Delegated to: | Richard Weinberger |
Headers | show |
Series | testing UBIFS with zstd | expand |
On Wed, Jun 6, 2018 at 2:55 PM, Michele Dionisio <michele.dionisio@gmail.com> wrote: > I'm testing ubifs with zstd compression. In my opinion it is a good > idea to start to insert this "new" compression algorithem because the > performance is paragonable with LZ4 and the compression is much > better. Numbers, please. :-) Also see: http://lists.infradead.org/pipermail/linux-mtd/2018-January/078742.html
Hi Michele, On Wed, Jun 6, 2018 at 5:56 AM Michele Dionisio <michele.dionisio@gmail.com> wrote: > > I'm testing ubifs with zstd compression. In my opinion it is a good > idea to start to insert this "new" compression algorithem because the > performance is paragonable with LZ4 and the compression is much > better. > > I attach patches for mtd-utils and linux too When you send patches upstream, please send them in-line as plain text in your email instead of as attachments. They can't be reviewed and commented on as blobs. https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html#no-mime-no-links-no-compression-no-attachments-just-plain-text Thanks, - Steve
I have 2 partitions on same NAND. one with LZ4 one with ztsd. Mount option are: root@flexaT:/# mount rootfs on / type rootfs (rw) ubi0:data on /opt/device/data type ubifs (rw,noatime,bulk_read) ubi1:spare on /opt/device/real_spare type ubifs (rw,noatime,bulk_read) I execute the following sequence of command for test INFILE="/dev/zero" rm -f /tmp/testfile dd if=${INFILE} of=/tmp/testfile bs=16k count=1k rm -f /opt/device/data/test df /opt/device/data sysctl -w vm.drop_caches=3 time dd if=/tmp/testfile of=/opt/device/data/test bs=64k count=1k sync sysctl -w vm.drop_caches=3 time dd if=/opt/device/data/test of=/dev/null bs=64k df /opt/device/data rm -f /opt/device/data/test rm -f /opt/device/real_spare/test df /opt/device/real_spare sysctl -w vm.drop_caches=3 time dd if=/tmp/testfile of=/opt/device/real_spare/test bs=64k count=1k sync sysctl -w vm.drop_caches=3 time dd if=/opt/device/real_spare/test of=/dev/null bs=64k df /opt/device/real_spare rm -f /opt/device/real_spare/test rm -f /tmp/testfile I have that the test 2 time using INFILE="/dev/zero" and INFILE="/dev/urandom". Both test is not realistic because zeros is too much comprimable and random is not comprimibile. The result is: ---------------------------------------------------------------------------------------- COMPRESSION | time write | time read | space used LZO (/dev/zero) | 2.39 | 1.40 | 624 blocks ZSTD (/dev/zero) | 2.63 | 1.43 | 596 blocks LZO (/dev/urandom) | 13.38 | 5.88 | 16024 blocks ZSTD (/dev/urandom) | 7.89 | 5.83 | 15896 blocks to have an idea about compression I have try to use one sqlite3 db and the size of the same file on both database is: LZO : 2688 bloks ZSTD: 1356 bloks Il giorno mer 6 giu 2018 alle ore 15:06 Richard Weinberger <richard.weinberger@gmail.com> ha scritto: > > On Wed, Jun 6, 2018 at 2:55 PM, Michele Dionisio > <michele.dionisio@gmail.com> wrote: > > I'm testing ubifs with zstd compression. In my opinion it is a good > > idea to start to insert this "new" compression algorithem because the > > performance is paragonable with LZ4 and the compression is much > > better. > > Numbers, please. :-) > Also see: > http://lists.infradead.org/pipermail/linux-mtd/2018-January/078742.html > > -- > Thanks, > //richard
Michele, Am Mittwoch, 6. Juni 2018, 18:01:17 CEST schrieb Michele Dionisio: > I have 2 partitions on same NAND. one with LZ4 one with ztsd. Mount option are: LZ4 or LZO? Below you write LZO... > root@flexaT:/# mount > rootfs on / type rootfs (rw) > ubi0:data on /opt/device/data type ubifs (rw,noatime,bulk_read) > ubi1:spare on /opt/device/real_spare type ubifs (rw,noatime,bulk_read) > > I execute the following sequence of command for test > > INFILE="/dev/zero" > rm -f /tmp/testfile > dd if=${INFILE} of=/tmp/testfile bs=16k count=1k > rm -f /opt/device/data/test > df /opt/device/data > sysctl -w vm.drop_caches=3 > time dd if=/tmp/testfile of=/opt/device/data/test bs=64k count=1k > sync > sysctl -w vm.drop_caches=3 > time dd if=/opt/device/data/test of=/dev/null bs=64k > df /opt/device/data > rm -f /opt/device/data/test > rm -f /opt/device/real_spare/test > df /opt/device/real_spare > sysctl -w vm.drop_caches=3 > time dd if=/tmp/testfile of=/opt/device/real_spare/test bs=64k count=1k > sync > sysctl -w vm.drop_caches=3 > time dd if=/opt/device/real_spare/test of=/dev/null bs=64k > df /opt/device/real_spare > rm -f /opt/device/real_spare/test > rm -f /tmp/testfile > > I have that the test 2 time using INFILE="/dev/zero" and > INFILE="/dev/urandom". Both test is not realistic because zeros is too > much comprimable and random is not comprimibile. The result is: > > ---------------------------------------------------------------------------------------- > COMPRESSION | time write | time read | space used > LZO (/dev/zero) | 2.39 | 1.40 | 624 blocks > ZSTD (/dev/zero) | 2.63 | 1.43 | 596 blocks > LZO (/dev/urandom) | 13.38 | 5.88 | 16024 blocks > ZSTD (/dev/urandom) | 7.89 | 5.83 | 15896 blocks What scale is time? ms? And what is block? LEBs? Thanks, //richard
From e4e484259296b1cfbaf14fe3f403df3b4f0b0849 Mon Sep 17 00:00:00 2001 From: Michele Dionisio <michele.dionisio@powersoft.com> Date: Wed, 6 Jun 2018 14:53:50 +0200 Subject: Add zstd compression on ubifs Signed-off-by: Michele Dionisio <michele.dionisio@gmail.com> diff --git a/fs/ubifs/Kconfig b/fs/ubifs/Kconfig index 83a961bf7280..1f44a6ff99cf 100644 --- a/fs/ubifs/Kconfig +++ b/fs/ubifs/Kconfig @@ -5,8 +5,10 @@ config UBIFS_FS select CRYPTO if UBIFS_FS_ADVANCED_COMPR select CRYPTO if UBIFS_FS_LZO select CRYPTO if UBIFS_FS_ZLIB + select CRYPTO if UBIFS_FS_ZSTD select CRYPTO_LZO if UBIFS_FS_LZO select CRYPTO_DEFLATE if UBIFS_FS_ZLIB + select CRYPTO_ZSTD if UBIFS_FS_ZSTD depends on MTD_UBI help UBIFS is a file system for flash devices which works on top of UBI. @@ -36,6 +38,14 @@ config UBIFS_FS_ZLIB help Zlib compresses better than LZO but it is slower. Say 'Y' if unsure. +config UBIFS_FS_ZSTD + bool "ZSTD compression support" if UBIFS_FS_ADVANCED_COMPR + depends on UBIFS_FS + default y + help + ZSTD compresses is a big win in speed over Zlib and + in compression ratio over LZO. Say 'Y' if unsure. + config UBIFS_ATIME_SUPPORT bool "Access time support" if UBIFS_FS depends on UBIFS_FS diff --git a/fs/ubifs/compress.c b/fs/ubifs/compress.c index 565cb56d7225..89183aeeeb7a 100644 --- a/fs/ubifs/compress.c +++ b/fs/ubifs/compress.c @@ -71,6 +71,24 @@ static struct ubifs_compressor zlib_compr = { }; #endif +#ifdef CONFIG_UBIFS_FS_ZSTD +static DEFINE_MUTEX(zstd_enc_mutex); +static DEFINE_MUTEX(zstd_dec_mutex); + +static struct ubifs_compressor zstd_compr = { + .compr_type = UBIFS_COMPR_ZSTD, + .comp_mutex = &zstd_enc_mutex, + .decomp_mutex = &zstd_dec_mutex, + .name = "zstd", + .capi_name = "zstd", +}; +#else +static struct ubifs_compressor zstd_compr = { + .compr_type = UBIFS_COMPR_ZSTD, + .name = "zstd", +}; +#endif + /* All UBIFS compressors */ struct ubifs_compressor *ubifs_compressors[UBIFS_COMPR_TYPES_CNT]; @@ -228,13 +246,19 @@ int __init ubifs_compressors_init(void) if (err) return err; - err = compr_init(&zlib_compr); + err = compr_init(&zstd_compr); if (err) goto out_lzo; + err = compr_init(&zlib_compr); + if (err) + goto out_zstd; + ubifs_compressors[UBIFS_COMPR_NONE] = &none_compr; return 0; +out_zstd: + compr_exit(&zstd_compr); out_lzo: compr_exit(&lzo_compr); return err; @@ -247,4 +271,5 @@ void ubifs_compressors_exit(void) { compr_exit(&lzo_compr); compr_exit(&zlib_compr); + compr_exit(&zstd_compr); } diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index 6c397a389105..2a725c8174b4 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -1035,6 +1035,8 @@ static int ubifs_parse_options(struct ubifs_info *c, char *options, c->mount_opts.compr_type = UBIFS_COMPR_LZO; else if (!strcmp(name, "zlib")) c->mount_opts.compr_type = UBIFS_COMPR_ZLIB; + else if (!strcmp(name, "zstd")) + c->mount_opts.compr_type = UBIFS_COMPR_ZSTD; else { ubifs_err(c, "unknown compressor \"%s\"", name); //FIXME: is c ready? kfree(name); diff --git a/fs/ubifs/ubifs-media.h b/fs/ubifs/ubifs-media.h index e8c23c9d4f4a..9f869c7c1532 100644 --- a/fs/ubifs/ubifs-media.h +++ b/fs/ubifs/ubifs-media.h @@ -341,12 +341,14 @@ enum { * UBIFS_COMPR_NONE: no compression * UBIFS_COMPR_LZO: LZO compression * UBIFS_COMPR_ZLIB: ZLIB compression + * UBIFS_COMPR_ZSTD: ZSTD compression * UBIFS_COMPR_TYPES_CNT: count of supported compression types */ enum { UBIFS_COMPR_NONE, UBIFS_COMPR_LZO, UBIFS_COMPR_ZLIB, + UBIFS_COMPR_ZSTD, UBIFS_COMPR_TYPES_CNT, };