diff mbox series

testing UBIFS with zstd

Message ID CAFwEtmo=tXCckkQFn2VDoNV4tto7MBr0Tj3xVqLroxfJzZ=Fkw@mail.gmail.com
State Awaiting Upstream
Delegated to: Richard Weinberger
Headers show
Series testing UBIFS with zstd | expand

Commit Message

Michele Dionisio June 6, 2018, 12:55 p.m. UTC 
I'm testing ubifs with zstd compression. In my opinion it is a good
idea to start to insert this "new" compression algorithem because the
performance is paragonable with LZ4 and the compression is much
better.

I attach patches for mtd-utils and linux too

(remember to add also https://lkml.org/lkml/2017/9/8/824)

Comments

Richard Weinberger June 6, 2018, 1:06 p.m. UTC | #1 
On Wed, Jun 6, 2018 at 2:55 PM, Michele Dionisio
<michele.dionisio@gmail.com> wrote:
> I'm testing ubifs with zstd compression. In my opinion it is a good
> idea to start to insert this "new" compression algorithem because the
> performance is paragonable with LZ4 and the compression is much
> better.

Numbers, please. :-)
Also see:
http://lists.infradead.org/pipermail/linux-mtd/2018-January/078742.html
Steve deRosier June 6, 2018, 2:38 p.m. UTC | #2 
Hi Michele,

On Wed, Jun 6, 2018 at 5:56 AM Michele Dionisio
<michele.dionisio@gmail.com> wrote:
>
> I'm testing ubifs with zstd compression. In my opinion it is a good
> idea to start to insert this "new" compression algorithem because the
> performance is paragonable with LZ4 and the compression is much
> better.
>
> I attach patches for mtd-utils and linux too

When you send patches upstream, please send them in-line as plain text
in your email instead of as attachments. They can't be reviewed and
commented on as blobs.
https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html#no-mime-no-links-no-compression-no-attachments-just-plain-text

Thanks,
- Steve
Michele Dionisio June 6, 2018, 4:01 p.m. UTC | #3 
I have 2 partitions on same NAND. one with LZ4 one with ztsd. Mount option are:

root@flexaT:/# mount
rootfs on / type rootfs (rw)
ubi0:data on /opt/device/data type ubifs (rw,noatime,bulk_read)
ubi1:spare on /opt/device/real_spare type ubifs (rw,noatime,bulk_read)

I execute the following sequence of command for test

INFILE="/dev/zero"
rm -f /tmp/testfile
dd if=${INFILE} of=/tmp/testfile bs=16k count=1k
rm -f /opt/device/data/test
df /opt/device/data
sysctl -w vm.drop_caches=3
time dd if=/tmp/testfile of=/opt/device/data/test bs=64k count=1k
sync
sysctl -w vm.drop_caches=3
time dd if=/opt/device/data/test of=/dev/null bs=64k
df /opt/device/data
rm -f /opt/device/data/test
rm -f /opt/device/real_spare/test
df /opt/device/real_spare
sysctl -w vm.drop_caches=3
time dd if=/tmp/testfile of=/opt/device/real_spare/test bs=64k count=1k
sync
sysctl -w vm.drop_caches=3
time dd if=/opt/device/real_spare/test of=/dev/null bs=64k
df /opt/device/real_spare
rm -f /opt/device/real_spare/test
rm -f /tmp/testfile

I have that the test 2 time using INFILE="/dev/zero" and
INFILE="/dev/urandom". Both test is not realistic because zeros is too
much comprimable and random is not comprimibile. The result is:

----------------------------------------------------------------------------------------
COMPRESSION           |  time write    | time read   |   space used
LZO (/dev/zero)             |  2.39          | 1.40        |   624 blocks
ZSTD (/dev/zero)           |  2.63          | 1.43        |   596 blocks
LZO (/dev/urandom)    |  13.38         | 5.88        |   16024 blocks
ZSTD (/dev/urandom)   |  7.89          | 5.83        |   15896 blocks

to have an idea about compression I have try to use one sqlite3 db and
the size of the same file on both database is:
LZO : 2688 bloks
ZSTD: 1356 bloks


Il giorno mer 6 giu 2018 alle ore 15:06 Richard Weinberger
<richard.weinberger@gmail.com> ha scritto:
>
> On Wed, Jun 6, 2018 at 2:55 PM, Michele Dionisio
> <michele.dionisio@gmail.com> wrote:
> > I'm testing ubifs with zstd compression. In my opinion it is a good
> > idea to start to insert this "new" compression algorithem because the
> > performance is paragonable with LZ4 and the compression is much
> > better.
>
> Numbers, please. :-)
> Also see:
> http://lists.infradead.org/pipermail/linux-mtd/2018-January/078742.html
>
> --
> Thanks,
> //richard
Richard Weinberger June 6, 2018, 5:30 p.m. UTC | #4 
Michele,

Am Mittwoch, 6. Juni 2018, 18:01:17 CEST schrieb Michele Dionisio:
> I have 2 partitions on same NAND. one with LZ4 one with ztsd. Mount option are:

LZ4 or LZO?
Below you write LZO...
 
> root@flexaT:/# mount
> rootfs on / type rootfs (rw)
> ubi0:data on /opt/device/data type ubifs (rw,noatime,bulk_read)
> ubi1:spare on /opt/device/real_spare type ubifs (rw,noatime,bulk_read)
> 
> I execute the following sequence of command for test
> 
> INFILE="/dev/zero"
> rm -f /tmp/testfile
> dd if=${INFILE} of=/tmp/testfile bs=16k count=1k
> rm -f /opt/device/data/test
> df /opt/device/data
> sysctl -w vm.drop_caches=3
> time dd if=/tmp/testfile of=/opt/device/data/test bs=64k count=1k
> sync
> sysctl -w vm.drop_caches=3
> time dd if=/opt/device/data/test of=/dev/null bs=64k
> df /opt/device/data
> rm -f /opt/device/data/test
> rm -f /opt/device/real_spare/test
> df /opt/device/real_spare
> sysctl -w vm.drop_caches=3
> time dd if=/tmp/testfile of=/opt/device/real_spare/test bs=64k count=1k
> sync
> sysctl -w vm.drop_caches=3
> time dd if=/opt/device/real_spare/test of=/dev/null bs=64k
> df /opt/device/real_spare
> rm -f /opt/device/real_spare/test
> rm -f /tmp/testfile
> 
> I have that the test 2 time using INFILE="/dev/zero" and
> INFILE="/dev/urandom". Both test is not realistic because zeros is too
> much comprimable and random is not comprimibile. The result is:
> 
> ----------------------------------------------------------------------------------------
> COMPRESSION           |  time write    | time read   |   space used
> LZO (/dev/zero)             |  2.39          | 1.40        |   624 blocks
> ZSTD (/dev/zero)           |  2.63          | 1.43        |   596 blocks
> LZO (/dev/urandom)    |  13.38         | 5.88        |   16024 blocks
> ZSTD (/dev/urandom)   |  7.89          | 5.83        |   15896 blocks

What scale is time? ms?
And what is block? LEBs?

Thanks,
//richard
diff mbox series

Patch

From e4e484259296b1cfbaf14fe3f403df3b4f0b0849 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio@powersoft.com>
Date: Wed, 6 Jun 2018 14:53:50 +0200
Subject: Add zstd compression on ubifs

Signed-off-by: Michele Dionisio <michele.dionisio@gmail.com>

diff --git a/fs/ubifs/Kconfig b/fs/ubifs/Kconfig
index 83a961bf7280..1f44a6ff99cf 100644
--- a/fs/ubifs/Kconfig
+++ b/fs/ubifs/Kconfig
@@ -5,8 +5,10 @@  config UBIFS_FS
 	select CRYPTO if UBIFS_FS_ADVANCED_COMPR
 	select CRYPTO if UBIFS_FS_LZO
 	select CRYPTO if UBIFS_FS_ZLIB
+	select CRYPTO if UBIFS_FS_ZSTD
 	select CRYPTO_LZO if UBIFS_FS_LZO
 	select CRYPTO_DEFLATE if UBIFS_FS_ZLIB
+	select CRYPTO_ZSTD if UBIFS_FS_ZSTD
 	depends on MTD_UBI
 	help
 	  UBIFS is a file system for flash devices which works on top of UBI.
@@ -36,6 +38,14 @@  config UBIFS_FS_ZLIB
 	help
 	  Zlib compresses better than LZO but it is slower. Say 'Y' if unsure.
 
+config UBIFS_FS_ZSTD
+	bool "ZSTD compression support" if UBIFS_FS_ADVANCED_COMPR
+	depends on UBIFS_FS
+	default y
+	help
+	  ZSTD compresses is a big win in speed over Zlib and
+	  in compression ratio over LZO. Say 'Y' if unsure.
+
 config UBIFS_ATIME_SUPPORT
 	bool "Access time support" if UBIFS_FS
 	depends on UBIFS_FS
diff --git a/fs/ubifs/compress.c b/fs/ubifs/compress.c
index 565cb56d7225..89183aeeeb7a 100644
--- a/fs/ubifs/compress.c
+++ b/fs/ubifs/compress.c
@@ -71,6 +71,24 @@  static struct ubifs_compressor zlib_compr = {
 };
 #endif
 
+#ifdef CONFIG_UBIFS_FS_ZSTD
+static DEFINE_MUTEX(zstd_enc_mutex);
+static DEFINE_MUTEX(zstd_dec_mutex);
+
+static struct ubifs_compressor zstd_compr = {
+	.compr_type = UBIFS_COMPR_ZSTD,
+	.comp_mutex = &zstd_enc_mutex,
+	.decomp_mutex = &zstd_dec_mutex,
+	.name = "zstd",
+	.capi_name = "zstd",
+};
+#else
+static struct ubifs_compressor zstd_compr = {
+	.compr_type = UBIFS_COMPR_ZSTD,
+	.name = "zstd",
+};
+#endif
+
 /* All UBIFS compressors */
 struct ubifs_compressor *ubifs_compressors[UBIFS_COMPR_TYPES_CNT];
 
@@ -228,13 +246,19 @@  int __init ubifs_compressors_init(void)
 	if (err)
 		return err;
 
-	err = compr_init(&zlib_compr);
+	err = compr_init(&zstd_compr);
 	if (err)
 		goto out_lzo;
 
+	err = compr_init(&zlib_compr);
+	if (err)
+		goto out_zstd;
+
 	ubifs_compressors[UBIFS_COMPR_NONE] = &none_compr;
 	return 0;
 
+out_zstd:
+	compr_exit(&zstd_compr);
 out_lzo:
 	compr_exit(&lzo_compr);
 	return err;
@@ -247,4 +271,5 @@  void ubifs_compressors_exit(void)
 {
 	compr_exit(&lzo_compr);
 	compr_exit(&zlib_compr);
+	compr_exit(&zstd_compr);
 }
diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index 6c397a389105..2a725c8174b4 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -1035,6 +1035,8 @@  static int ubifs_parse_options(struct ubifs_info *c, char *options,
 				c->mount_opts.compr_type = UBIFS_COMPR_LZO;
 			else if (!strcmp(name, "zlib"))
 				c->mount_opts.compr_type = UBIFS_COMPR_ZLIB;
+			else if (!strcmp(name, "zstd"))
+				c->mount_opts.compr_type = UBIFS_COMPR_ZSTD;
 			else {
 				ubifs_err(c, "unknown compressor \"%s\"", name); //FIXME: is c ready?
 				kfree(name);
diff --git a/fs/ubifs/ubifs-media.h b/fs/ubifs/ubifs-media.h
index e8c23c9d4f4a..9f869c7c1532 100644
--- a/fs/ubifs/ubifs-media.h
+++ b/fs/ubifs/ubifs-media.h
@@ -341,12 +341,14 @@  enum {
  * UBIFS_COMPR_NONE: no compression
  * UBIFS_COMPR_LZO: LZO compression
  * UBIFS_COMPR_ZLIB: ZLIB compression
+ * UBIFS_COMPR_ZSTD: ZSTD compression
  * UBIFS_COMPR_TYPES_CNT: count of supported compression types
  */
 enum {
 	UBIFS_COMPR_NONE,
 	UBIFS_COMPR_LZO,
 	UBIFS_COMPR_ZLIB,
+	UBIFS_COMPR_ZSTD,
 	UBIFS_COMPR_TYPES_CNT,
 };