From patchwork Sun Mar  8 16:46:10 2009
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: mkfs.jffs2 aborts with MALLOC_CHECK_=2 on x86_64
From: Stefan Seyfried <seife@suse.de>
X-Patchwork-Id: 24178
Message-Id: <20090308164610.GA25396@suse.de>
To: linux-mtd@lists.infradead.org
Date: Sun, 8 Mar 2009 17:46:10 +0100

On Fri, Jan 16, 2009 at 10:47:29PM +0100, Stefan Seyfried wrote:
> Hi,
> 
> current mtd-utils' mkfs.jffs2 aborts on me:
> seife@stoetzler:~> /dev/shm/mtd-utils/mkfs.jffs2 -L
> mkfs.jffs2:
>        lzo priority:80 disabled
>       zlib priority:60 enabled
>      rtime priority:50 enabled
> 
> seife@stoetzler:~> MALLOC_CHECK_=2 /dev/shm/mtd-utils/mkfs.jffs2 -U -b -e
> 131072 -p -r . -o /tmp/img.jffs2
> Aborted


> I looked around and found out that it happens, when both enabled compressors
> return -1 in compr.c line 246, and then the free in line 258 aborts.
> 
> doing
> 
> #define STREAM_END_SPACE 20
> 
> instead of the default of 12 in compr_zlib.c fixes it for me. However, I'm
> neither shure if this has any bad side effects, nor _why_ it fixes it.
> My host is 64bits (x86_64), maybe this is affecting the buffer sizes or
> something like that.
> Hope this is helpful.

valgrind was much more helpful than gdb in this case.
I'm pretty sure it's an integer underflow: it happens when
jffs2_rtime_compress is called with *dstlen = 1
The same in compr_zlib has not triggered for me yet, but is probably
worth fixing anyway.


diff --git a/compr_rtime.c b/compr_rtime.c
index 131536c..7353024 100644
--- a/compr_rtime.c
+++ b/compr_rtime.c
@@ -32,7 +32,7 @@ static int jffs2_rtime_compress(unsigned char *data_in, unsigned char *cpage_out
 
 	memset(positions,0,sizeof(positions));
 
-	while (pos < (*sourcelen) && outpos <= (*dstlen)-2) {
+	while (pos < (*sourcelen) && outpos+2 <= *dstlen) {
 		int backpos, runlen=0;
 		unsigned char value;
 
diff --git a/compr_zlib.c b/compr_zlib.c
index 400b18a..eb415b9 100644
--- a/compr_zlib.c
+++ b/compr_zlib.c
@@ -71,7 +71,7 @@ int jffs2_zlib_compress(unsigned char *data_in, unsigned char *cpage_out,
 	strm.next_out = cpage_out;
 	strm.total_out = 0;
 
-	while (strm.total_out < *dstlen - STREAM_END_SPACE && strm.total_in < *sourcelen) {
+	while (strm.total_out + STREAM_END_SPACE < *dstlen && strm.total_in < *sourcelen) {
 		strm.avail_out = *dstlen - (strm.total_out + STREAM_END_SPACE);
 		strm.avail_in = min((unsigned)(*sourcelen-strm.total_in), strm.avail_out);
 		ret = deflate(&strm, Z_PARTIAL_FLUSH);