From patchwork Mon Mar 21 08:34:35 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Corentin Chary X-Patchwork-Id: 87720 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id D69C5B6F2B for ; Mon, 21 Mar 2011 19:41:15 +1100 (EST) Received: from localhost ([127.0.0.1]:54894 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Q1afn-0005R4-8p for incoming@patchwork.ozlabs.org; Mon, 21 Mar 2011 04:41:11 -0400 Received: from [140.186.70.92] (port=52968 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Q1aZf-0002Un-Dz for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:34:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q1aZc-0007QH-VP for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:34:50 -0400 Received: from smtp5.tech.numericable.fr ([82.216.111.41]:46731) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q1aZc-0007Pu-Mx for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:34:48 -0400 Received: from localhost.localdomain (ip-138.net-82-216-39.nice.rev.numericable.fr [82.216.39.138]) by smtp5.tech.numericable.fr (Postfix) with ESMTP id 3280712401C; Mon, 21 Mar 2011 09:34:47 +0100 (CET) From: Corentin Chary To: anthony@codemonkey.ws Date: Mon, 21 Mar 2011 09:34:35 +0100 Message-Id: <1300696478-6051-2-git-send-email-corentin.chary@gmail.com> X-Mailer: git-send-email 1.7.3.4 In-Reply-To: <1300696478-6051-1-git-send-email-corentin.chary@gmail.com> References: <1300696478-6051-1-git-send-email-corentin.chary@gmail.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Received-From: 82.216.111.41 Cc: Michael Tokarev , qemu-devel , Blue Swirl , Corentin Chary , Paolo Bonzini Subject: [Qemu-devel] [PATCH 1/4] vnc: tight: Fix crash after 2GB of output X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Michael Tokarev fix 2Gb integer overflow in in VNC tight and zlib encodings As found by Roland Dreier (excellent catch!), when amount of VNC compressed data produced by zlib and sent to client exceeds 2Gb, integer overflow occurs because currently, we calculate amount of data produced at each step by comparing saved total_out with new total_out, and total_out is something which grows without bounds. Compare it with previous avail_out instead of total_out, and leave total_out alone. The same code is used in vnc-enc-tight.c and vnc-enc-zlib.c, so fix both cases. There, there's no actual need to save previous_out value, since capacity-offset (which is how that value is calculated) stays the same so it can be recalculated again after call to deflate(), but whole thing becomes less readable this way. Reported-by: Roland Dreier Signed-off-by: Michael Tokarev Signed-off-by: Corentin Chary --- ui/vnc-enc-tight.c | 5 +++-- ui/vnc-enc-zlib.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index 2522936..87fdf35 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -868,8 +868,8 @@ static int tight_compress_data(VncState *vs, int stream_id, size_t bytes, zstream->avail_in = vs->tight.tight.offset; zstream->next_out = vs->tight.zlib.buffer + vs->tight.zlib.offset; zstream->avail_out = vs->tight.zlib.capacity - vs->tight.zlib.offset; + previous_out = zstream->avail_out; zstream->data_type = Z_BINARY; - previous_out = zstream->total_out; /* start encoding */ if (deflate(zstream, Z_SYNC_FLUSH) != Z_OK) { @@ -878,7 +878,8 @@ static int tight_compress_data(VncState *vs, int stream_id, size_t bytes, } vs->tight.zlib.offset = vs->tight.zlib.capacity - zstream->avail_out; - bytes = zstream->total_out - previous_out; + /* ...how much data has actually been produced by deflate() */ + bytes = previous_out - zstream->avail_out; tight_send_compact_size(vs, bytes); vnc_write(vs, vs->tight.zlib.buffer, bytes); diff --git a/ui/vnc-enc-zlib.c b/ui/vnc-enc-zlib.c index 3c6e6ab..e32e4cd 100644 --- a/ui/vnc-enc-zlib.c +++ b/ui/vnc-enc-zlib.c @@ -103,8 +103,8 @@ static int vnc_zlib_stop(VncState *vs) zstream->avail_in = vs->zlib.zlib.offset; zstream->next_out = vs->output.buffer + vs->output.offset; zstream->avail_out = vs->output.capacity - vs->output.offset; + previous_out = zstream->avail_out; zstream->data_type = Z_BINARY; - previous_out = zstream->total_out; // start encoding if (deflate(zstream, Z_SYNC_FLUSH) != Z_OK) { @@ -113,7 +113,7 @@ static int vnc_zlib_stop(VncState *vs) } vs->output.offset = vs->output.capacity - zstream->avail_out; - return zstream->total_out - previous_out; + return previous_out - zstream->avail_out; } int vnc_zlib_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)