From patchwork Thu Nov 8 23:55:16 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: BALATON Zoltan X-Patchwork-Id: 197904 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 27FAD2C00F3 for ; Fri, 9 Nov 2012 10:55:37 +1100 (EST) Received: from localhost ([::1]:52827 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TWbwc-0002Vy-30 for incoming@patchwork.ozlabs.org; Thu, 08 Nov 2012 18:55:34 -0500 Received: from eggs.gnu.org ([208.118.235.92]:49588) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TWbwV-0002Vr-C9 for qemu-devel@nongnu.org; Thu, 08 Nov 2012 18:55:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TWbwU-0003BM-2x for qemu-devel@nongnu.org; Thu, 08 Nov 2012 18:55:27 -0500 Received: from mono.eik.bme.hu ([152.66.115.2]:45458) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TWbwT-00039q-Pj for qemu-devel@nongnu.org; Thu, 08 Nov 2012 18:55:26 -0500 Received: from localhost (localhost [127.0.0.1]) by mono.eik.bme.hu (Postfix) with ESMTP id 90F09AF1; Fri, 9 Nov 2012 00:55:17 +0100 (CET) X-Virus-Scanned: amavisd-new at eik.bme.hu Received: from mono.eik.bme.hu ([127.0.0.1]) by localhost (mono.eik.bme.hu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id u-mEFXTmEhlD; Fri, 9 Nov 2012 00:55:17 +0100 (CET) Received: by mono.eik.bme.hu (Postfix, from userid 432) id 23EDE25D; Fri, 9 Nov 2012 00:55:17 +0100 (CET) Date: Fri, 9 Nov 2012 00:55:16 +0100 (CET) From: BALATON Zoltan X-X-Sender: balaton@mono To: Gerd Hoffmann In-Reply-To: <509C1F16.2090808@redhat.com> Message-ID: References: <509C1F16.2090808@redhat.com> User-Agent: Alpine 2.00 (GSO 1167 2008-08-23) MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Solaris 10 X-Received-From: 152.66.115.2 Cc: Gerhard Wiesinger , Peter Maydell , Anthony Liguori , qemu-devel@nongnu.org Subject: Re: [Qemu-devel] [PATCH] ui/vnc.c: Fix crash with VNC X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org On Thu, 8 Nov 2012, Gerd Hoffmann wrote: >> I think this is fixing this at the wrong level. Either we >> should require that drivers (in this case vmware_vga.c) >> must not call dpy_gfx_update() with out of range values, >> or we should do the clipping in the console.c layer, but >> I don't think requiring every UI backend to clip is the >> right thing. Anthony? > > Agree. IMHO vmware_vga.c is at fault here and should be fixed. We can > add some asserts to console.[ch] to enforce this ... Would the attached patch help? Regards, BALATON Zoltan From e1ea12f3fa70298f630c0b829d0f304339ca9799 Mon Sep 17 00:00:00 2001 From: BALATON Zoltan Date: Fri, 9 Nov 2012 00:44:29 +0100 Subject: [PATCH 2/2] vmware_vga: Clip updates with negative out of range rects to visible area Added checks and clipping also for negative out of range values in update rects which have been seen to happen at least with VNC under Windows NT 4.0 when a window is outside the visible area. Signed-off-by: BALATON Zoltan --- hw/vmware_vga.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) -- 1.7.10 diff --git a/hw/vmware_vga.c b/hw/vmware_vga.c index 834588d..e59ab3a 100644 --- a/hw/vmware_vga.c +++ b/hw/vmware_vga.c @@ -296,6 +296,14 @@ static inline void vmsvga_update_rect(struct vmsvga_state_s *s, uint8_t *src; uint8_t *dst; + if (x < 0 || x + w < 0) { + fprintf(stderr, "%s: update negative x position: %d, w: %d\n", + __func__, x, w); + w -= x; + x = MAX(x, 0); + y = MAX(w, 0); + } + if (x + w > ds_get_width(s->vga.ds)) { fprintf(stderr, "%s: update width too large x: %d, w: %d\n", __func__, x, w); @@ -303,6 +311,14 @@ static inline void vmsvga_update_rect(struct vmsvga_state_s *s, w = ds_get_width(s->vga.ds) - x; } + if (y < 0 || y + h < 0) { + fprintf(stderr, "%s: update negative y position: %d, h: %d\n", + __func__, y, h); + h -= y; + y = MAX(y, 0); + h = MAX(h, 0); + } + if (y + h > ds_get_height(s->vga.ds)) { fprintf(stderr, "%s: update height too large y: %d, h: %d\n", __func__, y, h);