From patchwork Thu Dec 17 21:02:23 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roland Dreier X-Patchwork-Id: 41369 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 6F28BB6F0C for ; Fri, 18 Dec 2009 12:56:06 +1100 (EST) Received: from localhost ([127.0.0.1]:37404 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NLS4Z-0007KP-FM for incoming@patchwork.ozlabs.org; Thu, 17 Dec 2009 20:56:03 -0500 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NLNUX-0006jt-EP for qemu-devel@nongnu.org; Thu, 17 Dec 2009 16:02:33 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1NLNUS-0006Wz-BJ for qemu-devel@nongnu.org; Thu, 17 Dec 2009 16:02:32 -0500 Received: from [199.232.76.173] (port=34669 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NLNUS-0006Wl-6c for qemu-devel@nongnu.org; Thu, 17 Dec 2009 16:02:28 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]:4862) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.60) (envelope-from ) id 1NLNUR-0004fA-Is for qemu-devel@nongnu.org; Thu, 17 Dec 2009 16:02:27 -0500 Authentication-Results: sj-iport-4.cisco.com; dkim=neutral (message not signed) header.i=none X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEAFMrKkurRN+J/2dsb2JhbAC/L5cthC0E X-IronPort-AV: E=Sophos;i="4.47,415,1257120000"; d="scan'208";a="64354785" Received: from sj-core-3.cisco.com ([171.68.223.137]) by sj-iport-4.cisco.com with ESMTP; 17 Dec 2009 21:02:24 +0000 Received: from xbh-sjc-231.amer.cisco.com (xbh-sjc-231.cisco.com [128.107.191.100]) by sj-core-3.cisco.com (8.13.8/8.14.3) with ESMTP id nBHL2O3b023594 for ; Thu, 17 Dec 2009 21:02:24 GMT Received: from xfe-sjc-212.amer.cisco.com ([171.70.151.187]) by xbh-sjc-231.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 17 Dec 2009 13:02:24 -0800 Received: from roland-alpha.cisco.com ([10.33.42.9]) by xfe-sjc-212.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 17 Dec 2009 13:02:24 -0800 Received: by roland-alpha.cisco.com (Postfix, from userid 33217) id 0B61A1FD0B; Thu, 17 Dec 2009 13:02:23 -0800 (PST) From: Roland Dreier To: qemu-devel@nongnu.org X-Message-Flag: Warning: May contain useful information Date: Thu, 17 Dec 2009 13:02:23 -0800 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) MIME-Version: 1.0 X-OriginalArrivalTime: 17 Dec 2009 21:02:24.0259 (UTC) FILETIME=[3B547530:01CA7F5C] X-detected-operating-system: by monty-python.gnu.org: Genre and OS details not recognized. X-Mailman-Approved-At: Thu, 17 Dec 2009 20:30:12 -0500 Subject: [Qemu-devel] [PATCH] vmware_vga: Don't crash on too-big DEFINE_CURSOR command X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org QEMU crashes with vmware_vga when running a Linux guest with the latest X.org vmware video driver if QEMU is using SDL for video output. In this case, vmware_vga advertises cursor acceleration to the guest, and the crash comes when the guest does a DEFINE_CURSOR command with a 64x64 32bpp cursor. This request overruns the image[] array in struct vmsvga_cursor_definition_s and QEMU ends up segfaulting because of memory corruption caused by writing past the end of the array. Fix this by enlarging the image[] array to be able to hold 4096 32-bit pixels so we don't fail for the case of 64*64*32bpp, and also add error checking to avoid a crash if an even bigger request is sent by a guest. Signed-off-by: Roland Dreier --- hw/vmware_vga.c | 11 ++++++++++- 1 files changed, 10 insertions(+), 1 deletions(-) diff --git a/hw/vmware_vga.c b/hw/vmware_vga.c index f3e3749..d253a2e 100644 --- a/hw/vmware_vga.c +++ b/hw/vmware_vga.c @@ -462,7 +462,7 @@ struct vmsvga_cursor_definition_s { int hot_x; int hot_y; uint32_t mask[1024]; - uint32_t image[1024]; + uint32_t image[4096]; /* allow for 64x64 32bpp cursor */ }; #define SVGA_BITMAP_SIZE(w, h) ((((w) + 31) >> 5) * (h)) @@ -557,6 +557,15 @@ static void vmsvga_fifo_run(struct vmsvga_state_s *s) cursor.height = y = vmsvga_fifo_read(s); vmsvga_fifo_read(s); cursor.bpp = vmsvga_fifo_read(s); + + if (SVGA_BITMAP_SIZE(x, y) > sizeof cursor.mask / sizeof (uint32_t) || + SVGA_PIXMAP_SIZE(x, y, cursor.bpp) > sizeof cursor.image / sizeof (uint32_t)) { + fprintf(stderr, "%s: DEFINE_CSURSOR too large x: %d, y: %d bpp: %d\n", + __FUNCTION__, x, y, cursor.bpp); + args = SVGA_BITMAP_SIZE(x, y) + SVGA_PIXMAP_SIZE(x, y, cursor.bpp); + goto badcmd; + } + for (args = 0; args < SVGA_BITMAP_SIZE(x, y); args ++) cursor.mask[args] = vmsvga_fifo_read_raw(s); for (args = 0; args < SVGA_PIXMAP_SIZE(x, y, cursor.bpp); args ++)