From patchwork Fri Jan 27 04:42:28 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Korolev X-Patchwork-Id: 138122 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [140.186.70.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id CEA59B6EE7 for ; Fri, 27 Jan 2012 15:42:50 +1100 (EST) Received: from localhost ([::1]:46000 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RqdeB-000246-6L for incoming@patchwork.ozlabs.org; Thu, 26 Jan 2012 23:42:47 -0500 Received: from eggs.gnu.org ([140.186.70.92]:56701) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Rqde3-000241-0m for qemu-devel@nongnu.org; Thu, 26 Jan 2012 23:42:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Rqde1-0007Yn-S0 for qemu-devel@nongnu.org; Thu, 26 Jan 2012 23:42:38 -0500 Received: from usrksweb02.endace.com ([174.143.168.194]:45649) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Rqde1-0007Yg-KQ for qemu-devel@nongnu.org; Thu, 26 Jan 2012 23:42:37 -0500 Received: from mail.et.endace.com ([131.203.121.41]) by usrksweb02.endace.com (8.14.3/8.14.3/Debian-9.2ubuntu1) with ESMTP id q0R4bSBf003117; Fri, 27 Jan 2012 04:37:30 GMT Received: from [192.168.69.208] (192.168.69.208) by nzhmlmbx01.ad.endace.com (192.168.32.5) with Microsoft SMTP Server (TLS) id 14.1.218.12; Fri, 27 Jan 2012 17:42:29 +1300 Message-ID: <4F222B34.103@endace.com> Date: Fri, 27 Jan 2012 17:42:28 +1300 From: Alexey Korolev User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20111229 Thunderbird/9.0 MIME-Version: 1.0 To: Avi Kivity References: <4F1F971B.4020309@endace.com> <20120126091436.GB13974@redhat.com> <4F215A9B.6090204@redhat.com> <20120126143626.GE17198@redhat.com> <4F216D4E.7030405@redhat.com> In-Reply-To: <4F216D4E.7030405@redhat.com> X-Originating-IP: [192.168.69.208] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 174.143.168.194 Cc: sfd@endace.com, Kevin O'Connor , "qemu-devel@nongnu.org" , "Michael S. Tsirkin" Subject: Re: [Qemu-devel] [RFC/PATCH] Fix guest OS panic when 64bit BAR is present X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org On 27/01/12 04:12, Avi Kivity wrote: > On 01/26/2012 04:36 PM, Michael S. Tsirkin wrote: >> On Thu, Jan 26, 2012 at 03:52:27PM +0200, Avi Kivity wrote: >>> On 01/26/2012 11:14 AM, Michael S. Tsirkin wrote: >>>> On Wed, Jan 25, 2012 at 06:46:03PM +1300, Alexey Korolev wrote: >>>>> Hi, >>>>> In this post >>>>> http://lists.gnu.org/archive/html/qemu-devel/2011-12/msg03171.html I've >>>>> mentioned about the issues when 64Bit PCI BAR is present and 32bit >>>>> address range is selected for it. >>>>> The issue affects all recent qemu releases and all >>>>> old and recent guest Linux kernel versions. >>>>> >>>>> We've done some investigations. Let me explain what happens. >>>>> Assume we have 64bit BAR with size 32MB mapped at [0xF0000000 - >>>>> 0xF2000000] >>>>> >>>>> When Linux guest starts it does PCI bus enumeration. >>>>> The OS enumerates 64BIT bars using the following procedure. >>>>> 1. Write all FF's to lower half of 64bit BAR >>>>> 2. Write address back to lower half of 64bit BAR >>>>> 3. Write all FF's to higher half of 64bit BAR >>>>> 4. Write address back to higher half of 64bit BAR >>>>> >>>>> Linux code is here: >>>>> http://lxr.linux.no/#linux+v3.2.1/drivers/pci/probe.c#L149 >>>>> >>>>> What does it mean for qemu? >>>>> >>>>> At step 1. qemu pci_default_write_config() recevies all FFs for lower >>>>> part of the 64bit BAR. Then it applies the mask and converts the value >>>>> to "All FF's - size + 1" (FE000000 if size is 32MB). >>>>> Then pci_bar_address() checks if BAR address is valid. Since it is a >>>>> 64bit bar it reads 0x00000000FE000000 - this address is valid. So qemu >>>>> updates topology and sends request to update mappings in KVM with new >>>>> range for the 64bit BAR FE000000 - 0xFFFFFFFF. This usually means kernel >>>>> panic on boot, if there is another mapping in the FE000000 - 0xFFFFFFFF >>>>> range, which is quite common. >>>> Do you know why does it panic? As far as I can see >>>> from code at >>>> http://lxr.linux.no/#linux+v2.6.35.9/drivers/pci/probe.c#L162 >>>> >>>> 171 pci_read_config_dword(dev, pos, &l); >>>> 172 pci_write_config_dword(dev, pos, l | mask); >>>> 173 pci_read_config_dword(dev, pos, &sz); >>>> 174 pci_write_config_dword(dev, pos, l); >>>> >>>> BAR is restored: what triggers an access between lines 172 and 174? >>> Random interrupt reading the time, likely. >> Weird, what the backtrace shows is init, unrelated >> to interrupts. >> > It's a bug then. qemu doesn't undo the mapping correctly. > > If you have clear instructions, I'll try to reproduce it. > Well the easiest way to reproduce this is: 1. Get kernel bzImage (version < 2.6.36) 2. Apply patch to ivshmem.c --- --- 3. Launch qemu with a command like that /usr/bin/qemu-system-x86_64 -M pc-0.14 -enable-kvm -m 2048 -smp 1,socket=1,cores=1,threads=1 -name centos54 -uuid d37daefd-75bd-4387-cee1-7f0b153ee2af -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos54.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -drive file=/dev/dock200-1/centos54,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/data/CentOS-5.4-x86_64-bin-DVD.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -chardev file,id=charserial0,path=/home/alexey/cent54.log -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x4.0x0 --device ivshmem,size=32,shm="shm" -kernel bzImage -append "root=/dev/hda1 console=ttyS0,115200n8 console=tty0" in other words add: --device ivshmem,size=32,shm="shm" That is all. Note: it won't necessary cause panic message on some kernels it just hangs or reboots. diff --git a/hw/ivshmem.c b/hw/ivshmem.c index 1aa9e3b..71f8c21 100644 --- a/hw/ivshmem.c +++ b/hw/ivshmem.c @@ -341,7 +341,7 @@ static void create_shared_memory_BAR(IVShmemState *s, int fd) { memory_region_add_subregion(&s->bar, 0, &s->ivshmem); /* region for shared memory */ - pci_register_bar(&s->dev, 2, PCI_BASE_ADDRESS_SPACE_MEMORY, &s->bar); + pci_register_bar(&s->dev, 2, PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64, &s->bar) } static void close_guest_eventfds(IVShmemState *s, int posn)