[{"id":1794175,"web_url":"http://patchwork.ozlabs.org/comment/1794175/","msgid":"","list_archive_url":null,"date":"2017-10-26T10:45:44","subject":"Re: [Qemu-devel] [PATCH 0/3] vfio/pci: Add NVIDIA GPUDirect P2P\n\tclique support","submitter":{"id":72655,"url":"http://patchwork.ozlabs.org/api/people/72655/","name":"Bob Chen","email":"a175818323@gmail.com"},"content":"There seem to be some bugs in these patches, causing my VM failed to boot.\n\nTest case:\n\n0. Merge these 3 patches in to release 2.10.1\n\n1. qemu-system-x86_64_2.10.1 ... \\\n-device vfio-pci,host=04:00.0 \\\n-device vfio-pci,host=05:00.0 \\\n-device vfio-pci,host=08:00.0 \\\n-device vfio-pci,host=09:00.0 \\\n-device vfio-pci,host=85:00.0 \\\n-device vfio-pci,host=86:00.0 \\\n-device vfio-pci,host=89:00.0 \\\n-device vfio-pci,host=8a:00.0 ...\n\nThe guest was able to boot up.\n\n2. qemu-system-x86_64_2.10.1 ... \\\n-device vfio-pci,host=04:00.0,x-nv-gpudirect-clique=0 \\\n-device vfio-pci,host=05:00.0,x-nv-gpudirect-clique=0 \\\n-device vfio-pci,host=08:00.0,x-nv-gpudirect-clique=0 \\\n-device vfio-pci,host=09:00.0,x-nv-gpudirect-clique=0 \\\n-device vfio-pci,host=85:00.0,x-nv-gpudirect-clique=8 \\\n-device vfio-pci,host=86:00.0,x-nv-gpudirect-clique=8 \\\n-device vfio-pci,host=89:00.0,x-nv-gpudirect-clique=8 \\\n-device vfio-pci,host=8a:00.0,x-nv-gpudirect-clique=8 \\\n\nHang. VNC couldn't connect.\n\n\nMy personal patch used to work, although it was done by straightforward\nhacking and not that friendly to read.\n\n--- a/hw/vfio/pci.c\n+++ b/hw/vfio/pci.c\n\n @@static int vfio_initfn(PCIDevice *pdev)\n\n vfio_add_emulated_long(vdev, 0xc8, 0x50080009, ~0);\n if (count < 4)\n {\n vfio_add_emulated_long(vdev, 0xcc, 0x00005032, ~0);\n }\n else\n {\n vfio_add_emulated_long(vdev, 0xcc, 0x00085032, ~0);\n }\n vfio_add_emulated_word(vdev, 0x78, 0xc810, ~0);\n\n2017-08-30 6:05 GMT+08:00 Alex Williamson :\n\n> NVIDIA has a specification for exposing a virtual vendor capability\n> which provides a hint to guest drivers as to which sets of GPUs can\n> support direct peer-to-peer DMA. Devices with the same clique ID are\n> expected to support this. The user can specify a clique ID for an\n> NVIDIA graphics device using the new vfio-pci x-nv-gpudirect-clique=\n> option, where valid clique IDs are a 4-bit integer. It's entirely the\n> user's responsibility to specify sets of devices for which P2P works\n> correctly and provides some benefit. This is only useful for DMA\n> between NVIDIA GPUs, therefore it's only useful to specify cliques\n> comprised of more than one GPU. Furthermore, this does not enable DMA\n> between VMs, there is no change to VM DMA mapping, this only exposes\n> hints about existing DMA paths to the guest driver. Thanks,\n>\n> Alex\n>\n> ---\n>\n> Alex Williamson (3):\n> vfio/pci: Do not unwind on error\n> vfio/pci: Add virtual capabilities quirk infrastructure\n> vfio/pci: Add NVIDIA GPUDirect Cliques support\n>\n>\n> hw/vfio/pci-quirks.c | 114 ++++++++++++++++++++++++++++++\n> ++++++++++++++++++++\n> hw/vfio/pci.c | 17 +++++++\n> hw/vfio/pci.h | 4 ++\n> 3 files changed, 133 insertions(+), 2 deletions(-)\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"DJU2Vl1r\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yN3Zr3LtQz9sBW\n\tfor ;\n\tThu, 26 Oct 2017 21:46:12 +1100 (AEDT)","from localhost ([::1]:52026 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1e7ffu-000638-JG\n\tfor incoming@patchwork.ozlabs.org; Thu, 26 Oct 2017 06:46:10 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:55356)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1e7ffa-000632-RM\n\tfor qemu-devel@nongnu.org; Thu, 26 Oct 2017 06:45:52 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1e7ffW-0008IT-Cl\n\tfor qemu-devel@nongnu.org; Thu, 26 Oct 2017 06:45:50 -0400","from mail-oi0-x242.google.com ([2607:f8b0:4003:c06::242]:43415)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1e7ffW-0008HY-59\n\tfor qemu-devel@nongnu.org; Thu, 26 Oct 2017 06:45:46 -0400","by mail-oi0-x242.google.com with SMTP id c77so4882725oig.0\n\tfor ; Thu, 26 Oct 2017 03:45:45 -0700 (PDT)","by 10.157.26.69 with HTTP; Thu, 26 Oct 2017 03:45:44 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=DPiLHM5NRZ5TaYOLfdZP0CHJOwRdxIKTJiYvYMdEjHU=;\n\tb=DJU2Vl1r3YXd7JK5iaexntytdf1f51o/+EdFXBwnsikx0oTRavWJ7UuE+qMJzxHnUt\n\tjuOv8D+53eeLMfcYPfofhCS3NUac/EATHvSCyFducgq3gXydwNNbfuXvImVvhB6eI/lV\n\tN6b+cJAKZ+9zD9NTcij/7KSYo5PQvsrHxI4hOrLHAoG/E38BAmfghh4eNZhL1zSJX8Ey\n\tpXdRCc3SSrgw7vtV7UkS1Rm/yfAxO9GmijGGCT1EwYdOktvTr+hCBTjygPxjJDHBl7bK\n\tfQmDgPIum/SWAqCqXgvpQanDb5LV0aSKkjOC9hgogpXANmG50+F2ycS3KzmMVfWCHQJz\n\tk0Wg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=DPiLHM5NRZ5TaYOLfdZP0CHJOwRdxIKTJiYvYMdEjHU=;\n\tb=MTIaDG4v3xuuP6+igS10ndn4JY6RAU3qDvbBQI6cd4TkRLNKM06u4q/9WqZnzO9y2A\n\tX3uBaxwUgM63jWzUmpf7EWkI3trUi5JsXAoKuw1U2xcC+w3XPiRziFP1K007XFTfSxph\n\tcGCFZLVfK3+cLJTiIW3Sh/eHIRdm8dioSl8LZtaBA83gGm/8cbBR6b52EpSpIMAPjS+j\n\t9UuD39OzKOOAqVjhCcemsNgNMycLr+icGl2+OPTN83qB3moSzRSNR62Coeyn0zcPjwPl\n\tu+ofdmKeZdFtyzdHkQwgnEULbqQeDk94zq0Q6IbVr6lMgIvwoUuYPHYeMpF/5zwwR1SJ\n\tMOIg==","X-Gm-Message-State":"AMCzsaU2J3vlDmvG6g9Gfk6yU5Edffu40qZYT8HFQwNia4+wZvxb04uK\n\ttc0zXbS67B+Q1NS6uLKLDWIH9jFkslpV9936seY=","X-Google-Smtp-Source":"ABhQp+Qs4WmJSV1RFeUXQ/pJ8HckU+4v+xNMnNsGEVB63EWrOJj3Mw+yMGRhLrqFZdpdZeiMLgBXyZMofYrF31FR8KQ=","X-Received":"by 10.202.106.19 with SMTP id f19mr2267606oic.50.1509014744932; \n\tThu, 26 Oct 2017 03:45:44 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170829214929.31136.21144.stgit@gimli.home>","References":"<20170829214929.31136.21144.stgit@gimli.home>","From":"Bob Chen ","Date":"Thu, 26 Oct 2017 18:45:44 +0800","Message-ID":"","To":"Alex Williamson ","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4003:c06::242","Content-Type":"text/plain; charset=\"UTF-8\"","X-Content-Filtered-By":"Mailman/MimeDel 2.1.21","Subject":"Re: [Qemu-devel] [PATCH 0/3] vfio/pci: Add NVIDIA GPUDirect P2P\n\tclique support","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"qemu-devel@nongnu.org","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1807237,"web_url":"http://patchwork.ozlabs.org/comment/1807237/","msgid":"","list_archive_url":null,"date":"2017-11-20T10:45:09","subject":"Re: [Qemu-devel] [PATCH 0/3] vfio/pci: Add NVIDIA GPUDirect P2P\n\tclique support","submitter":{"id":72655,"url":"http://patchwork.ozlabs.org/api/people/72655/","name":"Bob Chen","email":"a175818323@gmail.com"},"content":"It's a mistake, please ignore. This patch is able to work.\n\n2017-10-26 18:45 GMT+08:00 Bob Chen :\n\n> There seem to be some bugs in these patches, causing my VM failed to boot.\n>\n> Test case:\n>\n> 0. Merge these 3 patches in to release 2.10.1\n>\n> 1. qemu-system-x86_64_2.10.1 ... \\\n> -device vfio-pci,host=04:00.0 \\\n> -device vfio-pci,host=05:00.0 \\\n> -device vfio-pci,host=08:00.0 \\\n> -device vfio-pci,host=09:00.0 \\\n> -device vfio-pci,host=85:00.0 \\\n> -device vfio-pci,host=86:00.0 \\\n> -device vfio-pci,host=89:00.0 \\\n> -device vfio-pci,host=8a:00.0 ...\n>\n> The guest was able to boot up.\n>\n> 2. qemu-system-x86_64_2.10.1 ... \\\n> -device vfio-pci,host=04:00.0,x-nv-gpudirect-clique=0 \\\n> -device vfio-pci,host=05:00.0,x-nv-gpudirect-clique=0 \\\n> -device vfio-pci,host=08:00.0,x-nv-gpudirect-clique=0 \\\n> -device vfio-pci,host=09:00.0,x-nv-gpudirect-clique=0 \\\n> -device vfio-pci,host=85:00.0,x-nv-gpudirect-clique=8 \\\n> -device vfio-pci,host=86:00.0,x-nv-gpudirect-clique=8 \\\n> -device vfio-pci,host=89:00.0,x-nv-gpudirect-clique=8 \\\n> -device vfio-pci,host=8a:00.0,x-nv-gpudirect-clique=8 \\\n>\n> Hang. VNC couldn't connect.\n>\n>\n> My personal patch used to work, although it was done by straightforward\n> hacking and not that friendly to read.\n>\n> --- a/hw/vfio/pci.c\n> +++ b/hw/vfio/pci.c\n>\n> @@static int vfio_initfn(PCIDevice *pdev)\n>\n> vfio_add_emulated_long(vdev, 0xc8, 0x50080009, ~0);\n> if (count < 4)\n> {\n> vfio_add_emulated_long(vdev, 0xcc, 0x00005032, ~0);\n> }\n> else\n> {\n> vfio_add_emulated_long(vdev, 0xcc, 0x00085032, ~0);\n> }\n> vfio_add_emulated_word(vdev, 0x78, 0xc810, ~0);\n>\n> 2017-08-30 6:05 GMT+08:00 Alex Williamson :\n>\n>> NVIDIA has a specification for exposing a virtual vendor capability\n>> which provides a hint to guest drivers as to which sets of GPUs can\n>> support direct peer-to-peer DMA. Devices with the same clique ID are\n>> expected to support this. The user can specify a clique ID for an\n>> NVIDIA graphics device using the new vfio-pci x-nv-gpudirect-clique=\n>> option, where valid clique IDs are a 4-bit integer. It's entirely the\n>> user's responsibility to specify sets of devices for which P2P works\n>> correctly and provides some benefit. This is only useful for DMA\n>> between NVIDIA GPUs, therefore it's only useful to specify cliques\n>> comprised of more than one GPU. Furthermore, this does not enable DMA\n>> between VMs, there is no change to VM DMA mapping, this only exposes\n>> hints about existing DMA paths to the guest driver. Thanks,\n>>\n>> Alex\n>>\n>> ---\n>>\n>> Alex Williamson (3):\n>> vfio/pci: Do not unwind on error\n>> vfio/pci: Add virtual capabilities quirk infrastructure\n>> vfio/pci: Add NVIDIA GPUDirect Cliques support\n>>\n>>\n>> hw/vfio/pci-quirks.c | 114 ++++++++++++++++++++++++++++++\n>> ++++++++++++++++++++\n>> hw/vfio/pci.c | 17 +++++++\n>> hw/vfio/pci.h | 4 ++\n>> 3 files changed, 133 insertions(+), 2 deletions(-)\n>>\n>\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"Iu8xVK/m\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3ygQNh2KHFz9s03\n\tfor ;\n\tMon, 20 Nov 2017 21:45:37 +1100 (AEDT)","from localhost ([::1]:56538 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1eGja2-0005AN-58\n\tfor incoming@patchwork.ozlabs.org; Mon, 20 Nov 2017 05:45:34 -0500","from eggs.gnu.org ([2001:4830:134:3::10]:59776)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1eGjZg-00059k-F5\n\tfor qemu-devel@nongnu.org; Mon, 20 Nov 2017 05:45:13 -0500","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1eGjZf-0005TH-6w\n\tfor qemu-devel@nongnu.org; Mon, 20 Nov 2017 05:45:12 -0500","from mail-ot0-x241.google.com ([2607:f8b0:4003:c0f::241]:46451)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1eGjZe-0005Sz-Vg\n\tfor qemu-devel@nongnu.org; Mon, 20 Nov 2017 05:45:11 -0500","by mail-ot0-x241.google.com with SMTP id j29so7252166oth.13\n\tfor ; Mon, 20 Nov 2017 02:45:10 -0800 (PST)","by 10.157.37.122 with HTTP; Mon, 20 Nov 2017 02:45:09 -0800 (PST)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=iIpbpoU/MHygqIak1xr7a3tycikrJn7oiY8KCDiuMXU=;\n\tb=Iu8xVK/mp7BtDi7i9WUJiANy/XnEcgEezPdCCAyMlUhhz9kcFS9+HbA/doTjzFNb8f\n\t1XfCXSKOF1jcRnWZxrU+4kpRjAAS0Wn1MJP1W87U+m9ev088oAQHd8tG6LZZkTX24fci\n\tBALUCCGu2cByVX/9vPQsn1D5izVvG1e8yHLsvpsKL10zSX17dxQviCmYH80SIZV955B3\n\t7lSc+BzXbYdtaCBY7JjFxxE8VwCUeCt5Zi5Gmr9x2z94nJ9oG/eMzp/oesngmh6V5ltj\n\t+JOPhyey82d0vUn/CXJbEfehXTtj9OX5K0tonrKEUH5zZM+K9GXuxG54VURUqvamFypC\n\tS2kw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=iIpbpoU/MHygqIak1xr7a3tycikrJn7oiY8KCDiuMXU=;\n\tb=kZRMh67SEe8EAGr04JKiMu7WHKyPSUareHdJg7oT14Puh+aXCKAAih166AwzofLuAT\n\tIFaUQHtlvF6aFdEeRuMIAxMQCMlLVoXfsCslHgiCbdWW/MkhhWFoVQ1wEc2cF9/mK3mh\n\tJgK+Yv7cLeXY+A9MAEf2pKurJCQ1PIT/pz3ow4uBNeclJ04XWfQQLbUd9dJf8zFzbkU0\n\t3F2y82oMDLhr4j7hZgvMUMTEzi1t1zK6WwAXCtcINvwSom9z7ucBMEBu7xWDHrdyuTC6\n\t+H1G6uepZfVpv9tk8PtkhS4/YQiHauhjOqxZSpTqwWPYbXqy7QvE0LRSBLAJcX7rvTal\n\tKIuQ==","X-Gm-Message-State":"AJaThX5KQu3l/0OwzK+fBGol2PbfaVrr9majky0/V5d568Y53tG9mSC+\n\tueUsAziRvkav8yMQIEHSaJBthTyRvkf2hTmA+go=","X-Google-Smtp-Source":"AGs4zMas08bfdAfV9WjrGycLK/NexGPvbHRly4o2XzWH7QFEtny9/H/utyelJqIR3fJ60ESOi9R9n7f71ETNZxbe5zU=","X-Received":"by 10.157.92.197 with SMTP id r5mr8004379oti.286.1511174709703; \n\tMon, 20 Nov 2017 02:45:09 -0800 (PST)","MIME-Version":"1.0","In-Reply-To":"","References":"<20170829214929.31136.21144.stgit@gimli.home>\n\t","From":"Bob Chen ","Date":"Mon, 20 Nov 2017 18:45:09 +0800","Message-ID":"","To":"Alex Williamson ","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4003:c0f::241","Content-Type":"text/plain; charset=\"UTF-8\"","X-Content-Filtered-By":"Mailman/MimeDel 2.1.21","Subject":"Re: [Qemu-devel] [PATCH 0/3] vfio/pci: Add NVIDIA GPUDirect P2P\n\tclique support","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"qemu-devel@nongnu.org","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]