{"id":817751,"url":"http://patchwork.ozlabs.org/api/covers/817751/?format=json","web_url":"http://patchwork.ozlabs.org/project/linux-imx/cover/20170923004207.22356-1-cdall@linaro.org/","project":{"id":19,"url":"http://patchwork.ozlabs.org/api/projects/19/?format=json","name":"Linux IMX development","link_name":"linux-imx","list_id":"linux-imx-kernel.lists.patchwork.ozlabs.org","list_email":"linux-imx-kernel@lists.patchwork.ozlabs.org","web_url":null,"scm_url":null,"webscm_url":null,"list_archive_url":"","list_archive_url_format":"","commit_url_format":""},"msgid":"<20170923004207.22356-1-cdall@linaro.org>","list_archive_url":null,"date":"2017-09-23T00:41:47","name":"[v3,00/20] KVM: arm/arm64: Optimize arch timer register handling","submitter":{"id":71350,"url":"http://patchwork.ozlabs.org/api/people/71350/?format=json","name":"Christoffer Dall","email":"cdall@linaro.org"},"mbox":"http://patchwork.ozlabs.org/project/linux-imx/cover/20170923004207.22356-1-cdall@linaro.org/mbox/","series":[{"id":4729,"url":"http://patchwork.ozlabs.org/api/series/4729/?format=json","web_url":"http://patchwork.ozlabs.org/project/linux-imx/list/?series=4729","date":"2017-09-23T00:41:47","name":"KVM: arm/arm64: Optimize arch timer register handling","version":3,"mbox":"http://patchwork.ozlabs.org/series/4729/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/covers/817751/comments/","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org header.b=\"q0+FpNOl\"; \n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"RlpIFEEu\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xzWm44v2rz9t32\n\tfor ;\n\tSat, 23 Sep 2017 10:43:00 +1000 (AEST)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1dvYWr-0003xB-El; Sat, 23 Sep 2017 00:42:45 +0000","from mail-wm0-x22e.google.com ([2a00:1450:400c:c09::22e])\n\tby bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux))\n\tid 1dvYWZ-0003ja-H8 for linux-arm-kernel@lists.infradead.org;\n\tSat, 23 Sep 2017 00:42:39 +0000","by mail-wm0-x22e.google.com with SMTP id m72so7631011wmc.1\n\tfor ;\n\tFri, 22 Sep 2017 17:42:04 -0700 (PDT)","from localhost.localdomain (xd93dd96b.cust.hiper.dk.\n\t[217.61.217.107]) by smtp.gmail.com with ESMTPSA id\n\tc25sm646085edb.57.2017.09.22.17.42.01\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128);\n\tFri, 22 Sep 2017 17:42:02 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe:\n\tList-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date:\n\tSubject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date:\n\tResent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:\n\tReferences:List-Owner; bh=/Efx0H/Ns9y8FkN6wwf0wzz7kYwjCYpQ5gp1tsBo0Oc=;\n\tb=q0+\n\tFpNOlTJU/Za7gTY8VPlRjjIcTjUXILgDkvri+kYUHLcj/juO6a0wiR1zye23vvfdyiP1SwdK1vo1w\n\tN3QFGft3CGcOJmwwWoQjFOeBceWjNc99PAUI7mVsDYF1vg/gLHFcWzdciWA+P8ZQgqMTHCXxCsLyz\n\tXRUWm65rqRTr58l8NbhkEyJsm+T4zhie+NUGmJjv7GeXbjkHARytjElx78tmcPFRR7lpDXus2hPSI\n\tt9Ytou6WRlyoyeAs4YZDOeP4FcaW+Cs7eGghuqR6YrHF6kAOWXIav94zo1UlnSOGb6bp8PmYkr/3k\n\tXnWCeAxSL5+yDG7hQjiBReiFh4F6O6g==;","v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=from:to:cc:subject:date:message-id;\n\tbh=idIgOe65csJ6ndI83skA8Qv289I6nYmHmkvt3cYU9SQ=;\n\tb=RlpIFEEu3Y1GHNHlCExjwiOiUS/Z8EBP4u6ppPj73Yz5l0hv550bLlX3+cljSrRNkz\n\trkKQkoRa5q2LDJJuWdbNE9qcSCVLvEhfRi126QdT+Oa4675cAxU9qzxIph/BBorKKAvx\n\tVYxEKuebwpBDEIGskPsyx9QVTr1onmkAjM5jE="],"X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id;\n\tbh=idIgOe65csJ6ndI83skA8Qv289I6nYmHmkvt3cYU9SQ=;\n\tb=TDI7GF/OYsmvgxMFxuHA44rdjMmJJW/ZBMPQXxNRoWCr1BndiGef9maxx0g0vWmQos\n\txKiYJFXYwo1v9cxLOtqa/JiP1o6G7LXZPj5+GcSyPYf8v1QabJb2TiRIveztk9rJn1kl\n\tSqtAYLPYtcgIAGVlz1akw+er5pzS5tAQ8l2HOuGpjSXOKCRzem+6/YEvIgMVy7K2ZNJT\n\tdDhvy9YI8OkyBt7ZJM6faytXnfDzYjbn5p3kE1tJXrFv2sJ2tHuivX7sPnk7kfdenpDL\n\tnSM1BCk700RuTdEBqnDMBVhgR2+K3/ww2yKwNJutLF/pf5nQeSqzQchzgWQjesusxkb+\n\tWXQA==","X-Gm-Message-State":"AHPjjUioRPZ14xA9PmBDX/FdEEVxY/0S7njLuZ9S8ZFDVlbfsKPwRIyi\n\tpA6KdADXWANOBzqRnnxhe4pgaw==","X-Google-Smtp-Source":"AOwi7QAJpq6mFMPXBQaCYQVfLjWBHFNliJ1JzmDYljX0RKSsnJjKxG2IRHwO+DBNApxuZJGq7Pqdmw==","X-Received":"by 10.80.212.9 with SMTP id t9mr6644444edh.240.1506127323149;\n\tFri, 22 Sep 2017 17:42:03 -0700 (PDT)","From":"Christoffer Dall ","To":"kvmarm@lists.cs.columbia.edu,\n\tlinux-arm-kernel@lists.infradead.org","Subject":"[PATCH v3 00/20] KVM: arm/arm64: Optimize arch timer register\n\thandling","Date":"Sat, 23 Sep 2017 02:41:47 +0200","Message-Id":"<20170923004207.22356-1-cdall@linaro.org>","X-Mailer":"git-send-email 2.9.0","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20170922_174228_176411_DBB2BF6B ","X-CRM114-Status":"GOOD ( 17.47 )","X-Spam-Score":"-2.7 (--)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-2.7 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/,\n\tlow\n\ttrust [2a00:1450:400c:c09:0:0:0:22e listed in] [list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]\n\t-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature\n\t0.1 DKIM_SIGNED Message has a DKIM or DK signature,\n\tnot necessarily valid\n\t-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from\n\tauthor's domain","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Marc Zyngier ,\n\tCatalin Marinas ,\n\tWill Deacon , \n\tkvm@vger.kernel.org, Christoffer Dall ","MIME-Version":"1.0","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"},"content":"We currently spend a measurable amount of time on each entry/exit to the\nguest dealing with arch timer registers, even when the timer is not\npending and not doing anything (on certain architectures).\n\nWe can do much better by moving the arch timer save/restore to the\nvcpu_load and vcpu_put functions, but this means that if we don't read\nback the timer state on every exit from the guest, then we have to be\nable to start taking timer interrupts for the virtual timer in KVM and\nhandle that properly.\n\nThat has a number of entertaining consequences, such as having to make\nsure we don't deadlock between any of the vgic code and interrupt\ninjection happening from an ISR. On the plus side, being able to inject\nvirtual interrupts corresponding to a physical interrupt directly from\nan ISR is probably a good system design change overall.\n\nWe also have to change the use of the physical vs. virtual counter in\nthe arm64 kernel to avoid having to save/restore the CNTVOFF_EL2\nregister on every return to the hypervisor. The only reason I could\nfind for using the virtual counter for the kernel on systems with access\nto the physical counter is to detect if firmware did not properly clear\nCNTVOFF_EL2, and this change has to weighed against the existing check\n(assuming I got this right).\n\nOn a non-VHE system (AMD Seattle) I have measured this to improve the\nworld-switch time by about ~100 cycles, but on an EL2 kernel (emulating\nVHE behavior on the same hardware) this gives us around ~250 cycles\nworth of improvement, and on Thunder-X we seem to get ~650 cycles\nimprovement, because we can avoid the extra configuration of trapping\naccesses to the physical timer from EL1 on every switch.\n\nThese patches require that the GICv2 hardware (on such systems) is\nproperly reported by firmware to have the extra CPU interface page for\nthe deactivate register.\n\nCode is also available here:\ngit://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git timer-optimize-v3\n\nBased on v4.14-rc1\n\nSome recent numbers I ran on Thunder-X with v4.14-rc1 with/without these\npatches (averaged over a few hundred thousand executions) for a base\nhypercall cost:\n\nWithout this series, avg. cycles: 12,476\nWithout this series, min. cycles: 12,052\n\nWith this series, avg. cycles: 11,782\nWith this series, min. cycles: 11,435\n\nImprovement ~650 cycles (over 5%)\n\nChanges since v2:\n - Removed RFC tag\n - Included Marc's patch to support EOI/deactivate on broken firmware\n systems\n - Simplified patch 6 (was patch 5 in RFC v2)\n - Clarify percpu_devid interrupts in patch 12 (was patch 11)\n\nThanks,\n Christoffer\n\nChristoffer Dall (19):\n arm64: Use physical counter for in-kernel reads\n arm64: Use the physical counter when available for read_cycles\n KVM: arm/arm64: Guard kvm_vgic_map_is_active against !vgic_initialized\n KVM: arm/arm64: Support calling vgic_update_irq_pending from irq\n context\n KVM: arm/arm64: Check that system supports split eoi/deactivate\n KVM: arm/arm64: Make timer_arm and timer_disarm helpers more generic\n KVM: arm/arm64: Rename soft timer to bg_timer\n KVM: arm/arm64: Use separate timer for phys timer emulation\n KVM: arm/arm64: Move timer/vgic flush/sync under disabled irq\n KVM: arm/arm64: Move timer save/restore out of the hyp code\n genirq: Document vcpu_info usage for percpu_devid interrupts\n KVM: arm/arm64: Set VCPU affinity for virt timer irq\n KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit\n KVM: arm/arm64: Support EL1 phys timer register access in set/get reg\n KVM: arm/arm64: Use kvm_arm_timer_set/get_reg for guest register traps\n KVM: arm/arm64: Move phys_timer_emulate function\n KVM: arm/arm64: Avoid phys timer emulation in vcpu entry/exit\n KVM: arm/arm64: Get rid of kvm_timer_flush_hwstate\n KVM: arm/arm64: Rework kvm_timer_should_fire\n\nMarc Zyngier (1):\n irqchip/gic: Deal with broken firmware exposing only 4kB of GICv2 CPU\n interface\n\n Documentation/admin-guide/kernel-parameters.txt | 7 +\n arch/arm/include/asm/kvm_asm.h | 2 +\n arch/arm/include/asm/kvm_hyp.h | 4 +-\n arch/arm/include/uapi/asm/kvm.h | 6 +\n arch/arm/kvm/hyp/switch.c | 7 +-\n arch/arm64/include/asm/arch_timer.h | 18 +-\n arch/arm64/include/asm/kvm_asm.h | 2 +\n arch/arm64/include/asm/kvm_hyp.h | 4 +-\n arch/arm64/include/asm/timex.h | 2 +-\n arch/arm64/include/uapi/asm/kvm.h | 6 +\n arch/arm64/kvm/hyp/switch.c | 6 +-\n arch/arm64/kvm/sys_regs.c | 41 +--\n drivers/clocksource/arm_arch_timer.c | 33 +-\n drivers/irqchip/irq-gic.c | 74 +++-\n include/kvm/arm_arch_timer.h | 19 +-\n kernel/irq/manage.c | 3 +-\n virt/kvm/arm/arch_timer.c | 446 ++++++++++++++++--------\n virt/kvm/arm/arm.c | 45 ++-\n virt/kvm/arm/hyp/timer-sr.c | 74 ++--\n virt/kvm/arm/vgic/vgic-its.c | 17 +-\n virt/kvm/arm/vgic/vgic-mmio-v2.c | 22 +-\n virt/kvm/arm/vgic/vgic-mmio-v3.c | 17 +-\n virt/kvm/arm/vgic/vgic-mmio.c | 44 ++-\n virt/kvm/arm/vgic/vgic-v2.c | 5 +-\n virt/kvm/arm/vgic/vgic-v3.c | 12 +-\n virt/kvm/arm/vgic/vgic.c | 63 ++--\n virt/kvm/arm/vgic/vgic.h | 3 +-\n 27 files changed, 648 insertions(+), 334 deletions(-)"}