From patchwork Mon Nov 13 08:16:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Balbir Singh X-Patchwork-Id: 837386 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb3RQ5W71z9sNw for ; Mon, 13 Nov 2017 19:17:54 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Tgz7K5vw"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb3RQ3TPCzDqYy for ; Mon, 13 Nov 2017 19:17:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Tgz7K5vw"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::244; helo=mail-pf0-x244.google.com; envelope-from=bsingharora@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Tgz7K5vw"; dkim-atps=neutral Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb3Pg5bvvzDqYw for ; Mon, 13 Nov 2017 19:16:23 +1100 (AEDT) Received: by mail-pf0-x244.google.com with SMTP id 17so11311408pfn.12 for ; Mon, 13 Nov 2017 00:16:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=f7lOPNjpSH12yK4G5FtWbRgPusqbxN2ZV+mM7sLQ9c8=; b=Tgz7K5vwR7U4jSBSqMzF0+KrYrhljyXE1AkFsvZi626DjJXxblaqaqiHPjUEoKbvwo OXXMi9/f6q1QUgR3t17MKRgVWnZJ+GPtA+C1zL4hqnNOTFoYt6hcQIdcGKDfcanC4md/ kO8V4oNWvU2nB3vJ/DukTrDnLU1n2gJ8w4/JmhOuw+m7g6kodporicyo+zMk4ys0X/Q+ B6M4epJKhxPtoTC5aV6U6q78WlWzRdXpr3bKn12h9hYy1xTTOPiG4t+fTmQxIJkwA+pA j4G3SliE9BHcQz/vsNIKMVOiJVPP0YTXCgGM6Vx9KzwHTXuR9AUi72Fx4FWXzmxg7p7s x21A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=f7lOPNjpSH12yK4G5FtWbRgPusqbxN2ZV+mM7sLQ9c8=; b=OSMUwBD4rQeKzcAg5yC1gJKW9DPFY2SC5A4m9U3sX8EVqOJpyJy2KsGghaMe6C4yqg GI4Fx55W9AvWN2qwKQANOYBBwstRM77v7zS+L19waA9LHN77INYs3rrsjRKbc/9vTLVP e3kcZ/LfGfDgi3k2SVrtrqQb2b+DZEqbNZvYSyBAy/zNQCYvn+UtN9MOH7i/qLFgZEkv KxmChb18LF5ANWBwUwffbTYlBg33hjmhslYko6F1wA13qAXD68xGYbqLvKCVvia4sVdG vT9hwmrq1mCGkNciRK365LLQn1zTdpufGZ2m/XmMplNE+RjwPEd2Ialqa7UAPnJW16X0 0UzA== X-Gm-Message-State: AJaThX5zip2kjJoWwdpAobIwGEpZFnT74nFjRKkmLC/xz2po9CWuT+o4 dl47HF/LnP5QEVO7XhJ2GMk= X-Google-Smtp-Source: AGs4zMY3UQ5WeuT78Av3LiMSwecnLZJGOi8oAvcjV1IlMktBoiWBn44R/ynBYU2IVOjgqDDQK491+w== X-Received: by 10.99.119.134 with SMTP id s128mr7852444pgc.409.1510560980645; Mon, 13 Nov 2017 00:16:20 -0800 (PST) Received: from firefly.ozlabs.ibm.com ([122.99.82.10]) by smtp.googlemail.com with ESMTPSA id b19sm33598973pfd.182.2017.11.13.00.16.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 13 Nov 2017 00:16:19 -0800 (PST) From: Balbir Singh To: alistair@popple.id.au Subject: [PATCH] powerpc/npu: Cleanup MMIO ATSD flushing Date: Mon, 13 Nov 2017 19:16:11 +1100 Message-Id: <20171113081611.27374-1-bsingharora@gmail.com> X-Mailer: git-send-email 2.13.6 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev@lists.ozlabs.org, aneesh.kumar@linux.vnet.ibm.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" While reviewing the code I found that the flush assumes all pages are of mmu_linear_psize, which is not correct. The patch uses find_linux_pte to find the right page size and uses that for launching the ATSD invalidation. A new helper is added to abstract the invalidation from the various notifiers. The patch also cleans up a bit by removing AP size from PID flushes. Signed-off-by: Balbir Singh --- Changelog since RFC - Implement review comments from Aneesh arch/powerpc/platforms/powernv/npu-dma.c | 66 ++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 17 deletions(-) diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c index 2cb6cbea4b3b..bf12a023a8de 100644 --- a/arch/powerpc/platforms/powernv/npu-dma.c +++ b/arch/powerpc/platforms/powernv/npu-dma.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -27,6 +28,7 @@ #include #include #include +#include #include "powernv.h" #include "pci.h" @@ -459,9 +461,6 @@ static int mmio_invalidate_pid(struct npu *npu, unsigned long pid, bool flush) /* PRS set to process-scoped */ launch |= PPC_BIT(13); - /* AP */ - launch |= (u64) mmu_get_ap(mmu_virtual_psize) << PPC_BITLSHIFT(17); - /* PID */ launch |= pid << PPC_BITLSHIFT(38); @@ -473,7 +472,8 @@ static int mmio_invalidate_pid(struct npu *npu, unsigned long pid, bool flush) } static int mmio_invalidate_va(struct npu *npu, unsigned long va, - unsigned long pid, bool flush) + unsigned long pid, bool flush, + unsigned int shift) { unsigned long launch; @@ -484,9 +484,8 @@ static int mmio_invalidate_va(struct npu *npu, unsigned long va, launch |= PPC_BIT(13); /* AP */ - launch |= (u64) mmu_get_ap(mmu_virtual_psize) << PPC_BITLSHIFT(17); + launch |= (u64) mmu_get_ap(shift) << PPC_BITLSHIFT(17); - /* PID */ launch |= pid << PPC_BITLSHIFT(38); /* No flush */ @@ -503,7 +502,8 @@ struct mmio_atsd_reg { }; static void mmio_invalidate_wait( - struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS], bool flush) + struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS], bool flush, + unsigned int shift) { struct npu *npu; int i, reg; @@ -536,7 +536,8 @@ static void mmio_invalidate_wait( * the value of va. */ static void mmio_invalidate(struct npu_context *npu_context, int va, - unsigned long address, bool flush) + unsigned long address, bool flush, + unsigned int shift) { int i, j; struct npu *npu; @@ -569,7 +570,7 @@ static void mmio_invalidate(struct npu_context *npu_context, int va, if (va) mmio_atsd_reg[i].reg = mmio_invalidate_va(npu, address, pid, - flush); + flush, shift); else mmio_atsd_reg[i].reg = mmio_invalidate_pid(npu, pid, flush); @@ -582,10 +583,43 @@ static void mmio_invalidate(struct npu_context *npu_context, int va, } } - mmio_invalidate_wait(mmio_atsd_reg, flush); + mmio_invalidate_wait(mmio_atsd_reg, flush, shift); if (flush) /* Wait for the flush to complete */ - mmio_invalidate_wait(mmio_atsd_reg, false); + mmio_invalidate_wait(mmio_atsd_reg, false, shift); +} + +static void pnv_npu2_invalidate_helper(struct npu_context *npu_context, + struct mm_struct *mm, unsigned long start, + unsigned long end, bool flush) +{ + unsigned long address; + bool is_thp = false; + unsigned int hshift = 0, shift; + + address = start; + do { + local_irq_disable(); + find_linux_pte(mm->pgd, address, &is_thp, &hshift); + if (!is_thp) + shift = PAGE_SHIFT; + else if (hshift && !is_thp) + shift = hshift; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + else + shift = HPAGE_PMD_SIZE; +#else + else { + shift = PAGE_SHIFT; + pr_warn_once("unsupport page size for mm %p,addr %lx\n", + mm, start); + } +#endif + mmio_invalidate(npu_context, address > 0, address, flush, + shift); + local_irq_enable(); + address += (1ull << shift); + } while (address < end); } static void pnv_npu2_mn_release(struct mmu_notifier *mn, @@ -601,7 +635,7 @@ static void pnv_npu2_mn_release(struct mmu_notifier *mn, * There should be no more translation requests for this PID, but we * need to ensure any entries for it are removed from the TLB. */ - mmio_invalidate(npu_context, 0, 0, true); + pnv_npu2_invalidate_helper(npu_context, mm, 0, PAGE_SIZE, true); } static void pnv_npu2_mn_change_pte(struct mmu_notifier *mn, @@ -611,7 +645,7 @@ static void pnv_npu2_mn_change_pte(struct mmu_notifier *mn, { struct npu_context *npu_context = mn_to_npu_context(mn); - mmio_invalidate(npu_context, 1, address, true); + pnv_npu2_invalidate_helper(npu_context, mm, address, address, true); } static void pnv_npu2_mn_invalidate_range(struct mmu_notifier *mn, @@ -619,13 +653,11 @@ static void pnv_npu2_mn_invalidate_range(struct mmu_notifier *mn, unsigned long start, unsigned long end) { struct npu_context *npu_context = mn_to_npu_context(mn); - unsigned long address; - for (address = start; address < end; address += PAGE_SIZE) - mmio_invalidate(npu_context, 1, address, false); + pnv_npu2_invalidate_helper(npu_context, mm, start, end, false); /* Do the flush only on the final addess == end */ - mmio_invalidate(npu_context, 1, address, true); + pnv_npu2_invalidate_helper(npu_context, mm, end, end, true); } static const struct mmu_notifier_ops nv_nmmu_notifier_ops = {