From patchwork Wed Jan 23 22:56:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030218 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="Qs/oOYay"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLMl6ZcQz9s7h for ; Thu, 24 Jan 2019 09:59:47 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43182 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRUn-0007P9-Fi for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 17:59:45 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40844) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSJ-0005mL-N1 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSI-00046c-Op for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:11 -0500 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:43864) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSI-00045h-Ip for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:10 -0500 Received: by mail-pl1-x641.google.com with SMTP id gn14so1897079plb.10 for ; Wed, 23 Jan 2019 14:57:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AGAWWFtB255WL3QD01Pm/ADB1O2A0MpRDbzf5uqk7D0=; b=Qs/oOYayVKdoLK4L/1M5G4Pl8zkA7GZGyO5vngVJqdlC5o2nbh4EAePtXxomAkzZsW BBtQJPdRve8fTfl9x3OiuID9+B+vcC6b5qpoOqxQpn3V6YbwP7JK7GtsF8zrZKtqlTF1 N4gUBDDGvzdgwaIn54lfYi5XeFGrVOr8Kxs0o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AGAWWFtB255WL3QD01Pm/ADB1O2A0MpRDbzf5uqk7D0=; b=Qak+EPArHZxfXTEKyh4QFXpOBg8gzFK/byDRaTxFtr6GIf42kQVNSHVE7f9Nt8J74R zV3F/0cgmLjnwHH+0q+JrjsZEOtH1DVJX00OjCp+wW/qvGv8lZBDLNAfvXEG0RNXrzzm 7CiiU0Z/h7ebjBDMxmjqlijMFkujayk9vv/Ury9yUobKMUgAoxeOOuElV9QIXfAvtsqr N9ecDVdD0mStdqtZS5IKZT+ILmeRslbR/d3/yL7bQe1f6rsOuH+OxL6T0cwIjpu7CF5w 2fx5fhm7dOJKxRotUcNnNSqIRxc2ihUN1uWM8yn/QTFTh6J/tWo7FVQXDzQVAVXHheSW SqsQ== X-Gm-Message-State: AJcUukf8+YPAuhESkejL0B8jV6xW6AEnEb4WzdBWpFYNJiuiB9PZW6iK B2XF8465IRtH2zjxrur61Eikh8g6Hjc= X-Google-Smtp-Source: ALg8bN6BdUSpgzMjgjhiXS6iluaJiiqizPXxGsrtUqrM+gqhk2wN+m3cNaDGbxN3vZ4G9pogr2Caew== X-Received: by 2002:a17:902:aa0a:: with SMTP id be10mr4117258plb.266.1548284229177; Wed, 23 Jan 2019 14:57:09 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:08 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:53 -0800 Message-Id: <20190123225705.28963-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::641 Subject: [Qemu-devel] [PATCH 01/13] cputlb: do not evict empty entries to the vtlb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Currently we evict an entry to the victim TLB when it doesn't match the current address. But it could be that there's no match because the current entry is empty (i.e. all -1's, for instance via tlb_flush). Do not evict the entry to the vtlb in that case. This change will help us keep track of the TLB's use rate, which we'll use to implement a policy for dynamic TLB sizing. Tested-by: Alex Bennée Reviewed-by: Alex Bennée Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Message-Id: <20190116170114.26802-2-cota@braap.org> Signed-off-by: Richard Henderson --- accel/tcg/cputlb.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index af6bd8ccf9..10f1150c62 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -224,6 +224,15 @@ static inline bool tlb_hit_page_anyprot(CPUTLBEntry *tlb_entry, tlb_hit_page(tlb_entry->addr_code, page); } +/** + * tlb_entry_is_empty - return true if the entry is not in use + * @te: pointer to CPUTLBEntry + */ +static inline bool tlb_entry_is_empty(const CPUTLBEntry *te) +{ + return te->addr_read == -1 && te->addr_write == -1 && te->addr_code == -1; +} + /* Called with tlb_c.lock held */ static inline void tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, target_ulong page) @@ -591,7 +600,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, * Only evict the old entry to the victim tlb if it's for a * different page; otherwise just overwrite the stale data. */ - if (!tlb_hit_page_anyprot(te, vaddr_page)) { + if (!tlb_hit_page_anyprot(te, vaddr_page) && !tlb_entry_is_empty(te)) { unsigned vidx = env->tlb_d[mmu_idx].vindex++ % CPU_VTLB_SIZE; CPUTLBEntry *tv = &env->tlb_v_table[mmu_idx][vidx]; From patchwork Wed Jan 23 22:56:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030222 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="TEyhRjJk"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLRV5fJZz9sCX for ; Thu, 24 Jan 2019 10:03:01 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43242 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRXp-0001S3-6E for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:02:53 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40889) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSP-0005qq-PN for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSK-00048N-NZ for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:15 -0500 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:46355) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSK-00047A-8b for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:12 -0500 Received: by mail-pg1-x544.google.com with SMTP id w7so1732559pgp.13 for ; Wed, 23 Jan 2019 14:57:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HLjl+n4SV78hr3c2b29Cbyc0sFVyUwvUdC3hlSZ/Kjs=; b=TEyhRjJklHKAP/h92wF+H0OqHLPJp0qVb8qfzb8xks5J0LMS72tIdXv528GXQgfJog yLbBi3WdEIbQSOEsAHAO2yU44gPPTHUxN0d2w5NTwxy131MFN9Stj+GPGlMXrOvhhMGq xthVujmimh45C1rKwrjBnYaPTsKXpDyzlVa5Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HLjl+n4SV78hr3c2b29Cbyc0sFVyUwvUdC3hlSZ/Kjs=; b=DNs4jNVzMxBEYrfTWiI+2+m20HC1nhcOEj2Wweui9iWe1cDGxThhriz87+hhsmIBwj 9XfJtWi+MfAa2dJ98kVa8rjkHHOVSE0lSQtM+LMUB1ARokQg/M1IxLhh8WzfAht3bUx0 Caaz8CZe0eNgXQKc4obwrqrpwcaG655OXTJetyiNhX54HnZ6Q1y9LeHKh40RRKfdVM0G 10mDpEpLjnvmFhHwmb6VjbjAt5IVkZfKTzz9hVCy4A0I1HPuJ6OghKhr2VcD0xzQtC5F FTg3+aL17naIRWCg3F70d1YMuLWVkfjpa3d6mSx4LAxjmyPyrKviUOC44NAvVFLGq1H2 M3DA== X-Gm-Message-State: AJcUukdgIhdLGI2wCfKNoKWEfKmYnU6rR/IfNHZa+q8OwUMxGfQWyEzj P6EZ0eCP14zdGEwA2Ju33ZUoNehiq4k= X-Google-Smtp-Source: ALg8bN4PSrcw20L8RkGXGQgniETX9EROyHIzWIwss3wqJnLPJ0guNoJrn7Yq+xpwHRW5rOYG0p1XIQ== X-Received: by 2002:a63:1766:: with SMTP id 38mr3664856pgx.299.1548284230463; Wed, 23 Jan 2019 14:57:10 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:09 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:54 -0800 Message-Id: <20190123225705.28963-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 Subject: [Qemu-devel] [PATCH 02/13] tcg: introduce dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Disabled in all TCG backends for now. Tested-by: Alex Bennée Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Message-Id: <20190116170114.26802-3-cota@braap.org> Signed-off-by: Richard Henderson --- include/exec/cpu-defs.h | 57 ++++++++++- include/exec/cpu_ldst.h | 21 ++++ tcg/aarch64/tcg-target.h | 1 + tcg/arm/tcg-target.h | 1 + tcg/i386/tcg-target.h | 1 + tcg/mips/tcg-target.h | 1 + tcg/ppc/tcg-target.h | 1 + tcg/riscv/tcg-target.h | 1 + tcg/s390/tcg-target.h | 1 + tcg/sparc/tcg-target.h | 1 + tcg/tci/tcg-target.h | 1 + accel/tcg/cputlb.c | 202 ++++++++++++++++++++++++++++++++++++++- 12 files changed, 282 insertions(+), 7 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index 6a60f94a41..191a1e021f 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -67,6 +67,28 @@ typedef uint64_t target_ulong; #define CPU_TLB_ENTRY_BITS 5 #endif +#if TCG_TARGET_IMPLEMENTS_DYN_TLB +#define CPU_TLB_DYN_MIN_BITS 6 +#define CPU_TLB_DYN_DEFAULT_BITS 8 + + +# if HOST_LONG_BITS == 32 +/* Make sure we do not require a double-word shift for the TLB load */ +# define CPU_TLB_DYN_MAX_BITS (32 - TARGET_PAGE_BITS) +# else /* HOST_LONG_BITS == 64 */ +/* + * Assuming TARGET_PAGE_BITS==12, with 2**22 entries we can cover 2**(22+12) == + * 2**34 == 16G of address space. This is roughly what one would expect a + * TLB to cover in a modern (as of 2018) x86_64 CPU. For instance, Intel + * Skylake's Level-2 STLB has 16 1G entries. + * Also, make sure we do not size the TLB past the guest's address space. + */ +# define CPU_TLB_DYN_MAX_BITS \ + MIN(22, TARGET_VIRT_ADDR_SPACE_BITS - TARGET_PAGE_BITS) +# endif + +#else /* !TCG_TARGET_IMPLEMENTS_DYN_TLB */ + /* TCG_TARGET_TLB_DISPLACEMENT_BITS is used in CPU_TLB_BITS to ensure that * the TLB is not unnecessarily small, but still small enough for the * TLB lookup instruction sequence used by the TCG target. @@ -98,6 +120,7 @@ typedef uint64_t target_ulong; NB_MMU_MODES <= 8 ? 3 : 4)) #define CPU_TLB_SIZE (1 << CPU_TLB_BITS) +#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ typedef struct CPUTLBEntry { /* bit TARGET_LONG_BITS to TARGET_PAGE_BITS : virtual address @@ -141,6 +164,18 @@ typedef struct CPUIOTLBEntry { MemTxAttrs attrs; } CPUIOTLBEntry; +/** + * struct CPUTLBWindow + * @begin_ns: host time (in ns) at the beginning of the time window + * @max_entries: maximum number of entries observed in the window + * + * See also: tlb_mmu_resize_locked() + */ +typedef struct CPUTLBWindow { + int64_t begin_ns; + size_t max_entries; +} CPUTLBWindow; + typedef struct CPUTLBDesc { /* * Describe a region covering all of the large pages allocated @@ -152,6 +187,10 @@ typedef struct CPUTLBDesc { target_ulong large_page_mask; /* The next index to use in the tlb victim table. */ size_t vindex; +#if TCG_TARGET_IMPLEMENTS_DYN_TLB + CPUTLBWindow window; + size_t n_used_entries; +#endif } CPUTLBDesc; /* @@ -176,6 +215,20 @@ typedef struct CPUTLBCommon { size_t elide_flush_count; } CPUTLBCommon; +#if TCG_TARGET_IMPLEMENTS_DYN_TLB +# define CPU_TLB \ + /* tlb_mask[i] contains (n_entries - 1) << CPU_TLB_ENTRY_BITS */ \ + uintptr_t tlb_mask[NB_MMU_MODES]; \ + CPUTLBEntry *tlb_table[NB_MMU_MODES]; +# define CPU_IOTLB \ + CPUIOTLBEntry *iotlb[NB_MMU_MODES]; +#else +# define CPU_TLB \ + CPUTLBEntry tlb_table[NB_MMU_MODES][CPU_TLB_SIZE]; +# define CPU_IOTLB \ + CPUIOTLBEntry iotlb[NB_MMU_MODES][CPU_TLB_SIZE]; +#endif + /* * The meaning of each of the MMU modes is defined in the target code. * Note that NB_MMU_MODES is not yet defined; we can only reference it @@ -184,9 +237,9 @@ typedef struct CPUTLBCommon { #define CPU_COMMON_TLB \ CPUTLBCommon tlb_c; \ CPUTLBDesc tlb_d[NB_MMU_MODES]; \ - CPUTLBEntry tlb_table[NB_MMU_MODES][CPU_TLB_SIZE]; \ + CPU_TLB \ CPUTLBEntry tlb_v_table[NB_MMU_MODES][CPU_VTLB_SIZE]; \ - CPUIOTLBEntry iotlb[NB_MMU_MODES][CPU_TLB_SIZE]; \ + CPU_IOTLB \ CPUIOTLBEntry iotlb_v[NB_MMU_MODES][CPU_VTLB_SIZE]; #else diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 959068495a..83b2907d86 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -135,6 +135,21 @@ static inline target_ulong tlb_addr_write(const CPUTLBEntry *entry) #endif } +#if TCG_TARGET_IMPLEMENTS_DYN_TLB +/* Find the TLB index corresponding to the mmu_idx + address pair. */ +static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, + target_ulong addr) +{ + uintptr_t size_mask = env->tlb_mask[mmu_idx] >> CPU_TLB_ENTRY_BITS; + + return (addr >> TARGET_PAGE_BITS) & size_mask; +} + +static inline size_t tlb_n_entries(CPUArchState *env, uintptr_t mmu_idx) +{ + return (env->tlb_mask[mmu_idx] >> CPU_TLB_ENTRY_BITS) + 1; +} +#else /* Find the TLB index corresponding to the mmu_idx + address pair. */ static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, target_ulong addr) @@ -142,6 +157,12 @@ static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, return (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); } +static inline size_t tlb_n_entries(CPUArchState *env, uintptr_t mmu_idx) +{ + return CPU_TLB_SIZE; +} +#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ + /* Find the TLB entry corresponding to the mmu_idx + address pair. */ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, target_ulong addr) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 2d93cf404e..68868a27eb 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -15,6 +15,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 24 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #undef TCG_TARGET_STACK_GROWSUP typedef enum { diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h index 16172f73a3..c5a7064bdc 100644 --- a/tcg/arm/tcg-target.h +++ b/tcg/arm/tcg-target.h @@ -60,6 +60,7 @@ extern int arm_arch; #undef TCG_TARGET_STACK_GROWSUP #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 typedef enum { TCG_REG_R0 = 0, diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 7995fe3eab..94aa4cef7c 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -27,6 +27,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 31 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #ifdef __x86_64__ # define TCG_TARGET_REG_BITS 64 diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h index 5cb8672470..8600eefd9a 100644 --- a/tcg/mips/tcg-target.h +++ b/tcg/mips/tcg-target.h @@ -37,6 +37,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index 52c1bb04b1..b51854b5cf 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -34,6 +34,7 @@ #define TCG_TARGET_NB_REGS 32 #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 typedef enum { TCG_REG_R0, TCG_REG_R1, TCG_REG_R2, TCG_REG_R3, diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h index 60918cacb4..1eb032626c 100644 --- a/tcg/riscv/tcg-target.h +++ b/tcg/riscv/tcg-target.h @@ -33,6 +33,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 20 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/s390/tcg-target.h b/tcg/s390/tcg-target.h index 853ed6e7aa..394b545369 100644 --- a/tcg/s390/tcg-target.h +++ b/tcg/s390/tcg-target.h @@ -27,6 +27,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 2 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 19 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 typedef enum TCGReg { TCG_REG_R0 = 0, diff --git a/tcg/sparc/tcg-target.h b/tcg/sparc/tcg-target.h index a0ed2a3342..dc0a227890 100644 --- a/tcg/sparc/tcg-target.h +++ b/tcg/sparc/tcg-target.h @@ -29,6 +29,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h index 086f34e69a..816dc4697c 100644 --- a/tcg/tci/tcg-target.h +++ b/tcg/tci/tcg-target.h @@ -43,6 +43,7 @@ #define TCG_TARGET_INTERPRETER 1 #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 #if UINTPTR_MAX == UINT32_MAX # define TCG_TARGET_REG_BITS 32 diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 10f1150c62..a3a1614f0e 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -74,6 +74,187 @@ QEMU_BUILD_BUG_ON(sizeof(target_ulong) > sizeof(run_on_cpu_data)); QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16); #define ALL_MMUIDX_BITS ((1 << NB_MMU_MODES) - 1) +#if TCG_TARGET_IMPLEMENTS_DYN_TLB +static inline size_t sizeof_tlb(CPUArchState *env, uintptr_t mmu_idx) +{ + return env->tlb_mask[mmu_idx] + (1 << CPU_TLB_ENTRY_BITS); +} + +static void tlb_window_reset(CPUTLBWindow *window, int64_t ns, + size_t max_entries) +{ + window->begin_ns = ns; + window->max_entries = max_entries; +} + +static void tlb_dyn_init(CPUArchState *env) +{ + int i; + + for (i = 0; i < NB_MMU_MODES; i++) { + CPUTLBDesc *desc = &env->tlb_d[i]; + size_t n_entries = 1 << CPU_TLB_DYN_DEFAULT_BITS; + + tlb_window_reset(&desc->window, get_clock_realtime(), 0); + desc->n_used_entries = 0; + env->tlb_mask[i] = (n_entries - 1) << CPU_TLB_ENTRY_BITS; + env->tlb_table[i] = g_new(CPUTLBEntry, n_entries); + env->iotlb[i] = g_new(CPUIOTLBEntry, n_entries); + } +} + +/** + * tlb_mmu_resize_locked() - perform TLB resize bookkeeping; resize if necessary + * @env: CPU that owns the TLB + * @mmu_idx: MMU index of the TLB + * + * Called with tlb_lock_held. + * + * We have two main constraints when resizing a TLB: (1) we only resize it + * on a TLB flush (otherwise we'd have to take a perf hit by either rehashing + * the array or unnecessarily flushing it), which means we do not control how + * frequently the resizing can occur; (2) we don't have access to the guest's + * future scheduling decisions, and therefore have to decide the magnitude of + * the resize based on past observations. + * + * In general, a memory-hungry process can benefit greatly from an appropriately + * sized TLB, since a guest TLB miss is very expensive. This doesn't mean that + * we just have to make the TLB as large as possible; while an oversized TLB + * results in minimal TLB miss rates, it also takes longer to be flushed + * (flushes can be _very_ frequent), and the reduced locality can also hurt + * performance. + * + * To achieve near-optimal performance for all kinds of workloads, we: + * + * 1. Aggressively increase the size of the TLB when the use rate of the + * TLB being flushed is high, since it is likely that in the near future this + * memory-hungry process will execute again, and its memory hungriness will + * probably be similar. + * + * 2. Slowly reduce the size of the TLB as the use rate declines over a + * reasonably large time window. The rationale is that if in such a time window + * we have not observed a high TLB use rate, it is likely that we won't observe + * it in the near future. In that case, once a time window expires we downsize + * the TLB to match the maximum use rate observed in the window. + * + * 3. Try to keep the maximum use rate in a time window in the 30-70% range, + * since in that range performance is likely near-optimal. Recall that the TLB + * is direct mapped, so we want the use rate to be low (or at least not too + * high), since otherwise we are likely to have a significant amount of + * conflict misses. + */ +static void tlb_mmu_resize_locked(CPUArchState *env, int mmu_idx) +{ + CPUTLBDesc *desc = &env->tlb_d[mmu_idx]; + size_t old_size = tlb_n_entries(env, mmu_idx); + size_t rate; + size_t new_size = old_size; + int64_t now = get_clock_realtime(); + int64_t window_len_ms = 100; + int64_t window_len_ns = window_len_ms * 1000 * 1000; + bool window_expired = now > desc->window.begin_ns + window_len_ns; + + if (desc->n_used_entries > desc->window.max_entries) { + desc->window.max_entries = desc->n_used_entries; + } + rate = desc->window.max_entries * 100 / old_size; + + if (rate > 70) { + new_size = MIN(old_size << 1, 1 << CPU_TLB_DYN_MAX_BITS); + } else if (rate < 30 && window_expired) { + size_t ceil = pow2ceil(desc->window.max_entries); + size_t expected_rate = desc->window.max_entries * 100 / ceil; + + /* + * Avoid undersizing when the max number of entries seen is just below + * a pow2. For instance, if max_entries == 1025, the expected use rate + * would be 1025/2048==50%. However, if max_entries == 1023, we'd get + * 1023/1024==99.9% use rate, so we'd likely end up doubling the size + * later. Thus, make sure that the expected use rate remains below 70%. + * (and since we double the size, that means the lowest rate we'd + * expect to get is 35%, which is still in the 30-70% range where + * we consider that the size is appropriate.) + */ + if (expected_rate > 70) { + ceil *= 2; + } + new_size = MAX(ceil, 1 << CPU_TLB_DYN_MIN_BITS); + } + + if (new_size == old_size) { + if (window_expired) { + tlb_window_reset(&desc->window, now, desc->n_used_entries); + } + return; + } + + g_free(env->tlb_table[mmu_idx]); + g_free(env->iotlb[mmu_idx]); + + tlb_window_reset(&desc->window, now, 0); + /* desc->n_used_entries is cleared by the caller */ + env->tlb_mask[mmu_idx] = (new_size - 1) << CPU_TLB_ENTRY_BITS; + env->tlb_table[mmu_idx] = g_try_new(CPUTLBEntry, new_size); + env->iotlb[mmu_idx] = g_try_new(CPUIOTLBEntry, new_size); + /* + * If the allocations fail, try smaller sizes. We just freed some + * memory, so going back to half of new_size has a good chance of working. + * Increased memory pressure elsewhere in the system might cause the + * allocations to fail though, so we progressively reduce the allocation + * size, aborting if we cannot even allocate the smallest TLB we support. + */ + while (env->tlb_table[mmu_idx] == NULL || env->iotlb[mmu_idx] == NULL) { + if (new_size == (1 << CPU_TLB_DYN_MIN_BITS)) { + error_report("%s: %s", __func__, strerror(errno)); + abort(); + } + new_size = MAX(new_size >> 1, 1 << CPU_TLB_DYN_MIN_BITS); + env->tlb_mask[mmu_idx] = (new_size - 1) << CPU_TLB_ENTRY_BITS; + + g_free(env->tlb_table[mmu_idx]); + g_free(env->iotlb[mmu_idx]); + env->tlb_table[mmu_idx] = g_try_new(CPUTLBEntry, new_size); + env->iotlb[mmu_idx] = g_try_new(CPUIOTLBEntry, new_size); + } +} + +static inline void tlb_table_flush_by_mmuidx(CPUArchState *env, int mmu_idx) +{ + tlb_mmu_resize_locked(env, mmu_idx); + memset(env->tlb_table[mmu_idx], -1, sizeof_tlb(env, mmu_idx)); + env->tlb_d[mmu_idx].n_used_entries = 0; +} + +static inline void tlb_n_used_entries_inc(CPUArchState *env, uintptr_t mmu_idx) +{ + env->tlb_d[mmu_idx].n_used_entries++; +} + +static inline void tlb_n_used_entries_dec(CPUArchState *env, uintptr_t mmu_idx) +{ + env->tlb_d[mmu_idx].n_used_entries--; +} + +#else /* !TCG_TARGET_IMPLEMENTS_DYN_TLB */ + +static inline void tlb_dyn_init(CPUArchState *env) +{ +} + +static inline void tlb_table_flush_by_mmuidx(CPUArchState *env, int mmu_idx) +{ + memset(env->tlb_table[mmu_idx], -1, sizeof(env->tlb_table[0])); +} + +static inline void tlb_n_used_entries_inc(CPUArchState *env, uintptr_t mmu_idx) +{ +} + +static inline void tlb_n_used_entries_dec(CPUArchState *env, uintptr_t mmu_idx) +{ +} +#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ + void tlb_init(CPUState *cpu) { CPUArchState *env = cpu->env_ptr; @@ -82,6 +263,8 @@ void tlb_init(CPUState *cpu) /* Ensure that cpu_reset performs a full flush. */ env->tlb_c.dirty = ALL_MMUIDX_BITS; + + tlb_dyn_init(env); } /* flush_all_helper: run fn across all cpus @@ -122,7 +305,7 @@ void tlb_flush_counts(size_t *pfull, size_t *ppart, size_t *pelide) static void tlb_flush_one_mmuidx_locked(CPUArchState *env, int mmu_idx) { - memset(env->tlb_table[mmu_idx], -1, sizeof(env->tlb_table[0])); + tlb_table_flush_by_mmuidx(env, mmu_idx); memset(env->tlb_v_table[mmu_idx], -1, sizeof(env->tlb_v_table[0])); env->tlb_d[mmu_idx].large_page_addr = -1; env->tlb_d[mmu_idx].large_page_mask = -1; @@ -234,12 +417,14 @@ static inline bool tlb_entry_is_empty(const CPUTLBEntry *te) } /* Called with tlb_c.lock held */ -static inline void tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, +static inline bool tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, target_ulong page) { if (tlb_hit_page_anyprot(tlb_entry, page)) { memset(tlb_entry, -1, sizeof(*tlb_entry)); + return true; } + return false; } /* Called with tlb_c.lock held */ @@ -250,7 +435,9 @@ static inline void tlb_flush_vtlb_page_locked(CPUArchState *env, int mmu_idx, assert_cpu_is_self(ENV_GET_CPU(env)); for (k = 0; k < CPU_VTLB_SIZE; k++) { - tlb_flush_entry_locked(&env->tlb_v_table[mmu_idx][k], page); + if (tlb_flush_entry_locked(&env->tlb_v_table[mmu_idx][k], page)) { + tlb_n_used_entries_dec(env, mmu_idx); + } } } @@ -267,7 +454,9 @@ static void tlb_flush_page_locked(CPUArchState *env, int midx, midx, lp_addr, lp_mask); tlb_flush_one_mmuidx_locked(env, midx); } else { - tlb_flush_entry_locked(tlb_entry(env, midx, page), page); + if (tlb_flush_entry_locked(tlb_entry(env, midx, page), page)) { + tlb_n_used_entries_dec(env, midx); + } tlb_flush_vtlb_page_locked(env, midx, page); } } @@ -444,8 +633,9 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length) qemu_spin_lock(&env->tlb_c.lock); for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { unsigned int i; + unsigned int n = tlb_n_entries(env, mmu_idx); - for (i = 0; i < CPU_TLB_SIZE; i++) { + for (i = 0; i < n; i++) { tlb_reset_dirty_range_locked(&env->tlb_table[mmu_idx][i], start1, length); } @@ -607,6 +797,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, /* Evict the old entry into the victim tlb. */ copy_tlb_helper_locked(tv, te); env->iotlb_v[mmu_idx][vidx] = env->iotlb[mmu_idx][index]; + tlb_n_used_entries_dec(env, mmu_idx); } /* refill the tlb */ @@ -658,6 +849,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, } copy_tlb_helper_locked(te, &tn); + tlb_n_used_entries_inc(env, mmu_idx); qemu_spin_unlock(&env->tlb_c.lock); } From patchwork Wed Jan 23 22:56:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030226 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="AXhYOESD"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLWS4ZyCz9s7h for ; Thu, 24 Jan 2019 10:06:28 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43297 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRbG-00043D-Je for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:06:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40898) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSR-0005s4-EZ for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSL-00049O-Lw for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:15 -0500 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:37950) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSL-00048E-7x for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:13 -0500 Received: by mail-pf1-x442.google.com with SMTP id q1so1934850pfi.5 for ; Wed, 23 Jan 2019 14:57:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G3R6M8lo+egOdVDHfgVuN8I78ohbr7aGRLQnNFv5cCY=; b=AXhYOESDPpTGWSvVzeNHsiaTlQWLH/KgTGDatOJUmkhtG91ma9VhIff7ydN3F947DB RWFuIYK4zVhjYtu4UQfYxBwFtKPadrP5TNXQ5UosNg1AbP91RMkM4IHsIOQCIvWVpRE4 3G9KZ6bOiJmBU0gBuOBYjpYPzVqGJm0JWbFEc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G3R6M8lo+egOdVDHfgVuN8I78ohbr7aGRLQnNFv5cCY=; b=Fb71q/ET2tLIKewM4qiF3GC+Ufgn4LTXpYAHpB5JZJFVTtV4zC7/mf195upt9Lr0YS QL9fXp9n9nda/MVEmPskKoAe0bDdSw3F2ep+u2DgLVO3A+8KvqP+zpV5Z8hMSk50yCLO dc0EKDcRMqvl+Fh3/52xK7u4OcNyv85Y37Z9XH9wF5RnGNiUrLhkCYPtlJ1T3ZLwjFcp fmU0yi3IO55Sg+Ua9sxDHqp0Mgqi0dk3xY1SxPXJW8R5kzP5UR4DdrsZx9CncExP/5A2 kU1A6X1yEvbQHI405SDYyexIMOIWTze1BGcpWzF8h/G00q3X2laILoKzHMAvIizA4FDP ugbw== X-Gm-Message-State: AJcUukejR4BsSM8qutbMxWDkoZCOx9Fo8qQ30DGeaUZ+GV1MROv20FvP Kjb58YRxhG9KBcR1ERiZ/aIvsHI4Eps= X-Google-Smtp-Source: ALg8bN4fmW3yjeUWdqP/AmYsRz+YGvGV6SDz4EAU7c7xOj75rJCpZ2LTztfvqUo7msddwQ6EC5jupw== X-Received: by 2002:a63:6b05:: with SMTP id g5mr3652382pgc.15.1548284231557; Wed, 23 Jan 2019 14:57:11 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:10 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:55 -0800 Message-Id: <20190123225705.28963-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH 03/13] tcg/i386: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" As the following experiments show, this series is a net perf gain, particularly for memory-heavy workloads. Experiments are run on an Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz. 1. System boot + shudown, debian aarch64: - Before (v3.1.0): Performance counter stats for './die.sh v3.1.0' (10 runs): 9019.797015 task-clock (msec) # 0.993 CPUs utilized ( +- 0.23% ) 29,910,312,379 cycles # 3.316 GHz ( +- 0.14% ) 54,699,252,014 instructions # 1.83 insn per cycle ( +- 0.08% ) 10,061,951,686 branches # 1115.541 M/sec ( +- 0.08% ) 172,966,530 branch-misses # 1.72% of all branches ( +- 0.07% ) 9.084039051 seconds time elapsed ( +- 0.23% ) - After: Performance counter stats for './die.sh tlb-dyn-v5' (10 runs): 8624.084842 task-clock (msec) # 0.993 CPUs utilized ( +- 0.23% ) 28,556,123,404 cycles # 3.311 GHz ( +- 0.13% ) 51,755,089,512 instructions # 1.81 insn per cycle ( +- 0.05% ) 9,526,513,946 branches # 1104.641 M/sec ( +- 0.05% ) 166,578,509 branch-misses # 1.75% of all branches ( +- 0.19% ) 8.680540350 seconds time elapsed ( +- 0.24% ) That is, a 4.4% perf increase. 2. System boot + shutdown, ubuntu 18.04 x86_64: - Before (v3.1.0): 56100.574751 task-clock (msec) # 1.016 CPUs utilized ( +- 4.81% ) 200,745,466,128 cycles # 3.578 GHz ( +- 5.24% ) 431,949,100,608 instructions # 2.15 insn per cycle ( +- 5.65% ) 77,502,383,330 branches # 1381.490 M/sec ( +- 6.18% ) 844,681,191 branch-misses # 1.09% of all branches ( +- 3.82% ) 55.221556378 seconds time elapsed ( +- 5.01% ) - After: 56603.419540 task-clock (msec) # 1.019 CPUs utilized ( +- 10.19% ) 202,217,930,479 cycles # 3.573 GHz ( +- 10.69% ) 439,336,291,626 instructions # 2.17 insn per cycle ( +- 14.14% ) 80,538,357,447 branches # 1422.853 M/sec ( +- 16.09% ) 776,321,622 branch-misses # 0.96% of all branches ( +- 3.77% ) 55.549661409 seconds time elapsed ( +- 10.44% ) No improvement (within noise range). Note that for this workload, increasing the time window too much can lead to perf degradation, since it flushes the TLB *very* frequently. 3. x86_64 SPEC06int: x86_64-softmmu speedup vs. v3.1.0 for SPEC06int (test set) Host: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz (Skylake) 5.5 +------------------------------------------------------------------------+ | +-+ | 5 |-+.................+-+...............................tlb-dyn-v5.......+-| | * * | 4.5 |-+.................*.*................................................+-| | * * | 4 |-+.................*.*................................................+-| | * * | 3.5 |-+.................*.*................................................+-| | * * | 3 |-+......+-+*.......*.*................................................+-| | * * * * | 2.5 |-+......*..*.......*.*.................................+-+*...........+-| | * * * * * * | 2 |-+......*..*.......*.*.................................*..*...........+-| | * * * * * * +-+ | 1.5 |-+......*..*.......*.*.................................*..*.*+-+.*+-+.+-| | * * *+-+ * * +-+ *+-+ +-+ +-+ * * * * * * | 1 |++++-+*+*++*+*++*++*+*++*+*+++-+*+*+-++*+-++++-++++-+++*++*+*++*+*++*+++| | * * * * * * * * * * * * * * * * * * * * * * * * * * | 0.5 +------------------------------------------------------------------------+ 400.perlb401.bzip403.g429445.g456.hm462.libq464.h471.omn47483.xalancbgeomean png: https://imgur.com/YRF90f7 That is, a 1.51x average speedup over the baseline, with a max speedup of 5.17x. Here's a different look at the SPEC06int results, using KVM as the baseline: x86_64-softmmu slowdown vs. KVM for SPEC06int (test set) Host: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz (Skylake) 25 +---------------------------------------------------------------------------+ | +-+ +-+ | | * * +-+ v3.1.0 | | * * +-+ tlb-dyn-v5 | | * * * * +-+ | 20 |-+.................*.*.............................*.+-+......*.*........+-| | * * * # # * * | | +-+ * * * # # * * | | * * * * * # # * * | 15 |-+......*.*........*.*.............................*.#.#......*.+-+......+-| | * * * * * # # * #|# | | * * * * +-+ * # # * +-+ | | * * +-+ * * ++-+ +-+ * # # * # # +-+ | | * * +-+ * * * ## *| +-+ * # # * # # +-+ | 10 |-+......*.*..*.+-+.*.*........*.##.......++-+.*.+-+*.#.#......*.#.#.*.*..+-| | * * * +-+ * * * ## +-+ *# # * # #* # # +-+ * # # * * | | * * * # # * * +-+ * ## * +-+ *# # * # #* # # * * * # # *+-+ | | * * * # # * * * +-+ * ## * # # *# # * # #* # # * * * # # * ## | 5 |-+......*.+-+*.#.#.*.*..*.#.#.*.##.*.#.#.*#.#.*.#.#*.#.#.*.*..*.#.#.*.##.+-| | * # #* # # * +-+* # # * ## * # # *# # * # #* # # * * * # # * ## | | * # #* # # * # #* # # * ## * # # *# # * # #* # # * +-+* # # * ## | | ++-+ * # #* # # * # #* # # * ## * # # *# # * # #* # # * # #* # # * ## | |+++*#+#+*+#+#*+#+#+*+#+#*+#+#+*+##+*+#+#+*#+#+*+#+#*+#+#+*+#+#*+#+#+*+##+++| 0 +---------------------------------------------------------------------------+ 400.perlbe401.bzi403.gc429445.go456.h462.libqu464.h471.omne4483.xalancbmgeomean png: https://imgur.com/YzAMNEV After this series, we bring down the average SPEC06int slowdown vs KVM from 11.47x to 7.58x. Tested-by: Alex Bennée Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Message-Id: <20190116170114.26802-4-cota@braap.org> Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.h | 2 +- tcg/i386/tcg-target.inc.c | 28 ++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 94aa4cef7c..eb40312e67 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -27,7 +27,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 31 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #ifdef __x86_64__ # define TCG_TARGET_REG_BITS 64 diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index dbf8253718..4d84aea3a9 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -329,6 +329,7 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_ARITH_GvEv (0x03) /* ... plus (ARITH_FOO << 3) */ #define OPC_ANDN (0xf2 | P_EXT38) #define OPC_ADD_GvEv (OPC_ARITH_GvEv | (ARITH_ADD << 3)) +#define OPC_AND_GvEv (OPC_ARITH_GvEv | (ARITH_AND << 3)) #define OPC_BLENDPS (0x0c | P_EXT3A | P_DATA16) #define OPC_BSF (0xbc | P_EXT) #define OPC_BSR (0xbd | P_EXT) @@ -1641,7 +1642,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, } if (TCG_TYPE_PTR == TCG_TYPE_I64) { hrexw = P_REXW; - if (TARGET_PAGE_BITS + CPU_TLB_BITS > 32) { + if (TARGET_PAGE_BITS + CPU_TLB_DYN_MAX_BITS > 32) { tlbtype = TCG_TYPE_I64; tlbrexw = P_REXW; } @@ -1649,6 +1650,15 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, } tcg_out_mov(s, tlbtype, r0, addrlo); + tcg_out_shifti(s, SHIFT_SHR + tlbrexw, r0, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + + tcg_out_modrm_offset(s, OPC_AND_GvEv + trexw, r0, TCG_AREG0, + offsetof(CPUArchState, tlb_mask[mem_index])); + + tcg_out_modrm_offset(s, OPC_ADD_GvEv + hrexw, r0, TCG_AREG0, + offsetof(CPUArchState, tlb_table[mem_index])); + /* If the required alignment is at least as large as the access, simply copy the address and mask. For lesser alignments, check that we don't cross pages for the complete access. */ @@ -1658,20 +1668,10 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, tcg_out_modrm_offset(s, OPC_LEA + trexw, r1, addrlo, s_mask - a_mask); } tlb_mask = (target_ulong)TARGET_PAGE_MASK | a_mask; - - tcg_out_shifti(s, SHIFT_SHR + tlbrexw, r0, - TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); - tgen_arithi(s, ARITH_AND + trexw, r1, tlb_mask, 0); - tgen_arithi(s, ARITH_AND + tlbrexw, r0, - (CPU_TLB_SIZE - 1) << CPU_TLB_ENTRY_BITS, 0); - - tcg_out_modrm_sib_offset(s, OPC_LEA + hrexw, r0, TCG_AREG0, r0, 0, - offsetof(CPUArchState, tlb_table[mem_index][0]) - + which); /* cmp 0(r0), r1 */ - tcg_out_modrm_offset(s, OPC_CMP_GvEv + trexw, r1, r0, 0); + tcg_out_modrm_offset(s, OPC_CMP_GvEv + trexw, r1, r0, which); /* Prepare for both the fast path add of the tlb addend, and the slow path function argument setup. */ @@ -1684,7 +1684,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, if (TARGET_LONG_BITS > TCG_TARGET_REG_BITS) { /* cmp 4(r0), addrhi */ - tcg_out_modrm_offset(s, OPC_CMP_GvEv, addrhi, r0, 4); + tcg_out_modrm_offset(s, OPC_CMP_GvEv, addrhi, r0, which + 4); /* jne slow_path */ tcg_out_opc(s, OPC_JCC_long + JCC_JNE, 0, 0, 0); @@ -1696,7 +1696,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, /* add addend(r0), r1 */ tcg_out_modrm_offset(s, OPC_ADD_GvEv + hrexw, r1, r0, - offsetof(CPUTLBEntry, addend) - which); + offsetof(CPUTLBEntry, addend)); } /* From patchwork Wed Jan 23 22:56:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030220 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="Kgq8eySU"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLPV2XdNz9s7h for ; Thu, 24 Jan 2019 10:01:18 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43229 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRWG-0000Ei-6C for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:01:16 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40926) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSj-00063F-Eg for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSa-0004FL-0T for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:31 -0500 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:36998) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSP-00049e-HV for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:19 -0500 Received: by mail-pg1-x543.google.com with SMTP id c25so1755424pgb.4 for ; Wed, 23 Jan 2019 14:57:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7A3cyVnhngUyFXbPZwkeSvHcXZO2amddRw53jDJwAWc=; b=Kgq8eySUOsDHnp+CzoE1Pe61OPYD8ckkQJCNIsBqDUUNq1tT98+fLcYxwTDA5puFbg PnTTfxykXHqCqshn13+xRWzgib5un8ODtPg4u54s9/3QFntcp5oGvr10cm6bpYvt9WzA 8aqaRu+BLJTcW6RF8tIejvs7nQevzIu22++38= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7A3cyVnhngUyFXbPZwkeSvHcXZO2amddRw53jDJwAWc=; b=Vpvz0IpvV/3JxayH8/umHcxGlhAFumyCXwEYE+fMHG6OTl3e7if97zyPE096VXtxst rVZGVyAvzHekmMGnndMeTb033hoix8SKZg4MhZrSnUrw3NAdqqdOwYJUlFEMgf+OL7km VEUmN1876iYTtIFKhOkE2lCq/1MkKY+lVzB/PQ0n72wvp+u5encnQJcdNe2qFVoZjyc7 /40d6Y3eHSxHUqnqjnLdgXw1ar61+UX/xc1hlzAUtVXLx2+Lk9apAHd3Jzq+zLXcn8n4 Mlv6AUiz4lP/4kBAfr4wM1wu81VPE2ieqqTBSEGAAGwLu6DubGRsX5DAW5hbNcnhyx9u 8lwA== X-Gm-Message-State: AJcUukf3Gg6j+KUh0O8nDmt8ADJhgSYoPSGQUpaw2Oyy2D1MtcjRd9uP pcUMMWYEj7xARb3w5/gTibD9GDp/8vk= X-Google-Smtp-Source: ALg8bN6iQX5z9OlSGv6oPVgnS3/6nhKibXvPjCrvqcSnQVMlNK93ArhJDboWH7EJfmV6mQQG9I81cA== X-Received: by 2002:a62:76cc:: with SMTP id r195mr3936611pfc.38.1548284233123; Wed, 23 Jan 2019 14:57:13 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:11 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:56 -0800 Message-Id: <20190123225705.28963-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH 04/13] tcg/aarch64: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson Reviewed-by: Alex Bennée Tested-by: Alex Bennée --- tcg/aarch64/tcg-target.h | 2 +- tcg/aarch64/tcg-target.inc.c | 100 +++++++++++++++++++++-------------- 2 files changed, 60 insertions(+), 42 deletions(-) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 68868a27eb..5085a81060 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -15,7 +15,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 24 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #undef TCG_TARGET_STACK_GROWSUP typedef enum { diff --git a/tcg/aarch64/tcg-target.inc.c b/tcg/aarch64/tcg-target.inc.c index ee0d5819af..d57f9e500f 100644 --- a/tcg/aarch64/tcg-target.inc.c +++ b/tcg/aarch64/tcg-target.inc.c @@ -498,6 +498,9 @@ typedef enum { I3510_EON = 0x4a200000, I3510_ANDS = 0x6a000000, + /* Logical shifted register instructions (with a shift). */ + I3502S_AND_LSR = I3510_AND | (1 << 22), + /* AdvSIMD copy */ I3605_DUP = 0x0e000400, I3605_INS = 0x4e001c00, @@ -1448,6 +1451,14 @@ static void add_qemu_ldst_label(TCGContext *s, bool is_ld, TCGMemOpIdx oi, label->label_ptr[0] = label_ptr; } +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + +/* We expect to use a 24-bit unsigned offset from ENV. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table[NB_MMU_MODES - 1]) + > 0xffffff); + /* Load and compare a TLB entry, emitting the conditional jump to the slow path for the failure case, which will be patched later when finalizing the slow path. Generated code returns the host addend in X1, @@ -1456,15 +1467,55 @@ static void tcg_out_tlb_read(TCGContext *s, TCGReg addr_reg, TCGMemOp opc, tcg_insn_unit **label_ptr, int mem_index, bool is_read) { - int tlb_offset = is_read ? - offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) - : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write); + int mask_ofs = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_ofs = offsetof(CPUArchState, tlb_table[mem_index]); unsigned a_bits = get_alignment_bits(opc); unsigned s_bits = opc & MO_SIZE; unsigned a_mask = (1u << a_bits) - 1; unsigned s_mask = (1u << s_bits) - 1; - TCGReg base = TCG_AREG0, x3; - uint64_t tlb_mask; + TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0, x3; + TCGType mask_type; + uint64_t compare_mask; + + if (table_ofs > 0xfff) { + int table_hi = table_ofs & ~0xfff; + int mask_hi = mask_ofs & ~0xfff; + + table_base = TCG_REG_X1; + if (mask_hi == table_hi) { + mask_base = table_base; + } else if (mask_hi) { + mask_base = TCG_REG_X0; + tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, + mask_base, TCG_AREG0, mask_hi); + } + tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, + table_base, TCG_AREG0, table_hi); + mask_ofs -= mask_hi; + table_ofs -= table_hi; + } + + mask_type = (TARGET_PAGE_BITS + CPU_TLB_DYN_MAX_BITS > 32 + ? TCG_TYPE_I64 : TCG_TYPE_I32); + + /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ + tcg_out_ld(s, mask_type, TCG_REG_X0, mask_base, mask_ofs); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_X1, table_base, table_ofs); + + /* Extract the TLB index from the address into X0. */ + tcg_out_insn(s, 3502S, AND_LSR, mask_type == TCG_TYPE_I64, + TCG_REG_X0, TCG_REG_X0, addr_reg, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + + /* Add the tlb_table pointer, creating the CPUTLBEntry address into X1. */ + tcg_out_insn(s, 3502, ADD, 1, TCG_REG_X1, TCG_REG_X1, TCG_REG_X0); + + /* Load the tlb comparator into X0, and the fast path addend into X1. */ + tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_X0, TCG_REG_X1, is_read + ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write)); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_X1, TCG_REG_X1, + offsetof(CPUTLBEntry, addend)); /* For aligned accesses, we check the first byte and include the alignment bits within the address. For unaligned access, we check that we don't @@ -1476,47 +1527,14 @@ static void tcg_out_tlb_read(TCGContext *s, TCGReg addr_reg, TCGMemOp opc, TCG_REG_X3, addr_reg, s_mask - a_mask); x3 = TCG_REG_X3; } - tlb_mask = (uint64_t)TARGET_PAGE_MASK | a_mask; - - /* Extract the TLB index from the address into X0. - X0 = - addr_reg */ - tcg_out_ubfm(s, TARGET_LONG_BITS == 64, TCG_REG_X0, addr_reg, - TARGET_PAGE_BITS, TARGET_PAGE_BITS + CPU_TLB_BITS); + compare_mask = (uint64_t)TARGET_PAGE_MASK | a_mask; /* Store the page mask part of the address into X3. */ tcg_out_logicali(s, I3404_ANDI, TARGET_LONG_BITS == 64, - TCG_REG_X3, x3, tlb_mask); - - /* Add any "high bits" from the tlb offset to the env address into X2, - to take advantage of the LSL12 form of the ADDI instruction. - X2 = env + (tlb_offset & 0xfff000) */ - if (tlb_offset & 0xfff000) { - tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, TCG_REG_X2, base, - tlb_offset & 0xfff000); - base = TCG_REG_X2; - } - - /* Merge the tlb index contribution into X2. - X2 = X2 + (X0 << CPU_TLB_ENTRY_BITS) */ - tcg_out_insn(s, 3502S, ADD_LSL, TCG_TYPE_I64, TCG_REG_X2, base, - TCG_REG_X0, CPU_TLB_ENTRY_BITS); - - /* Merge "low bits" from tlb offset, load the tlb comparator into X0. - X0 = load [X2 + (tlb_offset & 0x000fff)] */ - tcg_out_ldst(s, TARGET_LONG_BITS == 32 ? I3312_LDRW : I3312_LDRX, - TCG_REG_X0, TCG_REG_X2, tlb_offset & 0xfff, - TARGET_LONG_BITS == 32 ? 2 : 3); - - /* Load the tlb addend. Do that early to avoid stalling. - X1 = load [X2 + (tlb_offset & 0xfff) + offsetof(addend)] */ - tcg_out_ldst(s, I3312_LDRX, TCG_REG_X1, TCG_REG_X2, - (tlb_offset & 0xfff) + (offsetof(CPUTLBEntry, addend)) - - (is_read ? offsetof(CPUTLBEntry, addr_read) - : offsetof(CPUTLBEntry, addr_write)), 3); + TCG_REG_X3, x3, compare_mask); /* Perform the address comparison. */ - tcg_out_cmp(s, (TARGET_LONG_BITS == 64), TCG_REG_X0, TCG_REG_X3, 0); + tcg_out_cmp(s, TARGET_LONG_BITS == 64, TCG_REG_X0, TCG_REG_X3, 0); /* If not equal, we jump to the slow path. */ *label_ptr = s->code_ptr; From patchwork Wed Jan 23 22:56:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030217 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="FTzWzDte"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLLD2bq1z9s3l for ; Thu, 24 Jan 2019 09:58:28 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43176 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRTW-00068l-7u for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 17:58:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40984) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSl-00064i-M1 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSj-0004JM-UW for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:40497) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSj-0004AI-Hs for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:37 -0500 Received: by mail-pg1-x544.google.com with SMTP id z10so1747495pgp.7 for ; Wed, 23 Jan 2019 14:57:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UQit8ZLfJm/PCda/410nizx0XvPkSZMgAZZJczgwnZs=; b=FTzWzDtenileIpe8Tw651M0YTYtpLqp2FiKT1bJDuDWgjWM9gI5MpnqDOytzCm++Qx ZqFbfjT/Jc9uQ2szlBFHB/WF32fc2ALqszMsKmhbt05uIUbJdKMDGShY082qAabwedp1 9Mo4MForslJ+D8VUoSq7FjG4leugiYpZtMYz0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UQit8ZLfJm/PCda/410nizx0XvPkSZMgAZZJczgwnZs=; b=eaQpdZ4KPlQX4r24PGLKHpZrYgUiethzoIdUvw8ZRvF54+m4ObOWco/EfRpiZdQaBj bCRKZCD4GGdx67gaCW1bUssvrmA5OP5lEQlglCOEyHrFTqLY1qImM8Nsm5ItD46NaWGq KRlIX3x+KlX+TJI0yuE7Gyjax1B6hrb38KbmbaxuUSypf0nFleJ34jtDxz63pkYdIcQ0 uKIMwSCfKf3zGBy2Yy0UkhQTJJwU/au4iPP6YzkJ22ha4VMjMEjxLE8f+/P66AmsESFc jPgVP69BbvNcPnNj0a7HjJvoXEBDVoq5mNyEskMQh58FClJdo9lt6jen3AaBjU73YITt AQpg== X-Gm-Message-State: AJcUukcLqjapbvTWGeB/5JFlxeNRE6++UWIWsmeDqdRxj75ghksnm+q8 7/voir/ImjGP6PAIQA3/si72AHeZPdU= X-Google-Smtp-Source: ALg8bN4QhfMNNuqEITxXIpbBSORLWSc5A55ix+mRkDyoitvt05Qi1fH/lw2b06mOhJz6bVDhW5BPfg== X-Received: by 2002:a63:920a:: with SMTP id o10mr3646006pgd.141.1548284234397; Wed, 23 Jan 2019 14:57:14 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:13 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:57 -0800 Message-Id: <20190123225705.28963-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 Subject: [Qemu-devel] [PATCH 05/13] tcg/ppc: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/ppc/tcg-target.h | 2 +- tcg/ppc/tcg-target.inc.c | 91 ++++++++++++++++++++++------------------ 2 files changed, 52 insertions(+), 41 deletions(-) diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index b51854b5cf..95b735b0bb 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -34,7 +34,7 @@ #define TCG_TARGET_NB_REGS 32 #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum { TCG_REG_R0, TCG_REG_R1, TCG_REG_R2, TCG_REG_R3, diff --git a/tcg/ppc/tcg-target.inc.c b/tcg/ppc/tcg-target.inc.c index 8c1cfdd7ac..773690f1d9 100644 --- a/tcg/ppc/tcg-target.inc.c +++ b/tcg/ppc/tcg-target.inc.c @@ -327,6 +327,7 @@ static int tcg_target_const_match(tcg_target_long val, TCGType type, #define LHZ OPCD( 40) #define LHA OPCD( 42) #define LWZ OPCD( 32) +#define LWZUX XO31( 55) #define STB OPCD( 38) #define STH OPCD( 44) #define STW OPCD( 36) @@ -338,6 +339,7 @@ static int tcg_target_const_match(tcg_target_long val, TCGType type, #define LD XO58( 0) #define LDX XO31( 21) #define LDU XO58( 1) +#define LDUX XO31( 53) #define LWA XO58( 2) #define LWAX XO31(341) @@ -1503,6 +1505,10 @@ static void * const qemu_st_helpers[16] = { [MO_BEQ] = helper_be_stq_mmu, }; +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + /* Perform the TLB load and compare. Places the result of the comparison in CR7, loads the addend of the TLB into R3, and returns the register containing the guest address (zero-extended into R4). Clobbers R0 and R2. */ @@ -1513,61 +1519,63 @@ static TCGReg tcg_out_tlb_read(TCGContext *s, TCGMemOp opc, { int cmp_off = (is_read - ? offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) - : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write)); - int add_off = offsetof(CPUArchState, tlb_table[mem_index][0].addend); - TCGReg base = TCG_AREG0; + ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write)); + int mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_off = offsetof(CPUArchState, tlb_table[mem_index]); + TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0; unsigned s_bits = opc & MO_SIZE; unsigned a_bits = get_alignment_bits(opc); - /* Extract the page index, shifted into place for tlb index. */ - if (TCG_TARGET_REG_BITS == 64) { - if (TARGET_LONG_BITS == 32) { - /* Zero-extend the address into a place helpful for further use. */ - tcg_out_ext32u(s, TCG_REG_R4, addrlo); - addrlo = TCG_REG_R4; - } else { - tcg_out_rld(s, RLDICL, TCG_REG_R3, addrlo, - 64 - TARGET_PAGE_BITS, 64 - CPU_TLB_BITS); + if (table_off > 0x7fff) { + int mask_hi = mask_off - (int16_t)mask_off; + int table_hi = table_off - (int16_t)table_off; + + table_base = TCG_REG_R4; + if (mask_hi == table_hi) { + mask_base = table_base; + } else if (mask_hi) { + mask_base = TCG_REG_R3; + tcg_out32(s, ADDIS | TAI(mask_base, TCG_AREG0, mask_hi >> 16)); } + tcg_out32(s, ADDIS | TAI(table_base, TCG_AREG0, table_hi >> 16)); + mask_off -= mask_hi; + table_off -= table_hi; } - /* Compensate for very large offsets. */ - if (add_off >= 0x8000) { - int low = (int16_t)cmp_off; - int high = cmp_off - low; - assert((high & 0xffff) == 0); - assert(cmp_off - high == (int16_t)(cmp_off - high)); - assert(add_off - high == (int16_t)(add_off - high)); - tcg_out32(s, ADDIS | TAI(TCG_REG_TMP1, base, high >> 16)); - base = TCG_REG_TMP1; - cmp_off -= high; - add_off -= high; - } + /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_R3, mask_base, mask_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_R4, table_base, table_off); - /* Extraction and shifting, part 2. */ - if (TCG_TARGET_REG_BITS == 32 || TARGET_LONG_BITS == 32) { - tcg_out_rlw(s, RLWINM, TCG_REG_R3, addrlo, - 32 - (TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS), - 32 - (CPU_TLB_BITS + CPU_TLB_ENTRY_BITS), - 31 - CPU_TLB_ENTRY_BITS); + /* Extract the page index, shifted into place for tlb index. */ + if (TCG_TARGET_REG_BITS == 32) { + tcg_out_shri32(s, TCG_REG_TMP1, addrlo, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); } else { - tcg_out_shli64(s, TCG_REG_R3, TCG_REG_R3, CPU_TLB_ENTRY_BITS); + tcg_out_shri64(s, TCG_REG_TMP1, addrlo, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); } + tcg_out32(s, AND | SAB(TCG_REG_R3, TCG_REG_R3, TCG_REG_TMP1)); - tcg_out32(s, ADD | TAB(TCG_REG_R3, TCG_REG_R3, base)); - - /* Load the tlb comparator. */ - if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { - tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_R4, TCG_REG_R3, cmp_off); - tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_TMP1, TCG_REG_R3, cmp_off + 4); + /* Load the TLB comparator. */ + if (cmp_off == 0 && TCG_TARGET_REG_BITS >= TARGET_LONG_BITS) { + uint32_t lxu = (TCG_TARGET_REG_BITS == 32 || TARGET_LONG_BITS == 32 + ? LWZUX : LDUX); + tcg_out32(s, lxu | TAB(TCG_REG_TMP1, TCG_REG_R3, TCG_REG_R4)); } else { - tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_TMP1, TCG_REG_R3, cmp_off); + tcg_out32(s, ADD | TAB(TCG_REG_R3, TCG_REG_R3, TCG_REG_R4)); + if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { + tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_TMP1, TCG_REG_R3, cmp_off + 4); + tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_R4, TCG_REG_R3, cmp_off); + } else { + tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_TMP1, TCG_REG_R3, cmp_off); + } } /* Load the TLB addend for use on the fast path. Do this asap to minimize any load use delay. */ - tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_R3, TCG_REG_R3, add_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_R3, TCG_REG_R3, + offsetof(CPUTLBEntry, addend)); /* Clear the non-page, non-alignment bits from the address */ if (TCG_TARGET_REG_BITS == 32) { @@ -1600,6 +1608,9 @@ static TCGReg tcg_out_tlb_read(TCGContext *s, TCGMemOp opc, if (TARGET_LONG_BITS == 32) { tcg_out_rlw(s, RLWINM, TCG_REG_R0, t, 0, (32 - a_bits) & 31, 31 - TARGET_PAGE_BITS); + /* Zero-extend the address for use in the final address. */ + tcg_out_ext32u(s, TCG_REG_R4, addrlo); + addrlo = TCG_REG_R4; } else if (a_bits == 0) { tcg_out_rld(s, RLDICR, TCG_REG_R0, t, 0, 63 - TARGET_PAGE_BITS); } else { From patchwork Wed Jan 23 22:56:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030216 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="EfJoc+se"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLL84bFCz9s3l for ; Thu, 24 Jan 2019 09:58:24 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43174 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRTS-00065d-Kc for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 17:58:22 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40931) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSj-00063J-Gk for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSf-0004Gz-Ll for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:37 -0500 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:34632) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRST-0004Ah-66 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:26 -0500 Received: by mail-pl1-x643.google.com with SMTP id w4so1920755plz.1 for ; Wed, 23 Jan 2019 14:57:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Dw5nbSHMsvXdy8KVi1NIDw6faRz5GrF8p+oczK/QAi4=; b=EfJoc+sepSKhb/YDN8mDhcTpAQ5PQAxQHTU29L/Yz8Ix0kEpBP3MJWNjJXV/AT+Fn6 rXvf9l7DyZGW7HsCcbO4+rx16NKiMtIdtwxsxlswQ7FOz4z9t4eGS2niww2U5w4u4um/ FFT+IsqGTNc+x/JGkjUUsvO5TwXBPCoD0zNnE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Dw5nbSHMsvXdy8KVi1NIDw6faRz5GrF8p+oczK/QAi4=; b=H1GWdwJPsNewKTnXleg3hg69w1kpN0lx+Ocs4uV5LLljOQHbKKV+BZ0g98zzdWbN7h 8yCAjokicKyTxVHsuKu0DMazrpfS6FCIIIiihW9YrF+ArsYasvLp8pVytmRKnv01Z3iF 9iaaMdx+fFESHgGs1z3lidbPuAoXZb4uEqiOTS8wpGuhrzGxFOsVb/X5MdeTaNTEAnxa UmmshyE7aC1ZqRQbN2Tu706tzMVVyS2nVkeWRAUIX6KsDI5P5japy7qqIrz6h6ddmfuS jAwB7YroJ1M2Xr6HMWoznEup83J9Y60heuBysYYomh+gn0MNSmCGFEK96mlxTSpFRWhn 6rMA== X-Gm-Message-State: AJcUukepHABc2hn7fzwSLBhGvPfSzORlBOb3/qt2pafoRajn25NAU0Eo IsgN1j4KLDGGEUZgJCQlOzMGapgD+B8= X-Google-Smtp-Source: ALg8bN5GUcbPrhzGd83Jcxn9JRzATIC+c8FJFg2wJ//Xdk48t9FcPuu293+rZTsPJod5NFTJi7PvTA== X-Received: by 2002:a17:902:1101:: with SMTP id d1mr4126366pla.136.1548284235576; Wed, 23 Jan 2019 14:57:15 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:14 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:58 -0800 Message-Id: <20190123225705.28963-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH 06/13] tcg/sparc: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/sparc/tcg-target.h | 2 +- tcg/sparc/tcg-target.inc.c | 82 +++++++++++++++++++++++--------------- 2 files changed, 51 insertions(+), 33 deletions(-) diff --git a/tcg/sparc/tcg-target.h b/tcg/sparc/tcg-target.h index dc0a227890..6020a670c0 100644 --- a/tcg/sparc/tcg-target.h +++ b/tcg/sparc/tcg-target.h @@ -29,7 +29,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/sparc/tcg-target.inc.c b/tcg/sparc/tcg-target.inc.c index 55144c437c..7a61839dc1 100644 --- a/tcg/sparc/tcg-target.inc.c +++ b/tcg/sparc/tcg-target.inc.c @@ -1074,54 +1074,72 @@ static void tcg_out_nop_fill(tcg_insn_unit *p, int count) The result of the TLB comparison is in %[ix]cc. The sanitized address is in the returned register, maybe %o0. The TLB addend is in %o1. */ +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + +/* We expect tlb_mask to be "near" tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) - + offsetof(CPUArchState, tlb_mask) >= (1 << 13)); + static TCGReg tcg_out_tlb_load(TCGContext *s, TCGReg addr, int mem_index, TCGMemOp opc, int which) { + int mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_off = offsetof(CPUArchState, tlb_table[mem_index]); + TCGReg base = TCG_AREG0; const TCGReg r0 = TCG_REG_O0; const TCGReg r1 = TCG_REG_O1; const TCGReg r2 = TCG_REG_O2; unsigned s_bits = opc & MO_SIZE; unsigned a_bits = get_alignment_bits(opc); - int tlb_ofs; + tcg_target_long compare_mask; - /* Shift the page number down. */ - tcg_out_arithi(s, r1, addr, TARGET_PAGE_BITS, SHIFT_SRL); + if (!check_fit_i32(table_off, 13)) { + int table_hi; + + base = r1; + if (table_off <= 2 * 0xfff) { + table_hi = 0xfff; + tcg_out_arithi(s, base, TCG_AREG0, table_hi, ARITH_ADD); + } else { + table_hi = table_off & ~0x3ff; + tcg_out_sethi(s, base, table_hi); + tcg_out_arith(s, base, TCG_AREG0, base, ARITH_ADD); + } + mask_off -= table_hi; + table_off -= table_hi; + tcg_debug_assert(check_fit_i32(mask_off, 13)); + } + + /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ + tcg_out_ld(s, TCG_TYPE_PTR, r0, base, mask_off); + tcg_out_ld(s, TCG_TYPE_PTR, r1, base, table_off); + + /* Extract the page index, shifted into place for tlb index. */ + tcg_out_arithi(s, r2, addr, TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS, + SHIFT_SRL); + tcg_out_arith(s, r2, r2, r0, ARITH_AND); + + /* Add the tlb_table pointer, creating the CPUTLBEntry address into R2. */ + tcg_out_arith(s, r2, r2, r1, ARITH_ADD); + + /* Load the tlb comparator and the addend. */ + tcg_out_ld(s, TCG_TYPE_TL, r0, r2, which); + tcg_out_ld(s, TCG_TYPE_PTR, r1, r2, offsetof(CPUTLBEntry, addend)); /* Mask out the page offset, except for the required alignment. We don't support unaligned accesses. */ if (a_bits < s_bits) { a_bits = s_bits; } - tcg_out_movi(s, TCG_TYPE_TL, TCG_REG_T1, - TARGET_PAGE_MASK | ((1 << a_bits) - 1)); - - /* Mask the tlb index. */ - tcg_out_arithi(s, r1, r1, CPU_TLB_SIZE - 1, ARITH_AND); - - /* Mask page, part 2. */ - tcg_out_arith(s, r0, addr, TCG_REG_T1, ARITH_AND); - - /* Shift the tlb index into place. */ - tcg_out_arithi(s, r1, r1, CPU_TLB_ENTRY_BITS, SHIFT_SLL); - - /* Relative to the current ENV. */ - tcg_out_arith(s, r1, TCG_AREG0, r1, ARITH_ADD); - - /* Find a base address that can load both tlb comparator and addend. */ - tlb_ofs = offsetof(CPUArchState, tlb_table[mem_index][0]); - if (!check_fit_ptr(tlb_ofs + sizeof(CPUTLBEntry), 13)) { - if (tlb_ofs & ~0x3ff) { - tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, tlb_ofs & ~0x3ff); - tcg_out_arith(s, r1, r1, TCG_REG_T1, ARITH_ADD); - } - tlb_ofs &= 0x3ff; + compare_mask = (tcg_target_ulong)TARGET_PAGE_MASK | ((1 << a_bits) - 1); + if (check_fit_tl(compare_mask, 13)) { + tcg_out_arithi(s, r2, addr, compare_mask, ARITH_AND); + } else { + tcg_out_movi(s, TCG_TYPE_TL, r2, compare_mask); + tcg_out_arith(s, r2, addr, r2, ARITH_AND); } - - /* Load the tlb comparator and the addend. */ - tcg_out_ld(s, TCG_TYPE_TL, r2, r1, tlb_ofs + which); - tcg_out_ld(s, TCG_TYPE_PTR, r1, r1, tlb_ofs+offsetof(CPUTLBEntry, addend)); - - /* subcc arg0, arg2, %g0 */ tcg_out_cmp(s, r0, r2, 0); /* If the guest address must be zero-extended, do so now. */ From patchwork Wed Jan 23 22:56:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030221 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="VvEDyTbU"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLPb0SpQz9sCX for ; Thu, 24 Jan 2019 10:01:23 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43232 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRWL-0000I9-0T for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:01:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41052) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSy-0006EU-AE for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSl-0004LN-HV for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:43 -0500 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:33042) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSl-0004BC-3Y for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: by mail-pg1-x542.google.com with SMTP id z11so1765876pgu.0 for ; Wed, 23 Jan 2019 14:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u1h6Y+1jeyggZCDYjdrFQDBu8S/E6FnZ3feRJf5H+uM=; b=VvEDyTbUP7hZA0BNgknITMAKGI2CfA1RhZPzIlOPa6CWZXT+i9N3vvrk8ga+JTiZAs hjNNsQjBV5RIIFSdzo7t4v2SsG7iry8hfY5PhtcNYxaDkD4TMkyxRhzdoh+/lItHbvH3 KFUEV3KTMrB3p1pdlACTPyhvbciz6ZBNA1y/g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u1h6Y+1jeyggZCDYjdrFQDBu8S/E6FnZ3feRJf5H+uM=; b=Auv7DLZua7yPZKHcHH9lregHtrSPkoy3R77EmQ1xGKq5UwJoICje/bGjowwFe7HGQk x8JnbLa8eIvOgoJc1rZ9Mt78jizAsnkz5X1xCEdHmBnpw8f9Z63pAqdJ0KWfs8H+aTrn 7IL4lJugLmsKpZiQbC4huxwh3GV6DgiYXXITdBh7a03Ox8GwQIO30CO5QNw6ogkC+31d +jsLP50/GzpgfRoAXHU1mJj65SZn3ZLH3yDim7BIsgEJji2TISCXby5CTJrsBCDh+Wcl bJi6Adz5ZHZ80K5MO45SoJd8lRpjITLIV/2HtfCQr9saKlvBdMTYqoif+o1d+Ckzj3fi QfsA== X-Gm-Message-State: AJcUukdMQ2I3+ZO/ljKoWI6QDmnFzz/F7gKZoECAFlnghO1F12QQER2l ZiTVzRDo9dkbqJ9iAFHrlet4Dm0zw0A= X-Google-Smtp-Source: ALg8bN45dxQNoMQFPr3chMLNMMvpDG23RZ9LJzmRJOnTD9ZXOiYXfpDGc1uVtqN6PflDPY+zXgpluA== X-Received: by 2002:aa7:83c6:: with SMTP id j6mr3939855pfn.91.1548284236846; Wed, 23 Jan 2019 14:57:16 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.15 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:16 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:56:59 -0800 Message-Id: <20190123225705.28963-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 Subject: [Qemu-devel] [PATCH 07/13] tcg/s390: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/s390/tcg-target.h | 2 +- tcg/s390/tcg-target.inc.c | 45 +++++++++++++++++---------------------- 2 files changed, 20 insertions(+), 27 deletions(-) diff --git a/tcg/s390/tcg-target.h b/tcg/s390/tcg-target.h index 394b545369..357528dd97 100644 --- a/tcg/s390/tcg-target.h +++ b/tcg/s390/tcg-target.h @@ -27,7 +27,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 2 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 19 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum TCGReg { TCG_REG_R0 = 0, diff --git a/tcg/s390/tcg-target.inc.c b/tcg/s390/tcg-target.inc.c index 39ecf609a1..7db90b3bae 100644 --- a/tcg/s390/tcg-target.inc.c +++ b/tcg/s390/tcg-target.inc.c @@ -1537,10 +1537,10 @@ static void tcg_out_qemu_st_direct(TCGContext *s, TCGMemOp opc, TCGReg data, #if defined(CONFIG_SOFTMMU) #include "tcg-ldst.inc.c" -/* We're expecting to use a 20-bit signed offset on the tlb memory ops. - Using the offset of the second entry in the last tlb table ensures - that we can index all of the elements of the first entry. */ -QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table[NB_MMU_MODES - 1][1]) +/* We're expecting to use a 20-bit signed offset on the tlb memory ops. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_mask[NB_MMU_MODES - 1]) + > 0x7ffff); +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table[NB_MMU_MODES - 1]) > 0x7ffff); /* Load and compare a TLB entry, leaving the flags set. Loads the TLB @@ -1552,48 +1552,41 @@ static TCGReg tcg_out_tlb_read(TCGContext* s, TCGReg addr_reg, TCGMemOp opc, unsigned a_bits = get_alignment_bits(opc); unsigned s_mask = (1 << s_bits) - 1; unsigned a_mask = (1 << a_bits) - 1; + int mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_off = offsetof(CPUArchState, tlb_table[mem_index]); int ofs, a_off; uint64_t tlb_mask; + tcg_out_sh64(s, RSY_SRLG, TCG_REG_R2, addr_reg, TCG_REG_NONE, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + tcg_out_insn(s, RXY, NG, TCG_REG_R2, TCG_AREG0, TCG_REG_NONE, mask_off); + tcg_out_insn(s, RXY, AG, TCG_REG_R2, TCG_AREG0, TCG_REG_NONE, table_off); + /* For aligned accesses, we check the first byte and include the alignment bits within the address. For unaligned access, we check that we don't cross pages using the address of the last byte of the access. */ a_off = (a_bits >= s_bits ? 0 : s_mask - a_mask); tlb_mask = (uint64_t)TARGET_PAGE_MASK | a_mask; - - if (s390_facilities & FACILITY_GEN_INST_EXT) { - tcg_out_risbg(s, TCG_REG_R2, addr_reg, - 64 - CPU_TLB_BITS - CPU_TLB_ENTRY_BITS, - 63 - CPU_TLB_ENTRY_BITS, - 64 + CPU_TLB_ENTRY_BITS - TARGET_PAGE_BITS, 1); - if (a_off) { - tcg_out_insn(s, RX, LA, TCG_REG_R3, addr_reg, TCG_REG_NONE, a_off); - tgen_andi(s, TCG_TYPE_TL, TCG_REG_R3, tlb_mask); - } else { - tgen_andi_risbg(s, TCG_REG_R3, addr_reg, tlb_mask); - } + if ((s390_facilities & FACILITY_GEN_INST_EXT) && a_off == 0) { + tgen_andi_risbg(s, TCG_REG_R3, addr_reg, tlb_mask); } else { - tcg_out_sh64(s, RSY_SRLG, TCG_REG_R2, addr_reg, TCG_REG_NONE, - TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); tcg_out_insn(s, RX, LA, TCG_REG_R3, addr_reg, TCG_REG_NONE, a_off); - tgen_andi(s, TCG_TYPE_I64, TCG_REG_R2, - (CPU_TLB_SIZE - 1) << CPU_TLB_ENTRY_BITS); tgen_andi(s, TCG_TYPE_TL, TCG_REG_R3, tlb_mask); } if (is_ld) { - ofs = offsetof(CPUArchState, tlb_table[mem_index][0].addr_read); + ofs = offsetof(CPUTLBEntry, addr_read); } else { - ofs = offsetof(CPUArchState, tlb_table[mem_index][0].addr_write); + ofs = offsetof(CPUTLBEntry, addr_write); } if (TARGET_LONG_BITS == 32) { - tcg_out_mem(s, RX_C, RXY_CY, TCG_REG_R3, TCG_REG_R2, TCG_AREG0, ofs); + tcg_out_insn(s, RX, C, TCG_REG_R3, TCG_REG_R2, TCG_REG_NONE, ofs); } else { - tcg_out_mem(s, 0, RXY_CG, TCG_REG_R3, TCG_REG_R2, TCG_AREG0, ofs); + tcg_out_insn(s, RXY, CG, TCG_REG_R3, TCG_REG_R2, TCG_REG_NONE, ofs); } - ofs = offsetof(CPUArchState, tlb_table[mem_index][0].addend); - tcg_out_mem(s, 0, RXY_LG, TCG_REG_R2, TCG_REG_R2, TCG_AREG0, ofs); + tcg_out_insn(s, RXY, LG, TCG_REG_R2, TCG_REG_R2, TCG_REG_NONE, + offsetof(CPUTLBEntry, addend)); if (TARGET_LONG_BITS == 32) { tgen_ext32u(s, TCG_REG_R3, addr_reg); From patchwork Wed Jan 23 22:57:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030235 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="EV4x986J"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLdM4JqRz9s7h for ; Thu, 24 Jan 2019 10:11:35 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43368 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRgD-0007Yu-Gd for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:11:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41045) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSy-0006EP-7p for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSk-0004KV-UP for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:43 -0500 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:38980) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSk-0004Bw-IW for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:38 -0500 Received: by mail-pf1-x442.google.com with SMTP id r136so1937336pfc.6 for ; Wed, 23 Jan 2019 14:57:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=c8rpHMW1a3L6wvNx+bLvONc9BqIsy+JAMLEXQpO6PDg=; b=EV4x986JDUkV9bNMvkj4ImLe+xaWTK00qJXkMyyX/uSuONzuoVjSM6/BA662X7wPEU BLuL3jpbeVHaaCMv8k77gwjv7ZjNQAQPvNoTKWOuLCXUV4fOCzpO9yhzZJ9lJWdqhLU8 AzyhJz4t3/Q3QSqN/jZp0e3T9wDLBADMxkT0k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=c8rpHMW1a3L6wvNx+bLvONc9BqIsy+JAMLEXQpO6PDg=; b=N1BMMXJ1K9uJG/gVxif4kqhU6njVwChkaVuMSPXjKsqzZX6V1URtnfBoGuFeW+fssc w+N9QYFDQCYdKzEhmY/hHZMaepJ2IMRruoJEMmY6AQMFi3gNl52epUOg90T8G7F3cd7D qPcyFanjeSntUUuFbpzVLlZw++grukohurJ68O3Cr/cMkD1ZiC+73h6Og4KWvKOtWzKa 4q0FvR4zYbnWchbCCXlRiiGpzTZBt20XZfkfR8ly+0eqFLc5Mq9txIh0XDHsnJBcKDEX NJuaiBzQCxnjfG5M5xcReGBEyioK+3bTSXISKVH6qOZlr1TOIQSALF2LTEW+11f5IHW3 yjvQ== X-Gm-Message-State: AJcUukfTHmu6tpXVb3H9uZU2LjsAaKL3jDzxB44I7jHhHuxEpWicx0gY JsR5K+kZtfbBFg0R07Dymy8QWxIVfwg= X-Google-Smtp-Source: ALg8bN7ECRyPOS0aLU2blpUDv3Z29OZ4JbjU21BsZvQZgX1kkyaPPCqY0JY1nYfDSEp9k+iVYv67FA== X-Received: by 2002:a62:de06:: with SMTP id h6mr4058721pfg.158.1548284238021; Wed, 23 Jan 2019 14:57:18 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.16 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:17 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:00 -0800 Message-Id: <20190123225705.28963-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH 08/13] tcg/riscv: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson Reviewed-by: Alistair Francis --- tcg/riscv/tcg-target.h | 2 +- tcg/riscv/tcg-target.inc.c | 126 ++++++++++++++++--------------------- 2 files changed, 56 insertions(+), 72 deletions(-) diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h index 1eb032626c..83b123ca03 100644 --- a/tcg/riscv/tcg-target.h +++ b/tcg/riscv/tcg-target.h @@ -33,7 +33,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 20 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/riscv/tcg-target.inc.c b/tcg/riscv/tcg-target.inc.c index 6cf8de32b5..b785f4acb7 100644 --- a/tcg/riscv/tcg-target.inc.c +++ b/tcg/riscv/tcg-target.inc.c @@ -958,6 +958,17 @@ static void * const qemu_st_helpers[16] = { [MO_BEQ] = helper_be_stq_mmu, }; +/* We don't support oversize guests */ +QEMU_BUILD_BUG_ON(TCG_TARGET_REG_BITS < TARGET_LONG_BITS); + +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + +/* We expect tlb_mask to be "near" tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) - + offsetof(CPUArchState, tlb_mask) >= 0x800); + static void tcg_out_tlb_load(TCGContext *s, TCGReg addrl, TCGReg addrh, TCGMemOpIdx oi, tcg_insn_unit **label_ptr, bool is_load) @@ -965,94 +976,67 @@ static void tcg_out_tlb_load(TCGContext *s, TCGReg addrl, TCGMemOp opc = get_memop(oi); unsigned s_bits = opc & MO_SIZE; unsigned a_bits = get_alignment_bits(opc); - target_ulong mask; + tcg_target_long compare_mask; int mem_index = get_mmuidx(oi); - int cmp_off - = (is_load - ? offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) - : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write)); - int add_off = offsetof(CPUArchState, tlb_table[mem_index][0].addend); - RISCVInsn load_cmp_op = (TARGET_LONG_BITS == 64 ? OPC_LD : - TCG_TARGET_REG_BITS == 64 ? OPC_LWU : OPC_LW); - RISCVInsn load_add_op = TCG_TARGET_REG_BITS == 64 ? OPC_LD : OPC_LW; - TCGReg base = TCG_AREG0; + int mask_off, table_off; + TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0; - /* We don't support oversize guests */ - if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { - g_assert_not_reached(); + mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + table_off = offsetof(CPUArchState, tlb_table[mem_index]); + if (table_off > 0x7ff) { + int mask_hi = mask_off - sextreg(mask_off, 0, 12); + int table_hi = table_off - sextreg(table_off, 0, 12); + + if (likely(mask_hi == table_hi)) { + mask_base = table_base = TCG_REG_TMP1; + tcg_out_opc_upper(s, OPC_LUI, mask_base, mask_hi); + tcg_out_opc_reg(s, OPC_ADD, mask_base, mask_base, TCG_AREG0); + mask_off -= mask_hi; + table_off -= mask_hi; + } else { + mask_base = TCG_REG_TMP0; + table_base = TCG_REG_TMP1; + tcg_out_opc_upper(s, OPC_LUI, mask_base, mask_hi); + tcg_out_opc_reg(s, OPC_ADD, mask_base, mask_base, TCG_AREG0); + table_off -= mask_off; + mask_off -= mask_hi; + tcg_out_opc_imm(s, OPC_ADDI, table_base, mask_base, mask_off); + } } + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, mask_base, mask_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP1, table_base, table_off); + + tcg_out_opc_imm(s, OPC_SRLI, TCG_REG_TMP2, addrl, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP2, TCG_REG_TMP2, TCG_REG_TMP0); + tcg_out_opc_reg(s, OPC_ADD, TCG_REG_TMP2, TCG_REG_TMP2, TCG_REG_TMP1); + + /* Load the tlb comparator and the addend. */ + tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_TMP0, TCG_REG_TMP2, + is_load ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write)); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP2, TCG_REG_TMP2, + offsetof(CPUTLBEntry, addend)); + /* We don't support unaligned accesses. */ if (a_bits < s_bits) { a_bits = s_bits; } - mask = (target_ulong)TARGET_PAGE_MASK | ((1 << a_bits) - 1); - - - /* Compensate for very large offsets. */ - if (add_off >= 0x1000) { - int adj; - base = TCG_REG_TMP2; - if (cmp_off <= 2 * 0xfff) { - adj = 0xfff; - tcg_out_opc_imm(s, OPC_ADDI, base, TCG_AREG0, adj); - } else { - adj = cmp_off - sextreg(cmp_off, 0, 12); - tcg_debug_assert(add_off - adj >= -0x1000 - && add_off - adj < 0x1000); - - tcg_out_opc_upper(s, OPC_LUI, base, adj); - tcg_out_opc_reg(s, OPC_ADD, base, base, TCG_AREG0); - } - add_off -= adj; - cmp_off -= adj; - } - - /* Extract the page index. */ - if (CPU_TLB_BITS + CPU_TLB_ENTRY_BITS < 12) { - tcg_out_opc_imm(s, OPC_SRLI, TCG_REG_TMP0, addrl, - TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); - tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP0, TCG_REG_TMP0, - MAKE_64BIT_MASK(CPU_TLB_ENTRY_BITS, CPU_TLB_BITS)); - } else if (TARGET_PAGE_BITS >= 12) { - tcg_out_opc_upper(s, OPC_LUI, TCG_REG_TMP0, - MAKE_64BIT_MASK(TARGET_PAGE_BITS, CPU_TLB_BITS)); - tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP0, TCG_REG_TMP0, addrl); - tcg_out_opc_imm(s, OPC_SRLI, TCG_REG_TMP0, TCG_REG_TMP0, - CPU_TLB_BITS - CPU_TLB_ENTRY_BITS); - } else { - tcg_out_opc_imm(s, OPC_SRLI, TCG_REG_TMP0, addrl, TARGET_PAGE_BITS); - tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP0, TCG_REG_TMP0, - MAKE_64BIT_MASK(0, CPU_TLB_BITS)); - tcg_out_opc_imm(s, OPC_SLLI, TCG_REG_TMP0, TCG_REG_TMP0, - CPU_TLB_ENTRY_BITS); - } - - /* Add that to the base address to index the tlb. */ - tcg_out_opc_reg(s, OPC_ADD, TCG_REG_TMP2, base, TCG_REG_TMP0); - base = TCG_REG_TMP2; - - /* Load the tlb comparator and the addend. */ - tcg_out_ldst(s, load_cmp_op, TCG_REG_TMP0, base, cmp_off); - tcg_out_ldst(s, load_add_op, TCG_REG_TMP2, base, add_off); - /* Clear the non-page, non-alignment bits from the address. */ - if (mask == sextreg(mask, 0, 12)) { - tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP1, addrl, mask); + compare_mask = (tcg_target_long)TARGET_PAGE_MASK | ((1 << a_bits) - 1); + if (compare_mask == sextreg(compare_mask, 0, 12)) { + tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP1, addrl, compare_mask); } else { - tcg_out_movi(s, TCG_TYPE_REG, TCG_REG_TMP1, mask); + tcg_out_movi(s, TCG_TYPE_TL, TCG_REG_TMP1, compare_mask); tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP1, TCG_REG_TMP1, addrl); - } + } /* Compare masked address with the TLB entry. */ label_ptr[0] = s->code_ptr; tcg_out_opc_branch(s, OPC_BNE, TCG_REG_TMP0, TCG_REG_TMP1, 0); /* NOP to allow patching later */ tcg_out_opc_imm(s, OPC_ADDI, TCG_REG_ZERO, TCG_REG_ZERO, 0); - /* TODO: Move this out of line - * see: - * https://lists.nongnu.org/archive/html/qemu-devel/2018-11/msg02234.html - */ /* TLB Hit - translate address using addend. */ if (TCG_TARGET_REG_BITS > TARGET_LONG_BITS) { From patchwork Wed Jan 23 22:57:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030234 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="X4Bf9Vt2"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLbX44x4z9s7h for ; Thu, 24 Jan 2019 10:10:00 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43330 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmReg-0006Yx-EG for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:09:58 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41043) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSy-0006EN-7a for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSk-0004K1-K3 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:43 -0500 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:37951) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSj-0004CT-W2 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:38 -0500 Received: by mail-pf1-x442.google.com with SMTP id q1so1934978pfi.5 for ; Wed, 23 Jan 2019 14:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=C2BrQaGdJrcnzidVlPeIc02gojy3eZ+R8AGWPoviy0s=; b=X4Bf9Vt2GvNffN5ZGUv4/4PMbpAV0cb4/asD5qMElC+y2xwOOk2H4Pysi5E9fhlLUv sPto/M4OKnKs3sCzgyhDAPYVuOr8wnH8rq8RX6jFHb8QSO+S384HKYQcShvGBEKQ/plw AUMGdnm+5IX0do95iXnEKoCSfkh3brHCU9ceE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=C2BrQaGdJrcnzidVlPeIc02gojy3eZ+R8AGWPoviy0s=; b=Uby6JBzm37l0+BPfGAatx2HPzf6j5p/oiJ/6idSj/9+Q2azhIhbB/2zDy8i2DgTO/C Oo5Q1GvzlrhqEi4qmu3NGsPLtwKoE6RE8kYqCf51qqyh1RQEWrD9+S4ZaWkTV5VFqIZy ZoH2wp2zFMEHCP1ueyGNJeh5ZjBWipVAoHxSG68yyQeCr/Y7eKVKdxl5kEbbwYTpbmRF SCcYUb/5Vb3Wjrjq5iYh14Rxh7pUDe/EwHC7ROazv0Md7uIwINzgZRcc/zFJdR3A0Bny iMiqzmsUopUSt5zmrgYh9cYwTyVQzKWO3+WVaNenvMfcRHFUZChamvkX5aIf67BLTE2s SqPQ== X-Gm-Message-State: AJcUukeNl+hejGSb0aCS5XwSI7KX+CBfqVKv2Ad2sBovFv+PMbWps/K/ AdawjIKMxIlnWp89mEYAtt/dt2LgAE0= X-Google-Smtp-Source: ALg8bN4qGyZB0FMi4GwTphEgtoMrxnCSyydzXWYIY0C3Od715trIh+kJroaJSISOychcscqzqgnF7g== X-Received: by 2002:a62:6047:: with SMTP id u68mr3866634pfb.239.1548284239410; Wed, 23 Jan 2019 14:57:19 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:18 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:01 -0800 Message-Id: <20190123225705.28963-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH 09/13] tcg/arm: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/arm/tcg-target.h | 2 +- tcg/arm/tcg-target.inc.c | 143 +++++++++++++++++++-------------------- 2 files changed, 72 insertions(+), 73 deletions(-) diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h index c5a7064bdc..679aaf097e 100644 --- a/tcg/arm/tcg-target.h +++ b/tcg/arm/tcg-target.h @@ -60,7 +60,7 @@ extern int arm_arch; #undef TCG_TARGET_STACK_GROWSUP #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum { TCG_REG_R0 = 0, diff --git a/tcg/arm/tcg-target.inc.c b/tcg/arm/tcg-target.inc.c index 49f57d655e..2245a8aeb9 100644 --- a/tcg/arm/tcg-target.inc.c +++ b/tcg/arm/tcg-target.inc.c @@ -500,6 +500,12 @@ static inline void tcg_out_ldrd_r(TCGContext *s, int cond, TCGReg rt, tcg_out_memop_r(s, cond, INSN_LDRD_REG, rt, rn, rm, 1, 1, 0); } +static inline void tcg_out_ldrd_rwb(TCGContext *s, int cond, TCGReg rt, + TCGReg rn, TCGReg rm) +{ + tcg_out_memop_r(s, cond, INSN_LDRD_REG, rt, rn, rm, 1, 1, 1); +} + static inline void tcg_out_strd_8(TCGContext *s, int cond, TCGReg rt, TCGReg rn, int imm8) { @@ -1229,8 +1235,13 @@ static TCGReg tcg_out_arg_reg64(TCGContext *s, TCGReg argreg, #define TLB_SHIFT (CPU_TLB_ENTRY_BITS + CPU_TLB_BITS) -/* We're expecting to use an 8-bit immediate and to mask. */ -QEMU_BUILD_BUG_ON(CPU_TLB_BITS > 8); +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + +/* We expect to use a 20-bit unsigned offset from ENV. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table[NB_MMU_MODES - 1]) + > 0xfffff); /* Load and compare a TLB entry, leaving the flags set. Returns the register containing the addend of the tlb entry. Clobbers R0, R1, R2, TMP. */ @@ -1238,84 +1249,72 @@ QEMU_BUILD_BUG_ON(CPU_TLB_BITS > 8); static TCGReg tcg_out_tlb_read(TCGContext *s, TCGReg addrlo, TCGReg addrhi, TCGMemOp opc, int mem_index, bool is_load) { - TCGReg base = TCG_AREG0; - int cmp_off = - (is_load - ? offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) - : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write)); - int add_off = offsetof(CPUArchState, tlb_table[mem_index][0].addend); - int mask_off; + int cmp_off = (is_load ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write)); + int mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_off = offsetof(CPUArchState, tlb_table[mem_index]); + TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0; unsigned s_bits = opc & MO_SIZE; unsigned a_bits = get_alignment_bits(opc); - /* V7 generates the following: - * ubfx r0, addrlo, #TARGET_PAGE_BITS, #CPU_TLB_BITS - * add r2, env, #high - * add r2, r2, r0, lsl #CPU_TLB_ENTRY_BITS - * ldr r0, [r2, #cmp] - * ldr r2, [r2, #add] - * movw tmp, #page_align_mask - * bic tmp, addrlo, tmp - * cmp r0, tmp - * - * Otherwise we generate: - * shr tmp, addrlo, #TARGET_PAGE_BITS - * add r2, env, #high - * and r0, tmp, #(CPU_TLB_SIZE - 1) - * add r2, r2, r0, lsl #CPU_TLB_ENTRY_BITS - * ldr r0, [r2, #cmp] - * ldr r2, [r2, #add] - * tst addrlo, #s_mask - * cmpeq r0, tmp, lsl #TARGET_PAGE_BITS - */ - if (use_armv7_instructions) { - tcg_out_extract(s, COND_AL, TCG_REG_R0, addrlo, - TARGET_PAGE_BITS, CPU_TLB_BITS); - } else { - tcg_out_dat_reg(s, COND_AL, ARITH_MOV, TCG_REG_TMP, - 0, addrlo, SHIFT_IMM_LSR(TARGET_PAGE_BITS)); - } + if (table_off > 0xfff) { + int mask_hi = mask_off & ~0xfff; + int table_hi = table_off & ~0xfff; + int rot; - /* Add portions of the offset until the memory access is in range. - * If we plan on using ldrd, reduce to an 8-bit offset; otherwise - * we can use a 12-bit offset. */ - if (use_armv6_instructions && TARGET_LONG_BITS == 64) { - mask_off = 0xff; - } else { - mask_off = 0xfff; - } - while (cmp_off > mask_off) { - int shift = ctz32(cmp_off & ~mask_off) & ~1; - int rot = ((32 - shift) << 7) & 0xf00; - int addend = cmp_off & (0xff << shift); - tcg_out_dat_imm(s, COND_AL, ARITH_ADD, TCG_REG_R2, base, - rot | ((cmp_off >> shift) & 0xff)); - base = TCG_REG_R2; - add_off -= addend; - cmp_off -= addend; - } - - if (!use_armv7_instructions) { - tcg_out_dat_imm(s, COND_AL, ARITH_AND, - TCG_REG_R0, TCG_REG_TMP, CPU_TLB_SIZE - 1); - } - tcg_out_dat_reg(s, COND_AL, ARITH_ADD, TCG_REG_R2, base, - TCG_REG_R0, SHIFT_IMM_LSL(CPU_TLB_ENTRY_BITS)); - - /* Load the tlb comparator. Use ldrd if needed and available, - but due to how the pointer needs setting up, ldm isn't useful. - Base arm5 doesn't have ldrd, but armv5te does. */ - if (use_armv6_instructions && TARGET_LONG_BITS == 64) { - tcg_out_ldrd_8(s, COND_AL, TCG_REG_R0, TCG_REG_R2, cmp_off); - } else { - tcg_out_ld32_12(s, COND_AL, TCG_REG_R0, TCG_REG_R2, cmp_off); - if (TARGET_LONG_BITS == 64) { - tcg_out_ld32_12(s, COND_AL, TCG_REG_R1, TCG_REG_R2, cmp_off + 4); + table_base = TCG_REG_R2; + if (mask_hi == table_hi) { + mask_base = table_base; + } else if (mask_hi) { + mask_base = TCG_REG_TMP; + rot = encode_imm(mask_hi); + assert(rot >= 0); + tcg_out_dat_imm(s, COND_AL, ARITH_ADD, mask_base, TCG_AREG0, + rotl(mask_hi, rot) | (rot << 7)); } + rot = encode_imm(table_hi); + assert(rot >= 0); + tcg_out_dat_imm(s, COND_AL, ARITH_ADD, table_base, TCG_AREG0, + rotl(table_hi, rot) | (rot << 7)); + + mask_off -= mask_hi; + table_off -= table_hi; + } + + /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ + tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_TMP, mask_base, mask_off); + tcg_out_ld(s, TCG_TYPE_I32, TCG_REG_R2, table_base, table_off); + + /* Extract the tlb index from the address into TMP. */ + tcg_out_dat_reg(s, COND_AL, ARITH_AND, TCG_REG_TMP, TCG_REG_TMP, addrlo, + SHIFT_IMM_LSR(TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS)); + + /* + * Add the tlb_table pointer, creating the CPUTLBEntry address in R2. + * Load the tlb comparator into R0/R1 and the fast path addend into R2. + */ + if (cmp_off == 0) { + if (use_armv6_instructions && TARGET_LONG_BITS == 64) { + tcg_out_ldrd_rwb(s, COND_AL, TCG_REG_R0, TCG_REG_R2, TCG_REG_TMP); + } else { + tcg_out_ld32_rwb(s, COND_AL, TCG_REG_R0, TCG_REG_R2, TCG_REG_TMP); + } + } else { + tcg_out_dat_reg(s, COND_AL, ARITH_ADD, + TCG_REG_R2, TCG_REG_R2, TCG_REG_TMP, 0); + if (use_armv6_instructions && TARGET_LONG_BITS == 64) { + tcg_out_ldrd_8(s, COND_AL, TCG_REG_R0, TCG_REG_R2, cmp_off); + } else { + tcg_out_ld32_12(s, COND_AL, TCG_REG_R0, TCG_REG_R2, cmp_off); + } + } + if (!use_armv6_instructions && TARGET_LONG_BITS == 64) { + tcg_out_ld32_12(s, COND_AL, TCG_REG_R1, TCG_REG_R2, cmp_off + 4); } /* Load the tlb addend. */ - tcg_out_ld32_12(s, COND_AL, TCG_REG_R2, TCG_REG_R2, add_off); + tcg_out_ld32_12(s, COND_AL, TCG_REG_R2, TCG_REG_R2, + offsetof(CPUTLBEntry, addend)); /* Check alignment. We don't support inline unaligned acceses, but we can easily support overalignment checks. */ From patchwork Wed Jan 23 22:57:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030224 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="OlBD0Bua"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLTN6Hjcz9s7h for ; Thu, 24 Jan 2019 10:04:40 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43254 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRZW-0002gS-Qh for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:04:38 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41034) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSw-0006Cc-9y for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSk-0004K7-Kh for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:41 -0500 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:45759) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSk-0004Cu-1Y for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:38 -0500 Received: by mail-pg1-x541.google.com with SMTP id y4so1734274pgc.12 for ; Wed, 23 Jan 2019 14:57:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+L0wBkxKTUeVI/04FDDLFSjFaBD1pRVqKJYILvbKu24=; b=OlBD0Bua1gJcAX5WS+M1Zh9KxF0qv49FfyIIcFTfnyt6vQ/aii+TD6wAQDSHGYdrLt zxI5iR4tgi6aMuTOODl8xxvBwU26/g3c8xF3OrilDw0DhNq+OQYfoEQJAZc5mW0dpnMn qEtvPftDHK7d7HRJJo5NiDlaw6VLPwdvL/RLU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+L0wBkxKTUeVI/04FDDLFSjFaBD1pRVqKJYILvbKu24=; b=C/sR/nPTqgMutzm62Mg4wZc46vuQu8r4MV+8tofcS0JpOT+jOzGWn0oYFekMs6QQnX cCf1a3jpi2SU+VCAc9mA2o+CbumpaSrq3rbNPiPieaP0lbho7uxZgY3XFqWJZktxidAD 4kkJBlu1nAztJ7AeC8ub3h9cRzeh/rrFNNmAOaKrcrU07fcslP2sXvzehpbHZvIQGafz owsmIODGOMKsf6AKZjuuTSXiEuJn9sZ1+D3TJZDnM6iTq/t+ZL+C032I+de3lU/zwDEd KrPG7iFMi09SnRjeMFoUPGL7LPyMHEbwiK5z5SuIBcZe0caQusY1FUiRp3mDHhy7g8fE /IGg== X-Gm-Message-State: AJcUukcIn5kGKSSP30RKuUFBKJARTfPifRUNv5LB/l47LmpHwUzX/6Mi unZp8GyUt7ojickZiHP5kRL3rtlAx84= X-Google-Smtp-Source: ALg8bN67876t5WCD27Nddu5eZAgINGDDvdQe1GHUDJF5MHkZYLT3uJr+DHkRi6jQvdD++8MnmyzLbg== X-Received: by 2002:a62:7504:: with SMTP id q4mr3881097pfc.180.1548284240643; Wed, 23 Jan 2019 14:57:20 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:19 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:02 -0800 Message-Id: <20190123225705.28963-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH 10/13] tcg/mips: Fix tcg_out_qemu_ld_slow_path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Patch the branch after it has been emitted rather than before it exists. Signed-off-by: Richard Henderson --- tcg/mips/tcg-target.inc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tcg/mips/tcg-target.inc.c b/tcg/mips/tcg-target.inc.c index be0bc92e8e..c5d7067f89 100644 --- a/tcg/mips/tcg-target.inc.c +++ b/tcg/mips/tcg-target.inc.c @@ -1343,8 +1343,9 @@ static void tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) } } - reloc_pc16(s->code_ptr, l->raddr); tcg_out_opc_br(s, OPC_BEQ, TCG_REG_ZERO, TCG_REG_ZERO); + reloc_pc16(s->code_ptr - 1, l->raddr); + /* delay slot */ if (TCG_TARGET_REG_BITS == 64 && l->type == TCG_TYPE_I32) { /* we always sign-extend 32-bit loads */ From patchwork Wed Jan 23 22:57:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030219 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="ExW3r3xQ"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLNZ0FPNz9s7h for ; Thu, 24 Jan 2019 10:00:30 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43190 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRVT-0007zx-VK for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:00:28 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41048) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSy-0006EQ-88 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSk-0004Jk-BX for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:43 -0500 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:39884) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSj-0004DP-Q4 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:38 -0500 Received: by mail-pl1-x641.google.com with SMTP id 101so1905425pld.6 for ; Wed, 23 Jan 2019 14:57:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+jpzDaVjPXUUB7/HTmam1U+UM2kIhlzghJN+I1PSa8c=; b=ExW3r3xQ+inVCL7kGvAj9KNVPvNbwxfUyBAmxGjGH64uCR8dCFWp5alvYz0Sng6TaL Tz9Bfcb5mHx9h1OoZM6OuvKlIyPf75H+nBse/4i9UujAQ9hvEtWZvfsO+vgsYS+RopMq 4572xa3PKwZNTu9HTVne2byVM3n0tNu88QKCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+jpzDaVjPXUUB7/HTmam1U+UM2kIhlzghJN+I1PSa8c=; b=tJFXkhydp/t+UCaXU2peOzXh8tekRoDSGOO0Gy46xVro0L6NPyxSGV+oV6GuQwF4a0 pBeTr39yn3nhqWKZk3kcill8HDVyscco191xCLFKMou3FiFOsC1Ewkvq+kTa//B0HPVP 6ggAc6lTUS0+v87mIAsNo58KAsoUa/ca/Z2FKxB4D3mT+DkNoU3Xek71LtDRn5I2LLqx IJ+6Gje2+8WkrSzCzo1Zp7sSh9ntColGynprvidFFUNZP8qD096124tUcA+lIcpWZ5pE LKoCy80uqL9isvysIMfM6K61cngLpkbOKtFiY3kErw3+Y6LTbE55HND53vCtRmTP1gHS nuXg== X-Gm-Message-State: AJcUukeWOn+Y0InfTlvMiULXOoXNK+jVBae+ivmRqgv2Yo3GOvzIxY8X ymfqnno2h8JcN1FUsdMlpoJZfT1vf5g= X-Google-Smtp-Source: ALg8bN4GA6g4b4jchj65F29tLj/DvDiiZ8TGMdGECkTHLlvj8e/+7vYg7kBzfpFGIyigzaIrNsU5uA== X-Received: by 2002:a17:902:8ec9:: with SMTP id x9mr4196958plo.27.1548284241746; Wed, 23 Jan 2019 14:57:21 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:21 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:03 -0800 Message-Id: <20190123225705.28963-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::641 Subject: [Qemu-devel] [PATCH 11/13] tcg/mips: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/mips/tcg-target.h | 2 +- tcg/mips/tcg-target.inc.c | 94 ++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 32 deletions(-) diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h index 8600eefd9a..40adbe38cb 100644 --- a/tcg/mips/tcg-target.h +++ b/tcg/mips/tcg-target.h @@ -37,7 +37,7 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/mips/tcg-target.inc.c b/tcg/mips/tcg-target.inc.c index c5d7067f89..8a92e916dd 100644 --- a/tcg/mips/tcg-target.inc.c +++ b/tcg/mips/tcg-target.inc.c @@ -1201,8 +1201,19 @@ static int tcg_out_call_iarg_reg2(TCGContext *s, int i, TCGReg al, TCGReg ah) return i; } -/* Perform the tlb comparison operation. The complete host address is - placed in BASE. Clobbers TMP0, TMP1, TMP2, A0. */ +/* We expect tlb_mask to be before tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) < + offsetof(CPUArchState, tlb_mask)); + +/* We expect tlb_mask to be "near" tlb_table. */ +QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table) - + offsetof(CPUArchState, tlb_mask) >= 0x8000); + +/* + * Perform the tlb comparison operation. + * The complete host address is placed in BASE. + * Clobbers TMP0, TMP1, TMP2, TMP3. + */ static void tcg_out_tlb_load(TCGContext *s, TCGReg base, TCGReg addrl, TCGReg addrh, TCGMemOpIdx oi, tcg_insn_unit *label_ptr[2], bool is_load) @@ -1210,52 +1221,73 @@ static void tcg_out_tlb_load(TCGContext *s, TCGReg base, TCGReg addrl, TCGMemOp opc = get_memop(oi); unsigned s_bits = opc & MO_SIZE; unsigned a_bits = get_alignment_bits(opc); - target_ulong mask; int mem_index = get_mmuidx(oi); - int cmp_off - = (is_load - ? offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) - : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write)); - int add_off = offsetof(CPUArchState, tlb_table[mem_index][0].addend); + int mask_off = offsetof(CPUArchState, tlb_mask[mem_index]); + int table_off = offsetof(CPUArchState, tlb_table[mem_index]); + int add_off = offsetof(CPUTLBEntry, addend); + int cmp_off = (is_load ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write)); + TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0; + target_ulong mask; - tcg_out_opc_sa(s, ALIAS_TSRL, TCG_REG_A0, addrl, - TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); - tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_A0, TCG_REG_A0, - (CPU_TLB_SIZE - 1) << CPU_TLB_ENTRY_BITS); - tcg_out_opc_reg(s, ALIAS_PADD, TCG_REG_A0, TCG_REG_A0, TCG_AREG0); + if (table_off > 0x7fff) { + int mask_hi = mask_off - (int16_t)mask_off; + int table_hi = table_off - (int16_t)table_off; - /* Compensate for very large offsets. */ - while (add_off >= 0x8000) { - /* Most target env are smaller than 32k, but a few are larger than 64k, - * so handle an arbitrarily large offset. - */ - tcg_out_opc_imm(s, ALIAS_PADDI, TCG_REG_A0, TCG_REG_A0, 0x7ff0); - cmp_off -= 0x7ff0; - add_off -= 0x7ff0; + table_base = TCG_TMP1; + if (likely(mask_hi == table_hi)) { + mask_base = table_base; + tcg_out_opc_imm(s, OPC_LUI, mask_base, TCG_REG_ZERO, mask_hi >> 16); + tcg_out_opc_reg(s, ALIAS_PADD, mask_base, mask_base, TCG_AREG0); + mask_off -= mask_hi; + table_off -= mask_hi; + } else { + if (mask_hi != 0) { + mask_base = TCG_TMP0; + tcg_out_opc_imm(s, OPC_LUI, + mask_base, TCG_REG_ZERO, mask_hi >> 16); + tcg_out_opc_reg(s, ALIAS_PADD, + mask_base, mask_base, TCG_AREG0); + } + table_off -= mask_off; + mask_off -= mask_hi; + tcg_out_opc_imm(s, ALIAS_PADDI, table_base, mask_base, mask_off); + } } + /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ + tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, mask_base, mask_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP1, table_base, table_off); + + /* Extract the TLB index from the address into TMP3. */ + tcg_out_opc_sa(s, ALIAS_TSRL, TCG_TMP3, addrl, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + tcg_out_opc_reg(s, OPC_AND, TCG_TMP3, TCG_TMP3, TCG_TMP0); + + /* Add the tlb_table pointer, creating the CPUTLBEntry address in TMP3. */ + tcg_out_opc_reg(s, ALIAS_PADD, TCG_TMP3, TCG_TMP3, TCG_TMP1); + /* We don't currently support unaligned accesses. We could do so with mips32r6. */ if (a_bits < s_bits) { a_bits = s_bits; } + /* Mask the page bits, keeping the alignment bits to compare against. */ mask = (target_ulong)TARGET_PAGE_MASK | ((1 << a_bits) - 1); - /* Load the (low half) tlb comparator. Mask the page bits, keeping the - alignment bits to compare against. */ + /* Load the (low-half) tlb comparator. */ if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { - tcg_out_ld(s, TCG_TYPE_I32, TCG_TMP0, TCG_REG_A0, cmp_off + LO_OFF); + tcg_out_ld(s, TCG_TYPE_I32, TCG_TMP0, TCG_TMP3, cmp_off + LO_OFF); tcg_out_movi(s, TCG_TYPE_I32, TCG_TMP1, mask); } else { - tcg_out_ldst(s, - (TARGET_LONG_BITS == 64 ? OPC_LD - : TCG_TARGET_REG_BITS == 64 ? OPC_LWU : OPC_LW), - TCG_TMP0, TCG_REG_A0, cmp_off); + tcg_out_ldst(s, (TARGET_LONG_BITS == 64 ? OPC_LD + : TCG_TARGET_REG_BITS == 64 ? OPC_LWU : OPC_LW), + TCG_TMP0, TCG_TMP3, cmp_off); tcg_out_movi(s, TCG_TYPE_TL, TCG_TMP1, mask); /* No second compare is required here; load the tlb addend for the fast path. */ - tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP2, TCG_REG_A0, add_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP2, TCG_TMP3, add_off); } tcg_out_opc_reg(s, OPC_AND, TCG_TMP1, TCG_TMP1, addrl); @@ -1271,10 +1303,10 @@ static void tcg_out_tlb_load(TCGContext *s, TCGReg base, TCGReg addrl, /* Load and test the high half tlb comparator. */ if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { /* delay slot */ - tcg_out_ld(s, TCG_TYPE_I32, TCG_TMP0, TCG_REG_A0, cmp_off + HI_OFF); + tcg_out_ld(s, TCG_TYPE_I32, TCG_TMP0, TCG_TMP3, cmp_off + HI_OFF); /* Load the tlb addend for the fast path. */ - tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP2, TCG_REG_A0, add_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP2, TCG_TMP3, add_off); label_ptr[1] = s->code_ptr; tcg_out_opc_br(s, OPC_BNE, addrh, TCG_TMP0); From patchwork Wed Jan 23 22:57:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030230 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="B9OSu6f+"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLZ33p4jz9sCX for ; Thu, 24 Jan 2019 10:08:42 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43322 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRdO-0005pP-Kp for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:08:38 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40967) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSl-00064Q-8R for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSk-0004Jg-BK for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:39886) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSj-0004Dx-T9 for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:38 -0500 Received: by mail-pl1-x643.google.com with SMTP id 101so1905450pld.6 for ; Wed, 23 Jan 2019 14:57:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=chBVMwzjNPannioQJTQaKUW8hZTcHq/vz45yGlCAQkY=; b=B9OSu6f+yRBG5eo5ICQw63LqHfoDnzFw/evOA2gTQk07WjNvBDgeVJ7PheaFXDLKbP UjAPGEb7Cw8a9tXY+19jP6/df9Ei0ywDlylsaWiIA8lS4IEYqRTfLAEKouaKMIflWH8p /XRUkYXS0iOzvArVvSF8f0oN88ocKG7bZ7mn4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=chBVMwzjNPannioQJTQaKUW8hZTcHq/vz45yGlCAQkY=; b=nhZG/mk9bwRe7AGA5GdSwzEHZeNtFw6+Ty5NHX+ebC8jFDqg4LPZ9Xj+oQDWXj2pUR 8C2SJQlAe5NWMWCYtxxxkHJqCguSVp6tKHFLP1iofY1VDO13R2DKDQfpZ/MlnvHWC4/q mJhrHMqGTHoEY7KknIWf9sxtuXsLFRzsKMOpwD511fXeFUOcXMP74wG8LMGCLIwqR90X QLGG3GexK93VbSx0Vuff2GmR9Aa+tTiFHatAuaGdeJhRcu2QUWxR+ivovmmDDuK/2agZ Ds4PiK8myRz7GLq9u8voCUY6mzlT6fauwWgY1ptFLMeNijITjA2Wnmnfh8IbUYrseXUu MGgA== X-Gm-Message-State: AJcUukdRbcik+0cx857S0x+BkpZhHFNvZOznaPbc77284hm2K31pbEBG o7V5JHpqFg4LkxWTPPbL1IXC90FXddo= X-Google-Smtp-Source: ALg8bN5A92CFmGjsZxXhNojK75KqCQyQasTcGvtf2QX+GXdCb084QYaK2383xzsF0j0/hFu6wMlKrA== X-Received: by 2002:a17:902:2a0a:: with SMTP id i10mr4071167plb.323.1548284243021; Wed, 23 Jan 2019 14:57:23 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:22 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:04 -0800 Message-Id: <20190123225705.28963-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH 12/13] tcg/tci: enable dynamic TLB sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is automatic due to TCI using the other softtlb macros. Signed-off-by: Richard Henderson --- tcg/tci/tcg-target.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h index 816dc4697c..d9a28752c1 100644 --- a/tcg/tci/tcg-target.h +++ b/tcg/tci/tcg-target.h @@ -43,7 +43,7 @@ #define TCG_TARGET_INTERPRETER 1 #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 0 +#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #if UINTPTR_MAX == UINT32_MAX # define TCG_TARGET_REG_BITS 32 From patchwork Wed Jan 23 22:57:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1030223 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="CLuUYy1H"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43lLSP6Y5jz9sCX for ; Thu, 24 Jan 2019 10:03:49 +1100 (AEDT) Received: from localhost ([127.0.0.1]:43248 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRYh-000228-4U for incoming@patchwork.ozlabs.org; Wed, 23 Jan 2019 18:03:47 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41044) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gmRSy-0006EO-7d for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gmRSl-0004LI-Gk for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:43 -0500 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]:46572) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gmRSl-0004EU-0C for qemu-devel@nongnu.org; Wed, 23 Jan 2019 17:57:39 -0500 Received: by mail-pg1-x52f.google.com with SMTP id w7so1732816pgp.13 for ; Wed, 23 Jan 2019 14:57:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rYE8/3dXrA20kZd6I0RIW4CEgJyz6Qhl0RlK6m6isgE=; b=CLuUYy1HVDSMgyW0REkFCXKwtP2tfoeZp4Z7CouP9RJcYL7luG9NPDI6S6G00Td9zB ZMLFwrKQkq7ZpALqHZhEfxxcTvuk7Uv8BN5dq2536y0ZTwaLXf2c/ZBVKzoI4GC/sevI JhkI+RhcyLuDeuF9EqLTkyXJZhoJZTCjOcbFs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rYE8/3dXrA20kZd6I0RIW4CEgJyz6Qhl0RlK6m6isgE=; b=o3HctHi6rGgGOdkapKAPF1kVWXiZ2KSp4e/dTTnzC2mE9mzexumihG5h5F44yVFTk1 wk4OFCQAGjbUmh1n6t/rzn9sL0afBPEvYkj42VznYPUazsKFrDaPj+KXW41sEWxvzHtO XEEQL+UKiKF2k8tnMkY5899nZZp2V9H/RynZpa2QN0nhd2EnPOwlgNg/4cPjTavDchyZ ve4LcXl7HXmKR63xnAhSexEa9J2H6OdcrSjZcTjnUj3WpfPl9aUcCsWg73yuYVn0m8eq LsF+6PHqod+FWGUAaZVGHk7TxiOcC/sk2GSdNq/eaumjtcZ0fWK5wwGkn7v7txLUT31X O8UQ== X-Gm-Message-State: AJcUukehJzJVtFJ7SLX8LNzeimChqlA+zHMdaZ94zV3d49hKo3X3pENi jUyabkl/6jd/eVeBqmgYYgC5KHLFkms= X-Google-Smtp-Source: ALg8bN4W0dtrCDuoiSpZVH4/PtDivMnlZhyOcfCFvMIdYBfId9pp1Dqqg4qEyVkwQenDW0CICOLGhg== X-Received: by 2002:a65:4646:: with SMTP id k6mr3650690pgr.153.1548284244189; Wed, 23 Jan 2019 14:57:24 -0800 (PST) Received: from cloudburst.twiddle.net (97-126-115-157.tukw.qwest.net. [97.126.115.157]) by smtp.gmail.com with ESMTPSA id y9sm23858132pfi.74.2019.01.23.14.57.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 14:57:23 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 23 Jan 2019 14:57:05 -0800 Message-Id: <20190123225705.28963-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190123225705.28963-1-richard.henderson@linaro.org> References: <20190123225705.28963-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52f Subject: [Qemu-devel] [PATCH 13/13] cputlb: Remove static tlb sizing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Now that all tcg backends support TCG_TARGET_IMPLEMENTS_DYN_TLB, remove the define and the old code. Signed-off-by: Richard Henderson Reviewed-by: Alistair Francis --- include/exec/cpu-defs.h | 46 ---------------------------------------- include/exec/cpu_ldst.h | 14 ------------ tcg/aarch64/tcg-target.h | 1 - tcg/arm/tcg-target.h | 1 - tcg/i386/tcg-target.h | 1 - tcg/mips/tcg-target.h | 1 - tcg/ppc/tcg-target.h | 1 - tcg/riscv/tcg-target.h | 1 - tcg/s390/tcg-target.h | 1 - tcg/sparc/tcg-target.h | 1 - tcg/tci/tcg-target.h | 1 - accel/tcg/cputlb.c | 21 ------------------ 12 files changed, 90 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index 191a1e021f..8f2a848bf5 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -67,11 +67,9 @@ typedef uint64_t target_ulong; #define CPU_TLB_ENTRY_BITS 5 #endif -#if TCG_TARGET_IMPLEMENTS_DYN_TLB #define CPU_TLB_DYN_MIN_BITS 6 #define CPU_TLB_DYN_DEFAULT_BITS 8 - # if HOST_LONG_BITS == 32 /* Make sure we do not require a double-word shift for the TLB load */ # define CPU_TLB_DYN_MAX_BITS (32 - TARGET_PAGE_BITS) @@ -87,41 +85,6 @@ typedef uint64_t target_ulong; MIN(22, TARGET_VIRT_ADDR_SPACE_BITS - TARGET_PAGE_BITS) # endif -#else /* !TCG_TARGET_IMPLEMENTS_DYN_TLB */ - -/* TCG_TARGET_TLB_DISPLACEMENT_BITS is used in CPU_TLB_BITS to ensure that - * the TLB is not unnecessarily small, but still small enough for the - * TLB lookup instruction sequence used by the TCG target. - * - * TCG will have to generate an operand as large as the distance between - * env and the tlb_table[NB_MMU_MODES - 1][0].addend. For simplicity, - * the TCG targets just round everything up to the next power of two, and - * count bits. This works because: 1) the size of each TLB is a largish - * power of two, 2) and because the limit of the displacement is really close - * to a power of two, 3) the offset of tlb_table[0][0] inside env is smaller - * than the size of a TLB. - * - * For example, the maximum displacement 0xFFF0 on PPC and MIPS, but TCG - * just says "the displacement is 16 bits". TCG_TARGET_TLB_DISPLACEMENT_BITS - * then ensures that tlb_table at least 0x8000 bytes large ("not unnecessarily - * small": 2^15). The operand then will come up smaller than 0xFFF0 without - * any particular care, because the TLB for a single MMU mode is larger than - * 0x10000-0xFFF0=16 bytes. In the end, the maximum value of the operand - * could be something like 0xC000 (the offset of the last TLB table) plus - * 0x18 (the offset of the addend field in each TLB entry) plus the offset - * of tlb_table inside env (which is non-trivial but not huge). - */ -#define CPU_TLB_BITS \ - MIN(8, \ - TCG_TARGET_TLB_DISPLACEMENT_BITS - CPU_TLB_ENTRY_BITS - \ - (NB_MMU_MODES <= 1 ? 0 : \ - NB_MMU_MODES <= 2 ? 1 : \ - NB_MMU_MODES <= 4 ? 2 : \ - NB_MMU_MODES <= 8 ? 3 : 4)) - -#define CPU_TLB_SIZE (1 << CPU_TLB_BITS) -#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ - typedef struct CPUTLBEntry { /* bit TARGET_LONG_BITS to TARGET_PAGE_BITS : virtual address bit TARGET_PAGE_BITS-1..4 : Nonzero for accesses that should not @@ -187,10 +150,8 @@ typedef struct CPUTLBDesc { target_ulong large_page_mask; /* The next index to use in the tlb victim table. */ size_t vindex; -#if TCG_TARGET_IMPLEMENTS_DYN_TLB CPUTLBWindow window; size_t n_used_entries; -#endif } CPUTLBDesc; /* @@ -215,19 +176,12 @@ typedef struct CPUTLBCommon { size_t elide_flush_count; } CPUTLBCommon; -#if TCG_TARGET_IMPLEMENTS_DYN_TLB # define CPU_TLB \ /* tlb_mask[i] contains (n_entries - 1) << CPU_TLB_ENTRY_BITS */ \ uintptr_t tlb_mask[NB_MMU_MODES]; \ CPUTLBEntry *tlb_table[NB_MMU_MODES]; # define CPU_IOTLB \ CPUIOTLBEntry *iotlb[NB_MMU_MODES]; -#else -# define CPU_TLB \ - CPUTLBEntry tlb_table[NB_MMU_MODES][CPU_TLB_SIZE]; -# define CPU_IOTLB \ - CPUIOTLBEntry iotlb[NB_MMU_MODES][CPU_TLB_SIZE]; -#endif /* * The meaning of each of the MMU modes is defined in the target code. diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 83b2907d86..d78041d7a0 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -135,7 +135,6 @@ static inline target_ulong tlb_addr_write(const CPUTLBEntry *entry) #endif } -#if TCG_TARGET_IMPLEMENTS_DYN_TLB /* Find the TLB index corresponding to the mmu_idx + address pair. */ static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, target_ulong addr) @@ -149,19 +148,6 @@ static inline size_t tlb_n_entries(CPUArchState *env, uintptr_t mmu_idx) { return (env->tlb_mask[mmu_idx] >> CPU_TLB_ENTRY_BITS) + 1; } -#else -/* Find the TLB index corresponding to the mmu_idx + address pair. */ -static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, - target_ulong addr) -{ - return (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); -} - -static inline size_t tlb_n_entries(CPUArchState *env, uintptr_t mmu_idx) -{ - return CPU_TLB_SIZE; -} -#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ /* Find the TLB entry corresponding to the mmu_idx + address pair. */ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 5085a81060..2d93cf404e 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -15,7 +15,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 24 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #undef TCG_TARGET_STACK_GROWSUP typedef enum { diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h index 679aaf097e..16172f73a3 100644 --- a/tcg/arm/tcg-target.h +++ b/tcg/arm/tcg-target.h @@ -60,7 +60,6 @@ extern int arm_arch; #undef TCG_TARGET_STACK_GROWSUP #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum { TCG_REG_R0 = 0, diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index eb40312e67..7995fe3eab 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -27,7 +27,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 31 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #ifdef __x86_64__ # define TCG_TARGET_REG_BITS 64 diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h index 40adbe38cb..5cb8672470 100644 --- a/tcg/mips/tcg-target.h +++ b/tcg/mips/tcg-target.h @@ -37,7 +37,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index 95b735b0bb..52c1bb04b1 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -34,7 +34,6 @@ #define TCG_TARGET_NB_REGS 32 #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 16 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum { TCG_REG_R0, TCG_REG_R1, TCG_REG_R2, TCG_REG_R3, diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h index 83b123ca03..60918cacb4 100644 --- a/tcg/riscv/tcg-target.h +++ b/tcg/riscv/tcg-target.h @@ -33,7 +33,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 20 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/s390/tcg-target.h b/tcg/s390/tcg-target.h index 357528dd97..853ed6e7aa 100644 --- a/tcg/s390/tcg-target.h +++ b/tcg/s390/tcg-target.h @@ -27,7 +27,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 2 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 19 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 typedef enum TCGReg { TCG_REG_R0 = 0, diff --git a/tcg/sparc/tcg-target.h b/tcg/sparc/tcg-target.h index 6020a670c0..a0ed2a3342 100644 --- a/tcg/sparc/tcg-target.h +++ b/tcg/sparc/tcg-target.h @@ -29,7 +29,6 @@ #define TCG_TARGET_INSN_UNIT_SIZE 4 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #define TCG_TARGET_NB_REGS 32 typedef enum { diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h index d9a28752c1..086f34e69a 100644 --- a/tcg/tci/tcg-target.h +++ b/tcg/tci/tcg-target.h @@ -43,7 +43,6 @@ #define TCG_TARGET_INTERPRETER 1 #define TCG_TARGET_INSN_UNIT_SIZE 1 #define TCG_TARGET_TLB_DISPLACEMENT_BITS 32 -#define TCG_TARGET_IMPLEMENTS_DYN_TLB 1 #if UINTPTR_MAX == UINT32_MAX # define TCG_TARGET_REG_BITS 32 diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index a3a1614f0e..dad9b7796c 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -74,7 +74,6 @@ QEMU_BUILD_BUG_ON(sizeof(target_ulong) > sizeof(run_on_cpu_data)); QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16); #define ALL_MMUIDX_BITS ((1 << NB_MMU_MODES) - 1) -#if TCG_TARGET_IMPLEMENTS_DYN_TLB static inline size_t sizeof_tlb(CPUArchState *env, uintptr_t mmu_idx) { return env->tlb_mask[mmu_idx] + (1 << CPU_TLB_ENTRY_BITS); @@ -235,26 +234,6 @@ static inline void tlb_n_used_entries_dec(CPUArchState *env, uintptr_t mmu_idx) env->tlb_d[mmu_idx].n_used_entries--; } -#else /* !TCG_TARGET_IMPLEMENTS_DYN_TLB */ - -static inline void tlb_dyn_init(CPUArchState *env) -{ -} - -static inline void tlb_table_flush_by_mmuidx(CPUArchState *env, int mmu_idx) -{ - memset(env->tlb_table[mmu_idx], -1, sizeof(env->tlb_table[0])); -} - -static inline void tlb_n_used_entries_inc(CPUArchState *env, uintptr_t mmu_idx) -{ -} - -static inline void tlb_n_used_entries_dec(CPUArchState *env, uintptr_t mmu_idx) -{ -} -#endif /* TCG_TARGET_IMPLEMENTS_DYN_TLB */ - void tlb_init(CPUState *cpu) { CPUArchState *env = cpu->env_ptr;