From patchwork Mon Jun 15 11:51:24 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alvise Rigo X-Patchwork-Id: 484229 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 74844140273 for ; Mon, 15 Jun 2015 21:52:32 +1000 (AEST) Received: from localhost ([::1]:33573 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4SwI-0000II-My for incoming@patchwork.ozlabs.org; Mon, 15 Jun 2015 07:52:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4Str-0004h5-8m for qemu-devel@nongnu.org; Mon, 15 Jun 2015 07:50:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z4Stm-00040k-1c for qemu-devel@nongnu.org; Mon, 15 Jun 2015 07:49:59 -0400 Received: from mail-wi0-f174.google.com ([209.85.212.174]:35678) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4Stl-00040Y-PB for qemu-devel@nongnu.org; Mon, 15 Jun 2015 07:49:53 -0400 Received: by wiga1 with SMTP id a1so74764380wig.0 for ; Mon, 15 Jun 2015 04:49:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OGJ31KBXWTyS+JjjoPyTTnsMjYhc18Wr5TUDwjJuCQA=; b=Pua5+XFLQ/TzViE3T04VNUCBgh0W0hBkig0pPYlFEDGtlT0WlA5j3dR/WRPs2ioQSY IXDIwQ7rBe22Zew0jyICW8DU0VuWBt4v3eI4umVWKwYDmip9+SG/saHkm9FipfY8jfh0 Uy1eg9v6YL0xq+fSp96eaq+QXmDoqSiFHuh14UnRYgYh6Sb4J2gTu/gtn3B+zSR/krbS RC7BSVpJ8Ga5uDGXU8W3Yvg8wyAww7grlOCVvU5KX/WJDyxlxz3M3lvo1x9qphAtm+tx E8k/ujK7vMZ9Jbjk5zOOtl/5RqEKw/osDzMqQ1uw4V0EQ4Zitni9zPVVtBctxjSugXk9 +1ug== X-Gm-Message-State: ALoCoQmcxw3eN/+nPKPOug0ylRS3Mk6bQw7pEY5sc7SwrjNJFUrhBR21crBHli8QFA6nyl/QgmzE X-Received: by 10.194.240.8 with SMTP id vw8mr44833549wjc.114.1434368993157; Mon, 15 Jun 2015 04:49:53 -0700 (PDT) Received: from linarch.home (LPuteaux-656-1-278-113.w80-15.abo.wanadoo.fr. [80.15.154.113]) by mx.google.com with ESMTPSA id tl3sm18570524wjc.20.2015.06.15.04.49.51 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 15 Jun 2015 04:49:51 -0700 (PDT) From: Alvise Rigo To: qemu-devel@nongnu.org Date: Mon, 15 Jun 2015 13:51:24 +0200 Message-Id: <1434369088-15076-4-git-send-email-a.rigo@virtualopensystems.com> X-Mailer: git-send-email 2.4.3 In-Reply-To: <1434369088-15076-1-git-send-email-a.rigo@virtualopensystems.com> References: <1434369088-15076-1-git-send-email-a.rigo@virtualopensystems.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.85.212.174 Cc: mttcg@listserver.greensocs.com, claudio.fontana@huawei.com, cota@braap.org, jani.kokkonen@huawei.com, tech@virtualopensystems.com, alex.bennee@linaro.org, rth@twiddle.net Subject: [Qemu-devel] [RFC v2 3/7] Add new TLB_EXCL flag X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Add a new flag for the TLB entries to force all the accesses made to a page to follow the slow-path. In the case we remove a TLB entry marked as EXCL, we unset the corresponding exclusive bit in the bitmap. Mark the accessed page as dirty to invalidate any pending operation of LL/SC only if a vCPU writes to the protected address. Suggested-by: Jani Kokkonen Suggested-by: Claudio Fontana Signed-off-by: Alvise Rigo --- cputlb.c | 18 ++++- include/exec/cpu-all.h | 2 + include/exec/cpu-defs.h | 4 ++ softmmu_template.h | 187 ++++++++++++++++++++++++++++++------------------ 4 files changed, 142 insertions(+), 69 deletions(-) diff --git a/cputlb.c b/cputlb.c index a506086..630c11c 100644 --- a/cputlb.c +++ b/cputlb.c @@ -299,6 +299,16 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, env->tlb_v_table[mmu_idx][vidx] = *te; env->iotlb_v[mmu_idx][vidx] = env->iotlb[mmu_idx][index]; + if (te->addr_write & TLB_EXCL) { + /* We are removing an exclusive entry, if the corresponding exclusive + * bit is set, unset it. */ + hwaddr hw_addr = (env->iotlb[mmu_idx][index].addr & TARGET_PAGE_MASK) + + (te->addr_write & TARGET_PAGE_MASK); + if (cpu_physical_memory_excl_is_dirty(hw_addr)) { + cpu_physical_memory_set_excl_dirty(hw_addr); + } + } + /* refill the tlb */ env->iotlb[mmu_idx][index].addr = iotlb - vaddr; env->iotlb[mmu_idx][index].attrs = attrs; @@ -324,7 +334,13 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, + xlat)) { te->addr_write = address | TLB_NOTDIRTY; } else { - te->addr_write = address; + if (!(address & TLB_MMIO) && + !cpu_physical_memory_excl_is_dirty(section->mr->ram_addr + + xlat)) { + te->addr_write = address | TLB_EXCL; + } else { + te->addr_write = address; + } } } else { te->addr_write = -1; diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index ac06c67..632f6ce 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -311,6 +311,8 @@ extern RAMList ram_list; #define TLB_NOTDIRTY (1 << 4) /* Set if TLB entry is an IO callback. */ #define TLB_MMIO (1 << 5) +/* Set if TLB entry refers a page that requires exclusive access. */ +#define TLB_EXCL (1 << 6) void dump_exec_info(FILE *f, fprintf_function cpu_fprintf); void dump_opcount_info(FILE *f, fprintf_function cpu_fprintf); diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index d5aecaf..c73a75f 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -165,5 +165,9 @@ typedef struct CPUIOTLBEntry { #define CPU_COMMON \ /* soft mmu support */ \ CPU_COMMON_TLB \ + \ + /* Used for atomic instruction translation. */ \ + bool ll_sc_context; \ + hwaddr excl_protected_hwaddr; \ #endif diff --git a/softmmu_template.h b/softmmu_template.h index 39f571b..f1782f6 100644 --- a/softmmu_template.h +++ b/softmmu_template.h @@ -141,6 +141,21 @@ vidx >= 0; \ }) +#define lookup_cpus_ll_addr(addr) \ +({ \ + CPUState *cpu; \ + bool hit = false; \ + \ + CPU_FOREACH(cpu) { \ + if (cpu != current_cpu && env->excl_protected_hwaddr == addr) { \ + hit = true; \ + break; \ + } \ + } \ + \ + hit; \ +}) + #ifndef SOFTMMU_CODE_ACCESS static inline DATA_TYPE glue(io_read, SUFFIX)(CPUArchState *env, CPUIOTLBEntry *iotlbentry, @@ -409,43 +424,61 @@ void helper_le_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, tlb_addr = env->tlb_table[mmu_idx][index].addr_write; } - /* Handle an IO access. */ + /* Handle an IO access or exclusive access. */ if (unlikely(tlb_addr & ~TARGET_PAGE_MASK)) { - CPUIOTLBEntry *iotlbentry; - if ((addr & (DATA_SIZE - 1)) != 0) { - goto do_unaligned_access; - } - iotlbentry = &env->iotlb[mmu_idx][index]; - - /* ??? Note that the io helpers always read data in the target - byte ordering. We should push the LE/BE request down into io. */ - val = TGT_LE(val); - glue(io_write, SUFFIX)(env, iotlbentry, val, addr, retaddr); - return; - } - - /* Handle slow unaligned access (it spans two pages or IO). */ - if (DATA_SIZE > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 - >= TARGET_PAGE_SIZE)) { - int i; - do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); + CPUIOTLBEntry *iotlbentry = &env->iotlb[mmu_idx][index]; + if ((tlb_addr & ~TARGET_PAGE_MASK) == TLB_EXCL) { + /* The slow-path has been forced since we are writing to + * exclusive-protected memory. */ + hwaddr hw_addr = (iotlbentry->addr & TARGET_PAGE_MASK) + addr; + + bool set_to_dirty; + + /* Two cases of invalidation: the current vCPU is writing to another + * vCPU's exclusive address or the vCPU that issued the LoadLink is + * writing to it, but not through a StoreCond. */ + set_to_dirty = lookup_cpus_ll_addr(hw_addr); + set_to_dirty |= env->ll_sc_context && + (env->excl_protected_hwaddr == hw_addr); + + if (set_to_dirty) { + cpu_physical_memory_set_excl_dirty(hw_addr); + } /* the vCPU is legitimately writing to the protected address */ + } else { + if ((addr & (DATA_SIZE - 1)) != 0) { + goto do_unaligned_access; + } + + /* ??? Note that the io helpers always read data in the target + byte ordering. We should push the LE/BE request down into io. */ + val = TGT_LE(val); + glue(io_write, SUFFIX)(env, iotlbentry, val, addr, retaddr); + return; } - /* XXX: not efficient, but simple */ - /* Note: relies on the fact that tlb_fill() does not remove the - * previous page from the TLB cache. */ - for (i = DATA_SIZE - 1; i >= 0; i--) { - /* Little-endian extract. */ - uint8_t val8 = val >> (i * 8); - /* Note the adjustment at the beginning of the function. - Undo that for the recursion. */ - glue(helper_ret_stb, MMUSUFFIX)(env, addr + i, val8, - oi, retaddr + GETPC_ADJ); + } else { + /* Handle slow unaligned access (it spans two pages or IO). */ + if (DATA_SIZE > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 + >= TARGET_PAGE_SIZE)) { + int i; + do_unaligned_access: + if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, + mmu_idx, retaddr); + } + /* XXX: not efficient, but simple */ + /* Note: relies on the fact that tlb_fill() does not remove the + * previous page from the TLB cache. */ + for (i = DATA_SIZE - 1; i >= 0; i--) { + /* Little-endian extract. */ + uint8_t val8 = val >> (i * 8); + /* Note the adjustment at the beginning of the function. + Undo that for the recursion. */ + glue(helper_ret_stb, MMUSUFFIX)(env, addr + i, val8, + oi, retaddr + GETPC_ADJ); + } + return; } - return; } /* Handle aligned access or unaligned access in the same page. */ @@ -489,43 +522,61 @@ void helper_be_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, tlb_addr = env->tlb_table[mmu_idx][index].addr_write; } - /* Handle an IO access. */ + /* Handle an IO access or exclusive access. */ if (unlikely(tlb_addr & ~TARGET_PAGE_MASK)) { - CPUIOTLBEntry *iotlbentry; - if ((addr & (DATA_SIZE - 1)) != 0) { - goto do_unaligned_access; - } - iotlbentry = &env->iotlb[mmu_idx][index]; - - /* ??? Note that the io helpers always read data in the target - byte ordering. We should push the LE/BE request down into io. */ - val = TGT_BE(val); - glue(io_write, SUFFIX)(env, iotlbentry, val, addr, retaddr); - return; - } - - /* Handle slow unaligned access (it spans two pages or IO). */ - if (DATA_SIZE > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 - >= TARGET_PAGE_SIZE)) { - int i; - do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); + CPUIOTLBEntry *iotlbentry = &env->iotlb[mmu_idx][index]; + if ((tlb_addr & ~TARGET_PAGE_MASK) == TLB_EXCL) { + /* The slow-path has been forced since we are writing to + * exclusive-protected memory. */ + hwaddr hw_addr = (iotlbentry->addr & TARGET_PAGE_MASK) + addr; + + bool set_to_dirty; + + /* Two cases of invalidation: the current vCPU is writing to another + * vCPU's exclusive address or the vCPU that issued the LoadLink is + * writing to it, but not through a StoreCond. */ + set_to_dirty = lookup_cpus_ll_addr(hw_addr); + set_to_dirty |= env->ll_sc_context && + (env->excl_protected_hwaddr == hw_addr); + + if (set_to_dirty) { + cpu_physical_memory_set_excl_dirty(hw_addr); + } /* the vCPU is legitimately writing to the protected address */ + } else { + if ((addr & (DATA_SIZE - 1)) != 0) { + goto do_unaligned_access; + } + + /* ??? Note that the io helpers always read data in the target + byte ordering. We should push the LE/BE request down into io. */ + val = TGT_BE(val); + glue(io_write, SUFFIX)(env, iotlbentry, val, addr, retaddr); + return; } - /* XXX: not efficient, but simple */ - /* Note: relies on the fact that tlb_fill() does not remove the - * previous page from the TLB cache. */ - for (i = DATA_SIZE - 1; i >= 0; i--) { - /* Big-endian extract. */ - uint8_t val8 = val >> (((DATA_SIZE - 1) * 8) - (i * 8)); - /* Note the adjustment at the beginning of the function. - Undo that for the recursion. */ - glue(helper_ret_stb, MMUSUFFIX)(env, addr + i, val8, - oi, retaddr + GETPC_ADJ); + } else { + /* Handle slow unaligned access (it spans two pages or IO). */ + if (DATA_SIZE > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 + >= TARGET_PAGE_SIZE)) { + int i; + do_unaligned_access: + if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, + mmu_idx, retaddr); + } + /* XXX: not efficient, but simple */ + /* Note: relies on the fact that tlb_fill() does not remove the + * previous page from the TLB cache. */ + for (i = DATA_SIZE - 1; i >= 0; i--) { + /* Big-endian extract. */ + uint8_t val8 = val >> (((DATA_SIZE - 1) * 8) - (i * 8)); + /* Note the adjustment at the beginning of the function. + Undo that for the recursion. */ + glue(helper_ret_stb, MMUSUFFIX)(env, addr + i, val8, + oi, retaddr + GETPC_ADJ); + } + return; } - return; } /* Handle aligned access or unaligned access in the same page. */