diff mbox series

[4/8] Hugetlb: Migrating libhugetlbfs icache-hygiene

Message ID 20221201122844.142062-5-tsahu@linux.ibm.com
State Changes Requested
Headers show
Series Hugetlb:Migrating the libhugetlbfs tests | expand

Commit Message

Tarun Sahu Dec. 1, 2022, 12:28 p.m. UTC
Migrating the libhugetlbfs/testcases/icache-hygiene.c test

Test Description: Older ppc64 kernels don't properly flush dcache to
icache before giving a cleared page to userspace.  With some exceedingly
hairy code, this attempts to test for this bug.

This test will never trigger (obviously) on machines with coherent
icache and dcache (including x86 and POWER5).  On any given run,
even on a buggy kernel there's a chance the bug won't trigger -
either because we don't get the same physical page back when we
remap, or because the icache happens to get flushed in the interim.

Signed-off-by: Tarun Sahu <tsahu@linux.ibm.com>
---
 runtest/hugetlb                               |   1 +
 testcases/kernel/mem/.gitignore               |   1 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap15.c  | 239 ++++++++++++++++++
 3 files changed, 241 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap15.c

Comments

Richard Palethorpe Dec. 12, 2022, 2:08 p.m. UTC | #1
Hello,

Tarun Sahu <tsahu@linux.ibm.com> writes:

> Migrating the libhugetlbfs/testcases/icache-hygiene.c test
>
> Test Description: Older ppc64 kernels don't properly flush dcache to
> icache before giving a cleared page to userspace.  With some exceedingly
> hairy code, this attempts to test for this bug.
>
> This test will never trigger (obviously) on machines with coherent
> icache and dcache (including x86 and POWER5).  On any given run,
> even on a buggy kernel there's a chance the bug won't trigger -
> either because we don't get the same physical page back when we
> remap, or because the icache happens to get flushed in the interim.

Compiling with Clang 15.0.5 on x86_64 and upstream kernel 6.0.9 in a KVM
VM the test fails:

sh-5.2# ./hugemmap15
[13712.044617] hugemmap15 (2118): drop_caches: 3
tst_hugepage.c:83: TINFO: 3 hugepage(s) reserved
tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
hugemmap15.c:191: TFAIL: icache unclean

> +
> +static struct tst_test test = {
> +	.needs_root = 1,
> +	.mntpoint = MNTPOINT,
> +	.needs_hugetlbfs = 1,
> +	.needs_tmpdir = 1,
> +	.setup = setup,
> +	.cleanup = cleanup,
> +	.test_all = run_test,
> +	.hugepages = {3, TST_NEEDS},
> +};
> -- 
> 2.31.1

Again, it would be nice to have the git tag(s) of the fix commit(s) if
possible.
Tarun Sahu Dec. 13, 2022, 5:27 p.m. UTC | #2
Hi Richard,
Thanks for looking at it.

I am also trying to reproduce the same, In the meantime, I have found the
fix commit for this bug: https://lore.kernel.org/all/20051209032051.GA11744@localhost.localdomain/

I will update the tag in this and all tests moving forward,
Pardon me for missing such crucial details.

Thanks
Taurn
On Dec 12 2022, Richard Palethorpe wrote:
> Hello,
> 
> Tarun Sahu <tsahu@linux.ibm.com> writes:
> 
> > Migrating the libhugetlbfs/testcases/icache-hygiene.c test
> >
> > Test Description: Older ppc64 kernels don't properly flush dcache to
> > icache before giving a cleared page to userspace.  With some exceedingly
> > hairy code, this attempts to test for this bug.
> >
> > This test will never trigger (obviously) on machines with coherent
> > icache and dcache (including x86 and POWER5).  On any given run,
> > even on a buggy kernel there's a chance the bug won't trigger -
> > either because we don't get the same physical page back when we
> > remap, or because the icache happens to get flushed in the interim.
> 
> Compiling with Clang 15.0.5 on x86_64 and upstream kernel 6.0.9 in a KVM
> VM the test fails:
> 
> sh-5.2# ./hugemmap15
> [13712.044617] hugemmap15 (2118): drop_caches: 3
> tst_hugepage.c:83: TINFO: 3 hugepage(s) reserved
> tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
> hugemmap15.c:191: TFAIL: icache unclean
> 
> > +
> > +static struct tst_test test = {
> > +	.needs_root = 1,
> > +	.mntpoint = MNTPOINT,
> > +	.needs_hugetlbfs = 1,
> > +	.needs_tmpdir = 1,
> > +	.setup = setup,
> > +	.cleanup = cleanup,
> > +	.test_all = run_test,
> > +	.hugepages = {3, TST_NEEDS},
> > +};
> > -- 
> > 2.31.1
> 
> Again, it would be nice to have the git tag(s) of the fix commit(s) if
> possible.
> 
> -- 
> Thank you,
> Richard.
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp
Tarun Sahu Dec. 19, 2022, 9:01 p.m. UTC | #3
On Dec 13 2022, Tarun Sahu wrote:
Hi,

with clang (any version) it even fails on power (with/without VM).
while debugging with lldb, I found clang/llvm used an optmization which
caused goto *p to be ignored and it just jump to jumplabel. Tunrning off
the optimization makes the test work as expected.

I will post the v2 with change in Makefile for this test.

--- a/testcases/kernel/mem/hugetlb/hugemmap/Makefile
+++ b/testcases/kernel/mem/hugetlb/hugemmap/Makefile
@@ -8,4 +8,5 @@ include $(top_srcdir)/include/mk/testcases.mk
 include $(abs_srcdir)/../Makefile.inc
 include $(top_srcdir)/include/mk/generic_leaf_target.mk

+hugemmap15: CFLAGS+=-O0

Thanks
Tarun

> Hi Richard,
> Thanks for looking at it.
> 
> I am also trying to reproduce the same, In the meantime, I have found the
> fix commit for this bug: https://lore.kernel.org/all/20051209032051.GA11744@localhost.localdomain/
> 
> I will update the tag in this and all tests moving forward,
> Pardon me for missing such crucial details.
> 
> Thanks
> Taurn
> On Dec 12 2022, Richard Palethorpe wrote:
> > Hello,
> > 
> > Tarun Sahu <tsahu@linux.ibm.com> writes:
> > 
> > > Migrating the libhugetlbfs/testcases/icache-hygiene.c test
> > >
> > > Test Description: Older ppc64 kernels don't properly flush dcache to
> > > icache before giving a cleared page to userspace.  With some exceedingly
> > > hairy code, this attempts to test for this bug.
> > >
> > > This test will never trigger (obviously) on machines with coherent
> > > icache and dcache (including x86 and POWER5).  On any given run,
> > > even on a buggy kernel there's a chance the bug won't trigger -
> > > either because we don't get the same physical page back when we
> > > remap, or because the icache happens to get flushed in the interim.
> > 
> > Compiling with Clang 15.0.5 on x86_64 and upstream kernel 6.0.9 in a KVM
> > VM the test fails:
> > 
> > sh-5.2# ./hugemmap15
> > [13712.044617] hugemmap15 (2118): drop_caches: 3
> > tst_hugepage.c:83: TINFO: 3 hugepage(s) reserved
> > tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
> > hugemmap15.c:191: TFAIL: icache unclean
> > 
> > > +
> > > +static struct tst_test test = {
> > > +	.needs_root = 1,
> > > +	.mntpoint = MNTPOINT,
> > > +	.needs_hugetlbfs = 1,
> > > +	.needs_tmpdir = 1,
> > > +	.setup = setup,
> > > +	.cleanup = cleanup,
> > > +	.test_all = run_test,
> > > +	.hugepages = {3, TST_NEEDS},
> > > +};
> > > -- 
> > > 2.31.1
> > 
> > Again, it would be nice to have the git tag(s) of the fix commit(s) if
> > possible.
> > 
> > -- 
> > Thank you,
> > Richard.
> > 
> > -- 
> > Mailing list info: https://lists.linux.it/listinfo/ltp
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp
diff mbox series

Patch

diff --git a/runtest/hugetlb b/runtest/hugetlb
index 796ebe7fa..0714ed34c 100644
--- a/runtest/hugetlb
+++ b/runtest/hugetlb
@@ -16,6 +16,7 @@  hugemmap11 hugemmap11
 hugemmap12 hugemmap12
 hugemmap13 hugemmap13
 hugemmap14 hugemmap14
+hugemmap15 hugemmap15
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
diff --git a/testcases/kernel/mem/.gitignore b/testcases/kernel/mem/.gitignore
index 3106579ce..d59b60fd4 100644
--- a/testcases/kernel/mem/.gitignore
+++ b/testcases/kernel/mem/.gitignore
@@ -15,6 +15,7 @@ 
 /hugetlb/hugemmap/hugemmap12
 /hugetlb/hugemmap/hugemmap13
 /hugetlb/hugemmap/hugemmap14
+/hugetlb/hugemmap/hugemmap15
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
diff --git a/testcases/kernel/mem/hugetlb/hugemmap/hugemmap15.c b/testcases/kernel/mem/hugetlb/hugemmap/hugemmap15.c
new file mode 100644
index 000000000..8be4d194f
--- /dev/null
+++ b/testcases/kernel/mem/hugetlb/hugemmap/hugemmap15.c
@@ -0,0 +1,239 @@ 
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation.
+ * Author: David Gibson & Adam Litke
+ */
+
+/*\
+ * [Description]
+ *
+ * Older ppc64 kernels don't properly flush dcache to icache before
+ * giving a cleared page to userspace.  With some exceedingly
+ * hairy code, this attempts to test for this bug.
+ *
+ * This test will never trigger (obviously) on machines with coherent
+ * icache and dcache (including x86 and POWER5).  On any given run,
+ * even on a buggy kernel there's a chance the bug won't trigger -
+ * either because we don't get the same physical page back when we
+ * remap, or because the icache happens to get flushed in the interim.
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <setjmp.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/mman.h>
+#include <ucontext.h>
+#include <limits.h>
+#include <sys/param.h>
+#include <sys/types.h>
+
+#include "hugetlb.h"
+
+#define SUCC_JMP 1
+#define FAIL_JMP 2
+#define COPY_SIZE	128
+
+/* Seems to be enough to trigger reliably */
+#define NUM_REPETITIONS	64
+#define MNTPOINT "hugetlbfs/"
+static long hpage_size;
+static int  fd = -1;
+
+static void cacheflush(void *p)
+{
+#if defined(__powerpc__)
+	asm volatile("dcbst 0,%0; sync; icbi 0,%0; isync" : : "r"(p));
+#elif defined(__arm__) || defined(__aarch64__)
+	__clear_cache(p, p + COPY_SIZE);
+#else
+	(void)p;
+#endif
+}
+
+static void jumpfunc(int copy, void *p)
+{
+	/*
+	 * gcc bug workaround: if there is exactly one &&label
+	 * construct in the function, gcc assumes the computed goto
+	 * goes there, leading to the complete elision of the goto in
+	 * this case
+	 */
+	void *l = &&dummy;
+
+	l = &&jumplabel;
+
+	if (copy) {
+		memcpy(p, l, COPY_SIZE);
+		cacheflush(p);
+	}
+
+	goto *p;
+ dummy:
+	tst_res(TWARN, "unreachable?");
+
+ jumplabel:
+	return;
+}
+
+static sigjmp_buf sig_escape;
+static void *sig_expected;
+
+static void sig_handler(int signum, siginfo_t *si, void *uc)
+{
+#if defined(__powerpc__) || defined(__powerpc64__) || defined(__ia64__) || \
+	defined(__s390__) || defined(__s390x__) || defined(__sparc__) || \
+	defined(__aarch64__) || (defined(__riscv) && __riscv_xlen == 64)
+	/* On powerpc, ia64, s390 and Aarch64, 0 bytes are an illegal
+	 * instruction, so, if the icache is cleared properly, we SIGILL
+	 * as soon as we jump into the cleared page
+	 */
+	if (signum == SIGILL) {
+		tst_res(TINFO, "SIGILL at %p (sig_expected=%p)", si->si_addr,
+				sig_expected);
+		if (si->si_addr == sig_expected)
+			siglongjmp(sig_escape, SUCC_JMP);
+		siglongjmp(sig_escape, FAIL_JMP + SIGILL);
+	}
+#elif defined(__i386__) || defined(__x86_64__) || defined(__arm__)
+	/* On x86, zero bytes form a valid instruction:
+	 *	add %al,(%eax)		(i386)
+	 * or	add %al,(%rax)		(x86_64)
+	 *
+	 * So, behaviour depends on the contents of [ER]AX, which in
+	 * turn depends on the details of code generation.  If [ER]AX
+	 * contains a valid pointer, we will execute the instruction
+	 * repeatedly until we run off that hugepage and get a SIGBUS
+	 * on the second, truncated page.  If [ER]AX does not contain
+	 * a valid pointer, we will SEGV on the first instruction in
+	 * the cleared page.  We check for both possibilities
+	 * below.
+	 *
+	 * On 32 bit ARM, zero bytes are interpreted as follows:
+	 *  andeq	r0, r0, r0	(ARM state, 4 bytes)
+	 *  movs	r0, r0		(Thumb state, 2 bytes)
+	 *
+	 * So, we only expect to run off the end of the huge page and
+	 * generate a SIGBUS.
+	 */
+	if (signum == SIGBUS) {
+		tst_res(TINFO, "SIGBUS at %p (sig_expected=%p)", si->si_addr,
+				sig_expected);
+		if (sig_expected
+		    && (PALIGN(sig_expected, hpage_size)
+			== si->si_addr)) {
+			siglongjmp(sig_escape, SUCC_JMP);
+		}
+		siglongjmp(sig_escape, FAIL_JMP + SIGBUS);
+	}
+#if defined(__x86_64__) || defined(__i386__)
+	if (signum == SIGSEGV) {
+#ifdef __x86_64__
+		void *pc = (void *)((ucontext_t *)uc)->uc_mcontext.gregs[REG_RIP];
+#else
+		void *pc = (void *)((ucontext_t *)uc)->uc_mcontext.gregs[REG_EIP];
+#endif
+		tst_res(TINFO, "SIGSEGV at %p, PC=%p (sig_expected=%p)",
+				si->si_addr, pc, sig_expected);
+		if (sig_expected == pc)
+			siglongjmp(sig_escape, SUCC_JMP);
+		siglongjmp(sig_escape, FAIL_JMP + SIGSEGV);
+	}
+#endif
+#else
+#error "Need to setup signal conditions for this arch"
+#endif
+}
+
+static int test_once(int fd)
+{
+	void *p, *q;
+
+	SAFE_FTRUNCATE(fd, 0);
+
+	switch (sigsetjmp(sig_escape, 1)) {
+	case SUCC_JMP:
+		sig_expected = NULL;
+		SAFE_FTRUNCATE(fd, 0);
+		return 0;
+	case FAIL_JMP + SIGILL:
+		tst_res(TFAIL, "SIGILL somewhere unexpected");
+		return -1;
+	case FAIL_JMP + SIGBUS:
+		tst_res(TFAIL, "SIGBUS somewhere unexpected");
+		return -1;
+	case FAIL_JMP + SIGSEGV:
+		tst_res(TFAIL, "SIGSEGV somewhere unexpected");
+		return -1;
+	default:
+		break;
+	}
+	p = SAFE_MMAP(NULL, 2*hpage_size, PROT_READ|PROT_WRITE|PROT_EXEC,
+		 MAP_SHARED, fd, 0);
+
+	SAFE_FTRUNCATE(fd, hpage_size);
+
+	q = p + hpage_size - COPY_SIZE;
+
+	jumpfunc(1, q);
+
+	SAFE_FTRUNCATE(fd, 0);
+	p = SAFE_MMAP(p, hpage_size, PROT_READ|PROT_WRITE|PROT_EXEC,
+		 MAP_SHARED|MAP_FIXED, fd, 0);
+
+	q = p + hpage_size - COPY_SIZE;
+	sig_expected = q;
+
+	jumpfunc(0, q); /* This should blow up */
+
+	tst_res(TFAIL, "icache unclean");
+	return -1;
+}
+
+static void run_test(void)
+{
+	int i;
+
+	struct sigaction sa = {
+		.sa_sigaction = sig_handler,
+		.sa_flags = SA_SIGINFO,
+	};
+
+	SAFE_SIGACTION(SIGILL, &sa, NULL);
+	SAFE_SIGACTION(SIGBUS, &sa, NULL);
+	SAFE_SIGACTION(SIGSEGV, &sa, NULL);
+
+	fd = tst_creat_unlinked(MNTPOINT, 0);
+
+	for (i = 0; i < NUM_REPETITIONS; i++)
+		if (test_once(fd))
+			goto cleanup;
+
+	tst_res(TPASS, "Successfully tested dcache to icache flush");
+cleanup:
+	SAFE_CLOSE(fd);
+}
+
+static void setup(void)
+{
+	hpage_size = SAFE_READ_MEMINFO("Hugepagesize:")*1024;
+}
+
+static void cleanup(void)
+{
+	if (fd > 0)
+		SAFE_CLOSE(fd);
+}
+
+static struct tst_test test = {
+	.needs_root = 1,
+	.mntpoint = MNTPOINT,
+	.needs_hugetlbfs = 1,
+	.needs_tmpdir = 1,
+	.setup = setup,
+	.cleanup = cleanup,
+	.test_all = run_test,
+	.hugepages = {3, TST_NEEDS},
+};