Patchwork [BUG] Bad page map in process ibv_devinfo

login
register
mail settings
Submitter David Miller
Date Nov. 18, 2011, 6:55 a.m.
Message ID <20111118.015519.1069441245022002863.davem@davemloft.net>
Download mbox | patch
Permalink /patch/126351/
State Accepted
Delegated to: David Miller
Headers show

Comments

David Miller - Nov. 18, 2011, 6:55 a.m.
From: David Miller <davem@davemloft.net>
Date: Fri, 18 Nov 2011 01:17:08 -0500 (EST)

> That explains everything.  The problem is that we don't do the sparc64
> PTE handling code patching in modules.  So it's left at the default 4U
> versions.
 ...
> I'll work on a fix for this.

Ok, please test this out, thanks!

--------------------
[PATCH] sparc64: Patch sun4v code sequences properly on module load.

Some of the sun4v code patching occurs in inline functions visible
to, and usable by, modules.

Therefore we have to patch them up during module load.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc/kernel/entry.h    |    7 ++++++
 arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
 arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
 3 files changed, 63 insertions(+), 19 deletions(-)
Lukas Razik - Nov. 18, 2011, 1:07 p.m.
David Miller <davem@davemloft.net> wrote:
>>  That explains everything.  The problem is that we don't do the sparc64
>>  PTE handling code patching in modules.  So it's left at the default 4U
>>  versions.
> ...
>>  I'll work on a fix for this.
> 
> Ok, please test this out, thanks!
> 
> --------------------
> [PATCH] sparc64: Patch sun4v code sequences properly on module load.
> 
> Some of the sun4v code patching occurs in inline functions visible
> to, and usable by, modules.
> 
> Therefore we have to patch them up during module load.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/kernel/entry.h    |    7 ++++++
> arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
> arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
> 3 files changed, 63 insertions(+), 19 deletions(-)
> 

Hi David,

sorry that it took so long but I had to sleep to avoid mistakes - it was 6am here.
Thanks again for this fast patch!!!

Your first patch I could apply against linux-2.6.39.4. Your second patch I can't.
I've tried to apply it manually but I've seen too many differences and gave up.

Therefore I applied _both_ patches against linux-3.1.1.
Then I've tested ibv_devinfo with the builtin mlx drivers from linux-3.1.1 and it seems that the BUG message is away.

But I can't do further testing because I would need the mlx modules from OFED-1.5.4-rc4 (newest version) which doesn't compile with linux-3.1.1:
http://thread.gmane.org/gmane.linux.drivers.rdma/10192

Hence I would need your patch against 2.6.39.4 to say for sure if it works now. :-/
Please don't feel forced by me! It would be nice to have a patch but I see that you work on many areas needing improvement. Then I must wait for an OFED version that is compatible with linux-3.1.1 ...

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Razik - Nov. 18, 2011, 5:42 p.m.
----- Ursprüngliche Message -----

> Von: David Miller <davem@davemloft.net>
> An: linux@razik.name
> Cc: roland@purestorage.com; sparclinux@vger.kernel.org; linux-rdma@vger.kernel.org
> Gesendet: 7:55 Freitag, 18.November 2011
> Betreff: Re: [BUG] Bad page map in process ibv_devinfo
> 
> From: David Miller <davem@davemloft.net>
> Date: Fri, 18 Nov 2011 01:17:08 -0500 (EST)
> 
>>  That explains everything.  The problem is that we don't do the sparc64
>>  PTE handling code patching in modules.  So it's left at the default 4U
>>  versions.
> ...
>>  I'll work on a fix for this.
> 
> Ok, please test this out, thanks!
> 
> --------------------
> [PATCH] sparc64: Patch sun4v code sequences properly on module load.
> 
> Some of the sun4v code patching occurs in inline functions visible
> to, and usable by, modules.
> 
> Therefore we have to patch them up during module load.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/kernel/entry.h    |    7 ++++++
> arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
> arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
> 3 files changed, 63 insertions(+), 19 deletions(-)
> 

Hi David,

I could apply also your second patch to linux-2.6.39.4 manually. So you don't need to write another version.
Now I'll try to test it with the new OFED drivers...

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller - Nov. 18, 2011, 7:12 p.m.
From: Lukas Razik <linux@razik.name>
Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT)

> I could apply also your second patch to linux-2.6.39.4 manually. So
> you don't need to write another version.  Now I'll try to test it
> with the new OFED drivers...

Thanks for all of your testing so far.
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Razik - Nov. 18, 2011, 9:29 p.m.
David Miller <davem@davemloft.net> wrote:
> From: Lukas Razik <linux@razik.name>

> Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT)
> 
>>  I could apply also your second patch to linux-2.6.39.4 manually. So
>>  you don't need to write another version.  Now I'll try to test it
>>  with the new OFED drivers...
> 
> Thanks for all of your testing so far.

Hello David!

Oh, no! Please don't thank me. That's as if my grandma would have thanked after I've eaten her tasty cake... :)
I thank you and Roland very much for the fast and professional help! You did a very good job!
I still can't use OpenMPI over the ConnectX adapter (another error) but that's not for the SPARC list I think.

I think you've eliminated the BUG we talk about. :)
Again: I've applied the patch against 2.6.39.4 manually. I can't say if it also works with linux-3.1.1 because of OFED-1.5.4-rc4 .

Now if I run 'ibv_devinfo' I get no kernel BUG messages anymore and my 
system doesn't hang if I use one of the ibv_*_pingpong utilities:
---
cluster1:~# ibv_rc_pingpong -n 1000000
  local address:  LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID ::
  remote address: LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID ::
8192000000 bytes in 19.59 seconds = 3345.20 Mbit/sec
1000000 iters in 19.59 seconds = 19.59 usec/iter

cluster2:~# ibv_rc_pingpong -n 1000000 cluster1
  local address:  LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID ::
  remote address: LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID ::
8192000000 bytes in 19.60 seconds = 3344.02 Mbit/sec
1000000 iters in 19.60 seconds = 19.60 usec/iter
---

BTW: If you need someone in the future for testing a SPARC64 patch - just write me!

Regards and have a nice weekend,
Lukas


PS: That's the whole conversation:
http://thread.gmane.org/gmane.linux.drivers.rdma/10157
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/arch/sparc/kernel/entry.h b/arch/sparc/kernel/entry.h
index e27f8ea..0c218e4 100644
--- a/arch/sparc/kernel/entry.h
+++ b/arch/sparc/kernel/entry.h
@@ -42,6 +42,9 @@  extern void fpsave(unsigned long *fpregs, unsigned long *fsr,
 extern void fpload(unsigned long *fpregs, unsigned long *fsr);
 
 #else /* CONFIG_SPARC32 */
+
+#include <asm/trap_block.h>
+
 struct popc_3insn_patch_entry {
 	unsigned int	addr;
 	unsigned int	insns[3];
@@ -57,6 +60,10 @@  extern struct popc_6insn_patch_entry __popc_6insn_patch,
 	__popc_6insn_patch_end;
 
 extern void __init per_cpu_patch(void);
+extern void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *,
+				    struct sun4v_1insn_patch_entry *);
+extern void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *,
+				    struct sun4v_2insn_patch_entry *);
 extern void __init sun4v_patch(void);
 extern void __init boot_cpu_id_too_large(int cpu);
 extern unsigned int dcache_parity_tl1_occurred;
diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c
index da0c6c7..e551987 100644
--- a/arch/sparc/kernel/module.c
+++ b/arch/sparc/kernel/module.c
@@ -17,6 +17,8 @@ 
 #include <asm/processor.h>
 #include <asm/spitfire.h>
 
+#include "entry.h"
+
 #ifdef CONFIG_SPARC64
 
 #include <linux/jump_label.h>
@@ -203,6 +205,29 @@  int apply_relocate_add(Elf_Shdr *sechdrs,
 }
 
 #ifdef CONFIG_SPARC64
+static void do_patch_sections(const Elf_Ehdr *hdr,
+			      const Elf_Shdr *sechdrs)
+{
+	const Elf_Shdr *s, *sun4v_1insn = NULL, *sun4v_2insn = NULL;
+	char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
+
+	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
+		if (!strcmp(".sun4v_1insn_patch", secstrings + s->sh_name))
+			sun4v_1insn = s;
+		if (!strcmp(".sun4v_2insn_patch", secstrings + s->sh_name))
+			sun4v_2insn = s;
+	}
+
+	if (sun4v_1insn && tlb_type == hypervisor) {
+		void *p = (void *) sun4v_1insn->sh_addr;
+		sun4v_patch_1insn_range(p, p + sun4v_1insn->sh_size);
+	}
+	if (sun4v_2insn && tlb_type == hypervisor) {
+		void *p = (void *) sun4v_2insn->sh_addr;
+		sun4v_patch_2insn_range(p, p + sun4v_2insn->sh_size);
+	}
+}
+
 int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
@@ -210,6 +235,8 @@  int module_finalize(const Elf_Ehdr *hdr,
 	/* make jump label nops */
 	jump_label_apply_nops(me);
 
+	do_patch_sections(hdr, sechdrs);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
diff --git a/arch/sparc/kernel/setup_64.c b/arch/sparc/kernel/setup_64.c
index c965595a..a854a1c 100644
--- a/arch/sparc/kernel/setup_64.c
+++ b/arch/sparc/kernel/setup_64.c
@@ -234,40 +234,50 @@  void __init per_cpu_patch(void)
 	}
 }
 
-void __init sun4v_patch(void)
+void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *start,
+			     struct sun4v_1insn_patch_entry *end)
 {
-	extern void sun4v_hvapi_init(void);
-	struct sun4v_1insn_patch_entry *p1;
-	struct sun4v_2insn_patch_entry *p2;
-
-	if (tlb_type != hypervisor)
-		return;
+	while (start < end) {
+		unsigned long addr = start->addr;
 
-	p1 = &__sun4v_1insn_patch;
-	while (p1 < &__sun4v_1insn_patch_end) {
-		unsigned long addr = p1->addr;
-
-		*(unsigned int *) (addr +  0) = p1->insn;
+		*(unsigned int *) (addr +  0) = start->insn;
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  0));
 
-		p1++;
+		start++;
 	}
+}
 
-	p2 = &__sun4v_2insn_patch;
-	while (p2 < &__sun4v_2insn_patch_end) {
-		unsigned long addr = p2->addr;
+void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *start,
+			     struct sun4v_2insn_patch_entry *end)
+{
+	while (start < end) {
+		unsigned long addr = start->addr;
 
-		*(unsigned int *) (addr +  0) = p2->insns[0];
+		*(unsigned int *) (addr +  0) = start->insns[0];
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  0));
 
-		*(unsigned int *) (addr +  4) = p2->insns[1];
+		*(unsigned int *) (addr +  4) = start->insns[1];
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  4));
 
-		p2++;
+		start++;
 	}
+}
+
+void __init sun4v_patch(void)
+{
+	extern void sun4v_hvapi_init(void);
+
+	if (tlb_type != hypervisor)
+		return;
+
+	sun4v_patch_1insn_range(&__sun4v_1insn_patch,
+				&__sun4v_1insn_patch_end);
+
+	sun4v_patch_2insn_range(&__sun4v_2insn_patch,
+				&__sun4v_2insn_patch_end);
 
 	sun4v_hvapi_init();
 }