| Message ID | 20111118.015519.1069441245022002863.davem@davemloft.net |
|---|---|
| State | Accepted |
| Delegated to: | David Miller |
| Headers | show |
David Miller <davem@davemloft.net> wrote: >> That explains everything. The problem is that we don't do the sparc64 >> PTE handling code patching in modules. So it's left at the default 4U >> versions. > ... >> I'll work on a fix for this. > > Ok, please test this out, thanks! > > -------------------- > [PATCH] sparc64: Patch sun4v code sequences properly on module load. > > Some of the sun4v code patching occurs in inline functions visible > to, and usable by, modules. > > Therefore we have to patch them up during module load. > > Signed-off-by: David S. Miller <davem@davemloft.net> > --- > arch/sparc/kernel/entry.h | 7 ++++++ > arch/sparc/kernel/module.c | 27 +++++++++++++++++++++++ > arch/sparc/kernel/setup_64.c | 48 +++++++++++++++++++++++++---------------- > 3 files changed, 63 insertions(+), 19 deletions(-) > Hi David, sorry that it took so long but I had to sleep to avoid mistakes - it was 6am here. Thanks again for this fast patch!!! Your first patch I could apply against linux-2.6.39.4. Your second patch I can't. I've tried to apply it manually but I've seen too many differences and gave up. Therefore I applied _both_ patches against linux-3.1.1. Then I've tested ibv_devinfo with the builtin mlx drivers from linux-3.1.1 and it seems that the BUG message is away. But I can't do further testing because I would need the mlx modules from OFED-1.5.4-rc4 (newest version) which doesn't compile with linux-3.1.1: http://thread.gmane.org/gmane.linux.drivers.rdma/10192 Hence I would need your patch against 2.6.39.4 to say for sure if it works now. :-/ Please don't feel forced by me! It would be nice to have a patch but I see that you work on many areas needing improvement. Then I must wait for an OFED version that is compatible with linux-3.1.1 ... Regards, Lukas -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
----- Ursprüngliche Message ----- > Von: David Miller <davem@davemloft.net> > An: linux@razik.name > Cc: roland@purestorage.com; sparclinux@vger.kernel.org; linux-rdma@vger.kernel.org > Gesendet: 7:55 Freitag, 18.November 2011 > Betreff: Re: [BUG] Bad page map in process ibv_devinfo > > From: David Miller <davem@davemloft.net> > Date: Fri, 18 Nov 2011 01:17:08 -0500 (EST) > >> That explains everything. The problem is that we don't do the sparc64 >> PTE handling code patching in modules. So it's left at the default 4U >> versions. > ... >> I'll work on a fix for this. > > Ok, please test this out, thanks! > > -------------------- > [PATCH] sparc64: Patch sun4v code sequences properly on module load. > > Some of the sun4v code patching occurs in inline functions visible > to, and usable by, modules. > > Therefore we have to patch them up during module load. > > Signed-off-by: David S. Miller <davem@davemloft.net> > --- > arch/sparc/kernel/entry.h | 7 ++++++ > arch/sparc/kernel/module.c | 27 +++++++++++++++++++++++ > arch/sparc/kernel/setup_64.c | 48 +++++++++++++++++++++++++---------------- > 3 files changed, 63 insertions(+), 19 deletions(-) > Hi David, I could apply also your second patch to linux-2.6.39.4 manually. So you don't need to write another version. Now I'll try to test it with the new OFED drivers... Regards, Lukas -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Lukas Razik <linux@razik.name> Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT) > I could apply also your second patch to linux-2.6.39.4 manually. So > you don't need to write another version. Now I'll try to test it > with the new OFED drivers... Thanks for all of your testing so far. -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Miller <davem@davemloft.net> wrote: > From: Lukas Razik <linux@razik.name> > Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT) > >> I could apply also your second patch to linux-2.6.39.4 manually. So >> you don't need to write another version. Now I'll try to test it >> with the new OFED drivers... > > Thanks for all of your testing so far. Hello David! Oh, no! Please don't thank me. That's as if my grandma would have thanked after I've eaten her tasty cake... :) I thank you and Roland very much for the fast and professional help! You did a very good job! I still can't use OpenMPI over the ConnectX adapter (another error) but that's not for the SPARC list I think. I think you've eliminated the BUG we talk about. :) Again: I've applied the patch against 2.6.39.4 manually. I can't say if it also works with linux-3.1.1 because of OFED-1.5.4-rc4 . Now if I run 'ibv_devinfo' I get no kernel BUG messages anymore and my system doesn't hang if I use one of the ibv_*_pingpong utilities: --- cluster1:~# ibv_rc_pingpong -n 1000000 local address: LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID :: remote address: LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID :: 8192000000 bytes in 19.59 seconds = 3345.20 Mbit/sec 1000000 iters in 19.59 seconds = 19.59 usec/iter cluster2:~# ibv_rc_pingpong -n 1000000 cluster1 local address: LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID :: remote address: LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID :: 8192000000 bytes in 19.60 seconds = 3344.02 Mbit/sec 1000000 iters in 19.60 seconds = 19.60 usec/iter --- BTW: If you need someone in the future for testing a SPARC64 patch - just write me! Regards and have a nice weekend, Lukas PS: That's the whole conversation: http://thread.gmane.org/gmane.linux.drivers.rdma/10157 -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/arch/sparc/kernel/entry.h b/arch/sparc/kernel/entry.h index e27f8ea..0c218e4 100644 --- a/arch/sparc/kernel/entry.h +++ b/arch/sparc/kernel/entry.h @@ -42,6 +42,9 @@ extern void fpsave(unsigned long *fpregs, unsigned long *fsr, extern void fpload(unsigned long *fpregs, unsigned long *fsr); #else /* CONFIG_SPARC32 */ + +#include <asm/trap_block.h> + struct popc_3insn_patch_entry { unsigned int addr; unsigned int insns[3]; @@ -57,6 +60,10 @@ extern struct popc_6insn_patch_entry __popc_6insn_patch, __popc_6insn_patch_end; extern void __init per_cpu_patch(void); +extern void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *, + struct sun4v_1insn_patch_entry *); +extern void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *, + struct sun4v_2insn_patch_entry *); extern void __init sun4v_patch(void); extern void __init boot_cpu_id_too_large(int cpu); extern unsigned int dcache_parity_tl1_occurred; diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c index da0c6c7..e551987 100644 --- a/arch/sparc/kernel/module.c +++ b/arch/sparc/kernel/module.c @@ -17,6 +17,8 @@ #include <asm/processor.h> #include <asm/spitfire.h> +#include "entry.h" + #ifdef CONFIG_SPARC64 #include <linux/jump_label.h> @@ -203,6 +205,29 @@ int apply_relocate_add(Elf_Shdr *sechdrs, } #ifdef CONFIG_SPARC64 +static void do_patch_sections(const Elf_Ehdr *hdr, + const Elf_Shdr *sechdrs) +{ + const Elf_Shdr *s, *sun4v_1insn = NULL, *sun4v_2insn = NULL; + char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; + + for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) { + if (!strcmp(".sun4v_1insn_patch", secstrings + s->sh_name)) + sun4v_1insn = s; + if (!strcmp(".sun4v_2insn_patch", secstrings + s->sh_name)) + sun4v_2insn = s; + } + + if (sun4v_1insn && tlb_type == hypervisor) { + void *p = (void *) sun4v_1insn->sh_addr; + sun4v_patch_1insn_range(p, p + sun4v_1insn->sh_size); + } + if (sun4v_2insn && tlb_type == hypervisor) { + void *p = (void *) sun4v_2insn->sh_addr; + sun4v_patch_2insn_range(p, p + sun4v_2insn->sh_size); + } +} + int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, struct module *me) @@ -210,6 +235,8 @@ int module_finalize(const Elf_Ehdr *hdr, /* make jump label nops */ jump_label_apply_nops(me); + do_patch_sections(hdr, sechdrs); + /* Cheetah's I-cache is fully coherent. */ if (tlb_type == spitfire) { unsigned long va; diff --git a/arch/sparc/kernel/setup_64.c b/arch/sparc/kernel/setup_64.c index c965595a..a854a1c 100644 --- a/arch/sparc/kernel/setup_64.c +++ b/arch/sparc/kernel/setup_64.c @@ -234,40 +234,50 @@ void __init per_cpu_patch(void) } } -void __init sun4v_patch(void) +void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *start, + struct sun4v_1insn_patch_entry *end) { - extern void sun4v_hvapi_init(void); - struct sun4v_1insn_patch_entry *p1; - struct sun4v_2insn_patch_entry *p2; - - if (tlb_type != hypervisor) - return; + while (start < end) { + unsigned long addr = start->addr; - p1 = &__sun4v_1insn_patch; - while (p1 < &__sun4v_1insn_patch_end) { - unsigned long addr = p1->addr; - - *(unsigned int *) (addr + 0) = p1->insn; + *(unsigned int *) (addr + 0) = start->insn; wmb(); __asm__ __volatile__("flush %0" : : "r" (addr + 0)); - p1++; + start++; } +} - p2 = &__sun4v_2insn_patch; - while (p2 < &__sun4v_2insn_patch_end) { - unsigned long addr = p2->addr; +void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *start, + struct sun4v_2insn_patch_entry *end) +{ + while (start < end) { + unsigned long addr = start->addr; - *(unsigned int *) (addr + 0) = p2->insns[0]; + *(unsigned int *) (addr + 0) = start->insns[0]; wmb(); __asm__ __volatile__("flush %0" : : "r" (addr + 0)); - *(unsigned int *) (addr + 4) = p2->insns[1]; + *(unsigned int *) (addr + 4) = start->insns[1]; wmb(); __asm__ __volatile__("flush %0" : : "r" (addr + 4)); - p2++; + start++; } +} + +void __init sun4v_patch(void) +{ + extern void sun4v_hvapi_init(void); + + if (tlb_type != hypervisor) + return; + + sun4v_patch_1insn_range(&__sun4v_1insn_patch, + &__sun4v_1insn_patch_end); + + sun4v_patch_2insn_range(&__sun4v_2insn_patch, + &__sun4v_2insn_patch_end); sun4v_hvapi_init(); }