| Message ID | b1451438f7148ad0e03306a1f1409f4ad1d6ec7c.1581684263.git.christophe.leroy@c-s.fr (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | powerpc/kprobes: Fix trap address when trap happened in real mode | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | Successfully applied on branch powerpc/merge (a5bc6e124219546a81ce334dc9b16483d55e9abf) |
| snowpatch_ozlabs/build-ppc64le | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64be | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64e | success | Build succeeded |
| snowpatch_ozlabs/build-pmac32 | success | Build succeeded |
| snowpatch_ozlabs/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 16 lines checked |
| snowpatch_ozlabs/needsstable | warning | Please consider tagging this patch for stable! |
Hi, On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) Christophe Leroy <christophe.leroy@c-s.fr> wrote: > When a program check exception happens while MMU translation is > disabled, following Oops happens in kprobe_handler() in the following > test: > > } else if (*addr != BREAKPOINT_INSTRUCTION) { Thanks for the report and patch. I'm not so sure about powerpc implementation but at where the MMU translation is disabled, can the handler work correctly? (And where did you put the probe on?) Your fix may fix this Oops, but if the handler needs special care, it is an option to blacklist such place (if possible). Anyway, Naveen, can you review it? Thank you, > > [ 33.098554] BUG: Unable to handle kernel data access on read at 0x0000e268 > [ 33.105091] Faulting instruction address: 0xc000ec34 > [ 33.110010] Oops: Kernel access of bad area, sig: 11 [#1] > [ 33.115348] BE PAGE_SIZE=16K PREEMPT CMPC885 > [ 33.119540] Modules linked in: > [ 33.122591] CPU: 0 PID: 429 Comm: cat Not tainted 5.6.0-rc1-s3k-dev-00824-g84195dc6c58a #3267 > [ 33.131005] NIP: c000ec34 LR: c000ecd8 CTR: c019cab8 > [ 33.136002] REGS: ca4d3b58 TRAP: 0300 Not tainted (5.6.0-rc1-s3k-dev-00824-g84195dc6c58a) > [ 33.144324] MSR: 00001032 <ME,IR,DR,RI> CR: 2a4d3c52 XER: 00000000 > [ 33.150699] DAR: 0000e268 DSISR: c0000000 > [ 33.150699] GPR00: c000b09c ca4d3c10 c66d0620 00000000 ca4d3c60 00000000 00009032 00000000 > [ 33.150699] GPR08: 00020000 00000000 c087de44 c000afe0 c66d0ad0 100d3dd6 fffffff3 00000000 > [ 33.150699] GPR16: 00000000 00000041 00000000 ca4d3d70 00000000 00000000 0000416d 00000000 > [ 33.150699] GPR24: 00000004 c53b6128 00000000 0000e268 00000000 c07c0000 c07bb6fc ca4d3c60 > [ 33.188015] NIP [c000ec34] kprobe_handler+0x128/0x290 > [ 33.192989] LR [c000ecd8] kprobe_handler+0x1cc/0x290 > [ 33.197854] Call Trace: > [ 33.200340] [ca4d3c30] [c000b09c] program_check_exception+0xbc/0x6fc > [ 33.206590] [ca4d3c50] [c000e43c] ret_from_except_full+0x0/0x4 > [ 33.212392] --- interrupt: 700 at 0xe268 > [ 33.270401] Instruction dump: > [ 33.273335] 913e0008 81220000 38600001 3929ffff 91220000 80010024 bb410008 7c0803a6 > [ 33.280992] 38210020 4e800020 38600000 4e800020 <813b0000> 6d2a7fe0 2f8a0008 419e0154 > [ 33.288841] ---[ end trace 5b9152d4cdadd06d ]--- > > Check MSR and convert regs->nip to virtual address if the trap > happened with MSR_IR cleared. > > Reported-by: Larry Finger <Larry.Finger@lwfinger.net> > Fixes: 6cc89bad60a6 ("powerpc/kprobes: Invoke handlers directly") > Cc: stable@kernel.vger.org > Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> > Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> > > --- > The bug might have existed even before that commit from Naveen. > > Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> > --- > arch/powerpc/kernel/kprobes.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c > index 2d27ec4feee4..f8b848aa65bd 100644 > --- a/arch/powerpc/kernel/kprobes.c > +++ b/arch/powerpc/kernel/kprobes.c > @@ -23,6 +23,7 @@ > #include <asm/cacheflush.h> > #include <asm/sstep.h> > #include <asm/sections.h> > +#include <asm/io.h> > #include <linux/uaccess.h> > > DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; > @@ -264,6 +265,9 @@ int kprobe_handler(struct pt_regs *regs) > if (user_mode(regs)) > return 0; > > + if (!(regs->msr & MSR_IR)) > + addr = phys_to_virt(regs->nip); > + > /* > * We don't want to be preempted for the entire > * duration of kprobe processing > -- > 2.25.0 >
Le 14/02/2020 à 13:47, Christophe Leroy a écrit : > When a program check exception happens while MMU translation is > disabled, following Oops happens in kprobe_handler() in the following > test: > > } else if (*addr != BREAKPOINT_INSTRUCTION) { > > [ 33.098554] BUG: Unable to handle kernel data access on read at 0x0000e268 > [ 33.105091] Faulting instruction address: 0xc000ec34 > [ 33.110010] Oops: Kernel access of bad area, sig: 11 [#1] > [ 33.115348] BE PAGE_SIZE=16K PREEMPT CMPC885 > [ 33.119540] Modules linked in: > [ 33.122591] CPU: 0 PID: 429 Comm: cat Not tainted 5.6.0-rc1-s3k-dev-00824-g84195dc6c58a #3267 > [ 33.131005] NIP: c000ec34 LR: c000ecd8 CTR: c019cab8 > [ 33.136002] REGS: ca4d3b58 TRAP: 0300 Not tainted (5.6.0-rc1-s3k-dev-00824-g84195dc6c58a) > [ 33.144324] MSR: 00001032 <ME,IR,DR,RI> CR: 2a4d3c52 XER: 00000000 > [ 33.150699] DAR: 0000e268 DSISR: c0000000 > [ 33.150699] GPR00: c000b09c ca4d3c10 c66d0620 00000000 ca4d3c60 00000000 00009032 00000000 > [ 33.150699] GPR08: 00020000 00000000 c087de44 c000afe0 c66d0ad0 100d3dd6 fffffff3 00000000 > [ 33.150699] GPR16: 00000000 00000041 00000000 ca4d3d70 00000000 00000000 0000416d 00000000 > [ 33.150699] GPR24: 00000004 c53b6128 00000000 0000e268 00000000 c07c0000 c07bb6fc ca4d3c60 > [ 33.188015] NIP [c000ec34] kprobe_handler+0x128/0x290 > [ 33.192989] LR [c000ecd8] kprobe_handler+0x1cc/0x290 > [ 33.197854] Call Trace: > [ 33.200340] [ca4d3c30] [c000b09c] program_check_exception+0xbc/0x6fc > [ 33.206590] [ca4d3c50] [c000e43c] ret_from_except_full+0x0/0x4 > [ 33.212392] --- interrupt: 700 at 0xe268 > [ 33.270401] Instruction dump: > [ 33.273335] 913e0008 81220000 38600001 3929ffff 91220000 80010024 bb410008 7c0803a6 > [ 33.280992] 38210020 4e800020 38600000 4e800020 <813b0000> 6d2a7fe0 2f8a0008 419e0154 > [ 33.288841] ---[ end trace 5b9152d4cdadd06d ]--- > > Check MSR and convert regs->nip to virtual address if the trap > happened with MSR_IR cleared. > > Reported-by: Larry Finger <Larry.Finger@lwfinger.net> > Fixes: 6cc89bad60a6 ("powerpc/kprobes: Invoke handlers directly") > Cc: stable@kernel.vger.org Oops, I meant Cc: stable@vger.kernel.org > Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> > Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> > > --- > The bug might have existed even before that commit from Naveen. > > Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> > --- > arch/powerpc/kernel/kprobes.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c > index 2d27ec4feee4..f8b848aa65bd 100644 > --- a/arch/powerpc/kernel/kprobes.c > +++ b/arch/powerpc/kernel/kprobes.c > @@ -23,6 +23,7 @@ > #include <asm/cacheflush.h> > #include <asm/sstep.h> > #include <asm/sections.h> > +#include <asm/io.h> > #include <linux/uaccess.h> > > DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; > @@ -264,6 +265,9 @@ int kprobe_handler(struct pt_regs *regs) > if (user_mode(regs)) > return 0; > > + if (!(regs->msr & MSR_IR)) > + addr = phys_to_virt(regs->nip); > + > /* > * We don't want to be preempted for the entire > * duration of kprobe processing >
Hi, Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : > Hi, > > On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> When a program check exception happens while MMU translation is >> disabled, following Oops happens in kprobe_handler() in the following >> test: >> >> } else if (*addr != BREAKPOINT_INSTRUCTION) { > > Thanks for the report and patch. I'm not so sure about powerpc implementation > but at where the MMU translation is disabled, can the handler work correctly? > (And where did you put the probe on?) > > Your fix may fix this Oops, but if the handler needs special care, it is an > option to blacklist such place (if possible). I guess that's another story. Here we are not talking about a place where kprobe has been illegitimately activated, but a place where there is a valid trap, which generated a valid 'program check exception'. And kprobe was off at that time. As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, a debugger breakpoint, a perf breakpoint, etc...) calls kprobe_handler(), kprobe_handler() must be prepared to handle the case where the MMU translation is disabled, even if probes are not supposed to be set for functions running with MMU translation disabled. Christophe > > Anyway, Naveen, can you review it? > > Thank you, > >> >> [ 33.098554] BUG: Unable to handle kernel data access on read at 0x0000e268 >> [ 33.105091] Faulting instruction address: 0xc000ec34 >> [ 33.110010] Oops: Kernel access of bad area, sig: 11 [#1] >> [ 33.115348] BE PAGE_SIZE=16K PREEMPT CMPC885 >> [ 33.119540] Modules linked in: >> [ 33.122591] CPU: 0 PID: 429 Comm: cat Not tainted 5.6.0-rc1-s3k-dev-00824-g84195dc6c58a #3267 >> [ 33.131005] NIP: c000ec34 LR: c000ecd8 CTR: c019cab8 >> [ 33.136002] REGS: ca4d3b58 TRAP: 0300 Not tainted (5.6.0-rc1-s3k-dev-00824-g84195dc6c58a) >> [ 33.144324] MSR: 00001032 <ME,IR,DR,RI> CR: 2a4d3c52 XER: 00000000 >> [ 33.150699] DAR: 0000e268 DSISR: c0000000 >> [ 33.150699] GPR00: c000b09c ca4d3c10 c66d0620 00000000 ca4d3c60 00000000 00009032 00000000 >> [ 33.150699] GPR08: 00020000 00000000 c087de44 c000afe0 c66d0ad0 100d3dd6 fffffff3 00000000 >> [ 33.150699] GPR16: 00000000 00000041 00000000 ca4d3d70 00000000 00000000 0000416d 00000000 >> [ 33.150699] GPR24: 00000004 c53b6128 00000000 0000e268 00000000 c07c0000 c07bb6fc ca4d3c60 >> [ 33.188015] NIP [c000ec34] kprobe_handler+0x128/0x290 >> [ 33.192989] LR [c000ecd8] kprobe_handler+0x1cc/0x290 >> [ 33.197854] Call Trace: >> [ 33.200340] [ca4d3c30] [c000b09c] program_check_exception+0xbc/0x6fc >> [ 33.206590] [ca4d3c50] [c000e43c] ret_from_except_full+0x0/0x4 >> [ 33.212392] --- interrupt: 700 at 0xe268 >> [ 33.270401] Instruction dump: >> [ 33.273335] 913e0008 81220000 38600001 3929ffff 91220000 80010024 bb410008 7c0803a6 >> [ 33.280992] 38210020 4e800020 38600000 4e800020 <813b0000> 6d2a7fe0 2f8a0008 419e0154 >> [ 33.288841] ---[ end trace 5b9152d4cdadd06d ]--- >> >> Check MSR and convert regs->nip to virtual address if the trap >> happened with MSR_IR cleared. >> >> Reported-by: Larry Finger <Larry.Finger@lwfinger.net> >> Fixes: 6cc89bad60a6 ("powerpc/kprobes: Invoke handlers directly") >> Cc: stable@kernel.vger.org >> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> >> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> >> >> --- >> The bug might have existed even before that commit from Naveen. >> >> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> >> --- >> arch/powerpc/kernel/kprobes.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c >> index 2d27ec4feee4..f8b848aa65bd 100644 >> --- a/arch/powerpc/kernel/kprobes.c >> +++ b/arch/powerpc/kernel/kprobes.c >> @@ -23,6 +23,7 @@ >> #include <asm/cacheflush.h> >> #include <asm/sstep.h> >> #include <asm/sections.h> >> +#include <asm/io.h> >> #include <linux/uaccess.h> >> >> DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; >> @@ -264,6 +265,9 @@ int kprobe_handler(struct pt_regs *regs) >> if (user_mode(regs)) >> return 0; >> >> + if (!(regs->msr & MSR_IR)) >> + addr = phys_to_virt(regs->nip); >> + >> /* >> * We don't want to be preempted for the entire >> * duration of kprobe processing >> -- >> 2.25.0 >> > >
On Sat, 15 Feb 2020 11:28:49 +0100 Christophe Leroy <christophe.leroy@c-s.fr> wrote: > Hi, > > Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : > > Hi, > > > > On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) > > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > > > >> When a program check exception happens while MMU translation is > >> disabled, following Oops happens in kprobe_handler() in the following > >> test: > >> > >> } else if (*addr != BREAKPOINT_INSTRUCTION) { > > > > Thanks for the report and patch. I'm not so sure about powerpc implementation > > but at where the MMU translation is disabled, can the handler work correctly? > > (And where did you put the probe on?) > > > > Your fix may fix this Oops, but if the handler needs special care, it is an > > option to blacklist such place (if possible). > > I guess that's another story. Here we are not talking about a place > where kprobe has been illegitimately activated, but a place where there > is a valid trap, which generated a valid 'program check exception'. And > kprobe was off at that time. Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, it has to know the address where the breakpoint happens. OK. > > As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, > a debugger breakpoint, a perf breakpoint, etc...) calls > kprobe_handler(), kprobe_handler() must be prepared to handle the case > where the MMU translation is disabled, even if probes are not supposed > to be set for functions running with MMU translation disabled. Can't we check the MMU is disabled there (as same as checking the exception happened in user space or not)? Thank you,
Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : > On Sat, 15 Feb 2020 11:28:49 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> Hi, >> >> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : >>> Hi, >>> >>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) >>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>> >>>> When a program check exception happens while MMU translation is >>>> disabled, following Oops happens in kprobe_handler() in the following >>>> test: >>>> >>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { >>> >>> Thanks for the report and patch. I'm not so sure about powerpc implementation >>> but at where the MMU translation is disabled, can the handler work correctly? >>> (And where did you put the probe on?) >>> >>> Your fix may fix this Oops, but if the handler needs special care, it is an >>> option to blacklist such place (if possible). >> >> I guess that's another story. Here we are not talking about a place >> where kprobe has been illegitimately activated, but a place where there >> is a valid trap, which generated a valid 'program check exception'. And >> kprobe was off at that time. > > Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, > it has to know the address where the breakpoint happens. OK. > >> >> As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, >> a debugger breakpoint, a perf breakpoint, etc...) calls >> kprobe_handler(), kprobe_handler() must be prepared to handle the case >> where the MMU translation is disabled, even if probes are not supposed >> to be set for functions running with MMU translation disabled. > > Can't we check the MMU is disabled there (as same as checking the exception > happened in user space or not)? > What do you mean by 'there' ? At the entry of kprobe_handler() ? That's what my patch does, it checks whether MMU is disabled or not. If it is, it converts the address to a virtual address. Do you mean kprobe_handler() should bail out early as it does when the trap happens in user mode ? Of course we can do that, I don't know enough about kprobe to know if kprobe_handler() should manage events that happened in real-mode or just ignore them. But I tested adding an event on a function that runs in real-mode, and it (now) works. So, what should we do really ? Christophe
On Mon, 17 Feb 2020 10:03:22 +0100 Christophe Leroy <christophe.leroy@c-s.fr> wrote: > > > Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : > > On Sat, 15 Feb 2020 11:28:49 +0100 > > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > > > >> Hi, > >> > >> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : > >>> Hi, > >>> > >>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) > >>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>> > >>>> When a program check exception happens while MMU translation is > >>>> disabled, following Oops happens in kprobe_handler() in the following > >>>> test: > >>>> > >>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { > >>> > >>> Thanks for the report and patch. I'm not so sure about powerpc implementation > >>> but at where the MMU translation is disabled, can the handler work correctly? > >>> (And where did you put the probe on?) > >>> > >>> Your fix may fix this Oops, but if the handler needs special care, it is an > >>> option to blacklist such place (if possible). > >> > >> I guess that's another story. Here we are not talking about a place > >> where kprobe has been illegitimately activated, but a place where there > >> is a valid trap, which generated a valid 'program check exception'. And > >> kprobe was off at that time. > > > > Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, > > it has to know the address where the breakpoint happens. OK. > > > >> > >> As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, > >> a debugger breakpoint, a perf breakpoint, etc...) calls > >> kprobe_handler(), kprobe_handler() must be prepared to handle the case > >> where the MMU translation is disabled, even if probes are not supposed > >> to be set for functions running with MMU translation disabled. > > > > Can't we check the MMU is disabled there (as same as checking the exception > > happened in user space or not)? > > > > What do you mean by 'there' ? At the entry of kprobe_handler() ? > > That's what my patch does, it checks whether MMU is disabled or not. If > it is, it converts the address to a virtual address. > > Do you mean kprobe_handler() should bail out early as it does when the > trap happens in user mode ? Yes, that is what I meant. > Of course we can do that, I don't know > enough about kprobe to know if kprobe_handler() should manage events > that happened in real-mode or just ignore them. But I tested adding an > event on a function that runs in real-mode, and it (now) works. > > So, what should we do really ? I'm not sure how the powerpc kernel runs in real mode. But clearly, at least kprobe event can not handle that case because it tries to access memory by probe_kernel_read(). Unless that function correctly handles the address translation, I want to prohibit kprobes on such address. So what I would like to see is, something like below. diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 2d27ec4feee4..4771be152416 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) unsigned int *addr = (unsigned int *)regs->nip; struct kprobe_ctlblk *kcb; - if (user_mode(regs)) + if (user_mode(regs) || !(regs->msr & MSR_IR)) return 0; /* Thank you,
Le 17/02/2020 à 11:27, Masami Hiramatsu a écrit : > On Mon, 17 Feb 2020 10:03:22 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> >> >> Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : >>> On Sat, 15 Feb 2020 11:28:49 +0100 >>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>> >>>> Hi, >>>> >>>> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : >>>>> Hi, >>>>> >>>>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) >>>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>>>> >>>>>> When a program check exception happens while MMU translation is >>>>>> disabled, following Oops happens in kprobe_handler() in the following >>>>>> test: >>>>>> >>>>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { >>>>> >>>>> Thanks for the report and patch. I'm not so sure about powerpc implementation >>>>> but at where the MMU translation is disabled, can the handler work correctly? >>>>> (And where did you put the probe on?) >>>>> >>>>> Your fix may fix this Oops, but if the handler needs special care, it is an >>>>> option to blacklist such place (if possible). >>>> >>>> I guess that's another story. Here we are not talking about a place >>>> where kprobe has been illegitimately activated, but a place where there >>>> is a valid trap, which generated a valid 'program check exception'. And >>>> kprobe was off at that time. >>> >>> Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, >>> it has to know the address where the breakpoint happens. OK. >>> >>>> >>>> As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, >>>> a debugger breakpoint, a perf breakpoint, etc...) calls >>>> kprobe_handler(), kprobe_handler() must be prepared to handle the case >>>> where the MMU translation is disabled, even if probes are not supposed >>>> to be set for functions running with MMU translation disabled. >>> >>> Can't we check the MMU is disabled there (as same as checking the exception >>> happened in user space or not)? >>> >> >> What do you mean by 'there' ? At the entry of kprobe_handler() ? >> >> That's what my patch does, it checks whether MMU is disabled or not. If >> it is, it converts the address to a virtual address. >> >> Do you mean kprobe_handler() should bail out early as it does when the >> trap happens in user mode ? > > Yes, that is what I meant. > >> Of course we can do that, I don't know >> enough about kprobe to know if kprobe_handler() should manage events >> that happened in real-mode or just ignore them. But I tested adding an >> event on a function that runs in real-mode, and it (now) works. >> >> So, what should we do really ? > > I'm not sure how the powerpc kernel runs in real mode. > But clearly, at least kprobe event can not handle that case because > it tries to access memory by probe_kernel_read(). Unless that function > correctly handles the address translation, I want to prohibit kprobes > on such address. > > So what I would like to see is, something like below. > > diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c > index 2d27ec4feee4..4771be152416 100644 > --- a/arch/powerpc/kernel/kprobes.c > +++ b/arch/powerpc/kernel/kprobes.c > @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) > unsigned int *addr = (unsigned int *)regs->nip; > struct kprobe_ctlblk *kcb; > > - if (user_mode(regs)) > + if (user_mode(regs) || !(regs->msr & MSR_IR)) > return 0; > > /* > > With this instead change of my patch, I get an Oops everytime a kprobe event occurs in real-mode. This is because kprobe_handler() is now saying 'this trap doesn't belong to me' for a trap that has been installed by it. So the 'program check' exception handler doesn't find the owner of the trap hence generate an Oops. Even if we don't want kprobe() to proceed with the event entirely (allthough it works at least for simple events), I'd expect it to fail gracefully. Christophe
On 02/17/2020 03:38 PM, Christophe Leroy wrote: > > > Le 17/02/2020 à 11:27, Masami Hiramatsu a écrit : >> On Mon, 17 Feb 2020 10:03:22 +0100 >> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >> >>> >>> >>> Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : >>>> On Sat, 15 Feb 2020 11:28:49 +0100 >>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>>> >>>>> Hi, >>>>> >>>>> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : >>>>>> Hi, >>>>>> >>>>>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) >>>>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>>>>> >>>>>>> When a program check exception happens while MMU translation is >>>>>>> disabled, following Oops happens in kprobe_handler() in the >>>>>>> following >>>>>>> test: >>>>>>> >>>>>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { >>>>>> >>>>>> Thanks for the report and patch. I'm not so sure about powerpc >>>>>> implementation >>>>>> but at where the MMU translation is disabled, can the handler work >>>>>> correctly? >>>>>> (And where did you put the probe on?) >>>>>> >>>>>> Your fix may fix this Oops, but if the handler needs special care, >>>>>> it is an >>>>>> option to blacklist such place (if possible). >>>>> >>>>> I guess that's another story. Here we are not talking about a place >>>>> where kprobe has been illegitimately activated, but a place where >>>>> there >>>>> is a valid trap, which generated a valid 'program check exception'. >>>>> And >>>>> kprobe was off at that time. >>>> >>>> Ah, I got it. It is not a kprobe breakpoint, but to check that >>>> correctly, >>>> it has to know the address where the breakpoint happens. OK. >>>> >>>>> >>>>> As any 'program check exception' due to a trap (ie a BUG_ON, a >>>>> WARN_ON, >>>>> a debugger breakpoint, a perf breakpoint, etc...) calls >>>>> kprobe_handler(), kprobe_handler() must be prepared to handle the case >>>>> where the MMU translation is disabled, even if probes are not supposed >>>>> to be set for functions running with MMU translation disabled. >>>> >>>> Can't we check the MMU is disabled there (as same as checking the >>>> exception >>>> happened in user space or not)? >>>> >>> >>> What do you mean by 'there' ? At the entry of kprobe_handler() ? >>> >>> That's what my patch does, it checks whether MMU is disabled or not. If >>> it is, it converts the address to a virtual address. >>> >>> Do you mean kprobe_handler() should bail out early as it does when the >>> trap happens in user mode ? >> >> Yes, that is what I meant. >> >>> Of course we can do that, I don't know >>> enough about kprobe to know if kprobe_handler() should manage events >>> that happened in real-mode or just ignore them. But I tested adding an >>> event on a function that runs in real-mode, and it (now) works. >>> >>> So, what should we do really ? >> >> I'm not sure how the powerpc kernel runs in real mode. >> But clearly, at least kprobe event can not handle that case because >> it tries to access memory by probe_kernel_read(). Unless that function >> correctly handles the address translation, I want to prohibit kprobes >> on such address. >> >> So what I would like to see is, something like below. >> >> diff --git a/arch/powerpc/kernel/kprobes.c >> b/arch/powerpc/kernel/kprobes.c >> index 2d27ec4feee4..4771be152416 100644 >> --- a/arch/powerpc/kernel/kprobes.c >> +++ b/arch/powerpc/kernel/kprobes.c >> @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) >> unsigned int *addr = (unsigned int *)regs->nip; >> struct kprobe_ctlblk *kcb; >> - if (user_mode(regs)) >> + if (user_mode(regs) || !(regs->msr & MSR_IR)) >> return 0; >> /* >> >> > > With this instead change of my patch, I get an Oops everytime a kprobe > event occurs in real-mode. > > This is because kprobe_handler() is now saying 'this trap doesn't belong > to me' for a trap that has been installed by it. > > So the 'program check' exception handler doesn't find the owner of the > trap hence generate an Oops. > > Even if we don't want kprobe() to proceed with the event entirely > (allthough it works at least for simple events), I'd expect it to fail > gracefully. > What about something like that: @@ -264,6 +265,13 @@ int kprobe_handler(struct pt_regs *regs) if (user_mode(regs)) return 0; + if (!(regs->msr & MSR_IR)) { + if (!get_kprobe(phys_to_virt(regs->nip))) + return 0; + regs->nip += 4; + return 1; + } + /* * We don't want to be preempted for the entire * duration of kprobe processing Christophe
On Mon, 17 Feb 2020 16:38:50 +0100 Christophe Leroy <christophe.leroy@c-s.fr> wrote: > > > Le 17/02/2020 à 11:27, Masami Hiramatsu a écrit : > > On Mon, 17 Feb 2020 10:03:22 +0100 > > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > > > >> > >> > >> Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : > >>> On Sat, 15 Feb 2020 11:28:49 +0100 > >>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>> > >>>> Hi, > >>>> > >>>> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : > >>>>> Hi, > >>>>> > >>>>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) > >>>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>>>> > >>>>>> When a program check exception happens while MMU translation is > >>>>>> disabled, following Oops happens in kprobe_handler() in the following > >>>>>> test: > >>>>>> > >>>>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { > >>>>> > >>>>> Thanks for the report and patch. I'm not so sure about powerpc implementation > >>>>> but at where the MMU translation is disabled, can the handler work correctly? > >>>>> (And where did you put the probe on?) > >>>>> > >>>>> Your fix may fix this Oops, but if the handler needs special care, it is an > >>>>> option to blacklist such place (if possible). > >>>> > >>>> I guess that's another story. Here we are not talking about a place > >>>> where kprobe has been illegitimately activated, but a place where there > >>>> is a valid trap, which generated a valid 'program check exception'. And > >>>> kprobe was off at that time. > >>> > >>> Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, > >>> it has to know the address where the breakpoint happens. OK. > >>> > >>>> > >>>> As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, > >>>> a debugger breakpoint, a perf breakpoint, etc...) calls > >>>> kprobe_handler(), kprobe_handler() must be prepared to handle the case > >>>> where the MMU translation is disabled, even if probes are not supposed > >>>> to be set for functions running with MMU translation disabled. > >>> > >>> Can't we check the MMU is disabled there (as same as checking the exception > >>> happened in user space or not)? > >>> > >> > >> What do you mean by 'there' ? At the entry of kprobe_handler() ? > >> > >> That's what my patch does, it checks whether MMU is disabled or not. If > >> it is, it converts the address to a virtual address. > >> > >> Do you mean kprobe_handler() should bail out early as it does when the > >> trap happens in user mode ? > > > > Yes, that is what I meant. > > > >> Of course we can do that, I don't know > >> enough about kprobe to know if kprobe_handler() should manage events > >> that happened in real-mode or just ignore them. But I tested adding an > >> event on a function that runs in real-mode, and it (now) works. > >> > >> So, what should we do really ? > > > > I'm not sure how the powerpc kernel runs in real mode. > > But clearly, at least kprobe event can not handle that case because > > it tries to access memory by probe_kernel_read(). Unless that function > > correctly handles the address translation, I want to prohibit kprobes > > on such address. > > > > So what I would like to see is, something like below. > > > > diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c > > index 2d27ec4feee4..4771be152416 100644 > > --- a/arch/powerpc/kernel/kprobes.c > > +++ b/arch/powerpc/kernel/kprobes.c > > @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) > > unsigned int *addr = (unsigned int *)regs->nip; > > struct kprobe_ctlblk *kcb; > > > > - if (user_mode(regs)) > > + if (user_mode(regs) || !(regs->msr & MSR_IR)) > > return 0; > > > > /* > > > > > > With this instead change of my patch, I get an Oops everytime a kprobe > event occurs in real-mode. > > This is because kprobe_handler() is now saying 'this trap doesn't belong > to me' for a trap that has been installed by it. Hmm, on powerpc, kprobes is allowed to probe on the code which runs in the real mode? I think we should also prohibit it by blacklisting. (It is easy to add blacklist by NOKPROBE_SYMBOL(func)) Or, some parts are possble to run under both real mode and kernel mode? > > So the 'program check' exception handler doesn't find the owner of the > trap hence generate an Oops. > > Even if we don't want kprobe() to proceed with the event entirely > (allthough it works at least for simple events), I'd expect it to fail > gracefully. Agreed. I thought it was easy to identify real mode code. But if it is hard, we should apply your first patch and also skip user handlers if we are in the real mode (and increment missed count). BTW, can the emulater handle the real mode code correctly? Thank you,
Le 18/02/2020 à 01:44, Masami Hiramatsu a écrit : > On Mon, 17 Feb 2020 16:38:50 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> >> >> Le 17/02/2020 à 11:27, Masami Hiramatsu a écrit : >>> On Mon, 17 Feb 2020 10:03:22 +0100 >>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>> >>>> >>>> >>>> Le 16/02/2020 à 13:34, Masami Hiramatsu a écrit : >>>>> On Sat, 15 Feb 2020 11:28:49 +0100 >>>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Le 14/02/2020 à 14:54, Masami Hiramatsu a écrit : >>>>>>> Hi, >>>>>>> >>>>>>> On Fri, 14 Feb 2020 12:47:49 +0000 (UTC) >>>>>>> Christophe Leroy <christophe.leroy@c-s.fr> wrote: >>>>>>> >>>>>>>> When a program check exception happens while MMU translation is >>>>>>>> disabled, following Oops happens in kprobe_handler() in the following >>>>>>>> test: >>>>>>>> >>>>>>>> } else if (*addr != BREAKPOINT_INSTRUCTION) { >>>>>>> >>>>>>> Thanks for the report and patch. I'm not so sure about powerpc implementation >>>>>>> but at where the MMU translation is disabled, can the handler work correctly? >>>>>>> (And where did you put the probe on?) >>>>>>> >>>>>>> Your fix may fix this Oops, but if the handler needs special care, it is an >>>>>>> option to blacklist such place (if possible). >>>>>> >>>>>> I guess that's another story. Here we are not talking about a place >>>>>> where kprobe has been illegitimately activated, but a place where there >>>>>> is a valid trap, which generated a valid 'program check exception'. And >>>>>> kprobe was off at that time. >>>>> >>>>> Ah, I got it. It is not a kprobe breakpoint, but to check that correctly, >>>>> it has to know the address where the breakpoint happens. OK. >>>>> >>>>>> >>>>>> As any 'program check exception' due to a trap (ie a BUG_ON, a WARN_ON, >>>>>> a debugger breakpoint, a perf breakpoint, etc...) calls >>>>>> kprobe_handler(), kprobe_handler() must be prepared to handle the case >>>>>> where the MMU translation is disabled, even if probes are not supposed >>>>>> to be set for functions running with MMU translation disabled. >>>>> >>>>> Can't we check the MMU is disabled there (as same as checking the exception >>>>> happened in user space or not)? >>>>> >>>> >>>> What do you mean by 'there' ? At the entry of kprobe_handler() ? >>>> >>>> That's what my patch does, it checks whether MMU is disabled or not. If >>>> it is, it converts the address to a virtual address. >>>> >>>> Do you mean kprobe_handler() should bail out early as it does when the >>>> trap happens in user mode ? >>> >>> Yes, that is what I meant. >>> >>>> Of course we can do that, I don't know >>>> enough about kprobe to know if kprobe_handler() should manage events >>>> that happened in real-mode or just ignore them. But I tested adding an >>>> event on a function that runs in real-mode, and it (now) works. >>>> >>>> So, what should we do really ? >>> >>> I'm not sure how the powerpc kernel runs in real mode. >>> But clearly, at least kprobe event can not handle that case because >>> it tries to access memory by probe_kernel_read(). Unless that function >>> correctly handles the address translation, I want to prohibit kprobes >>> on such address. >>> >>> So what I would like to see is, something like below. >>> >>> diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c >>> index 2d27ec4feee4..4771be152416 100644 >>> --- a/arch/powerpc/kernel/kprobes.c >>> +++ b/arch/powerpc/kernel/kprobes.c >>> @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) >>> unsigned int *addr = (unsigned int *)regs->nip; >>> struct kprobe_ctlblk *kcb; >>> >>> - if (user_mode(regs)) >>> + if (user_mode(regs) || !(regs->msr & MSR_IR)) >>> return 0; >>> >>> /* >>> >>> >> >> With this instead change of my patch, I get an Oops everytime a kprobe >> event occurs in real-mode. >> >> This is because kprobe_handler() is now saying 'this trap doesn't belong >> to me' for a trap that has been installed by it. > > Hmm, on powerpc, kprobes is allowed to probe on the code which runs > in the real mode? I think we should also prohibit it by blacklisting. > (It is easy to add blacklist by NOKPROBE_SYMBOL(func)) Yes, I see a lot of them tagged with _ASM_NOKPROBE_SYMBOL() on PPC64, but none on PPC32. I suppose that's missing and have to be added. Nevertheless, if one symbol has been forgotten in the blacklist, I think it is a problem if it generate Oopses. > Or, some parts are possble to run under both real mode and kernel mode? I don't think so, at least on PPC32 > >> >> So the 'program check' exception handler doesn't find the owner of the >> trap hence generate an Oops. >> >> Even if we don't want kprobe() to proceed with the event entirely >> (allthough it works at least for simple events), I'd expect it to fail >> gracefully. > > Agreed. I thought it was easy to identify real mode code. But if it is > hard, we should apply your first patch and also skip user handlers > if we are in the real mode (and increment missed count). user handlers are already skipped. What do you think about my latest proposal below ? If a trap is encoutered in real mode, if checks if the matching virtual address corresponds to a valid kprobe. If it is, it skips it. If not, it returns 0 to tell "it's no me". You are also talking about incrementing the missed count. Who do we do that ? @@ -264,6 +265,13 @@ int kprobe_handler(struct pt_regs *regs) if (user_mode(regs)) return 0; + if (!(regs->msr & MSR_IR)) { + if (!get_kprobe(phys_to_virt(regs->nip))) + return 0; + regs->nip += 4; + return 1; + } + /* * We don't want to be preempted for the entire * duration of kprobe processing > > BTW, can the emulater handle the real mode code correctly? I don't know, how do I test that ? Christophe
On Tue, 18 Feb 2020 06:58:06 +0100 Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>>> > >>>> What do you mean by 'there' ? At the entry of kprobe_handler() ? > >>>> > >>>> That's what my patch does, it checks whether MMU is disabled or not. If > >>>> it is, it converts the address to a virtual address. > >>>> > >>>> Do you mean kprobe_handler() should bail out early as it does when the > >>>> trap happens in user mode ? > >>> > >>> Yes, that is what I meant. > >>> > >>>> Of course we can do that, I don't know > >>>> enough about kprobe to know if kprobe_handler() should manage events > >>>> that happened in real-mode or just ignore them. But I tested adding an > >>>> event on a function that runs in real-mode, and it (now) works. > >>>> > >>>> So, what should we do really ? > >>> > >>> I'm not sure how the powerpc kernel runs in real mode. > >>> But clearly, at least kprobe event can not handle that case because > >>> it tries to access memory by probe_kernel_read(). Unless that function > >>> correctly handles the address translation, I want to prohibit kprobes > >>> on such address. > >>> > >>> So what I would like to see is, something like below. > >>> > >>> diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c > >>> index 2d27ec4feee4..4771be152416 100644 > >>> --- a/arch/powerpc/kernel/kprobes.c > >>> +++ b/arch/powerpc/kernel/kprobes.c > >>> @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) > >>> unsigned int *addr = (unsigned int *)regs->nip; > >>> struct kprobe_ctlblk *kcb; > >>> > >>> - if (user_mode(regs)) > >>> + if (user_mode(regs) || !(regs->msr & MSR_IR)) > >>> return 0; > >>> > >>> /* > >>> > >>> > >> > >> With this instead change of my patch, I get an Oops everytime a kprobe > >> event occurs in real-mode. > >> > >> This is because kprobe_handler() is now saying 'this trap doesn't belong > >> to me' for a trap that has been installed by it. > > > > Hmm, on powerpc, kprobes is allowed to probe on the code which runs > > in the real mode? I think we should also prohibit it by blacklisting. > > (It is easy to add blacklist by NOKPROBE_SYMBOL(func)) > > Yes, I see a lot of them tagged with _ASM_NOKPROBE_SYMBOL() on PPC64, > but none on PPC32. I suppose that's missing and have to be added. Ah, you are using PPC32. > Nevertheless, if one symbol has been forgotten in the blacklist, I think > it is a problem if it generate Oopses. There is a long history also on x86 to make a blacklist. Anyway, how did you get this error on PPC32? Somewhere would you like to probe and it is a real mode function? Or, it happened unexpectedly? > > > Or, some parts are possble to run under both real mode and kernel mode? > > I don't think so, at least on PPC32 OK, that's a good news. Also, are there any independent section where such real mode functions are stored? (I can see start_real_trampolines in sections.h) If that kind of sections are defined, it is easy to make a blacklist in arch_populate_kprobe_blacklist(). See arch/arm64/kernel/probes/kprobes.c. > >> So the 'program check' exception handler doesn't find the owner of the > >> trap hence generate an Oops. > >> > >> Even if we don't want kprobe() to proceed with the event entirely > >> (allthough it works at least for simple events), I'd expect it to fail > >> gracefully. > > > > Agreed. I thought it was easy to identify real mode code. But if it is > > hard, we should apply your first patch and also skip user handlers > > if we are in the real mode (and increment missed count). > > user handlers are already skipped. Yes, if you don't put a kprobes on real mode code. However, if user (accidentally) puts a probe on real mode code, it might call a user handler? > > What do you think about my latest proposal below ? If a trap is > encoutered in real mode, if checks if the matching virtual address > corresponds to a valid kprobe. If it is, it skips it. If not, it returns > 0 to tell "it's no me". You are also talking about incrementing the > missed count. Who do we do that ? I rather like your first patch. If there is a kprobes, we can not skip the instruction, because there is an instruction which must be executed. (or single-skipped, but I'm not sure the emulator works correctly on real mode) Thank you, > > > > @@ -264,6 +265,13 @@ int kprobe_handler(struct pt_regs *regs) > if (user_mode(regs)) > return 0; > > + if (!(regs->msr & MSR_IR)) { > + if (!get_kprobe(phys_to_virt(regs->nip))) > + return 0; > + regs->nip += 4; > + return 1; > + } > + > /* > * We don't want to be preempted for the entire > * duration of kprobe processing > > > > > > BTW, can the emulater handle the real mode code correctly? > > I don't know, how do I test that ? > > Christophe
Le 18/02/2020 à 11:29, Masami Hiramatsu a écrit : > On Tue, 18 Feb 2020 06:58:06 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>>>>> >>>>>> What do you mean by 'there' ? At the entry of kprobe_handler() ? >>>>>> >>>>>> That's what my patch does, it checks whether MMU is disabled or not. If >>>>>> it is, it converts the address to a virtual address. >>>>>> >>>>>> Do you mean kprobe_handler() should bail out early as it does when the >>>>>> trap happens in user mode ? >>>>> >>>>> Yes, that is what I meant. >>>>> >>>>>> Of course we can do that, I don't know >>>>>> enough about kprobe to know if kprobe_handler() should manage events >>>>>> that happened in real-mode or just ignore them. But I tested adding an >>>>>> event on a function that runs in real-mode, and it (now) works. >>>>>> >>>>>> So, what should we do really ? >>>>> >>>>> I'm not sure how the powerpc kernel runs in real mode. >>>>> But clearly, at least kprobe event can not handle that case because >>>>> it tries to access memory by probe_kernel_read(). Unless that function >>>>> correctly handles the address translation, I want to prohibit kprobes >>>>> on such address. >>>>> >>>>> So what I would like to see is, something like below. >>>>> >>>>> diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c >>>>> index 2d27ec4feee4..4771be152416 100644 >>>>> --- a/arch/powerpc/kernel/kprobes.c >>>>> +++ b/arch/powerpc/kernel/kprobes.c >>>>> @@ -261,7 +261,7 @@ int kprobe_handler(struct pt_regs *regs) >>>>> unsigned int *addr = (unsigned int *)regs->nip; >>>>> struct kprobe_ctlblk *kcb; >>>>> >>>>> - if (user_mode(regs)) >>>>> + if (user_mode(regs) || !(regs->msr & MSR_IR)) >>>>> return 0; >>>>> >>>>> /* >>>>> >>>>> >>>> >>>> With this instead change of my patch, I get an Oops everytime a kprobe >>>> event occurs in real-mode. >>>> >>>> This is because kprobe_handler() is now saying 'this trap doesn't belong >>>> to me' for a trap that has been installed by it. >>> >>> Hmm, on powerpc, kprobes is allowed to probe on the code which runs >>> in the real mode? I think we should also prohibit it by blacklisting. >>> (It is easy to add blacklist by NOKPROBE_SYMBOL(func)) >> >> Yes, I see a lot of them tagged with _ASM_NOKPROBE_SYMBOL() on PPC64, >> but none on PPC32. I suppose that's missing and have to be added. > > Ah, you are using PPC32. > >> Nevertheless, if one symbol has been forgotten in the blacklist, I think >> it is a problem if it generate Oopses. > > There is a long history also on x86 to make a blacklist. Anyway, how did > you get this error on PPC32? Somewhere would you like to probe and > it is a real mode function? Or, it happened unexpectedly? The first Oops I got was triggered by a WARN_ON() kind of trap in real mode. The trap exception handler called kprobe_handler() which tried to read the instruction at the trap address (which was a real-mode address) so it triggered a Bad Access Fault. This was initially the purpose of my patch. After discussion with you, I started looking at what would be the effect of setting a kprobe event in a function which runs in real mode. > >> >>> Or, some parts are possble to run under both real mode and kernel mode? >> >> I don't think so, at least on PPC32 > > OK, that's a good news. Also, are there any independent section where such > real mode functions are stored? (I can see start_real_trampolines in > sections.h) If that kind of sections are defined, it is easy to make > a blacklist in arch_populate_kprobe_blacklist(). See arch/arm64/kernel/probes/kprobes.c. Part of them are in .head.text, and this section is already blacklisted throught function arch_within_kprobe_blacklist() But there are several other functions which are not there. For instance, many things within entry_32.S, and also things in hash_low.S On PPC64 (ie in entry_64.S) they were explicitely blacklisted with _ASM_NOKPROBE_SYMBOL(). We have to do the same on PPC64 > > >>>> So the 'program check' exception handler doesn't find the owner of the >>>> trap hence generate an Oops. >>>> >>>> Even if we don't want kprobe() to proceed with the event entirely >>>> (allthough it works at least for simple events), I'd expect it to fail >>>> gracefully. >>> >>> Agreed. I thought it was easy to identify real mode code. But if it is >>> hard, we should apply your first patch and also skip user handlers >>> if we are in the real mode (and increment missed count). >> >> user handlers are already skipped. > > Yes, if you don't put a kprobes on real mode code. However, if user > (accidentally) puts a probe on real mode code, it might call a > user handler? Are we talking about the same thing ? Only kernel code can run in real mode, so the following code at the beginning of kprobe_handler() does the job ? if (user_mode(regs)) return 0; > >> >> What do you think about my latest proposal below ? If a trap is >> encoutered in real mode, if checks if the matching virtual address >> corresponds to a valid kprobe. If it is, it skips it. If not, it returns >> 0 to tell "it's no me". You are also talking about incrementing the >> missed count. Who do we do that ? > > I rather like your first patch. If there is a kprobes, we can not skip > the instruction, because there is an instruction which must be executed. > (or single-skipped, but I'm not sure the emulator works correctly on > real mode) Oops, yes of course. Christophe
On Tue, 18 Feb 2020 12:04:41 +0100 Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> Nevertheless, if one symbol has been forgotten in the blacklist, I think > >> it is a problem if it generate Oopses. > > > > There is a long history also on x86 to make a blacklist. Anyway, how did > > you get this error on PPC32? Somewhere would you like to probe and > > it is a real mode function? Or, it happened unexpectedly? > > The first Oops I got was triggered by a WARN_ON() kind of trap in real > mode. The trap exception handler called kprobe_handler() which tried to > read the instruction at the trap address (which was a real-mode address) > so it triggered a Bad Access Fault. > > This was initially the purpose of my patch. OK, then filtering the trap reason in kprobe handler is a bit strange. It should be done in the previous stage (maybe in trap.c) Can we filter it by exception flag or only by checking the instruction which causes the exception, or needs get_kprobe()...? > After discussion with you, I started looking at what would be the effect > of setting a kprobe event in a function which runs in real mode. If the kprobe single-stepping (or emulation) works in real mode, just ignore the kprobes pre/post_handlers and increment nmissed count. If that doesn't work, we have to call a BUG_ON, because we can not continue the code execution. And also, you have to find a way to make a blacklist for real mode code. > >> > >>> Or, some parts are possble to run under both real mode and kernel mode? > >> > >> I don't think so, at least on PPC32 > > > > OK, that's a good news. Also, are there any independent section where such > > real mode functions are stored? (I can see start_real_trampolines in > > sections.h) If that kind of sections are defined, it is easy to make > > a blacklist in arch_populate_kprobe_blacklist(). See arch/arm64/kernel/probes/kprobes.c. > > Part of them are in .head.text, and this section is already blacklisted > throught function arch_within_kprobe_blacklist() Then, those are OK. > > But there are several other functions which are not there. For instance, > many things within entry_32.S, and also things in hash_low.S > On PPC64 (ie in entry_64.S) they were explicitely blacklisted with > _ASM_NOKPROBE_SYMBOL(). We have to do the same on PPC64 Agreed. Some of such unstable state code must not be probed. > >>>> So the 'program check' exception handler doesn't find the owner of the > >>>> trap hence generate an Oops. > >>>> > >>>> Even if we don't want kprobe() to proceed with the event entirely > >>>> (allthough it works at least for simple events), I'd expect it to fail > >>>> gracefully. > >>> > >>> Agreed. I thought it was easy to identify real mode code. But if it is > >>> hard, we should apply your first patch and also skip user handlers > >>> if we are in the real mode (and increment missed count). > >> > >> user handlers are already skipped. > > > > Yes, if you don't put a kprobes on real mode code. However, if user > > (accidentally) puts a probe on real mode code, it might call a > > user handler? > > Are we talking about the same thing ? Ah, sorry about that. "user handler" here I meant was "kprobe pre/post_handler function defined by the user of kprobes". > > Only kernel code can run in real mode, so the following code at the > beginning of kprobe_handler() does the job ? > > if (user_mode(regs)) > return 0; Yes, you're right. > >> What do you think about my latest proposal below ? If a trap is > >> encoutered in real mode, if checks if the matching virtual address > >> corresponds to a valid kprobe. If it is, it skips it. If not, it returns > >> 0 to tell "it's no me". You are also talking about incrementing the > >> missed count. Who do we do that ? > > > > I rather like your first patch. If there is a kprobes, we can not skip > > the instruction, because there is an instruction which must be executed. > > (or single-skipped, but I'm not sure the emulator works correctly on > > real mode) > > Oops, yes of course. Thank you,
Le 18/02/2020 à 13:33, Masami Hiramatsu a écrit : > On Tue, 18 Feb 2020 12:04:41 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >>>> Nevertheless, if one symbol has been forgotten in the blacklist, I think >>>> it is a problem if it generate Oopses. >>> >>> There is a long history also on x86 to make a blacklist. Anyway, how did >>> you get this error on PPC32? Somewhere would you like to probe and >>> it is a real mode function? Or, it happened unexpectedly? >> >> The first Oops I got was triggered by a WARN_ON() kind of trap in real >> mode. The trap exception handler called kprobe_handler() which tried to >> read the instruction at the trap address (which was a real-mode address) >> so it triggered a Bad Access Fault. >> >> This was initially the purpose of my patch. > > OK, then filtering the trap reason in kprobe handler is a bit strange. > It should be done in the previous stage (maybe in trap.c) See commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.6-rc2&id=6cc89bad60a673a24386f1ada83de8a068a78909 > Can we filter it by exception flag or only by checking the instruction > which causes the exception, or needs get_kprobe()...? The trap instruction used by kprobe is also used for other purposes like BUG_ON() or WARN_ON(), so needs get_kprobe() > >> After discussion with you, I started looking at what would be the effect >> of setting a kprobe event in a function which runs in real mode. > > If the kprobe single-stepping (or emulation) works in real mode, just > ignore the kprobes pre/post_handlers and increment nmissed count. > > If that doesn't work, we have to call a BUG_ON, because we can not > continue the code execution. And also, you have to find a way to make > a blacklist for real mode code. Yes, it has to be done function by function (hoppefully there's not more than a dozen). But I'd like something which can fails gracefully for the functions we will forget to mark noprobe. But as a first step I'd really like a bug fix in 5.6 to avoid Oopsing in kprobe_handler() at a non-kprobe trap. Christophe
Masami, Christophe, Apologies for pitching in late here... Masami Hiramatsu wrote: > On Tue, 18 Feb 2020 12:04:41 +0100 > Christophe Leroy <christophe.leroy@c-s.fr> wrote: > >> >> Nevertheless, if one symbol has been forgotten in the blacklist, I think >> >> it is a problem if it generate Oopses. >> > >> > There is a long history also on x86 to make a blacklist. Anyway, how did >> > you get this error on PPC32? Somewhere would you like to probe and >> > it is a real mode function? Or, it happened unexpectedly? >> >> The first Oops I got was triggered by a WARN_ON() kind of trap in real >> mode. The trap exception handler called kprobe_handler() which tried to >> read the instruction at the trap address (which was a real-mode address) >> so it triggered a Bad Access Fault. >> >> This was initially the purpose of my patch. > > OK, then filtering the trap reason in kprobe handler is a bit strange. > It should be done in the previous stage (maybe in trap.c) > Can we filter it by exception flag or only by checking the instruction > which causes the exception, or needs get_kprobe()...? I think Masami's earlier patch proposal to bail out early from kprobe_handler() is appropriate here. We don't support kprobe in real mode since we don't have a way to ensure that the pre/post handlers work properly. We will obviously also have to blacklist some of the real mode code from being probed to begin with. In addition, we will also have to blacklist any location where we can't take a trap (MSR_RI being unset, as an example) Christophe, See some of the below patch series: https://patchwork.ozlabs.org/patch/752336/ https://patchwork.ozlabs.org/patch/752333/ https://patchwork.ozlabs.org/patch/782399/ - Naveen
diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 2d27ec4feee4..f8b848aa65bd 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -23,6 +23,7 @@ #include <asm/cacheflush.h> #include <asm/sstep.h> #include <asm/sections.h> +#include <asm/io.h> #include <linux/uaccess.h> DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; @@ -264,6 +265,9 @@ int kprobe_handler(struct pt_regs *regs) if (user_mode(regs)) return 0; + if (!(regs->msr & MSR_IR)) + addr = phys_to_virt(regs->nip); + /* * We don't want to be preempted for the entire * duration of kprobe processing
When a program check exception happens while MMU translation is disabled, following Oops happens in kprobe_handler() in the following test: } else if (*addr != BREAKPOINT_INSTRUCTION) { [ 33.098554] BUG: Unable to handle kernel data access on read at 0x0000e268 [ 33.105091] Faulting instruction address: 0xc000ec34 [ 33.110010] Oops: Kernel access of bad area, sig: 11 [#1] [ 33.115348] BE PAGE_SIZE=16K PREEMPT CMPC885 [ 33.119540] Modules linked in: [ 33.122591] CPU: 0 PID: 429 Comm: cat Not tainted 5.6.0-rc1-s3k-dev-00824-g84195dc6c58a #3267 [ 33.131005] NIP: c000ec34 LR: c000ecd8 CTR: c019cab8 [ 33.136002] REGS: ca4d3b58 TRAP: 0300 Not tainted (5.6.0-rc1-s3k-dev-00824-g84195dc6c58a) [ 33.144324] MSR: 00001032 <ME,IR,DR,RI> CR: 2a4d3c52 XER: 00000000 [ 33.150699] DAR: 0000e268 DSISR: c0000000 [ 33.150699] GPR00: c000b09c ca4d3c10 c66d0620 00000000 ca4d3c60 00000000 00009032 00000000 [ 33.150699] GPR08: 00020000 00000000 c087de44 c000afe0 c66d0ad0 100d3dd6 fffffff3 00000000 [ 33.150699] GPR16: 00000000 00000041 00000000 ca4d3d70 00000000 00000000 0000416d 00000000 [ 33.150699] GPR24: 00000004 c53b6128 00000000 0000e268 00000000 c07c0000 c07bb6fc ca4d3c60 [ 33.188015] NIP [c000ec34] kprobe_handler+0x128/0x290 [ 33.192989] LR [c000ecd8] kprobe_handler+0x1cc/0x290 [ 33.197854] Call Trace: [ 33.200340] [ca4d3c30] [c000b09c] program_check_exception+0xbc/0x6fc [ 33.206590] [ca4d3c50] [c000e43c] ret_from_except_full+0x0/0x4 [ 33.212392] --- interrupt: 700 at 0xe268 [ 33.270401] Instruction dump: [ 33.273335] 913e0008 81220000 38600001 3929ffff 91220000 80010024 bb410008 7c0803a6 [ 33.280992] 38210020 4e800020 38600000 4e800020 <813b0000> 6d2a7fe0 2f8a0008 419e0154 [ 33.288841] ---[ end trace 5b9152d4cdadd06d ]--- Check MSR and convert regs->nip to virtual address if the trap happened with MSR_IR cleared. Reported-by: Larry Finger <Larry.Finger@lwfinger.net> Fixes: 6cc89bad60a6 ("powerpc/kprobes: Invoke handlers directly") Cc: stable@kernel.vger.org Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> --- The bug might have existed even before that commit from Naveen. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> --- arch/powerpc/kernel/kprobes.c | 4 ++++ 1 file changed, 4 insertions(+)