| Message ID | 1381120226-14838-1-git-send-email-khandual@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Michael Ellerman |
| Headers | show |
On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: > Right now the `config_bhrb` PMU specific call happens after write_mmcr0 > which actually enables the PMU for event counting and interrupt. So > there is a small window of time where the PMU and BHRB runs without the > required HW branch filter (if any) enabled in BHRB. This can cause some > of the branch samples to be collected through BHRB without any filter > being applied and hence affecting the correctness of the results. This > patch moves the BHRB config function call before enabling the interrupts. Patch looks good. But it reminds me I have an item in my TODO list: - "Why can't config_bhrb() be done in compute_mmcr()" ? cheers
On 10/08/2013 09:51 AM, Michael Ellerman wrote: > On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: >> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 >> which actually enables the PMU for event counting and interrupt. So >> there is a small window of time where the PMU and BHRB runs without the >> required HW branch filter (if any) enabled in BHRB. This can cause some >> of the branch samples to be collected through BHRB without any filter >> being applied and hence affecting the correctness of the results. This >> patch moves the BHRB config function call before enabling the interrupts. > > Patch looks good. > > But it reminds me I have an item in my TODO list: > - "Why can't config_bhrb() be done in compute_mmcr()" ? > compute_mmcr() function deals with generic MMCR* configs for normal PMU events. Even if BHRB config touches MMCRA register, it's configuration does not interfere with the PMU config for general events. So its best to keep them separate. Besides, we can always look at these code consolidation issues in future. But this patch solves a problem which is happening right now. Regards Anshuman
On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: > On 10/08/2013 09:51 AM, Michael Ellerman wrote: > > On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: > >> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 > >> which actually enables the PMU for event counting and interrupt. So > >> there is a small window of time where the PMU and BHRB runs without the > >> required HW branch filter (if any) enabled in BHRB. This can cause some > >> of the branch samples to be collected through BHRB without any filter > >> being applied and hence affecting the correctness of the results. This > >> patch moves the BHRB config function call before enabling the interrupts. > > > > Patch looks good. > > > > But it reminds me I have an item in my TODO list: > > - "Why can't config_bhrb() be done in compute_mmcr()" ? > > > > compute_mmcr() function deals with generic MMCR* configs for normal PMU > events. Even if BHRB config touches MMCRA register, it's configuration > does not interfere with the PMU config for general events. So its best > to keep them separate. I'm unconvinced. If they'd been together to begin with this bug never would have happened. And there's the added overhead of another indirect function call. > Besides, we can always look at these code consolidation > issues in future. The future is now. > But this patch solves a problem which is happening right now. Sure, I'm not saying we shouldn't merge it as a fix. But I think we should do the cleanup to move it into compute_mmcr() for 3.13. cheers
On 10/09/2013 06:51 AM, Michael Ellerman wrote: > On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: >> On 10/08/2013 09:51 AM, Michael Ellerman wrote: >>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: >>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 >>>> which actually enables the PMU for event counting and interrupt. So >>>> there is a small window of time where the PMU and BHRB runs without the >>>> required HW branch filter (if any) enabled in BHRB. This can cause some >>>> of the branch samples to be collected through BHRB without any filter >>>> being applied and hence affecting the correctness of the results. This >>>> patch moves the BHRB config function call before enabling the interrupts. >>> >>> Patch looks good. >>> >>> But it reminds me I have an item in my TODO list: >>> - "Why can't config_bhrb() be done in compute_mmcr()" ? >>> >> >> compute_mmcr() function deals with generic MMCR* configs for normal PMU >> events. Even if BHRB config touches MMCRA register, it's configuration >> does not interfere with the PMU config for general events. So its best >> to keep them separate. > > I'm unconvinced. If they'd been together to begin with this bug never > would have happened. This is an ordering of configuration problem. Putting them together in the same function does not rule out the chances of this ordering problem. Could you please kindly explain how this could have been avoided ? > > And there's the added overhead of another indirect function call. > This overhead should be minimal given the fact that we already call so many PMU specific indirect calls. BHRB is a different part of the PMU hardware, so a separate call for this purpose is not a bad idea. AFAIK, X86 does that too for LBR. But yes it is debatable. >> Besides, we can always look at these code consolidation >> issues in future. > > The future is now. What I meant was functional correctness has always more priority than code consolidation efforts. Yes I will look into this after book3s software branch filtering code has been merged. > >> But this patch solves a problem which is happening right now. > > Sure, I'm not saying we shouldn't merge it as a fix. But I think we > should do the cleanup to move it into compute_mmcr() for 3.13. yeah that sounds reasonable. Regards Anshuman
On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote: > On 10/09/2013 06:51 AM, Michael Ellerman wrote: > > On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: > >> On 10/08/2013 09:51 AM, Michael Ellerman wrote: > >>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: > >>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 > >>>> which actually enables the PMU for event counting and interrupt. So > >>>> there is a small window of time where the PMU and BHRB runs without the > >>>> required HW branch filter (if any) enabled in BHRB. This can cause some > >>>> of the branch samples to be collected through BHRB without any filter > >>>> being applied and hence affecting the correctness of the results. This > >>>> patch moves the BHRB config function call before enabling the interrupts. > >>> > >>> Patch looks good. > >>> > >>> But it reminds me I have an item in my TODO list: > >>> - "Why can't config_bhrb() be done in compute_mmcr()" ? > >>> > >> > >> compute_mmcr() function deals with generic MMCR* configs for normal PMU > >> events. Even if BHRB config touches MMCRA register, it's configuration > >> does not interfere with the PMU config for general events. So its best > >> to keep them separate. > > > > I'm unconvinced. If they'd been together to begin with this bug never > > would have happened. > > This is an ordering of configuration problem. Putting them together in the > same function does not rule out the chances of this ordering problem. Could > you please kindly explain how this could have been avoided ? The existing code already makes sure to write MMCRA before MMCR0. cheers
On 10/09/2013 11:33 AM, Michael Ellerman wrote: > On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote: >> On 10/09/2013 06:51 AM, Michael Ellerman wrote: >>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: >>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote: >>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: >>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 >>>>>> which actually enables the PMU for event counting and interrupt. So >>>>>> there is a small window of time where the PMU and BHRB runs without the >>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some >>>>>> of the branch samples to be collected through BHRB without any filter >>>>>> being applied and hence affecting the correctness of the results. This >>>>>> patch moves the BHRB config function call before enabling the interrupts. >>>>> >>>>> Patch looks good. >>>>> >>>>> But it reminds me I have an item in my TODO list: >>>>> - "Why can't config_bhrb() be done in compute_mmcr()" ? >>>>> >>>> >>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU >>>> events. Even if BHRB config touches MMCRA register, it's configuration >>>> does not interfere with the PMU config for general events. So its best >>>> to keep them separate. >>> >>> I'm unconvinced. If they'd been together to begin with this bug never >>> would have happened. >> >> This is an ordering of configuration problem. Putting them together in the >> same function does not rule out the chances of this ordering problem. Could >> you please kindly explain how this could have been avoided ? > > The existing code already makes sure to write MMCRA before MMCR0. > Thats not true. One example being here at power_pmu_enable function. write_mmcr0(cpuhw, mmcr0); /* * Enable instruction sampling if necessary */ if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) { mb(); mtspr(SPRN_MMCRA, cpuhw->mmcr[2]); } Even I think this is not right. Instruction sampling should have been enabled before we enable PMU interrupts. Else there is a small window of time where we could have the PMU enabled with events (which requires sampling) without the sampling itself being enabled in MMCRA. The only dependency BHRB and generic events have with each other is that they both are ready for action once the PMU interrupt has been enabled with MMCR0_PMXE bit. Regards Anshuman
On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote: > On 10/09/2013 11:33 AM, Michael Ellerman wrote: > > On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote: > >> On 10/09/2013 06:51 AM, Michael Ellerman wrote: > >>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: > >>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote: > >>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: > >>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 > >>>>>> which actually enables the PMU for event counting and interrupt. So > >>>>>> there is a small window of time where the PMU and BHRB runs without the > >>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some > >>>>>> of the branch samples to be collected through BHRB without any filter > >>>>>> being applied and hence affecting the correctness of the results. This > >>>>>> patch moves the BHRB config function call before enabling the interrupts. > >>>>> > >>>>> Patch looks good. > >>>>> > >>>>> But it reminds me I have an item in my TODO list: > >>>>> - "Why can't config_bhrb() be done in compute_mmcr()" ? > >>>>> > >>>> > >>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU > >>>> events. Even if BHRB config touches MMCRA register, it's configuration > >>>> does not interfere with the PMU config for general events. So its best > >>>> to keep them separate. > >>> > >>> I'm unconvinced. If they'd been together to begin with this bug never > >>> would have happened. > >> > >> This is an ordering of configuration problem. Putting them together in the > >> same function does not rule out the chances of this ordering problem. Could > >> you please kindly explain how this could have been avoided ? > > > > The existing code already makes sure to write MMCRA before MMCR0. > > Thats not true. One example being here at power_pmu_enable function. > > write_mmcr0(cpuhw, mmcr0); > > /* > * Enable instruction sampling if necessary > */ > if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) { > mb(); > mtspr(SPRN_MMCRA, cpuhw->mmcr[2]); > } The only example. The BHRB config would have been applied prior to that: mtspr(SPRN_MMCRA, cpuhw->mmcr[2] & ~MMCRA_SAMPLE_ENABLE); mtspr(SPRN_MMCR1, cpuhw->mmcr[1]); mtspr(SPRN_MMCR0, (cpuhw->mmcr[0] & ~(MMCR0_PMC1CE | MMCR0_PMCjCE)) | MMCR0_FC); So as I said, if the BHRB config was in cpuhw->mmcr[2] then the ordering would have been correct. > Even I think this is not right. Instruction sampling should have been > enabled before we enable PMU interrupts. Else there is a small window > of time where we could have the PMU enabled with events (which requires > sampling) without the sampling itself being enabled in MMCRA. Yes I agree. That's a separate bug, which we'll need to test on all the book3s platforms we have perf support for. cheers
On 10/11/2013 07:41 AM, Michael Ellerman wrote: > On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote: >> On 10/09/2013 11:33 AM, Michael Ellerman wrote: >>> On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote: >>>> On 10/09/2013 06:51 AM, Michael Ellerman wrote: >>>>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote: >>>>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote: >>>>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote: >>>>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0 >>>>>>>> which actually enables the PMU for event counting and interrupt. So >>>>>>>> there is a small window of time where the PMU and BHRB runs without the >>>>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some >>>>>>>> of the branch samples to be collected through BHRB without any filter >>>>>>>> being applied and hence affecting the correctness of the results. This >>>>>>>> patch moves the BHRB config function call before enabling the interrupts. >>>>>>> >>>>>>> Patch looks good. >>>>>>> >>>>>>> But it reminds me I have an item in my TODO list: >>>>>>> - "Why can't config_bhrb() be done in compute_mmcr()" ? >>>>>>> >>>>>> >>>>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU >>>>>> events. Even if BHRB config touches MMCRA register, it's configuration >>>>>> does not interfere with the PMU config for general events. So its best >>>>>> to keep them separate. >>>>> >>>>> I'm unconvinced. If they'd been together to begin with this bug never >>>>> would have happened. >>>> >>>> This is an ordering of configuration problem. Putting them together in the >>>> same function does not rule out the chances of this ordering problem. Could >>>> you please kindly explain how this could have been avoided ? >>> >>> The existing code already makes sure to write MMCRA before MMCR0. >> >> Thats not true. One example being here at power_pmu_enable function. >> >> write_mmcr0(cpuhw, mmcr0); >> >> /* >> * Enable instruction sampling if necessary >> */ >> if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) { >> mb(); >> mtspr(SPRN_MMCRA, cpuhw->mmcr[2]); >> } > > The only example. > > The BHRB config would have been applied prior to that: > > mtspr(SPRN_MMCRA, cpuhw->mmcr[2] & ~MMCRA_SAMPLE_ENABLE); > mtspr(SPRN_MMCR1, cpuhw->mmcr[1]); > mtspr(SPRN_MMCR0, (cpuhw->mmcr[0] & ~(MMCR0_PMC1CE | MMCR0_PMCjCE)) > | MMCR0_FC); > > > So as I said, if the BHRB config was in cpuhw->mmcr[2] then the ordering would > have been correct. I got your point. But ... > >> Even I think this is not right. Instruction sampling should have been >> enabled before we enable PMU interrupts. Else there is a small window >> of time where we could have the PMU enabled with events (which requires >> sampling) without the sampling itself being enabled in MMCRA. > > Yes I agree. That's a separate bug, which we'll need to test on all the book3s > platforms we have perf support for. Okay, I guess any platform which supports sampling will definitely want to have it enabled before we can set the events to count on PMU. Can you think of any problem which can arise if we move it before the enabling the PMU back ? Else we can fix this easily.
On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote: > On 10/11/2013 07:41 AM, Michael Ellerman wrote: > > On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote: > > > >> Even I think this is not right. Instruction sampling should have been > >> enabled before we enable PMU interrupts. Else there is a small window > >> of time where we could have the PMU enabled with events (which requires > >> sampling) without the sampling itself being enabled in MMCRA. > > > > Yes I agree. That's a separate bug, which we'll need to test on all the book3s > > platforms we have perf support for. > > Okay, I guess any platform which supports sampling will definitely want to have > it enabled before we can set the events to count on PMU. Can you think of any > problem which can arise if we move it before the enabling the PMU back ? Else > we can fix this easily. In theory it should be a trivial change. But hardware can behave in strange ways, it's possible on some old chip we need to do it the current way for some reason. So although I don't think it will be a problem, it could be, so we will need to test it thoroughly. cheers
On 10/14/2013 11:49 AM, Michael Ellerman wrote: > On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote: >> On 10/11/2013 07:41 AM, Michael Ellerman wrote: >>> On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote: >>> >>>> Even I think this is not right. Instruction sampling should have been >>>> enabled before we enable PMU interrupts. Else there is a small window >>>> of time where we could have the PMU enabled with events (which requires >>>> sampling) without the sampling itself being enabled in MMCRA. >>> >>> Yes I agree. That's a separate bug, which we'll need to test on all the book3s >>> platforms we have perf support for. >> >> Okay, I guess any platform which supports sampling will definitely want to have >> it enabled before we can set the events to count on PMU. Can you think of any >> problem which can arise if we move it before the enabling the PMU back ? Else >> we can fix this easily. > > In theory it should be a trivial change. But hardware can behave in > strange ways, it's possible on some old chip we need to do it the > current way for some reason. > > So although I don't think it will be a problem, it could be, so we > will need to test it thoroughly. So which are the HW chips, you would like to test this fix for possible problems ?
On 10/16/2013 10:00 AM, Anshuman Khandual wrote: > On 10/14/2013 11:49 AM, Michael Ellerman wrote: >> On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote: >>> On 10/11/2013 07:41 AM, Michael Ellerman wrote: >>>> On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote: >>>> >>>>> Even I think this is not right. Instruction sampling should have been >>>>> enabled before we enable PMU interrupts. Else there is a small window >>>>> of time where we could have the PMU enabled with events (which requires >>>>> sampling) without the sampling itself being enabled in MMCRA. >>>> >>>> Yes I agree. That's a separate bug, which we'll need to test on all the book3s >>>> platforms we have perf support for. >>> >>> Okay, I guess any platform which supports sampling will definitely want to have >>> it enabled before we can set the events to count on PMU. Can you think of any >>> problem which can arise if we move it before the enabling the PMU back ? Else >>> we can fix this easily. >> >> In theory it should be a trivial change. But hardware can behave in >> strange ways, it's possible on some old chip we need to do it the >> current way for some reason. >> >> So although I don't think it will be a problem, it could be, so we >> will need to test it thoroughly. > > So which are the HW chips, you would like to test this fix for possible problems ? > Michael, Any updates on this patch ?
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c index 29b89e8..7d2f13c 100644 --- a/arch/powerpc/perf/core-book3s.c +++ b/arch/powerpc/perf/core-book3s.c @@ -1147,6 +1147,9 @@ static void power_pmu_enable(struct pmu *pmu) mmcr0 = ebb_switch_in(ebb, cpuhw->mmcr[0]); mb(); + if (cpuhw->bhrb_users) + ppmu->config_bhrb(cpuhw->bhrb_filter); + write_mmcr0(cpuhw, mmcr0); /* @@ -1158,8 +1161,6 @@ static void power_pmu_enable(struct pmu *pmu) } out: - if (cpuhw->bhrb_users) - ppmu->config_bhrb(cpuhw->bhrb_filter); local_irq_restore(flags); }
Right now the `config_bhrb` PMU specific call happens after write_mmcr0 which actually enables the PMU for event counting and interrupt. So there is a small window of time where the PMU and BHRB runs without the required HW branch filter (if any) enabled in BHRB. This can cause some of the branch samples to be collected through BHRB without any filter being applied and hence affecting the correctness of the results. This patch moves the BHRB config function call before enabling the interrupts. Here are some data points captured via trace prints which depicts how we could get PMU interrupts with BHRB filter NOT enabled with a standard perf record command line (asking for branch record information as well). perf record -j any_call ls Before the patch:- ls-1962 [003] d... 2065.299590: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299603: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299611: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299618: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299625: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299632: .perf_event_interrupt: MMCRA: 40000000000 ls-1962 [003] d... 2065.299639: .perf_event_interrupt: MMCRA: 40000000000 --> All the PMU interrupts before this point did not have the --> requested HW branch filter enabled in the MMCRA. ls-1962 [003] d... 2065.299647: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.299662: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.299700: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.299798: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.299956: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.300145: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.300347: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.300556: .perf_event_interrupt: MMCRA: 40040000000 ls-1962 [003] d... 2065.300771: .perf_event_interrupt: MMCRA: 40040000000 After the patch:- ls-1850 [008] d... 190.311828: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311848: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311856: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311863: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311869: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311876: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311884: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311892: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311907: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.311945: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.312044: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.312206: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.312397: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d..2 190.312626: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.312814: .perf_event_interrupt: MMCRA: 40040000000 ls-1850 [008] d... 190.313004: .perf_event_interrupt: MMCRA: 40040000000 --> All the PMU interrupts have the requested --> HW BHRB branch filter enabled in MMCRA. Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com> --- arch/powerpc/perf/core-book3s.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)