| Message ID | 1460130851-29021-1-git-send-email-thuth@redhat.com |
|---|---|
| State | Changes Requested |
| Headers | show |
On 08.04.2016 17:54, Thomas Huth wrote: > The SIAR register is available twice, one time as SPR 780 (unprivileged, > but read-only), and one time as SPR 796 (privileged, but read and write). > The Linux kernel code currently uses SPR 780 - and while this is OK for > reading, writing to that register of course does not work. > Since the KVM code tries to write to this register, too (see the mtspr > in book3s_hv_rmhandlers.S), the contents of this register sometimes get > lost for the guests, e.g. during migration of a VM. > To fix this issue, simply switch to the other SPR numer 796 instead. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > Note: The perf code in core-book3s.c also seems to write to the SIAR > SPR, so that might be affected by this issue, too - but I did > not test the perf code, so I'm not sure about that part. > > arch/powerpc/include/asm/reg.h | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h > index f5f4c66..6630420 100644 > --- a/arch/powerpc/include/asm/reg.h > +++ b/arch/powerpc/include/asm/reg.h > @@ -752,13 +752,13 @@ > #define SPRN_PMC6 792 > #define SPRN_PMC7 793 > #define SPRN_PMC8 794 > -#define SPRN_SIAR 780 > #define SPRN_SDAR 781 > #define SPRN_SIER 784 > #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ > #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ > #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ > #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ > +#define SPRN_SIAR 796 > #define SPRN_TACR 888 > #define SPRN_TCSCR 889 > #define SPRN_CSIGR 890 Ping! Anybody any comments? Thomas -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Friday 08 April 2016 09:24 PM, Thomas Huth wrote: > The SIAR register is available twice, one time as SPR 780 (unprivileged, > but read-only), and one time as SPR 796 (privileged, but read and write). > The Linux kernel code currently uses SPR 780 - and while this is OK for > reading, writing to that register of course does not work. > Since the KVM code tries to write to this register, too (see the mtspr > in book3s_hv_rmhandlers.S), the contents of this register sometimes get > lost for the guests, e.g. during migration of a VM. > To fix this issue, simply switch to the other SPR numer 796 instead. IIUC, SIAR and SDAR are updated by hardware when we take a pmu exception with sampling mode enabled (based on instr). And these register contents are mainly for OS consumption. So, we dont need to restore these register values at all, kindly correct me if I missing something here. Maddy > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > Note: The perf code in core-book3s.c also seems to write to the SIAR > SPR, so that might be affected by this issue, too - but I did > not test the perf code, so I'm not sure about that part. > > arch/powerpc/include/asm/reg.h | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h > index f5f4c66..6630420 100644 > --- a/arch/powerpc/include/asm/reg.h > +++ b/arch/powerpc/include/asm/reg.h > @@ -752,13 +752,13 @@ > #define SPRN_PMC6 792 > #define SPRN_PMC7 793 > #define SPRN_PMC8 794 > -#define SPRN_SIAR 780 > #define SPRN_SDAR 781 > #define SPRN_SIER 784 > #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ > #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ > #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ > #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ > +#define SPRN_SIAR 796 > #define SPRN_TACR 888 > #define SPRN_TCSCR 889 > #define SPRN_CSIGR 890 -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> Am 25.04.2016 um 10:08 schrieb Madhavan Srinivasan <maddy@linux.vnet.ibm.com>: > > > >> On Friday 08 April 2016 09:24 PM, Thomas Huth wrote: >> The SIAR register is available twice, one time as SPR 780 (unprivileged, >> but read-only), and one time as SPR 796 (privileged, but read and write). >> The Linux kernel code currently uses SPR 780 - and while this is OK for >> reading, writing to that register of course does not work. >> Since the KVM code tries to write to this register, too (see the mtspr >> in book3s_hv_rmhandlers.S), the contents of this register sometimes get >> lost for the guests, e.g. during migration of a VM. >> To fix this issue, simply switch to the other SPR numer 796 instead. > > IIUC, SIAR and SDAR are updated by hardware when we take > a pmu exception with sampling mode enabled (based on instr). > And these register contents are mainly for OS consumption. > So, we dont need to restore these register values at all, > kindly correct me if I missing something here. What if you migrate between a pmu event firing and the os reading siar? Or what if the host gets pmu events? Or we migrate the guest to a different pcpu? In all those cases we need to ensure the register contents are consistent. > > Maddy > >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> Note: The perf code in core-book3s.c also seems to write to the SIAR >> SPR, so that might be affected by this issue, too - but I did >> not test the perf code, so I'm not sure about that part. Please write a small unit test that fires off pmu events constantly and checks whtether they arrive correctly. Run perf in parallel on the host to increase the chance for breakage. >> >> arch/powerpc/include/asm/reg.h | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h >> index f5f4c66..6630420 100644 >> --- a/arch/powerpc/include/asm/reg.h >> +++ b/arch/powerpc/include/asm/reg.h >> @@ -752,13 +752,13 @@ >> #define SPRN_PMC6 792 >> #define SPRN_PMC7 793 >> #define SPRN_PMC8 794 >> -#define SPRN_SIAR 780 >> #define SPRN_SDAR 781 >> #define SPRN_SIER 784 >> #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ >> #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ >> #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ >> #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ >> +#define SPRN_SIAR 796 I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. Alex >> #define SPRN_TACR 888 >> #define SPRN_TCSCR 889 >> #define SPRN_CSIGR 890 > > -- > To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 25.04.2016 10:15, Alexander Graf wrote: > > >> Am 25.04.2016 um 10:08 schrieb Madhavan Srinivasan <maddy@linux.vnet.ibm.com>: >> >> >> >>> On Friday 08 April 2016 09:24 PM, Thomas Huth wrote: >>> The SIAR register is available twice, one time as SPR 780 (unprivileged, >>> but read-only), and one time as SPR 796 (privileged, but read and write). >>> The Linux kernel code currently uses SPR 780 - and while this is OK for >>> reading, writing to that register of course does not work. >>> Since the KVM code tries to write to this register, too (see the mtspr >>> in book3s_hv_rmhandlers.S), the contents of this register sometimes get >>> lost for the guests, e.g. during migration of a VM. >>> To fix this issue, simply switch to the other SPR numer 796 instead. >> >> IIUC, SIAR and SDAR are updated by hardware when we take >> a pmu exception with sampling mode enabled (based on instr). >> And these register contents are mainly for OS consumption. >> So, we dont need to restore these register values at all, >> kindly correct me if I missing something here. > > What if you migrate between a pmu event firing and the os reading siar? Or what if the host gets pmu events? Or we migrate the guest to a different pcpu? In all those cases we need to ensure the register contents are consistent. Right. Or a guest could use the SIAR as a temporary scratch register while not using the performance monitoring stuff. In that case the contents of the register of course have to be preserved, too. >>> Signed-off-by: Thomas Huth <thuth@redhat.com> >>> --- >>> Note: The perf code in core-book3s.c also seems to write to the SIAR >>> SPR, so that might be affected by this issue, too - but I did >>> not test the perf code, so I'm not sure about that part. > > Please write a small unit test that fires off pmu events constantly and checks whtether they arrive correctly. Run perf in parallel on the host to increase the chance for breakage. I'm not very familiar with that PMU stuff yet, but I can have a try... >>> arch/powerpc/include/asm/reg.h | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h >>> index f5f4c66..6630420 100644 >>> --- a/arch/powerpc/include/asm/reg.h >>> +++ b/arch/powerpc/include/asm/reg.h >>> @@ -752,13 +752,13 @@ >>> #define SPRN_PMC6 792 >>> #define SPRN_PMC7 793 >>> #define SPRN_PMC8 794 >>> -#define SPRN_SIAR 780 >>> #define SPRN_SDAR 781 >>> #define SPRN_SIER 784 >>> #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ >>> #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ >>> #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ >>> #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ >>> +#define SPRN_SIAR 796 > > I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. Sure. Any suggestions on the naming? I could either rename the current SPRN_SIAR to SPRN_USIAR (so that it is named similar to other registers that behave that way, like SPRN_USPRG3 - and also QEMU uses USIAR for this already). Or I could leave the old name untouched and use something like "SPRN_SIAR_WR" for the 796 register. What do you prefer? By the way, I just noticed that SPRN_SDAR (781) seems to suffer from the same problem, too! Thomas -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 04/25/2016 11:16 AM, Thomas Huth wrote: > On 25.04.2016 10:15, Alexander Graf wrote: >> >>> Am 25.04.2016 um 10:08 schrieb Madhavan Srinivasan <maddy@linux.vnet.ibm.com>: >>> >>> >>> >>>> On Friday 08 April 2016 09:24 PM, Thomas Huth wrote: >>>> The SIAR register is available twice, one time as SPR 780 (unprivileged, >>>> but read-only), and one time as SPR 796 (privileged, but read and write). >>>> The Linux kernel code currently uses SPR 780 - and while this is OK for >>>> reading, writing to that register of course does not work. >>>> Since the KVM code tries to write to this register, too (see the mtspr >>>> in book3s_hv_rmhandlers.S), the contents of this register sometimes get >>>> lost for the guests, e.g. during migration of a VM. >>>> To fix this issue, simply switch to the other SPR numer 796 instead. >>> IIUC, SIAR and SDAR are updated by hardware when we take >>> a pmu exception with sampling mode enabled (based on instr). >>> And these register contents are mainly for OS consumption. >>> So, we dont need to restore these register values at all, >>> kindly correct me if I missing something here. >> What if you migrate between a pmu event firing and the os reading siar? Or what if the host gets pmu events? Or we migrate the guest to a different pcpu? In all those cases we need to ensure the register contents are consistent. > Right. Or a guest could use the SIAR as a temporary scratch register > while not using the performance monitoring stuff. In that case the > contents of the register of course have to be preserved, too. > >>>> Signed-off-by: Thomas Huth <thuth@redhat.com> >>>> --- >>>> Note: The perf code in core-book3s.c also seems to write to the SIAR >>>> SPR, so that might be affected by this issue, too - but I did >>>> not test the perf code, so I'm not sure about that part. >> Please write a small unit test that fires off pmu events constantly and checks whtether they arrive correctly. Run perf in parallel on the host to increase the chance for breakage. > I'm not very familiar with that PMU stuff yet, but I can have a try... > >>>> arch/powerpc/include/asm/reg.h | 2 +- >>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>> >>>> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h >>>> index f5f4c66..6630420 100644 >>>> --- a/arch/powerpc/include/asm/reg.h >>>> +++ b/arch/powerpc/include/asm/reg.h >>>> @@ -752,13 +752,13 @@ >>>> #define SPRN_PMC6 792 >>>> #define SPRN_PMC7 793 >>>> #define SPRN_PMC8 794 >>>> -#define SPRN_SIAR 780 >>>> #define SPRN_SDAR 781 >>>> #define SPRN_SIER 784 >>>> #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ >>>> #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ >>>> #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ >>>> #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ >>>> +#define SPRN_SIAR 796 >> I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. > Sure. Any suggestions on the naming? I could either rename the current > SPRN_SIAR to SPRN_USIAR (so that it is named similar to other registers > that behave that way, like SPRN_USPRG3 - and also QEMU uses USIAR for > this already). Or I could leave the old name untouched and use something > like "SPRN_SIAR_WR" for the 796 register. What do you prefer? I'd defer that decision to Michael :). > By the way, I just noticed that SPRN_SDAR (781) seems to suffer from the > same problem, too! Great! The more the merrier :) Alex -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Monday 25 April 2016 01:45 PM, Alexander Graf wrote: > >> Am 25.04.2016 um 10:08 schrieb Madhavan Srinivasan <maddy@linux.vnet.ibm.com>: >> >> >> >>> On Friday 08 April 2016 09:24 PM, Thomas Huth wrote: >>> The SIAR register is available twice, one time as SPR 780 (unprivileged, >>> but read-only), and one time as SPR 796 (privileged, but read and write). >>> The Linux kernel code currently uses SPR 780 - and while this is OK for >>> reading, writing to that register of course does not work. >>> Since the KVM code tries to write to this register, too (see the mtspr >>> in book3s_hv_rmhandlers.S), the contents of this register sometimes get >>> lost for the guests, e.g. during migration of a VM. >>> To fix this issue, simply switch to the other SPR numer 796 instead. >> IIUC, SIAR and SDAR are updated by hardware when we take >> a pmu exception with sampling mode enabled (based on instr). >> And these register contents are mainly for OS consumption. >> So, we dont need to restore these register values at all, >> kindly correct me if I missing something here. > What if you migrate between a pmu event firing and the os reading siar? Or what if the host gets pmu events? Or we migrate the guest to a different pcpu? In all those cases we need to ensure the register contents are consistent. Ok got it. Let me try perf record with sample_addr type. Maddy > >> Maddy >> >>> Signed-off-by: Thomas Huth <thuth@redhat.com> >>> --- >>> Note: The perf code in core-book3s.c also seems to write to the SIAR >>> SPR, so that might be affected by this issue, too - but I did >>> not test the perf code, so I'm not sure about that part. > Please write a small unit test that fires off pmu events constantly and checks whtether they arrive correctly. Run perf in parallel on the host to increase the chance for breakage. > >>> arch/powerpc/include/asm/reg.h | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h >>> index f5f4c66..6630420 100644 >>> --- a/arch/powerpc/include/asm/reg.h >>> +++ b/arch/powerpc/include/asm/reg.h >>> @@ -752,13 +752,13 @@ >>> #define SPRN_PMC6 792 >>> #define SPRN_PMC7 793 >>> #define SPRN_PMC8 794 >>> -#define SPRN_SIAR 780 >>> #define SPRN_SDAR 781 >>> #define SPRN_SIER 784 >>> #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ >>> #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ >>> #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ >>> #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ >>> +#define SPRN_SIAR 796 > I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. > > Alex > >>> #define SPRN_TACR 888 >>> #define SPRN_TCSCR 889 >>> #define SPRN_CSIGR 890 >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Apr 08, 2016 at 05:54:11PM +0200, Thomas Huth wrote: > The SIAR register is available twice, one time as SPR 780 (unprivileged, > but read-only), and one time as SPR 796 (privileged, but read and write). > The Linux kernel code currently uses SPR 780 - and while this is OK for > reading, writing to that register of course does not work. > Since the KVM code tries to write to this register, too (see the mtspr > in book3s_hv_rmhandlers.S), the contents of this register sometimes get > lost for the guests, e.g. during migration of a VM. > To fix this issue, simply switch to the other SPR numer 796 instead. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > Note: The perf code in core-book3s.c also seems to write to the SIAR > SPR, so that might be affected by this issue, too - but I did > not test the perf code, so I'm not sure about that part. EBBs mean we need to context-switch the SIAR between user tasks (among other registers). I notice that SDAR is also wrong, and the MMCR2 definition is also using the user-accessible number (though it at least is writable as well as readable). Paul. -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 25, 2016 at 10:15:47AM +0200, Alexander Graf wrote: > > >> +#define SPRN_SIAR 796 > > I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. I don't think we ever did any performance monitoring on iSeries, and in any case, we have removed the iSeries code. I think we should just use the privileged, read/write number (i.e. 796). Paul. -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12.05.2016 06:57, Paul Mackerras wrote: > On Mon, Apr 25, 2016 at 10:15:47AM +0200, Alexander Graf wrote: >> >>>> +#define SPRN_SIAR 796 >> >> I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. > > I don't think we ever did any performance monitoring on iSeries, and > in any case, we have removed the iSeries code. I think we should just > use the privileged, read/write number (i.e. 796). Ok, then I'll simply add the fix for SDAR to my patch as well, test it and submit it again. Thomas -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, May 12, 2016 at 09:27:40AM +0200, Thomas Huth wrote: > On 12.05.2016 06:57, Paul Mackerras wrote: > > On Mon, Apr 25, 2016 at 10:15:47AM +0200, Alexander Graf wrote: > >> > >>>> +#define SPRN_SIAR 796 > >> > >> I'm sure there's a reason (iSeries?) we used the r/o version before. Better introduce a new constant that gives us rw access and use that in the kvm entry/exit code. > > > > I don't think we ever did any performance monitoring on iSeries, and > > in any case, we have removed the iSeries code. I think we should just > > use the privileged, read/write number (i.e. 796). > > Ok, then I'll simply add the fix for SDAR to my patch as well, test it > and submit it again. Great! Could you switch MMCR2 to the privileged number as well while you're at it? Thanks, Paul. -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h index f5f4c66..6630420 100644 --- a/arch/powerpc/include/asm/reg.h +++ b/arch/powerpc/include/asm/reg.h @@ -752,13 +752,13 @@ #define SPRN_PMC6 792 #define SPRN_PMC7 793 #define SPRN_PMC8 794 -#define SPRN_SIAR 780 #define SPRN_SDAR 781 #define SPRN_SIER 784 #define SIER_SIPR 0x2000000 /* Sampled MSR_PR */ #define SIER_SIHV 0x1000000 /* Sampled MSR_HV */ #define SIER_SIAR_VALID 0x0400000 /* SIAR contents valid */ #define SIER_SDAR_VALID 0x0200000 /* SDAR contents valid */ +#define SPRN_SIAR 796 #define SPRN_TACR 888 #define SPRN_TCSCR 889 #define SPRN_CSIGR 890
The SIAR register is available twice, one time as SPR 780 (unprivileged, but read-only), and one time as SPR 796 (privileged, but read and write). The Linux kernel code currently uses SPR 780 - and while this is OK for reading, writing to that register of course does not work. Since the KVM code tries to write to this register, too (see the mtspr in book3s_hv_rmhandlers.S), the contents of this register sometimes get lost for the guests, e.g. during migration of a VM. To fix this issue, simply switch to the other SPR numer 796 instead. Signed-off-by: Thomas Huth <thuth@redhat.com> --- Note: The perf code in core-book3s.c also seems to write to the SIAR SPR, so that might be affected by this issue, too - but I did not test the perf code, so I'm not sure about that part. arch/powerpc/include/asm/reg.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)