| Message ID | 20231030165739.3092137-1-gerald.yang@canonical.com |
|---|---|
| State | New |
| Headers | show |
| Series | [SRU,Jammy,lowlatency] Ubuntu: [Config] lowlatency: enhance desktop responsiveness | expand |
On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: > BugLink: https://bugs.launchpad.net/bugs/2023007 > > [Impact] > > The lowlatency kernel in Ubuntu is specifically designed to prioritize > high responsiveness, making it ideal for multimedia environments like > DAWs and audio processing platforms, as well as soft real-time > environments. > > With the introduction of a real-time kernel, it might be worth > reconsidering the role of the lowlatency kernel and potentially > including it as the default kernel in desktop images, focusing on its > suitability for desktop-oriented usage. > > To achieve this, we can enable additional configuration settings and > make it more focused for a low-latency and highly responsive desktop > environment. > > Optionally (for the future) provide also an additional user-space > package that would enable specific run-time kernel settings focused at > certain preset workload profiles (e.g, web navigation, gaming, audio > processing, etc.). > > [Test case] > > Use linux-lowlatency in a desktop environment and measure responsiveness > of interactive applications. > > [Fix] > > Enable the following additional .config settings to make this kernel > more suitable for a low-latency desktop kernel: > > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown > clock tick when possible across all the enabled CPUs if they are > either idle or running 1 task - reduce kernel jitter of running tasks > due to the periodic clock tick, must be enabled at boot time passing > `nohz_full=<cpu_list>`) > > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU > callbacks from softirq context to kthread context (reduce time spent > in softirqs with preemption disabled to improve the overall system > responsiveness, at the cost of introducing a potential performance > penalty, because RCU callbacks are not processed by kernel threads) > > [Regression potential] > > Enabling all these settings can introduce a potential performance > regression, but the kernel should result more responsive and make > it more suitable for a desktop/multimedia/gaming/audio processing > context. > > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> It's worth mentioning that we already applied these configs to the lowlatency kernels in mantic and in general I don't see any downside if we apply this. However, I'm not sure if a change like this is proper SRU material, we are basically saying "now lowlatency is a desktop kernel everywhere". Certain workloads may experience regressions (especially with the RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be enabled/disabled via boot options). We also need to keep in mind that Jammy will get these changes with the lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport this and apply it everywhere... opinions? -Andrea > --- > debian.lowlatency/config/annotations | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations > index 694ae26b6745..3e1556d60809 100644 > --- a/debian.lowlatency/config/annotations > +++ b/debian.lowlatency/config/annotations > @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i > CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_LATENCYTOP note<'https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, LP#1655986'> > > +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'> > + > +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}> > +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> > + > CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_PREEMPT note<'Enable fully preemptible kernel'> > > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'> > > +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'> > > # ---- Annotations without notes ---- > > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': ' > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}> > CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': ' > CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}> > -- > 2.25.1 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
Hi Andrea, Thanks for the feedback! please let me explain this in more detail I sent out this SRU because some of our customers need to use NO_HZ_FULL. In their scenario, the CPU time is pretty sensitive, and must avoid as much noise as possible to keep most CPUs doing only one single task. I found some CPU steal time happen on their VM because RCU callback softirqs are still triggered on CPUs with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find NO_HZ_FULL is not built into generic kernel. I think the use case for NO_HZ_FULL should be different than generic kernel, e.g. performance vs low latency On generic kernel, we also did some tests with NO_HZ_FULL built-in here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154 The results show performance degradation on AMD EPYC series machines only when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on Intel and arm64 machines. Because one of the customer is still on Focal, if it's possible to merge this into 5.15 kernel, it would be more convenient for them to use HWE lowlatency kernel. And if there are users need to use NO_HZ_FULL on Focal, lowlatency kernel could be a proper choice for them, because so far there is no kernel with NO_HZ_FULL built-in on Focal. Thanks, Gerald On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <andrea.righi@canonical.com> wrote: > On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: > > BugLink: https://bugs.launchpad.net/bugs/2023007 > > > > [Impact] > > > > The lowlatency kernel in Ubuntu is specifically designed to prioritize > > high responsiveness, making it ideal for multimedia environments like > > DAWs and audio processing platforms, as well as soft real-time > > environments. > > > > With the introduction of a real-time kernel, it might be worth > > reconsidering the role of the lowlatency kernel and potentially > > including it as the default kernel in desktop images, focusing on its > > suitability for desktop-oriented usage. > > > > To achieve this, we can enable additional configuration settings and > > make it more focused for a low-latency and highly responsive desktop > > environment. > > > > Optionally (for the future) provide also an additional user-space > > package that would enable specific run-time kernel settings focused at > > certain preset workload profiles (e.g, web navigation, gaming, audio > > processing, etc.). > > > > [Test case] > > > > Use linux-lowlatency in a desktop environment and measure responsiveness > > of interactive applications. > > > > [Fix] > > > > Enable the following additional .config settings to make this kernel > > more suitable for a low-latency desktop kernel: > > > > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown > > clock tick when possible across all the enabled CPUs if they are > > either idle or running 1 task - reduce kernel jitter of running tasks > > due to the periodic clock tick, must be enabled at boot time passing > > `nohz_full=<cpu_list>`) > > > > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU > > callbacks from softirq context to kthread context (reduce time spent > > in softirqs with preemption disabled to improve the overall system > > responsiveness, at the cost of introducing a potential performance > > penalty, because RCU callbacks are not processed by kernel threads) > > > > [Regression potential] > > > > Enabling all these settings can introduce a potential performance > > regression, but the kernel should result more responsive and make > > it more suitable for a desktop/multimedia/gaming/audio processing > > context. > > > > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > > It's worth mentioning that we already applied these configs to the > lowlatency kernels in mantic and in general I don't see any downside if > we apply this. > > However, I'm not sure if a change like this is proper SRU material, we > are basically saying "now lowlatency is a desktop kernel everywhere". > > Certain workloads may experience regressions (especially with the > RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be > enabled/disabled via boot options). > > We also need to keep in mind that Jammy will get these changes with the > lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport > this and apply it everywhere... opinions? > > -Andrea > > > --- > > debian.lowlatency/config/annotations | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > > > diff --git a/debian.lowlatency/config/annotations > b/debian.lowlatency/config/annotations > > index 694ae26b6745..3e1556d60809 100644 > > --- a/debian.lowlatency/config/annotations > > +++ b/debian.lowlatency/config/annotations > > @@ -15,12 +15,20 @@ CONFIG_HZ_250 > note<'Override default HZ used i > > CONFIG_LATENCYTOP policy<{'amd64': 'y', > 'arm64': 'y'}> > > CONFIG_LATENCYTOP note<' > https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, > LP#1655986'> > > > > +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', > 'arm64': 'y'}> > > +CONFIG_NO_HZ_FULL note<'Enable access to > "Full tickless mode" (LP: #2023007)'> > > + > > +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', > 'arm64': 'n'}> > > +CONFIG_NO_HZ_IDLE note<'Disabled in favor > of CONFIG_NO_HZ_FULL (LP: #2023007)'> > > + > > CONFIG_PREEMPT policy<{'amd64': 'y', > 'arm64': 'y'}> > > CONFIG_PREEMPT note<'Enable fully > preemptible kernel'> > > > > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', > 'arm64': 'n'}> > > CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary > preemption model'> > > > > +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', > 'arm64': 'y'}> > > +CONFIG_RCU_NOCB_CPU note<'Move RCU > callbacks from softirq context to kthread context (LP: #2023007)'> > > > > # ---- Annotations without notes ---- > > > > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN > policy<{'amd64': 'y', 'arm64': ' > > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', > 'arm64': 'n'}> > > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', > 'arm64': 'm'}> > > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', > 'arm64': 'm'}> > > +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', > 'arm64': 'y'}> > > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', > 'arm64': 'n'}> > > CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', > 'arm64': 'n'}> > > CONFIG_HZ policy<{'amd64': > '1000', 'arm64': '1000'}> > > CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> > > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU > policy<{'amd64': 'y', 'arm64': ' > > CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', > 'arm64': 'n'}> > > CONFIG_TASKS_RCU policy<{'amd64': 'y', > 'arm64': 'y'}> > > CONFIG_TEST_DIV64 policy<{'amd64': 'm', > 'arm64': 'm'}> > > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', > 'arm64': '-'}> > > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', > 'arm64': 'y'}> > > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', > 'arm64': 'y'}> > > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', > 'arm64': 'y'}> > > -- > > 2.25.1 > > > > > > -- > > kernel-team mailing list > > kernel-team@lists.ubuntu.com > > https://lists.ubuntu.com/mailman/listinfo/kernel-team >
Hi, On Tue, 31 Oct 2023 at 10:54, Gerald Yang <gerald.yang@canonical.com> wrote: > > Hi Andrea, > > Thanks for the feedback! please let me explain this in more detail > > I sent out this SRU because some of our customers need to use NO_HZ_FULL. > In their scenario, the CPU time is pretty sensitive, and must avoid as much noise as possible to keep > most CPUs doing only one single task. > I found some CPU steal time happen on their VM because RCU callback softirqs are still triggered on CPUs > with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find NO_HZ_FULL is not built into generic kernel. > > I think the use case for NO_HZ_FULL should be different than generic kernel, e.g. performance vs low latency > > On generic kernel, we also did some tests with NO_HZ_FULL built-in here: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154 > > The results show performance degradation on AMD EPYC series machines only > when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on Intel and arm64 machines. > > Because one of the customer is still on Focal, if it's possible to merge this into 5.15 kernel, it would be more convenient > for them to use HWE lowlatency kernel. > And if there are users need to use NO_HZ_FULL on Focal, lowlatency kernel could be a proper choice for them, > because so far there is no kernel with NO_HZ_FULL built-in on Focal. I am ok having this enabled in focal:lowlatency-hwe-5.15 and jammy:lowlatency-hwe-6.5 but not change anything in focal:lowlatency (5.4 / ga) jammy:lowlatency (5.15 / ga). Would that help you? As an exception to our policy. I really feel uncomfortable changing default GA flavours, but risk levels of making hwe kernels more like the future sounds safer. > > Thanks, > Gerald > > On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <andrea.righi@canonical.com> wrote: >> >> On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: >> > BugLink: https://bugs.launchpad.net/bugs/2023007 >> > >> > [Impact] >> > >> > The lowlatency kernel in Ubuntu is specifically designed to prioritize >> > high responsiveness, making it ideal for multimedia environments like >> > DAWs and audio processing platforms, as well as soft real-time >> > environments. >> > >> > With the introduction of a real-time kernel, it might be worth >> > reconsidering the role of the lowlatency kernel and potentially >> > including it as the default kernel in desktop images, focusing on its >> > suitability for desktop-oriented usage. >> > >> > To achieve this, we can enable additional configuration settings and >> > make it more focused for a low-latency and highly responsive desktop >> > environment. >> > >> > Optionally (for the future) provide also an additional user-space >> > package that would enable specific run-time kernel settings focused at >> > certain preset workload profiles (e.g, web navigation, gaming, audio >> > processing, etc.). >> > >> > [Test case] >> > >> > Use linux-lowlatency in a desktop environment and measure responsiveness >> > of interactive applications. >> > >> > [Fix] >> > >> > Enable the following additional .config settings to make this kernel >> > more suitable for a low-latency desktop kernel: >> > >> > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown >> > clock tick when possible across all the enabled CPUs if they are >> > either idle or running 1 task - reduce kernel jitter of running tasks >> > due to the periodic clock tick, must be enabled at boot time passing >> > `nohz_full=<cpu_list>`) >> > >> > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU >> > callbacks from softirq context to kthread context (reduce time spent >> > in softirqs with preemption disabled to improve the overall system >> > responsiveness, at the cost of introducing a potential performance >> > penalty, because RCU callbacks are not processed by kernel threads) >> > >> > [Regression potential] >> > >> > Enabling all these settings can introduce a potential performance >> > regression, but the kernel should result more responsive and make >> > it more suitable for a desktop/multimedia/gaming/audio processing >> > context. >> > >> > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> >> >> It's worth mentioning that we already applied these configs to the >> lowlatency kernels in mantic and in general I don't see any downside if >> we apply this. >> >> However, I'm not sure if a change like this is proper SRU material, we >> are basically saying "now lowlatency is a desktop kernel everywhere". >> >> Certain workloads may experience regressions (especially with the >> RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be >> enabled/disabled via boot options). >> >> We also need to keep in mind that Jammy will get these changes with the >> lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport >> this and apply it everywhere... opinions? >> >> -Andrea >> >> > --- >> > debian.lowlatency/config/annotations | 13 +++++++++++++ >> > 1 file changed, 13 insertions(+) >> > >> > diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations >> > index 694ae26b6745..3e1556d60809 100644 >> > --- a/debian.lowlatency/config/annotations >> > +++ b/debian.lowlatency/config/annotations >> > @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i >> > CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_LATENCYTOP note<'https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, LP#1655986'> >> > >> > +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'> >> > + >> > +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}> >> > +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> >> > + >> > CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_PREEMPT note<'Enable fully preemptible kernel'> >> > >> > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'> >> > >> > +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'> >> > >> > # ---- Annotations without notes ---- >> > >> > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': ' >> > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}> >> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}> >> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}> >> > CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> >> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': ' >> > CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}> >> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}> >> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}> >> > -- >> > 2.25.1 >> > >> > >> > -- >> > kernel-team mailing list >> > kernel-team@lists.ubuntu.com >> > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
Hi Dimitri, Thanks for checking this! And yes, that would be very helpful to enable it in focal lowlatency-hwe and jammy lowlatency-hwe kernels, at least users/customers could have a way to use this feature. Another question I'd like to ask is roughly when will 6.5 lowlatency-hwe kernel be landed in Jammy? Should I also submit this to 6.2 lowlatency kernel? Thanks, Gerald On Tue, Oct 31, 2023 at 8:36 PM Dimitri John Ledkov < dimitri.ledkov@canonical.com> wrote: > Hi, > > On Tue, 31 Oct 2023 at 10:54, Gerald Yang <gerald.yang@canonical.com> > wrote: > > > > Hi Andrea, > > > > Thanks for the feedback! please let me explain this in more detail > > > > I sent out this SRU because some of our customers need to use NO_HZ_FULL. > > In their scenario, the CPU time is pretty sensitive, and must avoid as > much noise as possible to keep > > most CPUs doing only one single task. > > I found some CPU steal time happen on their VM because RCU callback > softirqs are still triggered on CPUs > > with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find > NO_HZ_FULL is not built into generic kernel. > > > > I think the use case for NO_HZ_FULL should be different than generic > kernel, e.g. performance vs low latency > > > > On generic kernel, we also did some tests with NO_HZ_FULL built-in here: > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154 > > > > The results show performance degradation on AMD EPYC series machines only > > when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on > Intel and arm64 machines. > > > > Because one of the customer is still on Focal, if it's possible to merge > this into 5.15 kernel, it would be more convenient > > for them to use HWE lowlatency kernel. > > And if there are users need to use NO_HZ_FULL on Focal, lowlatency > kernel could be a proper choice for them, > > because so far there is no kernel with NO_HZ_FULL built-in on Focal. > > I am ok having this enabled in focal:lowlatency-hwe-5.15 and > jammy:lowlatency-hwe-6.5 > > but not change anything in focal:lowlatency (5.4 / ga) > jammy:lowlatency (5.15 / ga). > > Would that help you? As an exception to our policy. I really feel > uncomfortable changing default GA flavours, but risk levels of making > hwe kernels more like the future sounds safer. > > > > > Thanks, > > Gerald > > > > On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <andrea.righi@canonical.com> > wrote: > >> > >> On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: > >> > BugLink: https://bugs.launchpad.net/bugs/2023007 > >> > > >> > [Impact] > >> > > >> > The lowlatency kernel in Ubuntu is specifically designed to prioritize > >> > high responsiveness, making it ideal for multimedia environments like > >> > DAWs and audio processing platforms, as well as soft real-time > >> > environments. > >> > > >> > With the introduction of a real-time kernel, it might be worth > >> > reconsidering the role of the lowlatency kernel and potentially > >> > including it as the default kernel in desktop images, focusing on its > >> > suitability for desktop-oriented usage. > >> > > >> > To achieve this, we can enable additional configuration settings and > >> > make it more focused for a low-latency and highly responsive desktop > >> > environment. > >> > > >> > Optionally (for the future) provide also an additional user-space > >> > package that would enable specific run-time kernel settings focused at > >> > certain preset workload profiles (e.g, web navigation, gaming, audio > >> > processing, etc.). > >> > > >> > [Test case] > >> > > >> > Use linux-lowlatency in a desktop environment and measure > responsiveness > >> > of interactive applications. > >> > > >> > [Fix] > >> > > >> > Enable the following additional .config settings to make this kernel > >> > more suitable for a low-latency desktop kernel: > >> > > >> > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" > (shutdown > >> > clock tick when possible across all the enabled CPUs if they are > >> > either idle or running 1 task - reduce kernel jitter of running > tasks > >> > due to the periodic clock tick, must be enabled at boot time > passing > >> > `nohz_full=<cpu_list>`) > >> > > >> > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU > >> > callbacks from softirq context to kthread context (reduce time > spent > >> > in softirqs with preemption disabled to improve the overall system > >> > responsiveness, at the cost of introducing a potential performance > >> > penalty, because RCU callbacks are not processed by kernel threads) > >> > > >> > [Regression potential] > >> > > >> > Enabling all these settings can introduce a potential performance > >> > regression, but the kernel should result more responsive and make > >> > it more suitable for a desktop/multimedia/gaming/audio processing > >> > context. > >> > > >> > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > >> > >> It's worth mentioning that we already applied these configs to the > >> lowlatency kernels in mantic and in general I don't see any downside if > >> we apply this. > >> > >> However, I'm not sure if a change like this is proper SRU material, we > >> are basically saying "now lowlatency is a desktop kernel everywhere". > >> > >> Certain workloads may experience regressions (especially with the > >> RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be > >> enabled/disabled via boot options). > >> > >> We also need to keep in mind that Jammy will get these changes with the > >> lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport > >> this and apply it everywhere... opinions? > >> > >> -Andrea > >> > >> > --- > >> > debian.lowlatency/config/annotations | 13 +++++++++++++ > >> > 1 file changed, 13 insertions(+) > >> > > >> > diff --git a/debian.lowlatency/config/annotations > b/debian.lowlatency/config/annotations > >> > index 694ae26b6745..3e1556d60809 100644 > >> > --- a/debian.lowlatency/config/annotations > >> > +++ b/debian.lowlatency/config/annotations > >> > @@ -15,12 +15,20 @@ CONFIG_HZ_250 > note<'Override default HZ used i > >> > CONFIG_LATENCYTOP policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > CONFIG_LATENCYTOP note<' > https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, > LP#1655986'> > >> > > >> > +CONFIG_NO_HZ_FULL policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > +CONFIG_NO_HZ_FULL note<'Enable access > to "Full tickless mode" (LP: #2023007)'> > >> > + > >> > +CONFIG_NO_HZ_IDLE policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > +CONFIG_NO_HZ_IDLE note<'Disabled in > favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> > >> > + > >> > CONFIG_PREEMPT policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > CONFIG_PREEMPT note<'Enable fully > preemptible kernel'> > >> > > >> > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > CONFIG_PREEMPT_VOLUNTARY note<'Disable > voluntary preemption model'> > >> > > >> > +CONFIG_RCU_NOCB_CPU policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > +CONFIG_RCU_NOCB_CPU note<'Move RCU > callbacks from softirq context to kthread context (LP: #2023007)'> > >> > > >> > # ---- Annotations without notes ---- > >> > > >> > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN > policy<{'amd64': 'y', 'arm64': ' > >> > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': > 'n', 'arm64': 'm'}> > >> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': > 'n', 'arm64': 'm'}> > >> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > CONFIG_DEBUG_PREEMPT policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > CONFIG_HZ policy<{'amd64': > '1000', 'arm64': '1000'}> > >> > CONFIG_INLINE_READ_LOCK policy<{'arm64': > '-'}> > >> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU > policy<{'amd64': 'y', 'arm64': ' > >> > CONFIG_PREEMPT_TRACER policy<{'amd64': > 'n', 'arm64': 'n'}> > >> > CONFIG_TASKS_RCU policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > CONFIG_TEST_DIV64 policy<{'amd64': > 'm', 'arm64': 'm'}> > >> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': > '-', 'arm64': '-'}> > >> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': > 'y', 'arm64': 'y'}> > >> > -- > >> > 2.25.1 > >> > > >> > > >> > -- > >> > kernel-team mailing list > >> > kernel-team@lists.ubuntu.com > >> > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > -- > > kernel-team mailing list > > kernel-team@lists.ubuntu.com > > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > -- > okurrr, > > Dimitri >
On Tue, Oct 31, 2023 at 09:48:55PM +0800, Gerald Yang wrote: > Hi Dimitri, > > Thanks for checking this! > And yes, that would be very helpful to enable it in focal lowlatency-hwe > and jammy lowlatency-hwe kernels, > at least users/customers could have a way to use this feature. > > Another question I'd like to ask is roughly when will 6.5 lowlatency-hwe > kernel be landed in Jammy? Should I also submit this to 6.2 lowlatency > kernel? We are working on hwe-6.5/lowlatency-hwe-6.5 right now, we are in the process of fixing the usual dkms breakage, reviewing test results, etc... at this point I think we need to wait after the sprint to have the 6.5's out in jammy. -Andrea > > Thanks, > Gerald > > > On Tue, Oct 31, 2023 at 8:36 PM Dimitri John Ledkov < > dimitri.ledkov@canonical.com> wrote: > > > Hi, > > > > On Tue, 31 Oct 2023 at 10:54, Gerald Yang <gerald.yang@canonical.com> > > wrote: > > > > > > Hi Andrea, > > > > > > Thanks for the feedback! please let me explain this in more detail > > > > > > I sent out this SRU because some of our customers need to use NO_HZ_FULL. > > > In their scenario, the CPU time is pretty sensitive, and must avoid as > > much noise as possible to keep > > > most CPUs doing only one single task. > > > I found some CPU steal time happen on their VM because RCU callback > > softirqs are still triggered on CPUs > > > with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find > > NO_HZ_FULL is not built into generic kernel. > > > > > > I think the use case for NO_HZ_FULL should be different than generic > > kernel, e.g. performance vs low latency > > > > > > On generic kernel, we also did some tests with NO_HZ_FULL built-in here: > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154 > > > > > > The results show performance degradation on AMD EPYC series machines only > > > when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on > > Intel and arm64 machines. > > > > > > Because one of the customer is still on Focal, if it's possible to merge > > this into 5.15 kernel, it would be more convenient > > > for them to use HWE lowlatency kernel. > > > And if there are users need to use NO_HZ_FULL on Focal, lowlatency > > kernel could be a proper choice for them, > > > because so far there is no kernel with NO_HZ_FULL built-in on Focal. > > > > I am ok having this enabled in focal:lowlatency-hwe-5.15 and > > jammy:lowlatency-hwe-6.5 > > > > but not change anything in focal:lowlatency (5.4 / ga) > > jammy:lowlatency (5.15 / ga). > > > > Would that help you? As an exception to our policy. I really feel > > uncomfortable changing default GA flavours, but risk levels of making > > hwe kernels more like the future sounds safer. > > > > > > > > Thanks, > > > Gerald > > > > > > On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <andrea.righi@canonical.com> > > wrote: > > >> > > >> On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: > > >> > BugLink: https://bugs.launchpad.net/bugs/2023007 > > >> > > > >> > [Impact] > > >> > > > >> > The lowlatency kernel in Ubuntu is specifically designed to prioritize > > >> > high responsiveness, making it ideal for multimedia environments like > > >> > DAWs and audio processing platforms, as well as soft real-time > > >> > environments. > > >> > > > >> > With the introduction of a real-time kernel, it might be worth > > >> > reconsidering the role of the lowlatency kernel and potentially > > >> > including it as the default kernel in desktop images, focusing on its > > >> > suitability for desktop-oriented usage. > > >> > > > >> > To achieve this, we can enable additional configuration settings and > > >> > make it more focused for a low-latency and highly responsive desktop > > >> > environment. > > >> > > > >> > Optionally (for the future) provide also an additional user-space > > >> > package that would enable specific run-time kernel settings focused at > > >> > certain preset workload profiles (e.g, web navigation, gaming, audio > > >> > processing, etc.). > > >> > > > >> > [Test case] > > >> > > > >> > Use linux-lowlatency in a desktop environment and measure > > responsiveness > > >> > of interactive applications. > > >> > > > >> > [Fix] > > >> > > > >> > Enable the following additional .config settings to make this kernel > > >> > more suitable for a low-latency desktop kernel: > > >> > > > >> > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" > > (shutdown > > >> > clock tick when possible across all the enabled CPUs if they are > > >> > either idle or running 1 task - reduce kernel jitter of running > > tasks > > >> > due to the periodic clock tick, must be enabled at boot time > > passing > > >> > `nohz_full=<cpu_list>`) > > >> > > > >> > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU > > >> > callbacks from softirq context to kthread context (reduce time > > spent > > >> > in softirqs with preemption disabled to improve the overall system > > >> > responsiveness, at the cost of introducing a potential performance > > >> > penalty, because RCU callbacks are not processed by kernel threads) > > >> > > > >> > [Regression potential] > > >> > > > >> > Enabling all these settings can introduce a potential performance > > >> > regression, but the kernel should result more responsive and make > > >> > it more suitable for a desktop/multimedia/gaming/audio processing > > >> > context. > > >> > > > >> > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > > >> > > >> It's worth mentioning that we already applied these configs to the > > >> lowlatency kernels in mantic and in general I don't see any downside if > > >> we apply this. > > >> > > >> However, I'm not sure if a change like this is proper SRU material, we > > >> are basically saying "now lowlatency is a desktop kernel everywhere". > > >> > > >> Certain workloads may experience regressions (especially with the > > >> RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be > > >> enabled/disabled via boot options). > > >> > > >> We also need to keep in mind that Jammy will get these changes with the > > >> lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport > > >> this and apply it everywhere... opinions? > > >> > > >> -Andrea > > >> > > >> > --- > > >> > debian.lowlatency/config/annotations | 13 +++++++++++++ > > >> > 1 file changed, 13 insertions(+) > > >> > > > >> > diff --git a/debian.lowlatency/config/annotations > > b/debian.lowlatency/config/annotations > > >> > index 694ae26b6745..3e1556d60809 100644 > > >> > --- a/debian.lowlatency/config/annotations > > >> > +++ b/debian.lowlatency/config/annotations > > >> > @@ -15,12 +15,20 @@ CONFIG_HZ_250 > > note<'Override default HZ used i > > >> > CONFIG_LATENCYTOP policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > CONFIG_LATENCYTOP note<' > > https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, > > LP#1655986'> > > >> > > > >> > +CONFIG_NO_HZ_FULL policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > +CONFIG_NO_HZ_FULL note<'Enable access > > to "Full tickless mode" (LP: #2023007)'> > > >> > + > > >> > +CONFIG_NO_HZ_IDLE policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > +CONFIG_NO_HZ_IDLE note<'Disabled in > > favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> > > >> > + > > >> > CONFIG_PREEMPT policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > CONFIG_PREEMPT note<'Enable fully > > preemptible kernel'> > > >> > > > >> > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > CONFIG_PREEMPT_VOLUNTARY note<'Disable > > voluntary preemption model'> > > >> > > > >> > +CONFIG_RCU_NOCB_CPU policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > +CONFIG_RCU_NOCB_CPU note<'Move RCU > > callbacks from softirq context to kthread context (LP: #2023007)'> > > >> > > > >> > # ---- Annotations without notes ---- > > >> > > > >> > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN > > policy<{'amd64': 'y', 'arm64': ' > > >> > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': > > 'n', 'arm64': 'm'}> > > >> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': > > 'n', 'arm64': 'm'}> > > >> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > CONFIG_DEBUG_PREEMPT policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > CONFIG_HZ policy<{'amd64': > > '1000', 'arm64': '1000'}> > > >> > CONFIG_INLINE_READ_LOCK policy<{'arm64': > > '-'}> > > >> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU > > policy<{'amd64': 'y', 'arm64': ' > > >> > CONFIG_PREEMPT_TRACER policy<{'amd64': > > 'n', 'arm64': 'n'}> > > >> > CONFIG_TASKS_RCU policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > CONFIG_TEST_DIV64 policy<{'amd64': > > 'm', 'arm64': 'm'}> > > >> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': > > '-', 'arm64': '-'}> > > >> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': > > 'y', 'arm64': 'y'}> > > >> > -- > > >> > 2.25.1 > > >> > > > >> > > > >> > -- > > >> > kernel-team mailing list > > >> > kernel-team@lists.ubuntu.com > > >> > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > > > -- > > > kernel-team mailing list > > > kernel-team@lists.ubuntu.com > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > > > > > -- > > okurrr, > > > > Dimitri > >
Thanks Andrea for the information In this case, I'd like to confirm if this will be merged into 5.15 lowlatency-hwe kernel on Focal? not GA on Jammy So I can provide some feedbacks to the customer Thanks, Gerald On Tue, Oct 31, 2023 at 10:57 PM Andrea Righi <andrea.righi@canonical.com> wrote: > On Tue, Oct 31, 2023 at 09:48:55PM +0800, Gerald Yang wrote: > > Hi Dimitri, > > > > Thanks for checking this! > > And yes, that would be very helpful to enable it in focal lowlatency-hwe > > and jammy lowlatency-hwe kernels, > > at least users/customers could have a way to use this feature. > > > > Another question I'd like to ask is roughly when will 6.5 lowlatency-hwe > > kernel be landed in Jammy? Should I also submit this to 6.2 lowlatency > > kernel? > > We are working on hwe-6.5/lowlatency-hwe-6.5 right now, we are in the > process of fixing the usual dkms breakage, reviewing test results, > etc... at this point I think we need to wait after the sprint to have > the 6.5's out in jammy. > > -Andrea > > > > > Thanks, > > Gerald > > > > > > On Tue, Oct 31, 2023 at 8:36 PM Dimitri John Ledkov < > > dimitri.ledkov@canonical.com> wrote: > > > > > Hi, > > > > > > On Tue, 31 Oct 2023 at 10:54, Gerald Yang <gerald.yang@canonical.com> > > > wrote: > > > > > > > > Hi Andrea, > > > > > > > > Thanks for the feedback! please let me explain this in more detail > > > > > > > > I sent out this SRU because some of our customers need to use > NO_HZ_FULL. > > > > In their scenario, the CPU time is pretty sensitive, and must avoid > as > > > much noise as possible to keep > > > > most CPUs doing only one single task. > > > > I found some CPU steal time happen on their VM because RCU callback > > > softirqs are still triggered on CPUs > > > > with isolcpus, nohz_full and rcu_nocbs configured, and lead me to > find > > > NO_HZ_FULL is not built into generic kernel. > > > > > > > > I think the use case for NO_HZ_FULL should be different than generic > > > kernel, e.g. performance vs low latency > > > > > > > > On generic kernel, we also did some tests with NO_HZ_FULL built-in > here: > > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154 > > > > > > > > The results show performance degradation on AMD EPYC series machines > only > > > > when NO_HZ_FULL is built-in but not enabled, but this doesn't happen > on > > > Intel and arm64 machines. > > > > > > > > Because one of the customer is still on Focal, if it's possible to > merge > > > this into 5.15 kernel, it would be more convenient > > > > for them to use HWE lowlatency kernel. > > > > And if there are users need to use NO_HZ_FULL on Focal, lowlatency > > > kernel could be a proper choice for them, > > > > because so far there is no kernel with NO_HZ_FULL built-in on Focal. > > > > > > I am ok having this enabled in focal:lowlatency-hwe-5.15 and > > > jammy:lowlatency-hwe-6.5 > > > > > > but not change anything in focal:lowlatency (5.4 / ga) > > > jammy:lowlatency (5.15 / ga). > > > > > > Would that help you? As an exception to our policy. I really feel > > > uncomfortable changing default GA flavours, but risk levels of making > > > hwe kernels more like the future sounds safer. > > > > > > > > > > > Thanks, > > > > Gerald > > > > > > > > On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi < > andrea.righi@canonical.com> > > > wrote: > > > >> > > > >> On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: > > > >> > BugLink: https://bugs.launchpad.net/bugs/2023007 > > > >> > > > > >> > [Impact] > > > >> > > > > >> > The lowlatency kernel in Ubuntu is specifically designed to > prioritize > > > >> > high responsiveness, making it ideal for multimedia environments > like > > > >> > DAWs and audio processing platforms, as well as soft real-time > > > >> > environments. > > > >> > > > > >> > With the introduction of a real-time kernel, it might be worth > > > >> > reconsidering the role of the lowlatency kernel and potentially > > > >> > including it as the default kernel in desktop images, focusing on > its > > > >> > suitability for desktop-oriented usage. > > > >> > > > > >> > To achieve this, we can enable additional configuration settings > and > > > >> > make it more focused for a low-latency and highly responsive > desktop > > > >> > environment. > > > >> > > > > >> > Optionally (for the future) provide also an additional user-space > > > >> > package that would enable specific run-time kernel settings > focused at > > > >> > certain preset workload profiles (e.g, web navigation, gaming, > audio > > > >> > processing, etc.). > > > >> > > > > >> > [Test case] > > > >> > > > > >> > Use linux-lowlatency in a desktop environment and measure > > > responsiveness > > > >> > of interactive applications. > > > >> > > > > >> > [Fix] > > > >> > > > > >> > Enable the following additional .config settings to make this > kernel > > > >> > more suitable for a low-latency desktop kernel: > > > >> > > > > >> > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" > > > (shutdown > > > >> > clock tick when possible across all the enabled CPUs if they > are > > > >> > either idle or running 1 task - reduce kernel jitter of running > > > tasks > > > >> > due to the periodic clock tick, must be enabled at boot time > > > passing > > > >> > `nohz_full=<cpu_list>`) > > > >> > > > > >> > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move > RCU > > > >> > callbacks from softirq context to kthread context (reduce time > > > spent > > > >> > in softirqs with preemption disabled to improve the overall > system > > > >> > responsiveness, at the cost of introducing a potential > performance > > > >> > penalty, because RCU callbacks are not processed by kernel > threads) > > > >> > > > > >> > [Regression potential] > > > >> > > > > >> > Enabling all these settings can introduce a potential > performance > > > >> > regression, but the kernel should result more responsive and > make > > > >> > it more suitable for a desktop/multimedia/gaming/audio > processing > > > >> > context. > > > >> > > > > >> > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > > > >> > > > >> It's worth mentioning that we already applied these configs to the > > > >> lowlatency kernels in mantic and in general I don't see any > downside if > > > >> we apply this. > > > >> > > > >> However, I'm not sure if a change like this is proper SRU material, > we > > > >> are basically saying "now lowlatency is a desktop kernel > everywhere". > > > >> > > > >> Certain workloads may experience regressions (especially with the > > > >> RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be > > > >> enabled/disabled via boot options). > > > >> > > > >> We also need to keep in mind that Jammy will get these changes with > the > > > >> lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to > backport > > > >> this and apply it everywhere... opinions? > > > >> > > > >> -Andrea > > > >> > > > >> > --- > > > >> > debian.lowlatency/config/annotations | 13 +++++++++++++ > > > >> > 1 file changed, 13 insertions(+) > > > >> > > > > >> > diff --git a/debian.lowlatency/config/annotations > > > b/debian.lowlatency/config/annotations > > > >> > index 694ae26b6745..3e1556d60809 100644 > > > >> > --- a/debian.lowlatency/config/annotations > > > >> > +++ b/debian.lowlatency/config/annotations > > > >> > @@ -15,12 +15,20 @@ CONFIG_HZ_250 > > > note<'Override default HZ used i > > > >> > CONFIG_LATENCYTOP policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > CONFIG_LATENCYTOP note<' > > > https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, > > > LP#1655986'> > > > >> > > > > >> > +CONFIG_NO_HZ_FULL policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > +CONFIG_NO_HZ_FULL note<'Enable > access > > > to "Full tickless mode" (LP: #2023007)'> > > > >> > + > > > >> > +CONFIG_NO_HZ_IDLE policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > +CONFIG_NO_HZ_IDLE note<'Disabled in > > > favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> > > > >> > + > > > >> > CONFIG_PREEMPT policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > CONFIG_PREEMPT note<'Enable > fully > > > preemptible kernel'> > > > >> > > > > >> > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > CONFIG_PREEMPT_VOLUNTARY note<'Disable > > > voluntary preemption model'> > > > >> > > > > >> > +CONFIG_RCU_NOCB_CPU policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > +CONFIG_RCU_NOCB_CPU note<'Move RCU > > > callbacks from softirq context to kthread context (LP: #2023007)'> > > > >> > > > > >> > # ---- Annotations without notes ---- > > > >> > > > > >> > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN > > > policy<{'amd64': 'y', 'arm64': ' > > > >> > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': > > > 'n', 'arm64': 'm'}> > > > >> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': > > > 'n', 'arm64': 'm'}> > > > >> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > CONFIG_DEBUG_PREEMPT policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > CONFIG_HZ policy<{'amd64': > > > '1000', 'arm64': '1000'}> > > > >> > CONFIG_INLINE_READ_LOCK policy<{'arm64': > > > '-'}> > > > >> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU > > > policy<{'amd64': 'y', 'arm64': ' > > > >> > CONFIG_PREEMPT_TRACER policy<{'amd64': > > > 'n', 'arm64': 'n'}> > > > >> > CONFIG_TASKS_RCU policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > CONFIG_TEST_DIV64 policy<{'amd64': > > > 'm', 'arm64': 'm'}> > > > >> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': > > > '-', 'arm64': '-'}> > > > >> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': > > > 'y', 'arm64': 'y'}> > > > >> > -- > > > >> > 2.25.1 > > > >> > > > > >> > > > > >> > -- > > > >> > kernel-team mailing list > > > >> > kernel-team@lists.ubuntu.com > > > >> > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > > > > > -- > > > > kernel-team mailing list > > > > kernel-team@lists.ubuntu.com > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > > > > > > > > > -- > > > okurrr, > > > > > > Dimitri > > > >
On Mon, 30 Oct 2023 at 19:09, Gerald Yang <gerald.yang@canonical.com> wrote: > > BugLink: https://bugs.launchpad.net/bugs/2023007 > > [Impact] > > The lowlatency kernel in Ubuntu is specifically designed to prioritize > high responsiveness, making it ideal for multimedia environments like > DAWs and audio processing platforms, as well as soft real-time > environments. > > With the introduction of a real-time kernel, it might be worth > reconsidering the role of the lowlatency kernel and potentially > including it as the default kernel in desktop images, focusing on its > suitability for desktop-oriented usage. > > To achieve this, we can enable additional configuration settings and > make it more focused for a low-latency and highly responsive desktop > environment. > > Optionally (for the future) provide also an additional user-space > package that would enable specific run-time kernel settings focused at > certain preset workload profiles (e.g, web navigation, gaming, audio > processing, etc.). > > [Test case] > > Use linux-lowlatency in a desktop environment and measure responsiveness > of interactive applications. > > [Fix] > > Enable the following additional .config settings to make this kernel > more suitable for a low-latency desktop kernel: After careful assessment, this is not suitable for SRU and will regress the available portfolio of kernels we make available on Focal. As it is a feature, not a bug. Whilst all of the above is true, and this will ship in the GA in noble and in jammy-lowlatency-hwe, it is not an appropriate change for jammy-ga, nor for focal backports. This is primarily due to lack of an Ubuntu Realtime Kernel offering on the focal series. My recommendation is to upgrade to jammy, and opt into either lowlatency-hwe or realtime kernel in jammy as appropriate. Or wait a little bit and upgrade to noble. The jammy GA and focal HWE are otherwise frozen for changes like this, at the very least since summer 2022. > > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown > clock tick when possible across all the enabled CPUs if they are > either idle or running 1 task - reduce kernel jitter of running tasks > due to the periodic clock tick, must be enabled at boot time passing > `nohz_full=<cpu_list>`) > > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU > callbacks from softirq context to kthread context (reduce time spent > in softirqs with preemption disabled to improve the overall system > responsiveness, at the cost of introducing a potential performance > penalty, because RCU callbacks are not processed by kernel threads) > > [Regression potential] > > Enabling all these settings can introduce a potential performance > regression, but the kernel should result more responsive and make > it more suitable for a desktop/multimedia/gaming/audio processing > context. > > Signed-off-by: Gerald Yang <gerald.yang@canonical.com> > --- > debian.lowlatency/config/annotations | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations > index 694ae26b6745..3e1556d60809 100644 > --- a/debian.lowlatency/config/annotations > +++ b/debian.lowlatency/config/annotations > @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i > CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_LATENCYTOP note<'https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, LP#1655986'> > > +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'> > + > +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}> > +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> > + > CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_PREEMPT note<'Enable fully preemptible kernel'> > > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'> > > +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'> > > # ---- Annotations without notes ---- > > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': ' > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}> > CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': ' > CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}> > CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}> > CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}> > -- > 2.25.1 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations index 694ae26b6745..3e1556d60809 100644 --- a/debian.lowlatency/config/annotations +++ b/debian.lowlatency/config/annotations @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}> CONFIG_LATENCYTOP note<'https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, LP#1655986'> +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'> + +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> + CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}> CONFIG_PREEMPT note<'Enable fully preemptible kernel'> CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'> +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'> # ---- Annotations without notes ---- @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': ' CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}> CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}> +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}> CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': ' CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}> CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}> +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}> CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}>
BugLink: https://bugs.launchpad.net/bugs/2023007 [Impact] The lowlatency kernel in Ubuntu is specifically designed to prioritize high responsiveness, making it ideal for multimedia environments like DAWs and audio processing platforms, as well as soft real-time environments. With the introduction of a real-time kernel, it might be worth reconsidering the role of the lowlatency kernel and potentially including it as the default kernel in desktop images, focusing on its suitability for desktop-oriented usage. To achieve this, we can enable additional configuration settings and make it more focused for a low-latency and highly responsive desktop environment. Optionally (for the future) provide also an additional user-space package that would enable specific run-time kernel settings focused at certain preset workload profiles (e.g, web navigation, gaming, audio processing, etc.). [Test case] Use linux-lowlatency in a desktop environment and measure responsiveness of interactive applications. [Fix] Enable the following additional .config settings to make this kernel more suitable for a low-latency desktop kernel: - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`) - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads) [Regression potential] Enabling all these settings can introduce a potential performance regression, but the kernel should result more responsive and make it more suitable for a desktop/multimedia/gaming/audio processing context. Signed-off-by: Gerald Yang <gerald.yang@canonical.com> --- debian.lowlatency/config/annotations | 13 +++++++++++++ 1 file changed, 13 insertions(+)