| Message ID | 1503899802-13248-1-git-send-email-mpe@ellerman.id.au (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig | expand |
On 08/28/2017 02:56 AM, Michael Ellerman wrote: > Some Power9 boxes will have this adapter installed, so add it to the > defconfig so we can boot on those machines without an initrd. Michael, not sure if this affects Petitboot (I know it has its own default config files), but in the past we had some issues with this driver and it's nice to have the possibility to blacklist it. What is the rationale for this change? I mean, how being able to boot without initrd is interesting here? Sorry if it's silly question! Thanks, Guilherme P.S. If this config was *unset* and you're setting, I'd be glad to see it as module! > > Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> > --- > arch/powerpc/configs/powernv_defconfig | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig > index f89d9b1799b6..dde87c623b74 100644 > --- a/arch/powerpc/configs/powernv_defconfig > +++ b/arch/powerpc/configs/powernv_defconfig > @@ -113,6 +113,7 @@ CONFIG_SCSI_CXGB3_ISCSI=m > CONFIG_SCSI_CXGB4_ISCSI=m > CONFIG_SCSI_BNX2_ISCSI=m > CONFIG_BE2ISCSI=m > +CONFIG_SCSI_AACRAID=y > CONFIG_SCSI_MPT2SAS=m > CONFIG_SCSI_SYM53C8XX_2=m > CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0 >
"Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes: > On 08/28/2017 02:56 AM, Michael Ellerman wrote: >> Some Power9 boxes will have this adapter installed, so add it to the >> defconfig so we can boot on those machines without an initrd. > > Michael, not sure if this affects Petitboot (I know it has its own > default config files), but in the past we had some issues with this > driver and it's nice to have the possibility to blacklist it. Why we were black listing it? > What is the rationale for this change? I mean, how being able to boot > without initrd is interesting here? Sorry if it's silly question! It is a silly question! :) When you boot as many kernels as me it's preferable to be able to build a kernel completely remote from the box, throw the kernel on a tftp server and have the machine boot from that kernel all the way to userspace. That requires not using an initrd, and therefore having all the drivers built-in that are needed to get to the boot disk. (Yeah it is "possible" to build an initrd remotely but it's a pain). And the one P9 box I have here uses that driver for its root disk, so I do really want it to be =y. cheers
On 08/29/2017 08:22 AM, Michael Ellerman wrote: > "Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes: > >> On 08/28/2017 02:56 AM, Michael Ellerman wrote: >>> Some Power9 boxes will have this adapter installed, so add it to the >>> defconfig so we can boot on those machines without an initrd. >> >> Michael, not sure if this affects Petitboot (I know it has its own >> default config files), but in the past we had some issues with this >> driver and it's nice to have the possibility to blacklist it. > > Why we were black listing it? Basically for the same reason you want it built-in - debug purposes heheh In fact, this driver takes some time in its initialization process, so if my rootfs is not under it, to speedup my boot-time I could blacklist it; also, to debug the aacraid driver we could blacklist it in Petitboot and debug on distro (easier to build kernels there), since the issue could be in first probe only (or be affected by kexec-out from Petitboot). Anyway, since make it a module will harm your workflow, I agree built-in is the best option! Thanks for your detailed explanation. Cheers, Guilherme > >> What is the rationale for this change? I mean, how being able to boot >> without initrd is interesting here? Sorry if it's silly question! > > It is a silly question! :) > > When you boot as many kernels as me it's preferable to be able to build > a kernel completely remote from the box, throw the kernel on a tftp > server and have the machine boot from that kernel all the way to > userspace. > > That requires not using an initrd, and therefore having all the drivers > built-in that are needed to get to the boot disk. > > (Yeah it is "possible" to build an initrd remotely but it's a pain). > > And the one P9 box I have here uses that driver for its root disk, so I > do really want it to be =y. > > cheers >
"Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes: > On 08/29/2017 08:22 AM, Michael Ellerman wrote: >> "Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes: >> >>> On 08/28/2017 02:56 AM, Michael Ellerman wrote: >>>> Some Power9 boxes will have this adapter installed, so add it to the >>>> defconfig so we can boot on those machines without an initrd. >>> >>> Michael, not sure if this affects Petitboot (I know it has its own >>> default config files), but in the past we had some issues with this >>> driver and it's nice to have the possibility to blacklist it. >> >> Why we were black listing it? > > Basically for the same reason you want it built-in - debug purposes heheh > > In fact, this driver takes some time in its initialization process, so > if my rootfs is not under it, to speedup my boot-time I could blacklist > it; also, to debug the aacraid driver we could blacklist it in Petitboot > and debug on distro (easier to build kernels there), since the issue > could be in first probe only (or be affected by kexec-out from Petitboot). > > Anyway, since make it a module will harm your workflow, I agree built-in > is the best option! OK. So maybe when the petitboot kernel moves up to 4.14 they may want to flip it back to being a module. I think the majority of machines that have one of these adapters will be using it for their root disk, so I think the slow initialisation is just something we have to suffer. cheers
On 08/30/2017 08:07 AM, Michael Ellerman wrote: >[...] > OK. So maybe when the petitboot kernel moves up to 4.14 they may want to > flip it back to being a module. > OK, I'll be tuned! Thanks > I think the majority of machines that have one of these adapters will be > using it for their root disk, so I think the slow initialisation is just > something we have to suffer. > > cheers >
Michael Ellerman <mpe@ellerman.id.au> writes: > OK. So maybe when the petitboot kernel moves up to 4.14 they may want to > flip it back to being a module. Yeah. We tend to keep all RAID adapters as modules because they almost all uniquely try to undo all boot time optimization done by anybody, anywhere, ever. > I think the majority of machines that have one of these adapters will be > using it for their root disk, so I think the slow initialisation is just > something we have to suffer. It does seem that way currently.
Stewart Smith <stewart@linux.vnet.ibm.com> writes: > Michael Ellerman <mpe@ellerman.id.au> writes: >> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to >> flip it back to being a module. > > Yeah. We tend to keep all RAID adapters as modules because they almost > all uniquely try to undo all boot time optimization done by anybody, > anywhere, ever. Yes I know what you mean. I am thinking of one In ParticulaR. cheers
On 08/31/2017 07:37 AM, Michael Ellerman wrote: > Stewart Smith <stewart@linux.vnet.ibm.com> writes: > >> Michael Ellerman <mpe@ellerman.id.au> writes: >>> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to >>> flip it back to being a module. >> >> Yeah. We tend to keep all RAID adapters as modules because they almost >> all uniquely try to undo all boot time optimization done by anybody, >> anywhere, ever. > > Yes I know what you mean. I am thinking of one In ParticulaR. Ouch. In all fairness over the life of ipr I've done a couple things to improve initialization time. 1. On a PowerVM machine, the ipr boot device should not get reset between the FCODE driver and the Linux driver, so init is extremely fast. If its a non-boot device, then you'll have to wait for the firmware to initialize on those adapters. 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot kernel, then we should also avoid resetting the ipr adapter, so ipr init on the kernel being kexec booted from skiboot should be extremely fast. As to improving ipr init time in the skiboot kernel, that becomes more difficult to do safely. The firmware stack still needs to boot, do device discovery, potentially spin up disks, etc, which all takes time. Additionally, we put in changes in the firmware and driver to address complaints largely from Anton about it taking too long for ipr to timeout on bad adapters. If you've got cases where ipr init is taking a long time, I'd be interested to know what scenarios are the most annoying to see if there is any opportunity to improve. -Brian
Brian King <brking@linux.vnet.ibm.com> writes: > On 08/31/2017 07:37 AM, Michael Ellerman wrote: >> Stewart Smith <stewart@linux.vnet.ibm.com> writes: >> >>> Michael Ellerman <mpe@ellerman.id.au> writes: >>>> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to >>>> flip it back to being a module. >>> >>> Yeah. We tend to keep all RAID adapters as modules because they almost >>> all uniquely try to undo all boot time optimization done by anybody, >>> anywhere, ever. >> >> Yes I know what you mean. I am thinking of one In ParticulaR. > > Ouch. In all fairness over the life of ipr I've done a couple things to > improve initialization time. Sorry that wasn't meant as a dig at you, more the hardware. It might have needed a smiley :) > 1. On a PowerVM machine, the ipr boot device should not get reset between > the FCODE driver and the Linux driver, so init is extremely fast. If its > a non-boot device, then you'll have to wait for the firmware to initialize > on those adapters. Yeah I don't see any problem on my LPARs, which just have a single IPR for the boot device. Init takes ~1 second. > 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot > kernel, then we should also avoid resetting the ipr adapter, so ipr > init on the kernel being kexec booted from skiboot should be extremely fast. OK, I didn't know that was an option, so that might help. > ... > If you've got cases where ipr init is taking a long time, I'd be > interested to know what scenarios are the most annoying to see if there > is any opportunity to improve. Yeah booting bare metal is where I see it (not using ipr.fast_reboot). eg booting a Tuleta: [ 1.517752] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017) ... [ 1.519884] ipr 0001:08:00.0: Initializing IOA. [ 1.519941] scsi host0: IBM 0 Storage Adapter [ 25.695744] ipr 0001:08:00.0: Starting IOA initialization sequence. ... [ 26.329614] sd 0:2:0:0: [sda] Spinning up disk... [ 26.329752] sd 0:2:1:0: [sdb] Spinning up disk... [ 26.329990] sd 0:2:2:0: [sdc] Spinning up disk... [ 26.330065] sd 0:2:3:0: [sdd] Spinning up disk... ... [ 26.349962] sr 0:0:6:0: Attached scsi generic sg15 type 5 Those 24 seconds are a bit of a shame. I assume we're just waiting for the card firmware to come up? I don't think there's much we can do in the kernel, PCI has just come up, so we can't start the IPR init earlier. And the rest of the boot only takes ~3 seconds, so even if we did everything else in parallel, we'd still be waiting ~21 seconds for IPR. cheers
Michael Ellerman <mpe@ellerman.id.au> writes: >> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot >> kernel, then we should also avoid resetting the ipr adapter, so ipr >> init on the kernel being kexec booted from skiboot should be extremely fast. > > OK, I didn't know that was an option, so that might help. > >> ... >> If you've got cases where ipr init is taking a long time, I'd be >> interested to know what scenarios are the most annoying to see if there >> is any opportunity to improve. > > Yeah booting bare metal is where I see it (not using ipr.fast_reboot). Hrm... We should probably enable that by default for petitboot then. It'd at least cut some time off booting straight through to OS.
On 09/03/2017 06:19 PM, Stewart Smith wrote: > Michael Ellerman <mpe@ellerman.id.au> writes: >>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot >>> kernel, then we should also avoid resetting the ipr adapter, so ipr >>> init on the kernel being kexec booted from skiboot should be extremely fast. >> >> OK, I didn't know that was an option, so that might help. >> >>> ... >>> If you've got cases where ipr init is taking a long time, I'd be >>> interested to know what scenarios are the most annoying to see if there >>> is any opportunity to improve. >> >> Yeah booting bare metal is where I see it (not using ipr.fast_reboot). > > Hrm... We should probably enable that by default for petitboot then. > > It'd at least cut some time off booting straight through to OS. Agreed. I'd be interested to hear if that helps address the issue Michael is seeing. You can easily test this by exiting to a petitboot shell: echo 1 > /sys/module/ipr/parameters/fast_reboot Then go back to petitboot and boot the OS. Thanks, Brian
Brian King <brking@linux.vnet.ibm.com> writes: > On 09/03/2017 06:19 PM, Stewart Smith wrote: >> Michael Ellerman <mpe@ellerman.id.au> writes: >>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot >>>> kernel, then we should also avoid resetting the ipr adapter, so ipr >>>> init on the kernel being kexec booted from skiboot should be extremely fast. >>> >>> OK, I didn't know that was an option, so that might help. >>> >>>> ... >>>> If you've got cases where ipr init is taking a long time, I'd be >>>> interested to know what scenarios are the most annoying to see if there >>>> is any opportunity to improve. >>> >>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot). >> >> Hrm... We should probably enable that by default for petitboot then. >> >> It'd at least cut some time off booting straight through to OS. > > Agreed. I'd be interested to hear if that helps address the issue > Michael is seeing. > > You can easily test this by exiting to a petitboot shell: > > echo 1 > /sys/module/ipr/parameters/fast_reboot > > Then go back to petitboot and boot the OS. Just following up on this (!). This does work, and I've now been running it in my CI for about a month (~1000 boots) with no problems. You can also make it persistent by doing: $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1" cheers
Michael Ellerman <mpe@ellerman.id.au> writes: > Brian King <brking@linux.vnet.ibm.com> writes: >> On 09/03/2017 06:19 PM, Stewart Smith wrote: >>> Michael Ellerman <mpe@ellerman.id.au> writes: >>>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot >>>>> kernel, then we should also avoid resetting the ipr adapter, so ipr >>>>> init on the kernel being kexec booted from skiboot should be extremely fast. >>>> >>>> OK, I didn't know that was an option, so that might help. >>>> >>>>> ... >>>>> If you've got cases where ipr init is taking a long time, I'd be >>>>> interested to know what scenarios are the most annoying to see if there >>>>> is any opportunity to improve. >>>> >>>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot). >>> >>> Hrm... We should probably enable that by default for petitboot then. >>> >>> It'd at least cut some time off booting straight through to OS. >> >> Agreed. I'd be interested to hear if that helps address the issue >> Michael is seeing. >> >> You can easily test this by exiting to a petitboot shell: >> >> echo 1 > /sys/module/ipr/parameters/fast_reboot >> >> Then go back to petitboot and boot the OS. > > Just following up on this (!). > > This does work, and I've now been running it in my CI for about a month > (~1000 boots) with no problems. > > You can also make it persistent by doing: > > $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1" Okay, cool. https://github.com/open-power/op-build/pull/1900 will set it in firmware - we may as well run with this and fix any bugs we find. Any reason why it isn't the default behaviour?
On 02/22/2018 06:16 PM, Stewart Smith wrote: > Michael Ellerman <mpe@ellerman.id.au> writes: >> Brian King <brking@linux.vnet.ibm.com> writes: >>> On 09/03/2017 06:19 PM, Stewart Smith wrote: >>>> Michael Ellerman <mpe@ellerman.id.au> writes: >>>>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot >>>>>> kernel, then we should also avoid resetting the ipr adapter, so ipr >>>>>> init on the kernel being kexec booted from skiboot should be extremely fast. >>>>> >>>>> OK, I didn't know that was an option, so that might help. >>>>> >>>>>> ... >>>>>> If you've got cases where ipr init is taking a long time, I'd be >>>>>> interested to know what scenarios are the most annoying to see if there >>>>>> is any opportunity to improve. >>>>> >>>>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot). >>>> >>>> Hrm... We should probably enable that by default for petitboot then. >>>> >>>> It'd at least cut some time off booting straight through to OS. >>> >>> Agreed. I'd be interested to hear if that helps address the issue >>> Michael is seeing. >>> >>> You can easily test this by exiting to a petitboot shell: >>> >>> echo 1 > /sys/module/ipr/parameters/fast_reboot >>> >>> Then go back to petitboot and boot the OS. >> >> Just following up on this (!). >> >> This does work, and I've now been running it in my CI for about a month >> (~1000 boots) with no problems. >> >> You can also make it persistent by doing: >> >> $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1" > > Okay, cool. https://github.com/open-power/op-build/pull/1900 will set it > in firmware - we may as well run with this and fix any bugs we find. > > Any reason why it isn't the default behaviour? The primary reason this isn't the default setting is because it would cause undesired behavior on system reboots. On a kexec reboot, since the PCIe slots don't get power cycled or even hit with PERST, we can simply quiesce devices and pick them up at the other side of the kexec. On a real reboot where the firmware or hardware may end up doing a PERST or power cycle, we need to tell the ipr adapters that is coming and let them shutdown gracefully. If we don't, we will likely be OK in the single adapter configuration in most scenarios, but for a dual ipr adapter configuration, we can then end up with undesired adapter failovers occurring due to the unexpected power offs / resets, which can then end up extending the next boot. -Brian
diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig index f89d9b1799b6..dde87c623b74 100644 --- a/arch/powerpc/configs/powernv_defconfig +++ b/arch/powerpc/configs/powernv_defconfig @@ -113,6 +113,7 @@ CONFIG_SCSI_CXGB3_ISCSI=m CONFIG_SCSI_CXGB4_ISCSI=m CONFIG_SCSI_BNX2_ISCSI=m CONFIG_BE2ISCSI=m +CONFIG_SCSI_AACRAID=y CONFIG_SCSI_MPT2SAS=m CONFIG_SCSI_SYM53C8XX_2=m CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0
Some Power9 boxes will have this adapter installed, so add it to the defconfig so we can boot on those machines without an initrd. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> --- arch/powerpc/configs/powernv_defconfig | 1 + 1 file changed, 1 insertion(+)