| Message ID | 20180714164906.GQ23412@localhost.localdomain (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | Improvements for the PS3 | expand |
Hi Fredrik, On 07/14/2018 09:49 AM, Fredrik Noring wrote: > so I added a sleep with > > + msleep(10000); > + > return 0; > > et voilà, the screen came alive and the kernel panic was revealed! It seems > the kernel panics so fast that the PS3 frame buffer is unprepared. This is, > of course, very unfortunate because trying to debug the boot process without > a screen or any other means of obtaining console text is quite difficult. We could add a fixed delay there, but I'd like to avoid waiting that long on every boot. Why don't you add a kernel module_param named something like ps3fb_delay that takes a value in milliseconds and a default of zero. See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/video/fbdev/ps3fb.c?h=v4.17#n260 > I suppose the problem is that it relies on interrupts for ps3fb_sync_image > to regularly copy the image, hence without them the screen isn't updated to > show kernel panics, etc. Perhaps one way to fix that is to implement the > struct fb_tile_ops API, so that the console is synchronously updated? Would > that be acceptable? I'm not sure if that would work or not. Maybe Geert is more familiar with it. -Geoff
Hi Geoff, Frederik, On Thu, Jul 19, 2018 at 12:40 AM Geoff Levand <geoff@infradead.org> wrote: > On 07/14/2018 09:49 AM, Fredrik Noring wrote: > > so I added a sleep with > > > > + msleep(10000); I can't see where you added the sleep, but 10s seems excessive. If the real reason is the need to wait for an interrupt for ps3fb_sync_image(), then waiting for 40 ms should be sufficient? Or am I missing something? > > + > > return 0; > > > > et voilà, the screen came alive and the kernel panic was revealed! It seems > > the kernel panics so fast that the PS3 frame buffer is unprepared. This is, > > of course, very unfortunate because trying to debug the boot process without > > a screen or any other means of obtaining console text is quite difficult. > > We could add a fixed delay there, but I'd like to avoid waiting that > long on every boot. Why don't you add a kernel module_param named > something like ps3fb_delay that takes a value in milliseconds and a > default of zero. See: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/video/fbdev/ps3fb.c?h=v4.17#n260 > > > I suppose the problem is that it relies on interrupts for ps3fb_sync_image > > to regularly copy the image, hence without them the screen isn't updated to > > show kernel panics, etc. Perhaps one way to fix that is to implement the > > struct fb_tile_ops API, so that the console is synchronously updated? Would > > that be acceptable? > > I'm not sure if that would work or not. Maybe Geert is more familiar with it. That sounds like a complex solution, slowing down the console a lot. What about letting ps3fb register a panic notifier to sync the screen, like hyperv_fb does? Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Hi Geert, Fredrik, On 07/19/2018 12:45 AM, Geert Uytterhoeven wrote: >> On 07/14/2018 09:49 AM, Fredrik Noring wrote: >>> >>> et voilà, the screen came alive and the kernel panic was revealed! It seems >>> the kernel panics so fast that the PS3 frame buffer is unprepared. This is, >>> of course, very unfortunate because trying to debug the boot process without >>> a screen or any other means of obtaining console text is quite difficult. >> > What about letting ps3fb register a panic notifier to sync the screen, like > hyperv_fb does? Seems like the thing to do. Fredrik, do you want to try it? Otherwise, I'll work on it when I have some time. -Geoff
Hi Geert, Geoff, > > > so I added a sleep with > > > > > > + msleep(10000); > > I can't see where you added the sleep, but 10s seems excessive. > If the real reason is the need to wait for an interrupt for ps3fb_sync_image(), > then waiting for 40 ms should be sufficient? Or am I missing something? It's at the end of ps3fb_probe, as shown in the original post: https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-July/175771.html I thought 100 ms or so would work, but evidently it didn't. In fact, 1 s for even 5 s didn't seem to work either. In any case, I would like to develop a solution that does not need to sleep at all, so that will be my first approach for a proper implementation. > > > I suppose the problem is that it relies on interrupts for ps3fb_sync_image > > > to regularly copy the image, hence without them the screen isn't updated to > > > show kernel panics, etc. Perhaps one way to fix that is to implement the > > > struct fb_tile_ops API, so that the console is synchronously updated? Would > > > that be acceptable? > > > > I'm not sure if that would work or not. Maybe Geert is more familiar with it. > > That sounds like a complex solution, slowing down the console a lot. Why would that be slow? I have implemented a similar technique for the PlayStation 2 frame buffer, and (without any measurements at hand now) it appears to be about as fast as is possible, and reasonably easy too. :) It works like this: The PS2 frame buffer can operate in two distinct modes: virtual mode or console mode. Virtual mode is very similar to the PS3 in that the whole visible kernel memory frame buffer is copied to the Graphics Synthesizer (GS) via DMA regularly at vertical blank intervals. Console mode is different. No kernel memory is allocated for the frame buffer at all (which is a significant improvement in itself given that the PS2 is limited to 32 MiB of main memory) and mmap is disabled. Some struct fb_ops such as fb_fillrect and fb_copyarea are directly accelerated by GS hardware. The GS has two CRT merge circuits made for picture-in-picture that are used to accelerate YWRAP in hardware, which is particularly effective for console text scrolling. Additionally, the tiled API is implemented, and it turned out to be a rather good fit. The GS has 4 MiB of local frame buffer video memory (not directly accessible by the kernel). The maximum practical resolution 1920x1080p at 16 bits per pixel is 4147200 bytes, which leaves 47104 bytes for a tiled font which at 8x8 pixels and a minimum 4 bits indexed texture palette is at most 1464 characters. The indexed palette makes it easy to switch colors. struct fb_tile_ops such as fb_tileblit are accelerated by GS texture sprites which are fast (GS local copy) for the kernel via simple DMA (GIF) GS commands. Console text is always synchronous and therefore works properly in interrupt contexts and with kernel panics, etc. which is essential for debuggability. A buffered version could be faster, possibly, but I think that might as well be implemented by a user space console driver using a /dev/gs interface that can do zero-copy GS commands. The PS2 frame buffer implementation is nearly complete: https://github.com/frno7/linux/blob/ps2-v4.17-n3/drivers/video/fbdev/ps2fb.c Some adjustments and feature reductions seem to be needed for a PS3 version of anything similar. The simplest implementation is probably to just mirror characters as they are printed synchronously. I don't know the overhead of the hypervisor calls for copying graphics though, but the typical areas are quite small. Perhaps one could avoid allocating the kernel frame buffer as well when it's not needed. I have to investigate these things to be sure. > What about letting ps3fb register a panic notifier to sync the screen, like > hyperv_fb does? That would not work with kernel freezes unfortunately. Debugging those with nondeterministicly invisible kernel prints would be painful, I believe. Fredrik
Hi Fredrik, On Thu, Jul 19, 2018 at 10:14 PM Fredrik Noring <noring@nocrew.org> wrote: > > > > so I added a sleep with > > > > > > > > + msleep(10000); > > > > I can't see where you added the sleep, but 10s seems excessive. > > If the real reason is the need to wait for an interrupt for ps3fb_sync_image(), > > then waiting for 40 ms should be sufficient? Or am I missing something? > > It's at the end of ps3fb_probe, as shown in the original post: > > https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-July/175771.html Thanks, I had found that one in the mean time... > I thought 100 ms or so would work, but evidently it didn't. In fact, 1 s > for even 5 s didn't seem to work either. In any case, I would like to > develop a solution that does not need to sleep at all, so that will be my > first approach for a proper implementation. Hmm... > > > I suppose the problem is that it relies on interrupts for ps3fb_sync_image > > > > to regularly copy the image, hence without them the screen isn't updated to > > > > show kernel panics, etc. Perhaps one way to fix that is to implement the > > > > struct fb_tile_ops API, so that the console is synchronously updated? Would > > > > that be acceptable? > > > > > > I'm not sure if that would work or not. Maybe Geert is more familiar with it. > > > > That sounds like a complex solution, slowing down the console a lot. > > Why would that be slow? I have implemented a similar technique for the > PlayStation 2 frame buffer, and (without any measurements at hand now) it > appears to be about as fast as is possible, and reasonably easy too. :) [...] OK, I retract my statement ;-) > > What about letting ps3fb register a panic notifier to sync the screen, like > > hyperv_fb does? > > That would not work with kernel freezes unfortunately. Debugging those with > nondeterministicly invisible kernel prints would be painful, I believe. Unfortunately AFAIK the PS3 doesn't have any other "synchronous" output we can use for debugging (like flashing the power LED). Gr{oetje,eeting}s, Geert
Hi Geert, > > That would not work with kernel freezes unfortunately. Debugging those with > > nondeterministicly invisible kernel prints would be painful, I believe. > > Unfortunately AFAIK the PS3 doesn't have any other "synchronous" output we > can use for debugging (like flashing the power LED). Well, one can use lv1_panic to bisect the PowerPC boot process. It either freezes, in which case the call to lv1_panic isn't reached, or it reboots. This method yields 1 bit of information for every reflash and reboot, with a procedure taking about 5 minutes per turn, so 12 bits/hour sustained. ;) The prime advantage is that lv1_panic works everywhere, including the earliest boot stages. I've since learned about the PS3GELIC_UDBG option to broadcast UDP via the Ethernet port, but I don't know how well it works or its limitations. One useful option for debugging is to setup a very early graphical console, more or less from head.S, before any kernel functions are invoked. I have a simple proof of concept implemented for the PS2 here: https://github.com/frno7/linux/blob/ps2-v4.17-n3/arch/mips/boot/compressed/dbg.c This could be done for the PS3 as well. The OtherOS demo http://mc.pp.se/ps3/oodemo.xhtml by Marcus Comstedt in 2007 contains the essentials with MMU and hypervisor initialisation for graphics. I discovered that a small patch is required to make his demo work with modern GCC: --- a/source/script.lds +++ b/source/script.lds @@ -50,6 +50,7 @@ SECTIONS .opd : { *(.opd) } + . = ALIGN(256); .got : { __toc_start = .; *(.got) Fredrik
--- a/arch/powerpc/boot/wrapper +++ b/arch/powerpc/boot/wrapper @@ -550,5 +550,10 @@ ps3) bld="otheros.bld" [ $size -le 16777216 ] || bld="otheros-too-big.bld" gzip -n --force -9 --stdout "$ofile.bin" > "$odir/$bld" + if [ $size -gt 16777216 ] + then + echo "ERROR: Image size $size bytes is too large for 16 MiB limit" >&2 + exit 1 + fi ;; esac I then proceeded with disabling STACKTRACE_SUPPORT and LOCKDEP_SUPPORT, which reduced the size to about 8 MB, well below the 16 MiB limit. Perhaps disabling these entirely is a bit heavy-handed? Is there a smarter way? Trying to start the kernel results in a completely black screen. Nothing happens. To have a chance of seeing anything I had configured: CONFIG_FB_PS3=y CONFIG_FB_PS3_DEFAULT_SIZE_M=9 CONFIG_CMDLINE="video=ps3fb:mode:10" I decided to proceed by using lv1_panic to bisect the PowerPC boot process. It either froze, in which case the call to lv1_panic was not reached, or it rebooted. Interestingly, it turned out that ps3fb_probe was actually called, so I added a sleep with --- a/drivers/video/fbdev/ps3fb.c +++ b/drivers/video/fbdev/ps3fb.c @@ -1178,6 +1179,8 @@ static int ps3fb_probe(struct ps3_system_bus_device *dev) ps3fb.task = task; + msleep(10000); + return 0; err_unregister_framebuffer: