| Message ID | A88094362DE0AE49A118AB9B4EB3612403F9DEBE@dekaexchange.deka.local (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | Grant Likely |
| Headers | show |
On Wed, May 20, 2009 at 9:42 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: >> > I am able to reproduce the error using 2.6.29.2-rt11. I was able to >> > mitigate the problem by raising the priority of the transmit irq. >> > However when running an NFS server on the pcm030 under high cpu load I >> > now get >> > >> > [ 132.477503] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 132.892329] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 133.884109] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 134.876059] net eth0: FEC_IEVENT_RFIFO_ERROR >> > >> > Raising the priority of the rx irq does not seem to fix this problem >> > though. >> >> Hi Eric, >> >> This error has been seen before in non-rt kernels. I haven't had the >> chance to track it down and kill it yet. I believe there are locking >> issues associated with it. > > Uuuh, I recall this message. Kept me busy for some time :( > > You might try this patch which helped in our situation. > > =========== > > Subject: Enable XLB pipelining for MPC5200B > From: Wolfram Sang <w.sang@pengutronix.de> > > Enable pipelining as it helps getting rid of FEC problems. > Not intended for upstream, this must be dealt differently there. > > This patch is disabled by default. The bootloader should enable this feature. > So, this patch is only intended to be used where the bootloader does it in a > wrong manner and can't be replaced. > > Signed-off-by: Wolfram Sang <w.sang@pengutronix.de> > Acked-by: Juergen Beisert <j.beisert@pengutronix.de> > > --- > arch/powerpc/platforms/52xx/mpc52xx_common.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > Index: arch/powerpc/platforms/52xx/mpc52xx_common.c > =================================================================== > --- arch/powerpc/platforms/52xx/mpc52xx_common.c.orig > +++ arch/powerpc/platforms/52xx/mpc52xx_common.c > @@ -107,6 +107,13 @@ mpc5200_setup_xlb_arbiter(void) > */ > if ((mfspr(SPRN_SVR) & MPC5200_SVR_MASK) == MPC5200_SVR) > out_be32(&xlb->config, in_be32(&xlb->config) | MPC52xx_XLB_CFG_PLDIS); > +#if 0 > + /* > + * Enable pipelining, fixes FEC problems. The previous workaround is not > + * needed, as we have an MPC5200B (not A). > + */ > + out_be32(&xlb->config, in_be32(&xlb->config) & ~MPC52xx_XLB_CFG_PLDIS); > +#endif > > iounmap(xlb); > } > > -- > Pengutronix e.K. | Wolfram Sang | > Industrial Linux Solutions | http://www.pengutronix.de/ | > > Wolfram, > > Thanks, but no luck with this patch. It was already setup correctly by U-Boot. I don't see where this gets enabled in the u-boot source. Have you added it locally? These are the only two I see: cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 13); cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 15);
-----Original Message----- From: Jon Smirl [mailto:jonsmirl@gmail.com] Sent: Wednesday, May 20, 2009 11:15 To: Eric Millbrandt Cc: Wolfram Sang; Grant Likely; linuxppc-dev@ozlabs.org Subject: Re: mpc5200 fec error On Wed, May 20, 2009 at 9:42 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: >> > I am able to reproduce the error using 2.6.29.2-rt11. I was able to >> > mitigate the problem by raising the priority of the transmit irq. >> > However when running an NFS server on the pcm030 under high cpu load I >> > now get >> > >> > [ 132.477503] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 132.892329] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 133.884109] net eth0: FEC_IEVENT_RFIFO_ERROR >> > [ 134.876059] net eth0: FEC_IEVENT_RFIFO_ERROR >> > >> > Raising the priority of the rx irq does not seem to fix this problem >> > though. >> >> Hi Eric, >> >> This error has been seen before in non-rt kernels. I haven't had the >> chance to track it down and kill it yet. I believe there are locking >> issues associated with it. > > Uuuh, I recall this message. Kept me busy for some time :( > > You might try this patch which helped in our situation. > > =========== > > Subject: Enable XLB pipelining for MPC5200B > From: Wolfram Sang <w.sang@pengutronix.de> > > Enable pipelining as it helps getting rid of FEC problems. > Not intended for upstream, this must be dealt differently there. > > This patch is disabled by default. The bootloader should enable this feature. > So, this patch is only intended to be used where the bootloader does it in a > wrong manner and can't be replaced. > > Signed-off-by: Wolfram Sang <w.sang@pengutronix.de> > Acked-by: Juergen Beisert <j.beisert@pengutronix.de> > > --- > arch/powerpc/platforms/52xx/mpc52xx_common.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > Index: arch/powerpc/platforms/52xx/mpc52xx_common.c > =================================================================== > --- arch/powerpc/platforms/52xx/mpc52xx_common.c.orig > +++ arch/powerpc/platforms/52xx/mpc52xx_common.c > @@ -107,6 +107,13 @@ mpc5200_setup_xlb_arbiter(void) > */ > if ((mfspr(SPRN_SVR) & MPC5200_SVR_MASK) == MPC5200_SVR) > out_be32(&xlb->config, in_be32(&xlb->config) | MPC52xx_XLB_CFG_PLDIS); > +#if 0 > + /* > + * Enable pipelining, fixes FEC problems. The previous workaround is not > + * needed, as we have an MPC5200B (not A). > + */ > + out_be32(&xlb->config, in_be32(&xlb->config) & ~MPC52xx_XLB_CFG_PLDIS); > +#endif > > iounmap(xlb); > } > > -- > Pengutronix e.K. | Wolfram Sang | > Industrial Linux Solutions | http://www.pengutronix.de/ | > > Wolfram, > > Thanks, but no luck with this patch. It was already setup correctly by U-Boot. I don't see where this gets enabled in the u-boot source. Have you added it locally? These are the only two I see: cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 13); cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 15);
On Wed, May 20, 2009 at 11:28 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: > -----Original Message----- > From: Jon Smirl [mailto:jonsmirl@gmail.com] > Sent: Wednesday, May 20, 2009 11:15 > To: Eric Millbrandt > Cc: Wolfram Sang; Grant Likely; linuxppc-dev@ozlabs.org > Subject: Re: mpc5200 fec error > > On Wed, May 20, 2009 at 9:42 AM, Eric Millbrandt > <emillbrandt@dekaresearch.com> wrote: >>> > I am able to reproduce the error using 2.6.29.2-rt11. I was able to >>> > mitigate the problem by raising the priority of the transmit irq. >>> > However when running an NFS server on the pcm030 under high cpu load I >>> > now get >>> > >>> > [ 132.477503] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 132.892329] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 133.884109] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 134.876059] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > >>> > Raising the priority of the rx irq does not seem to fix this problem >>> > though. >>> >>> Hi Eric, >>> >>> This error has been seen before in non-rt kernels. I haven't had the >>> chance to track it down and kill it yet. I believe there are locking >>> issues associated with it. >> >> Uuuh, I recall this message. Kept me busy for some time :( >> >> You might try this patch which helped in our situation. >> >> =========== >> >> Subject: Enable XLB pipelining for MPC5200B >> From: Wolfram Sang <w.sang@pengutronix.de> >> >> Enable pipelining as it helps getting rid of FEC problems. >> Not intended for upstream, this must be dealt differently there. >> >> This patch is disabled by default. The bootloader should enable this feature. >> So, this patch is only intended to be used where the bootloader does it in a >> wrong manner and can't be replaced. >> >> Signed-off-by: Wolfram Sang <w.sang@pengutronix.de> >> Acked-by: Juergen Beisert <j.beisert@pengutronix.de> >> >> --- >> arch/powerpc/platforms/52xx/mpc52xx_common.c | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> Index: arch/powerpc/platforms/52xx/mpc52xx_common.c >> =================================================================== >> --- arch/powerpc/platforms/52xx/mpc52xx_common.c.orig >> +++ arch/powerpc/platforms/52xx/mpc52xx_common.c >> @@ -107,6 +107,13 @@ mpc5200_setup_xlb_arbiter(void) >> */ >> if ((mfspr(SPRN_SVR) & MPC5200_SVR_MASK) == MPC5200_SVR) >> out_be32(&xlb->config, in_be32(&xlb->config) | MPC52xx_XLB_CFG_PLDIS); >> +#if 0 >> + /* >> + * Enable pipelining, fixes FEC problems. The previous workaround is not >> + * needed, as we have an MPC5200B (not A). >> + */ >> + out_be32(&xlb->config, in_be32(&xlb->config) & ~MPC52xx_XLB_CFG_PLDIS); >> +#endif >> >> iounmap(xlb); >> } >> >> -- >> Pengutronix e.K. | Wolfram Sang | >> Industrial Linux Solutions | http://www.pengutronix.de/ | >> >> Wolfram, >> >> Thanks, but no luck with this patch. It was already setup correctly by U-Boot. > > > I don't see where this gets enabled in the u-boot source. Have you > added it locally? > > These are the only two I see: > cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 13); > cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 15); > > > -- > Jon Smirl > jonsmirl@gmail.com > > If you look a few more lines down you should see > # if defined(CFG_XLB_PIPELINING) > /* Enable piplining */ > *(vu_long *)(MPC5XXX_XLBARB + 0x40) &= ~(1 << 31); > # endif > > I obtained the u-boot sources directly from phytec, for my phyCore, so I am not sure if they added it. It is ok in main u-boot source. I just missed the lower section.
> On Wed, May 20, 2009 at 9:42 AM, Eric Millbrandt > <emillbrandt@dekaresearch.com> wrote: >>> > I am able to reproduce the error using 2.6.29.2-rt11. I was able to >>> > mitigate the problem by raising the priority of the transmit irq. >>> > However when running an NFS server on the pcm030 under high cpu load I >>> > now get >>> > >>> > [ 132.477503] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 132.892329] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 133.884109] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > [ 134.876059] net eth0: FEC_IEVENT_RFIFO_ERROR >>> > >>> > Raising the priority of the rx irq does not seem to fix this problem >>> > though. >>> >>> Hi Eric, >>> >>> This error has been seen before in non-rt kernels. I haven't had the >>> chance to track it down and kill it yet. I believe there are locking >>> issues associated with it. >> >> Uuuh, I recall this message. Kept me busy for some time :( >> >> You might try this patch which helped in our situation. >> >> =========== >> >> Subject: Enable XLB pipelining for MPC5200B >> From: Wolfram Sang <w.sang@pengutronix.de> >> >> Enable pipelining as it helps getting rid of FEC problems. >> Not intended for upstream, this must be dealt differently there. >> >> This patch is disabled by default. The bootloader should enable this feature. >> So, this patch is only intended to be used where the bootloader does it in a >> wrong manner and can't be replaced. >> >> Signed-off-by: Wolfram Sang <w.sang@pengutronix.de> >> Acked-by: Juergen Beisert <j.beisert@pengutronix.de> >> >> --- >> arch/powerpc/platforms/52xx/mpc52xx_common.c | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> Index: arch/powerpc/platforms/52xx/mpc52xx_common.c >> =================================================================== >> --- arch/powerpc/platforms/52xx/mpc52xx_common.c.orig >> +++ arch/powerpc/platforms/52xx/mpc52xx_common.c >> @@ -107,6 +107,13 @@ mpc5200_setup_xlb_arbiter(void) >> */ >> if ((mfspr(SPRN_SVR) & MPC5200_SVR_MASK) == MPC5200_SVR) >> out_be32(&xlb->config, in_be32(&xlb->config) | MPC52xx_XLB_CFG_PLDIS); >> +#if 0 >> + /* >> + * Enable pipelining, fixes FEC problems. The previous workaround is not >> + * needed, as we have an MPC5200B (not A). >> + */ >> + out_be32(&xlb->config, in_be32(&xlb->config) & ~MPC52xx_XLB_CFG_PLDIS); >> +#endif >> >> iounmap(xlb); >> } >> >> -- >> Pengutronix e.K. | Wolfram Sang | >> Industrial Linux Solutions | http://www.pengutronix.de/ | >> >> Wolfram, >> >> Thanks, but no luck with this patch. It was already setup correctly by U-Boot. > > > I don't see where this gets enabled in the u-boot source. Have you > added it locally? > > These are the only two I see: > cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 13); > cpu/mpc5xxx/cpu_init.c: *(vu_long *)(MPC5XXX_XLBARB + 0x40) |= (1 << 15); > > > -- > Jon Smirl > jonsmirl@gmail.com > > If you look a few more lines down you should see > # if defined(CFG_XLB_PIPELINING) > /* Enable piplining */ > *(vu_long *)(MPC5XXX_XLBARB + 0x40) &= ~(1 << 31); > # endif > > I obtained the u-boot sources directly from phytec, for my phyCore, so I am not sure if they added it. It is ok in main u-boot source. I just missed the lower section.
On Wed, May 20, 2009 at 10:41 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: > It looks like the phy is never getting reset properly after the FEC_IEVENT_RFIFO_ERROR. I threw some printk's into the fec mdio driver Yes, that sounds familiar. Most likely, the value of the MDIO bus control register got clobbered and not reset when the FEC was reset. Try adding this line to the beginning of mpc52xx_fec_mdio_transfer(): out_be32(&fec->mii_speed, 0x7e); It's a dirty ugly hack, but it should help. If that works, then I can come up with a better solution. Part of the problem is that the MDIO handling in the current code really isn't very good. I've got changes queued up in -next which cleans it up quite a bit which should make it easier to fix properly. g. > > /drivers/net/fec_mpc52xx_phy.c > 25 static int mpc52xx_fec_mdio_transfer(struct mii_bus *bus, int phy_id, > 26 int reg, u32 value) > 27 { > 28 struct mpc52xx_fec_mdio_priv *priv = bus->priv; > 29 struct mpc52xx_fec __iomem *fec; > 30 int tries = 100; > 31 > 32 value |= (phy_id << FEC_MII_DATA_PA_SHIFT) & FEC_MII_DATA_PA_MSK; > 33 value |= (reg << FEC_MII_DATA_RA_SHIFT) & FEC_MII_DATA_RA_MSK; > 34 > 35 fec = priv->regs; > 36 out_be32(&fec->ievent, FEC_IEVENT_MII); > 37 out_be32(&priv->regs->mii_data, value); > 38 > 39 /* wait for it to finish, this takes about 23 us on lite5200b */ > 40 while (!(in_be32(&fec->ievent) & FEC_IEVENT_MII) && --tries) > 41 udelay(5); > 42 > 43 if (!tries) { > 44 printk("Unable to reset phy\n"); > 45 return -ETIMEDOUT; > 46 } > 47 > 48 return value & FEC_MII_DATA_OP_RD ? > 49 in_be32(&priv->regs->mii_data) & FEC_MII_DATA_DATAMSK : 0; > 50 } > > and got this > > [ 127.085632] net eth0: FEC_IEVENT_RFIFO_ERROR > [ 127.093401] Resetting FEC > [ 127.096620] Unable to reset phy > [ 127.322224] Unable to reset phy > [ 128.321987] PHY: f0003000:00 - Link is Down > [ 129.221034] net eth0: FEC_IEVENT_RFIFO_ERROR > [ 129.228165] Resetting FEC > [ 129.231384] Unable to reset phy > [ 129.322286] Unable to reset phy > [ 132.885863] net eth0: FEC_IEVENT_RFIFO_ERROR > [ 132.893002] Resetting FEC > [ 132.896176] Unable to reset phy > [ 133.322202] Unable to reset phy > [ 135.684793] net eth0: FEC_IEVENT_RFIFO_ERROR > [ 135.691969] Resetting FEC > [ 135.695192] Unable to reset phy > [ 136.322530] Unable to reset phy > [ 138.490805] net eth0: FEC_IEVENT_RFIFO_ERROR > [ 138.497963] Resetting FEC > [ 138.501135] Unable to reset phy > [ 139.334200] Unable to reset phy > > I guess I have to spend some quality time with the 5200 manual. > > > _________________________________________________________________________________________ > > This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. > > Thank you. > > Please consider the environment before printing this email. >
-----Original Message----- From: Grant Likely [mailto:grant.likely@secretlab.ca] Sent: Wednesday, May 20, 2009 12:49 To: Eric Millbrandt Cc: Jon Smirl; Wolfram Sang; linuxppc-dev@ozlabs.org Subject: Re: mpc5200 fec error On Wed, May 20, 2009 at 10:41 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: > It looks like the phy is never getting reset properly after the FEC_IEVENT_RFIFO_ERROR. I threw some printk's into the fec mdio driver Yes, that sounds familiar. Most likely, the value of the MDIO bus control register got clobbered and not reset when the FEC was reset. Try adding this line to the beginning of mpc52xx_fec_mdio_transfer(): out_be32(&fec->mii_speed, 0x7e); It's a dirty ugly hack, but it should help. If that works, then I can come up with a better solution. Part of the problem is that the MDIO handling in the current code really isn't very good. I've got changes queued up in -next which cleans it up quite a bit which should make it easier to fix properly. g. That worked! I'm still getting the fifo receive errors, but at least now the fec recovers. [ 127.761365] net eth0: FEC_IEVENT_RFIFO_ERROR [ 129.274341] PHY: f0003000:00 - Link is Down [ 130.274266] PHY: f0003000:00 - Link is Up - 100/Full [ 134.955324] net eth0: FEC_IEVENT_RFIFO_ERROR [ 136.273959] PHY: f0003000:00 - Link is Down [ 137.274090] PHY: f0003000:00 - Link is Up - 100/Full [ 140.521462] net eth0: FEC_IEVENT_RFIFO_ERROR [ 142.273955] PHY: f0003000:00 - Link is Down [ 143.273954] PHY: f0003000:00 - Link is Up - 100/Full [ 148.471582] net eth0: FEC_IEVENT_RFIFO_ERROR [ 150.273984] PHY: f0003000:00 - Link is Down [ 151.273901] PHY: f0003000:00 - Link is Up - 100/Full Thanks Grant.
[ed: quoting repaired] On Wed, May 20, 2009 at 11:26 AM, Eric Millbrandt <emillbrandt@dekaresearch.com> wrote: > Grant Likely wrote: > > Yes, that sounds familiar. Most likely, the value of the MDIO bus > > control register got clobbered and not reset when the FEC was reset. > > Try adding this line to the beginning of mpc52xx_fec_mdio_transfer(): > > > > out_be32(&fec->mii_speed, 0x7e); > > > > It's a dirty ugly hack, but it should help. If that works, then I can > > come up with a better solution. Part of the problem is that the MDIO > > handling in the current code really isn't very good. I've got changes > > queued up in -next which cleans it up quite a bit which should make it > > easier to fix properly. > > That worked! I'm still getting the fifo receive errors, but at least now the fec recovers. Okay, I'll see if I can work that into a cleaner patch. In fact, I should look into reworking it so that the PHY doesn't get reset on a FIFO error. It shouldn't need to be reset at all AFAIKT. That way even when FIFO errors occur, they should not cause an expensive renegotiate time. g.
> Yes, that sounds familiar. Most likely, the value of the MDIO bus > control register got clobbered and not reset when the FEC was reset. I recall that I wondered about the RFIFO-error case back then. The manual states === Receive FIFO Error - indicates error occurred within the RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared, halting FEC frame processing. When this occurs, software must ensure both the FIFO Controller and BestComm are soft-reset. === It does not say that the whole FEC needs to be reset. (BTW doing a full FEC reset in IRQ context caused OOPSes back then, have to dig up details, as it doesn't seem to happen here...) Regards, Wolfram
=========== Subject: Enable XLB pipelining for MPC5200B From: Wolfram Sang <w.sang@pengutronix.de> Enable pipelining as it helps getting rid of FEC problems. Not intended for upstream, this must be dealt differently there. This patch is disabled by default. The bootloader should enable this feature. So, this patch is only intended to be used where the bootloader does it in a wrong manner and can't be replaced. Signed-off-by: Wolfram Sang <w.sang@pengutronix.de> Acked-by: Juergen Beisert <j.beisert@pengutronix.de> --- arch/powerpc/platforms/52xx/mpc52xx_common.c | 7 +++++++ 1 file changed, 7 insertions(+) Index: arch/powerpc/platforms/52xx/mpc52xx_common.c =================================================================== --- arch/powerpc/platforms/52xx/mpc52xx_common.c.orig +++ arch/powerpc/platforms/52xx/mpc52xx_common.c @@ -107,6 +107,13 @@ mpc5200_setup_xlb_arbiter(void) */ if ((mfspr(SPRN_SVR) & MPC5200_SVR_MASK) == MPC5200_SVR) out_be32(&xlb->config, in_be32(&xlb->config) | MPC52xx_XLB_CFG_PLDIS); +#if 0 + /* + * Enable pipelining, fixes FEC problems. The previous workaround is not + * needed, as we have an MPC5200B (not A). + */ + out_be32(&xlb->config, in_be32(&xlb->config) & ~MPC52xx_XLB_CFG_PLDIS); +#endif iounmap(xlb); }