[net-next,00/13] ice: implement support for PTP on E822 hardware

Extend the ice driver implementation to support PTP for the E822 based
devices.

This includes a few cleanup patches, that fix some minor issues spotted
while preparing them. In addition, there are some slight refactors to ease
the addition of E822 support, followed by adding the new hardware
implementation ice_ptp_hw.c.

There are a few major differences with E822 support compared to E810
support:

*) The E822 PHY is a bit different and requires a more complex
initialization procedure that requires delaying the PHY start until link is
up
*) The E822 device has a Clock Generation Unit which must be initialized in
order to generate proper clock frequencies on the output that drives the PTP
hardware clock registers
*) The E822 devices support enhanced timestamp calibration by making use of
a process called Vernier offset measurement. This allows the hardware to
measure phase offset related to the PHY clocks for Serdes and FEC, reducing
the inaccuracy of the timestamp relative to the actual packet transmission.
Making use of this requires programming the PHY to enable vernier
calibration and then detecting when the PHY has completed calibration. This
is done as part of a new kthread, ov_work.
*) E822 devices support crosstimestamping via PCIe PTM, which we enable when
available on the platform.

There is a fair amount of logic required to perform PHY and CGU
initialization, which is the vast majority of the new code, but it is fairly
self contained within ice_ptp_hw.c, with the exception of monitoring for
offset validity being handled by a kthread.

Jacob Keller (13):
  ice: fix Tx queue iteration for Tx timestamp enablement
  ice: remove dead code for allocating pin_config
  ice: add lock around Tx timestamp tracker flush
  ice: restart periodic outputs around time changes
  ice: introduce ice_base_incval function
  ice: PTP: move setting of tstamp_config
  ice: use 'int err' instead of 'int status'
  ice: introduce ice_ptp_init_phc function
  ice: convert clk_freq capability into time_ref
  ice: implement basic E822 PTP support
  ice: ensure the hardware Clock Generation Unit is configured
  ice: exit bypass mode once hardware finishes timestamp calibration
  ice: support crosstimestamping on E822 devices if supported

 drivers/net/ethernet/intel/ice/ice_cgu_regs.h |  116 +
 drivers/net/ethernet/intel/ice/ice_common.c   |   12 +
 .../net/ethernet/intel/ice/ice_hw_autogen.h   |    9 +
 drivers/net/ethernet/intel/ice/ice_main.c     |    7 +
 drivers/net/ethernet/intel/ice/ice_ptp.c      |  694 +++-
 drivers/net/ethernet/intel/ice/ice_ptp.h      |   30 +-
 .../net/ethernet/intel/ice/ice_ptp_consts.h   |  374 +++
 drivers/net/ethernet/intel/ice/ice_ptp_hw.c   | 2794 ++++++++++++++++-
 drivers/net/ethernet/intel/ice/ice_ptp_hw.h   |  345 ++
 drivers/net/ethernet/intel/ice/ice_type.h     |   23 +-
 10 files changed, 4193 insertions(+), 211 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ice/ice_cgu_regs.h
 create mode 100644 drivers/net/ethernet/intel/ice/ice_ptp_consts.h


base-commit: b6df00789e2831fff7a2c65aa7164b2a4dcbe599

Dear Jacob,


Thank you for sending the patches.


Am 01.07.21 um 02:27 schrieb Jacob Keller:
> Extend the ice driver implementation to support PTP for the E822 based
> devices.
> 
> This includes a few cleanup patches, that fix some minor issues spotted
> while preparing them. In addition, there are some slight refactors to ease
> the addition of E822 support, followed by adding the new hardware
> implementation ice_ptp_hw.c.
> 
> There are a few major differences with E822 support compared to E810
> support:
> 
> *) The E822 PHY is a bit different and requires a more complex
> initialization procedure that requires delaying the PHY start until link is
> up

It’d be great, if you gave concrete numbers.

[…]


Kind regards,

Paul

> -----Original Message-----
> From: Paul Menzel <pmenzel@molgen.mpg.de>
> Sent: Monday, July 05, 2021 12:47 AM
> To: Keller, Jacob E <jacob.e.keller@intel.com>
> Cc: intel-wired-lan@lists.osuosl.org
> Subject: Re: [Intel-wired-lan] [net-next 00/13] ice: implement support for PTP on
> E822 hardware
> 
> Dear Jacob,
> 
> 
> Thank you for sending the patches.
> 
> 
> Am 01.07.21 um 02:27 schrieb Jacob Keller:
> > Extend the ice driver implementation to support PTP for the E822 based
> > devices.
> >
> > This includes a few cleanup patches, that fix some minor issues spotted
> > while preparing them. In addition, there are some slight refactors to ease
> > the addition of E822 support, followed by adding the new hardware
> > implementation ice_ptp_hw.c.
> >
> > There are a few major differences with E822 support compared to E810
> > support:
> >
> > *) The E822 PHY is a bit different and requires a more complex
> > initialization procedure that requires delaying the PHY start until link is
> > up
> 
> It’d be great, if you gave concrete numbers.
> 

Concrete numbers on which part? I'm not sure if we have concrete numbers on everything here. For the calibration process, according to the data sheet I have, it indicates that uncalibrated timestamps (i.e. in bypass mode) have an error of up to 1 clock cycle and calibrated timestamps should have an error of less than 1/8th of a clock cycle. Here clock cycle refers to the length of one tick on the clock source, and error refers to difference between actual start of reception or transmission vs time of when the timstamp is captured. I unfortunately am not at all sure how this was measured by the hardware folks...

> […]
> 
> 
> Kind regards,
> 
> Paul

Dear Jacob,


Thank you for your reply.

Am 06.07.21 um 21:53 schrieb Keller, Jacob E:

>> -----Original Message-----
>> From: Paul Menzel <pmenzel@molgen.mpg.de>
>> Sent: Monday, July 05, 2021 12:47 AM
>> To: Keller, Jacob E <jacob.e.keller@intel.com>
>> Cc: intel-wired-lan@lists.osuosl.org
>> Subject: Re: [Intel-wired-lan] [net-next 00/13] ice: implement support for PTP on
>> E822 hardware

>> Am 01.07.21 um 02:27 schrieb Jacob Keller:
>>> Extend the ice driver implementation to support PTP for the E822 based
>>> devices.
>>>
>>> This includes a few cleanup patches, that fix some minor issues spotted
>>> while preparing them. In addition, there are some slight refactors to ease
>>> the addition of E822 support, followed by adding the new hardware
>>> implementation ice_ptp_hw.c.
>>>
>>> There are a few major differences with E822 support compared to E810
>>> support:
>>>
>>> *) The E822 PHY is a bit different and requires a more complex
>>> initialization procedure that requires delaying the PHY start until link is
>>> up
>>
>> It’d be great, if you gave concrete numbers.
>> 
> 
> Concrete numbers on which part? I'm not sure if we have concrete
> numbers on everything here. For the calibration process, according to
> the data sheet I have, it indicates that uncalibrated timestamps
> (i.e. in bypass mode) have an error of up to 1 clock cycle and
> calibrated timestamps should have an error of less than 1/8th of a
> clock cycle. Here clock cycle refers to the length of one tick on the
> clock source, and error refers to difference between actual start of
> reception or transmission vs time of when the timstamp is captured. I
> unfortunately am not at all sure how this was measured by the
> hardware folks...
Thank you for the explanation. I meant, how big the delay is until the 
link is up.


Kind regards,

Paul

> -----Original Message-----
> From: Paul Menzel <pmenzel@molgen.mpg.de>
> Sent: Wednesday, July 07, 2021 12:52 AM
> To: Keller, Jacob E <jacob.e.keller@intel.com>
> Cc: intel-wired-lan@lists.osuosl.org
> Subject: Re: [Intel-wired-lan] [net-next 00/13] ice: implement support for PTP on
> E822 hardware
> 
> Dear Jacob,
> 
> 
> Thank you for your reply.
> 
> Am 06.07.21 um 21:53 schrieb Keller, Jacob E:
> 
> >> -----Original Message-----
> >> From: Paul Menzel <pmenzel@molgen.mpg.de>
> >> Sent: Monday, July 05, 2021 12:47 AM
> >> To: Keller, Jacob E <jacob.e.keller@intel.com>
> >> Cc: intel-wired-lan@lists.osuosl.org
> >> Subject: Re: [Intel-wired-lan] [net-next 00/13] ice: implement support for PTP
> on
> >> E822 hardware
> 
> >> Am 01.07.21 um 02:27 schrieb Jacob Keller:
> >>> Extend the ice driver implementation to support PTP for the E822 based
> >>> devices.
> >>>
> >>> This includes a few cleanup patches, that fix some minor issues spotted
> >>> while preparing them. In addition, there are some slight refactors to ease
> >>> the addition of E822 support, followed by adding the new hardware
> >>> implementation ice_ptp_hw.c.
> >>>
> >>> There are a few major differences with E822 support compared to E810
> >>> support:
> >>>
> >>> *) The E822 PHY is a bit different and requires a more complex
> >>> initialization procedure that requires delaying the PHY start until link is
> >>> up
> >>
> >> It’d be great, if you gave concrete numbers.
> >>
> >
> > Concrete numbers on which part? I'm not sure if we have concrete
> > numbers on everything here. For the calibration process, according to
> > the data sheet I have, it indicates that uncalibrated timestamps
> > (i.e. in bypass mode) have an error of up to 1 clock cycle and
> > calibrated timestamps should have an error of less than 1/8th of a
> > clock cycle. Here clock cycle refers to the length of one tick on the
> > clock source, and error refers to difference between actual start of
> > reception or transmission vs time of when the timstamp is captured. I
> > unfortunately am not at all sure how this was measured by the
> > hardware folks...
> Thank you for the explanation. I meant, how big the delay is until the
> link is up.
> 


Ah, sorry for the confusion. So there's two things here, and I'll reword this block a bit in my next submission.

So first, we are waiting until hardware link is up, which could be down due to factors outside our control such as no cable, in which case it is going to be down until a cable is plugged in etc. I have no data on how long it takes link to go up after the cable is inserted. This isn't really a problem for general use though, as the delay is likely to be low when compared to observable user interaction. The issue is that if there is no link we can't start the calibration process. The expectation here is that either link is up, or it is down due to no cable, no link partner, or similar.

In addition, I guess not covered here, that I will expand on, is that the calibration process itself requires data from the first sent and received packets, which are outside the control of the driver, and could easily be a long time until the first packets are sent. We handle this by enabling the PHY in a bypass mode that enables the timestamps immediately with a slightly reduced accuracy due to missing calibration data.

I'll rephrase this section and do my best to clarify it in v3 which I'm sending soon.

Thanks,
Jake

> 
> Kind regards,
> 
> Paul