Precision Time Protocol (PTP)

Synchronizing network applications require their wall clock time to be aligned precisely with a reference time source (to the order of micro seconds or less). To achieve such accuracy, the application needs the support of networking HW (switch and adapter card), to provide the means to stamp time-sensitive packets. It also requires a time synchronization protocol which would make use of the HW time stamping to adjust its wall clock time to an accurate clock in the network.

The basic principle of PTP is as follows: Slave time = master time + propagation delay + offset.

The purpose of the protocol is to align the slave and the master time so that the gap between them is the propagation delay of the packet. Or in other words, the purpose of the protocol is to use the offset to correct the slave time so the offset between the master sending the packet and the slave receiving the packet is the propagation delay.

Master time is sent periodically by a reliable clock source named Master Clock (MC). In a PTP network, one single reference source is elected called Grand Master Clock (GMC). Propagation delay is calculated between each node and the MC by one of the two methods provided by the standard and further explained below.

To reach sub-microsecond resolutions, all the time stamps which record when a packet is sent and received should be done in the HW. This may impose interaction between SW and HW to query the HW time and send follow-up messages. This issue is further explained below in 2 step section.

Assuming that the propagation delay in the network is symmetric, the propagation time is the average time that took the sync and delay req messages to be switched.

Propagation delay = (T4-T1-(T3-T2))/2=(T4-T1+T2-T3)/2

T1 represents the time that the packet left the master which is actually the master time.

The following figure provides an example of the stages required by a slave clock to align its time to the master clock:

The following table presents the PTP message formats:

The types of clocks available are as follows:

• Grand Master Clock (GMC)—the reference time source derived from an accurate clock such as a GNSS driven clock (i.e. GPS, GLONASS, GALILEO)

• Boundary Clock (BC)—a network device that acts as slave to its master and as master to its slaves. (NVIDIA Onyx implements only this)

• Ordinary Clock (OC)—a clock that operates either as a Master or a Slave. In the case of a slave, the end point whose clock is been synced (normally a host/server).

• Master Clock (MC)—a clock which operates as a Master and derives its timing capabilities from the clock chain up to the GMC. It typically serves as a port on a BC connected to a host running as a slave.

• Transparent Clock (TC)—a PTP aware switch capable of measuring the PTP packet switching delay (transient time) and updating the data in the packet. In peer-to-peer (P2P) delay calculation mechanism, a TC device is also required to calculate its delay from the next hop toward the MC and add the value to the switching delay.

Two modes of delay calculations are defined:

• End-to-End (E2E)—each slave calculates its delay from the MC by running Delay request/ delay response sequence (NVIDIA Onyx implements only this)

• Peer-to-Peer—propagation delay (Pdelay) is calculated periodically on each link between the slave and the MC independently. The time synchronization packet sent from the MC to all the slaves in the network is updated by each of the downstream nodes with both switching delay (the time that the packet traversed the switch) and upstream hop Pdelay.

A domain consists of one or more PTP devices communicating with each other. PTP domain defines the scope of PTP message communication, state, operations, data sets, and timescale.

Boundary Clock

In a full E2E PTP deployment, the GMC needs to respond to each slave’s delay request message. A normal profile of PTP may require a few delay calculations per second. An average GMC is capable of addressing few thousands of messages per second. This imposes that direct slave/GMC communication limits the number of overall OCs to ~8K. To scale beyond that, there is a need for a hierarchy between the GMC and the slave. This is achieved by implementing BC, either in the TOR switches or on all the switches in the DC.

The following figure shows the master/slave role that a boundary clock implements between the MC and the Slave (OC).

Each BC acts as a slave towards the GMC and as GMC to its local slaves. Although adding a BC device introduces accuracy degradation as explained above, it becomes mandatory when the number of slaves on a single MC exceeds few thousand devices.

Another use of BC is to bridge between networks. When running PTP over native Ethernet packets, to create larger PTP domains, there is a need to bridge between the broadcast domains. This is done by BC switches.

Default PTP Profile Attributes (SMPTE 2059-2)

Configuring PTP

IEEE 1588 Precision Time Protocol (PTP) may be configured either on router or switch interfaces.

To enable PTP on a router interface you could simply enable it on the selected interface.

The process of configuring PTP on a switch interface is slightly different, however. PTP should be enabled on the interface itself as well as on the respective VLAN interface(s).

All PTP configuration for switch interfaces is taken from those defined on the VLAN interface.

To configure PTP on a router interface:

1. Enable the PTP CLI commands. Run:

   Copy

   Copied!

               
   
               
   switch (config) # protocol ptp
           

2. Configure the router interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 no switchport force
           

3. Add the primary IP address. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 ip address 172.16.1.1/24
           

4. Enable PTP on the interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 ptp enable
           

To verify the PTP configuration:

            

            
switch (config) # show ptp
PTP mode                : Boundary Clock
Message format          : Mixed
Acceptable Master Table : Enabled
Domain                  : 127
Clock identity          : 7c:fe:90:ff:fe:fa:21:88
GMC identity            : 7c:fe:90:ff:fe:fa:21:88
Number of master ports  : 1
Slave port interface    : N/A
 
PTP enabled interfaces:
  ----------------------------------------------------
  Port           VLAN      State        Forced Master
  ----------------------------------------------------
  Eth1/1         N/A       MASTER       no
        

To configure PTP on a switch interface:

1. Enable the PTP CLI commands. Run:

   Copy

   Copied!

               
   
               
   switch (config) # protocol ptp
           

2. Add the VLANs. Run:

   Copy

   Copied!

               
   
               
   switch (config) # vlan 2-3
           

3. Configure VLAN membership.
   For access interfaces, run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/2 switchport mode access
   switch (config) # interface ethernet 1/2 switchport access vlan 2
           

   For trunked interfaces, run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 switchport mode trunk
           

4. Enable PTP on the VLAN interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface vlan 2 ptp enable
   switch (config) # interface vlan 3 ptp enable
           

5. Enable PTP on the interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 ptp enable
           

   Warning

   The interface must be a member of the PTP enabled VLAN(s).

To verify the PTP configuration:

            

            
switch (config) # show ptp
PTP mode                : Boundary Clock
Message format          : Mixed
Acceptable Master Table : Enabled
Domain                  : 127
Clock identity          : 7c:fe:90:ff:fe:fa:21:88
GMC identity            : 7c:fe:90:ff:fe:fa:21:88
Number of master ports  : 2
Slave port interface    : N/A
 
PTP enabled interfaces:
  ----------------------------------------------------
  Port           VLAN      State        Forced Master
  ----------------------------------------------------
  Eth1/1         2         MASTER       no
  Eth1/2         2         MASTER       no
  Eth1/1         3         SLAVE        no
        

To protect the switch from rogue or mis-configured PTP endpoints, you may secure your Boundary Clock ports by creating an Acceptable Master Table (AMT) and configuring known PTP ports to always behave as a master port via the Forced Master option.

The AMT is a whitelist of up to 8 clock identities that are admissible to take part as valid GrandMasters in the Best Master Clock Algorithm (BMCA).

The Forced Master is enabled on a per-port basis to prevent processing announce messages from a PTP endpoint connected to it, in order for it to always stay in a Master state.

To configure Forced Master on a switch interface, you must enable it on the interface itself as well as on the respective VLAN interface(s).

To configure Acceptable Master Table, add the validated clock identities:

            

            
switch (config) # ptp amt E4:1D:2D:FF:FE:46:13:88
switch (config) # ptp amt E4:1D:2D:FF:FE:44:23:B7
        

To verify the Acceptable Master Table configuration:

            

            
switch (config) # show ptp amt
 
Clock Identities:
  E4:1D:2D:FF:FE:44:23:B7
  E4:1D:2D:FF:FE:46:13:88
        

To enable Forced Master on a router interface:

            

            
switch (config) # interface ethernet 1/2 ptp enable forced-master
        

To verify PTP configuration:

            

            
switch (config) # show ptp
PTP mode                : Boundary Clock
Message format          : Mixed
Acceptable Master Table : Enabled
Domain                  : 127
Clock identity          : 7c:fe:90:ff:fe:fa:21:88
GMC identity            : 7c:fe:90:ff:fe:fa:21:88
Number of master ports  : 1
Slave port interface    : N/A
 
PTP enabled interfaces:
  ----------------------------------------------------
  Port           VLAN      State        Forced Master
  ----------------------------------------------------
  Eth1/2         N/A       MASTER       yes
        

To configure Forced Master on a switch interface:

1. Enable Forced Master on the VLAN interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface vlan 2 ptp enable forced-master
           

2. Enable Forced Master on the interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ethernet 1/1 ptp enable forced-master
           

   Warning

   The interface should be a member in the PTP enabled VLAN(s).

To verify PTP configuration:

            

            
switch (config) # show ptp
PTP mode                : Boundary Clock
Message format          : Mixed
Acceptable Master Table : Enabled
Domain                  : 127
Clock identity          : 7c:fe:90:ff:fe:fa:21:88
GMC identity            : 7c:fe:90:ff:fe:fa:21:88
Number of master ports  : 2
Slave port interface    : N/A
 
PTP enabled interfaces:
  ----------------------------------------------------
  Port           VLAN      State        Forced Master
  ----------------------------------------------------
  Eth1/1         2         MASTER       yes
  Eth1/1         3         SLAVE        no
        

For more information about this feature and its potential applications, please refer to the following:

• IEEE 1588 Precision Time Protocol Design Guide → Guides (scroll to the bottom of the page)

• IEEE 1588 PTP on Spectrum Switches