NVIDIA Onyx User Manual v3.10.2002
NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance v8.2.2200

Precision Time Protocol (PTP)

Warning

This feature is currently not supported in Spectrum-3 based switches.

Synchronizing network applications require their wall clock time to be aligned precisely with a reference time source (to the order of micro seconds or less). To achieve such accuracy, the application needs the support of networking HW (switch and adapter card), to provide the means to stamp time-sensitive packets. It also requires a time synchronization protocol which would make use of the HW time stamping to adjust its wall clock time to an accurate clock in the network.

The basic principle of PTP is as follows: Slave time = master time + propagation delay + offset.

The purpose of the protocol is to align the slave and the master time so that the gap between them is the propagation delay of the packet. Or in other words, the purpose of the protocol is to use the offset to correct the slave time so the offset between the master sending the packet and the slave receiving the packet is the propagation delay.

Master time is sent periodically by a reliable clock source named Master Clock (MC). In a PTP network, one single reference source is elected called Grand Master Clock (GMC). Propagation delay is calculated between each node and the MC by one of the two methods provided by the standard and further explained below.

To reach sub-microsecond resolutions, all the time stamps which record when a packet is sent and received should be done in the HW. This may impose interaction between SW and HW to query the HW time and send follow-up messages. This issue is further explained below in 2 step section.

Assuming that the propagation delay in the network is symmetric, the propagation time is the average time that took the sync and delay req messages to be switched.

Propagation delay = (T4-T1-(T3-T2))/2=(T4-T1+T2-T3)/2

T1 represents the time that the packet left the master which is actually the master time.

The following figure provides an example of the stages required by a slave clock to align its time to the master clock:

PTP_Clock_Synchronization_Example.png

The following table presents the PTP message formats:

Message Type

Hex Value

Class

Sync

0

Event

Follow-up

8

General

Delay_Req

1

Event

Delay_Resp

9

General

Pdelay_Req

2

Event

Pdelay_Resp

3

Event

Pdelay_Resp follow-up

A

General

Announce

B

General

Signaling

C

General

Management

D

General

The types of clocks available are as follows:

  • Grand Master Clock (GMC)—the reference time source derived from an accurate clock such as a GNSS driven clock (i.e. GPS, GLONASS, GALILEO)

  • Boundary Clock (BC)—a network device that acts as slave to its master and as master to its slaves. (NVIDIA Onyx implements only this)

  • Ordinary Clock (OC)—a clock that operates either as a Master or a Slave. In the case of a slave, the end point whose clock is been synced (normally a host/server).

  • Master Clock (MC)—a clock which operates as a Master and derives its timing capabilities from the clock chain up to the GMC. It typically serves as a port on a BC connected to a host running as a slave.

  • Transparent Clock (TC)—a PTP aware switch capable of measuring the PTP packet switching delay (transient time) and updating the data in the packet. In peer-to-peer (P2P) delay calculation mechanism, a TC device is also required to calculate its delay from the next hop toward the MC and add the value to the switching delay.

Two modes of delay calculations are defined:

  • End-to-End (E2E)—each slave calculates its delay from the MC by running Delay request/ delay response sequence (NVIDIA Onyx implements only this)

  • Peer-to-Peer—propagation delay (Pdelay) is calculated periodically on each link between the slave and the MC independently. The time synchronization packet sent from the MC to all the slaves in the network is updated by each of the downstream nodes with both switching delay (the time that the packet traversed the switch) and upstream hop Pdelay.

A domain consists of one or more PTP devices communicating with each other. PTP domain defines the scope of PTP message communication, state, operations, data sets, and timescale.

Boundary Clock

In a full E2E PTP deployment, the GMC needs to respond to each slave’s delay request message. A normal profile of PTP may require a few delay calculations per second. An average GMC is capable of addressing few thousands of messages per second. This imposes that direct slave/GMC communication limits the number of overall OCs to ~8K. To scale beyond that, there is a need for a hierarchy between the GMC and the slave. This is achieved by implementing BC, either in the TOR switches or on all the switches in the DC.

The following figure shows the master/slave role that a boundary clock implements between the MC and the Slave (OC).

Boundary_Clock_Master_or_Slave_Functionality.png

Each BC acts as a slave towards the GMC and as GMC to its local slaves. Although adding a BC device introduces accuracy degradation as explained above, it becomes mandatory when the number of slaves on a single MC exceeds few thousand devices.

Another use of BC is to bridge between networks. When running PTP over native Ethernet packets, to create larger PTP domains, there is a need to bridge between the broadcast domains. This is done by BC switches.

Default PTP Profile Attributes (SMPTE 2059-2)

Name

Range

Default

Announce interval

-3 (0.125s), 1 (2s)

-2 (0.25s)

Announce timeout interval

2, 10

3

Sync interval (logSyncInt)

-7, -1

-3

Delay request interval

logSyncInt, logSyncInt +5

logSyncInt

PTP domain

0, 127

127

Priority 1

0, 255

128

Priority 2

0, 255

128

Configuring PTP

IEEE 1588 Precision Time Protocol (PTP) may be configured either on router or switch interfaces.

To enable PTP on a router interface you could simply enable it on the selected interface.

The process of configuring PTP on a switch interface is slightly different, however. PTP should be enabled on the interface itself as well as on the respective VLAN interface(s).

All PTP configuration for switch interfaces is taken from those defined on the VLAN interface.

Warning
  • Prior to enabling PTP, NTP must be disabled.

  • When changing PTP configurations, PTP process restarts.

  • PTP restarts under the following conditions:

    1. Removing/adding PTP-enabled VLAN from switchport configuration.

    2. Removing/adding physical interface to PTP-enabled list.

    3. Any direct PTP configuration change (forced-master, AMT, intervals, and so forth).

To configure PTP on a router interface:

  1. Enable the PTP CLI commands. Run:

    Copy
    Copied!
                

    switch (config) # protocol ptp

  2. Configure the router interface. Run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 no switchport force

  3. Add the primary IP address. Run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 ip address 172.16.1.1/24

  4. Enable PTP on the interface. Run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 ptp enable

To verify the PTP configuration:

Copy
Copied!
            

switch (config) # show ptp PTP mode : Boundary Clock Message format : Mixed Acceptable Master Table : Enabled Domain : 127 Clock identity : 7c:fe:90:ff:fe:fa:21:88 GMC identity : 7c:fe:90:ff:fe:fa:21:88 Number of master ports : 1 Slave port interface : N/A   PTP enabled interfaces: ---------------------------------------------------- Port VLAN State Forced Master ---------------------------------------------------- Eth1/1 N/A MASTER no

To configure PTP on a switch interface:

  1. Enable the PTP CLI commands. Run:

    Copy
    Copied!
                

    switch (config) # protocol ptp

  2. Add the VLANs. Run:

    Copy
    Copied!
                

    switch (config) # vlan 2-3

  3. Configure VLAN membership.
    For access interfaces, run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/2 switchport mode access switch (config) # interface ethernet 1/2 switchport access vlan 2

    For trunked interfaces, run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 switchport mode trunk

  4. Enable PTP on the VLAN interface. Run:

    Copy
    Copied!
                

    switch (config) # interface vlan 2 ptp enable switch (config) # interface vlan 3 ptp enable

  5. Enable PTP on the interface. Run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 ptp enable

    Warning

    The interface must be a member of the PTP enabled VLAN(s).

To verify the PTP configuration:

Copy
Copied!
            

switch (config) # show ptp PTP mode : Boundary Clock Message format : Mixed Acceptable Master Table : Enabled Domain : 127 Clock identity : 7c:fe:90:ff:fe:fa:21:88 GMC identity : 7c:fe:90:ff:fe:fa:21:88 Number of master ports : 2 Slave port interface : N/A   PTP enabled interfaces: ---------------------------------------------------- Port VLAN State Forced Master ---------------------------------------------------- Eth1/1 2 MASTER no Eth1/2 2 MASTER no Eth1/1 3 SLAVE no

To protect the switch from rogue or mis-configured PTP endpoints, you may secure your Boundary Clock ports by creating an Acceptable Master Table (AMT) and configuring known PTP ports to always behave as a master port via the Forced Master option.

The AMT is a whitelist of up to 8 clock identities that are admissible to take part as valid GrandMasters in the Best Master Clock Algorithm (BMCA).

The Forced Master is enabled on a per-port basis to prevent processing announce messages from a PTP endpoint connected to it, in order for it to always stay in a Master state.

To configure Forced Master on a switch interface, you must enable it on the interface itself as well as on the respective VLAN interface(s).

To configure Acceptable Master Table, add the validated clock identities:

Copy
Copied!
            

switch (config) # ptp amt E4:1D:2D:FF:FE:46:13:88 switch (config) # ptp amt E4:1D:2D:FF:FE:44:23:B7

To verify the Acceptable Master Table configuration:

Copy
Copied!
            

switch (config) # show ptp amt   Clock Identities: E4:1D:2D:FF:FE:44:23:B7 E4:1D:2D:FF:FE:46:13:88

To enable Forced Master on a router interface:

Copy
Copied!
            

switch (config) # interface ethernet 1/2 ptp enable forced-master

To verify PTP configuration:

Copy
Copied!
            

switch (config) # show ptp PTP mode : Boundary Clock Message format : Mixed Acceptable Master Table : Enabled Domain : 127 Clock identity : 7c:fe:90:ff:fe:fa:21:88 GMC identity : 7c:fe:90:ff:fe:fa:21:88 Number of master ports : 1 Slave port interface : N/A   PTP enabled interfaces: ---------------------------------------------------- Port VLAN State Forced Master ---------------------------------------------------- Eth1/2 N/A MASTER yes

To configure Forced Master on a switch interface:

  1. Enable Forced Master on the VLAN interface. Run:

    Copy
    Copied!
                

    switch (config) # interface vlan 2 ptp enable forced-master

  2. Enable Forced Master on the interface. Run:

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 ptp enable forced-master

    Warning

    The interface should be a member in the PTP enabled VLAN(s).

To verify PTP configuration:

Copy
Copied!
            

switch (config) # show ptp PTP mode : Boundary Clock Message format : Mixed Acceptable Master Table : Enabled Domain : 127 Clock identity : 7c:fe:90:ff:fe:fa:21:88 GMC identity : 7c:fe:90:ff:fe:fa:21:88 Number of master ports : 2 Slave port interface : N/A   PTP enabled interfaces: ---------------------------------------------------- Port VLAN State Forced Master ---------------------------------------------------- Eth1/1 2 MASTER yes Eth1/1 3 SLAVE no

Warning

Forced Master is indicated as “yes” only if enabled on the interface and the corresponding VLAN interface.

For more information about this feature and its potential applications, please refer to the following:

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.