1. Purpose[edit source]
This article describes how to configure TSN (Time-Sensitive Networking) endpoint
This article provides different steps to execute TSN scenario endpoint:
- Need synchronization across the network. TSN requires precise time synchronization to ensure that time-critical data is delivered on time and in the correct order.
- IEEE 802.1Qav
- IEEE 802.1Qbv
2. IEEE 802.1AS-Rev or gPTP synchronization[edit source]
Testing using ptp4l tool from linuxptp project
To check the ptp clock adjustment with PTP protocol, a PTP with ordinary clock (client) and a PTP Grandmaster (server) applications are needed to run on separate devices. Open source application package linuxptp can be used.
Create file /etc/gPTP.cfg with content as below:
[global] gmCapable 1 priority1 248 priority2 248 logAnnounceInterval 0 logSyncInterval -3 syncReceiptTimeout 3 neighborPropDelayThresh 800 min_neighbor_prop_delay -20000000 assume_two_step 1 path_trace_enabled 1 follow_up_info 1 transportSpecific 0x1 ptp_dst_mac 01:80:C2:00:00:0E network_transport L2 delay_mechanism P2P
Hence TX timestamp generation can be delayed (especially with low speed links) the ptp4l “tx_timestamp_timeout” parameter need to be set for ptp4l to work. The default value for "tx_timestamp_timeout" is 5 seconds To change the value of "tx_timestamp_timeout", you can add or modify the following line in the configuration file:
tx_timestamp_timeout <timeout_value>
PTP synchro between two boards:
ptp4l -i end0 -p /dev/ptp0 -m -f /etc/gPTP.cfg
PPS (Pulse Per Second) is a signal that is commonly used to check the synchronization of a system clock with a reference clock, such as a PTP grandmaster clock. To check the synchronization of the system clock with the reference clock using PPS, you can use a tool such as ptp4l or phc2sys to capture the PPS signal and compare it with the PTP timestamp generated by the system clock. If the system clock is synchronized with the reference clock, the PTP timestamp should be aligned with the PPS signal. More details in Ethernet_device_tree_configuration#How_to_configure_PTP_PPS_out_-3F
3. TSN 802.1Qav[edit source]
An example configuration with 4 reserved bandwidth given below:
- Uses 4 Queues (Q0-Q3). Each Q has a bandwidth reserved in h/w. Maximum 4 Queues supported
- Queue 0 has No Bandwidth Restriction, general traffic
- Queue 1 has 10% of Maximum bandwidth reserved (100Mbps)
- Queue 2 has 20% of Maximum bandwidth reserved (200Mbps)
- Queue 3 has 30% of Maximum bandwidth reserved (300Mbps)
- Q3 is the highest priority Queue and Q0 is the lowest priority
Here are the steps to configure this setup.
Bring up end0 interface
ip link add link end0 name end0.200 type vlan id 200 ip link set end0.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 ifconfig end0.200 192.168.200.60 netmask 255.255.255.0
CBS is used to enforce a Quality of Service by limiting the data rate of a traffic class, to separate packets into traffic classes the user may choose mqprio(8) https://man7.org/linux/man-pages/man8/tc-mqprio.8.html, and configure it like this:
tc qdisc add dev end0 clsact tc qdisc add dev end0 handle 100: parent root mqprio \ num_tc 4 \ map 0 0 1 1 2 2 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0
Configure Bandwidth allocated for each queue (https://man7.org/linux/man-pages/man8/tc-cbs.8.html)
// Set bandwidth of queue 1 to be 100Mbps tc qdisc replace dev end0 parent 100:2 cbs \ idleslope 100000 \ sendslope -900000 \ hicredit 150 \ locredit -1350 \ offload 1
// Set bandwidth of queue 2 to be 200Mbps tc qdisc replace dev end0 parent 100:3 cbs \ idleslope 200000 \ sendslope -800000 \ hicredit 300 \ locredit -1200 \ offload 1
// Set bandwidth of queue 3 to be 300Mbps tc qdisc replace dev end0 parent 100:4 cbs \ idleslope 300000 \ sendslope -700000 \ hicredit 450 \ locredit -1050 \ offload 1
So for queue 0 the bandwidth should be around 400Mbps (port_transmit_rate - sum of all bandwidth of each queue)
sendslope = idleslope - port_transmit_rate
hicredit = max_interference_size * (idleslope / port_transmit_rate)
locredit = max_frame_size * (sendslope / port_transmit_rate)
with max_interference_size = max_frame_size = 1500
For more details: https://en.wikipedia.org/wiki/Time-Sensitive_Networking#IEEE_802.1Qav_Forwarding_and_queuing_enhancements_for_time-sensitive_streams
Network core and Driver uses the skb priority to deliver frames to specific h/w queues. In the above case, priority 6 to 15 SKB (packet) goes to Q3 (4th entry in map in the tc qdisc command), priority 4 to 5 SKB goes to Q2 (3rd entry in map) priority 2 to 3 SKB goes to Q1 (2nd entry in map) and priority 0 to 1 SKB goes to Q0 (1st entry in map)
Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to end0 with IP address 192.168.200.30
iperf3 -s -p 5003& iperf3 -s -p 5002& iperf3 -s -p 5001& iperf3 -s -p 5000&
At DUT, start transmission of stream using iperf3
//Stream with SKB priority 6 : TOS=0x14 == SO_PRIORITY=6 // https:man7.org/linux/man-pages/man8/tc-prio.8.html iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4 iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &
//Stream with SKB priority 2 iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort) iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60 &
Result without tc rule apply :
For Traffic with priority 6:
[ 5] 0.00-60.00 sec 1.40 GBytes 200 Mbits/sec 0.046 ms 11841/1031880 (1.1%) receiver
For Traffic with priority 4:
[ 5] 0.00-60.00 sec 40.1 MBytes 5.60 Mbits/sec 39.651 ms 936/29490 (3.2%) receiver
For Traffic with priority 2:
[ 5] 0.00-60.00 sec 4.94 GBytes 707 Mbits/sec 0.035 ms 164701/3768910 (4.4%) receiver
For Traffic with priority 0 (Best effort):
[ 5] 0.00-60.00 sec 106 MBytes 14.8 Mbits/sec 0.017 ms 1765/76956 (2.3%) receiver
Without CBS TSN rules in place, multiple traffic streams may contend for network resources, causing congestion and possible packet loss. This can lead to poor network performance and lower quality of service for users. Furthermore, without CBS TSN rules, there no prioritization of important traffic, such as real-time audio or video streams, which can cause delays and interruptions in these applications. In summary, the absence of CBS TSN rules can create an unstable and undependable network environment.
Result with tc rule apply:
For Queue 3 with priority 6 and with BW of 300Mbps:
[ 5] 0.00-60.00 sec 1.72 GBytes 246 Mbits/sec 0.052 ms 189322/1444777 (13%) receiver
For Queue 2 with priority 4 and with BW of 200Mbps:
[ 5] 0.00-60.00 sec 1.16 GBytes 167 Mbits/sec 0.064 ms 110342/959769 (11%) receiver
For Queue 1 with priority 2 and with BW of 100Mbps:
[ 5] 0.00-60.00 sec 641 MBytes 89.6 Mbits/sec 0.266 ms 32740/489287 (6.7%) receiver
For Queue0 with priority 0 (Best effort):
[ 5] 0.00-60.00 sec 2.31 GBytes 331 Mbits/sec 0.006 ms 234299/1921970 (12%) receiver
When TSN rules are implemented, the network can provide guaranteed bandwidth allocation and prioritization for different traffic streams. CBS TSN rules allow for the allocation of credits to different traffic streams, which can be used to transmit data over the network. This ensures that critical traffic, such as real-time audio or video streams, receive the necessary bandwidth and are not affected by other traffic streams. CBS TSN rules also prevent congestion and packet loss by limiting the amount of data that can be transmitted by each traffic stream. This results in a more predictable and reliable network environment, with improved quality of service for users.
4. TSN 802.1Qbv[edit source]
An example configuration with 4 schedule entries given below:
- Uses 4 Queues (Q0-Q3). Each Q has a Gate associated in h/w. Maximum 4 Queues/Gates supported
- 3 higher priority Gates open for 125usec (Q1, Q2 and Q3)
- Q3 is the higest priority Queue and Q0 is the lowest priority
- 1 lower priority (Q0) opens remaining gates for 250 usec
Here are the steps to configure this schedule.
Bring up end0 interface
ip link add link end0 name end0.200 type vlan id 200 ip link set end0.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 ifconfig end0.200 192.168.200.60 netmask 255.255.255.0
Setup EST schedule with 4 Gates (Q0-Q3). For description of Command parameters, see manual page for taprio. https://man7.org/linux/man-pages/man8/tc-taprio.8.html TC0 <-> Q0, TC1 <-> Q1, TC2 <-> Q2 and TC3 <-> Q3
tc qdisc replace dev end0 parent root handle 100 taprio \ num_tc 4 \ map 0 0 1 1 2 2 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 \ base-time 0000 \ sched-entry S 8 125000 \ sched-entry S 4 125000 \ sched-entry S 2 125000 \ sched-entry S 1 250000 \ sched-entry S 0 375000 \ flags 2
Where num_tc is same as number of queues = 4, maps 16 priorities to one of 4 TCs, queues specify the Queue associated with each TC, TC0 - One queue @0, TC1 - One queue @1, TC2 - One queue @2 and TC3 - One queue @3
sched-entry S 8 125000 S - SetGateStates operation 8 - Bit mask showing bit 4 set (Q3/TC3) 125000 - 125000 nsecs (125 usecs ) duration of Gate open The cycle-time is 1 msec (which is the sum of the intervals of individual schedule)
Enable classifier. Classifier is used to mark the packet based on packet meta data.
tc qdisc add dev end0 clsact
Network core and Driver uses the skb priority to deliver frames to specific h/w queues. In the above case, priority 6 to 15 SKB (packet) goes to Q3 (4th entry in map in the tc qdisc command), priority 4 to 5 SKB goes to Q2 (3rd entry in map) priority 2 to 3 SKB goes to Q1 (2nd entry in map) and priority 0 to 1 SKB goes to Q0 (1st entry in map)
Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to end0 with IP address 192.168.200.30
iperf3 -s -p 5003& iperf3 -s -p 5002& iperf3 -s -p 5001& iperf3 -s -p 5000&
At DUT, start transmission of stream using iperf3
//Stream with SKB priority 6 : TOS=0x14 == SO_PRIORITY=6 // https:man7.org/linux/man-pages/man8/tc-prio.8.html iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4 iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &
//Stream with SKB priority 2 iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort) iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60 &
Result without tc rule apply :
Result with tc rule apply:
5. References[edit source]