Last edited one month ago

How to configure TSN switch

Applicable for STM32MP25x lines

1. Purpose[edit source]

This article describes how to configure TSN (Time-Sensitive Networking) on switch
This article provides different steps to execute TSN scenario with switch:

  • Need synchronization across the network. TSN requires precise time synchronization to ensure that time-critical data is delivered on time and in the correct order.
  • IEEE 802.1Qav
  • IEEE 802.1Qbv

2. IEEE 802.1AS-Rev or gPTP synchronization[edit source]

Proprietary PTP stack used for gPTP synchronization, which is provided by TTTech

Deptp is started by default

Basic configuration of the PTP stack is done via configuration file, which has XML format. See /etc/deptp/ptp_config.xml for stack config This stack:

  • Start gPTP protocol
  • synchronize system time to PTP clock

PTP Management tool called deptp tool can be used to access PTP runtime data from Linux command line. Useful commands:

// Show offset from master
watch -n1 deptp_tool --get-current-dataset
 
// Show PTP state of sw0p2
deptp_tool -t1 --get-port-dataset

// Show PTP state of sw0p3
deptp_tool -t2 --get-port-dataset
Warning white.png Warning
Be careful if you are using PHY which managed “low power mode” (i.e EEE)

Most of the time this feature is not compatible with TSN. To be considered “asCapable” the maximum acceptable value for neighborPropDelayThresh is 800ns (default value), if you need to tune this value you can add in your Interface describe in XML ptp_config file: <delay_thresh>800</delay_thresh> and update value


3. IEEE 802.1Qav[edit source]

Maximum 4 Queues supported by GMAC and maximum 8 Queues supported by switch

An example configuration with 4 reserved bandwidth across Switch given below:

  • Uses 4 Queues (Q0-Q3). Each Q has a bandwidth reserved in h/w.
    • Queue 0 (Yellow stream) has No Bandwidth Restriction, general traffic (Prio 0)
    • Queue 1 (Blue stream) has 40% of Maximum bandwidth reserved (300Mbps) (Prio 2)
    • Queue 2 (Green stream) has 30% of Maximum bandwidth reserved (200Mbps) (Prio 4)
    • Queue 3 (Red stream) has 20% of Maximum bandwidth reserved (100Mbps) (Prio 6)

Q3 is the highest priority Queue and Q0 is the lowest priority

Alternate text
Streams across the switch

Here are the steps to configure this setup.

Warning white.png Warning
If you want to perform CBS test, gPTP synchronization is not needed



Bring up end1/sw0ep interface

ip link add link sw0ep name sw0ep.200 type vlan id 200
ip link set sw0ep.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
ifconfig sw0ep.200 192.168.200.60 netmask 255.255.255.0
bridge vlan add vid 200 dev sw0p1
bridge vlan add vid 200 dev sw0p2
bridge vlan add vid 200 dev sw0p3

CBS is used to enforce a Quality of Service by limiting the data rate of a traffic class, to separate packets into traffic classes the user may choose mqprio(8) https://man7.org/linux/man-pages/man8/tc-mqprio.8.html, and configure it like this:


Network core and Driver uses the skb priority to deliver frames to specific h/w queues. In the above case, priority 6 (packet) goes to Q3, priority 4 goes to Q2, priority 2 goes to Q1 and priority 0 SKB goes to Q0.

 ./set-cbs-idleslope.sh sw0p3 6 100000000
 + tsntool fqtss tsa 6 cbs sw0p3
 + tsntool fqtss slope 6 100000000 sw0p3
 + tsntool fqtss show sw0p3
 Traffic Class        Algorithm            operIdleSlope
       0              Strict prio          n/a
       1              Strict prio          n/a
       2              Strict prio          n/a
       3              Strict prio          n/a
       4              Strict prio          n/a
       5              Strict prio          n/a
       6              CBS                  100000000 bps
       7              Strict prio          n/a
 ./set-cbs-idleslope.sh sw0p3 4 200000000
 + tsntool fqtss tsa 4 cbs sw0p3
 + tsntool fqtss slope 4 200000000 sw0p3
 + tsntool fqtss show sw0p3
 Traffic Class        Algorithm            operIdleSlope
       0              Strict prio          n/a
       1              Strict prio          n/a
       2              Strict prio          n/a
       3              Strict prio          n/a
       4              CBS                  200000000 bps
       5              Strict prio          n/a
       6              CBS                  100000000 bps
       7              Strict prio          n/a
 ./set-cbs-idleslope.sh sw0p3 2 300000000
 + tsntool fqtss tsa 2 cbs sw0p3
 + tsntool fqtss slope 2 300000000 sw0p3
 + tsntool fqtss show sw0p3
 Traffic Class        Algorithm            operIdleSlope
       0              Strict prio          n/a
       1              Strict prio          n/a
       2              CBS                  300000000 bps
       3              Strict prio          n/a
       4              CBS                  200000000 bps
       5              Strict prio          n/a
       6              CBS                  100000000 bps
       7              Strict prio          n/a

With set-cbs-idleslope.sh :

#!/bin/bash

#implemented for tsntool 2.4.0


if [[ "$#" -ne 3 ]] ; then
    echo "Usage: $0 port traffic-class idleSlope"
    exit 1
fi
port=$1
trafficClass=$2
idleSlope=$3
set -x
tsntool fqtss tsa $trafficClass cbs $port
tsntool fqtss slope $trafficClass $idleSlope $port
tsntool fqtss show $port

Run 4 iperf sessions to generate several streams with different priorities. Remote PC or other device connected to sw0p3 (ETH1) with IP address 192.168.200.30, and other device/PC connected to sw0p2 (ETH3) to launch 2 x iperf3 client to generate streams.

Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to sw0p3 with IP address 192.168.200.30

iperf3 -s -p 5003&
iperf3 -s -p 5002&
iperf3 -s -p 5001&
iperf3 -s -p 5000&

Remote PC or other device connected to sw0p2 with IP address 192.168.200.40

//Stream with SKB priority 6  : TOS=0x14 == SO_PRIORITY=6
// https:man7.org/linux/man-pages/man8/tc-prio.8.html
iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4
iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &

At DUT, start transmission of stream using iperf3

//Stream with SKB priority 2
iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort)
iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60  &

Result without set-cbs-idleslope.sh rule apply :
For Traffic with priority 6:

[  5]   0.00-60.85  sec  2.65 GBytes   374 Mbits/sec  0.196 ms  437454/2371320 (18%)  receiver

For Traffic with priority 4:

[  5]   0.00-60.21  sec  2.61 GBytes   372 Mbits/sec  0.046 ms  421125/2321380 (18%)  receiver

For Traffic with priority 2:

[  5]   0.00-60.00  sec   518 MBytes  72.4 Mbits/sec  0.047 ms  1826344/2195179 (83%)  receiver

For Traffic with priority 0 (Best effort):

[  5]   0.00-60.00  sec   831 MBytes   116 Mbits/sec  0.013 ms  2355974/2948076 (80%)  receiver

Without CBS TSN rules in place, multiple traffic streams may contend for network resources, causing congestion and possible packet loss. This can lead to poor network performance and lower quality of service for users. Furthermore, without CBS TSN rules, there no prioritization of important traffic, such as real-time audio or video streams, which can cause delays and interruptions in these applications. In summary, the absence of CBS TSN rules can create an unstable and undependable network environment.


Result with set-cbs-idleslope.sh rule apply:
For Queue 3 with priority 6 and with BW of 100Mbps:

[  5]   0.00-56.00  sec   647 MBytes  97.0 Mbits/sec  0.108 ms  3766160/4227385 (89%)  receiver

For Queue 2 with priority 4 and with BW of 200Mbps:

[  5]   0.00-60.00  sec  1.35 GBytes   193 Mbits/sec  0.066 ms  3438066/4422599 (78%)  receiver

For Queue 1 with priority 2 and with BW of 300Mbps:

[  5]   0.00-60.00  sec  2.02 GBytes   290 Mbits/sec  0.041 ms  2994134/4470767 (67%)  receiver

For Queue0 with priority 0 (Best effort):

[  5]   0.00-60.00  sec  1.11 GBytes   131 Mbits/sec  0.006 ms  134299/1921970 (12%)  receiver

When TSN rules are implemented, the network can provide guaranteed bandwidth allocation and prioritization for different traffic streams. CBS TSN rules allow for the allocation of credits to different traffic streams, which can be used to transmit data over the network. This ensures that critical traffic, such as real-time audio or video streams, receive the necessary bandwidth and are not affected by other traffic streams. CBS TSN rules also prevent congestion and packet loss by limiting the amount of data that can be transmitted by each traffic stream. This results in a more predictable and reliable network environment, with improved quality of service for users.

4. IEEE 802.1Qbv[edit source]

An example configuration with 4 streams across Switch given below: (same topology as described in previous chapter)

  • Uses 4 Queues (Q0-Q3). Each Q has a schedule reserved in h/w.
    • Queue 0 (Yellow stream) considered best effort which have gate open for 250usec (Prio 0)
    • Queue 1 (Blue stream) which have gate open for 125usec (Prio 2)
    • Queue 2 (Green stream) which have gate open for 125usec (Prio 4)
    • Queue 3 (Red stream) which have gate open for 125usec (Prio 6)

Q3 is the highest priority Queue and Q0 is the lowest priority

Here are the steps to configure this schedule.

Warning white.png Warning
Be careful if you want to perform QBV test, you need to start gPTP synchronization before, for switch it is deptp and it is started by default



Bring up end1/sw0ep interface

ip link add link sw0ep name sw0ep.200 type vlan id 200
ip link set sw0ep.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
ifconfig sw0ep.200 192.168.200.60 netmask 255.255.255.0
bridge vlan add vid 200 dev sw0p1
bridge vlan add vid 200 dev sw0p2
bridge vlan add vid 200 dev sw0p3

Example Qbv configuration: sgs is used for the configuration of Qbv. Save the following configuration to qbv.cfg.

with qbv.cfg:

sgs 125000 0x40
sgs 125000 0x10
sgs 125000 0x4
sgs 250000 0x1
sgs 250000 0
sgs 125000 0 

This configuration sets

  • set gate for traffic classes 6 for 125000 nanoseconds
  • set gate for traffic classes 4 for 125000 nanoseconds
  • set gate for traffic classes 2 for 125000 nanoseconds
  • set gate for traffic classes 0 for 250000 nanoseconds
  • all gates close for 375000 nanoseconds

Once the schedule is created, it should be written to the AdminControlList , before loading it to the OperCon- trolList. For writing it to the AdminControlList, call the tool tsntool as follows:

 tsntool st wrcl sw0p3 qbv.cfg

To check that the configuration was written to the AdminControlList, the AdminControlList can be read as follows: tsntool st rdacl <interface> Example of printing to the terminal:

tsntool st rdacl sw0p3
sgs  125000  0x40
sgs  125000  0x10
sgs  125000  0x04
sgs  250000  0x01
sgs  250000  0x00
sgs  125000  0x00

To load this schedule to the OperControlList so that it gets taken into use, call this command:
tsntool st configure <basetime> <cycletime> <cycletime-extension> <interface>
The basetime is formatted as <seconds>.<nanoseconds>, as the current time of the switch. Note: The current time of the switch can be read as follows:
cat /sys/class/net/sw0p3/ieee8021ST/CurrentTime
The parameter cycletime is formatted as <numerator>/<denominator> in seconds.
The cycletime-extension is formatted as plain nanoseconds. Note: The cycletime-extension parameter has to be specified for legacy reasons. It is ignored by the driver.
An alternative is to use a relative time to apply the configuration. It’s in the same format, but with a ’+’ or a ’-’ just before.

 tsntool st configure 0.0 1/1000 0 sw0p3

A cycle time of 1/1000 seconds = 1 millisecond is a lot longer than the time needed to complete one schedule run-through. After the schedule has finished within a cycle, the gates remain in the last configured state until the end of the cycle.
After calling the above commands, all the necessary parameters are set to have a basic Qbv configuration up and running. It can be confirmed by running a command:

tsntool st show sw0p3
GatesEnabled:            YES
ConfigPending:           NO
ConfigChangeError:       1

AdminCycleTime:          1/1000 sec
AdminCycleTimeExtension: 0 nsec
AdminControlListLength:  6 entries
OperCycleTime:           1/1000 sec
OperCycleTimeExtension:  0 nsec
OperControlListLength:   6 entries

CurrentTime (TAI):       2023-03-03 10:41:46.737801550 (1677840106.737801550)
AdminBaseTime (TAI):     1970-01-01 00:00:00.000000000
OperBaseTime (TAI):      1970-01-01 00:00:00.000000000
ConfigChangeTime (TAI):  2023-03-03 10:00:04.278000000 

SupportedCycleMax:       335211525/1000000000 sec
SupportedIntervalMax:    327675 nsec
SupportedListMax:        1023 entries
TickGranularity:         100000 1/10 nsec

portNumTrafficClasses:   8

queueMaxSDUTable:
TrafficClass   queueMaxSDU   TransmissionOverruns
 0                   1506   6961
 1                   1506   6961
 2                   1506   6961
 3                   1506   6961
 4                   1506   6961
 5                   1506   6961
 6                   1506   6961
 7                   1506   6961

Run 4 iperf sessions to generate several streams with different priorities. Remote PC or other device connected to sw0p3 (ETH1) with IP address 192.168.200.30, and other device/PC connected to sw0p2 (ETH3) to launch 2 x iperf3 client to generate streams.

Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to sw0p3 with IP address 192.168.200.30

iperf3 -s -p 5003&
iperf3 -s -p 5002&
iperf3 -s -p 5001&
iperf3 -s -p 5000&

Remote PC or other device connected to sw0p2 with IP address 192.168.200.40

//Stream with SKB priority 6  : TOS=0x14 == SO_PRIORITY=6
// https:man7.org/linux/man-pages/man8/tc-prio.8.html
iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4
iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &

At DUT, start transmission of stream using iperf3

//Stream with SKB priority 2
iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort)
iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60  &

Result without tsntool rule apply :

Alternate text
schedule without QBV rule: it's anarchy

Result with tsntool rule apply:

Alternate text
schedule with QBV rule: each traffic has its time frame

5. References[edit source]