1. Article purpose
This article provides the basic information needed to start using the sysstat[1] tool suite providing a collection of Linux tools:
- main tools: iostat, mpstat and pidstat
- others tools: sar, sadf
2. Introduction
The following table provides a brief description of the tool, as well as its availability depending on the software packages:
: this tool is either present (ready to use or to be activated), or can be integrated and activated on the software package.
: this tool is not present and cannot be integrated, or it is present but cannot be activated on the software package.
| Tool | STM32MPU Embedded Software distribution | STM32MPU Embedded Software distribution for Android™ | ||||||
|---|---|---|---|---|---|---|---|---|
| Name | Category | Purpose | Starter Package | Developer Package | Distribution Package | Starter Package | Developer Package | Distribution Package |
| sysstat | Monitoring tools | The sysstat [1] tool suite contains utilities to monitor the system performance and usage activity. It contains various utilities, common to many commercial Unix distributions, as well as tools that can be scheduled (via a scheduler such as cron) to collect and historize performance and activity data:
|
|
|
|
|
|
|
3. Installing the trace and debug tool on the target board
3.1. Using the STM32MPU Embedded Software distribution
The Sysstat tools are installed by default (/usr/bin/) and ready to be used with all STM32MPU OpenSTLinux Packages.
It is integrated in weston image distribution through meta-st package: meta-st/meta-st-openstlinux/recipes-st/packagegroups/packagegroup-framework-tools.bb.
RDEPENDS:packagegroup-framework-tools-core = "\ grep \ ... e2fsprogs \ e2fsprogs-resize2fs \ sysstat \ minicom \ ... "
4. Getting started
4.1. Using iostat
The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.[2]
/proc filesystem must be mounted for iostat to work. This is the case on STM32MPU Embedded Software.
- Command line options. Please refer to manual page[2] for details.
iostat --help
Usage: iostat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -s ] [ -t ] [ -V ] [ -x ] [ -y ] [ -z ]
[ { -f | +f } <directory> ] [ -j { ID | LABEL | PATH | UUID | ... } ]
[ --compact ] [ --dec={ 0 | 1 | 2 } ] [ --human ] [ --pretty ] [ -o JSON ]
[ [ -H ] -g <group_name> ] [ -p [ <device> [,...] | ALL ] ]
[ <device> [...] | ALL ]
- Basic command to display a single history since boot report for all CPU and devices:
iostat
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.11 0.00 0.60 0.05 0.00 99.25
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
mmcblk0 0.59 5.44 0.85 58539 9128
mmcblk1 0.03 0.31 0.00 3320 0
mmcblk1boot1 0.00 0.01 0.00 104 0
mmcblk1boot0 0.00 0.01 0.00 104 0
Note: by adding <interval> and <count> parameters, you can have a regular refresh during a given time interval:
iostat <interval> <count>
- Complex command to display statistics for block devices and all their partitions that are used by the system:
iostat -N -p ALL
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.11 0.00 0.61 0.05 0.00 99.23
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
ram0 0.00 0.00 0.00 0 0
ram1 0.00 0.00 0.00 0 0
ram2 0.00 0.00 0.00 0 0
ram3 0.00 0.00 0.00 0 0
ram4 0.00 0.00 0.00 0 0
ram5 0.00 0.00 0.00 0 0
ram6 0.00 0.00 0.00 0 0
ram7 0.00 0.00 0.00 0 0
ram8 0.00 0.00 0.00 0 0
ram9 0.00 0.00 0.00 0 0
ram10 0.00 0.00 0.00 0 0
ram11 0.00 0.00 0.00 0 0
ram12 0.00 0.00 0.00 0 0
ram13 0.00 0.00 0.00 0 0
ram14 0.00 0.00 0.00 0 0
ram15 0.00 0.00 0.00 0 0
loop0 0.00 0.00 0.00 0 0
loop1 0.00 0.00 0.00 0 0
loop2 0.00 0.00 0.00 0 0
loop3 0.00 0.00 0.00 0 0
loop4 0.00 0.00 0.00 0 0
loop5 0.00 0.00 0.00 0 0
loop6 0.00 0.00 0.00 0 0
loop7 0.00 0.00 0.00 0 0
mtdblock0 0.00 0.00 0.00 0 0
mtdblock1 0.00 0.00 0.00 0 0
mtdblock2 0.00 0.00 0.00 0 0
mtdblock3 0.00 0.00 0.00 0 0
mtdblock4 0.00 0.00 0.00 0 0
mtdblock5 0.00 0.00 0.00 0 0
mtdblock6 0.00 0.00 0.00 0 0
mtdblock7 0.00 0.00 0.00 0 0
mmcblk0 0.66 6.02 0.94 58539 9128
mmcblk0p1 0.00 0.04 0.00 344 0
mmcblk0p2 0.00 0.02 0.00 220 0
mmcblk0p3 0.00 0.01 0.00 72 0
mmcblk0p4 0.03 0.21 0.00 2065 1
mmcblk0p5 0.60 5.46 0.94 53175 9126
mmcblk0p6 0.02 0.21 0.00 2087 1
mmcblk1 0.04 0.34 0.00 3320 0
mmcblk1p1 0.00 0.01 0.00 64 0
mmcblk1p2 0.00 0.00 0.00 44 0
mmcblk1p3 0.00 0.00 0.00 44 0
mmcblk1p4 0.01 0.11 0.00 1032 0
mmcblk1p5 0.01 0.11 0.00 1032 0
mmcblk1boot1 0.00 0.01 0.00 104 0
mmcblk1boot0 0.00 0.01 0.00 104 0
4.2. Using mpstat
The mpstat command writes to standard output activities for each available processor, processor 0 being the first one.[3]
- Command line options. Please refer to manual page[3] for details.
mpstat --help
Usage: mpstat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -A ] [ -H ] [ -n ] [ -T ] [ -u ] [ -V ]
[ -I { SUM | CPU | SCPU | ALL } ] [ -N { <node_list> | ALL } ]
[ --dec={ 0 | 1 | 2 } ] [ -o JSON ] [ -P { <cpu_list> | ALL } ]
- Basic command to display three reports of global statistics for all processors at one second intervals:
mpstat 1 3
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
15:45:26 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
15:45:27 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:45:28 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:45:29 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
- Other command to display three reports of statistics for all processors at one second intervals:
mpstat -P ALL 1 3
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
15:48:32 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
15:48:33 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:48:33 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:48:33 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:48:33 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
15:48:34 all 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 99.50
15:48:34 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:48:34 1 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 99.01
15:48:34 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
15:48:35 all 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 99.50
15:48:35 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
15:48:35 1 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 99.01
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 99.67
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: 1 0.00 0.00 0.66 0.00 0.00 0.00 0.00 0.00 0.00 99.34
4.3. Using pidstat
The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel.[4]
- Command line options. Please refer to manual page[4] for details.
pidstat --help
Usage: pidstat [ options ] [ <interval> [ <count> ] ] [ -e <program> <args> ]
Options are:
[ -d ] [ -H ] [ -h ] [ -I ] [ -l ] [ -R ] [ -r ] [ -s ] [ -t ] [ -U [ <username> ] ]
[ -u ] [ -V ] [ -v ] [ -w ] [ -C <command> ] [ -G <process_name> ]
[ -p { <pid> [,...] | SELF | ALL } ] [ -T { TASK | CHILD | ALL } ]
[ --dec={ 0 | 1 | 2 } ] [ --human ]
- Basic command to display CPU statistic for all tasks:
pidstat
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
15:58:48 UID PID %usr %system %guest %wait %CPU CPU Command
15:58:48 0 1 0.03 0.12 0.00 0.05 0.14 0 systemd
15:58:48 0 2 0.00 0.00 0.00 0.00 0.00 1 kthreadd
15:58:48 0 7 0.00 0.01 0.00 0.01 0.01 0 ksoftirqd/0
15:58:48 0 8 0.04 0.00 0.00 0.02 0.04 1 rcu_preempt
15:58:48 0 15 0.00 0.01 0.00 0.00 0.01 1 ksoftirqd/1
15:58:48 0 18 0.00 0.00 0.00 0.00 0.00 1 kdevtmpfs
15:58:48 0 19 0.00 0.00 0.00 0.00 0.00 0 kworker/0:1
15:58:48 0 72 0.00 0.00 0.00 0.00 0.00 0 irq/82-mfx
15:58:48 0 77 0.00 0.06 0.00 0.01 0.06 1 mmcqd/0
15:58:48 0 83 0.00 0.00 0.00 0.00 0.00 0 mmcqd/1
15:58:48 0 85 0.00 0.00 0.00 0.00 0.00 0 mmcqd/1boot0
15:58:48 0 86 0.00 0.00 0.00 0.00 0.00 0 mmcqd/1boot1
15:58:48 0 91 0.00 0.63 0.00 0.00 0.63 0 kmemleak
15:58:48 0 93 0.00 0.00 0.00 0.00 0.00 0 kworker/0:3
15:58:48 0 95 0.00 0.01 0.00 0.00 0.01 1 irq/73-5a001000
15:58:48 0 98 0.00 0.00 0.00 0.00 0.00 0 kworker/0:1H
15:58:48 0 99 0.00 0.00 0.00 0.00 0.00 1 kworker/1:1H
15:58:48 0 100 0.00 0.00 0.00 0.00 0.00 0 jbd2/mmcblk0p5-
15:58:48 0 150 0.01 0.02 0.00 0.02 0.02 0 systemd-journal
15:58:48 0 189 0.01 0.01 0.00 0.03 0.01 1 systemd-udevd
15:58:48 0 201 0.00 0.00 0.00 0.00 0.00 0 kworker/u4:2
15:58:48 993 203 0.01 0.09 0.00 0.00 0.11 1 systemd-timesyn
15:58:48 998 273 0.09 0.04 0.00 0.02 0.14 0 avahi-daemon
15:58:48 0 276 0.00 0.00 0.00 0.00 0.00 1 systemd-logind
15:58:48 0 277 0.00 0.00 0.00 0.00 0.00 1 klogd
15:58:48 996 279 0.00 0.00 0.00 0.00 0.00 0 dbus-daemon
15:58:48 0 301 0.00 0.00 0.00 0.00 0.00 1 syslogd
15:58:48 992 312 0.00 0.00 0.00 0.00 0.00 1 systemd-network
15:58:48 0 314 0.00 0.00 0.00 0.00 0.00 0 weston-launch
15:58:48 0 319 0.01 0.01 0.00 0.00 0.01 1 weston
15:58:48 991 324 0.00 0.00 0.00 0.00 0.00 0 systemd-resolve
15:58:48 0 327 0.03 0.03 0.00 0.00 0.05 0 tcf-agent
15:58:48 0 332 0.00 0.00 0.00 0.00 0.00 1 login
15:58:48 0 333 0.00 0.00 0.00 0.00 0.00 0 agetty
15:58:48 997 334 0.00 0.00 0.00 0.00 0.00 1 named
15:58:48 0 340 0.00 0.00 0.00 0.00 0.00 0 weston-keyboard
15:58:48 0 342 0.01 0.00 0.00 0.00 0.01 0 weston-desktop-
15:58:48 0 345 0.00 0.01 0.00 0.00 0.01 1 systemd
15:58:48 0 350 0.00 0.00 0.00 0.00 0.00 0 sh
15:58:48 0 374 0.02 0.00 0.00 0.00 0.02 1 kworker/1:1
15:58:48 0 420 0.00 0.00 0.00 0.00 0.00 1 kworker/1:0
15:58:48 0 423 0.00 0.00 0.00 0.00 0.00 1 kworker/1:2
15:58:48 0 424 0.00 0.00 0.00 0.00 0.00 1 pidstat
- Basic command to display 3 reports of CPU statistics for every active task in the system at one second intervals.:
pidstat 1 3
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
15:59:56 UID PID %usr %system %guest %wait %CPU CPU Command
15:59:57 0 426 3.67 4.59 0.00 0.00 8.26 1 pidstat
15:59:57 UID PID %usr %system %guest %wait %CPU CPU Command
15:59:58 0 426 0.99 6.93 0.00 0.00 7.92 1 pidstat
15:59:58 UID PID %usr %system %guest %wait %CPU CPU Command
15:59:59 0 15 0.00 0.99 0.00 0.00 0.99 1 ksoftirqd/1
15:59:59 0 426 1.98 5.94 0.00 0.99 7.92 1 pidstat
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 0 15 0.00 0.32 0.00 0.00 0.32 - ksoftirqd/1
Average: 0 426 2.25 5.79 0.00 0.32 8.04 - pidstat
- Basic command to report statistics for a specific process with all associated threads:
pidstat -t -p `pidof weston-launch`
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
16:00:59 UID TGID TID %usr %system %guest %wait %CPU CPU Command
16:00:59 0 314 - 0.00 0.00 0.00 0.00 0.00 0 weston-launch
16:00:59 0 - 314 0.00 0.00 0.00 0.00 0.00 0 |__weston-launch
4.4. Using sar
The sar command collects, reports, or saves system activity information.[5]
- Command line options. Please refer to manual page[5] for details.
sar --help
Usage: sar [ options ] [ <interval> [ <count> ] ]
Options are:
[ -A ] [ -B ] [ -b ] [ -C ] [ -D ] [ -d ] [ -F [ MOUNT ] ] [ -H ] [ -h ]
[ -p ] [ -r [ ALL ] ] [ -S ] [ -t ] [ -u [ ALL ] ] [ -V ]
[ -v ] [ -W ] [ -w ] [ -x ] [ -y ] [ -z ]
[ -I [ SUM | ALL ] ] [ -P { <cpu_list> | ALL } ]
[ -m { <keyword> [,...] | ALL } ] [ -n { <keyword> [,...] | ALL } ]
[ -q [ <keyword> [,...] | ALL ] ]
[ --dev=<dev_list> ] [ --fs=<fs_list> ] [ --iface=<iface_list> ] [ --int=<int_list> ]
[ --dec={ 0 | 1 | 2 } ] [ --help ] [ --human ] [ --pretty ] [ --sadc ]
[ -j { SID | ID | LABEL | PATH | UUID | ... } ]
[ -f [ <filename> ] | -o [ <filename> ] | -[0-9]+ ]
[ -i <interval> ] [ -s [ <start_time> ] ] [ -e [ <end_time> ] ]
Main used options for activity report:
| sar option | Description |
|---|---|
| -u | CPU utilization of ALL CPUs |
| -P | CPU utilization of individual CPU or core |
| -r | Statistics for memory utilization |
| -S | Statistics for swap space utilization |
| -b | Statistics for I/O and transfer rate |
| -F | Statistics for currently mounted filesystems |
| -d | Activities for individual Block Device I/O |
| -w | Activities for task creation and system switching |
| -q | Information about task run queue and load average |
| -n | Statistics for network activities |
| -B | Report paging statistics |
4.4.1. Interactive mode
Two modes for using sar:
- Interactive mode (detailed here): sar is launched by the user using a command line on the console
- Non-interactive mode (detailed in To go further chapter): sar is launched by a script (/usr/libexec/sa/sa1, which is part of sysstat tools suite) that can be managed by a scheduler daemon like cron[6]
Interactive mode is used as soon as [ <interval> [ <count> ] ] parameters are present, otherwise non-interactive mode is used. In interactive mode, the command is executed immediately by taking into account the values.
As example, for getting CPU utilization of ALL CPUs, 5 times every 3s:
sar -u 3 5
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
12:30:20 CPU %user %nice %system %iowait %steal %idle
12:30:23 all 0.00 0.00 0.50 0.00 0.00 99.50
12:30:26 all 0.00 0.00 0.50 0.00 0.00 99.50
12:30:29 all 0.17 0.00 0.17 0.00 0.00 99.67
12:30:32 all 0.66 0.00 2.49 0.00 0.00 96.84
12:30:35 all 0.33 0.00 3.16 0.00 0.00 96.51
Average: all 0.23 0.00 1.36 0.00 0.00 98.41
-o <filename> can be used to store information into a file in binary format, which cat be later read with sadf.
4.5. Using sadf
The sadf command is used for displaying the contents of data files created by the sar command[7]: sar in interactive mode by using -o option, or file created in non-interactive mode in /var/log/sa/ directory.
- Command line options. Please refer to manual page[7] for details.
sadf --help
Usage: sadf [ options ] [ <interval> [ <count> ] ] [ <datafile> | -[0-9]+ ]
Options are:
[ -C ] [ -c | -d | -g | -j | -l | -p | -r | -x ] [ -H ] [ -h ] [ -T | -t | -U ] [ -V ]
[ -O <opts> [,...] ] [ -P { <cpu> [,...] | ALL } ]
[ --dev=<dev_list> ] [ --fs=<fs_list> ] [ --iface=<iface_list> ] [ --int=<int_list> ]
[ -s [ <start_time> ] ] [ -e [ <end_time> ] ]
[ -- <sar_options> ]
Main used format for activity report:
| sadf option | Description |
|---|---|
| -d | Print the contents of the data file in a format that can be easily ingested by a relational database system. The output consists of fields separated by a semicolon (i.e. MSExcel) |
| -g | From v11.3.1 Print the contents of the data file in SVG (Scalable Vector Graphics) format. Use the follwing syntax: sadf -g sar_datafile [ -- <sar_options> ] > output.svg |
| -j | Print the contents of the data file in JSON (JavaScript Object Notation) format |
| -t | Display timestamp in the original local time of the data file creator instead of UTC (Coordinated Universal Time) |
| -x | Print the contents of the data file in XML format |
5. To go further
5.1. Using sar non-interactive mode
In non-interactive mode a scheduler daemon is used to plan activity record.
Recommended environment is using cron[6] daemon and sysstat script /usr/libexec/sa/sa1 present when the tool suite is installed. Please refer to sa1 man page[8] for detail.
cron is not installed by default, and is part of cronie yocto package to be integrated:
- With the Distribution Package of the STM32MPU Embedded Software distribution (st-image-weston):
echo 'IMAGE_INSTALL += "cronie"' >> meta-st/meta-st-openstlinux/recipes-st/images/st-image-weston.bbappend
5.1. Format datas to be imported in MS-Excel
Example using sar command to get ALL CPUs usage, format datas into semicolumn text, then make a graph using MS-Excel:
- Record CPUs usage data (10 iterations every 3-second interval)
sar -u 3 10 -o test
Linux 4.14.48 (stm32mp1) 06/18/18 _armv7l_ (2 CPU)
13:03:26 CPU %user %nice %system %iowait %steal %idle
13:03:29 all 4.82 0.00 3.16 0.66 0.00 91.36
13:03:32 all 3.82 0.00 3.49 0.00 0.00 92.69
13:03:35 all 18.77 0.00 7.81 5.81 0.00 67.61
13:03:38 all 11.96 0.00 4.98 0.00 0.00 83.06
13:03:41 all 36.21 0.00 8.97 0.00 0.00 54.82
13:03:44 all 1.50 0.00 1.66 0.83 0.00 96.01
13:03:47 all 1.00 0.00 1.83 1.33 0.00 95.85
13:03:50 all 9.63 0.00 4.82 0.00 0.00 85.55
13:03:53 all 3.49 0.00 5.32 0.00 0.00 91.20
13:03:56 all 4.15 0.00 4.98 4.65 0.00 86.21
Average: all 9.53 0.00 4.70 1.33 0.00 84.44
File test is also created (binary format).
- Create readable report format from test file, i.e. in semicolumn format, which can be imported in MSExcel graph:
sadf -d test > test.cvs
cat test.cvs
# hostname;interval;timestamp;CPU;%user;%nice;%system;%iowait;%steal;%idle
stm32mp1;3;2018-06-18 13:03:29 UTC;-1;4.82;0.00;3.16;0.66;0.00;91.36
stm32mp1;3;2018-06-18 13:03:32 UTC;-1;3.82;0.00;3.49;0.00;0.00;92.69
stm32mp1;3;2018-06-18 13:03:35 UTC;-1;18.77;0.00;7.81;5.81;0.00;67.61
stm32mp1;3;2018-06-18 13:03:38 UTC;-1;11.96;0.00;4.98;0.00;0.00;83.06
stm32mp1;3;2018-06-18 13:03:41 UTC;-1;36.21;0.00;8.97;0.00;0.00;54.82
stm32mp1;3;2018-06-18 13:03:44 UTC;-1;1.50;0.00;1.66;0.83;0.00;96.01
stm32mp1;3;2018-06-18 13:03:47 UTC;-1;1.00;0.00;1.83;1.33;0.00;95.85
stm32mp1;3;2018-06-18 13:03:50 UTC;-1;9.63;0.00;4.82;0.00;0.00;85.55
stm32mp1;3;2018-06-18 13:03:53 UTC;-1;3.49;0.00;5.32;0.00;0.00;91.20
stm32mp1;3;2018-06-18 13:03:56 UTC;-1;4.15;0.00;4.98;4.65;0.00;86.21
6. References
- ↑ Jump up to: 1.0 1.1 https://sysstat.github.io/
- ↑ Jump up to: 2.0 2.1 https://man7.org/linux/man-pages/man1/iostat.1.html
- ↑ Jump up to: 3.0 3.1 https://man7.org/linux/man-pages/man1/mpstat.1.html
- ↑ Jump up to: 4.0 4.1 https://man7.org/linux/man-pages/man1/pidstat.1.html
- ↑ Jump up to: 5.0 5.1 https://man7.org/linux/man-pages/man1/sar.1.html
- ↑ Jump up to: 6.0 6.1 https://en.wikipedia.org/wiki/Cron
- ↑ Jump up to: 7.0 7.1 https://man7.org/linux/man-pages/man1/sadf.1.html
- ↑ https://www.man7.org/linux/man-pages/man8/sa1.8.html