Linux tracing, monitoring and debugging

Applicable for STM32MP13x lines, STM32MP15x lines

1 Article purpose[edit]

This article provides useful information to start using Linux® tracing, monitoring and debugging environments.

Two entry points are proposed in this article:

  • Linux tracing, monitoring and debugging tools, which gives an overview of some Linux® tools including usage and application domain. This chapter is useful when you already know the domain or the interface to search for.
  • Trace and debug overview per Linux software frameworks, which points to articles explaining how to get trace and debug information about the Linux® software frameworks that are relevant for the STM32MPU Embedded Software. This chapter is useful when you know the Linux® framework to search for.

2 Linux trace architecture overview[edit]

The Linux® trace architecture can be organized into four levels as shown in the figure below (inspired by Brendan Gregg presentation[1]):

Linux kernel tracing architecture

2.1 Back-end instrumentation[edit]

The back-end instrumentation provides tracing sources built in the Linux® kernel. They are split into three categories:

  • tracepoints: kernel static tracing, statically placed at logical places in the kernel. It provides key event details as a "format" string.
  • kprobes: kernel dynamic tracing. It allows to trace function calls, returns and line numbers.
  • uprobes: dynamic user-level tracing.

2.2 Tracing framework[edit]

Also named tracers, they use tracing sources.

Tracing frameworks include kernel in-tree tracers such as ftrace and perf_events, and out-of-tree tracers such as SystemTap and sysdig.

2.3 Front-end tools[edit]

Front-end tools come on top of tracers and help to configure them. For example:

  • trace-cmd or LTTng for ftrace
  • perf or perf-Tools for perf_events

2.4 Add-on tools and viewer[edit]

Add-on tools are also on top of tracers. However, they are not embedded inside the Linux® kernel.

Viewer tools propose Visual interpretation of trace data. For example:

3 Linux tracing, monitoring and debugging tools[edit]

Linux® provides many tools that are either dedicated to one function or multifunction (generic).

They cover both Linux® kernel and Linux® user space.

3.1 Domain mapping[edit]

The following mapping, done by Brendan Gregg [4], shows the different existing tools associated to the different Linux® frameworks.

blktraceiostatperfftraceLTTngpidstatstraceltraceperfnetstatsysdigethtoolperfpidstattopperfmpstattcpdumpnetstatipsar/procFilesytem
Note: The above image has been created by Brendan Gregg (Netflix) and can be found on his official web site.


3.2 Tool overview[edit]

The following table provides a brief description of the tool, as well as its availability depending on the software packages:

Yes: this tool is either present (ready to use or to be activated), or can be integrated and activated on the software package.

No: this tool is not present and cannot be integrated, or it is present but cannot be activated on the software package.

Tool STM32MPU Embedded Software distribution STM32MPU Embedded Software distribution for Android™
Warning white.png Warning
STM32MPU Embedded Software distribution for Android™ is no more supported by ST. You can contact our ST partner, Witekio, who can help you to port and maintain it on STM32MP15 platform.
Name Category Purpose Starter Package Developer Package Distribution Package Starter Package Developer Package Distribution Package
blktrace Tracing tools blktrace[5] generates traces of the I/O traffic on block devices (SD card, USB, eMMC...) No Yes Yes No No Yes
systemd core dump Debugging tools systemd core dump: generates core dump files on Linux
Yes No Yes No No No
ethtool Monitoring tools ethtool[6] allows to query or control network driver and hardware settings, in particular for wired Ethernet devices. Yes Yes Yes No No No
ftrace Tracing tools ftrace[7] (Function Tracer) is a powerful kernel tracing utility that is able, for instance, to trace every kernel function calls and kernel events without adding any extra code in your kernel source code No No Yes No No Yes
GDB Debugging tools The GNU Project debugger, GDB[8], allows monitoring program execution, or what the program was doing at the moment it crashed.
No* Yes No** No No No
* Cross compile gdb and openocd binaries are required and only available from Developer Package.
** It is recommended to use the Developer Package to run the gdb debug session, which provided all dependencies
Warning white.png Warning

STM32MPU Embedded Software distribution for Android™ is no more supported by ST. You can contact our ST partner, Witekio, who can help you to port and maintain it on STM32MP15 platform.

ifconfig Monitoring tools ifconfig[9] is a system administration utility for network interface configuration.

ifconfig is deprecated and has been replaced by ip (A web page provides a comparison between ifconfig and ip [10])
Yes Yes Yes Yes Yes Yes
ip Monitoring tools ip[11] shows / manipulates routing, devices, policy routing and tunnels of network interfaces.

ip replaces the deprecated command ifconfig

Yes Yes Yes Yes Yes Yes
kmemleak Monitoring tools kmemleak[12] provides a means to detect possible kernel memory leaks in a similar way to a tracing garbage collector, with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak. No Yes Yes No No Yes
trace-cmd Tracing tools trace-cmd[13] command interacts with the Ftrace tracer that is built inside the Linux kernel. It interfaces with the Ftrace specific files found in the debugfs file system under the tracing directory.

kernelshark[14] is a front-end reader of trace-cmd output. "trace-cmd record" and "trace-cmd extract" create a trace.dat (trace-cmd.dat) file. kernelshark can read this file, and produce a graph and list view of the corresponding data.

No No Yes No No No
ltrace Tracing tools ltrace[15] is used to display the calls to shared libraries made by a userspace application. ltrace is a userspace application.

Its use is very similar to strace.

No No Yes No No No
LTTng Tracing tools LTTng[16] is an open source tracing framework for Linux kernel and user spaces. It is a powerful tool that can be used for many purposes. LTTng traces need to be processed/displayed with a host tool such as Trace Compass[17], based on Eclipse plugin for increased portability. No No Yes No No No
netdata Monitoring tools netdata[18] is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards. Yes Yes Yes No No No
netstat Monitoring tools netstat[19] prints network connections, routing tables, interface statistics, masquerade connections, and multicast membership information. Yes Yes Yes Yes Yes Yes
perf Monitoring tools perf[20] is a Linux user space tool, which allows getting system performance figures Yes Yes Yes No* No* No*
Note: simpleperf[21] is present as equivalent but with less options
strace Tracing tools strace[22] is able to intercept and record the system calls which are called by a process and the signals which are received by another process. Yes Yes Yes Yes Yes Yes
sysdig Monitoring tools sysdig[23] is a tool for system troubleshooting, analysis and exploration. It can be used to capture, filter and decode system calls and other OS events. No No Yes No No No
sysprof Monitoring tools sysprof[24] is a statistical, system-wide profiler for Linux. It helps in finding the functions in which a program spends most of its time.

sysprof proposes a user interface available directly on the board display screen.

Yes Yes Yes No No No
sysstat Monitoring tools The sysstat [25] tool suite contains utilities to monitor the system performance and usage activity.

It contains various utilities, common to many commercial Unix distributions, as well as tools that can be scheduled (via a scheduler such as cron) to collect and historize performance and activity data:

  • iostat: reports CPU statistics and input/output statistics for block devices and partitions.
  • mpstat: reports individual or combined processor related statistics.
  • pidstat: reports statistics for Linux tasks (processes): I/O, CPU, memory, etc.
  • sar: collects, reports and saves system activity information (CPU, memory, disks, interrupts, network interfaces, TTY, kernel tables,etc.)
  • sadf: displays data collected by sar in multiple formats (CSV, XML, JSON, etc.). This command can also be used to exchange data with other programs or to draw graphs illustrating the various activities collected by sar using SVG (Scalable Vector Graphics) format.
Yes Yes Yes No No No
tcpdump Monitoring tools tcpdump[26] is a common packet analyzer that runs under the command line. It allows the user to display TCP/IP and other packets being transmitted or received over a network to which the computer is connected. Yes Yes Yes Yes Yes Yes
top Monitoring tools The top[27] program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The types of system summary information shown and the types, order and size of information displayed for tasks are all user configurable and that configuration can be made persistent across restarts. (Extracted from man page[27]) Yes Yes Yes Yes Yes Yes
valgrind Monitoring tools valgrind[28] is an instrumentation framework for building dynamic analysis tools. Some Valgrind tools can automatically detect many memory management and threading bugs, and profile your programs in detail.

This is tool for Linux application analysis.

Yes Yes Yes No No No[29]

4 Trace and debug overview per Linux software frameworks[edit]

4.1 On STM32MP13x lines More info.png[edit]

The pictures below allow accessing to different Linux software frameworks which provide specific trace and debug information in their "How to trace and debug the framework" dedicated chapter.


Documentation/arm/stm32/overview.rstPWM overviewOverview of GPIO pinsInterrupt overviewDmaengine overviewWatchdog overviewLinux kernel device treeRTC overviewReset overviewClock overviewCAN overviewEthernet overviewWLAN overviewBluetooth overviewALSA overviewV4L2 camera overviewDRM KMS overviewSPI overviewI2C overviewSerial TTY overviewMTD overviewMMC overviewPower overviewRegulator overviewThermal overviewNVMEM overviewOP-TEE overviewCrypto API overviewHardware random overviewUSB overviewMMC overviewIIO overview
STM32MP13 Linux kernel overview


OpenSTLinux BSP legend.png


4.2 On STM32MP15x lines More info.png[edit]

ALSA overviewDocumentation/arm/stm32/overview.rstBluetooth overviewClock overviewCEC overviewResource manager for coprocessingremoteproc framework overviewMailbox framework overviewRPMsg framework overviewHardware spinlock overviewCoprocessor management overviewHardware random overviewCrypto API overviewOP-TEE overviewLinux kernel device treeDmaengine overviewDRM KMS overviewI2C overviewIIO overviewInterrupt overviewOverview of GPIO pinsMMC overviewMTD overviewNVMEM overviewCAN overviewEthernet overviewWLAN overviewBluetooth overviewPower overviewThermal overviewRegulator overviewReset overviewRTC overviewSerial TTY overviewSPI overviewPWM overviewUSB overviewMMC overviewV4L2 camera overviewWatchdog overview
STM32MP15 Linux kernel overview


OpenSTLinux BSP legend.png


5 Tips[edit]

How to find Linux kernel driver associated to a device.

How to use the kernel dynamic debug.

6 Documentation and web articles[edit]

A lot of articles on the web mention Linux® kernel tracing and profiling. The following links provide a good introduction to these topics:

Info white.png Information
More general Linux performance information are available in nice slideshare presentation (Brendan Gregg), on the Brendan Gregg official web site or in LinuxCon2014 article.

Reference list: