Linux tracing, monitoring and debugging

1 Article purpose[edit]

This article provides useful information to start using Linux® tracing, monitoring and debugging environments.

Two entry points are proposed in this article:

  • Linux tracing, monitoring and debugging tools, which gives an overview of some Linux® tools including usage and application domain. This chapter is useful when you already know the domain or the interface to search for.
  • Trace and debug overview per Linux software frameworks, which points to articles explaining how to get trace and debug information about the Linux® software frameworks that are relevant for the STM32MPU Embedded Software. This chapter is useful when you know the Linux® framework to search for.

2 Linux trace architecture overview[edit]

The Linux® trace architecture can be organized into four levels as shown in the figure below (inspired by Brendan Gregg presentation[1]):

Linux kernel tracing architecture

2.1 Back-end instrumentation[edit]

The back-end instrumentation provides tracing sources built in the Linux® kernel. They are split into three categories:

  • tracepoints: kernel static tracing, statically placed at logical places in the kernel. It provides key event details as a "format" string.
  • kprobes: kernel dynamic tracing. It allows to trace function calls, returns and line numbers.
  • uprobes: dynamic user-level tracing.

2.2 Tracing framework[edit]

Also named tracers, they use tracing sources.

Tracing frameworks include kernel in-tree tracers such as ftrace and perf_events, and out-of-tree tracers such as SystemTap and sysdig.

2.3 Front-end tools[edit]

Front-end tools come on top of tracers and help to configure them. For example:

  • trace-cmd or LTTng for ftrace
  • perf or perf-Tools for perf_events

2.4 Add-on tools and viewer[edit]

Add-on tools are also on top of tracers. However, they are not embedded inside the Linux® kernel.

Viewer tools propose Visual interpretation of trace data. For example:

3 Linux tracing, monitoring and debugging tools[edit]

Linux® provides many tools that are either dedicated to one function or multifunction (generic).

They cover both Linux® kernel and Linux® user space.

3.1 Domain mapping[edit]

The following mapping, done by Brendan Gregg [4], shows the different existing tools associated to the different Linux® frameworks.

Note: The above image has been created by Brendan Gregg (Netflix) and can be found on his official web site.

3.2 Tool overview[edit]

The following table provides a brief description of the tool, as well as its availability depending on the software packages:

Yes: this tool is either present (ready to use or to be activated), or can be integrated and activated on the software package.

No: this tool is not present and cannot be integrated, or it is present but cannot be activated on the software package.

Tool STM32MPU Embedded Software distribution STM32MPU Embedded Software distribution for Android™
Name Category Purpose Starter Package Developer Package Distribution Package Starter Package Developer Package Distribution Package
blktrace Tracing tools blktrace[5] generates traces of the I/O traffic on block devices (SD card, USB, eMMC...) No Yes Yes
Under construction.png Coming soon
systemd core dump Debugging tools systemd core dump: generates core dump files on Linux
Yes No Yes No No No
ethtool Monitoring tools ethtool[6] allows to query or control network driver and hardware settings, in particular for wired Ethernet devices. Yes Yes Yes
Under construction.png Coming soon
ftrace Tracing tools ftrace[7] (Function Tracer) is a powerful kernel tracing utility that is able, for instance, to trace every kernel function calls and kernel events without adding any extra code in your kernel source code No No Yes
Under construction.png Coming soon
GDB Debugging tools The GNU Project debugger, GDB[8], allows monitoring program execution, or what the program was doing at the moment it crashed.
No* Yes No**
Under construction.png Coming soon
* Cross compile gdb binary is required and only available from Developer Package.

** It is recommended to use the Developer Package to run the gdb debug session, which provided all dependencies

ifconfig Monitoring tools ifconfig[9] is a system administration utility for network interface configuration.
Warning.png ifconfig is deprecated and has been replaced by ip.

A web page provides a comparison between ifconfig and ip [10]

Yes Yes Yes
Under construction.png Coming soon
ip Monitoring tools ip[11] shows / manipulates routing, devices, policy routing and tunnels of network interfaces.

ip replaces the deprecated command ifconfig

Yes Yes Yes
Under construction.png Coming soon
kmemleak Monitoring tools kmemleak[12] provides a means to detect possible kernel memory leaks in a similar way to a tracing garbage collector, with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak.

It is not installed by default on all STM32MPU Software Packages.

No Yes Yes
Under construction.png Coming soon
trace-cmd Tracing tools trace-cmd[13] command interacts with the Ftrace tracer that is built inside the Linux kernel. It interfaces with the Ftrace specific files found in the debugfs file system under the tracing directory.

kernelshark[14] is a front-end reader of trace-cmd output. "trace-cmd record" and "trace-cmd extract" create a trace.dat (trace-cmd.dat) file. kernelshark can read this file, and produce a graph and list view of the corresponding data.

No No Yes
Under construction.png Coming soon
ltrace Tracing tools ltrace[15] is used to display the calls to shared libraries made by a userspace application. ltrace is a userspace application.

Its use is very similar to strace.

No No Yes
Under construction.png Coming soon
LTTng Tracing tools LTTng[16] is an open source tracing framework for Linux kernel and user spaces. It is a powerful tool that can be used for many purposes. LTTng traces need to be processed/displayed with a host tool such as Trace Compass[17], based on Eclipse plugin for increased portability. No No Yes
Under construction.png Coming soon
netdata Monitoring tools netdata[18] is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards. Yes Yes Yes
Under construction.png Coming soon
netstat Monitoring tools netstat[19] prints network connections, routing tables, interface statistics, masquerade connections, and multicast membership information. Yes Yes Yes
Under construction.png Coming soon
perf Monitoring tools perf[20] is a Linux user space tool, which allows getting system performance figures Yes Yes Yes
Under construction.png Coming soon
strace Tracing tools strace[21] is able to intercept and record the system calls which are called by a process and the signals which are received by another process. Yes Yes Yes
Under construction.png Coming soon
Under construction.png Coming soon

sysprof Monitoring tools sysprof[22] is a statistical, system-wide profiler for Linux. It helps in finding the functions in which a program spends most of its time.

sysprof proposes a user interface available directly on the board display screen.

Yes Yes Yes

Under construction.png Coming soon

sysstat Monitoring tools The sysstat [23] tool suite contains utilities to monitor the system performance and usage activity.

It contains various utilities, common to many commercial Unix distributions, as well as tools that can be scheduled (via a scheduler such as cron) to collect and historize performance and activity data:

  • iostat: reports CPU statistics and input/output statistics for block devices and partitions.
  • mpstat: reports individual or combined processor related statistics.
  • pidstat: reports statistics for Linux tasks (processes): I/O, CPU, memory, etc.
  • sar: collects, reports and saves system activity information (CPU, memory, disks, interrupts, network interfaces, TTY, kernel tables,etc.)
  • sadf: displays data collected by sar in multiple formats (CSV, XML, JSON, etc.). This command can also be used to exchange data with other programs or to draw graphs illustrating the various activities collected by sar using SVG (Scalable Vector Graphics) format.

Yes Yes Yes

Under construction.png Coming soon

tcpdump Monitoring tools tcpdump[24] is a common packet analyzer that runs under the command line. It allows the user to display TCP/IP and other packets being transmitted or received over a network to which the computer is connected. Yes Yes Yes

Under construction.png Coming soon

top Monitoring tools The top[25] program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The types of system summary information shown and the types, order and size of information displayed for tasks are all user configurable and that configuration can be made persistent across restarts. (Extracted from man page[25]) Yes Yes Yes

Under construction.png Coming soon

valgrind Monitoring tools valgrind[26] is an instrumentation framework for building dynamic analysis tools. Some Valgrind tools can automatically detect many memory management and threading bugs, and profile your programs in detail.

This is tool for Linux application analysis.

Yes Yes Yes

Under construction.png Coming soon

4 Trace and debug overview per Linux software frameworks[edit]

The picture below allows accessing to different Linux software frameworks which provide specific trace and debug information in their "How to trace and debug the framework" dedicated chapter.

ALSA overviewCategory:Arm architectureBluetooth overviewClock overviewCEC overviewResource manager for coprocessingremoteproc framework overviewMailbox framework overviewRPMsg framework overviewHardware spinlock overview overviewCoprocessor management overviewHardware random overviewCrypto API overviewOP-TEE overviewLinux kernel device treeDmaengine overviewDRM KMS overviewVivante GCNANO GPU overviewI2C overviewIIO overviewInterrupt overviewOverview of GPIO pinsMMC overviewMTD overviewNVMEM overviewCAN overviewEthernet overviewWLAN overviewBluetooth overviewPower overviewPower overviewRegulator overviewThermal overviewPower overviewThermal overviewReset overviewRTC overviewSerial TTY overviewSPI overviewPWM overviewUSB overviewMMC overviewV4L2 camera overviewWatchdog overview
STM32MP15 Linux kernel overview

OpenSTLinux BSP legend.png

5 Tips[edit]

How to find Linux kernel driver associated to a device.

How to use the kernel dynamic debug.

6 Documentation and web articles[edit]

A lot of articles on the web mention Linux® kernel tracing and profiling. The following links provide a good introduction to these topics:

Info.png More general Linux performance information are available in nice slideshare presentation (Brendan Gregg), on the Brendan Gregg official web site or in LinuxCon2014 article.

Reference list:

Secure digital

former spelling for e•MMC ('e' in italic)

GNU dedugger, a portable debugger that runs on many Unix-like systems

Debug File System (See for more details)

Central processing unit