Difference between revisions of "Linux tracing, monitoring and debugging"

[quality revision] [quality revision]
m (Tool overview)





1 Article purpose[edit]

This article provides useful information to start using Linux® tracing, monitoring and debugging environments.

Two entry points are proposed in this article:

  • Linux tracing, monitoring and debugging tools, which gives an overview of some Linux® tools including usage and application domain. This chapter is useful when you already know the domain or the interface to search for.
  • Trace and debug overview per Linux software frameworks, which points to articles explaining how to get trace and debug information about the Linux® software frameworks that are relevant for the STM32MPU Embedded Software. This chapter is useful when you know the Linux® framework to search for.

2 Linux trace architecture overview[edit]

The Linux® trace architecture can be organized into four levels as shown in the figure below (inspired by Brendan Gregg presentation[1]):

Linux kernel tracing architecture

2.1 Back-end instrumentation[edit]

The back-end instrumentation provides tracing sources built in the Linux® kernel. They are split into three categories:

  • tracepoints: kernel static tracing, statically placed at logical places in the kernel. It provides key event details as a "format" string.
  • kprobes: kernel dynamic tracing. It allows to trace function calls, returns and line numbers.
  • uprobes: dynamic user-level tracing.

2.2 Tracing framework[edit]

Also named tracers, they use tracing sources.

Tracing frameworks include kernel in-tree tracers such as ftrace and perf_events, and out-of-tree tracers such as SystemTap and sysdig.

2.3 Front-end tools[edit]

Front-end tools come on top of tracers and help to configure them. For example:

  • trace-cmd or LTTng for ftrace
  • perf or perf-Tools for perf_events

2.4 Add-on tools and viewer[edit]

Add-on tools are also on top of tracers. However, they are not embedded inside the Linux® kernel.

Viewer tools propose Visual interpretation of trace data. For example:

3 Linux tracing, monitoring and debugging tools[edit]

Linux® provides many tools that are either dedicated to one function or multifunction (generic).

They cover both Linux® kernel and Linux® user space.

3.1 Domain mapping[edit]

The following mapping, done by Brendan Gregg [4], shows the different existing tools associated to the different Linux® frameworks.

blktrace iostat perf ftrace stap LTTng pidstat strace ltrace perf netstat ethtool sysdig perf pidstat top perf mpstat tcpdump netstat ip sar /proc Filesytem
Note: The above image has been created by Brendan Gregg (Netflix) and can be found on his official web site.
Template:WarningImageMapOverlay


3.2 Tool overview[edit]

The following table provides a brief description of the tool, as well as its availability depending on the software packages:

Yes: this tool is either present (ready to use or to be activated), or can be integrated and activated on the software package.

No: this tool is not present and cannot be integrated, or it is present but cannot be activated on the software package.

Tool STM32MPU Embedded Software distribution STM32MPU Embedded Software distribution for Android™
Name Category Purpose Starter Package Developer Package Distribution Package Starter Package Developer Package Distribution Package
blktrace Tracing tools blktrace[5] generates traces of the I/O traffic on block devices (SD card, USB, eMMC...) No Yes Yes No No Yes
systemd core dump Debugging tools systemd core dump: generates core dump files on Linux
Yes No Yes No No No
ethtool Monitoring tools ethtool[6] allows to query or control network driver and hardware settings, in particular for wired Ethernet devices. Yes Yes Yes No No No
ftrace Tracing tools ftrace[7] (Function Tracer) is a powerful kernel tracing utility that is able, for instance, to trace every kernel function calls and kernel events without adding any extra code in your kernel source code No No Yes No No Yes
GDB Debugging tools The GNU Project debugger, GDB[8], allows monitoring program execution, or what the program was doing at the moment it crashed.
No* Yes No** No No Yes**
* Cross compile gdb and openocd binaries are required and only available from Developer Package.
** It is recommended to use the Developer Package to run the gdb debug session, which provided all dependencies
* Cross compile gdb and openocd binaries are required and only available from Distribution Package.
ifconfig Monitoring tools ifconfig[9] is a system administration utility for network interface configuration.

ifconfig is deprecated and has been replaced by ip (A web page provides a comparison between ifconfig and ip [10])
Yes Yes Yes Yes Yes Yes
ip Monitoring tools ip[11] shows / manipulates routing, devices, policy routing and tunnels of network interfaces.

ip replaces the deprecated command ifconfig

Yes Yes Yes Yes Yes Yes
kmemleak Monitoring tools kmemleak[12] provides a means to detect possible kernel memory leaks in a similar way to a tracing garbage collector, with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak. No Yes Yes No No Yes
trace-cmd Tracing tools trace-cmd[13] command interacts with the Ftrace tracer that is built inside the Linux kernel. It interfaces with the Ftrace specific files found in the debugfs file system under the tracing directory.

kernelshark[14] is a front-end reader of trace-cmd output. "trace-cmd record" and "trace-cmd extract" create a trace.dat (trace-cmd.dat) file. kernelshark can read this file, and produce a graph and list view of the corresponding data.

No No Yes No No No
ltrace Tracing tools ltrace[15] is used to display the calls to shared libraries made by a userspace application. ltrace is a userspace application.

Its use is very similar to strace.

No No Yes No No No
LTTng Tracing tools LTTng[16] is an open source tracing framework for Linux kernel and user spaces. It is a powerful tool that can be used for many purposes. LTTng traces need to be processed/displayed with a host tool such as Trace Compass[17], based on Eclipse plugin for increased portability. No No Yes No No No
netdata Monitoring tools netdata[18] is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards. Yes Yes Yes No No No
netstat Monitoring tools netstat[19] prints network connections, routing tables, interface statistics, masquerade connections, and multicast membership information. Yes Yes Yes Yes Yes Yes
perf Monitoring tools perf[20] is a Linux user space tool, which allows getting system performance figures Yes Yes Yes No* No* No*
Note: simpleperf[21] is present as equivalent but with less options
strace Tracing tools strace[22] is able to intercept and record the system calls which are called by a process and the signals which are received by another process. Yes Yes Yes Yes Yes Yes
Under construction.png Coming soon


sysprof Monitoring tools sysprof[23] is a statistical, system-wide profiler for Linux. It helps in finding the functions in which a program spends most of its time.

sysprof proposes a user interface available directly on the board display screen.

Yes Yes Yes No No No sysstat Monitoring tools The sysstat [24] tool suite contains utilities to monitor the system performance and usage activity.

It contains various utilities, common to many commercial Unix distributions, as well as tools that can be scheduled (via a scheduler such as cron) to collect and historize performance and activity data:

  • iostat: reports CPU statistics and input/output statistics for block devices and partitions.
  • mpstat: reports individual or combined processor related statistics.
  • pidstat: reports statistics for Linux tasks (processes): I/O, CPU, memory, etc.
  • sar: collects, reports and saves system activity information (CPU, memory, disks, interrupts, network interfaces, TTY, kernel tables,etc.)
  • sadf: displays data collected by sar in multiple formats (CSV, XML, JSON, etc.). This command can also be used to exchange data with other programs or to draw graphs illustrating the various activities collected by sar using SVG (Scalable Vector Graphics) format.
Yes Yes Yes No No No
Under construction.png Coming soon


tcpdump Monitoring tools tcpdump[25] is a common packet analyzer that runs under the command line. It allows the user to display TCP/IP and other packets being transmitted or received over a network to which the computer is connected. Yes Yes Yes Yes Yes Yes top Monitoring tools The top[26] program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The types of system summary information shown and the types, order and size of information displayed for tasks are all user configurable and that configuration can be made persistent across restarts. (Extracted from man page[26]) Yes Yes Yes Yes Yes Yes valgrind Monitoring tools valgrind[27] is an instrumentation framework for building dynamic analysis tools. Some Valgrind tools can automatically detect many memory management and threading bugs, and profile your programs in detail.

This is tool for Linux application analysis.

Yes Yes Yes No No Yes

4 Trace and debug overview per Linux software frameworks[edit]

The picture below allows accessing to different Linux software frameworks which provide specific trace and debug information in their "How to trace and debug the framework" dedicated chapter.


ALSA overview Category:Arm architecture Bluetooth overview Clock overview CEC overview Resource manager for coprocessing remoteproc framework overview Mailbox framework overview RPMsg framework overview Hardware spinlock overview Coprocessor management overview Hardware random overview Crypto API overview OP-TEE overview Linux kernel device tree Dmaengine overview DRM KMS overview Vivante GCNANO GPU overview I2C overview IIO overview Interrupt overview Overview of GPIO pins MMC overview MTD overview NVMEM overview CAN overview Ethernet overview WLAN overview Bluetooth overview Power overview Power overview Regulator overview Thermal overview Power overview Thermal overview Reset overview RTC overview Serial TTY overview SPI overview PWM overview USB overview MMC overview V4L2 camera overview Watchdog overview
STM32MP15 Linux kernel overview


OpenSTLinux BSP legend.png


5 Tips[edit]

How to find Linux kernel driver associated to a device.

How to use the kernel dynamic debug.

6 Documentation and web articles[edit]

A lot of articles on the web mention Linux® kernel tracing and profiling. The following links provide a good introduction to these topics:

Info.png More general Linux performance information are available in nice slideshare presentation (Brendan Gregg), on the Brendan Gregg official web site or in LinuxCon2014 article.

Reference list:

<noinclude>

{{ArticleMainWriter | Jean-PhilippeR}}

{{ReviewersList|Jean-ChristopheT, GeraldB, LoicP, AlexandreT, BernardP}}

{{ ArticleApprovedVersion | Jean-PhilippeR | Jean-ChristopheT, AlexandreT, PascalP  | Jean-PhilippeR - 13Nov'18 | AnneJ - 12Nov'18 - 9566 | 11Jan'19 }}

[[Category:Trace and debug tools]]</noinclude>


== Article purpose ==
This article provides useful information to start using Linux<sup>&reg;</sup> tracing, monitoring and debugging environments.

Two entry points are proposed in this article:
*[[#Linux tracing, monitoring and debugging tools|Linux tracing, monitoring and debugging tools]], which gives an overview of some Linux<sup>&reg;</sup> tools including usage and application domain. {{highlight|This chapter is useful when you already know the domain or the interface to search for}}.
*[[#Trace and debug overview per Linux software frameworks|Trace and debug overview per Linux software frameworks]], which points to articles explaining how to get trace and debug information about the Linux<sup>&reg;</sup> software frameworks that are relevant for the STM32MPU Embedded Software.  {{highlight|This chapter is useful when you know the Linux<sup>&reg;</sup> framework to search for}}.

== Linux trace architecture overview ==

The Linux<sup>&reg;</sup> trace architecture can be organized into four levels as shown in the figure below (''inspired by Brendan Gregg presentation''<ref>http://www.brendangregg.com/linuxperf.html</ref>):
[[File:Linux tracing architecture.png|center|link=|Linux kernel tracing architecture]]

=== Back-end instrumentation ===
The back-end instrumentation provides tracing sources built in the Linux<sup>&reg;</sup> kernel. They are split into three categories:
*'''tracepoints''': kernel static tracing, statically placed at logical places in the kernel. It provides key event details as  a "format" string.
*'''kprobes''': kernel dynamic tracing. It allows to trace function calls, returns and line numbers.
*'''uprobes''': dynamic user-level tracing.

=== Tracing framework ===
Also named tracers, they use tracing sources.

Tracing frameworks include kernel in-tree tracers such as ftrace and perf_events, and out-of-tree tracers such as SystemTap and sysdig.

=== Front-end tools ===
Front-end tools come on top of tracers and help to configure them. For example:
* trace-cmd or LTTng for ftrace
* perf or perf-Tools for perf_events

=== Add-on tools and viewer ===
Add-on tools are also on top of tracers. However, they are not embedded inside the Linux<sup>&reg;</sup> kernel.

Viewer tools propose Visual interpretation of trace data. For example:
* '''[[Trace-cmd and kernelshark trace viewer#Visualizing trace using kernelshark|kernelshark]]''' for [[Ftrace|ftrace]]/[[Trace-cmd and kernelshark trace viewer|trace-cmd]]
* '''[[LTTng#Import LTTng traces in a Trace Compass trace viewer|Trace Compass]]'''<ref>http://www.tracecompass.org</ref> for [[LTTng|LTTng]] (and more)
* '''[[Perf#Visualizing trace using Flame Graphs|Flame Graph]]'''<ref>http://www.brendangregg.com/flamegraphs.html</ref> for [[Perf|perf]]

== Linux tracing, monitoring and debugging tools ==
Linux<sup>&reg;</sup> provides many tools that are either dedicated to one function or multifunction (generic).

They cover both Linux<sup>&reg;</sup> kernel and Linux<sup>&reg;</sup> user space.

=== Domain mapping ===
The following mapping, done by Brendan Gregg <ref>http://www.brendangregg.com/linuxperf.html </ref>, shows the different existing tools associated to the different Linux<sup>&reg;</sup> frameworks.
{{ImageMap|
Image:linux_tracing_observability_tools.png {{!}} frame {{!}} center {{!}} Note: The above image has been created by '''Brendan Gregg''' (Netflix) and can be found on his [http://www.brendangregg.com/linuxperf.html official web site]. <br/> {{WarningImageMapOverlay}}
rect 12 374 100 392 [[Blktrace | blktrace]]
rect 12 334 100 354 [[Sysstat tool suite| iostat]]
rect 36 166 104 184 [[Perf | perf]]
rect 36 184 104 202 [[Ftrace | ftrace]]
rect 36 205 104 222 [[Systemtap | stap]]
rect 36 280 104 300 [[LTTng | LTTng]]
rect 50 120 130 140 [[Sysstat tool suite| pidstat]]
rect 72 80 140 100 [[Strace | strace]]
rect 148 80 220 100 [[Ltrace | ltrace]]
rect 180 332 232 354 [[Perf | perf]]
rect 380 80 465 100 [[Netstat | netstat]]
rect 380 475 460 495 [[Ethtool | ethtool]]
rect 472 80 540 100 [[Sysdig | sysdig]]
rect 528 109 578 129 [[Perf | perf]]
rect 528 256 605 276 [[Sysstat tool suite| pidstat]]
rect 532 234 574 254 [[Top Linux command line | top]]
rect 685 256 740 276 [[Perf | perf]]
rect 545 160 615 180 [[Sysstat tool suite| mpstat]]
rect 500 358 580 380 [[Tcpdump | tcpdump]]
rect 650 418 730 438 [[Netstat | netstat]]
rect 650 438 730 458 [[IP Linux command line | ip]]
rect 660 97 722 117 [[Sysstat tool suite| sar]]
rect 660 137 722 157 [[Pseudo filesystem | /proc]]
rect 150 232 280 252 [[OpenSTLinux filesystem | Filesytem]]
}}

{{InternalInfo|To check if required to address, vmstat,free, dtrace and ebpf}}

=== Tool overview ===
{{:Trace_and_debug_tools_assignment_table_template}}
{{:Blktrace}}
|-
{{:Core dump}}
|-
{{:Ethtool}}
|-
{{:Ftrace}}
|-
{{:GDB}}
|-
{{:Ifconfig}}
|-
{{:IP Linux command line | IP}}
|-
{{:Kmemleak}}
|-
{{:Trace-cmd and kernelshark trace viewer}}
|-
{{:Ltrace}}
|-
{{:LTTng}}
|-
{{:Netdata}}
|-
{{:Netstat}}
|-
{{:Perf}}
|-
{{:Strace}}
|-
{{:Sysdig}}
|-
{{:Sysprof}}
|-
{{:Sysstat tool suite}}
|-
{{:Systemtap}}
|-
{{:Tcpdump}}
|-
{{:Top Linux command line | Top}}
|-
{{:Valgrind}}
|}

== Trace and debug overview per Linux software frameworks ==
The picture below allows accessing to different Linux software frameworks which provide specific trace and debug information in their {{highlight|"''How to trace and debug the framework''"}} dedicated chapter.

{{:STM32MP15 Linux kernel overview}}

== Tips ==
[[How to find Linux kernel driver associated to a device]].

[[How to use the kernel dynamic debug]].

== Documentation and web articles ==

A lot of articles on the web mention Linux<sup>&reg;</sup> kernel tracing and profiling. The following links provide a good introduction to these topics:
* [http://www.brendangregg.com/Slides/LISA2014_LinuxPerfAnalysisNewTools.pdf Linux Performance Analysis - New Tools and Old Secrets]: description of the Linux<sup>&reg;</sup> technology and of the different tools available.
* [https://wiki.yoctoproject.org/wiki/Tracing_and_Profiling Yocto project: Tracing and profiling]: How to enable tracing and profiling tools using Yocto
* [http://www.brendangregg.com/linuxperf.html#Tools Brendan Gregg Linux performance page]

{{Info| More general Linux performance information are available in [http://fr.slideshare.net/brendangregg/linux-performance-tools-2014 nice slideshare presentation (Brendan Gregg)], on the [http://www.brendangregg.com/linuxperf.html Brendan Gregg official web site] or in LinuxCon2014 [http://www.linux.com/news/enterprise/systems-management/785426-a-template-for-monitoring-linux-performance-tools-like-from-netflix article].}}

Reference list:<references />
Line 120: Line 120:
 
|-
 
|-
 
{{:Sysstat tool suite}}
 
{{:Sysstat tool suite}}
|-
 
{{:Systemtap}}
 
 
|-
 
|-
 
{{:Tcpdump}}
 
{{:Tcpdump}}

Attachments

Discussions