Trace-cmd and kernelshark trace viewer

1. Article purpose

This article provides the basic information needed to start using the Linux tool: trace-cmd and host trace viewer kernelshark.

2. Introduction

The following table provides a brief description of the tool, as well as its availability depending on the software packages:

: this tool is either present (ready to use or to be activated), or can be integrated and activated on the software package.

: this tool is not present and cannot be integrated, or it is present but cannot be activated on the software package.

Tool			STM32MPU Embedded Software distribution			STM32MPU Embedded Software distribution for Android™
Name	Category	Purpose	Starter Package	Developer Package	Distribution Package	Starter Package	Developer Package	Distribution Package
trace-cmd	Tracing tools	trace-cmd command interacts with the Ftrace tracer that is built inside the Linux kernel. It interfaces with the Ftrace specific files found in the debugfs file system under the tracing directory. kernelshark is a front-end reader of trace-cmd output. "trace-cmd record" and "trace-cmd extract" create a trace.dat (trace-cmd.dat) file. kernelshark can read this file, and produce a graph and list view of the corresponding data.

Note: in case you want to trace at boot time, it is recommended to use ftrace.

3. Installing the trace and debug tool on your target board

3.1. Using the STM32MPU Embedded Software distribution

3.1.1. Distribution Package

Since Yocto Thud version, trace-cmd recipe is no more present in openembedded-core meta package and so is not installed by default in all STM32MPU Software Packages.

Following recipes are proposed to integrate trace-cmd in STM32MPU Embedded Software distribution. It has to be added in layers/meta-st/meta-st-stm32mp/recipes-kernel.

• Create trace-cmd directory:

 mkdir layers/meta-st/meta-st-stm32mp/recipes-kernel/trace-cmd
 cd layers/meta-st/meta-st-stm32mp/recipes-kernel/trace-cmd

• Create trace-cmd.inc file with the following content:

SRCREV = "57371aaa2f469d0ba15fd85276deca7bfdd7ce36"
PV = "2.6.2"

inherit pkgconfig

FILESEXTRAPATHS =. "${FILE_DIRNAME}/trace-cmd:"

SRC_URI = "git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git;branch=trace-cmd-stable-v2.6"

S = "${WORKDIR}/git"

• Create trace-cmd_git.bb file with the following content:

SUMMARY = "User interface to Ftrace"
HOMEPAGE = "http://git.kernel.org/"
LICENSE = "GPLv2 & LGPLv2.1"

require trace-cmd.inc

LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe \
                    file://trace-cmd.c;beginline=6;endline=8;md5=2c22c965a649ddd7973d7913c5634a5e \
                    file://COPYING.LIB;md5=edb195fe538e4552c1f6ca0fd7bf4f0a \
                    file://trace-input.c;beginline=5;endline=8;md5=3ec82f43bbe0cfb5951ff414ef4d44d0 \
"

EXTRA_OEMAKE = "\
    'prefix=${prefix}' \
    'bindir=${bindir}' \
    'man_dir=${mandir}' \
    'html_install=${datadir}/kernelshark/html' \
    'img_install=${datadir}/kernelshark/html/images' \
    \
    'bindir_relative=${@oe.path.relative(prefix, bindir)}' \
    'libdir=${libdir}' \
    \
    NO_PYTHON=1 \
"

do_compile_prepend() {
    # Make sure the recompile is OK
    rm -f ${B}/.*.d
}

do_install() {
        oe_runmake DESTDIR="${D}" install
}

• Add trace-cmd package install to the st-image-weston image:

 echo 'IMAGE_INSTALL_append += "trace-cmd"' >> layers/meta-st/meta-st-openstlinux/recipes-st/images/st-image-weston.bbappend

• Rebuilt the st-image-weston target:

 cd <build_directory>
 bitbake st-image-weston

When testing the new image on the target board, trace-cmd can be found in the following path:

 which trace-cmd
/usr/bin/trace-cmd

However, an additional configuration of the Linux kernel must be performed to obtain the trace.

This is the configuration used for ftrace. Please refer to Ftrace install to complete the configuration of the Linux kernel.

This is the reason why you can not use this tool in Starter and Developer Packages context.

4. Getting started

trace-cmd inherits all available configurations from Ftrace.

It is possible to get filtered trace with both function and graph function mode.

It is also possible to trace specific prints done in the source file.

You first need to mount the tracefs:

 mount -t tracefs nodev /sys/kernel/tracing

To find out which tracers are available, simply cat the available_tracers file in the tracing directory:

  cat /sys/kernel/tracing/available_tracers
 function_graph function nop

Note that you must be placed on the target board within a directory path where you have access rights for writing the generated trace file.

4.1. Making a trace using function tracer mode

The example below shows how to filter on a set of functions that contains uart:

- To get a trace on UART, you can connect to the target board with the new console (i.e. through ssh), and then type a simple command, like "ls"

- Then press Ctrl-C to stop the trace

 trace-cmd record -p function -l *uart*
  plugin 'function'
Hit Ctrl^C to stop recording
^CCPU0 data recorded at offset=0x15d000
    4096 bytes in size
CPU1 data recorded at offset=0x15e000
    4096 bytes in size

See below for reading the trace.

4.2. Making a trace using graph function tracer mode

The example below shows how to filter on a set of graph functions that contains uart:

- To get trace on UART, you can connect to the target board with the new console (i.e. through ssh), and then type a simple command, like "ls"

- Then press Ctrl-C to stop the trace

 trace-cmd record -p function_graph -l *uart*
  plugin 'function_graph'
Hit Ctrl^C to stop recording
^CCPU0 data recorded at offset=0x15d000
    4096 bytes in size
CPU1 data recorded at offset=0x15e000
    4096 bytes in size

See below for reading the trace.

4.3. Reading the trace

Command to read the trace:

 trace-cmd report

By default, it reads the previously created file trace.dat at the position where the command is entered.

If you want to specify an other file name:

 trace-cmd report -i mytrace.dat
or
 trace-cmd report mytrace.dat

5. To go further

5.1. Visualizing trace using kernelshark

5.1.1. Host installation (Linux environment)

 sudo apt-get install kernelshark

5.1.2. Opening and reading trace-cmd trace

kernelshark can read trace-cmd report and produce a graph and list view of the data. Please refer to kernelshark home page for detail.

• Get the trace from the board. It is possible to get the trace directly from the SDCard under use, or remotely from the target board:

 scp root@<board_ip_address>/<path_to>/trace.dat <path_on_host>

• Open trace with kernelshark

It can be done by opening kernelshark and then opening trace.dat file from the menu, or using the following command:

 kernelshark <path_on_host>/trace.dat

Here is the example of a trace filtered with UART function_graph mode:

6. References

• Useful external links

Document link	Document Type	Description
Using KernelShark	User Guide	https://lwn.net
Trace-cmd presentation	User Guide	https://lwn.net
KernelShark quick tutorial	Presentation	https://elinux.org

Debug File System (See https://en.wikipedia.org/wiki/Debugfs for more details)

Universal Asynchronous Receiver/Transmitter

Debug File System (See https://en.wikipedia.org/wiki/Debugfs for more details)

Universal Asynchronous Receiver/Transmitter