This article gives information about the Linux® PCIe framework.
It explains how to activate PCIe interface and, based on examples, how to access it from user space.
1. Framework purpose[edit source]
The PCIe (Peripheral Component Interconnect express) Linux® framework supports two types of operating modes:
- Root Complex controller mode to connect peripheral devices
- Endpoint controller mode to be used as a peripheral
Linux can be used on the host machine. In this case various type of PCIe peripherals can be plugged in, such as:
- Mass storage (SATA or USB bridge...)
- Network interface (ethernet, wifi..)
Linux can also be used as a device on the peripheral side.
2. System overview[edit source]
2.1. Component description[edit source]
- PCIe userland (User space)
- Host-Side userland
- - pciutils[1] is a set of PCI utilities for collecting information about the PCI devices that are connected to the PCIe host.
- - One of the well-known utility is lspci, used to display information about PCIe buses and the devices connected to them.
- Common userland
- - sysfs provides an information interface available through the user terminal. See How to monitor with sysfs below.
- PCIe framework (Kernel space): PCIe RootComplex which rely on the PCIe core with specific APIs to support PCIe devices
- Host-Side provides API interface to class drivers and forwards the request from class drivers to host controller driver.
- PCIe controller drivers (Kernel space)
- PCIe Host controller drivers such as STM32 PCIe PCIe Host controllers in the PCIe Host-Side framework. STM32 PCIe uses kernel community drivers (kernel space), based on the PCIe framework.
- PCIe controller drivers can rely on Generic PHY framework to manage the physical layer for PCIe data transmissions and reference clock. STM32 ComboPHY PHY provider is a PHY driver in the Generic PHY framework:
2.2. API description[edit source]
The PCIe EPC library describes the set of API used to write a PCIe driver for the controller operating in End Point mode. See PCI kernel documentation for more details on API functions.
3. Configuration[edit source]
3.1. Kernel configuration[edit source]
PCIe support, STM32 PCIe driver is activated by default in ST deliveries. Nevertheless, if a specific configuration is required, this section indicates how the PCIe framework can be activated/deactivated in the kernel.
Activate PCI support (CONFIG_PCI=y) in the kernel configuration with the Linux Menuconfig tool: Menuconfig or how to configure kernel then select:
Device Drivers ---> [*] PCI support --->
Then activate PCI controllers drivers.
To activate the STM32 PCIe driver in Host mode, select:
Device Drivers --->
--- PCI support
---- PCI controller drivers
----- DesignWare PCI Core Support
------ STMicroelectronics PCIe Controller Host Mode for STM32 MP25
To activate the STM32 PCIe driver in Endpoint mode, select:
Device Drivers --->
--- PCI support
---- PCI controller drivers
----- DesignWare PCI Core Support
------ STMicroelectronics PCIe Controller Endpoint Mode for STM32 MP25
Then to activate the STM32 COMBOPHY driver, select:
Device Drivers --->
PHY Subsystem --->
<*> STMicroelectronics ComboPHY driver for STM32 MP25
3.2. Device tree configuration[edit source]
Detailed DT configurations for STM32 PCIe internal peripherals:
- for STM32 PCIe Host controller: PCIe device tree configuration
- for STM32 COMBOPHY PHY: ComboPHY device tree configuration
4. How to use the framework[edit source]
4.1. How to use the root complex framework[edit source]
4.1.1. How to list PCIe devices[edit source]
lspci displays information about the attached PCIe bridges and devices.
The example above shows a PCIe SATA controller, and a WIFI network controller device attached to a
4 ports PCIe switch.
lspci 00:00.0 PCI bridge: Synopsys, Inc. Device 0550 (rev 01) 01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 05:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
In the example:
- 00:00.0 represents the Host PCIe RC controller
- 01:00.0 represents inbound switch interface,
- 02:01-07 represents the outbound switch interface with 4 ports numbered 01,03,05,07
- 03:00.0 is the SATA controller outbound interface
- 05:00.0 is the WIFI controller outbound interface
The PCIe tree structure is best shown with the -P option:
lspci -P 00:00.0 PCI bridge: Synopsys, Inc. Device 0550 (rev 01) 00:00.0/00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 00:00.0/00.0/01.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 00:00.0/00.0/03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 00:00.0/00.0/05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 00:00.0/00.0/07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 00:00.0/00.0/01.0/00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 00:00.0/00.0/05.0/00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
To limit lspci to a specific device, uses its BDF address: for example, to check the SATA speed information:
lspci -vv -s 03:00.0 | grep Speed LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported LnkSta: Speed 5GT/s, Width x1 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
4.1.2. How to wakeup from PCIe[edit source]
|
Coming soon |
4.1.3. How to configure ASPM[edit source]
4.2. How to use the endpoint framework[edit source]
|
|
Coming soon |
5. How to trace and debug the framework[edit source]
5.1. How to monitor[edit source]
5.1.1. How to monitor with sysfs[edit source]
The same BDF format can be used to display information about devices.
cat /sys/bus/pci/devices/0000:03:00.0/current_link_speed
5.0 GT/s PCIe
5.1.2. Other ways of monitoring[edit source]
5.2. How to trace[edit source]
5.2.1. Activating PCIe framework debug messages[edit source]
A detailed dynamic trace is available in How to use the kernel dynamic debug
echo 8 > /proc/sys/kernel/printk echo "file drivers/pci/* +p" > /sys/kernel/debug/dynamic_debug/control
This command enables all the traces related to the PCIe core and drivers at runtime.
A finer selection can be made by choosing only the files to trace.
dynamic debug can also be activated at boot from the boot command line, for instance:
loglevel=8 dyndbg="file drivers/pci/* +p"
5.3. How to debug[edit source]
5.3.1. AER error messages[edit source]
Link errors are notified on the console with AER messages. Such errors usually indicate wrong electrical quality and can lead to speed demotion (demoted to Gen1 during link training) or timeouts due to the error logs on the display.
Recovered AER messages can flood the console and prevent correct use of the PCIe device. They can be disabled with pci=noaer command line parameter. In this case, link error statistics are logged in the host aer sysfs entries:
cat /sys/bus/pci/devices/0000:00:00.0/aer_dev_fatal Undefined 0 DLP 0 SDES 0 TLP 0 FCP 0 CmpltTO 0 CmpltAbrt 0 UnxCmplt 0 RxOF 0 MalfTLP 0 ECRC 0 UnsupReq 0 ACSViol 0 UncorrIntErr 0 BlockedTLP 0 AtomicOpBlocked 0 TLPBlockedErr 0 PoisonTLPBlocked 0 TOTAL_ERR_FATAL 0
The AER cause can be failing device, or an unsatisfactory electrical link (loss, jitter, extern aggressor, ...) Electrical compliance characterization must be performed, and the reference clock eye diagram performed to check Jitter or power ranges characteristics.
5.3.2. How to trace using a protocol analyzer[edit source]
Link-up recoveries or protocol error events can also be debugged with PCIe protocol analyzer. With the help of an exerciser, a PCIe protocol analyzer is a PCIe traffic sniffer that decodes PCIe packets and displays bus states and packets sent. Refer to you PCIe protocol analyzer user manual.
5.4. Troubleshooting[edit source]
Some common issues related to the PCIe feature are listed in STM32 PCIe troubleshooting
6. Source code location[edit source]
The source files are located inside the Linux kernel.
- The PCIe framework is under drivers/pci/
- The drivers used for STM32 PCIe are under drivers/pci/controller/dwc/pcie-stm32.c , drivers/pci/controller/dwc/pcie-stm32-ep.c
- The drivers used for STM32 ComboPHY are under drivers/phy/st/phy-stm32-combophy.c
7. To go further[edit source]
PCIe specifications can be found in PCISIG[2]
Very useful documentation can be found in the Linux kernel PCI documentation
An introduction to PCIe driver provided by Bootlin PCIe drivers[3]
8. References[edit source]