Difference between revisions of "Coprocessor management troubleshooting grid"

[quality revision] [quality revision]
 

Some typical issues related to the management of a coprocessor are listed below. Solutions or debugging methods are proposed for these issues.

If your issue is not listed, try looking at in the articles in the Coprocessor management Linux, Coprocessor management STM32Cube or troubleshooting grids categories.

1 Coprocessor firmware loading and control[edit]

Symptom Resolution

The coprocessor traces are not available on the Linux® side

Board $> cat  /sys/kernel/debug/remoteproc/remoteproc0/trace0
No such file or directory

This may happen for two reasons:

  • The firmware does not include any resource table or the resource table does not define any trace.
Update the firmware resource table and rebuild the firmware
  • The ".resource_table" section is empty or not defined in the elf file.
Use the following command to verify it in the generated elf file generated :
 
PC $> readelf -l <elf file>
 ...
 02     .data .resource_table .bss ._user_heap_stack
 ....

In case When starting the coprocessor from the boot loader bootloader (u-boot)  :

"unsupported fw ver: 0
Remote Processor 0 resource table Not found : 0x00000000-0x0 "

The firmware does not include any resource table. This is only a warning that do does not prevent the firmware to start from starting properly. "rproc load_rsc" step can be bypassed.

2 Inter processor communication[edit]

Symptom Resolution

Frozen firmware as consequence of a deadlock in OpenAMP during IP communication withe with the the main processor.

This Issue probably comes from rpmsg_virtio_rx_callback or rpmsg_virtio_send_offchannel_raw (rpmsg_virtio.c) functions which is that are called in interrupt context. They are using These functions use a mutex lock in rpmsg_device struct when accessing the index of the virtio queue index. Rework your code so theses that these functions are not called in interrupt context.

Linux kernel trace:

stm32-ipcc 4c001000.mailbox: Try increasing MBOX_TX_QUEUE_LEN

On each IPCC interrupt, the coprocessor treats all the RPMsg buffered RPMsgs (one IPCC event for several RPMsgRPMsgs). On Linux Side side, one IPCC signal is programmed for each RPMsg sent. This can result is an overflow warning on Linux because since too many IPCC events are queued. No RPMsg RPMsgs are dropped , but this message could be interpreted as the coprocessor reaches its capacity to treat the received messages in time.
Consider to rework reworking the code so that the coprocessor to process processes messages more efficiently , or to decrease decreasing the message rate of messages sent by the Linux.

Linux kernel trace ( example with rpmsg_tty driver):

 rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: timeout waiting for a tx buffer

This message means that there is no more TX buffer available to transmit a message messages to the remote processor. This may happen for two reasons:

  • The firmware implementation is incomplete and the coprocessor does not process any IPCC interrupt (eg interrupt disabled , or interrupt handler not defined, ...). Fix the firmware code of the firmware : refer to IPCC_internal_peripheral for detail details on the peripheral.
  • The coprocessor is busy, frozen or crashed. This can be confirmed by performing a debug tool analysis.

Linux kernel trace:

rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: No memory for tty_prepare_flip_string

There is no more space in the TTY temporary buffer on reception. The root cause is probably that the Linux application does did not read the tty device in on time to treat the incoming TTY stream. Consider to rework reworking the Linux application so that it takes less time to process messages , or decrease decreasing the message exchange frequencyrate.

Linux kernel trace:

remoteproc remoteproc0: stm32_rproc_kick: failed (<mbx>, <error value>)

The Linux remoteproc driver can't cannot use the IPCC mailbox which . This may happen for two reasons:

  • The Linux kernel is built without the support of the stm32 IPCC mailbox: to enable it, see refer to IPCC configuration.
  • The mailboxes are not / or incorrectly defined in the Linux kernel DeviceTree: fix it as described in mailbox DeviceTree.


<onlyinclude>

Some typical issues related to the management of a coprocessor are listed below. Solutions or debugging methods are proposed for these issues.

If your issue is not listed, try looking at in the articles in the [[:Category:Coprocessor management Linux|  Coprocessor management Linux]], [[:Category:Coprocessor management STM32Cube | Coprocessor management STM32Cube]] or [[:Category:Troubleshooting grids|troubleshooting grids]] categories.

== Coprocessor  firmware loading and control ==
{{:Troubleshooting grid table template}}<onlyinclude>
<includeonly>

! colspan="2" | coprocessor  firmware loading and control
|-</includeonly>

|
The coprocessor traces are not available on the Linux<sup>&reg;</sup> side
 {{Board$}} cat  /sys/kernel/debug/remoteproc/remoteproc0/trace0
 No such file or directory
||
This may happen for two reasons:
* The firmware does not include any [[Coprocessor_resource_table|resource table]] or the resource table does not define any [[Coprocessor_resource_table#How_to_add_trace_for_the_log_buffer|trace]]. 
: Update the firmware resource table and rebuild the firmware
* The ".resource_table" section  is empty or not defined in the elf file. 
:Use the following command to verify it in the generated elf file generated :
  {{PC$}} readelf -l <elf file>

  ...
  02     .data .resource_table .bss ._user_heap_stack
  ....
|-
|In case When starting the coprocessor from the boot loader bootloader (u-boot)  : 
 "unsupported fw ver: 0
 Remote Processor 0 resource table Not found : 0x00000000-0x0 "
||
The firmware does not include any [[Coprocessor_resource_table|resource table]].
This is only a warning that dodoes not prevent the firmware to startfrom starting properly.
"rproc load_rsc" step can be bypassed 
.</onlyinclude>

|}

== Inter processor communication ==
{{:Troubleshooting grid table template}}<onlyinclude>
<includeonly>

! colspan="2" | Inter processor communication
|-</includeonly>

|
Frozen firmware as consequence of a deadlock in OpenAMP during IP communication withewith the the main processor.
||This Issue probably comes from rpmsg_virtio_rx_callback or rpmsg_virtio_send_offchannel_raw (rpmsg_virtio.c) functions which isthat are called in interrupt context. They are usingThese functions use a mutex lock in rpmsg_device struct when accessing the index of the virtio queue index. Rework your code  so theses that these functions are not called in interrupt context.
|-
|
Linux kernel trace: 
 stm32-ipcc 4c001000.mailbox: Try increasing MBOX_TX_QUEUE_LEN
||
On each IPCC interrupt, the coprocessor treats all the RPMsg buffered RPMsgs (one IPCC event for several RPMsgRPMsgs). On Linux Sideside, one IPCC signal is programmed for each RPMsg sent. This can result is an overflow warning on Linux becausesince too many IPCC events are queued. No RPMsgRPMsgs are dropped,   but this message could be interpreted as the coprocessor  reaches its capacity to treat the received messages in time.<br>

Consider to rework reworking the code so that the coprocessor to process processes messages more efficiently, or to decrease decreasing the message rate of messages sent by the Linux.
|-
|
Linux kernel trace ( example with rpmsg_tty driver): 
  rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: timeout waiting for a tx buffer
||
This message means that there is no more TX buffer available to transmit a message messages to the remote processor.
This may happen for two reasons:
* The firmware implementation is incomplete and the coprocessor does not process any IPCC interrupt (eg interrupt disabled,  or interrupt handler not defined, ...). Fix the code of the firmware firmware code: refer to [[IPCC_internal_peripheral]] for detaildetails on the peripheral.
* The coprocessor is busy, frozen or crashed. This can be confirmed by performing a debug tool analysis.
|-
|
Linux kernel trace: 
 rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: No memory for tty_prepare_flip_string
||
There is no more space in the TTY temporary buffer on reception. The root cause is probably that the Linux application doesdid not read the tty device inon time to treat the incoming TTY stream.
Consider to rework reworking the Linux application so that it takes less time to process messages, or decreasedecreasing the message exchange frequencyrate.
|-
|
Linux kernel trace: 
 remoteproc remoteproc0: stm32_rproc_kick: failed (<mbx>, <error value>)
||
The Linux remoteproc driver can'tcannot use the [[IPCC_internal_peripheral | IPCC]] mailbox which. This may happen for two reasons:
* The Linux kernel is built without the support of the stm32 IPCC mailbox: to enable it, see refer to  [[Linux_Mailbox_framework_overview#Configuration|IPCC configuration]].
* The mailboxes are not /or incorrectly defined in the Linux kernel DeviceTree: fix it as described in [[Linux_remoteproc_framework_overview#Device_tree_configuration|mailbox DeviceTree]].</onlyinclude>

|}
<noinclude>

[[Category:Coprocessor management Linux]]
[[Category:Coprocessor management STM32Cube]]
[[Category:Troubleshooting grids]]{{PublicationRequestId | 14609 | 2020-01-15 |}}</noinclude>
(10 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
Some typical issues related to the management of a coprocessor are listed below. Solutions or debugging methods are proposed for these issues.
 
Some typical issues related to the management of a coprocessor are listed below. Solutions or debugging methods are proposed for these issues.
   
If your issue is not listed, try looking at in the articles in the [[:Category:Coprocessor management Linux|  Coprocessor management Linux]], [[:Category:Coprocessor management STM32Cube | Coprocessor management STM32Cube]] or [[:Category:Troubleshooting grids|troubleshooting grids]] categories.
+
If your issue is not listed, try looking in the articles in the [[:Category:Coprocessor management Linux|  Coprocessor management Linux]], [[:Category:Coprocessor management STM32Cube | Coprocessor management STM32Cube]] or [[:Category:Troubleshooting grids|troubleshooting grids]] categories.
   
 
== Coprocessor  firmware loading and control ==
 
== Coprocessor  firmware loading and control ==
Line 12: Line 12:
 
</includeonly>
 
</includeonly>
 
|
 
|
The coprocessor traces are not available on the Linux side
+
The coprocessor traces are not available on Linux<sup>&reg;</sup> side
 
  {{Board$}} cat  /sys/kernel/debug/remoteproc/remoteproc0/trace0
 
  {{Board$}} cat  /sys/kernel/debug/remoteproc/remoteproc0/trace0
 
  No such file or directory
 
  No such file or directory
Line 20: Line 20:
 
: Update the firmware resource table and rebuild the firmware
 
: Update the firmware resource table and rebuild the firmware
 
* The ".resource_table" section  is empty or not defined in the elf file.  
 
* The ".resource_table" section  is empty or not defined in the elf file.  
:Use following command to verify it in elf file generated :
+
:Use the following command to verify it in the generated elf file:
 
   {{PC$}} readelf -l <elf file>
 
   {{PC$}} readelf -l <elf file>
 
   ...
 
   ...
Line 27: Line 27:
 
|-
 
|-
 
|
 
|
In case starting the coprocessor from the boot loader (u-boot) :  
+
When starting the coprocessor from the bootloader (u-boot):  
 
  "unsupported fw ver: 0
 
  "unsupported fw ver: 0
 
  Remote Processor 0 resource table Not found : 0x00000000-0x0 "
 
  Remote Processor 0 resource table Not found : 0x00000000-0x0 "
 
||
 
||
 
The firmware does not include any [[Coprocessor_resource_table|resource table]].
 
The firmware does not include any [[Coprocessor_resource_table|resource table]].
This is only a warning that do not prevent the firmware to start properly.
+
This is only a warning that does not prevent the firmware from starting properly.
"rproc load_rsc" step can be bypassed  
+
"rproc load_rsc" step can be bypassed.
 
</onlyinclude>
 
</onlyinclude>
 
|}
 
|}
Line 45: Line 45:
 
</includeonly>
 
</includeonly>
 
|
 
|
Frozen firmware as consequence of a deadlock in OpenAMP during IP communication withe the main processor.
+
Frozen firmware as consequence of a deadlock in OpenAMP during IP communication with the the main processor.
 
||
 
||
Issue probably comes from rpmsg_virtio_rx_callback or rpmsg_virtio_send_offchannel_raw (rpmsg_virtio.c) functions which is called in interrupt context. They are using a mutex lock in rpmsg_device struct when accessing the index of the virtio queue index. Rework your code  so theses functions are not called in interrupt context.
+
This Issue probably comes from rpmsg_virtio_rx_callback or rpmsg_virtio_send_offchannel_raw (rpmsg_virtio.c) functions that are called in interrupt context. These functions use a mutex lock in rpmsg_device struct when accessing the index of the virtio queue index. Rework your code  so that these functions are not called in interrupt context.
 
|-
 
|-
 
|
 
|
Line 53: Line 53:
 
  stm32-ipcc 4c001000.mailbox: Try increasing MBOX_TX_QUEUE_LEN
 
  stm32-ipcc 4c001000.mailbox: Try increasing MBOX_TX_QUEUE_LEN
 
||
 
||
On each IPCC interrupt, the coprocessor treats all the RPMsg buffered (one IPCC event for several RPMsg). On Linux Side one IPCC signal is programmed for each RPMsg sent. This can result is an overflow warning on Linux because too many IPCC events are queued. No RPMsg are droppedbut this message could be interpreted as the coprocessor  reaches its capacity to treat the received messages in time.<br>
+
On each IPCC interrupt, the coprocessor treats all the buffered RPMsgs (one IPCC event for several RPMsgs). On Linux side, one IPCC signal is programmed for each RPMsg sent. This can result is an overflow warning on Linux since too many IPCC events are queued. No RPMsgs are dropped but this message could be interpreted as the coprocessor  reaches its capacity to treat the received messages in time.<br>
Consider to rework the code so the coprocessor to process messages more efficiently, or to decrease the message rate sent by the Linux.
+
Consider reworking the code so that the coprocessor processes messages more efficiently or decreasing the rate of messages sent by Linux.
 
|-
 
|-
 
|
 
|
Line 60: Line 60:
 
   rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: timeout waiting for a tx buffer
 
   rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: timeout waiting for a tx buffer
 
||
 
||
This message means that there is no more TX buffer available to transmit a message to the remote processor.
+
This message means that there is no more TX buffer available to transmit messages to the remote processor.
 
This may happen for two reasons:
 
This may happen for two reasons:
* The firmware implementation is incomplete and the coprocessor does not process any IPCC interrupt (interrupt disabled, interrupt handler not defined, ...). Fix the code of the firmware : refer to [[IPCC_internal_peripheral]] for detail on the peripheral.
+
* The firmware implementation is incomplete and the coprocessor does not process any IPCC interrupt (eg interrupt disabled or interrupt handler not defined). Fix the firmware code: refer to [[IPCC_internal_peripheral]] for details on the peripheral.
* The coprocessor is busy, frozen or crashed. This can be confirmed by a debug tool analysis.
+
* The coprocessor is busy, frozen or crashed. This can be confirmed by performing a debug tool analysis.
 
|-
 
|-
 
|
 
|
Line 69: Line 69:
 
  rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: No memory for tty_prepare_flip_string
 
  rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: No memory for tty_prepare_flip_string
 
||
 
||
There is no more space in the TTY temporary buffer on reception. The root cause is probably that the Linux application does not read the tty device in time to treat the incoming TTY stream.
+
There is no more space in the TTY temporary buffer on reception. The root cause is probably that the Linux application did not read the tty device on time to treat the incoming TTY stream.
Consider to rework the Linux application so it takes less time to process messages, or decrease the message exchange frequency.
+
Consider reworking the Linux application so that it takes less time to process messages or decreasing the message exchange rate.
 
|-
 
|-
 
|
 
|
Line 76: Line 76:
 
  remoteproc remoteproc0: stm32_rproc_kick: failed (<mbx>, <error value>)
 
  remoteproc remoteproc0: stm32_rproc_kick: failed (<mbx>, <error value>)
 
||
 
||
The Linux remoteproc driver can't use the [[IPCC_internal_peripheral | IPCC]] mailbox which may happen for two reasons:
+
The Linux remoteproc driver cannot use the [[IPCC_internal_peripheral | IPCC]] mailbox. This may happen for two reasons:
* The Linux kernel is built without the support of the stm32 IPCC mailbox: enable it, see [[Linux_Mailbox_framework_overview#Configuration|IPCC configuration]].
+
* The Linux kernel is built without the support of the stm32 IPCC mailbox: to enable it, refer to  [[Linux_Mailbox_framework_overview#Configuration|IPCC configuration]].
* The mailboxes are not / incorrectly defined in the Linux kernel DeviceTree: fix it as described in [[Linux_remoteproc_framework_overview#Device_tree_configuration|mailbox DeviceTree]].
+
* The mailboxes are not or incorrectly defined in the Linux kernel DeviceTree: fix it as described in [[Linux_remoteproc_framework_overview#Device_tree_configuration|mailbox DeviceTree]].
 
</onlyinclude>
 
</onlyinclude>
 
|}
 
|}
Line 86: Line 86:
 
[[Category:Coprocessor management STM32Cube]]
 
[[Category:Coprocessor management STM32Cube]]
 
[[Category:Troubleshooting grids]]
 
[[Category:Troubleshooting grids]]
  +
{{PublicationRequestId | 14609 | 2020-01-15 |}}
 
</noinclude>
 
</noinclude>