ISP middleware

1. Description

The ISP middleware is the embedded software lying between the drivers (mainly the DCMIPP-ISP HAL and sensor drivers) and the applications. Essentially functioning as a hidden translation layer, it enables image quality (IQ) for the applications running camera module pipelines.

It implements the following key features:

• DCMIPP ISP features control (for example: contrast, black level)
• Advanced IQ algorithms (for example: Auto exposure control)
• ISP and sensor configuration with the support of the remote STM32 ISP IQTune
• Application control for user-defined ISP settings (for example: exposure update)

The ISP middleware relies on two concepts:

• Its usage is as simple as possible. Basically, it exposes an Init(), Start(), and BackgroundProcess() API, which are responsible for controlling all the ISP and sensor features with the support of advanced IQ algorithms.
• It applies predefined ISP settings, which are tuned upstream with the STM32 ISP IQTune.

2. Services

The ISP middleware offers the following services:

• Configuration of the DCMIPP ISP and the camera module sensor (for example, demosaicing, decimation)
• Configuration and control of some IQ algorithms (for example: auto exposure contol, auto white balance)
• Communication with the remote STM32 ISP IQTune to configure the ISP and sensor settings. This service is only available if the middleware is built in engineering mode (tuning tool configuration).

3. Architecture overview

The components are:

• Application : user application controlling a camera module which:
  • Configures and starts the camera module pipeline
  • Controls the frame post processing like downsizing
  • Initializes the ISP middleware and calls its background loop

• ISP middleware: middleware controlling the image quality of the camera module pipeline which:
  • Configures the DCMIPP ISP and the sensor
  • Configures and controls 2A IQ algorithms (auto exposure and auto white balance)
  • Communicates with the remote STM32 ISP IQTune to configure the ISP and sensor settings. However, this is only available if the middleware is built in engineering mode.

• USB middleware:
  • the STM32 middleware USB Device enables the USB communication with STM32 ISP IQTune. Note that this is only available if the middleware is built in engineering mode.

• Camera middleware :
  • Middleware that provides the control of several camera module sensors and calls sensor drivers (for example: IMX335) API.

• HAL drivers:
  • the abstraction layers that interface with the different STM32 IP.

• Sensor driver:
  • Controls the camera module sensor.

• STM32 internal peripherals (hardware)
  • IP used by the ISP middleware:
    • DCMIPP. The ISP part of this IP is controlled by the ISP middleware.
    • USB is used to communicate with the remote STM32 ISP IQTune (only available if the middleware is built in engineering mode).
  • IP used by the ISP application:
    • DCMIPP. The non-ISP part (postprocessing) is controlled by the application.
    • CSI-2 connects the camera module CSI interface to the DCMIPP input.
    • I²C is used to control and configure the camera module sensor.

• STM32 ISP IQTune (peripheral):
  • STM32 ISP IQTune runs on a remote PC and allows the user to perform sensor and ISP fine-tuning. It generates IQ tuning parameters, which are used at runtime by the ISP middleware.

• IMX335 camera sensor (peripheral) :
  • The camera module sensor produces raw Bayer frames without any internal ISP support.

4. API

The ISP middleware exposes a set of control services:

Function	Description
ISP_Init()	Initialize the ISP middleware. The IQ tuning parameters structure is provided as an input parameter. A list of helpers providing non-ISP controls (for example: sensor configuration) are also provided as input parameter.
ISP_DeInit()	Deinitialize the ISP middleware
ISP_Start()	Start the ISP processing. This lets the middleware configure the ISP and the sensor according to the IQ tuning parameters, and then starts the IQ algorithms.
ISP_BackgroundProcess()	Ask the middleware to run its background task, such as algorithm control and STM32 ISP IQTune message exchange (if the middleware is built in engineering mode). This function needs to be scheduled on a regular basis (at least one call per VSYNC to get the best performance). With a bare-metal OS that supports either process, task, or thread, the application must call this function from its main loop.

It also exposes a limited number of applicative services:

Function	Description
ISP_SetExposureTarget()	Adjust the exposure target setting.
ISP_GetExposureTarget()	Read the exposure target setting.
ISP_ListWBRefModes()	Return the list of predefined white balance modes.
ISP_SetWBRefMode()	Apply some predefined ISP gains and color conversion settings as defined by a white balance mode.
ISP_GetWBRefMode()	Read the current white balance mode.
ISP_SetAECState()	Enable or disable the Auto Exposure Control algorithm.
ISP_GetAECState()	Read the state of the Auto Exposure Control algorithm.
ISP_GetDecimationFactor()	Read the ISP decimation factor.
ISP_SetStatArea()	Define the statistic window area.
ISP_GetStatArea()	Read the statistic window area.

5. Project structure

Here is the general structure of a project using the ISP middleware:

Application
|--> main.c                    Main application entry point
|--> isp_conf.c                Camera pipeline event callback implementation
|--> ...
Middleware
|--> ISP_Library               ISP middleware
|    |--> isp_algo.c           IQ algorithms
|    |--> isp_core.c           Core and API
|    |--> isp_services.c       Internal services
|--> Camera_Middleware
Drivers
Includes
|--> project_path/Inc
|    |--> isp_param_conf.h     ISP IQ configuration parameters

The isp_param_conf.h file, which holds the ISP and sensor settings can be generated by STM32 ISP IQTune.

A project using the ISP middleware built in engineering mode (build option ISP_MW_TUNING_TOOL_SUPPORT) includes some more files. It can connect to STM32 ISP IQTune to configure the ISP and sensor settings.

Application
|--> main.c
|--> isp_conf.c
|--> usbd_conf.c               USB device configuration and interface
|--> ...
Middleware
|--> ISP_Library
|    |--> USB_DEVICE           Interface with USB
          |--> ...
|    |--> isp_algo.c
|    |--> isp_cmd_parser.c     STM32 ISP IQTune command parser
|    |--> isp_core.c
|    |--> isp_services.c
|    |--> isp_tool_com.c       STM32 ISP IQTune communication
|--> Camera_Middleware
|--> USB_Device_Library        USB device middleware