ISP middleware

1. Description

The ISP middleware is the embedded software situated between the drivers (mainly the DCMIPP-ISP HAL and sensor drivers) and the applications. Essentially functioning as a hidden translation layer, it enables image quality (IQ) for applications running camera module pipelines.

It implements the following key features:

• Control of DCMIPP ISP features (for example, contrast or black level)
• Advanced IQ algorithms (for example, auto exposure control)
• ISP and sensor configuration with the support of the remote STM32 ISP IQTune
• Application control for user-defined ISP settings (for example, exposure updates)

The ISP middleware relies on two main concepts:

• Its usage is as simple as possible. Basically, it exposes an Init(), Start(), and BackgroundProcess() APIs, which control all the ISP and sensor features with support from advanced IQ algorithms.
• It applies predefined ISP settings, which are tuned upstream using STM32 ISP IQTune.

2. Services

The ISP middleware offers the following services:

• Configuration of the DCMIPP ISP and the camera module sensor (for example, demosaicing, decimation)
• Configuration and control of some IQ algorithms (for example, auto exposure control, auto white balance)
• Communication with the remote STM32 ISP IQTune to configure the ISP and sensor settings. This service is only available if the middleware is built in engineering mode (tuning tool configuration).

3. Architecture overview

The components are:

• Application:
  User application controlling a camera module which:
  • Configures and starts the camera module pipeline
  • Controls frame post processing, like downsizing
  • Initializes the ISP middleware and calls its background loop
• ThreadX:
  • ThreadX is a real-time operating system (RTOS) that manages multitasking and system resources. It is required for the application to function properly with USBX, as USBX depends on ThreadX to handle USB communication tasks efficiently.
• ISP middleware:
  Middleware controlling the image quality of the camera module pipeline which:
  • Configures the DCMIPP ISP and the sensor
  • Configures and controls 2A IQ algorithms (auto exposure and auto white balance)
  • Communicates with the remote STM32 ISP IQTune to configure ISP and sensor settings. This feature is available only when the middleware is built with the ISP_MW_TUNING_TOOL_SUPPORT option enabled. Otherwise, the application that uses the middleware functions as a basic preview application.
• USBX middleware:
  • USBX middleware enables the USB communication with STM32 ISP IQTune. It is included only when building with the ISP_MW_TUNING_TOOL_SUPPORT compile-time flag (tuning tool mode). It also provides a UVC camera service, available only in tuning tool mode if the ISP_ENABLE_UVC compile-time flag is defined.
• Camera middleware:
  • Middleware that provides the control of several camera module sensors and calls sensor drivers' (for example, IMX335) APIs.
• HAL drivers:
  • Abstraction layers that interface with the different STM32 IPs.
• Sensor driver:
  • Controls the camera module sensor.
• STM32 internal peripherals (hardware):
  • IP used by the ISP middleware:
    • DCMIPP. The ISP part of this IP is controlled by the ISP middleware.
    • USB is used to communicate with the remote STM32 ISP IQTune (only available if the middleware is built with the ISP_MW_TUNING_TOOL_SUPPORT option enabled).
  • IP used by the ISP application:
    • DCMIPP. The non-ISP part (postprocessing) is controlled by the application.
    • CSI-2 connects the camera module CSI interface to the DCMIPP input.
    • I²C is used to control and configure the camera module sensor.
• STM32 ISP IQTune (peripheral):
  • Runs on a remote PC and allows the user to perform sensor and ISP fine-tuning. It generates IQ tuning parameters used at runtime by the ISP middleware.
• IMX335 camera sensor or any other supported sensor (peripheral):
  • The camera module sensor produces raw Bayer frames without any internal ISP support.

4. API

The ISP middleware exposes a set of control services:

Function	Description
ISP_Init()	Initialize the ISP middleware. The IQ tuning parameters structure is provided as an input parameter. A list of helpers providing non-ISP controls (for example: sensor configuration) are also provided as input parameter.
ISP_DeInit()	Deinitialize the ISP middleware
ISP_Start()	Start the ISP processing. This lets the middleware configure the ISP and the sensor according to the IQ tuning parameters, and then starts the IQ algorithms.
ISP_BackgroundProcess()	Ask the middleware to run its background task, such as algorithm control and STM32 ISP IQTune message exchange (if the middleware is built in engineering mode). This function needs to be scheduled on a regular basis (at least one call per VSYNC to get the best performance). With a bare-metal OS that supports either process, task, or thread, the application must call this function from its main loop.
ISP_GetStatus()	Get the overall ISP status (as returned by ISP_BackgroundProcess).

It also exposes a limited number of applicative services:

Function	Description
ISP_SetExposureTarget()	Adjust the exposure target setting.
ISP_GetExposureTarget()	Read the exposure target setting.
ISP_ListWBRefModes()	Return the list of predefined white balance modes.
ISP_SetWBRefMode()	Apply some predefined ISP gains and color conversion settings as defined by a white balance mode.
ISP_GetWBRefMode()	Read the current white balance mode.
ISP_SetAECState()	Enable or disable the Auto Exposure Control algorithm.
ISP_GetAECState()	Read the state of the Auto Exposure Control algorithm.
ISP_GetDecimationFactor()	Read the ISP decimation factor.
ISP_SetStatArea()	Define the statistic window area.
ISP_GetStatArea()	Read the statistic window area.
ISP_GetLuxEstimation()	Read the estimated lux value of the current scene.
ISP_EnableRestartState()	Enable the Restart State mode. When enabled, when the system restarts the ISP middleware, its configuration is set just as it was at its latest update.
ISP_DisableRestartState()	Disable the Restart State mode.

5. Project structure

Here is the general structure of a project using the ISP middleware:

Application
|--> main.c                    Main application entry point
|--> isp_conf.c                Camera pipeline event callback implementation
|--> ...
Middleware
|--> ISP_Library               ISP middleware
|    |--> isp_algo.c           IQ algorithms
|    |--> isp_ae_algo.c        IQ AEC algorithm
|    |--> isp_awb_algo.c       IQ AWB algorithm
|    |--> isp_core.c           Core and API
|    |--> isp_services.c       Internal services
|--> Camera_Middleware
Drivers
Includes
|--> project_path/Inc
|    |--> isp_param_conf.h     ISP IQ configuration parameters

The isp_param_conf.h file, which holds the ISP and sensor settings can be generated by STM32 ISP IQTune.

A project using the ISP middleware built in engineering mode (build option ISP_MW_TUNING_TOOL_SUPPORT) includes some more files. It can connect to STM32 ISP IQTune to configure the ISP and sensor settings.

Application
|--> main.c
|--> isp_conf.c
|--> usbx_conf.c               USB device configuration and interface
|--> ...
Middleware
|--> ISP_Library
|    |--> isp_algo.c
|    |--> isp_ae_algo.c
|    |--> isp_awb_algo.c
|    |--> isp_cmd_parser.c     STM32 ISP IQTune command parser
|    |--> isp_core.c
|    |--> isp_services.c
|    |--> isp_tool_com.c       STM32 ISP IQTune communication
|--> Camera_Middleware
|--> usbx                      Azure RTOS USBX for STM32 devices
|--> threadx                   Azure RTOS Threadx for STM32 devices