How to tune ISP using the STM32 ISP IQTune

1. Article purpose

Unlike a camera module that embeds the lens, the image sensor, and the image signal processor (ISP), the RAW sensor module only embeds the lens and the image sensor.

Information

The image signal processor (ISP) processes the raw data from the sensor to produce a viewable image. It handles tasks such as white balance and color correction.

For camera modules, the ISP is tuned by the camera module provider.
For sensor modules, the ISP needs to be tuned by the application provider.

The purpose of this article is to provide step-by-step guidance to tune the DCMIPP ISP pipeline for a given RAW sensor and its associated lens.

2. Overview of DCMIPP ISP block diagram

The DCMIPP ISP (instantiated only once) is shared by both pipe1 and pipe2. It includes capabilities such as:

• Statistic removal: Eliminates unnecessary metadata, such as statistics, from the images received.
• Bad pixel: Detects and corrects artifacts generated by a bad pixel on the sensor array.
• Decimation: Downsizes the frame by selecting one pixel out of every 1, 2, 4, or 8. It ensures that the RAW frame size does not exceed 2688 pixels in width before demosaicing.
• Black level: Suppresses a potential sensor positive black offset by adjusting the black level for each pixel in the red, green, and blue components.
• ISP gain control: Sets the gains applied to the R, G, and B components.
• Demosaicing: Transforms the RAW Bayer format to RGB.
• Statistics extraction: Computes, from a statistic area, the average values for red, green, and blue components or histograms. Statistics are used by the software image quality algorithms.
• Color correction: Corrects the color with a fully programmable 3x3 coefficients matrix.
• Contrast: Enhances contrast through luminance adjustment.
• Gamma conversion: Implements a gamma compression on each R, G, and B component, using a static gamma exponent of 2.

Information

The decimation value is not configurable during the tuning procedure. This value is configured according to the sensor resolution to ensure that the RAW frame size does not exceed 2688 pixels in width before demosaicing.

Information

Although the gamma conversion block is not part of the ISP pipeline, the gamma conversion is configurable through the tuning procedure.

Refer to the STM32 reference manuals for the complete list of features.

3. Why tune the ISP for a RAW sensor?

Different light sources, also called illuminants, have different spectral distributions and color temperatures, measured in Kelvin, that can affect how the colors are captured by the camera sensor. The white balance adjustment (ISP gain) ensures that the whites appear white, and the color adjustment (color correction matrix) ensures that the colors in the captured image closely match the actual scene.

Tuning ISP parameters for different illuminants is a mandatory step in the camera application development process. It ensures that the sensor and lens connected to the device can produce good image quality in terms of white balance and color accuracy under different lighting conditions, meeting user application needs. The results of the tuning procedure are a set of parameters that are programmed in the ISP and controlled by ISP algorithms.

Important

To achieve good image quality, it is necessary to calibrate the ISP with light sources that have profiles close to real situations. Most of the time, the calibration is done with a light box capable of simulating different illuminant profiles.

Important

The ISP needs to be tuned when used with an RGB RAW sensor. With a monochrome RAW sensor, the ISP does not need to be tuned.

4. Tuning procedure

4.1. Prerequisite

The STM32 ISP IQTune application must be installed.

4.2. Needed materials

• X-Rite ColorChecker^® chart

A color checker is used in sensor tuning to ensure accurate color reproduction by providing a reference for adjusting the sensor's color response. It helps calibrate the sensor to match real-world colors, improving image quality and consistency. In the rest of the procedure, the term neutral gray patches refers to the six patches starting from the white patch to the black patch. Patch 1 is the top-left patch, and patch 24 is the bottom-right patch.

Important

The STM32 ISP IQTune analyzes the color rendering based on the X-Rite ColorChecker® edited after November 2014. The version before November 2014 has slight differences in the reference color values. For the rest of the procedure, it is expected to use the X-Rite ColorChecker® edited after November 2014.

• Spectrometer

A spectrometer is used in ISP tuning to measure the color values of light sources accurately. It provides precise color data, which is essential for calibrating the ISP to achieve accurate color reproduction. It is mainly used to obtain the color temperature of the scene.

• Light box

A light box is used in ISP tuning to provide a consistent and controlled lighting environment, ensuring uniform illumination for accurate color and exposure calibration. It helps eliminate external lighting variables, enabling precise adjustments.

4.3. Tuning step by step

To execute the tuning procedure, start from a blank page:

1. Click on the "Start tuning from scratch" button to begin with a blank configuration of the ISP.

Warning

The Start tuning from scratch must be executed twice to fully reset all the parameters. This is due to a wrong behavior of the reset function that will be fixed in the version 1.1.0 of the STM32 ISP IQTune tool.

4.3.1. Step 1: Configure the statistics removal

Description:
This module removes potential statistical data inserted by a sensor before starting the demosaicing conversion, which could be impacted by remaining statistics. It is typically used by sensors with a parallel interface, which cannot distinguish between statistics and pixel data. CSI-2 sensors typically split these flows using different virtual channel IDs.

Configuration procedure:

1. Enable or disable the statistic removal module.
2. When enabled, you can configure the number of first lines to skip and the number of lines to keep. These parameters depend on the sensor datasheet.
3. Apply the configuration.

4.3.2. Step 2: Configure the bad pixel

Description:
This module aims to detect and correct artifacts generated by a bad pixel on the sensor array. It is based on correlation extraction, not fixed locations.
It works for RAW Bayer flows only, as RGB flows are assumed to be already cleaned.
All arriving components are scanned, and the horizontal continuity correlation versus the two left and two right neighbors, on both the local component and the adjacent RAW Bayer component, is extracted:

• If a component is considered too light, it is declared as bad.
• Components that are too dark are not extracted to avoid false positives.

If a component is assumed to be bad, it is replaced by the horizontal average of its two neighbors (with the same component).
When the module is enabled, the user can configure the strength from 0 (high tolerance) to 7 (aggressive detection). In return, the 12-bit counter reports the number of bad components corrected. The counter resets every frame.

Important

A bad pixel software algorithm has been implemented by STMicroelectronics. This algorithm take as input a maximum number of pixels that the user might expect to detect; This is the threshold. The algorithm then updates the strength value of the bad pixel module to keep the amount of bad pixels detected close to the desired threshold.

Configuration procedure:

1. Enable the bad pixel module.
2. Once enabled, enable or disable the bad pixel software algorithm.
3. If the bad pixel algorithm is enabled, you can configure the threshold and in return be informed of the strength selected by the algorithm and the number of bad pixels detected. If the bad pixel algorithm is disabled, you can choose a fixed strength and in return be informed of the number of bad pixels detected.
4. Apply the configuration.

Information

The current manufacturing processes for image sensors minimize or eliminate defective pixels. Additionally, some image sensors feature automatic defective pixel correction, reducing the need for the ISP bad pixel module. We advise disabling this module if no bad pixel artifacts are detected.

4.3.3. Step 3: Configure the black level

Description:
This module suppresses a potential positive black-level offset, which occurs when the sensor sends non-null pixels while capturing a black picture. R, G, and B component offsets are subtracted from the R, G, and B Bayer components to achieve a black picture: R = G = B = 0.

Information

Most of the time, the black-level offset information can be found in the datasheet of the sensor. The following procedure can be used if this information is missing or to confirm it.

Configuration procedure:

1. Enable the black level module
2. Place a cap on the sensor (the preview frame is now black)
3. Open in the "Pixel Luminance" panel
4. Click on "Before demosaicing" to get the statistics before the demosaicing (in the RAW world)
5. Click on "Recalculate stats"
6. The "Average value of RGB & Luminance" graph is updated with the R, G, and B component averages that correspond to the black offset to be applied for each R, G, and B component. In this example, with the IMX335, we obtain R=12, G=12, and B=12.
7. Set the R, G, and B offset values
8. Apply the values by clicking the play button

Information

Multiple offset values can be tested by adding a new configuration using the "Add new Offset" button. To select the configuration, click the play button. The applied configuration appears highlighted in blue.

4.3.4. Step 4: Configure the demosaicing

Description:
The module operates on the pixel pipes. It implements a medium-quality conversion from RAW Bayer 8 bpp to RGB 24 bpp. The output RGB pixel rate matches the input component rate.
The demosaicing process takes the surrounding 3x3 components as input, applies linear heuristics to determine the most likely missing components, and generates an output RGB888 pixel. The underlying heuristic filters try to extract and adapt to edges, horizontal and vertical lines, and lone pixels (0: no detection, pure linear interpolation; 1 to 7: relative detection strength).

Configuration procedure:

1. Enable the demosaicing module.
2. Once enabled, you can update the strength of the filters used to detect edges, horizontal and vertical lines, and lone pixels. The default values provide good results.
3. Apply the configuration.

Information

The demosaicing "RAW config", "Bayer pattern", and "Color depth" are retrieved from the sensor information and are automatically set. If one of the parameters is incorrect, it is likely that the sensor driver does not provide the correct information.

4.3.5. Step 5: Configure the gamma correction

Description:
This module implements a gamma compression on each R, G, and B component using a static gamma exponent of 2.2 to store gamma-compressed pictures, as defined in the sRGB standard.

Information

To achieve good color rendering while dumping the buffer, it is important to enable gamma on both pipe1 and pipe2.

Configuration procedure:

1. Enable gamma correction for the pipe1
2. Enable gamma correction for the pipe2

4.3.6. Step 6: Configure auto white balance (AWB) profiles

Information

In the following description of the AWB profile configuration, a white balance profile for a given illuminant refers to a combination of two sets of parameters: the ISP gains associated with a color correction matrix (CCM).

Description:
This module interacts with the ISP gains and the color correction matrix block:

• The ISP gain equalizes the R, G, and B components by applying independent gains to each R, G, and B component to ensure white appears white. The G gain is the reference and is always 1. The R and B gains are updated using the G gain as reference.
• The color correction matrix converts the frame from the sensor RGB space to the sRGB space to achieve good color rendering.

Configuration procedure:

4.3.6.1. Prepare your white balance profile

1. Enable the white balance correction module.
2. Ensure that the AWB algorithm is disabled.
3. Create a new white balance profile. Newly created profiles always have all gains set to 1 and a CCM set with an identity matrix.
4. Name your profile and set the color temperature information box with the value retrieved using the spectrometer.
5. Apply the white balance profile.
6. Set the statistics area on the third neutral gray patch of the color checker.

Information

To configure the statistics area, click on the Live Feedback picture and drag the mouse to draw the statistics area. The statistics region coordinates are then updated and the red dashed rectangle is displayed on the Live Feedback picture as well as on the preview in the LCD display. The color of the statistics area can be changed by clicking on the button above the Live Feedback picture. Possible values are: Default, Light, or Dark.

4.3.6.2. Ensure the color checker is well exposed using the exposure control panel

1. Verify that the auto exposure algorithm is deactivated.
2. Open the "Pixel Luminance" panel.
3. Click "After demosaicing" to get the statistics after the demosaicing process (in the RGB world)
4. Update the "Sensor exposure time" to ensure that the luminance L (displayed in the "Average value of RGB & Luminance" graph) is around 56. The sensor gain must be as low as possible.

4.3.6.3. Configure the ISP gain to ensure white appears white

1. Take a RAW Bayer picture of the color checker (picture is saved as a .pgm file).
2. Open the "Analysis" panel.
3. Select "White Balance Gain" analysis.
4. Upload the RAW Bayer picture (.pgm file).
5. Start the white balance gain analysis. If the analysis fails, it is likely that the color checker is not detected because it is underexposed. In that case, slightly update the sensor exposure time and perform the steps again from step 1.
6. Get the R, G, and B ISP gains computed by the analysis tool.
7. Go back to the "White Balance" panel and set the R, G, and B gains of your white balance profile.
8. Apply the white balance profile.

4.3.6.4. Configure the color correction matrix for optimal color rendering

1. Take an RGB picture of the color checker with ISP gain applied (frame is saved as a .png file).
2. Open the "Analysis" panel.
3. Select "Color Correction Matrix" analysis.
4. Upload the RGB picture (.png file)
5. Start the CCM analysis.
6. Get the CCM coefficient that have been computed (you can use the copy button to save those values).
7. Go back to the "White Balance" panel and set the CCM coefficients (you can paste the coefficient values that have been copied)
8. Apply the white balance profile.

Important

Before applying the new coefficients of the CCM, verify that the CCM coefficients correspond to the identity matrix. If the CCM coefficients do not correspond to the identity matrix before applying the new coefficients, it is likely that you performed the CCM analysis based on a non-identity matrix, which will lead to an incorrect final color.

Information

At this stage, the color rendering for the selected illuminant is close to the reference.

4.3.6.5. Analyze the final ISP configuration to check color accuracy versus the reference colors

1. Take an RGB picture of the color checker with ISP gain and CCM applied (frame is saved as a .png file).
2. Open the "Analysis" panel.
3. Select "Advanced Image" analysis.
4. Upload the RGB picture (.png file).
5. Start the advanced image analysis.
6. The color analysis is complete, and you can verify the color accuracy with the different graph and information provided.
7. Optionally, you can download the analysis results as a zip file containing the graphs and .csv file.

Information

The color analysis provides information and metrics on how the colors are rendered according to the ISP configuration: DeltaE00 (ΔE00): color difference metric based on the CIEDE2000 formula measuring the overall color difference, including lightness, chroma, and hue. ΔE00 is informative, acceptance criteria is based on ΔC00 (see below) DeltaC00 (ΔC00): color difference metric based on the CIEDE2000 formula measuring only the chroma difference, ignoring lightness and hue differences. ΔC00 neutral gray patches acceptance criteria: ΔC00 < 5 ΔC00 other patches acceptance criteria: ΔC00 < 10 R/G and B/G ratio: neutrality of the gray patches in a color checker. For a neutral gray patch, the ideal ratios should be close to 1:1, indicating that the red, green, and blue channels are balanced, resulting in a neutral gray color. R/G ratio acceptance criteria: typically between 0.9 and 1.1 B/G ratio acceptance criteria: typically between 0.9 and 1.1 Exposure error: refers to the deviation of an image's exposure from the ideal exposure level. When using color checker charts, the reference color values are only accurate if the exposure is correct. Exposure error acceptance criteria: ± 0.250 f-stop If exposure error is not in the acceptance criteria, the sensor exposure needs to be increased or decreased accordingly. Description of the graph results: 3D bar graph representing the ΔC00 value of each patch compared to the reference value. A low bar indicates better color accuracy. Mean and max ΔE00 and ΔC00 are also provided. Graph providing information about the neutral gray patches: The top graph represents the log pixel level of each component (R, G, B) versus the neutral gray patch density. R, G, and B curves must be as close as possible for a good neutral gray patch rendering. This graph also provides the exposure error value and the measured value of the gamma. The bottom graph represents the ΔC00 versus the neutral gray patch density. Note: Density represents the extent to which light is absorbed. Visual representation of the ISP color rendering compared to the reference value (ISP color rendering is represented by the square in the middle of the reference color). .csv file with the complete data that permit to draw the graphs A, B, and C.

4.3.6.6. Enable the AWB algorithm

1. Select the white balance profile to be managed by the AWB algorithm.
2. Activate the AWB algorithm.

Information

The yellow tick box turns into the AWB logo when AWB is active. When AWB is active, the profile selected by the AWB algorithm is highlighted in blue.

4.3.7. Step 7: Configure the contrast (optional)

Description:
This module concludes the image processing pipeline. It enhances the contrast of the frame by non-linear emphasis of pixel luminance. It takes the RGB values from the pipeline, computes the luminance of the local pixel, and determines the desired luminance adjustment factor based on the input luminance using a lookup table. The output RGB components are clamped to 8 bits.
Configuration procedure:

1. Enable the contrast module
2. Select a predefined contrast profile
3. Optionally tune the LUT
4. Apply the contrast settings

4.3.8. Step 8: Configure the exposure control

Description:
This module aims to configure the sensor exposure to ensure the frame is well-exposed (neither too dark nor too bright). The exposure is not controlled by a parameter inside the ISP but by the sensor itself by updating the exposure time or the sensor gain, or both.
The user has the possibility to:

• Control the exposure manually by updating the gain or the exposure time, or both
• Enable the auto exposure algorithm
  • The algorithm automatically sets the appropriate sensor gain and exposure time based on statistics retrieved by the ISP and the ideal exposure target.
  • In this configuration, the user can control only the exposure compensation to shift the exposure target value.

Information

The exposure compensation is expressed in exposure value (EV). +1EV doubles the exposure target -1EV halves the exposure target

Configuration procedure:

1. Enable the auto exposure algorithm
2. Optionally change the exposure compensation value

4.3.9. Step 9: Export the ISP configuration

Description:
Once the ISP tuning is finalized, the ISP configuration needs to be exported so that the ISP settings and behaviors are reflected in the final application.

Configuration procedure:

1. Open the "My Configuration" panel
2. Click on "Export Configuration" and chose the .h code file format.

Information

The isp_param_conf.h generated file must be copied in the application project to take into account the ISP configuration created by the user.

4.4. Tips

4.4.1. Save and load your current work

4.4.2. Load a saved work

4.4.3. Sensor information at a glance

All the sensor information is summarized in the "Sensor Information" panel.

4.4.4. Current configuration at a glance

All ISP module status is summarized in the "My Configuration" panel.