1. Article purpose[edit | edit source]
The purpose of this article is to provide guidance on managing the ISP block within the DCMIPP, utilizing the Video4Linux2 (V4L2) framework on the Linux® OS.
2. Overview of DCMIPP ISP block diagram[edit | edit source]
2.1. Block diagram[edit | edit source]
The DCMIPP ISP is shared by both pipe1 and pipe2, is instantiated only once, and includes capabilities such as:
- Statistic removal: Eliminates unnecessary metadata, like statistics, from the images received.
- Bad-pixel removal: Automatically identifies and fixes bad pixels in the sensor array.
- Decimation: Reduces the number of pixels to a maximum of 2688 by selecting one pixel out of every 1, 2, 4, or 8.
- Black level correction: Adjusts the black level for each pixel in the red, green, and blue channels.
- Exposure control: Increases the exposure from 0.0 up to 255.99 times.
- Demosaicing: Transforms RawBayer format to RGB and supports NIR patterns of sizes 2x2 and 4x4.
- Statistics extract: Computes average values for red, green, blue, and luminance, including dynamic bins.
- Color conversion: Offers color conversion with flexible, fully programmable coefficients.
- Contrast: Enhances contrast through luminance adjustment.
Refer to the STM32 MPU reference manuals for the complete list of features.
2.2. The software topology[edit | edit source]
The diagram presented above illustrates the complete media-controller topology of the DCMIPP, including the ISP subdevice called dcmipp_main_isp.
Full control of the ISP is done by using both the v4l2-subdev pad API and dedicated video devices linked to the dcmipp_main_isp subdev.
All settings affecting the frame size or format are done via the dcmipp_main_isp pad (such as format, compose, and crop settings), while other features are managed through two video devices. The dcmipp_main_isp_params_output video output device allows to inject ISP block configuration while dcmipp_main_isp_stat_capture video capture device allows for control and capture of statistics data generated by the ISP.
3. DCMIPP ISP frame size & format control[edit | edit source]
3.1. Mapping between hardware features and v4l2-subdev pad API[edit | edit source]
The table below outlines the association of features that can be accessed using the v4l2 subdevice pad API.
| Feature / block | Integration with dcmipp_main_isp |
|---|---|
| Statistic removal | crop the selection on pad[0] (SINK), with only the top and height fields valid. |
| Decimation | compose the selection on pad[0] (SINK), with only width and height fields valid. |
| Demosaicing | Automatically activated when ISP block input format on pad[0] (SINK) is a RAW format. |
3.1. Example (demosaicing / stat removal / decimation)[edit | edit source]
Following media-ctl tool output illustrate a configuration of the dcmipp_main_isp subdev allowing:
- Demosaicing.
- Removal of the top 10 lines and bottom 2 lines of statistics.
- 2 time decimation of the frame.
- entity 13: dcmipp_main_isp (4 pads, 5 links, 0 routes)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[stream:0 fmt:RGB565_2X8_LE/640x480 field:none colorspace:rec709
crop.bounds:(0,0)/640x480
crop:(0,0)/0x0
compose.bounds:(0,0)/0x0
compose:(0,0)/0x0]
<- "dcmipp_input":2 []
pad1: Source
[stream:0 fmt:RGB888_1X24/640x480 field:none colorspace:rec709]
-> "dcmipp_main_postproc":0 [ENABLED,IMMUTABLE]
-> "dcmipp_aux_postproc":0 []
pad2: Sink
[stream:0 fmt:unknown/0x0]
<- "dcmipp_main_isp_params_output":0 [ENABLED,IMMUTABLE]
pad3: Source
[stream:0 fmt:unknown/0x0]
-> "dcmipp_main_isp_stat_capture":0 [ENABLED,IMMUTABLE]
- On pad0, crop:(0,10)/2592x1928 indicates that the original frame should be cropped to a size of 2592x1928 starting at line 10. Since original frame size is 2592x1940, this leads to cut of the first 10 lines and bottom 2 lines. The above example can be set using the following command:
media-ctl -d /dev/media1 --set-v4l2 "'dcmipp_main_isp':0[crop:(0,10)/2592x1928]"
- On same pad0, compose:(0,0)/1296x964 indicates to resize the frame (after the crop) to a size of 1296x964. On the dcmipp_main_isp subdev, this is done by decimation of the frame. The above example can be set using the following command:
media-ctl -d /dev/media1 --set-v4l2 "'dcmipp_main_isp':0[compose:(0,0)/1296x964]"
- pad0 format SRGGB10_1X10 is a RawBayer format, leading to automatic activation of the demosaicing block, which will thus generate a RGB888_1X24 on the output pad (pad1) of the subdev. Frame size is also automatically set to the size of the compose.
4. DCMIPP ISP block parameter video output device dcmipp_main_isp_params_output[edit | edit source]
The dcmipp_main_isp_params_output video output device allows for configuration of other features of the DCMIPP ISP not already configurated via the v4l2_subdev pad API. That is:
- Bad pixel removal
- Black level correction
- Exposure control
- Demosaicing filters control
- Color conversion
- Contrast enhancement
Parameters are injected into the DCMIPP ISP via V4L2 buffer of type V4L2_META_FMT_ST_DCMIPP_ISP_PARAMS negotiated with the dcmipp_main_isp_params_output video device.
#define V4L2_META_FMT_ST_DCMIPP_ISP_PARAMS v4l2_fourcc('S', 'T', 'I', 'P') /* STM32 DCMIPP ISP Parameters */
Once the application have allocated buffers via the driver V4L2_REQBUFS ioctl, it needs to fill in those buffers with struct stm32_dcmipp_params_cfg structures. Requesting setting application is done by using the V4L2_QBUF ioctl, using a traditional V4L2 buffer handling mechanism.
4.1. Parameter structure description (struct stm32_dcmipp_params_cfg)[edit | edit source]
The main structure is made a bitfield module_cfg_update indicating which parameter is valid within the structure, allowing to apply one or more parameters at once. Valid fields are described below.
/* Bad Pixel Removal */
#define STM32_DCMIPP_ISP_BPR (1U << 0)
/* Black Level Correction */
#define STM32_DCMIPP_ISP_BLC (1U << 1)
/* Exposure Control */
#define STM32_DCMIPP_ISP_EX (1U << 2)
/* Demosaicing filters */
#define STM32_DCMIPP_ISP_DM (1U << 3)
/* Color conversion Control */
#define STM32_DCMIPP_ISP_CC (1U << 4)
/* Contrast Enhancement */
#define STM32_DCMIPP_ISP_CE (1U << 5)
The other element of the structure is struct stm32_dcmipp_isp_ctrls_cfg, which contains structures for each configurable feature.
/**
* struct stm32_dcmipp_params_cfg - STM32 DCMIPP ISP Input Parameters Meta Data
*
* @module_cfg_update: mask the config bits of which module should be updated
* @ctrls: configuration of other ISP blocks
*/
struct stm32_dcmipp_params_cfg {
__u32 module_cfg_update;
struct stm32_dcmipp_isp_ctrls_cfg ctrls;
};
/**
* struct stm32_dcmipp_isp_ctrls_cfg - STM32 DCMIPP ISP Controls
*
* @bpr_cfg: configuration of the bad pixel removal block
* @blc_cfg: configuration of the black level correction block
* @ex_cfg: configuration of the exposure block
* @dm_cfg: configuration of the demosaicing filters block
* @cc_cfg: configuration of the color conversion block
* @ce_cfg: configuration of the contrast enhancement block
*/
struct stm32_dcmipp_isp_ctrls_cfg {
struct stm32_dcmipp_isp_bpr_cfg bpr_cfg;
struct stm32_dcmipp_isp_blc_cfg blc_cfg;
struct stm32_dcmipp_isp_ex_cfg ex_cfg;
struct stm32_dcmipp_isp_dm_cfg dm_cfg;
struct stm32_dcmipp_isp_cc_cfg cc_cfg;
struct stm32_dcmipp_isp_ce_cfg ce_cfg;
};
Further details about each element of this structure can be found within the stm32-dcmipp-config.h [1] file.
4.2. Example[edit | edit source]
The application dcmipp-isp-ctrl gives an example of the usage of ISP params interface. Function set_profile within dcmipp-isp-ctrl/dcmipp-isp-ctrl.c contains buffer handling allowing to set exposure and color conversion parameters of the DCMIPP ISP. As an example, see the code below.
/* Exposure / colorconv settings for D50 profile */
const float exposure_D50[3] = { 2.2, 1.0, 1.8 };
const float colorconv_D50[3][3] =
{ { 1.8008, -0.6484, -0.1523 },
{ -0.3555, 1.6992, -0.3438 },
{ 0.0977, -0.957, 1.8594 } };
float exposure[3], colorconv[3][3];
struct stm32_dcmipp_params_cfg params = {
.module_cfg_update = STM32_DCMIPP_ISP_BLC |
STM32_DCMIPP_ISP_EX |
STM32_DCMIPP_ISP_CC,
.ctrls = {
/* IMX335 black level set to 12 */
.blc_cfg = {
.en = 1,
.blc_r = 12,
.blc_g = 12,
.blc_b = 12,
},
},
};
struct stm32_dcmipp_isp_ex_cfg *exposure_wb = ¶ms.ctrls.ex_cfg;
struct stm32_dcmipp_isp_cc_cfg *cconv = ¶ms.ctrls.cc_cfg;
int i, j, ret;
memcpy(exposure, exposure_D50, sizeof(exposure));
memcpy(colorconv, colorconv_D50, sizeof(colorconv));
/* Set exposure */
to_shift_mult_float(exposure[0], &exposure_wb->shift_r, &exposure_wb->mult_r);
to_shift_mult_float(exposure[1], &exposure_wb->shift_g, &exposure_wb->mult_g);
to_shift_mult_float(exposure[2], &exposure_wb->shift_b, &exposure_wb->mult_b);
exposure_wb->en = 1;
/* Set colorconv */
cconv->rr = to_cconv_reg(colorconv[0][0]);
cconv->rg = to_cconv_reg(colorconv[0][1]);
cconv->rb = to_cconv_reg(colorconv[0][2]);
cconv->gr = to_cconv_reg(colorconv[1][0]);
cconv->gg = to_cconv_reg(colorconv[1][1]);
cconv->gb = to_cconv_reg(colorconv[1][2]);
cconv->br = to_cconv_reg(colorconv[2][0]);
cconv->bg = to_cconv_reg(colorconv[2][1]);
cconv->bb = to_cconv_reg(colorconv[2][2]);
cconv->en = 1;
cconv->clamp = STM32_DCMIPP_ISP_CC_CLAMP_DISABLED;
ret = apply_params(isp_desc, ¶ms);
if (ret)
printf("Failed to apply exposure\n");
5. DCMIPP ISP statistics control and capture video device dcmipp_main_isp_stat_capture[edit | edit source]
The dcmipp_main_isp_stat_capture video capture device allows for capture of statistics data generated by the DCMIPP ISP. Statistics information contains average / bins and bad pixel count information. Depending on the profile selected (cf V4L2_CID_ISP_STAT_PROFILE control below), statistics are not available during several frames upon starting a pipeline.
Statistics can be captured at two places within the pipeline, either before the black level correction block or after the demosaicing block. Since not all statistics can be generated at once, a control V4L2_CID_ISP_STAT_PROFILE is available to select the amount of statistics to be made available within a capture buffer, thus affecting the necessary period to get a fully updated statistics structure.
5.1. Statistics buffer format[edit | edit source]
Statistics generated by the DCMIPP are available via the dcmipp_main_isp_stat_capture V4L2 device. The DCMIPP generated two types of statistics AVERAGE and BINS that are made available via the struct stm32_dcmipp_stat_buf described below.
/**
* struct stm32_dcmipp_stat_avr_bins - average & bins statistics
*
* @average_rgb[3]: average value of R/G/B components
* @bins[12]: 12 values histogram
*/
struct stm32_dcmipp_stat_avr_bins {
__u32 average_RGB[3];
__u32 bins[12];
};
/**
* struct stm32_dcmipp_stat_buf - statistics buffer
*
* @pre: average & bins statistics at pre-demosaicing location
* @post: average & bins statistics at post-demosaicing location
* @bad_pixel_count: number of bad pixels detected in the frame
*/
struct stm32_dcmipp_stat_buf {
struct stm32_dcmipp_stat_avr_bins pre;
struct stm32_dcmipp_stat_avr_bins post;
__u32 bad_pixel_count;
};
Each buffer extract of the V4L2 device (of type V4L2_BUF_TYPE_META_CAPTURE) contains one struct stm32_dcmipp_stat_buf, using a dedicated data format V4L2_META_FMT_ST_ISP_STAT.
5.2. Statistics controls[edit | edit source]
The dcmipp_main_isp_stat_capture V4L2 device also exposes controls to tune the statistics extraction:
- ISP stat region control (V4L2_CID_ISP_STAT_REGION): control the statistics extraction area. Allows to define the area of the frame from which statistics are processed. The following struct v4l2_ctrl_isp_stat_region is used to set the control. For the time being there must be only a unique region hence nb_regions should be set to 1.
/**
* struct v4l2_ctrl_isp_stat_region - Region where ISP statistics are collected
*
* @nb_regions: number of regions
* @top: top coordinate of a region
* @left: left coordinate of a region
* @width: width of a region
* @height: height of a region
*/
struct v4l2_ctrl_isp_stat_region {
__u8 nb_regions;
__u32 top[25];
__u32 left[25];
__u32 width[25];
__u32 height[25];
};
- ISP stat profile selection (V4L2_CID_ISP_STAT_PROFILE): control the amount of statistics to be captured.
- V4L2_STAT_PROFILE_FULL: all statistics are captured.
- V4L2_STAT_PROFILE_AVERAGE_PRE: only average statistics, taken before black level correction.
- V4L2_STAT_PROFILE_AVERAGE_POST: only average statistics, taken after demosaicing.
- ISP stat average filter control (V4L2_CID_ISP_STAT_AVG_FILTER): control the filtering of pixels in the statistics AVERAGE processing.
- V4L2_STAT_AVG_FILTER_NONE: no filtering performed, all pixels are used for average calculation.
- V4L2_STAT_AVG_FILTER_EXCL16: pixels with value < 16 or >= 240 are discarded from average calculation.
- V4L2_STAT_AVG_FILTER_EXCL32: pixels with value < 32 or >= 224 are discarded from average calculation.
- V4L2_STAT_AVG_FILTER_EXCL64: pixels with value < 64 or >= 192 are discarded from average calculation.
- ISP stat bin component control (V4L2_CID_ISP_STAT_BIN_COMP): control which component is used for BINS processing (R, G, B, or luminance).
- V4L2_STAT_BIN_COMP_R: R component is used.
- V4L2_STAT_BIN_COMP_G: G component is used.
- V4L2_STAT_BIN_COMP_B: B component is used.
- V4L2_STAT_BIN_COMP_L: luminance is used.
5.3. Example[edit | edit source]
The application dcmipp-isp-ctrl gives an example of the usage of ISP statistics interface. Function get_stat within dcmipp-isp-ctrl/dcmipp-isp-ctrl.c contains buffer handling allowing to read statistics from the DCMIPP ISP.