1. Article purpose[edit | edit source]
The purpose of this article is to provide guidance on managing the ISP block within the DCMIPP , utilizing the Video4Linux2 (V4L2) framework on the Linux® OS.
2. Overview of DCMIPP ISP Block Diagram[edit | edit source]
2.1. Block diagram[edit | edit source]
The DCMIPP ISP, which is shared by both pipe1 and pipe2 and is instantiated only once, includes capabilities such as:
- Statistic removal: Eliminates unnecessary metadata, like statistics, from the images received.
- Bad-pixel removal: Automatically identifies and fixes bad pixels in the sensor array.
- Decimation: Reduces the number of pixels to a maximum of 2688 by selecting one pixel out of every 1, 2, 4, or 8.
- Black level correction: Adjusts the black level for each pixel in the red, green, and blue channels.
- Exposure control: Increases the exposure from 0.0 up to 255.99 times.
- Demosaicing: Transforms RawBayer format to RGB and supports NIR patterns of sizes 2x2 and 4x4.
- Statistics extract: Computes average values for red, green, blue, and luminance, including dynamic bins.
- Color conversion: Offers color conversion with flexible, fully programmable coefficients.
- Contrast:Enhances contrast through luminance adjustment.
Refer to the STM32 MPU reference manuals for the complete list of features.
2.2. The software topology[edit | edit source]
The diagram presented above illustrates the complete media-controller topology of the DCMIPP, including the ISP subdevice called dcmipp_main_isp.
Full control of the ISP is done by using both the v4l2-subdev pad API and dedicated video devices linked to the dcmipp_main_isp subdev.
All settings affecting the frame size or format are done via the dcmipp_main_isp pad (such as format, compose and crop settings), while others features are managed through two video device. The dcmipp_main_isp_params_output video output device allows to inject ISP block configuration while dcmipp_main_isp_stat_capture video capture device allows for control and capture of statistics data generated by the ISP.
3. DCMIPP ISP frame size & format control[edit | edit source]
3.1. Mapping between HW features and v4l2-subdev pad api[edit | edit source]
The table below outlines the association of features that can be accessed using the v4l2 sub-device pad API.
| Feature / Block | Integration with dcmipp_main_isp |
|---|---|
| Statistic removal | crop selection on pad[0] (SINK), with only the top and height fields valid |
| Decimation | compose selection on pad[0], with only width and height fields valid (SINK) |
| Demosaicing | Automatically activated when ISP block input format on pad[0] (SINK) is a RAW format |
3.1. Example (demosaicing / stat removal / decimation)[edit | edit source]
Following media-ctl tool output illustrate a configuration of the dcmipp_main_isp subdev allowing:
- demosaicing
- removal of top 10 lines and bottom 2 lines of statistics
- 2 time decimation of the frame
- entity 13: dcmipp_main_isp (4 pads, 5 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:SRGGB10_1X10/2592x1940 field:none colorspace:rec709
crop.bounds:(0,0)/2592x1940
crop:(0,10)/2592x1928
compose:(0,0)/1296x964]
<- "dcmipp_input":2 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:RGB888_1X24/1296x964 field:none colorspace:rec709]
-> "dcmipp_main_postproc":0 [ENABLED,IMMUTABLE]
-> "dcmipp_aux_postproc":0 []
pad2: Sink
[fmt:RGB888_1X24/1296x964 field:none colorspace:rec709]
<- "dcmipp_main_isp_params_output":0 [ENABLED,IMMUTABLE]
pad3: Source
[fmt:RGB888_1X24/1296x964 field:none colorspace:rec709]
-> "dcmipp_main_isp_stat_capture":0 [ENABLED,IMMUTABLE]
- On pad0, crop:(0,10)/2592x1928 indicates that the original frame should be cropped to a size of 2592x1928 starting at line 10. Since original frame size is 2592x1940, this leads to cut of first 10 lines and bottom 2 lines. Above example can be set using the following command:
media-ctl -d /dev/media1 --set-v4l2 "'dcmipp_main_isp':0[crop:(0,10)/2592x1928]"
- On same pad0, compose:(0,0)/1296x964 indicates to resize the frame (after the crop) to a size of 1296x964. On the dcmipp_main_isp subdev, this is done by decimation of the frame. Above example can be set using the following command:
media-ctl -d /dev/media1 --set-v4l2 "'dcmipp_main_isp':0[compose:(0,0)/1296x964]"
- pad0 format SRGGB10_1X10 is a RawBayer format, leading to automatic activation of the demosaicing block, which will thus generate a RGB888_1X24 on the output pad (pad1) of the subdev. Frame size is also automatically set to the size of the compose.
4. DCMIPP ISP block parameter video output device dcmipp_main_isp_params_output[edit | edit source]
The dcmipp_main_isp_params_output video output device allows for configuration of other features of the DCMIPP ISP not already configurated via the v4l2_subdev pad api. That is:
- bad pixel removal
- black level correction
- exposure control
- demosaicing filters control
- color conversion
- contrast enhancement
Parameters are injected into the DCMIPP ISP via V4L2 buffer of type V4L2_META_FMT_ST_DCMIPP_ISP_PARAMS negociated with the dcmipp_main_isp_params_output video device.
#define V4L2_META_FMT_ST_DCMIPP_ISP_PARAMS v4l2_fourcc('S', 'T', 'I', 'P') /* STM32 DCMIPP ISP Parameters */
Once the application have allocated buffers via the driver V4L2_REQBUFS ioctl, it needs to fill in those buffers with struct stm32_dcmipp_params_cfg structures. Requesting setting application is done by using the V4L2_QBUF ioctl, using a traditionnal V4L2 buffer handling mechanism.
4.1. Parameter structure description (struct stm32_dcmipp_params_cfg)[edit | edit source]
The main structure is made a bitfield module_cfg_update indicating which parameter is valid within the structure, allowing to apply one or more parameters at once. Valid fields are described below.
/* Bad Pixel Removal */ #define STM32_DCMIPP_ISP_BPR (1U << 0) /* Black Level Correction */ #define STM32_DCMIPP_ISP_BLC (1U << 1) /* Exposure Control */ #define STM32_DCMIPP_ISP_EX (1U << 2) /* Demosaicing filters */ #define STM32_DCMIPP_ISP_DM (1U << 3) /* Color conversion Control */ #define STM32_DCMIPP_ISP_CC (1U << 4) /* Contrast Enhancement */ #define STM32_DCMIPP_ISP_CE (1U << 5)
The other element of the structure is struct stm32_dcmipp_isp_ctrls_cfg which contains structures for each configurable feature.
/** * struct stm32_dcmipp_params_cfg - STM32 DCMIPP ISP Input Parameters Meta Data * * @module_cfg_update: mask the config bits of which module should be updated * @ctrls: configuration of other ISP blocks */ struct stm32_dcmipp_params_cfg { __u32 module_cfg_update; struct stm32_dcmipp_isp_ctrls_cfg ctrls; }; /** * struct stm32_dcmipp_isp_ctrls_cfg - STM32 DCMIPP ISP Controls * * @bpr_cfg: configuration of the bad pixel removal block * @blc_cfg: configuration of the black level correction block * @ex_cfg: configuration of the exposure block * @dm_cfg: configuration of the demosaicing filters block * @cc_cfg: configuration of the color conversion block * @ce_cfg: configuration of the contrast enhancement block */ struct stm32_dcmipp_isp_ctrls_cfg { struct stm32_dcmipp_isp_bpr_cfg bpr_cfg; struct stm32_dcmipp_isp_blc_cfg blc_cfg; struct stm32_dcmipp_isp_ex_cfg ex_cfg; struct stm32_dcmipp_isp_dm_cfg dm_cfg; struct stm32_dcmipp_isp_cc_cfg cc_cfg; struct stm32_dcmipp_isp_ce_cfg ce_cfg; };
Further details about each element of this structure can be found within the stm32-dcmipp-config.h [1] file.
4.2. Example[edit | edit source]
The application Unsupported domain! gives an example of the usage of ISP params interface. Function set_profile within Unsupported domain! contains buffer handling allowing to set exposure & color conversion parameters of the DCMIPP ISP. As an example below code
/* Exposure / colorconv settings for D50 profile */
const float exposure_D50[3] = { 2.2, 1.0, 1.8 };
const float colorconv_D50[3][3] =
{ { 1.8008, -0.6484, -0.1523 },
{ -0.3555, 1.6992, -0.3438 },
{ 0.0977, -0.957, 1.8594 } };
float exposure[3], colorconv[3][3];
struct stm32_dcmipp_params_cfg params = {
.module_cfg_update = STM32_DCMIPP_ISP_BLC |
STM32_DCMIPP_ISP_EX |
STM32_DCMIPP_ISP_CC,
.ctrls = {
/* IMX335 black level set to 12 */
.blc_cfg = {
.en = 1,
.blc_r = 12,
.blc_g = 12,
.blc_b = 12,
},
},
};
struct stm32_dcmipp_isp_ex_cfg *exposure_wb = ¶ms.ctrls.ex_cfg;
struct stm32_dcmipp_isp_cc_cfg *cconv = ¶ms.ctrls.cc_cfg;
int i, j, ret;
memcpy(exposure, exposure_D50, sizeof(exposure));
memcpy(colorconv, colorconv_D50, sizeof(colorconv));
/* Set exposure */
to_shift_mult_float(exposure[0], &exposure_wb->shift_r, &exposure_wb->mult_r);
to_shift_mult_float(exposure[1], &exposure_wb->shift_g, &exposure_wb->mult_g);
to_shift_mult_float(exposure[2], &exposure_wb->shift_b, &exposure_wb->mult_b);
exposure_wb->en = 1;
/* Set colorconv */
cconv->rr = to_cconv_reg(colorconv[0][0]);
cconv->rg = to_cconv_reg(colorconv[0][1]);
cconv->rb = to_cconv_reg(colorconv[0][2]);
cconv->gr = to_cconv_reg(colorconv[1][0]);
cconv->gg = to_cconv_reg(colorconv[1][1]);
cconv->gb = to_cconv_reg(colorconv[1][2]);
cconv->br = to_cconv_reg(colorconv[2][0]);
cconv->bg = to_cconv_reg(colorconv[2][1]);
cconv->bb = to_cconv_reg(colorconv[2][2]);
cconv->en = 1;
cconv->clamp = STM32_DCMIPP_ISP_CC_CLAMP_DISABLED;
ret = apply_params(isp_desc, ¶ms);
if (ret)
printf("Failed to apply exposure\n");
5. DCMIPP ISP statistics control & capture video device dcmipp_main_isp_stat_capture[edit | edit source]
The dcmipp_main_isp_stat_capture video capture device allows for capture of statistics data generated by the DCMIPP ISP. Statistics information contains average / bins and bad pixel count information. Depending on the profile selected (cf V4L2_CID_ISP_STAT_PROFILE control below), statistics are not available during several frames upon starting a pipeline.
Statistics can be captured at two places within the pipeline, either before black level correction block or after demosaicing block. Since not all statistics can be generated at once, a control V4L2_CID_ISP_STAT_PROFILE is available to select the amount of statistics to be made available within a capture buffer, thus affecting the necessary period in order to get a fully updated statistics structure.
5.1. Statistics buffer format[edit | edit source]
Statistics generated by the DCMIPP are available via the dcmipp_main_isp_stat_capture V4L2 device. The DCMIPP generated two types of statistics AVERAGE and BINS which are made available via the struct stm32_dcmipp_stat_buf described below.
/**
* struct stm32_dcmipp_stat_avr_bins - average & bins statistics
*
* @average_rgb[3]: average value of R/G/B components
* @bins[12]: 12 values histogram
*/
struct stm32_dcmipp_stat_avr_bins {
__u32 average_RGB[3];
__u32 bins[12];
};
/**
* struct stm32_dcmipp_stat_buf - statistics buffer
*
* @pre: average & bins statistics at pre-demosaicing location
* @post: average & bins statistics at post-demosaicing location
* @bad_pixel_count: number of bad pixels detected in the frame
*/
struct stm32_dcmipp_stat_buf {
struct stm32_dcmipp_stat_avr_bins pre;
struct stm32_dcmipp_stat_avr_bins post;
__u32 bad_pixel_count;
};
Each buffer extract of the V4L2 device (of type V4L2_BUF_TYPE_META_CAPTURE) will contain one struct stm32_dcmipp_stat_buf, using a dedicated dataformat V4L2_META_FMT_ST_ISP_STAT.
5.2. Statistics controls[edit | edit source]
The dcmipp_main_isp_stat_capture V4L2 device also exposes controls in order to tune the statistics extraction:
- ISP stat region control (V4L2_CID_ISP_STAT_REGION): control the statistics extraction area. Allows to define the area of the frame from which statistics are processed. The following struct v4l2_ctrl_isp_stat_region is used to set the control. For the time being there must be only a unique region hence nb_regions should be set to 1.
/**
* struct v4l2_ctrl_isp_stat_region - Region where ISP statistics are collected
*
* @nb_regions: number of regions
* @top: top coordinate of a region
* @left: left coordinate of a region
* @width: width of a region
* @height: height of a region
*/
struct v4l2_ctrl_isp_stat_region {
__u8 nb_regions;
__u32 top[25];
__u32 left[25];
__u32 width[25];
__u32 height[25];
};
- ISP stat profile selection (V4L2_CID_ISP_STAT_PROFILE): control the amount of statistics to be captured.
- V4L2_STAT_PROFILE_FULL: all statistics are captured
- V4L2_STAT_PROFILE_AVERAGE_PRE: only average statistics, taken before black level correction
- V4L2_STAT_PROFILE_AVERAGE_POST: only average statistics, taken after demosaicing
- ISP stat average filter control (V4L2_CID_ISP_STAT_AVG_FILTER): control the filtering of pixels in the statistics AVERAGE processing
- V4L2_STAT_AVG_FILTER_NONE: no filtering performed, all pixels used for average calculation
- V4L2_STAT_AVG_FILTER_EXCL16: pixels with value < 16 or >= 240 are discarded from average calculation
- V4L2_STAT_AVG_FILTER_EXCL32: pixels with value < 32 or >= 224 are discarded from average calculation
- V4L2_STAT_AVG_FILTER_EXCL64: pixels with value < 64 or >= 192 are discarded from average calculation
- ISP stat bin component control (V4L2_CID_ISP_STAT_BIN_COMP): control which component is used for BINS processing (R/G/B or Luminance)
- V4L2_STAT_BIN_COMP_R: R component used
- V4L2_STAT_BIN_COMP_G: G component used
- V4L2_STAT_BIN_COMP_B: B component used
- V4L2_STAT_BIN_COMP_L: luminance used
5.3. Example[edit | edit source]
The application Unsupported domain! gives an example of the usage of ISP statistics interface. Function get_stat within Unsupported domain! contains buffer handling allowing to read statistics from the DCMIPP ISP.