Simple ISP preview

This article explains how to launch a simple preview with ISP running for a better image quality rendering on the STM32N6570-DK board.

1. Description

To stream a camera and to display a preview with ISP enabled and 2A algorithms running, two examples are available.

The first one is part of the N6 STM32Cube Firmware applications and calls directly APIs of ISP middleware.
The second one is included in the X-Cube-ISP package and uses Camera middlleware instead of ISP middleware. Camera Middleware will then take care of running ISP services.

2. Prerequisites

• Hardware
  • STM32N6 discovery board
  • Discovery MB1860- STM32N6 (need USBC cable)
  • IMX335 camera module MB1854B

• Required tools
  • STM32CubeN6 FW version 1.0.0 or later for the first example
  • X-CUBE-ISP package v1.0.0 for the second example
  • STM32CubeIDE (STM32CubeIDE 1.15.1.24-N6-A2)

3. STM32Cube Firmware Simple preview

This application streams the IMX335 sensor, runs ISP 2A algorithms and display a stretched preview on full screen LCD in 800x480.

3.1. How to get application project

• Download the STM32Cube_FW_N6 Cube firmware
• Application example is located in "STM32Cube_FW_N6\Projects"\STM32N6570-DK\Applications\DCMIPP\DCMIPP_ContinuousMode"
• Import DCMIPP_ContinuousMode project in the STM32CubeIDE to run the application

3.2. Project structure

Here is the general structure of the DCMIPP_ContinuousMode project:

Application
|--> main.c                    Main application entry point, ISP helpers and DCMIPP callbacks
|--> ...
Middleware
|--> ISP_Library               ISP middleware
|    |--> evision              AE and AWB libraries
|    |--> isp_algo.c           IQ algorithms
|    |--> isp_core.c           Core and API
|    |--> isp_services.c       Internal services
Drivers                        Including IMX335 sensor driver, DCMIPP, CSI and LCD HAL drivers
Includes
|--> project_path/Inc
|    |--> imx335_E27_isp_param_conf.h     ISP IQ configuration parameters

3.3. Code implementation

The main.c file of the application contains the minimal implementation for simple ISP preview with the IMX335 sensor. This paragraph details the most important points of the code implementation for using ISP middleware.

It starts the camera and the ISP services and defines the mandatory helper functions of the ISP for Camera controls.

For more information, the main ISP APIs are detailed in the ISP midlleware article.

3.3.1. main function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/**
* @brief  The application entry point.
* @retval int
*/
int main(void)
{
  ISP_AppliHelpersTypeDef appliHelpers = {0};
  ISP_StatAreaTypeDef statArea = {0};

  ... /* Other system initializations here */

  /* Initialize all configured peripherals */
   MX_DCMIPP_Init();

  /* Initialize the IMX335 Sensor ----------------------------- */
  IMX335_Probe(IMX335_R2592_1944, IMX335_RAW_RGGB10);

  LCD_Init(FRAME_WIDTH, FRAME_HEIGHT);

  /* Fill init struct with Camera driver helpers */
  appliHelpers.GetSensorInfo = GetSensorInfoHelper;
  appliHelpers.SetSensorGain = SetSensorGainHelper;
  appliHelpers.GetSensorGain = GetSensorGainHelper;
  appliHelpers.SetSensorExposure = SetSensorExposureHelper;
  appliHelpers.GetSensorExposure = GetSensorExposureHelper;

  statArea.X0 = 0;
  statArea.Y0 = 0;
  statArea.XSize = 2592;
  statArea.YSize = 1944;

  /* Initialize the Image Signal Processing middleware */
  if(ISP_Init(&hcamera_isp, &hdcmipp, 0, &appliHelpers, &statArea, ISP_IQParamCacheInit[0]) != ISP_OK)
  {
    Error_Handler();
  }

  if (HAL_DCMIPP_CSI_PIPE_Start(&hdcmipp, DCMIPP_PIPE1, DCMIPP_VIRTUAL_CHANNEL0 , BUFFER_ADDRESS, DCMIPP_MODE_CONTINUOUS) != HAL_OK)
  {
    Error_Handler();
  }

  /* Start the Image Signal Processing */
  if (ISP_Start(&hcamera_isp) != ISP_OK)
  {
    Error_Handler();
  }

  /* Infinite loop */
  while (1)
  { ...
    if (ISP_BackgroundProcess(&hcamera_isp) != ISP_OK)
    { ...
    }
  }
}

• "MX_DCMIPP_Init": Initialize CSI and DCMIPP (only configure non-ISP settings)
• "IMX335_Probe": Read component ID and initialize camera in full resolution
• "HAL_DCMIPP_CSI_PIPE_Start": Direct call to CSI interface to start streaming
  
• ISP:
  • "appliHelpers.Get/Setxxx": Define ISP application helpers providing control of non ISP features (mandatory for sensor info and gain/exposure control)
  • "statArea.xxx": Set statistics windows top-left and size
  • "ISP_Init": Initialize the ISP middleware instance using isp parameters from imx335_E27_isp_param_conf.h
  • "ISP_Start": Start processing
  • "ISP_BackgroundProcess": Run the background process

3.3.2. helper functions

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/**
* @brief  ISP Middleware helper. Camera sensor info getter
* @retval ISP Status
*/
static ISP_StatusTypeDef GetSensorInfoHelper(uint32_t Instance, ISP_SensorInfoTypeDef *SensorInfo)
{
  UNUSED(Instance);
  return (ISP_StatusTypeDef) IMX335_GetSensorInfo(&IMX335Obj, (IMX335_SensorInfo_t *) SensorInfo);
}

/**
* @brief  ISP Middleware helper. Camera gain setter
* @retval ISP Status
*/
static ISP_StatusTypeDef SetSensorGainHelper(uint32_t Instance, int32_t Gain)
{
  UNUSED(Instance);
  isp_gain = Gain;
  return (ISP_StatusTypeDef) IMX335_SetGain(&IMX335Obj, Gain);
}

/**
* @brief  ISP Middleware helper. Camera gain getter
* @retval ISP Status
*/
static ISP_StatusTypeDef GetSensorGainHelper(uint32_t Instance, int32_t *Gain)
{
  UNUSED(Instance);
  *Gain = isp_gain;
  return ISP_OK;
}

/**
* @brief  ISP Middleware helper. Camera exposure setter
* @retval ISP Status
*/
static ISP_StatusTypeDef SetSensorExposureHelper(uint32_t Instance, int32_t Exposure)
{
  UNUSED(Instance);
  isp_exposure = Exposure;
  return (ISP_StatusTypeDef) IMX335_SetExposure(&IMX335Obj, Exposure);
}

/**
* @brief  ISP Middleware helper. Camera exposure getter
* @retval ISP Status
*/
static ISP_StatusTypeDef GetSensorExposureHelper(uint32_t Instance, int32_t *Exposure)
{
  UNUSED(Instance);
  *Exposure = isp_exposure;
  return ISP_OK;
}

• "SetSensorxxx": Call camera interface for setting gain and exposure controls
• "GetSensorxxx": Get sensor info and gain or exposure value

3.3.3. DCMIPP callback functions

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

void HAL_DCMIPP_PIPE_FrameEventCallback(DCMIPP_HandleTypeDef *hdcmipp, uint32_t Pipe)
{
  NbMainFrames++;
}

/**
* @brief  Vsync Event callback on pipe
* @param  hdcmipp DCMIPP device handle
*         Pipe    Pipe receiving the callback
* @retval None
*/
void HAL_DCMIPP_PIPE_VsyncEventCallback(DCMIPP_HandleTypeDef *hdcmipp, uint32_t Pipe)
{
  UNUSED(hdcmipp);
  /* Update the frame counter and call the ISP statistics handler */
  switch (Pipe)
  {
    case DCMIPP_PIPE0 :
      ISP_IncDumpFrameId(&hcamera_isp);
      break;
    case DCMIPP_PIPE1 :
      ISP_IncMainFrameId(&hcamera_isp);
      ISP_GatherStatistics(&hcamera_isp);
      break;
    case DCMIPP_PIPE2 :
      ISP_IncAncillaryFrameId(&hcamera_isp);
      break;
  }
}

DCMIPP component uses two callback functions. In this implementation, only "HAL_DCMIPP_PIPE_VsyncEventCallback" is enriched with ISP interface calls for each pipe. In simple preview application, only main pipe is used.

• "ISP_IncMainFrameId": Increment ISP Middleware frame counter for main pipe
• "ISP_GatherStatistics": Call the ISP statistics handler

4. Camera Middleware Simple preview

This application streams the IMX335 sensor, runs ISP 2A algorithms and display a preview that fits the LCD in 640x480 using Camera Middleware services.

4.1. How to get application project

• Presently, a simple preview application using ISP services through Camera Middleware is available in X-CUBE-ISP package.
• Download the package then follow instructions from Path: X-CUBE-ISP_v1.0.0/README.md to import STM32N6_ISP_IQTune_App project in STM32CubeIDE.
• Remove ISP_MW_TUNING_TOOL_SUPPORT definition from the project preprocessor settings as follow:

4.2. Project structure

Here is the general structure of the STM32N6_ISP_IQTune_App project without ISP_MW_TUNING_TOOL_SUPPORT:

Application
|--> main.c                    Main application entry point, ISP helpers and DCMIPP callbacks
|--> ...
Middleware
|--> Camera_Middleware         Camera middleware
|    |--> ISP_Library          ISP middleware included in Camera Middleware
|    |--> sensors              Interfaces and drivers of supported sensors
|    |    |--> drivers         Sensor drivers
|    |    |    |--> imx335.c   IMX335 sensor driver
|    |    |--> cmw_imx335.c    IMX335 sensor interface
|    |--> cmw_camera.c         Core and API of Camera middleware
|    |--> cmw_utils.c          Camera Middleware Utils functions
Drivers                        Including DCMIPP, CSI and LCD HAL drivers
Includes
|--> project_path/Inc
|    |--> isp_param_conf.h     ISP IQ configuration parameters

4.3. Code implementation

The approach here is completely different as the Camera Middleware includes the ISP_Library and directly makes the required calls to the ISP APIs. It also handles the helper functions and the DCMIPP callback functions.

4.3.1. main function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

...
uint8_t Main_DestBuffer[MAX_PREVIEW_BUFFER_WIDTH * MAX_PREVIEW_BUFFER_HEIGHT * BPP_RGB888];
...
/**
* @brief  Main program
* @retval None
*/
int main(void)
{
  uint32_t camera_instance = 0;
  ISP_StatusTypeDef ret;

  ... /* Other system initializations here */

  printf("**** Initializing ISP preview application  ****\r\n");

  /* Initialize the DCMIPP device and the camera */
  if (Camera_Config(&phDcmipp, camera_instance) != 0)
  {
    printf("ERROR: can't configure camera\r\n");
   Error_Handler();
}

  /* Configure the display */
  Display_Config();

  (void)ISP_IQParamCacheInit; /* unused */

  /* Start the main pipe */
  if (CMW_CAMERA_Start(DCMIPP_PIPE1, (uint8_t *) Main_DestBuffer, CMW_MODE_CONTINUOUS) != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to start CAMERA\r\n");
    Error_Handler();
  }

  printf("Camera and ISP started\r\n");

  /* Application main loop */
  while (1)
  {
    ret = CMW_ERROR_NONE;
    ret = CMW_CAMERA_Run();
    assert(ret == CMW_ERROR_NONE);
  }
}

• "Camera_Config": This is implemented on application side to initialize the Camera Middleware Instance (See description below).
• "CMW_CAMERA_Start": Start the camera capture in continuous mode. ISP_Init and ISP_Start are called here (in CMW_IMX335_Start).
• "CMW_CAMERA_Run": Run the background process. ISP_BackgroundProcess is called here (in CMW_IMX335_Run).

4.3.2. Camera_Config function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/**
* @brief  Configure the camera
* @param  hDcmipp Pointer to the dcmipp device
* @param  Instance Camera instance
* @retval 0 if success, 1 otherwise
*/
int Camera_Config(DCMIPP_HandleTypeDef **hDcmipp, uint32_t Instance)
{
  int32_t  ret;
  CMW_CameraInit_t initConf = {0};
  UNUSED(Instance);

  initConf.width = 0;  /* width and height not specified => camera full resolution is set */
  initConf.height = 0; /* width and height not specified => camera full resolution is set */
  initConf.fps = 30;
  initConf.mirror_flip = CMW_MIRRORFLIP_NONE; /* CMW_MIRRORFLIP_NONE or CMW_MIRRORFLIP_FLIP or CMW_MIRRORFLIP_MIRROR or CMW_MIRRORFLIP_FLIP_MIRROR */
  initConf.pixel_format = 0; /* Not yet implemented */
  initConf.anti_flicker = 0; /* Not yet implemented */

  ret = CMW_CAMERA_Init(&initConf);
  if (ret != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to Initialize camera\r\n");
  return 1;
  }

  *hDcmipp = CMW_CAMERA_GetDCMIPPHandle();

  /* Make sure manual exposure is set on camera sensor side
   * This has no effect if the camera does not support it.
   */
  ret = CMW_CAMERA_SetExposureMode(CMW_EXPOSUREMODE_MANUAL); /* CMW_EXPOSUREMODE_AUTO or CMW_EXPOSUREMODE_AUTOFREEZE */
  if ((ret != CMW_ERROR_NONE) && (ret != CMW_ERROR_FEATURE_NOT_SUPPORTED))
  {
    printf("ERROR: Failed to set manual exposure\r\n");
  return 1;
  }

  /* Get the sensor information to fill the Camera_SensorConf structure */
  ISP_SensorInfoTypeDef info;
  ret = CMW_CAMERA_GetSensorInfo(&info);
  if (ret != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to get the sensor information\r\n");
    return 1;
  }
  strncpy(Camera_SensorConf.name, info.name, sizeof(Camera_SensorConf.name));
  Camera_SensorConf.CamImgWidth = info.width;
  Camera_SensorConf.CamImgHeight = info.height;
  ComputePreviewSize(Camera_SensorConf.CamImgWidth,
  Camera_SensorConf.CamImgHeight,
  &Camera_SensorConf.PreviewWidth,
  &Camera_SensorConf.PreviewHeight);
  switch (info.color_depth)
  {
    case 8:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW8;
      Camera_SensorConf.BytePerPixel = BPP_RAW8;
      break;
    case 10:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW10;
      Camera_SensorConf.BytePerPixel = BPP_RAW10;
      break;
    case 12:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW12;
      Camera_SensorConf.BytePerPixel = BPP_RAW12;
      break;
    case 14:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW14;
      Camera_SensorConf.BytePerPixel = BPP_RAW14;
      break;
    default:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW8;
      Camera_SensorConf.BytePerPixel = BPP_RAW8;
  }

  /* Configure the DCMIPP pipes */
  DCMIPP_Pipe1Config_Preview();

  return 0;
}

• "CMW_CAMERA_Init": It handles DCMIPP initialization, IMX335 sensor initialization, specific pipe configuration and preview display settings
• "CMW_CAMERA_GetSensorInfo": Sensor info used for respecting aspect ratio on display
• "CMW_CAMERA_SetExposureMode": This interface allows to disable sensor native AE algorithm if you want to run ST AE algorithm from ISP middleware.