Simple ISP preview

This article explains how to launch a simple preview with ISP running for a better image quality rendering on the STM32N6570-DK board.

1. Description

This example demonstrates how to stream a camera and to display a preview with ISP enabled and 2A algorithms running.
This example included in the X-Cube-ISP package uses the Camera middlleware on top of the ISP middleware.

2. Prerequisites

• Hardware
  • STM32N6 discovery board
  • Discovery MB1860- STM32N6 (need USBC cable)
  • IMX335 camera module MB1854B

• Required tools
  • X-CUBE-ISP package v2.0.0
  • STM32CubeIDE (STM32CubeIDE 1.15.1.24-N6-A2)

3. Simple preview

This application streams the IMX335 sensor, runs ISP 2A algorithms and display a preview that fits the LCD in 640x480 using Camera Middleware services.

3.1. How to get application project

• Presently, a simple preview application using ISP services through Camera Middleware is available in X-CUBE-ISP package.
• Download the package then follow instructions from Path: X-CUBE-ISP_v2.0.0/README.md to import STM32N6_ISP_IQTune_App project in STM32CubeIDE.
• Remove ISP_MW_TUNING_TOOL_SUPPORT definition from the project preprocessor settings as follow:

• Then, clean the project and rebuild index and project.

3.2. Project structure

Here is the general structure of the STM32N6_ISP_IQTune_App project without ISP_MW_TUNING_TOOL_SUPPORT:

Application
|--> main.c                    Main application entry point, ISP helpers and DCMIPP callbacks
|--> ...
Middleware
|--> Camera_Middleware         Camera middleware
|    |--> ISP_Library          ISP middleware included in Camera Middleware
|    |--> sensors              Interfaces and drivers of supported sensors
|    |    |--> drivers         Sensor drivers
|    |    |    |--> imx335.c   IMX335 sensor driver
|    |    |--> cmw_imx335.c    IMX335 sensor interface
|    |--> cmw_camera.c         Core and API of Camera middleware
|    |--> cmw_utils.c          Camera Middleware Utils functions
Drivers                        Including DCMIPP, CSI and LCD HAL drivers
Includes
|--> project_path/Inc
|    |--> isp_param_conf.h     ISP IQ configuration parameters

3.3. Code implementation

The approach here is completely different as the Camera Middleware includes the ISP_Library and directly makes the required calls to the ISP APIs. It also handles the helper functions and the DCMIPP callback functions.

3.3.1. main function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

...
uint8_t Main_DestBuffer[MAX_PREVIEW_BUFFER_WIDTH * MAX_PREVIEW_BUFFER_HEIGHT * BPP_YUV422];
...
/**
* @brief  Main program
* @retval None
*/
static void main_thread_fct(ULONG arg)
{
  ...
  uint32_t camera_instance = 0;
  ISP_StatusTypeDef ret;

  ... /* Other system initializations here */

  printf("**** Initializing ISP preview application  ****\r\n");

  /* Initialize the DCMIPP device and the camera */
  if (Camera_Config(&phDcmipp, camera_instance) != 0)
  {
    printf("ERROR: can't configure camera\r\n");
   Error_Handler();
}

  /* Configure the display */
  Display_Config();

  (void)ISP_IQParamCacheInit; /* unused */

  /* Start the main pipe */
  if (CMW_CAMERA_Start(DCMIPP_PIPE1, (uint8_t *) Main_DestBuffer, CMW_MODE_CONTINUOUS) != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to start CAMERA\r\n");
    Error_Handler();
  }

  printf("Camera and ISP started\r\n");

  /* Application main loop */
  while (1)
  {
    ret = CMW_ERROR_NONE;
    ret = CMW_CAMERA_Run();
    assert(ret == CMW_ERROR_NONE);
  }
}

• "Camera_Config": This is implemented on application side to initialize the Camera Middleware Instance (See description below).
• "CMW_CAMERA_Start": Start the camera capture in continuous mode. ISP_Init and ISP_Start are called here (in CMW_IMX335_Start).
• "CMW_CAMERA_Run": Run the background process. ISP_BackgroundProcess is called here (in CMW_IMX335_Run).

3.3.2. Camera_Config function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/**
* @brief  Configure the camera
* @param  hDcmipp Pointer to the dcmipp device
* @param  Instance Camera instance
* @retval 0 if success, 1 otherwise
*/
int Camera_Config(DCMIPP_HandleTypeDef **hDcmipp, uint32_t Instance)
{
  int32_t  ret;
  CMW_CameraInit_t initConf = {0};
  UNUSED(Instance);

  initConf.width = 0;  /* width and height not specified => camera full resolution is set */
  initConf.height = 0; /* width and height not specified => camera full resolution is set */
  initConf.fps = 30;
  initConf.mirror_flip = CMW_MIRRORFLIP_NONE; /* CMW_MIRRORFLIP_NONE or CMW_MIRRORFLIP_FLIP or CMW_MIRRORFLIP_MIRROR or CMW_MIRRORFLIP_FLIP_MIRROR */

  ret = CMW_CAMERA_Init(&initConf);
  if (ret != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to Initialize camera\r\n");
  return 1;
  }

  *hDcmipp = CMW_CAMERA_GetDCMIPPHandle();

  /* Make sure manual exposure is set on camera sensor side
   * This has no effect if the camera does not support it.
   */
  ret = CMW_CAMERA_SetExposureMode(CMW_EXPOSUREMODE_MANUAL); /* CMW_EXPOSUREMODE_AUTO or CMW_EXPOSUREMODE_AUTOFREEZE */
  if ((ret != CMW_ERROR_NONE) && (ret != CMW_ERROR_FEATURE_NOT_SUPPORTED))
  {
    printf("ERROR: Failed to set manual exposure\r\n");
  return 1;
  }

  /* Get the sensor information to fill the Camera_SensorConf structure */
  ISP_SensorInfoTypeDef info;
  ret = CMW_CAMERA_GetSensorInfo(&info);
  if (ret != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to get the sensor information\r\n");
    return 1;
  }
  strncpy(Camera_SensorConf.name, info.name, sizeof(Camera_SensorConf.name));
  Camera_SensorConf.CamImgWidth = info.width;
  Camera_SensorConf.CamImgHeight = info.height;
  ComputePreviewSize(Camera_SensorConf.CamImgWidth,
  Camera_SensorConf.CamImgHeight,
  &Camera_SensorConf.PreviewWidth,
  &Camera_SensorConf.PreviewHeight);
  switch (info.color_depth)
  {
    case 8:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW8;
      Camera_SensorConf.BytePerPixel = BPP_RAW8;
      break;
    case 10:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW10;
      Camera_SensorConf.BytePerPixel = BPP_RAW10;
      break;
    case 12:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW12;
      Camera_SensorConf.BytePerPixel = BPP_RAW12;
      break;
    case 14:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW14;
      Camera_SensorConf.BytePerPixel = BPP_RAW14;
      break;
    default:
      Camera_SensorConf.SensorDataType = DCMIPP_DT_RAW8;
      Camera_SensorConf.BytePerPixel = BPP_RAW8;
  }

  /* Configure the DCMIPP pipes */
  DCMIPP_Pipe1Config_Preview();

  return 0;
}

• "CMW_CAMERA_Init": It handles DCMIPP initialization, IMX335 sensor initialization, specific pipe configuration and preview display settings
• "CMW_CAMERA_SetExposureMode": This interface allows to disable sensor native AE algorithm if you want to run ST AE algorithm from ISP middleware.
• "CMW_CAMERA_GetSensorInfo": Sensor info used for respecting aspect ratio on display