Simple ISP preview

This article explains how to launch a simple preview with ISP running for a better image quality rendering on the STM32N6570-DK board.

1. Description

This example demonstrates how to stream a camera and to display a preview with ISP enabled and 2A algorithms running.
This example included in the X-Cube-ISP package uses the Camera middlleware on top of the ISP middleware.

2. Prerequisites

• Hardware
  • STM32N6 discovery board
  • Discovery MB1860- STM32N6 (need USBC cable)
  • IMX335 camera module MB1854B

• Required tools
  • X-CUBE-ISP package v2.1.0
  • STM32CubeIDE (STM32CubeIDE 2.1.1)

3. Simple preview

This application streams the IMX335 sensor, runs ISP 2A algorithms and display a preview that fits the LCD in 640x480 using Camera Middleware services.

3.1. How to get application project

• Presently, a simple preview application using ISP services through Camera Middleware is available in X-CUBE-ISP package.
• Download the package then follow instructions from Path: X-CUBE-ISP_v2.1.0/README.md to import STM32N6_ISP_Preview_App project in STM32CubeIDE.
• Then, build index and project.

3.2. Project structure

Here is the general structure of the STM32N6_ISP_Preview_App project without ISP_MW_TUNING_TOOL_SUPPORT:

Application
|--> common                     Common directory
|    |--> main.c                Main entry point for all applications of X-CUBE-ISP project
|    |--> ...
|--> ISP_Preview_App            Preview application directory
|    |--> app.c                 Preview application entry point, Camera Middleware callbacks
|    |--> ...
Middleware
|--> Camera_Middleware          Camera middleware
|    |--> ISP_Library           ISP middleware included in Camera Middleware
|    |    |--> isp_algo.c       IQ algorithms
|    |    |--> isp_ae_algo.c    IQ AEC algorithm
|    |    |--> isp_awb_algo.c   IQ AWB algorithm
|    |    |--> isp_core.c       Core and API
|    |    |--> isp_services.c   Internal services
|    |--> sensors               Interfaces and drivers of supported sensors
|    |    |--> drivers          Sensor drivers
|    |    |    |--> imx335.c    IMX335 sensor driver
|    |    |    |--> vd1943.c    VD1943 sensor driver
|    |    |    |--> ...         Other supported sensor drivers
|    |    |--> cmw_imx335.c     IMX335 sensor interface
|    |    |--> cmw_vd1943.c     IMX335 sensor interface
|    |    |--> ...              Other supported sensor sensor interfaces
|    |--> cmw_camera.c          Core and API of Camera middleware
|    |--> cmw_utils.c           Camera Middleware Utils functions
|    |--> cmw_sensor_registry.c Camera Middleware registry of available sensors
Drivers                         Including DCMIPP, CSI and LCD HAL drivers
Includes
|--> project_path/Inc
|    |--> cmw_camera_conf.h     ISP list of supported camera modules
|    |--> isp_param_conf.h      ISP IQ configuration parameters

3.3. Code implementation

The approach here is completely different as the Camera Middleware includes the ISP_Library and directly makes the required calls to the ISP APIs. It also handles the helper functions and the DCMIPP callback functions.

3.3.1. main function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

...
static uint8_t Main_DestBuffer[2][MAX_PREVIEW_BUFFER_WIDTH * MAX_PREVIEW_BUFFER_HEIGHT * BPP_RGB888];
...
void App_Run()
{
  CAMERA_SensorConfTypeDef sensor_conf = {0};
  uint32_t frame_pitch;
  int ret;

  printf("**** Initializing ISP preview application  ****\r\n");
  /* Initialize the DCMIPP device and the camera */
  if (Camera_Config(&phDcmipp, &sensor_conf, &frame_pitch) != 0)
  {
    printf("ERROR: can't configure camera\r\n");
    Error_Handler();
  }

  /* Configure the display */
  Display_Config(&sensor_conf, frame_pitch);

  /* Start the main pipe */
  if (CMW_CAMERA_DoubleBufferStart(DCMIPP_PIPE1,
  (uint8_t *) Main_DestBuffer[0],
  (uint8_t *) Main_DestBuffer[1],
  CMW_CAPTUREMODE_CONTINUOUS) != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to start CAMERA\r\n");
    Error_Handler();
  }
  printf("Camera and ISP started\r\n");

  /* Application main loop */
  while (1)
  {
    tx_thread_sleep(1);
    ret = CMW_CAMERA_Run();
    assert(ret == CMW_ERROR_NONE);
  }
}

• "Camera_Config": This is implemented on application side to initialize the Camera Middleware Instance (See description below).
• "CMW_CAMERA_DoubleBufferStart": Start the camera capture in continuous mode with double buffering to avoid tearing effect. ISP_Init and ISP_Start are called here (in CMW_IMX335_Start).
• "CMW_CAMERA_Run": Run the background process. ISP_BackgroundProcess is called here (in CMW_IMX335_Run).

3.3.2. Camera_Config function

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/**
* @brief  Configure the camera
* @param  hDcmipp Pointer to the dcmipp device
* @param  Instance Camera instance
* @retval 0 if success, 1 otherwise
*/
static int Camera_Config(DCMIPP_HandleTypeDef **hDcmipp, CAMERA_SensorConfTypeDef *sensor_conf, uint32_t *frame_pitch)
{
  CMW_CameraInit_t initConf = {0};
  int32_t ret;

  initConf.width = 0;  /* width and height not specified => camera full resolution is set */
  initConf.height = 0; /* width and height not specified => camera full resolution is set */
  initConf.fps = 30;
  initConf.mirror_flip = CMW_MIRRORFLIP_NONE; /* CMW_MIRRORFLIP_NONE or CMW_MIRRORFLIP_FLIP or CMW_MIRRORFLIP_MIRROR or CMW_MIRRORFLIP_FLIP_MIRROR */

  ret = CMW_CAMERA_Init(&initConf, NULL);
  if (ret != CMW_ERROR_NONE)
  {
    printf("ERROR: Failed to Initialize camera\r\n");
    return 1;
  }
  *hDcmipp = CMW_CAMERA_GetDCMIPPHandle();

  sensor_conf->CamImgWidth = initConf.width;
  sensor_conf->CamImgHeight = initConf.height;
  ComputePreviewSize(sensor_conf->CamImgWidth, sensor_conf->CamImgHeight, MAX_PREVIEW_BUFFER_WIDTH,
                     MAX_PREVIEW_BUFFER_HEIGHT, &sensor_conf->PreviewWidth, &sensor_conf->PreviewHeight);

  /* Configure the DCMIPP pipes */
  DCMIPP_Pipe1Config_Preview(sensor_conf, frame_pitch);

  return 0;
}

• "CMW_CAMERA_Init": It handles DCMIPP initialization, IMX335 sensor initialization, specific pipe configuration and preview display settings
• "ComputePreviewSize": Get the preview size to respect aspect ratio of the camera