Camera support for Android™ is achieved through a generic Android camera HAL implementation on top of the libcamera[1] library that embeds a set of pipeline handler and image processing algorithm (IPA) modules to perform STM32MP25-specific configuration and processing.
1. libcamera overview[edit | edit source]
libcamera[1] is an open source complex camera support library for Linux®, Android and ChromeOS. It provides a complete userspace camera stack for Linux-based systems to abstract the configuration of hardware and image control algorithms required to obtain desirable results from the camera: it interfaces with Linux kernel device drivers and exposes a native C++ API to upper layers.
The libcamera documentation[3] details the four software layers: the kernel drivers, the libcamera framework, the libcamera adaptation (including the Android camera HAL), and the applications and upper level frameworks.
libcamera isn’t monolithic: it exposes multiple components through its public API, hardware abstractions are handled through the use of device-specific components, and dynamically loadable plugins are used to separate image processing algorithms from the core libcamera codebase.
Notably, the pipeline handler abstracts pipeline handling to hide device-specific details from the other components, and the image processing algorithms (IPAs) implement 3A (auto-exposure, auto-white balance and auto-focus) and other algorithms.
2. Android architecture[edit | edit source]
As explained in the How to use libcamera article, for the STM32MP25 DCMIPP, a specific pipeline handler and specific IPA modules are available. For Android, the AIDL camera HAL is implemented on top of the camera HAL v3 from libcamera.
Some explanations about the components in the above figure:
- Android frameworks and applications: considering the Android-based OpenSTDroid embedded software, one such application is “STCamera”.
- Android AIDL camera HAL[4]: it Implements an AIDL-based HAL by calling the Android camera HAL v3 component of libcamera. Thus, the following interfaces are implemented on top of libcamera:
- ICameraProvider: for enumerating individual devices and managing their status.
- ICameraDevice: the camera device interface.
- ICameraDeviceSession: the active camera device session interface.
- Android camera HAL v3: it exposes all the features required to implement an Android Camera HAL v3 on top of libcamera. It also implements internally features required by Android and missing from libcamera, such as JPEG encoding support or YUV reprocessing. The implementation initially targets the LIMITED hardware level, with support for the FULL level then being gradually implemented.
- libcamera core: it contains the libcamera components that are device-agnostic (e.g., Camera Manager, Camera Device, :Helpers and Support Classes).
- libcamera device-specific components:
- DCMIPP pipeline handler: it is the abstraction layer for the DCMIPP that accesses and controls the hardware through the kernel interface. It controls the ISP and captures components of a pipeline.
- DCMIPP IPA: it provides DCMIPP ISP functionalities that the pipeline handler can use for image processing.
- DCMIPP algoX/Y: it is a specialized algorithm (e.g., Auto Exposure), from ST or third party, that computes the values to be programmed into the DCMIPP ISP. It relies on the reported ISP statistics and provides static (fixed) values (e.g., black level constant value). For the Android-based OpenSTDroid embedded software, only ST algorithms are delivered.
- Linux kernel: it controls the DCMIPP driver and exposes the Linux kernel V4L2 family of APIs (Media Controller API, V4L2 Video Device API and V4L2 Subdev API).
- DCMIPP: it is the STM32MP25 peripheral used to receive video data from an external camera sensor device. It integrates an image signal processor (ISP) to get the desired output quality.
- CSI / I2C / Sensor: it is the camera sensor that produces the raw Bayer frames.
3. Android integration[edit | edit source]
3.1. Specificities[edit | edit source]
The code of libcamera (device/stm/stm32mp2/hardware/camera/libcamera) for Android and STM32MP25 (DCMIPP) is based on the code from libcamera official repository[5], and the code of the DCMIPP pipeline handler and image processing algorithms from the Yocto-based OpenSTLinux embedded software and the X-LINUX-ISP expansion package (improvements for the DCMIPP image processing algorithms).
As for integration into Android, the following points should be noted:
- the build process is deeply modified (see next chapter).
- a camera HAL configuration file is added (see below chapter).
- the RGB565 format is defined and the "IMPLEMENTATION_DEFINED" format is translated into this RGB565 format (
./src/android/camera_capabilities.cpp). - to get a "0-copy" mechanism for the preview frames from camera capture to display, the buffers' management has been modified as follows:
- the
GRALLOC_USAGE_HW_FBflag is added to the usage flags to get contiguous buffers, and theGRALLOC_USAGE_HW_COMPOSERis added to the usage flags so that buffers might be used by the hardware composer (./src/android/camera_device.cpp). - a cache mechanism (
./src/android/camera_device.cpp) between the set "stream and buffer handle", and the buffer file descriptor is added to ensure the 1:1 mapping between V4L2 buffer indexes and the buffer physical addresses, that the camera drivers assume. This cache ensures that each set "stream and buffer handle" uses a stable file descriptor across the lifetime of the stream, and avoids inconsistent V4L2 index reuse and preview blinking. - for the configuration of a stream, a
maxBuffersfield (maximum number of buffers that could be dequeued concurrently) complements thebufferCountfield (number of buffers to allocate). For the DCMIPP pipeline,bufferCountis set tomaxBuffersplus an extra margin, so that it matches the maximum number of buffers that the camera framework may allocate (./src/libcamera/pipeline/dcmipp/dcmipp_path.cpp). - "direct" buffers (i.e. buffers provided by the client of the camera HAL v3) instead of "internal" buffers (i.e. buffers allocated by the camera HAL v3 itself) are used for the preview stream, avoiding an extra copy for each captured frame (
./src/android/camera_device.cpp).
- the
3.2. libcamera build process[edit | edit source]
The libcamera build process relies on the Meson build system. During the build process, some files (e.g. ./include/libcamera/libcamera.h or ./include/libcamera/formats.h) are generated through shell scripts (e.g., ./utils/gen-header.sh) or Python scripts (e.g., ./utils/gen-formats.py). The scripts are mainly located in the ./utils directory.
For each IPA module (<ipa_module_name>.so), a signature is calculated, and stored alongside the IPA module in a file with a ".sign" suffix (<ipa_module_name.so.sign),
At runtime, when the IPA module instance is created, the signature is recalculated and compared to the stored one.
Since Android does not support the Meson build system, but uses the Soong build system instead, a Python script (gen-android-files.py) is added to generate the following files that are added in the libcamera tree structure:
├── include
│ └── libcamera
│ ├── control_ids.h
│ ├── formats.h
│ ├── internal
│ │ └── tracepoints.h
│ ├── ipa
│ │ ├── core_ipa_interface.h
│ │ ├── core_ipa_serializer.h
│ │ ├── dcmipp_ipa_interface.h
│ │ ├── dcmipp_ipa_proxy.h
│ │ └── dcmipp_ipa_serializer.h
│ ├── libcamera.h
│ ├── property_ids.h
│ └── version.h
├──src
└── libcamera
├── controls_ids.cpp
├── property_ids.cpp
├── proxy
│ ├── dcmipp_ipa_proxy.cpp
│ └── worker
│ └── dcmipp_ipa_proxy_worker.cpp
└── version.cpp
To build libcamera, the following Blueprint files are added:
./src/apps/Android.bpfile to compile the "cam" test application is added; it builds the "cam" binary (only for userdebug build)../Android.bpfile to compile the libcamera framework is added; it builds the:libcamera.soshared library (libcamera core and STM32 DCMIPP pipeline handler).ipa_dcmipp.soshared library (STM32 DCMIPP IPA).ipa_dcmipp.so.signsignature file (of STM32 DCMIPP IPA shared library).dcmipp_ipa_proxybinary (STM32 DCMIPP IPA proxy worker).imx335.yamlconfiguration file (imx335 sensor for STM32 DCMIPP IPA).
./src/android/Android.bpfile to compile the libcamera Android camera HAL v3 is added; it builds thecamera.stm.soshared library.
3.3. Sensor and camera configurations[edit | edit source]
Two configuration files shall be provided to tune the camera sensor and to specify the camera position.
Thus, for the MB1854 camera board that is proposed as accessory of the STM32MP257x-EV1 Evaluation board 
and STM32MP257x-DK Discovery kit , and that embeds the IMX335QLN 5Mpix Image Sensor, the following files are delivered:
imx335.yaml: IMX335 sensor tuning file for the STM32MP25 DCMIPP IPA. If another sensor than the IMX335 is used, the How to tune ISP using the STM32 ISP IQTune article provides step-by-step guidance to tune the DCMIPP ISP pipeline for a given RAW sensor and its associated lens.camera_hal.yaml: identifier, location and rotation of the available camera(s).
3.4. Formats, resolutions and performances[edit | edit source]
libcamera introduces notion of "StreamRole" to define predefined ways an application intends to use a camera. These roles can be either "Raw" (targeting raw Bayer extraction of the image from a sensor), "StillImage" (targeting best image capture with JPEG output via a software encoder), "VideoRecording" (targeting a video encoder; currently not supported with Android) and "ViewFinder" (targeting a display).
In DCMIPP pipeline (./src/libcamera/pipeline/dcmipp/dcmipp.cpp), those roles are mapped to the following formats and resolutions:
- "Raw": capture in raw Bayer format (identical to the format sent by the sensor: SBGGR8 for the IMX335) up to sensor resolution (2592x1944 for the IMX335).
- "ViewFinder": capture in RGB888 format at resolution up to 1024x600 that is the resolution of the B-LVDS7-WSVGA module proposed as accessory of the STM32MP257x-EV1 Evaluation board and STM32MP257x-DK Discovery kit .
- "StillCapture": capture in YUV420 format up to 1920x1440 resolution (4/3 aspect ratio) in 4K page configuration, which is the default shipped configuration (see STM32 MPU OpenSTDroid release note). If the 16K page configuration is selected, then capture is possible up to the sensor maximum resolution (2592x1944 for the IMX335), provided that the libcamera patch that restricts the maximum resolution is reverted.
Obviously, the camera application can request different configurations for the preview stream (e.g. 640x480 in HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED format) and the still capture stream (e.g. 1600x1200 in HAL_PIXEL_FORMAT_BLOB format).
The camera HAL v3 module converts the requested configurations in the following formats and resolutions for the libcamera "ViewFinder" and "StillCapture" roles:
- "ViewFinder": capture at the requested resolution in RGB565 format at 30 frames per second.
- "StillCapture": capture at the requested resolution in NV12 format.
When a still capture is requested, the camera HAL v3 module converts the captured NV12 frame into a JPEG file via its software JPEG encoder.
The buffers management implemented in the camera HAL v3 module allows a "0-copy" mechanism for the frames shared between the hardware composer, the camera frameworks, the camera HAL v3, libcamera and the LTDC and DCMIPP internal peripherals. The LTDC support of upscaling for the RGB formats, allows to "convert", for example, 640x480 RGB565 preview frames in 1024x600 RGB565 displayed frames. Therefore, the camera use cases let the GPU free for any other parallel use case. Thus, it is recommanded to request, for the preview, resolutions that are lower or equal to the resolution of the display module.
4. How to debug[edit | edit source]
Per default, only the warning and error logs of libcamera are available. To increase libcamera log level, the following modification shall be done in
./device/stm/stm32mp2/hardware/camera/provider/android.hardware.camera.provider-service.libcamera.rc:
service vendor.camera.provider.libcamera /vendor/bin/hw/android.hardware.camera.provider-service.libcamera
[...]
setenv LIBCAMERA_LOG_LEVELS *:DEBUG
[...]
For the userdebug build, the "cam" test application and the "media-ctl" V4L2 utility are built (see ./device/stm/stm32mp2/eval/device.mk):
- See in the How to use libcamera article information about the "cam" command line application, or use
cam -hto get the list of the options supported by this test application. - See in the STM32MP2 V4L2 camera overview article how to use the "media-ctl" V4L2 utility to get information about the camera sensor capabilities; especially, the command provided in this this chapter prints the topology of the camera subsystem, which might be useful for debugging.
5. References[edit | edit source]