This is my Google Summer of Code 2023 project with libcamera.

It aims to move the current V4L2 implementation of accessing and controlling cameras in OpenCV to libcamera, this will allow for better functionality for more advanced and complex cameras. This will be done by creating a videoio plugin for OpenCV to support libcamera and then creating suitable classes and functions to add the desired functionality.

Project Details

Contributor Name: Advait Dhamorikar

Mentors: Umang Jain, Paul Elder

Organization: libcamera

Project: Integrating libcamera into OpenCV

Links: * Github * Project Page

Workflow Overview

OpenCV is a popular computer-vision library used that offers ease of usage, libcamera is a camera-stack that can handle the complexity of modern embedded cameras with support for open and closed source IPAs. The project aimed to integrate libcamera into the OpenCV backend.

Adding libcamera support

If you have libcamera installed [docs] and use the libcamera backend when creating a VideoCapture object using:

cv::VideoCapture cap(0, cv::CAP_LIBCAMERA);

and if you used this flag with CMake:

-D WITH_LIBCAMERA=ON

Macros have been setup which check if your installation is working. Thus, the HAVE_LIBCAMERA flag will be set to true.

OpenCV backend

The call to libcamera is primarily made by cap_libcamera.cpp, which handles all major function calls. The core class CvCapture_libcamera_proxy is derived from OpenCV's IVideoCapture class. Its key member functions include:

• Constructor: Initializes the CameraManager thread and calls cam_init(), which handles error checking and opens the camera.
• open(int index): Acquires the camera of the specified index. A configuration is generated based on the chosen stream role (e.g., ViewFinder, VideoRecording, Raw, StillCapture). Pixelformat (e.g., YUYV, MJPEG, NV12) and stream dimensions can be customized here. If the configuration is valid, buffers are allocated and requests are queued to the camera.
• grabFrame(): Retrieves a frame from the camera's internal queue.
• retrieveFrame(): Processes the retrieved frame. It calls convertToRgb() to transform raw stream data into BGR/RGB format, then requeues the buffer for reuse.
• convertToRgb(): Checks the stream configuration and accesses its buffers. These buffers are mmap'd to get their data, which is processed according to the pixelformat and returned as a cv::Mat.

Contributions

You can check my commits and code here: https://github.com/advait-0/opencv/

Known issues

The backend has been integrated and supports majority of the IVideoCapture functions and the stream data is successfully being mmaped, however the format and colour conversion of this mapped data is still in testing for YUYV at the time of writing this blog.

Future Work

• Add support for additional formats like NV12.
• Add multi-camera support.
• Utilize dma to have a zero copy workflow.

Acknowledgements

A very special thanks to my mentors Umang Jain(uajain), Paul Elder (epoll) for always answering my doubts and supporting me throughout. It was a great experience working with them and getting to learn so many new things as a part of the Google Summer of Code. I would also like to extend thanks to Laurent Pinchart(pinchartl), Kieran Bingham(kbingham) and other members of the libcamera organization who helped with my queries.