Google Summer of Code'23

Final blog of my project integrating libcamera into OpenCV

This is my Google Summer of Code 2023 project with libcamera.

It aims to move the current V4L2 implementation of accessing and controlling cameras in OpenCV to libcamera, this will allow for better functionality for more advanced and complex cameras. This will be done by creating a videoio plugin for OpenCV to support libcamera and then creating suitable classes and functions to add the desired functionality.

Project Details

Contributor Name: Advait Dhamorikar

Mentors: Umang Jain, Paul Elder

Organization: libcamera

Project: Integrating libcamera into OpenCV

Links:

• Github
• Project Page

Workflow Overview

OpenCV is a popular computer-vision library used that offers ease of usage, libcamera is a camera-stack that can handle the complexity of modern embedded cameras with support for open and closed source IPAs. The project aimed to integrate libcamera into the OpenCV backend.

Adding libcamera support

If you have libcamera installedhttps://libcamera.org/getting-started.html and use the libcamera backend when creating a VideoCapture object using

cv::VideoCapture cap(0,cv::CAP_LIBCAMERA)

and if you had used this flag when using CMake

-D WITH_LIBCAMERA=ON

macros have been setup which check if your installation is working, thus the

HAVE_LIBCAMERA

flag will be set to true.

OpenCV backend

The call to libcamera is made primarily by the cap_libcamera.cpp which handles all of the function calls, Where the Class CvCapture_libcamera_proxy is derived from OpenCVs IVideoCapture Class its member functions include:

• Constructor: This is where the Camera Manager thread is started and cam_init is called which is responsible for error handling and then finally opening the camera.
• open(int index): The camera of the specified index gets acquired here, a configuration for it is generated based on a chosen streamrole which include ViewFinder, VideoRecording, Raw or StillCapture. We can make additional changes to the configuration here like choosing the desired pixelformat(YUYV, MJPEG, NV12 etc) and setting the dimensions of the stream. The configuration if supported is then validated and buffer allocation is done. Created requests are queued to camera.
• grabFrame(): Used to get the frames from the camera.
• retrieveFrame(): Processes the frames retrieved from the queue one by one calls the convertToRgb() functions which converts the planar data of various stream format into supported BGR/RGB, queuing the buffers for reuse again.
  • convertToRgb(): This function checks the streamconfig for the format and finds its buffers. It then mmaps those buffers data and returns a file descriptor. The mapped data is then processed according to it’s pixelformat and returned to a cv::Mat container.

Contributions

You can check my commits and code here:

1. Adding necessary libcamera addons to backend
2. Implement open functionality
3. Add buffer allocation capabilities
4. Add all necessary IVideoCapture functions framework
5. Add pixelformat and colour conversion capabilities
6. Bug fixes and optimizations

Known issues

The backend has been integrated and supports majority of the IVideoCapture functions and the stream data is successfully being mmaped, however the format and colour conversion of this mapped data is still in testing for YUYV at the time of writing this blog.

Future Work

•  Add support for additional formats like NV12.
•  Utilize dma to have a zero copy workflow.

Acknowledgements

A very special thanks to my mentors Umang Jain(uajain), Paul Elder (epoll) for always answering my doubts and supporting me throughout. It was a great experience working with them and getting to learn so many new things as a part of the Google Summer of Code. I would also like to extend thanks to Laurent Pinchart(pinchartl), Kieran Bingham(kbingham) and other members of the libcamera organization who helped with my queries.

libcamera + GSoC = :-)