19 - Complex cameras are complex - Jacopo Mondi

The ‘complex camera’ user-space library is a fairly new and hot topic in the Linux kernel video community. The issue is debated and targets a real technical debt of Linux systems.

The Linux kernel video input subsystem and its APIs changed greatly in the last years to keep up with the increasing complexity and processing power embedded in modern SoCs. Namely, the biggest game-changing feature introduced already 5 years ago is the media controller subsystem and its pad oriented APIs, that allows composing pipelines of processing blocks to model the acquisition and re-processing of video and images.

While most of the media subsystem drivers in mainline Linux have been ported to fully support media-controller, the real missing component is now user space support to automate setting up and controlling the image processing pipelines.

The Video4Linux community is now tackling the issue by implementing support for “complex camera” systems by providing a user space library that aims to support the most recent use cases represented by mobile consumer devices and high end laptops.

This talks provides an overview of the in-kernel APIs of modern media-controller capable video device drivers, their userspace APIs and the challenges the currently in-development library has to face.

It also aims to provide to both driver and application developers an overview of the most modern implementation of the Linux video input stack architecture, that will likely be found in most system in next years.