Google Summer of Code'24
Final blog of my project adding AAC via S/PDIF and Bluetooth passthrough
This is my Google Summer of Code 2024 project with VideoLAN.
VideoLAN is a not-for-profit organization that develops and maintains a lot of open source projects which are used by billions worldwide including VLC Player, libVLC, x264, x265 and more.
This project aims to add passthrough for the AAC codec for hardware decoding over S/PDIF and Bluetooth for
enchanced audio as quality is often lost due to the decoding and re-encoding due to the lossy nature of the codec.
Project Details
Contributor Name: Advait Dhamorikar
Mentors: Thomas Guillem
Organization: VideoLAN
Project: Adding AAC via S/PDIF and Bluetooth passthrough
Links:
Workflow Overview
What is passthrough and why does it matter?
Passthrough is sending your digital data to be decoded by your receiver hardware which could include your headphones, earphones or amplifier instead of decoding it at the source.
In our case considering VLC player, it doesn’t currently support AAC passthrough.
Consider an AAC encoded 2 channel(stereo sound setup) sampled at 44.1KHz Whenever an AAC encoded stream is played from a container say .aac or .mp4a then we interpret the data in it whether they’re subtitles or actual audio.
These samples are then packetized by a packetizer. A packetizer interprets the data headers which carry information about the stream that helps us in interpreting it correctly and helps us in synchronization.
The decoder then deciphers the information held by the buffers which in our case is the Lavc59.37.100. It then is pre-buffered for playback.
With audio passthrough you can significantly improve your playback by avoiding lossy re-encoding and reduce lag in time sensitive applications allowing you to use your soundbars to their full potential.
Behind the Scenes
VLC is a beautiful, practical and modular software. Here is an explanation of the tasks happening to achieve this passthrough.
Here is what we are trying to do to achieve passthrough:
When you open a .aac or an AAC encoded .mp4a file, VLC uses a demuxer to separate different types of content which might be enclosed in the container e.g; subtitle files and the actual audio data.
[0000000114019000] main input debug: creating demux "mp4a", URL: file:///Users/advait/Downloads/twotigers.aac, path: /Users/advait/Downloads/twotigers.aac
[000000016c9cabf0] main generic debug: looking for demux module matching "mp4a": 1 candidates
[000060000103e4c0] es demux debug: detected format mp4a
[000000016c9ca870] main generic debug: looking for packetizer module matching "any": 25 candidates
The packetizer module is then called
[0000000158eac080] mpeg4audio demux packetizer debug: running MPEG4 audio packetizer
[0000000158eac080] mpeg4audio demux packetizer debug: AAC Type Unknown
[000000016c9ca870] main generic debug: using packetizer module "mpeg4audio"
This packetizer module is responsible for interpreting the data within the AAC file. Whether it has an ADTS, LOAS/LATM header or not, and then passes the processed data blocks further.
The blocks are then sent to the decoder.
[0000000149825a00] spdif decoder debug: spdif.c setting properties
[0000000158f16dd0] main player debug: reusing audio output
[0000000158e05780] main audio output debug: Entered aout_OutputNew
[0000000158e05780] avsamplebuffer audio output debug: Entered Start
Now the decoder in spdif.c isn’t an actual decoder but cleverly developed to forward the audio data to maintain the flow of data. This supposed decoder just forwards the data and sets the output properties.
The OS specidic audio output module then handles this data buffer. For this project I worked with Audiotrack for Android and Core Audio, AVSamplebuffer for MacOS and iOS.
You can read some of my more in-depth blogs that might be helpful here:
Contributions
You can check my commits and code here:
- Add AAC passthrough avsb componenets for MacOS and iOS
- Android audiotrack passthrough components
- Check for empty buffers being passed
- Add Android VLC 4.0 build instructions
Ongoing work
The backend changes for MacOS, iOS and Android are mostly complete however we are still facing minor hiccups in achieving proper audio playback.
We are currently in contact with the Apple developer team to tackle the issues involving their Core Audio API.
Acknowledgements
A very special thanks to my mentor Thomas Guillem for always answering my doubts and supporting me throughout. It was a great experience working with the him and getting to learn from his mentorship. I would also like to extend thanks to Jean-Baptiste Kempf and other members of the VideoLAN community who helped with my queries on irc.
It was an honour to work with team VideoLAN, to develop for the player I have used almost my entire life. It makes you realize the power of libre open source :)
