DeepAFx: Differentiable Signal Processing with Black-Box Audio Effects

Audio signal processing effects (FX) are used to manipulate sound characteristics across a variety of media. Many FX, however, can be difficult or tedious to use, particularly for novice users. In our work, we aim to simplify how audio FX are used by training a machine to use FX directly and perform automatic audio production tasks.

To combine deep learning and audio plugins together, we have developed a new method to incorporate third-party, audio signal processing effects (FX) plugins as layers within deep neural networks. We then use a deep encoder to analyze sounds and learn to control audio FX that themselves performs signal manipulation. To train our network with non-differentiable FX layers, we compute FX layer gradients via a fast, parallel stochastic approximation scheme within a standard auto differentiable graph, enabling efficient end-to-end backpropagation for deep learning training. For technical details of the work, please see:

“Differentiable Signal Processing with Black-Box Audio Effects."
Marco A. Martínez Ramírez, Oliver Wang, Paris Smaragdis, and Nicholas J.Bryan. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.

ArXiv Paper: https://arxiv.org/abs/2105.04752
Web Page: https://mchijmma.github.io/DeepAFx
Code: https://github.com/adobe-research/Dee...