A neural audio codec learns how to compress and reconstruct sound from data, using deep neural networks in place of the hand-designed transforms and psychoacoustic models of traditional codecs. The common design is an autoencoder: a neural encoder maps audio to a compact latent representation, a quantizer (often residual vector quantization, RVQ) discretizes it to a target bitrate, and a neural decoder regenerates the waveform. Lyra, EnCodec, and SoundStream are the landmark examples, achieving usable quality at bitrates far below what classical codecs can reach, especially for speech. They also produce the discrete tokens that modern generative audio and speech models operate on. The cost is heavier compute, but neural codecs are reshaping the low-bitrate and generative frontier of audio.
The book · Volume 5 of 7
Audio for Video: A Complete Guide to Sound in Video Products: From Jitter Buffers and Lip Sync to Atmos, Quality Metrics, and Neural Codecs
A neural audio codec can carry speech at a few kbps, but it costs compute on every device and changes how loss and quality behave. Volume 5 covers neural codecs in detail, with quality metrics such as PESQ, POLQA and ViSQOL, packet loss concealment and FEC, so you can judge where a neural codec beats Opus in your product. Written by Nikolay Sapunov, CEO at Fora Soft.
Considering a neural audio codec like Lyra or EnCodec for low-bandwidth calls?
We will test neural codecs against Opus on your network conditions and send a recommendation with quality scores and device CPU cost. 250+ video products shipped since 2005.
