The Future: End-To-End Learned Audio, AI-Driven Loudness, Personalized Mixes

Why this matters

If you build or own a product that carries sound — a streaming service, a conferencing tool, an e-learning platform, a telemedicine app — the audio decisions you make today assume a world that is ending. You ship one mix and trust the viewer's device to play it; you hit one loudness target and hope it suits every room; you produce one language track and pay a studio for each dub. The shift this article describes turns each of those into something a model decides per listener, in real time, and the products that plan for it will feel a generation ahead of the ones that do not. This piece is for the product manager, founder, or operations lead who has to tell the difference between a real near-term capability and a conference-stage promise — what to build for now, what to watch, and where the consent, loudness, and disclosure rules are about to bind. It closes the Audio for Video section, so it leans on the vocabulary the earlier articles built; where a term needs refreshing, it is refreshed here.

The old contract: one mix, fixed forever

Start with the thing that is ending, because the future only makes sense against it. For almost the entire history of recorded media, the audio you heard was a mix — a single combined recording where an engineer had already decided how loud the dialogue sits against the music and the sound effects, and baked those decisions in permanently. A film, a news broadcast, a lecture recording: each shipped as one finished mix, identical for the cinema, the living room, the phone on a noisy train, the viewer who is hard of hearing, and the viewer who speaks a different language. If the dialogue was too quiet under the score, every viewer got dialogue that was too quiet. Your only controls were the master volume and, if you were lucky, a choice of pre-made language tracks and subtitles.

This worked because the production chain was a sequence of separate, hand-built tools, each doing one fixed job. A microphone captured sound; a codec — the software that compresses sound into fewer bits — squeezed it for delivery; a loudness normalizer pushed the whole mix to a standard target; a player decoded it and sent it to speakers. Each tool was designed by engineers who studied the problem and wrote the rules by hand. The chain was reliable, predictable, and completely rigid. Once the mix left the studio, no one downstream could change the balance inside it, because the separate ingredients — the voice, the music, the effects — had already been melted into one stream.

Two forces are dissolving that rigidity at once. The first is that the separate hand-built tools are being replaced by learned systems — neural networks that figure out the job from data instead of from a rulebook — and a learned system can do several jobs in one pass and adapt them to context. The second is that audio is moving back toward shipping as separate ingredients plus instructions rather than one melted mix, so the final balance can be decided at playback, by the viewer or by a model acting for them. Put those together and the fixed product becomes a flexible one. The rest of this article is the map of what that unlocks.

Shift one: end-to-end learned audio

The phrase end-to-end learned audio means replacing the chain of separate, hand-built tools with one neural network — or a small stack of them — trained to take sound in at one end and produce the result you want at the other, learning all the steps in between from examples rather than from hand-written rules. "End-to-end" is the key part: instead of a microphone feeding a hand-built noise filter feeding a hand-built codec feeding a hand-built error-concealer, a single learned system is trained on the whole journey at once, so it can trade off the steps against each other in ways a chain of separate tools never could.

We have already seen the first member of this family in detail. A neural audio codec is compression software where a network, not an engineer, learns to squeeze sound into a tiny stream of numbers and rebuild it — and the best ones reach the quality of the old codecs at three to four times fewer bits. The important thing for the future is not the bit saving. It is what the same machinery turned out to be able to do. A network that can rebuild a convincing voice from a thin stream of data is, mechanically, almost the same as a network that can predict the next piece of a voice that does not exist yet. Compression and generation are the same skill pointed in two directions: compression throws away everything the decoder can predict, and generation is nothing but prediction. This is why the line between a codec and a generator is dissolving, and it is the single most important idea in this article.

Watch how it cascades through the older tools. The job of packet loss concealment — hiding the gap when a chunk of audio goes missing on a bad network — used to be a hand-built guess that repeated the last good sound. Google's WaveNetEQ replaced that guess with a network that generates plausible audio to fill the gap, because a model that predicts sound is exactly what concealment needs. The same is happening to noise suppression, echo cancellation, and the codec itself — and the end state is not four separate networks but tendencies toward one. A learned front end that captures, cleans, compresses, conceals losses, and hands a clean token stream to whatever comes next, all in a single trained system. That is the end-to-end picture, and pieces of it ship today.

There is one more frontier worth naming because it changes the shape of the systems. The neural codecs we covered turn sound into discrete tokens — a short list of menu choices, like the words in a sentence, that a language model can predict. The newest research questions whether discreteness is even necessary: some 2025 systems generate audio from continuous internal representations and skip the rounding-to-a-menu step entirely. Whether the future keeps discrete audio tokens or moves to continuous ones is unsettled, and it matters because it decides how cleanly sound plugs into the large language models that increasingly sit at the center of everything. Hold the uncertainty; do not bet a roadmap on either answer yet.

Shift two: AI-driven loudness and the dialogue you can actually hear

Here is the shift that is furthest along in production, and the one your users are most likely to notice first, because it fixes a complaint almost everyone has: "I can't hear what they're saying over the music."

First, refresh the vocabulary. Loudness is how loud audio actually seems to a human ear, and the modern way to measure it is a number called LUFS — Loudness Units relative to Full Scale — defined by the standard ITU-R BS.1770. For fifteen years, "getting loudness right" meant pushing a whole finished mix to one agreed LUFS target — in Europe, the EBU R128 recommendation sets that at −23 LUFS — so that programs and channels matched and no one had to lurch for the remote between shows. That is loudness as a single number applied to the whole mix, and it is genuinely useful: it ended the era of adverts screaming louder than the film.

But a single loudness number for the whole mix cannot solve the dialogue problem, because the problem is not that the mix is too quiet — it is that the voice is too quiet relative to the music and effects inside that mix. To fix that, you need to reach inside the finished mix, find the speech, and turn only the speech up. For decades that was impossible after the mix was baked, because the voice and the music were one inseparable stream. Two approaches now make it possible, and they map exactly onto the two forces from the start of this article.

The first approach is object-based audio: ship the dialogue, the music, and the effects as separate ingredients — "objects" — each with instructions, instead of one melted mix, so the player can adjust the dialogue level on its own. This is built into MPEG-H 3D Audio, the immersive-audio standard carried in the ATSC 3.0 broadcast system now on air in the United States and South Korea. On a NextGen TV today, a viewer can raise the dialogue level inside the broadcast, within limits the broadcaster set and sent as metadata. The broadcaster can also author preset mixes — a "clear dialogue" preset, a "home theatre" preset, a sports preset that lets you turn the commentator down and the crowd up. The mix is no longer one fixed thing; it is a set of ingredients plus a few dials.

The second approach uses a learned system to solve the same problem for the enormous library of content that was already mixed the old way, where the separate ingredients are gone. Fraunhofer's Dialog+ is a deep neural network — a neural network being a model with millions of adjustable numbers — trained to listen to a finished mix and pull the dialogue back out of it, so the speech level becomes adjustable even though no one shipped separate objects. German public broadcasters WDR and BR ran the first large-scale public field tests of it starting in September 2020. The numbers from those tests are the reason this matters commercially, not just technically. In a survey of more than two thousand viewers, 90% of people over sixty reported difficulty understanding TV speech "often" or "very often", and 83% of all participants — including those who do not normally struggle — said they liked having the option to switch Dialog+ on. A feature that four out of five viewers want is not an accessibility niche; it is a mainstream expectation arriving.

The same idea has reached consumer devices through Amazon's Dialogue Boost. It launched on Prime Video in 2022 using cloud processing, then was compressed to under one percent of its original size while keeping nearly identical quality, so it can run on the device — on a Fire TV or an Echo speaker — and apply to audio from any app, including Netflix, YouTube, and Disney+. In Amazon's testing, over 86% of participants preferred the dialogue-boosted audio, rising to 100% approval among users with hearing loss. The machinery is the neural source separation we met in the AI-in-audio survey: a network that splits a mix into its parts, here so it can turn one part up.

Show the loudness arithmetic once, because it makes the difference concrete. Suppose a documentary is mixed so the dialogue averages −27 LUFS and the music averages −24 LUFS. The music is louder than the voice:

−24 LUFS (music) − (−27 LUFS) = 3 LU louder

A traditional loudness normalizer that pushes the whole mix to −23 LUFS keeps that 3 LU gap exactly — every part moves up together, and the voice stays buried. A dialogue-aware system, object-based or learned, instead lifts only the speech, say by 6 LU:

−27 LUFS + 6 LU = −21 LUFS (dialogue), with music unchanged at −24 LUFS

Now the voice sits 3 LU above the music, and the viewer can follow it. Same content, two philosophies: loudness as one number for the whole mix, versus loudness as a per-ingredient control the listener can steer. The future is decisively the second.

Figure 1. The core shift in one picture. The old contract melts ingredients into one fixed mix; the new contract keeps them separate so the balance — especially dialogue level — is chosen per listener at playback.

Shift three: the personalized mix

Once audio ships as ingredients plus instructions instead of one melted mix, dialogue level is only the first dial. A personalized mix is the general case: the language, the accent, the dialogue level, the descriptive narration, the spatial layout, and the device-appropriate balance are all selected for the individual listener at the moment of playback. The decade ahead turns "the mix" from a noun the studio ships into a verb the player performs.

The pieces already exist as separate features, and the future is mostly their convergence. ATSC 3.0's object model already lets a broadcaster offer alternate-language dialogue, assistive audio services, special commentary, and music-and-effects-only tracks, mixed on the device to the viewer's preference. Netflix added more than thirteen thousand hours of audio description — narration of the on-screen action for blind and low-vision viewers — across thirty-four languages in 2025 alone, a 30% year-over-year jump, and built a "search by language" feature so a viewer can find content by its dubbing, subtitle, or accessibility attributes. These are personalization primitives: the viewer is no longer accepting one mix but assembling the one they want from a menu.

The frontier that turns a menu into something genuinely new is real-time language and voice conversion — re-speaking content into another language while keeping the original speaker's voice and emotional delivery. Meta's Seamless family is the clearest public marker of where this is going. SeamlessExpressive translates speech while preserving vocal style and prosody — the rhythm, stress, and melody of speech, including the speaker's speech rate and pauses — so the translation sounds like the same person speaking a new language rather than a flat robotic dub. SeamlessStreaming does it without waiting for the sentence to finish, using an attention mechanism that emits the translation as the speaker talks, low enough in latency for live conversation. Put those together and you get the capability that reshapes localization: a viewer in São Paulo and a viewer in Seoul watch the same lecture, each hearing the instructor's own voice in their own language, decided at playback.

Accent conversion is the same technology aimed at a different axis. In March 2026, Krisp launched customer-side accent conversion that reshapes an incoming speaker's accent to be clearer to the listener, in real time, while — this is the hard part — preserving the sarcasm, urgency, hesitation, and warmth that carry meaning. The technical challenge across all of these is identical: change how the sound is produced while leaving what it means intact. That is exactly the semantic-versus-acoustic token split we met in neural codecs — separating what is said from how it sounds — now doing load-bearing work in a product, because once you can edit the two independently, you can swap the language or the accent (the acoustic side) while holding the meaning and emotion (the semantic side) fixed.

There is a quieter inverse problem that completes the picture, and it is where audio reaches back into video. If you re-speak a film in a new language, the actor's lips no longer match the words — the lip-sync breaks, and the human eye is unforgiving about it. The 2025 answer is visual dubbing: AI that re-shapes the actor's mouth in the video to match the new dubbed audio. Flawless's TrueSync builds a 3D model of the performer's face and adapts their mouth movements to the new dialogue in post-production, working up to 8K and holding up on a cinema screen; its DeepEditor, released February 2025, lets editors change a line and keep the original emotional performance. A Swedish feature, Watch the Skies, played US theatres in May 2025 visually dubbed into English. This is audio driving video — the inverse of the lip-sync problems Block 5 spent nine articles on, and proof that in the learned-media future, the audio and video pipelines stop being separate.

What's real now versus what's still a promise

The honest map matters more than the hype, so here it is in one table, sorted by how close each capability is to something you can ship.

Read it top to bottom as a timeline. The top rows are buy-or-build decisions you can make today; the middle rows are things to pilot and watch closely over the next two years; the bottom rows are roadmap-shaping ideas that are not yet products. The single most common mistake is reading a bottom-row demo and budgeting as if it were a top-row product.

Pitfall — "the demo works, so the product is ready." Learned audio is unusually good at producing a stunning thirty-second demo and an unusable round-the-clock service. A voice-cloning translation that is flawless on a clean studio sample falls apart on a noisy call; a dialogue separator that is crisp on a film trailer smears on a dense action scene; a visual dub that holds on one face fails on a crowd. The gap is not the model quality you saw — it is the long tail of real conditions the demo avoided. Before you commit a roadmap to any capability in the lower half of the table, run it on your worst inputs: your noisiest call, your densest mix, your hardest accent, your weakest client device. The question is never "can it do this once" but "what does it cost to do this every time, for everyone, including the cases the demo skipped." For most live products in 2026, the boring, hand-built tools — Opus for transport, a tuned jitter buffer for resilience — still beat the learned ones on cost and reliability, and the learned pieces earn their place one proven feature at a time.

The trust problem rides along with the capability

Every capability in the lower half of that table shares one engine, and that engine creates a problem the old fixed-mix world never had. A model that can rebuild a voice from a thin stream of tokens is the same kind of model that can generate that voice saying things the person never said. A model that re-speaks a line in a new language is one tuning step from one that puts new words in the speaker's mouth. A system that re-shapes lips to match dubbed audio is, mechanically, a system that can make anyone appear to say anything. When your compression, your translation, and your synthesis are all the same neural network pointed in different directions, "is this audio real" stops being a question the technology can answer for you.

The response is arriving on two fronts at once. The technical front is provenance — marking synthetic audio so it can be detected downstream. Meta's Seamless work shipped an inaudible, localized watermark specifically to dampen the impact of deepfakes, and watermarking the output of generative audio systems is becoming standard practice rather than an afterthought. The legal front is disclosure. The EU AI Act's Article 50 transparency obligations — which apply from 2 August 2026 — require providers of generative systems to machine-mark synthetic audio and require deployers to disclose audio deepfakes. As we covered in the AI-in-audio survey, the United States is moving on the same axis through proposed federal and state likeness laws. The practical upshot for a product builder: any feature that generates or transforms a recognizable voice now carries a consent-and-disclosure obligation, and the place that obligation attaches is moving from the obvious tools (a dubbing studio) into the plumbing (the codec, the translation layer, the conferencing pipeline). Build the labeling in from the start; it is far cheaper than retrofitting it.

Figure 2. The capability timeline with the trust layer underneath it. As features move from "shipping now" toward "fully generative," the provenance and disclosure obligation grows, not shrinks.

The 2030 audio pipeline, sketched

Pull the three shifts together and you can sketch the pipeline a video product plausibly runs by 2030. It is not science fiction; every block below exists in some shipping or near-shipping form today, and the change is in how they connect.

At capture, a single learned front end replaces the chain of separate tools. One network takes the microphone signal and, in one pass, suppresses noise, cancels echo, detects speech, and produces a clean token stream — the captured sound already in the compact, model-ready form we met as neural-codec tokens. There is no separate noise filter, echo canceller, and encoder; there is one trained system that does all of it, because training them together beats chaining them apart.

At transport, the pragmatic answer is still split. For plain real-time voice between known people, Opus or its traditional successor carries the stream, because it decodes almost for free on every device. For anything touching a model — assistants, translation, generated narration — the audio travels as tokens, because that is the form the model thinks in. The pipeline carries both, and chooses per feature.

At playback, the fixed mix is gone. The content arrives as ingredients plus instructions, and a model on or near the device assembles the mix for this listener: their language (re-spoken in the original voice), their dialogue level, their accent preference, their device's ideal spatial layout, their accessibility needs. The loudness target is no longer one number for everyone but a personalized one — content-aware, room-aware, and listener-aware. And riding alongside every generated or transformed element is a provenance mark and, where the law requires it, a disclosure.

The honest caveat closes the sketch. The blockers to this pipeline are not imagination; they are compute, control, and trust. Running learned models on every frame for every listener costs processor time and battery that not every device can spend, which is why the boring tools keep winning the cost-sensitive jobs. Giving a model fine control — turn this voice up this much without smearing the music — is still imperfect. And the trust layer has to be solved in lockstep, or the capability outruns the public's willingness to believe what they hear. The future is arriving feature by feature, from the top of that readiness table downward, and the teams that win are the ones who adopt each piece the moment it crosses from demo to dependable — not a year early, and not a year late.

Where Fora Soft fits in

We build conferencing, e-learning, telemedicine, OTT streaming, and surveillance products, and our stance on learned audio is the same discipline we bring to every hyped technology: adopt the proven piece, pilot the promising piece, and refuse to ship the demo piece on a customer's budget. For live calling, we still reach for the boring, reliable stack — Opus for transport, a tuned WebRTC audio pipeline for resilience — because it wins on cost and reliability today. Where we watch and pilot hardest is the personalized layer: dialogue clarity for e-learning and telemedicine, where being understood is the entire product; per-listener language and accessibility for OTT, where the audience is global; and the provenance and disclosure plumbing, because any product that touches a recognizable voice now inherits a consent obligation that is cheaper to design in than to bolt on. The right question for a client is never "should we use AI audio" but "which listener problem does a learned model solve well enough, cheaply enough, to ship this year" — and that frontier moves every quarter.

What to read next

• AI in audio for video: voice cloning, dubbing, restoration, generative music
• Neural audio codecs: Lyra, EnCodec, SoundStream and what comes next
• Loudness normalization: EBU R128, ITU-R BS.1770, ATSC A/85

Call to action

• Talk to a audio engineer — book a 30-minute scoping call to talk through your future of audio for video plan.
• See our case studies — 250+ shipped projects across video streaming, WebRTC, OTT, telemedicine, e-learning, surveillance, and AR/VR.
• Download the Future of Audio for Video: Readiness Cheat Sheet — One page on what's shipping now, what to pilot, the 2030 pipeline, and the trust layer that rides along.

References

1. M. Torcoli, J. Herre, et al. (Fraunhofer IIS), Dialog+ in Broadcasting: First Field Tests Using Deep-Learning-Based Dialogue Enhancement, arXiv:2112.09494, December 2021. https://arxiv.org/abs/2112.09494 (Primary source, read directly: deep-learning dialogue enhancement for legacy non-object content; WDR and BR public field tests from September 2020; survey of 2,000+ viewers — 90% of over-60s struggle with TV speech "often"/"very often", 83% of all participants liked the option to switch Dialog+ on. The load-bearing adoption evidence in this article.)
2. Seamless Communication (Meta AI), Seamless: Multilingual Expressive and Streaming Speech Translation, arXiv:2312.05187, December 2023. https://arxiv.org/abs/2312.05187 (Primary source, read directly: SeamlessExpressive preserves vocal style and prosody including speech rate and pauses; SeamlessStreaming uses Efficient Monotonic Multihead Attention for low-latency simultaneous speech-to-speech translation; ships an inaudible localized watermark to dampen deepfake impact. Source for the expressive-translation and watermarking claims.)
3. Regulation (EU) 2024/1689 (EU AI Act), Article 50 — Transparency Obligations for Providers and Deployers of Certain AI Systems. https://artificialintelligenceact.eu/article/50/ (Standards/legal primary source, read directly: Art. 50(2) requires providers to machine-mark synthetic audio/video/text output; Art. 50(4) requires deployers to disclose deep-fake content. The disclosure obligation this article cites.)
4. Regulation (EU) 2024/1689, Article 113 — Entry into Force and Application. https://artificialintelligenceact.eu/article/113/ (Legal primary source: confirms the Article 50 transparency obligations apply from 2 August 2026 — the date cited in the trust section.)
5. ITU-R BS.1770-5, Algorithms to measure audio programme loudness and true-peak audio level, ITU-R, 2023. https://www.itu.int/rec/R-REC-BS.1770 (Standards primary source: defines the LUFS loudness measurement used throughout; the per-mix loudness number the personalized-loudness future moves beyond.)
6. EBU R128 v5.0, Loudness normalisation and permitted maximum level of audio signals, EBU, 2023; with supplements R128s1 (short-form), s2 (streaming), s3 (radio), s4 (cinematic). https://tech.ebu.ch/publications/r128 (Standards primary source: the −23 LUFS broadcast target and the streaming supplement; EBU PLOUD's stated exploration of speech-level personalisation and accessibility services is the future-direction citation.)
7. ATSC A/342-3:2021, MPEG-H System, Advanced Television Systems Committee. https://www.atsc.org/wp-content/uploads/2021/03/A342-2021-Part-3-MPEG-H.pdf (Standards primary source, read directly: object-based audio with receiver-side dialogue level adjustment bounded by broadcaster-transmitted min/max metadata; broadcaster-authored preset mixes and user object-gain control — the object-based personalization mechanism.)
8. R. Bleidt et al. (Fraunhofer IIS), Development of the MPEG-H TV Audio System for ATSC 3.0, IEEE Transactions on Broadcasting, 2017. https://www.iis.fraunhofer.de/content/dam/iis/en/doc/ame/Conference-Paper/BleidtR-IEEE-2017-Development-of-MPEG-H-TV-Audio-System-for-ATSC-3-0.pdf (Peer-reviewed source: the MPEG-H personalization model — immersive + interactive objects, preset mixes, dialogue enhancement, alternate languages — as deployed in ATSC 3.0.)
9. Amazon Science / Prime Video Tech, Dialogue Boost: AI-powered dialogue enhancement, 2023–2025 (engineering descriptions of the on-device model). https://www.amazon.science/ (Vendor engineering source: cloud-launched 2022, compressed to <1% of original size for on-device use across apps; >86% of testers preferred boosted dialogue, 100% approval among users with hearing loss. Re-verify exact figures against the current Amazon publication at review.)
10. Netflix, Enhanced Accessibility Features (2025 announcement): 13,000+ hours of audio description across 34 languages, +30% YoY; "search by language" feature. https://about.netflix.com/ (Vendor primary source: the audio-description-at-scale and personalization-primitive claims. Trade-press corroboration via Cord Cutters News, 2025.)
11. Flawless AI, TrueSync visual dubbing and DeepEditor (February 2025); Watch the Skies US theatrical release, May 2025. https://flawlessai.com/localization-and-dubbing (Vendor source for the audio-drives-video "visual dubbing" inverse problem: 3D facial model, mouth re-shaping to new dialogue, up to 8K; DeepEditor preserves emotional performance. Theatrical-use claim corroborated by PetaPixel/New Atlas, 2025. Trade source — re-verify at review.)
12. Krisp, Customer-side accent conversion, launch March 2026. https://krisp.ai/ (Vendor source: real-time bidirectional accent conversion preserving prosody and emotion; the "change how it sounds, keep what it means" framing. Recent vendor announcement — flag for primary-doc confirmation at review.)
13. N. Zeghidour et al., SoundStream: An End-to-End Neural Audio Codec, arXiv:2107.03312, 2021. https://arxiv.org/abs/2107.03312 (Primary source: the end-to-end learned codec that grounds the compression-equals-generation argument; the same RVQ token stack that compresses sound is what generative and tokenizer systems reuse.)
14. Low-Resource Audio Codec Challenge (LRAC) 2025, Challenge Description, arXiv:2510.23312, 2025. https://arxiv.org/abs/2510.23312 (Current research source: frames the 2026 frontier as edge-deployable learned audio under tight compute/latency/bitrate limits — the "compute is the blocker" claim in the 2030-pipeline section. Also the pointer to continuous-representation, quantization-free directions.)

Living note: the shipping facts (Dialog+ field tests, MPEG-H/ATSC dialogue control, EU AI Act dates, EBU/ITU loudness targets) are stable standards-grade references. The product facts (Amazon Dialogue Boost figures, Netflix 2025 hours, Krisp March-2026 launch, Flawless theatrical use) are vendor and trade-press sources dated 2025–2026 and should be re-verified on the next refresh; several are flagged for primary-document confirmation at review.