ASR · Phase 7: Speech meets LLMs
SALMONN / Qwen-Audio / Qwen2-Audio
SALMONN (2023), Qwen-Audio (2023), Qwen2-Audio (2024): audio encoder + adapter + LLM
In one sentence
These three papers define the "audio encoder + adapter + LLM" pattern: a pretrained speech encoder (usually Whisper's) turns audio into feature vectors, a small adapter maps them into the LLM's embedding space, and a text LLM then does ASR, translation, audio question answering and chat from natural-language instructions.
Why this paper exists
Whisper (Whisper) transcribes well but only does a few fixed tasks selected by special tokens. Text LLMs can follow any instruction but cannot hear. Training a speech-and-language model from scratch is enormously expensive. The question: can you bolt hearing onto an existing LLM cheaply and get one model that handles speech, sounds and music with open-ended prompts?
The core idea
Keep both big pretrained parts, connect them with a thin trainable bridge. The encoder already knows audio, the LLM already knows language; the adapter only learns to translate encoder outputs into "soft tokens" the LLM can read, alongside the text prompt.
How it works
audio -> [speech encoder] -> frames (e.g. ~25-50/s)
|
[adapter: Q-Former or
pooling + linear projection]
|
audio embeddings + text prompt embeddings
\ /
[ text LLM ] -> answer textSALMONN (2023)
- Two encoders: Whisper's encoder for speech and BEATs (an audio-event encoder) for general sounds and music, with outputs combined frame by frame.
- Window-level Q-Former: a Q-Former (a small Transformer with learned query vectors that cross-attend to encoder outputs) applied to short windows of audio, so the number of audio tokens scales with duration.
- LLM: Vicuna, adapted with LoRA (low-rank adapter weights) instead of full fine-tuning.
- Task over-fitting: after instruction tuning, the model tended to just transcribe whatever it heard instead of following the prompt. SALMONN adds few-shot activation tuning to restore emergent abilities, such as answering questions about speech or translating into languages not seen in training.
Qwen-Audio (2023)
- Audio encoder initialised from Whisper-large-v2, LLM from Qwen-7B.
- Trained jointly on 30+ audio tasks (ASR, translation, sound and music captioning, speaker and emotion tasks and more).
- Hierarchical tags: like Whisper's special tokens but richer (task, language, timestamps, etc.), so tasks with different label styles share what they can and do not interfere.
- Qwen-Audio-Chat: an instruction-tuned version for multi-turn chat with audio and text inputs.
Qwen2-Audio (2024)
- Encoder initialised from Whisper-large-v3, LLM Qwen-7B.
- Replaces hierarchical tags with natural-language prompts for pretraining and uses more data.
- Two modes, voice chat (talk to it) and audio analysis (give audio plus a text instruction), with no system prompt needed to switch.
- DPO (direct preference optimisation) to improve factuality and instruction following.
- Reported better than prior systems, including Gemini-1.5-pro, on AIR-Bench audio instruction following.
Build the intuition
Analogy: hiring an interpreter (the adapter) between a sound engineer who hears everything but cannot talk about it (the encoder) and a brilliant writer who has never heard a sound (the LLM). You only train the interpreter, plus a little coaching for the writer.
Why not feed every 20 ms Whisper frame to the LLM? A 30 s clip would be 1,500 tokens; Q-Formers and pooling cut that so the LLM's context and compute stay manageable.
Numbers worth remembering
- Qwen-Audio: 30+ pretraining tasks; Whisper-large-v2 encoder + Qwen-7B.
- Qwen2-Audio: Whisper-large-v3 encoder + Qwen-7B, natural-language prompts, DPO.
- SALMONN: Whisper + BEATs encoders, Vicuna LLM with LoRA (13B is the main size; check the paper).
- Whisper's encoder emits 50 frames/s on 30 s windows (1,500 frames); adapters reduce this. Check each paper for the exact token rate.
Where it shows up today
This is the dominant design for "speech LLMs" and many new ASR systems: Qwen2.5-Omni and similar omni models, NVIDIA's Canary-Qwen-style SALM models, and many Indian-language research models that put an Indic speech encoder in front of an LLM. Contrast with Moshi (Moshi), which models audio tokens directly and speaks back, and DSM (Kyutai DSM), which streams.
What it got wrong, and what came next
- Not streaming: the encoder sees whole windows and the LLM reads the full audio before answering (Cache-aware streaming Conformer and Kyutai DSM are the streaming routes).
- Hallucination: an LLM decoder can produce fluent text that was never said, as Whisper does (Whisper).
- Heavy: a 7B+ LLM per request is slow and expensive for plain transcription compared with a CTC or TDT model (TDT). It does not fit a 4 GB GPU without quantization.
- Text out only (for these three); speaking back needs TTS or a model like Moshi.
Check yourself
Q1What are the three parts of the pattern and what is usually trained?
A pretrained audio encoder, an adapter (Q-Former or pooling plus projection), and a pretrained text LLM. Usually the adapter is trained fully, the LLM gets LoRA or light fine-tuning, and the encoder may be frozen or fine-tuned.
Q2Why did SALMONN need few-shot activation tuning?
Instruction tuning on mostly ASR and captioning made the model over-fit to those tasks, so it ignored new instructions and transcribed instead. A small extra tuning step reduced that bias and brought back abilities like spoken QA.
Q3What changed from Qwen-Audio to Qwen2-Audio in how tasks are specified?
Qwen-Audio used hierarchical tags, a structured tag format like Whisper's special tokens. Qwen2-Audio uses natural-language prompts, which is simpler and matches how users talk to it.
Q4For a low-latency Hindi call-centre transcriber, would you pick this pattern? Why or why not?
Probably not as the main path: it is not streaming, is expensive per second of audio, and can hallucinate. A streaming Fast Conformer or Indic CTC model is cheaper and faster; the LLM pattern suits post-call analysis or audio QA.
When you're back at a laptop
- Run Qwen2-Audio-7B-Instruct (4-bit, or on a cloud GPU) on a Hindi or Indian-English clip with prompts "Transcribe this" and "What is the speaker's mood?". Compare its transcript with Whisper's.
- Print the processor's audio feature shape and the number of audio tokens it inserts into the prompt.