Types of models
What are speech and audio models, and how do ASR and TTS differ?
Speech and audio models are the type that works with sound. Speech recognition (ASR) transcribes audio into text, text-to-speech (TTS) does the reverse, and audio understanding and generation models handle music, sound effects, and non-speech audio. Each is defined by the direction it travels between sound and meaning.
Speech and audio models are the type that deals in sound. The clearest way to sort them is by which direction they travel between audio and meaning. Automatic speech recognition, ASR, listens to spoken audio and produces text. Text-to-speech, TTS, takes written text and produces spoken audio. A third group works with audio that is not speech at all, understanding or generating music and sound effects. Same medium, different jobs.
ASR is the most familiar, the engine behind live captions, dictation, and meeting transcripts. A well-known example is Whisper, trained on 680,000 hours of varied audio so that it holds up across accents, background noise, and many languages without being tuned for each one[1]. Whisper was released openly, which is part of why this kind of transcription became commonplace[2]. The output is just text, so an ASR model is often the first link in a longer chain rather than the whole product.
TTS runs the other way, turning a string of text into a waveform that sounds like a voice. The hard part is not the words but the delivery: where to pause, which syllable to stress, how to carry a natural rhythm so the result does not sound flat or robotic. Good modern TTS gets close enough that it is used for narration, accessibility, and the spoken side of voice assistants.
This is also where the type meets the others. A voice assistant has traditionally been three models in sequence: ASR to hear you, a language model to work out a reply, and TTS to speak it. The newer direction collapses that chain, with a single multimodal model that takes audio in and produces audio out directly, which makes responses faster and lets the model keep tone and timing that a text-only middle step would throw away.
References
- Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) — Radford et al. (arXiv)
- Whisper large-v3 model card — OpenAI / Hugging Face
Common questions
- What is the difference between ASR and TTS?
- ASR, automatic speech recognition, listens to audio and writes down the words. TTS, text-to-speech, reads written words aloud as audio. They are mirror images: one goes from sound to text, the other from text to sound.
- Is a voice assistant a single model?
- Often it is a chain: speech recognition turns your words into text, a language model decides what to say back, and text-to-speech voices the reply. Newer systems are starting to fold these into one model that hears and speaks directly.