The voice is one of the most powerful and intimate tools of human interaction, capable of conveying emotions, intentions and nuances through variations in tone, rhythm and intensity. However, current voice assistance technologies often lack this depth, coming across as flat and lacking empathy. This limitation not only reduces the effectiveness of interactions, but can also lead to an uncomfortable feeling known as ‘uncanny valley’.
Sesame, a company at the forefront of artificial intelligence research, is directly addressing this challenge. Their goal is to achieve what they call ‘vocal presence’: the magical quality that makes spoken interactions real, understood and valued. To do this, they are developing conversational partners that do not just process requests, but engage in genuine dialogues that build trust and confidence over time.
Key components of this revolution include:
- Emotional intelligence: the ability to read and respond to emotional contexts.
- Conversational dynamics: natural management of timing, pauses, interruptions and emphasis.
- Contextual awareness: adapting tone and style according to the situation.
- Consistent personality: maintaining a reliable and appropriate presence.
To achieve these goals, Sesame introduced the Conversational Speech Model (CSM), a model that frames the problem as an end-to-end multimodal learning task using transformers. This approach exploits conversation history to produce a more natural and coherent voice. Unlike traditional text-to-speech (TTS) models, which generate spoken output directly from text but lack contextual awareness, CSM operates as a single-stage model, improving efficiency and expressiveness.
An innovative aspect of CSM is its ability to address the ‘one-to-many’ problem, where there are innumerable valid ways to pronounce a sentence, but only a few adapt to a given context. Without additional information, such as tone, rhythm and conversation history, models lack the necessary data to choose the best option. CSM overcomes this limitation by integrating the conversational context, allowing for more natural speech generation.
To demonstrate progress, Sesame presented a demo with two virtual assistants, Maya and Miles, optimised for friendliness and expressiveness. This demo illustrates the potential of Sesame’s approach to conversational speech generation, giving users a taste of more engaging and realistic voice interactions.
Here is the link to the official press release: https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice
