A research team at Meta has developed an artificial intelligence model capable of directly translating speech from one language to another. The model, named SEAMLESSM4T, addresses limitations in existing machine translation systems, which typically rely on multi-step processes such as speech recognition, text translation, and text-to-speech conversion.

SEAMLESSM4T enables direct translations for up to 101 languages, potentially paving the way for faster and more efficient multilingual communication. According to a study published in Nature, the model can:

• Translate speech-to-speech in 101 languages to 36 target languages.

• Translate speech-to-text from 101 languages to 96.

• Translate text-to-speech from 96 languages to 36.

• Translate text-to-text in 96 languages.

• Perform automatic speech recognition for 96 languages.

The research team highlighted that SEAMLESSM4T achieves up to 23% greater accuracy in speech-to-speech translations compared to existing systems.

In a commentary published alongside the study, Tanel Alumäe, Associate Professor at Tallinn University of Technology, emphasized the model’s greatest strength: its publicly available data and code, which allow for further optimization and application.

However, Alumäe noted certain challenges remain, such as: limited support for some languages and difficulty translating conversations in noisy environments or with speakers who have strong accents, areas where human translators excel.

Alison Kennecke, Assistant Professor in the Computer Science Department at Cornell University, praised the research for quantifying the potential for toxic, harmful, or offensive language in translations. The researchers also analyzed potential gender bias within the model’s output.

Kennecke stressed the importance of understanding how speech technologies can disproportionately fail certain demographics, despite being more efficient and cost-effective than human transcription and translation.