On January 15, 2025, researchers from Meta published a groundbreaking paper in Nature detailing their new AI translation system, SeamlessM4T v2, which translates speech to speech from 101 source languages into 36 output languages.

The SeamlessM4T model boasts a 23% improvement in accuracy over existing translation systems, while also functioning as an automatic speech recognition system in 96 languages.

Currently, SeamlessM4T is being utilized for automatic video dubbing on Instagram and real-time translations through Ray-Ban glasses, translating from languages like Spanish, French, or Italian to English.

Meta has positioned itself as a significant supporter of open-source AI, enhancing its foundational models with this new technology, which can be fine-tuned for specific tasks.

The model was trained on an extensive dataset of 4.5 million hours of multilingual audio, utilizing self-supervised and semi-supervised learning techniques to learn from vast amounts of unannotated data.

To address data scarcity, the Seamless team developed a method called SONAR, which vectorizes texts from multiple languages into a single embedding space, facilitating automatic data alignment.

Despite its advancements, the model faces challenges, as it cannot cover the nearly 7,000 spoken languages and struggles in noisy environments or with strong accents.

Experts emphasize that human translators remain essential for conveying cultural nuances and context that AI may not fully grasp, particularly in critical fields like medicine and law.

The performance of the model regarding gender bias in translations was analyzed, revealing that significant improvements are still needed in this area.

While SeamlessM4T represents a major step toward universal translation, there are still approximately 6,500 languages worldwide, indicating that further development is necessary.

Meta plans to make the resources for SeamlessM4T publicly available for non-commercial use, encouraging further research in inclusive speech translation technologies.

The ultimate goal of the Seamless project is to create translation systems that feel natural and organic, akin to the fictional Babel Fish from Douglas Adams' 'The Hitchhiker's Guide to the Galaxy.'