Key Takeaways
- SL2T is the first consumer sign language AI, shipping on Pixel 11 in Gboard and Live Transcribe.
- It uses MediaPipe Holistic landmarks, discarding raw video and skipping glosses for direct sign-to-text translation.
- Trained on 100,000+ hours across 50+ sign languages, it outperforms single-language models for 70 million users.
Table of Contents
Sign Language AI Exits the Lab
Google DeepMind has shipped a multilingual sign-language-to-text model called SL2T directly into consumer hardware — starting with American Sign Language dictation on the Pixel 11, inside Gboard and Live Transcribe, at no additional cost.
The model, unveiled by the company on August 12, translates signing into fluent English text in real time.
It is the first time a major AI lab has embedded sign language translation into widely available consumer products rather than keeping it confined to research papers and prototype demos.
As Google DeepMind detailed, the model processes more than 100,000 hours of training data spanning over 50 sign languages, with roughly a quarter of that corpus dedicated to ASL alone.
A Deaf user can now sign into their phone to search the web, draft messages, compose documents, or issue commands to Gemini — replacing typed input with a modality that testers describe as faster and more natural than typing in English.
In Live Transcribe, the same technology enables signed responses during conversations, removing the friction of typing back and forth when two people communicate across different language modalities.
Landmarks, Not Pixels: Inside the SL2T Architecture
SL2T does not see raw video.
An on-device model called MediaPipe Holistic extracts pose landmark coordinates — tracking points across the signer’s hands, arms, torso, head, and face — and only these geometric data points leave the device for server-side translation.
The original camera feed is discarded immediately, a design choice that hardens privacy by default.
What makes the architecture genuinely novel is its refusal to use glosses, the intermediate annotation layer that most prior sign language translation systems depend on.
Glosses act as a manual word-mapping step between sign and text, but they collapse the non-linear, simultaneous grammar of sign languages into a linear sequence that strips away spatial constructions and non-manual markers — facial expressions, body shifts, and other carriers of meaning that have no gloss equivalent.
By translating directly from landmark sequences to text, SL2T eliminates that bottleneck.
The model learns to map three-dimensional movement trajectories straight to fluent English sentences, which means its vocabulary is not artificially capped and its translation quality scales with data volume rather than annotation bandwidth.
On the FLEURS-ASL benchmark, SL2T posts a zero-shot BLEURT score of 70 — a figure the engineering team at Google DeepMind describes as significantly higher than any previously reported result.
Yet the team pushed well beyond academic metrics, tuning for streaming latency, hallucination resistance on non-signing inputs, fairness for left-handed signers who make up roughly 10% of users, and robust performance on one-handed signing — the grip someone uses while holding a phone with their other hand.
Training jointly across languages, dialects, and proficiency levels forces the model to learn shared structural representations.
In Google DeepMind’s experiments, this multilingual approach outperformed single-language models, a pattern that mirrors findings from spoken language AI where cross-lingual transfer consistently strengthens generalization.
What 70 Million Users Gain When AI Learns to See Language
The global Deaf and hard-of-hearing population numbers an estimated 70 million people communicating across more than 200 distinct sign languages — each with its own grammar, lexicon, and cultural context, none of which is simply a gestural encoding of the dominant spoken language in its region.
For decades, the AI industry’s language revolution treated this entire linguistic landscape as an edge case.
Spoken language transcription, machine translation, voice dictation, and conversational assistants became commodity features for hearing users while sign languages remained locked out of the stack — not because the need was smaller, but because the technical challenge was genuinely harder.
Sign language translation demands true machine translation between structurally unrelated languages, not sequential phoneme-to-text mapping.
It also demands that computer vision systems track fine-grained, simultaneous whole-body movements at frame rates high enough to preserve meaning — a compute-intensive problem that earlier attempts, including sensor-glove approaches, fundamentally misunderstood by treating sign as hand-shapes alone.
SL2T’s arrival on a consumer device signals a broader shift in how major AI labs allocate resources toward accessibility.
Google DeepMind established the AI Sign Language Advisory Committee, or AISLAC, bringing together Deaf organizations and subject-matter experts to govern deployment priorities through a participatory model — not as an after-the-fact ethics review, but as a structural input from conceptualization onward.
Sam Sepah, a Deaf Googler, drove the initial vision, and the team co-authored a joint impact report transparently cataloging the technology’s capabilities and current limitations, a practice they intend to repeat for every major release.
That governance structure matters because the stakes are unusually high: a translation error in a conversational AI is an annoyance; a translation error in a medical or legal setting for a Deaf user who relies on the tool for access is a different category of failure entirely.
The remaining gaps — rare signs, rapid fingerspelling, passive constructions, classifier depictions, and tense inference without surrounding context — are documented openly in the impact report.
No gloss-based pipeline could have surfaced those edge cases so precisely, because gloss-based pipelines obscure where meaning degrades.
The direct landmark-to-text path makes failure modes transparent, which is itself a safety property.
A BLEURT Score Is Not the Point
The real benchmark is whether a technology that sat in research limbo for decades finally reaches the people who have been waiting for it — on a device they already carry, without a separate purchase, without a calibration ritual, and without the indignity of wearing specialized hardware to be understood.
SL2T on Pixel 11 is a starting line, not a finish line — and the multilingual architecture underneath it suggests the roadmap is measured in languages, not years.
For organizations building AI-powered accessibility tools that must scale across modalities without compromising privacy or latency, Andres SEO Expert approaches programmatic SEO and AI automation with the same architectural rigor — reach out to explore what that infrastructure looks like for your deployment.
Frequently Asked Questions
What is SL2T and how does it translate sign language to text?
SL2T is a multilingual sign-language-to-text model developed by Google DeepMind. It uses MediaPipe Holistic to extract pose landmark coordinates from the signer’s hands, arms, torso, head, and face, then translates those geometric data points directly into fluent English text on device or via server-side processing, without relying on raw video or intermediate gloss annotations.
How does SL2T protect user privacy during translation?
SL2T discards the original camera feed immediately after extracting landmark coordinates. Only the geometric landmark data leaves the device for server-side translation, ensuring that raw video never leaves the phone and privacy is hardened by default.
Which sign languages does SL2T support?
SL2T processes more than 100,000 hours of training data spanning over 50 sign languages, with roughly a quarter dedicated to American Sign Language (ASL). It is designed to support multilingual and cross-lingual translation, starting with ASL on consumer devices like the Pixel 11.
How is SL2T different from earlier sign language translation systems?
Previous systems relied on glosses, an intermediate annotation layer that forces sign language into a linear word-by-word structure, losing spatial and non-manual grammatical markers. SL2T eliminates glosses entirely, translating directly from landmark sequences to text, which allows it to capture the full grammar and scale with data rather than annotation effort.
What benchmark results has SL2T achieved?
On the FLEURS-ASL benchmark, SL2T achieves a zero-shot BLEURT score of 70, which Google DeepMind reports is significantly higher than any previously published result. The team also tuned for streaming latency, hallucination resistance, fairness for left-handed signers, and robust one-handed signing performance.
How were Deaf communities involved in developing SL2T?
Google DeepMind established the AI Sign Language Advisory Committee (AISLAC), bringing together Deaf organizations and subject-matter experts to govern deployment priorities from the start. A Deaf Googler, Sam Sepah, drove the initial vision, and the team co-authored a joint impact report that transparently documents capabilities and limitations.
What are the current limitations of SL2T?
Documented limitations include handling of rare signs, rapid fingerspelling, passive constructions, classifier depictions, and tense inference without surrounding context. These edge cases are openly acknowledged in the impact report, and the direct translation path makes these failure modes more transparent than gloss-based systems.
