Credits & licences
Built on open research.
LipScribe runs open models and libraries in your browser. Its three lip-reading models are licensed for research and non-commercial use only, and so is LipScribe.
Lip-reading models
- English, lips, the smaller model (optional): Auto-AVSR (Ma et al., ICASSP 2023; code Apache-2.0), LRS3 visual model, in our own 4-bit ONNX export (encoder + CTC + attention-decoder rescoring), fine-tuned by LipScribe on ~1,480 h of English clips from AVYT (AVSRCocktail, Interspeech 2025; YouTube videos, CC BY-NC-SA 4.0), European Commission press briefings and speeches from its Audiovisual Service (© European Union, CC BY 4.0) and Creative Commons videos.
- Spanish, lips: ViSpeR (Djilali et al., TII), CC BY-NC 2.0, in our own 4-bit ONNX export, fine-tuned by LipScribe on public recordings from the Parlamento de Canarias, the Congreso de los Diputados, RTVE and Creative Commons videos.
- English, lips: FLARE Large (Zhang et al., NeurIPS 2026; code MIT, weights CC BY-NC 4.0), its lip stream and its face stream, in our own 4-bit ONNX export (encoder + attention decoder read by a beam search). A small layer on top, for searching the lips for a word, was trained by LipScribe on 500 h of English clips (AVYT and our own sources).
- English, lips + sound (optional): AVSRCocktail (Nguyen et al., Interspeech 2025), CC BY-NC 4.0, built on Meta’s AV-HuBERT: “AV-HuBERT is licensed under the AV-HuBERT license, Copyright (c) Meta Platforms, Inc. All Rights Reserved.” (licence: non-commercial research only).
- Spanish, lips + sound (optional): ViSpeR’s audio-visual model (Djilali et al., TII), CC BY-NC 2.0, in our own 4-bit ONNX export.
All four models: research and non-commercial use only. Commercial use would need models trained on rights-cleared video.
Libraries
- Face detection: MediaPipe BlazeFace (Apache-2.0) on LiteRT.js (Apache-2.0).
- Telling people apart across cuts: SFace (OpenCV Zoo, Apache-2.0).
- Inference: ONNX Runtime Web (MIT).
- Video decoding: Mediabunny (MPL-2.0).
The front page’s demo
Real LipScribe output, read with the sound off, on “The Standard Model” by Fermilab, CC BY 3.0 (cut, cropped, muted).