Lip reading is hard. LipScribe tells you how sure it is.
Many sounds look the same on the lips — “pat”, “bat” and “mat” are one mouth shape — and even trained human lip readers miss a lot. So instead of pretending to be perfect, LipScribe gives every word a probability and shows you the other likely readings. Expect the gist, not a verbatim record.
The numbers
Lips alone: our test set of 67 real-world clips, 60 minutes (word error rate: English 34.5%, Spanish 35.4%). Lips + sound: 46 English clips, clean and with a crowd or a second voice mixed in as loud as the speaker (word error rate 8.9%, 13.9% and 10.8%). Lips + sound in Spanish: 21 clips, 8.6% clean and 11.7% with a crowd; a second voice as loud as the speaker is not handled (62%).
What the colours and percentages mean
Each word’s percentage is its chance of being right, estimated from what the model saw around that word. White words are likely (80%+), amber ones unsure, and underlined coral ones doubtful (under 50%). Tap a word in the app to see what else it could be; line readings show how the probability is shared among the readings the decoder kept.
Thank you all for coming tonight
- 44%Thank you all for coming tonight
- 23%Thank you all for coming to night
- 11%Thank you all for becoming tonight
What makes a clip easy or hard
Works best with
- one speaker facing the camera
- a face that fills a good part of the frame
- steady light and little motion blur
- 25–30 frames per second
Gets harder with
- turned heads and profile views
- small or distant faces
- hands, microphones or beards over the mouth
- low frame rates, heavy compression, rare names and numbers
Before reading, the app checks the face size, head turn, detection coverage and frame rate, and tells you if a clip is likely to be hard.