Honest accuracy

Lip reading is hard. LipScribe tells you how sure it is.

Many sounds look the same on the lips — “pat”, “bat” and “mat” are one mouth shape — and even trained human lip readers miss a lot. So instead of pretending to be perfect, LipScribe gives every word a probability and shows you the other likely readings. Expect the gist, not a verbatim record.

The numbers

~2 in 3English words right on varied real-world clips, from the lips alone.
~85%of English words right on a clear, front-facing studio speaker.
~65%of Spanish words right — Spanish is in beta.
~9 in 10English words right when LipScribe also listens to the sound, even over a crowd or a second voice.
93%of the words shown at 80–100% were right; words shown at ~50% were right about half the time.

Lips alone: our test set of 67 real-world clips, 60 minutes (word error rate: English 34.5%, Spanish 35.4%). Lips + sound: 46 English clips, clean and with a crowd or a second voice mixed in as loud as the speaker (word error rate 8.9%, 13.9% and 10.8%). Lips + sound in Spanish: 21 clips, 8.6% clean and 11.7% with a crowd; a second voice as loud as the speaker is not handled (62%).

What the colours and percentages mean

Each word’s percentage is its chance of being right, estimated from what the model saw around that word. White words are likely (80%+), amber ones unsure, and underlined coral ones doubtful (under 50%). Tap a word in the app to see what else it could be; line readings show how the probability is shared among the readings the decoder kept.

What a line looks like
0:07

Thank you all for coming tonight

  1. 44%Thank you all for coming tonight
  2. 23%Thank you all for coming to night
  3. 11%Thank you all for becoming tonight
likely 80%+unsure 50–80%doubtful under 50%

What makes a clip easy or hard

Works best with

  • one speaker facing the camera
  • a face that fills a good part of the frame
  • steady light and little motion blur
  • 25–30 frames per second

Gets harder with

  • turned heads and profile views
  • small or distant faces
  • hands, microphones or beards over the mouth
  • low frame rates, heavy compression, rare names and numbers

Before reading, the app checks the face size, head turn, detection coverage and frame rate, and tells you if a clip is likely to be hard.