Kind: captions
Language: en
this video was filmed without sound is
it possible to use only these images to
reconstruct the sound that is can you
hear pictures in this video I'm gonna
try to demonstrate that it's possible to
get sound from pictures but it's not
gonna be easy so I'm gonna need some
help this episode was sponsored by
LastPass which allowed me to fly to the
Bay Area to meet with one of my science
heroes so now on with the video
what you doing I'm sorry my play part of
my concern isn't my I like through all
this crap over here so this is the
experiment that I came up with it's like
a crumpled up ball of tinfoil yeah like
if I had a more powerful camera and I
had like the right lens then you know we
could do something that looks more like
you're spying on somebody but we should
be able to demonstrate that you can
recover like a rhythm or a sound from
you know whatever camera you have now
you might think it would be easy to
record sound in video because after all
sound is just vibrations so the air is
vibrating back and forth and everything
it hits should vibrate back and forth
too so you think all we need to do is
video that motion and plot displacement
versus time and then recover the sound
but it's not that simple because for one
thing I mean these sound vibrations are
incredibly tiny they move objects only
about one micrometer and even if you're
super zoomed in that is way less than a
pixel we're talking a hundredths or a
thousandth of a pixel we're not seeing
something that is at one pixel moved to
an adjacent pixel you're seeing one
pixel will get slightly darker and the
next pixel gets slightly brighter what
objects work best for recovering soon so
the things that work best are things
that have a lot of damping but are also
very light so that they move very
readily with changes in the air pressure
so what are some good examples well like
a bag of chips you know the initial
experiments were very like contrived in
a way you know we had these objects on
optical benches we were blasting light
at them the sounds were like as loud as
we could make them shave and a haircut
let's put that camera on a tripod
oh yeah that's I mean I'd be great all
right let's give it a shot this is the
actual clip I recorded and I want you to
notice two things first you can't really
see much motion and second there are
plenty of pixels getting dimmer and
brighter because of image noise I mean
it's not a pristine perfect image so how
do you tell the difference between
pixels getting brighter and dimmer due
to tiny movements versus noise it's
actually you look for edges in the image
and then you say well if the object
moves by some fraction of a fraction of
a pixel in one particular direction
then pixels on one side of that edge
will get a little bit brighter pixels
the other side of that edge will get a
little bit darker and so basically what
we do is we sum together all the ones
that are supposed to get brighter and
subtract all the ones that are supposed
to get darker and then that gives us or
one number right and if you track that
number over time then that gives you an
estimate of the displacement we're
dealing this is time but in samples and
then this is position hmm so what are
you doing well I try to do some
filtering on that it's clipping I mean
it's not much it's not much but you can
recognize two of the beats this is what
we've recovered from a hundred and
eighty frames per second which isn't
really a whole lot within the range of
like audible frequencies so that's why
you know kind of the most we can hope
for here is rhythm
now of course the main limiting factor
is framerate because we can hear sounds
from 20 Hertz to 20,000 Hertz but most
cameras only shoot 30 frames per second
so they miss virtually all of these
sound frequencies imagine this is the
motion created by a 30 Hertz sound if
you try to capture this with a camera at
30 frames per second you would end up
seeing the object always in the same
position because it's at the same point
in the wave cycle so your perception
would be that the object is not moving
at all so in order to measure a
frequency of sound you actually need to
sample at least twice that frequency
which is why a lot of music is sampled
at 44 or 48 kilohertz that's more than
twice as much as the highest sound we
can hear at any rate if you want to get
a something more intelligible you're
going to need some higher frame rate
camera
so we just went to the camera store and
picked up a new camera that should be
able to shoot a thousand frames per
second or there abouts is that gonna be
enough it'll be enough for something I
love that confidence this is one
modulation away from dubstep right here
just a little bit more of a bump bump
and then we have the next big track I've
set it to a thousand frames per second
now we're talking yeah
okay
okay so you have the footage there and
your cropping in a bit tell me about
that well we're running this on my
laptop as opposed to the servers that I
had back at MIT and that is gonna mean
that if we run it on the full video of
my laptop will crash okay so we're gonna
crop it and try it on that I can see a
little bit of motion yeah I mean I think
that one question is whether that's like
resonant motion yeah well in this case
would be kind of like the equivalent of
a rocking chair like if the if the foil
has a like a rocking mode then that's
actually not gonna give us a lot of
sound information mm-hmm can you tell
whether this is gonna work or not I am
optimistic
I think because I know what I'm
listening for I can hear it in there but
yeah you haven't careful though really
that you're well just gonna be careful
that you're not like confirmation bias
sure let's try it
that's about 60 Hertz that seems a
little much
okay we're gonna try one more time we
have basically put the piece of foil on
top of the speaker we're dialing up the
volume to 11 well I mean it's a shower
speaker so cool so what do you think of
that image there it's beautiful gorgeous
it has been like a couple years since
I've looked at this code so I suspect I
might have forgotten something and I'm
using it wrong but I want to point
something out here which is head here's
what we recovered and here is here is
the piano roll of the signal that we
sent this looks like dent to me and
that's the done yeah yeah that's it that
is the intent intent yeah okay so let's
see if we can actually get it to me in
here yeah we can see it we just can't
hear it I think I know what it is what
is it I don't think my laptop can play
those frequencies hold on a second let
me get some heavy unreal speakers or
something
[Laughter]
what are you hearing
hold on I hear shave and a haircut two
bits here
listen this helpful bit thank you yeah
and
[Laughter]
yep there you go
vision microphone this was a basic proof
of concept but Abe showed that with more
powerful equipment he could recover
human speech from outside soundproof
glass have you ever considered that your
computer is a physical system that gives
off vibrations each key on the keyboard
produces a unique sound due to its
unique location in fact research has
shown that audio recordings of typing
reveal 96% of keystrokes accurately now
this portion of the video was sponsored
by LastPass an app that stores all your
usernames and passwords so you never
have to type them in and this prevents
people from stealing your passwords by
say recording audio of your keystrokes
or just looking over your shoulder
what's even better is the convenience of
never having to fill in usernames and
passwords or getting locked out again
with LastPass you don't have to write
remember or reset passwords you get
unlimited password storage and free
cross-device sync when you open an app
or site on your computer or on an iOS or
Android device LastPass fills in your
username and password this saves you
valuable brain space so put your
passwords on autopilot with LastPass now
something I particularly like is if you
upgrade to LastPass premium you get
advanced multi-factor authentication and
it works like magic so click the link
below to find out more about LastPass
and thanks to LastPass for sponsoring
today's show