Kind: captions
Language: en
the following is a conversation with
Suresh Carmen a professor at MIT
co-founder of the autonomous vehicle
company optimist ride and is one of the
top roboticists in the world including
robots that drive and robots that fly to
me personally he has been a mentor a
colleague and a friend he's one of the
smartest most generous people I know so
it was a pleasure and honor to finally
sit down with him for this recorded
conversation this is the artificial
intelligence podcast if you enjoy it
subscribe I knew to review five stars in
Apple podcast supported on patreon or
simply connect with me on Twitter and
Lex Friedman spelled Fri D ma n as usual
I'll do a few minutes of ads now and
never any ads in the middle that can
break the flow of the conversation I
hope that works for you it doesn't hurt
the listening experience this show is
presented by cash app the number one
finance app in the App Store when you
get it use the code lex podcast cash app
lets you send money to friends buy
Bitcoin and invest in the stock market
with as little as $1 this cash app
allows you to send and receive money
digitally let me mention a surprising
fact about physical money it costs 2.4
cents to produce a single penny in fact
I think it costs 85 million dollars
annually to produce them that's a crazy
little fact about physical money so
again if you get cash out from the App
Store Google Play and use the collects
podcast you get $10 and cash Apple also
donate $10 the first an organization
that is helping to advanced robotics and
STEM education for young people around
the world and now here's my conversation
with Sir - Carmen
since you have worked extensively on
both what is the more difficult task
autonomous flying or autonomous driving
that's a good question
I think that autonomous flying just kind
of doing it for consumer drones and so
on the kinds of applications that we're
looking at right now is probably easier
and so I think that that's maybe one of
the reasons why it took off like
literally a little earlier than the
autonomous cars but I think if we look
ahead I would think that you know the
real benefits of autonomous flying
unleashing them in like transportation
logistics and so on I think it's a lot
harder than autonomous driving so I
think my guess is that you know we've
seen a few kind of machines fly here and
there but we really haven't yet seen any
kind of you know machine like like at
massive scale large-scale being deployed
and flown and so on and I think that's
gonna be after we kind of resolve some
of the large scale deployments of
autonomous driving it was the hard part
what's your intuition behind why at
scale when consumer-facing drones are
tough so I think in general its scale is
tough like for example I mean you think
about it we have actually deployed a lot
of robots in the let's say the past 50
years we academics or we business I
think we as humanity deployed a lot of
robots and I think that when you think
about it you know robots they're
autonomous they work and they work on
their own but they are either like in
isolated environments or they are in
sort of you know they may be at scale
but they're really confined to a certain
environment that they don't interact so
much with humans and so you know they
work in
I don't know factory floors their houses
they work on Mars you know they are
fully autonomous over there but I think
that the real challenge of our time is
to take these vehicles and put them into
places where humans are present so now I
know that there's a lot of like human
robot interaction type of things that
need to be done and so on that's that's
one thing but even just from the
fundamental algorithms and systems and
and the business cases or maybe the
business models even like architecture
planning societal issues legally there's
a whole bunch of pack of things that are
related to us putting robotic vehicles
into human present environments and
these humans you know they will not
potentially be even trained to interact
with them they may not even be using the
services that are provided by these
vehicles they may not even know that
they're autonomous they're just doing
their thing living in environments that
are designed for humans not for robots
and that I think is one of the biggest
challenges I think over our time to put
vehicles there and you know to go back
to your question I think doing that at
scale meaning you know you go out in a
city and you have you know like
thousands or tens of thousands of
autonomous vehicles that are going
around it is so dance to the point where
if you see one of them you look around
you see another one it is that dance and
that density we've never done anything
like that before and I would I would bet
that that kind of density will first
happen with autonomous cars because I
think you know we can ban the
environment a little bit we can
especially kind of making them safe is a
lot easier when they're like on the
ground when they're in the air it's a
little bit more complicated but I don't
see that there's gonna be a big
separation I think that you know there
will come a time that we're gonna
quickly see these things unfold do you
think there will be a time where there's
tens of thousands of delivery drones
they fill this guy you know I think I
think it's possible to be honest
delivery drones is one thing but you
know you can imagine for transportation
like a like an important use cases but
you know we're in Boston you want to go
from Boston to New York and you want to
do it from the top of this building to
the top of another building in Manhattan
and you're gonna do it in one and a half
hours and that's that's a big
opportunity I think personal transport
so like you and your friend like oh yeah
or almost like I like like an uber so
like four people six people a people in
our work in autonomous vehicles I see
that so there's kind of like a bit of a
need for you know one person transport
but also like like a few people so you
and I could take the trip together we
could have lunch
that you know I think kind of sounds
crazy maybe even sounds a bit cheesy but
I think that those kinds of things are
some of the real opportunities and I
think you know it's not like the typical
airplane and the airport would disappear
very quickly but I would think that you
know many people would feel like they
would spend an extra hundred dollars on
doing that and cutting that for our
travel down to one and a half hours so
how feasible are flying cars it's been
the dream that's like when people
imagine the future for 50 plus years
they think fine cars it's a it's like
all technology is just cheesy to think
about now because it seems so far away
but overnight it can change but just
technically speaking in your view how
feasible is it to make that happen I'll
get to that question but just one thing
is that I think you know sometimes we
think about what's gonna happen in the
next 50 years it's just really hard to
guess right next 50 years I don't know I
mean we could yet what's gonna happen in
transportation in the next 50 we could
get flying saucers I I could bet on that
I think there's a 50/50 chance that you
know like you can build machines that
can ionize the air around them and push
it down with magnets and they would fly
like a flying saucer that is possible
and it might happen in the next 50 years
so it's a bit hard to guess like when
you think about 50 years before but I
would think that you know there's this
this kind of notion where there's a
certain type of air space that we call
the agile airspace and there is there's
good amount of opportunities in that
airspace
so that would be the space that is kind
of a little bit higher than the place
where you can throw a stone because
that's a tough thing when you think
about it you know it takes a kid on a
stone to take an aircraft down and then
what happens but you know imagine the
airspace that's high enough so that you
cannot throw a stone but it is low
enough that you're not interacting with
the with the very large aircraft that
are you know flying several thousand
feet above and that airspace is
underutilized or it's actually kind of
not utilized at all yeah that's right so
there's you know there's like
recreational people kind of fly every
now and then but it's very few
if you look up in the sky you may not
see any of them at any given time every
night now you'll see one airplane
utilizing that space and you'll be
surprised and the moment you're outside
of an airport a little bit like it's
just kind of flies off and it goes out
and I think utilizing that airspace the
technical challenge is there is you know
building an autonomy and ensuring that
that kind of autonomy is safe ultimately
I think it is going to be building in
complex software are complicated so that
it's maybe a few orders of magnitude
more complicated than what we have on
aircraft today and at the same time
ensuring just like we ensure on aircraft
ensuring that it's safe and so that
becomes like building that kind of
complicated hardware and a software
becomes a challenge especially when you
know you build that hardware
I mean you build that software with data
and so you know it's it's of course
there's some rule by software in there
that kind of do a certain set of things
but but then you know there's a lot of
training merit machine learning will be
key to these guys to delivering safe
vehicles in the future especially not
maybe the safe part but I think the
intelligent part um I mean there are
certain things that we do it with
machine learning and it's just there's
like right now all the way and and I
don't I don't know how else they could
be done and you know there's there's
always this conundrum I mean we could I
could be like we could maybe gather
billions of programmers humans who
program perception algorithms that
detect things in the sky and whatever or
you know we I don't know we maybe even
have robots like learning a simulation
environment and transfer and they might
be learning a lot better in a simulation
environment than a billion humans put
their brains together and try to program
humans pretty limited what's uh what's
the role of simulations withdrawals if
you've done quite a bit of work there
how promising just the very thing you
said just now how promising is the
possibility of training
and developing a safe flying robot in
simulation and deploying it and having
that work pretty well in the real world
I think that you know a lot of people
when they hear simulation they will
focus on training immediately but I
think one thing that you said which was
interesting it's developing
I think simulation environments are
actually could be key and great for
development and that's not new like for
example you know there's people in the
automotive industry have been using
dynamic simulation for like decades now
and and it's pretty standard that you
know you would build and you would
simulate if you want to build an
embedded controller you plug that kind
of embedded computer into another
computer that other computer would
simulate tiny and so on and I think you
know fast forward these things you can
create pretty crazy simulation
environments
like for instance one of the things that
has happened recently and that you know
we can do now is that we can simulate
cameras a lot better than we used to
simulate them we were able to simulate
them before and that's I think we just
hit the elbow on that kind of
improvement I would imagine that really
improvements in hardware especially and
with improvements and machine learning I
think that we would get to a point where
we can simulate cameras very very much
similar cameras means simulate how a
real camera would see the real world
therefore you can explore the
limitations of that you can train
perception algorithms on the in
simulation all that kind of stuff
exactly so you know it's it has been
easier to simulate what we will called
interoceptive sensors like internal
sensors so for example inertial sensing
has been easy to simulate it has also
been easily simulate dynamics like like
physics that are governed by ordinary
differential equations I mean like how a
car goes around maybe have it rolls on
the road how they interact with it
interacts with the road or even an
aircraft flying around like the dynamic
the physics of that what has been really
hard has been to simulate extra Sept of
sensors sensors that kind of like look
out from the vehicle and that's a new
thing that's coming like laser
rangefinders they're a little bit
easier cameras radars are a little bit
tougher I think once we nail that down
the the next challenge I think in
simulation will be to simulate human
behavior that's also extremely hard even
when you imagine like how a human driven
car would act around even that is hard
but imagine trying to simulate you know
a a model of a human just doing a bunch
of gestures and so on and and you know
it's it's actually simulated it's not
captured like with a motion capture but
it is similarly that's that's very in
fact today I get involved a lot with
like sort of this kind of very high-end
rendering projects and I have like this
test that I've pass it to my friends or
my mom you know ice and like two photos
two kind of pictures and I say rendered
which one is rendered which one is real
and it's pretty hard to distinguish
except I realized except when we put
humans in there it's possible that our
brains are trained in a way that we
recognize humans extremely well but we
don't so much recognize the built
environments because built an alarm sort
of came after per se we evolved into
sort of being humans but but humans were
always there same thing happens for
example you look at like monkeys and you
can't distinguish one from another but
they sort of do and it's very possible
that they look at humans it's kind of
pretty hard to distinguish one from
another but we do and so our eyes are
pretty well trained to look at humans
and understand if something is off we
will get it we may not be able to
pinpoint it so in my typical friend test
or mom test what would happen is that we
put like a human walking in you know
anything and they they say you know this
is not right something is off in this
video I don't know what but I can tell
you it's too human I can take the human
and I can show you like inside of a
building or like an apartment and it
will look like if we had time to render
it it will look great and this should be
no surprise a lot of movies that people
are watching it's all computer-generated
you know even nowadays when you watch a
drama movie and like there's nothing
going on action wise but it turns out
it's kinda like cheaper I guess to
render the background and so they would
but how do we get there how do we get a
human
would pass the mom / friend test a
simulation of a human walking do you
think that's something we can creep up
do by just doing kind of a comparison
learning or you have humans annotate
what's more realistic and not just by
watching they go what what's the path is
it seems totally mysterious how we thing
right simulate human behavior
it's it's hard because a lot of the
other things that I mentioned to you
including simulating cameras right it is
the the thing there is that you know we
know the physics we know how it works
like in the real world and we can write
some rules and we can do that like for
example simulating cameras there's this
thing called ray tracing I mean you
literally just kind of imagine it's very
similar to it's not exactly the same but
it's very similar to tracing photon by
photon they're going around bouncing on
things and coming your eye a human
behavior developing a dynamic like like
like a model of that that is
mathematical so that you can put it into
a processor that would go through that
that's gonna be hard and so so what else
do you got you can collect data right
and you can try to match the data or
another thing that you can do is that
you know you can show the Frant test you
know you can say this or that and this
or that and that will be labeling
anything that requires human labeling
ultimately we're limited by the number
of humans that you know we have
available at a heart disposal and the
things that they can do you know they
have to do a lot of other things than
also labeling this data so so that
modeling human behavior part is is I
think going we're gonna realize it's
very tough and I think that also effects
you know our development of autonomous
vehicles I see them self-driving in
smile like you want to use so you're
building self-driving you know it the
first time like right after urban
challenge I think everybody focused on
localization mapping and localization
you know as slam algorithms came in
Google was just doing that and so
building these HD maps basically that's
about knowing where you are and then
five years later in 2012-2013 came the
kind of coding code AI revolution and
that started telling us about everybody
else's but we're still missing what
everybody else is gonna do next and so
you want to know where you are you want
to know what everybody else is hopefully
you know about what you're gonna do next
and then you want to predict what other
people are going to do and that last bit
has been a real real challenge what do
you think is the role your own of you of
your the ego vehicle the robot you the
the you the robotic you in controlling
and having some control of how the
future on roles of what's going to
happen in the future that seems to be a
little bit ignored in trying to predict
the future is how you yourself can
affect that future by being either
aggressive or less aggressive or
signaling in some kind of way so this
kind of game theoretic dance seems to be
ignored for the moment it's yeah it's
it's totally ignored I mean it's it's
quite interesting actually like how we
how we interact with things versus we
interact with humans like so if if you
see a vehicle that's completely empty
and it's trying to do something all of a
sudden it becomes a thing
so interacted with like you interact
with this table and so you can throw
your backpack or you can kick your kick
it put your feet on it and things like
that but when it's a human there's all
kinds of ways of interacting with a
human so if you know like you and I are
face to face we're very civil you know
we talk understand each other for the
most part you'll see you just but but
the thing is that like for example you
and I might interact through YouTube
comments and you know the conversation
may go a totally different angle and so
I think people kind of abusing these
autonomous vehicles is a real issue in
some sense and so when you're an ego
vehicle you're trying to you know
coordinate your way make your way it's
actually kind of harder than being a
human you know it's like it's you you
you not only need to be as smart as kind
of humans are but you also you're a
thing so they're gonna abuse you a
little bit so you need to make sure that
you can get around and do something so
yeah III in general believe in that sort
of game theoretic aspects I've actually
personally have done you know quite a
few papers both on that kind of game
theory and also like this this kind of
understanding people's social value
orientation for example you know some
people are aggressive some people not so
much and and you know like you robot
could understand that by just looking at
how people drive and as they kind of
come and approach you can actually
understand like if someone is gonna be
aggressive or or not as a robot and you
can make certain decisions well in terms
of predicting what they're going to do
the hard question is you as a robot
should you be aggressive or not when
faced with it was an aggressive role but
right now it seems like aggressive is a
very dangerous thing to do because it's
costly from a societal perspective how
you're perceived people are not very
accepting of aggressive robots emotive
Society I think that's accurate so that
is really is and so I'm not entirely
sure like how to have to go about but it
I know I know for a fact that how these
robots interact with other people in
there is going to be and then
interaction is always gonna be there I
mean you could be interacting with other
vehicles or other just people kind of
like walking around and like I said the
moment there's like nobody in the seat
it's like an empty thing just rolling
off the street it becomes like no
different than like any other thing
that's not human and so so people and
maybe abuse is the wrong word but you
know people may be rightfully even they
feel like you know this is a human
present environments designed for humans
to be and and they they kind of they
want to own it and then you know the
robots they would they would need to
understand and they would need to
respond in a certain way and I think
that you know this actually opens up
like quite a few interesting societal
questions for us as we deploy like we
talk robots at large scale so what would
happen when we try to deploy robots at
large scale I think is that we can
design systems in a way that they're
very efficient or we can design them
that they're very sustainable but
ultimately the sustainability efficiency
trade-offs like they're gonna be right
in there
we're gonna have to make some choices
like we're not going to be able to just
kind of put it aside so for example we
can be very aggressive and we can reduce
transportation delays increase capacity
of transportation or you know we can we
can be a lot nicer and other people to
kind of coding code on the environment
and live in a nice place and then
efficiency will drop so when you think
about it I think sustainability gets
attached to energy consumption or I
wanna have the impact immediately and
those are those are there but like
livability is another sustainability
impact so you create an environment that
people want to live in and if robots are
going around being aggressive and you
don't want to live in that environment
maybe however you should note that if
you're not being aggressive then you
know you're probably taking up some some
delays in transportation and listen that
so you're always balancing that and I
think this this choice has always been
there in transportation but I think the
more autonomy comes in the more explicit
the choice becomes yeah when it becomes
explicit that we can start to optimize
it and I will get to ask the very
difficult societal questions of what do
we value more efficiency or
sustainability it's kind of interesting
there will happen like I think we're
gonna have to like I think that the the
interesting thing about like the whole
autonomous vehicles question I think is
also kind of I think a lot of times you
know we have we have focused on
technology development like hundreds of
years and you know the products somehow
followed and and then you know we got to
make these choices and things like that
but this is this is a good time that you
know we even think about you know
autonomous taxi type of deployments and
the systems that would evolve from there
and you realize the business models are
different the impact on architecture is
different urban planning you get into
like regulations and then you get into
like these issues that you didn't think
about before but like sustainability and
ethics is like right in the middle of it
I mean even testing autonomous vehicles
like think about it you're testing
autonomous vehicles in human present
environments I mean the risk may be very
small but still you know it's it's a
it's a it's it's a you know strictly
greater than zero risk that you're
putting peep
and so then you have that innovation you
know risk trade-off that you're in that
somewhere and we understand that pretty
now they pretty well now is that if we
don't test the beast day the development
will be slower I mean it it doesn't mean
that we're not gonna be able to develop
I think it's gonna be pretty hard
actually
maybe we can we don't real I don't know
but well the thing is that those kinds
of trade-offs we already are making and
as these systems become more ubiquitous
I think those trade-offs will just
really hit so you are one of the
founders of optimist ride and town
vehicle company and we'll talk about it
well let me and that point ask maybe
good examples
keeping optimist right out of this
question sort of exemplars of different
strategies on the spectrum of innovation
and safety or caution so you dick way
Moe Google self-driving car way Moe
represents maybe a more cautious
approach and then you have Tesla on the
other side added by Elon Musk that
represents some more however which
adjectives you want to use aggressive
innovative I don't know but what what do
you think about the difference between
its two strategies in your view what's
more likely what's needed and is more
likely to succeed in the short term in
the long term definitely some sort of
balance is is kind of the right way to
go but I do think that the thing that is
the most important is actually like an
informed public so I don't I don't mind
you know I personally like if I were in
some place I wouldn't mind so much like
taking a certain amount of risk some
other people might and so I think the
key is for people to be informed and so
that they can ideally they can make a
choice in some cases that kind of choice
making that anonymously is of course
very hard but I don't think it's
actually that hard to inform people
so I think in in in one case like for
example even the Tesla approach I don't
know it's hard to judge how he informed
it is but it is somewhat informed I mean
you know things kind of come out I think
people know what they're taking and
things like that and so on but I think
the underlying I do think that these two
companies are a little bit kind of
representing like babe of course that
you know one of them seems a bit safer
the other one or you know whatever the
objective for that is and the other one
seems more aggressive or whatever the
ejector for that is but but I think you
know when you turn the tables they're
actually they're two other orthogonal
dimensions that these two are focusing
on on the one hand for remo I can see
that you know they're I mean they I
think they're a little bit see it as
research as well so they kind of they
don't I'm not sure if they're like
really interested in like an immediate
product you know they talk about it
sometimes there's some pressure to talk
about it so they kind of go for it but I
think I think that they're thinking
maybe in the back of their minds maybe
they don't put it this way but I think
they they realize that we're building
like a new engine it's kind of like call
it the AI engine or whatever that is and
you know an autonomous vehicles is a
very interesting embodiment of that
engine that allows you to understand
where the ego vehicle is the ego thing
is where everything else is what
everything else is gonna do and how do
you react how do you actually you know
interact with humans the right way how
do you build these systems and I think
they want to know that they want to
understand that and so they keep going
and doing that and so on the other
dimension Tesla is doing something
interesting I mean I think that they
have a good product people use it think
that you know like it's not for me
but I can totally see people people like
it and and people I think they have a
good product outside of automation but I
was just referring to the the automation
itself I mean you know like it kind of
drives itself you still have to be kind
of you still have to pay attention to it
right you know people seem to use it so
it works for something and so people I
think people are willing to pay for it
people are willing to buy it I think it
it's it's one of the other reasons why
people buy a Tesla car
maybe one of those reasons is Elon Musk
is the CEO and you know he seems like a
visionary person that's what people
think you know it seems like a visionary
person and so it adds like 5k to the
value of the car and then maybe another
5k is the autopilot and and you know
it's it's useful
I mean it's useful in the sense that
like people are using it and so III can
see Tesla and sure of course they want
to be visionary they want to kind of put
out a certain approach and they may
actually get there but I think that
there's also a primary benefit of doing
all these updates and rolling it out
because you know people pay for it and
it's it's your home it's basic you know
demand supply market and people like it
they're happy to pay another 5k 10k for
that novelty or whatever that is they
and they use it it's not like they get
it and they try it a couple times it's a
novelty but they use it a lot of the
time and so I think that's what Tesla is
doing it's actually pretty different
like they are on pretty orthogonal
dimensions of what kind of things that
they're building they are using the same
AI engine so it's very possible that you
know they're both gonna be sort of one
day kind of using a similar almost like
an internal internal combustion engine
it's a very bad metaphor but similar
internal combustion engine and maybe one
of them is building like a car the other
one is building a truck or something so
ultimately the use case is very
different so you like I said or one of
the founders of Optimus rad let's take a
step back it's one of the success
stories in the autonomous vehicle space
it's a great attack vehicle company
let's go from the very beginning what
does it take to start autonomous vehicle
company how do you go from idea to
deploying vehicles like you are and a
few a bunch of places including New York
I would say that I think that you know
what happened to us is it was was the
following I think we've realized a lot
of kind of talk in the autonomous
vehicle industry back in like 2014 even
when we wanted to kind of get started
and I don't know like I kind of I would
hear things like fully autonomous
vehicles two years from now three years
from now I kind of never bought it you
know I was a part of MIT zorbing Channel
Gentry it kind of like it has an
interesting history so I did in college
and in high school sort of a lot of
mathematically oriented work and I think
I kind of you know at some point it kind
of hit me I wanted to build something
and so I came to MIT mechanical
engineering program and I now realize I
think my advisor hired me because I
could do like really good math but I
told him that no no no I want to work on
that urban challenge car I want to build
the autonomous car and I think that was
that was kind of like a process why we
really learned I mean what the
challenges are and and what kind of
limitations are we up against you know
like having the limitations of computers
or understanding human behavior there's
so many of these things and I think it's
just kind of didn't and so so we said
hey you know like why don't we take a
more like a market-based approach so we
focus on a certain kind of market and we
build a system for that what we're
building is not so much of like an
autonomous vehicle only I would say so
we build full autonomy into the vehicles
but you know the way we kind of see it
is that we think that the approach
should actually involve humans operating
that not just just not sitting in the
vehicle and I think today what we have
is today we have one person operate one
vehicle no matter what that vehicle it
could be a forklift it could be a truck
it could be a car whatever that is and
we want to go from that to ten people
operate 50 vehicles how do we do that
you're referring to a world of maybe
perhaps teleoperation so can you just
say what it means for 10 might be
confusing for people listening what does
it mean for ten people to control 50
vehicles that's a good point so I think
it's am I very deliberately didn't call
it a law operation because people what
people think then is that people think
away from the vehicle sits a person sees
like maybe put some goggles or something
ER and drives the car so that's not at
all what we need but we mean the kind of
intelligence bye-bye
humans are in control except in certain
places the vehicles can execute on their
own and so imagine
like like a room where people can see
what the other vehicles are doing and
everything and you know there will be
some people who are more like more like
air traffic controllers call them like
AV controllers yeah
and so these AV controllers would
actually see kind of like like a whole
map and they would understand where
vehicles are really confident and where
they kind of you know need a little bit
more help and the help shouldn't be for
safety how it should be for efficiency
vehicles should be safe no matter what
if you had zero people they could be
very safe but they be going five miles
an hour and so if you want them to go
around 25 miles an hour then you need
people to come in and and for example
you know the vehicle come to an
intersection and the vehicle can say you
know I can wait
I can inch forward a little bit show my
intent or I can turn left and right now
it's clear I can turn I know that but
before you give me the go I won't and so
that's one example this doesn't mean
necessarily we're doing that actually I
think I think if you go down all them
all that much detail that every
intersection you're kind of expecting a
person to press a button then I don't
think you'll get the efficiency benefits
you want you need to be able to kind of
go around and be able to do these things
but but I think you need people to be
able to set high level behavior to
vehicles that's the other thing with
autonomous vehicles you know I think a
lot of people kind of think about it as
follows I mean this happens with
technology a lot you know you think
alright so I know about cars and I heard
robots so I think how this is gonna work
out is that I'm gonna buy a car press a
button and it's gonna drive itself and
when is that gonna happen you know and
people kind of tend to think about it
that way but when you think about what
really happens is that something comes
in in a way that you didn't even expect
if asked you might have said I don't
think I need that or I don't think it
should be that and so on and then and
then that that becomes the next big
thing coding code and so I think that
this kind of different ways of humans
operating vehicles could be really
powerful I think that sooner than then
later we might open our eyes up to a
world in which you go around walk in a
mall and there's a bunch of security
they're exactly operated in this way you
go into a factory or a warehouse there's
a whole bunch of robots they're pretty
exactly in this way you go to a you go
to the Brooklyn Navy Yard you see a
whole bunch of autonomous vehicles
Optimus right and they're operated maybe
in this way yes but I think people kind
of don't see that III sincerely think
that it's it's there's a possibility
that we may almost see like like a whole
mushrooming of this technology in all
kinds of places that we didn't expect
before and then maybe the real surprise
and then one day when your car actually
drives itself it may not be all that
much of a surprise at all
because you see it all the time you
interact with them you take the Optimus
ride hopefully that's your choice and
then you know you you hear a bunch of
things you go around you in your act
with them I don't know like you have a
little delivery vehicle that goes around
the sidewalks and delivers you things
and then you take it it says thank you
and then you get used to that and one
day your car actually drives itself and
the regulation goes by and you know you
can hit the button asleep and it
wouldn't be a surprise at all I think
that maybe the real reality so there's
gonna be a bunch of applications that
pop up around autonomous vehicles some
some of which maybe many of which we
don't expect at all so if we look at
Optimus ride what do you think you know
the viral application that the one that
like really works for people in mobility
what do you think optimus ride will
connect with in in in near future first
um I think that the first place is that
that I like the target honestly is like
these places where transportation is
required within an environment like
people typically call a geofence so you
can imagine like a roughly two mile by
two mile could be bigger could be
smaller type of an environment and
there's a lot of these kinds of
environments they're typically
transportation deprived the Brooklyn
Navy Yard that you know we're in today
we're in a few different places but
that's that was the one that was less
publicized that's a good example so
there's not a lot of transportation
there and you wouldn't expect like I
don't know I think maybe operating an
uber there ends up being sort of a
little too expensive or when you compare
it with operating uber
that becomes the elsewhere becomes the
priority and these people whose place
has become totally transportation
deprived and then what happens is that
you know people drive into these places
and to go from point A to point B inside
this place within that day they use
their cars and so we end up building
more parking for them to for example
take their cars and go to the lunch
place and I think that one of the things
that can be done is that you know you
can put in efficient safe sustainable
transportation systems into these types
of places first and I think that you
know you could deliver mobility in an
affordable way affordable accessible you
know sustainable way but I think what
also enables is that this kind of effort
money area land that we spend on parking
we could reclaim some of that and that
is on the order of like even for a small
environment like two mile by two mile it
doesn't have to be smack in the middle
of New York I mean anywhere else you're
talking tens of millions of dollars if
you're smack in the middle of New York
you're looking at billions of dollars of
savings just by doing that and that's
the economic part of it and there's a
societal part right I mean just look
around I mean the places that we live
are like built for cars it didn't look
like this just like a hundred years ago
like today no one walks in the middle of
the street it's four cars we no one
tells you that growing up but you grow
into that reality and so sometimes they
close the road
it happens here you know like the
celebration they close the road still
people don't walk in the middle of the
road like just walk in and people don't
but I think it has so much impact the
the car in in the space that we have and
and I think we talked about
sustainability livability I mean
ultimately these kinds of places that
parking spots at the very least could
change into something more useful or
maybe just like park areas recreational
and so I think that's the first thing
that that we're targeting and I think
that we're getting like a really good
response both from an economic societal
point of view especially places that are
a little bit forward-looking
and like for example Brooklyn Navy Yard
they have tenants there
distinct I recall like new lab it's kind
of like an Innovation Center there's a
bunch of startups there and so you know
you get those kinds of people and you
know that they're really interested in
sort of making that environment more
livable and these kinds of solutions
that Optimus tried provides almost kind
of comes in and and becomes that and
many of these places that are
transportation deprived you know they
have they actually ran shuttles and so
you know you can ask anybody the shuttle
experience is like terrible people hate
shuttles and I can tell you why it's
because you know like the driver is very
expensive in a shuttle business so what
makes sense is to attach 2030 seats to a
driver and a lot of people have this
misconception they think that shuttle
should be big sometimes we get that our
optimist right we tell them we're gonna
give you like four seater six Cedars and
we get asked like how about like twenty
Cedars like you know you don't need
twenty Cedars you want to split up those
seeds so that they can travel faster and
the transportation delays would go down
that's what you want if you make it big
not only you will get delays in
transportation but you won't have an
agile vehicle it will take a long time
to speed up slow down and so on
it'll you need to climb up to the thing
so it's kind of like really hard to
interact with and scheduling too perhaps
when you have more smaller vehicles
because closer to BER where you can
actually get a personal I mean just the
logistics of getting the vehicle to you
it becomes easier when you have a giant
shadow there's fewer of them and it
probably goes on a route a specific
route that's supposed to hit and when
you go on a specific route and all seats
travel together versus you know you have
a whole bunch of them you can imagine
the route you can still have but you can
imagine you split up the seats and
instead of you know damn traveling like
I don't know a mile apart they could be
like you know half a mile apart if you
split them into two that basically would
mean that your delays when you go out
you want wait for them for a long time
and that's one of the main reasons or
you don't have to climb up the other
thing is that I think if you split them
up in a nice way and if you can actually
know where people are going to be
somehow
you don't even need the app a lot of
people ask us the app we say why don't
you just walk into the vehicle how about
you just walk into the vehicle it
recognizes who you are
and it gives you a bunch of options of
places that you go and you just kind of
go there
I mean people kind of also internalize
the apps everybody needs a nap it's like
you don't need an app you just walk into
the place walk up but I think I think
one of the things that you know we
really try to do is to take that shuttle
experience that no one likes and tilt it
into something that everybody loves and
so I think that's another important
thing I would like to say that carefully
just like today operationally we don't
do shuttles you know we're really kind
of thinking of this as a system or a
network that we're designing but but
ultimately we go to places that would
normally rent the shuttle service that
people wouldn't like as much and we want
to tilt it into something that people
love so you virtually the second earlier
but how many optimist ride vehicles do
you think would be needed for any person
in Boston or New York if they step
outside there will be this this is like
a mathematical question there'll be two
optimist ride vehicles within line of
sight
is that the right number - well these
for example um that's that's the density
so meaning that if you see one vehicle
you look around you see another one -
imagine like you know Tesla will tell
you they collect a lot of data do you
see that with Tesla like you just walk
around and you look on you see Tesla
probably not very specific areas of
California maybe maybe you're right like
there's a couple zip codes that you know
just but I think but I think that's kind
of important because you know like maybe
the couple zip codes um the one thing
that we kind of depend on I'll get to
your question in a second but now like
we're taking a lot of tangents today oh
yeah so so so I think that this is
actually important people call this data
density or data velocity so it's very
good to collect data in a way that you
know you see the same place so many
times like you can drive 10,000 miles
around the country or you drive 10,000
miles in a confined environment you'll
see the same intersection hundreds of
times and when it comes to
dick ting what people are gonna do in
that specific intersection we become
really good at it versus if you draw in
like ten thousand miles around the
country you sing that only once and so
trying to predict what people do become
sorry and I think that you know you said
what is needed it's tens of thousands of
vehicles you know you really need to be
like a specific fraction or vehicle like
for example in good times in Singapore
you can go and you can just grab a cab
and they are like you know 10% 20% of
traffic those taxis ultimately that's
why you need to get to so that you know
you you get to a certain place where you
really the benefits really kick off and
like orders of magnitude type of a point
but once you get there you actually get
the benefits and you can certainly carry
people I think that's one of the things
people really don't like to wait for
themselves but for example they can wait
a lot more for the goods if they order
something like there you were sitting at
home and you want to wait half an hour
that sounds great people say it's great
you want to you're gonna take a cab
you're waiting half an hour like that's
crazy you don't want to wait that much
but I think you know you you can I think
really get to a point where the system
at peak times really focuses on kind of
transporting humans around and then it's
it's really it's a good fraction of
traffic to the point where you know you
go you look around there's something
there and you just kind of basically get
in there and it's already waiting for
you or something like that and then you
take it if you do it at that scale like
today for instance uber if you talk to a
driver right I mean uber takes a certain
cut it's a small cut or drivers would
argue that it's a large cut but you know
it's it's it's when you look at the
grand scheme of things most of that
money that you pay Hueber
kind of goes to the driver and if you
talk to the driver the driver will claim
that most of it is their time you know
they it's not spent on gas they think
it's not spent on the the car per se as
much it's like their time and if you
didn't have a have a person driving or
if you're in a scenario where you know
like point one person is driving the car
a fraction of a person is kind of
operating the car because you know your
one operates several if you're in that
situation you realize that the internal
combustion engine type of cars are very
inefficient you know we built them to go
on highways they pass crash tests
they're like really heavy they really
don't need to be like 25 times the
weight of its passengers or or you know
like area wise and so on and but if you
get through those inefficiencies and if
you really build like urban cars and
things like that I think the economics
really starts to check out like to the
point where I mean I don't know you may
be able to get into a car and it may be
less than a dollar to go from A to B as
long as you don't change your
destination you just pay 99 cents and go
that if you share it if you take another
stop somewhere it becomes a lot better
you know these kinds of things at least
four models at least for mathematics and
theory they start to really check out so
I think it's really exciting what
Optimus Art is doing in terms of it
feels the most reachable like they'll
actually be here and have an impact yeah
that is the idea and if we contrast that
again we'll go back to our old friends
way Moe and Tesla so way Moe seems to
have sort of technically similar
approaches as Optimus ride but a
different they're not as interested it
has having an impact today these in
nature they have a longer term sort of
investments almost more of a research
project still meaning they're trying to
solve as far as I understand maybe you
can you can differentiate but they seem
to want to do more unrestricted movement
meaning move from A to B where A to B is
all over the place versus Optimus right
is really nicely geofence and really
sort of established mobility in a
particular environment before you expand
it and then Tesla is like the complete
opposite which is you know the entirety
of the world actually is going to be
automated highway driving urban driving
every kind of driving
you know you kind of creep up to it by
incrementally improving the capabilities
of the autopilot system so when you
contrast all of these and on top of that
let me throw a question that nobody
likes
but his timeline when do you think each
of these approaches loosely speaking
nobody can predict the future will see
mass deployment so yah mosque predicts
the the craziest approach is at the I've
heard figures like at the end of this
year right so that's probably wildly
inaccurate but how wildly inaccurate is
it I mean first thing to lay out like
everybody else it's really it's really
hard to guess I mean I don't know I
don't know where where Tesla can look at
or Elon Musk can look at and say hey you
know it's the end of this year I mean I
don't know what you can look at you know
even the data that you know you I mean
if you look at the data even kind of
trying to extrapolate the end state
without knowing what exactly is gonna go
especially for like a machine learning
approach I mean it's just kind of very
hard to predict but I do think the
following does happen I think a lot of
people you know what they do is that
there's something that I called a couple
times time dilation in technology
prediction happens let me try to
describe a little bit there's a lot of
things that are so far ahead people
think they're close and there's a lot of
things that are actually close people
think it's far ahead
people tries to kind of look at a whole
landscape of technology development
admit needs chaos anything can happen in
any order at any time and there's a
whole bunch of things in that people
take it clamp it and put it into the
next three years and so then what
happens is that there's some things that
maybe can happen by the end of the year
or next year and so on and they push
that into like few years ahead because
it's just hard to explain and there are
things that are like we were looking at
20 years more maybe you know hopefully
in my lifetime type of things and cuz
you know we don't know I mean we don't
know how hard it is even like that's a
problem we don't know like if some of
these problems are actually AI complete
like we have no idea what's going on and
and you know we we take all of that and
then we clump it and then we say three
years from now and then some of us are
more optimistic so they're shooting at
the at the end of the year and some of
us are more realistic they say like five
years but you know we all I think it's
just hard to know and and I think trying
to predict like products ahead to three
years it's it's hard to know in the
following sense you know like we
typically say okay this is a technology
company but sometimes sometimes really
you're trying to build something where
technology does like there's a
technology gap you know like and Tesla
had that with electric vehicles you know
like when they first started they would
look at a chart much like a moose law
type of chart and they would just kind
of extrapolate that out and they'd say
we want to be here what's the technology
to get that we don't know it goes like
this so it's probably just gonna you
know keep going yeah um with bit AI that
goes into the cars we don't even have
that like we can't I mean what can you
quantify yeah like what kind of chart
are you looking at you know but so but
so I think when there's the technology
gap it's just kind of really hard to
predict so now I realize I talk like
five minutes and avoid your question I
didn't tell you anything about and I
don't think you I think you've actually
argued that it's not used even NES you
provide now is not that used to be very
hard there's one thing that I really
believe in and and you know this is not
my idea and it's been you know discussed
several times but but this this this
kind of like something like a startup or
a kind of an innovative company
including definitely may want may vary
more Tesla maybe even some of the other
big companies that are kind of trying
things this kind of like iterated
learning is very important the fact that
we're over there and we're trying things
and so on I think that's that's
important we try to understand and and I
think that you know the coding code
Silicon Valley has done that with
business models pretty well
and now I think we're trying to get to
do it well there's a little technology
gap
I mean before like you know you're
trying to build I'm not trying to you
know I think these companies are
building great technology to for example
enable internet search to do it so
quickly and that kind of didn't what
wasn't there so much but at least like
it was a kind of a technology that you
could predict to some degree and so on
and now we're just kind of trying to
build you know things that it's kind of
hard to quantify what kind of a metric
are we looking at so psychologically is
a sort of as a leader of graduate
students and an optimist ride a bunch of
brilliant engineers just curiosity
psychologically do you think it's good
to think that you know whatever
technology gap we're talking about can
be closed by the end of the year or do
you you know because we don't know so
the way do you want to say that
everything is going to improve
exponentially to yourself and to others
around you as a leader or do you want to
be more sort of maybe not cynical but I
don't want to use realistic because it's
hard to predict but yeah maybe more
cynical pessimistic about the ability to
close again yeah I I think that you know
going back I think that iterated
learning is like key that you know
you're out there you're running
experiments to learn and that doesn't
mean sort of like you know you like like
your optimist right you're kind of doing
something but I like in an environment
but like what Tesla is doing I think is
also kind of like this this kind of
notion and and you know people can go
around and say like you know this year
next year the other year and so on but
but I think that the nice thing about it
is that they're out there they're
pushing this technology in I think what
they should do more of I think that kind
of informed people about what kind of
technology that they're providing you
know the good and the bad and then you
know not just sort of you know if it
works very well but I think you know I'm
not saying they're not doing bad and
informing I think they're kind of trying
they you know they put up certain things
or at the very least YouTube videos
comes out on on how the summon function
works every now and then and and you
know people get informed
and so that that kind of cycle continues
but you know I I admired I think they're
kind of go out there and they do great
things they do their own kind of
experiment I think we do our own and I
think we're closing some similar
technology gaps but some also some are
orthogonal as well you know I think like
like we talked about you know people
being remote like it's something or in
the kind of environments that we're in
or think about a test the car maybe
maybe you can enable it one day like
there's you know low traffic like you're
kind of the stuff on go emotion you just
hit the button and the you can really
say or maybe there's another you know
Lane that you can pass into you going
that I think they can enable these kinds
of pride believe it and so I think that
that part that is really important and
that is really key and and beyond that I
think you know when is it exactly gonna
happen and and and so on I mean it's
like I said it's very hard to predict
and I would I would imagine that it
would be good to do some sort of like a
like a one or two year plan when it's a
little bit more predictable that you
know you the technology gaps you close
and and there and the kind of sort of
product that would answer so I know that
from optimist ride or you know other
companies that I get involved in I mean
at some point you find yourself in a
situation where you're trying to build a
product and and people are investing in
that in that you know building effort
and those investors that they do want to
know as they compare the investments
they want to make they do want to know
what happens in the next one or two
years and I think that's good to
communicate that but I think beyond that
it becomes it becomes a vision that we
want to get to someday and saying five
years ten years I don't think it means
anything but iterative learning is key
though you do and learn I think that is
key you know I got a sort of throwback
right at you criticism in terms of you
know like Tesla or somebody
communicating you know how someone works
and so on I got a chance to visit
Optimus ride and you guys are doing some
awesome stuff and yet the internet
doesn't know about it so you should also
communicate more showing off in
showing off some of the awesome stuff
the stuff that works and stuff that
doesn't work I mean it's just the stuff
I saw with the tracking different
objects and pedestrians so I'm
incredible stuff going on there just
cool maybe it's just the nerd of me but
I think the world would love to see that
kind of stuff yeah that's that's well
taken I think you know I should say that
it's not like you know we we weren't
able to I think we made a decision at
some point that decision did involve me
quite a bit on kind of sort of doing
this in kind of coding called stealth
mode for a bit but I think that you know
we will open it up quite a lot more and
I think that we are also that optimist
right kind of hitting when you new era
you know we're big now we're doing a lot
of interesting things and and I think
you know some of the deployments that we
kind of announced were some of the first
bits bits of information that we kind of
put out into the world we'll also put
out our technology a lot of the things
that we've been developing is really
amazing and you know we're gonna we're
gonna start putting it out now we're
especially interested in sort of like
being able to work with the best people
and I think and I think it's it's good
to not just kind of show them and they
come to our office for an interview but
just put it out there in terms of like
you know get people excited about what
we're doing so on Thomas vehicle space
let me ask one last question so yah
mosque famously said that lighter is a
crutch so uh I've talked to a bunch of
people bought it got asked you you use
that crutch quite a bit in the DARPA
days so you know and is that his idea in
general sort of you know more
provocative and fun I think than a
technical discussion but the idea is
that camera based can't primarily camera
based systems is going to be what
defines the future of autonomous
vehicles so what do you think of this
idea ladders a crutch versus primarily
uh camera based systems first things
first I think you know I'm a big
believer in just camera based autonomous
vehicle systems like I think that you
know you can put in a lot of autonomy
and
and you can do great things and and it's
it's it's very possible that at the time
scales like we said we can't predict
twenty years from now like you may be
able to do do things that we're doing
today only what lidar and you may be
will do them just with cameras and I
think that you know you can just I I
think that I will put my name on it to
like you know there will be a time when
you can only use cameras and you'll be
fine at that time though it's very
possible that you know you find the
lidar system as another robusta fire or
or it's so affordable that it's stupid
not to you know just kind of put it
there and I think and I think we may be
looking at a future like that do you
think we're over relying on lidar
right now because we understand it
better it's more reliable anyways
internment from a safety easier to build
with that's the other that's the other
thing I think to be very frank with you
I mean you know we've seen a lot of sort
of autonomous vehicles companies come
and go and the approach has been you
know you slap a lidar on a car and it's
kind of easy to build with when you have
a lighter are you know you just kind of
coat it up and and you hit the button
and you do a demo so I think there's
admittedly there's a lot of people they
focus on the lidar because it's easier
to build with that doesn't mean that you
know without the cameras just cameras
you can you cannot do what they're doing
but it's just kind of a lot harder and
so you need to have certain kind of
expertise to exploit that what we
believe in and you know you may be
seeing some of it is that we believe in
computer vision we certainly work on
computer vision and optimist ride by a
lot like um and and we've been doing
that from day one and we also believe in
sensor fusion so you know we do we have
a relatively minimal use of light ours
but but we do use them and I think you
know in the future I really believe that
the following sequence of events may
happen first things first number one
there may be a future in which you know
there's like cars with light hours and
everything and the cameras but you know
this in this 50 year ahead future they
can
drive with cameras as well especially in
some isolated environments and cameras
they go and they do the thing in the
same future it's very possible that you
know the white ARS are so cheap and
frankly make the software may be a
little less compute-intensive
at the very least or maybe less
complicated so that they can be
certified or or insured there of their
safety and things like that that it's
kind of stupid not to put the lidar like
imagine this you either put pay money
for the lidar or you pay money for the
compute and if you don't put the lidar
it's a more expensive system because you
have to put in a lot of compute like
this is another possibility I do think
that a lot of the sort of initial
deployments of self-driving vehicles I
think they will involve light ARS and
especially either low range or short
either short range or low resolution
light ARS are actually not that hard to
build in solid state they're still
scanning but like MEMS type of scanning
light ours and things like that they're
like they're actually not that hard I
think they will may be kind of playing
with the spectrum and the phaser eyes
they're a little bit harder but but I
think like you know putting your mom's
mirror in there that kind of scans the
environment it's not hard the only thing
is that you know you just like with a
lot of the things that we do nowadays in
developing technology you hit
fundamental limits of the universe the
speed of light becomes a problem in when
you're trying to scan the environment so
you don't get either good resolution or
you don't get range but but you know
it's still it's something that you can
put in that affordably so let me jump
back to drones you've uh you have a role
in the Lockheed Martin alpha pilot
Innovation Challenge where teams compete
in drone racing a super cool super
intense interesting application of AI so
can you tell me about the very basics of
the challenge and where you fit in well
your thoughts are on this problem and
it's sort of echoes of the early DARPA
challenge in the through the desert that
we're seeing now now with drone racing
yeah I mean one interesting thing about
it is that you know people drone racing
a
this is an eSport and so it's much like
you're playing a game but there's a real
drone going in an environment the human
being is controlling it with goggles on
so there's no it is a robot but there's
no AI there's no way I am human being is
controlling it and so that's already
there and and I've been interested in
this problem for quite a while actually
from a robot assist point of view and
that's what's happening in alpha pilot
which which probably of aggressive
flight of aggressive flight fully
autonomous aggressive flight the problem
that I'm interested in you asked about
alpha pod and I'll get there in a second
but the problem that I'm interested in
I'd love to build autonomous vehicles
like like drones that can go far faster
than any human possibly can I think we
should recognize that we as humans have
you know limitations in how fast we can
process information and those are some
biological limitations like we think
about this AI this way too I mean this
has been discussed a lot and this is not
sort of my idea per se but a lot of
people kind of think about human level
III and they think that you know AI is
not human level one day it'll be human
level and humans in the eyes they kind
of interact versus I think that the
situation really is that humans are at a
certain place and AI keeps improving and
at some point just crosses off and then
you know it gets smarter and smarter and
smarter and so drone releasing the same
issue humans play this game and you know
you have to like react in milliseconds
and there's really you know you see
something with your eyes and then that
information just flows through your
brain into your hands so that you can
command it and there's some also delays
and you know getting information back
and forth but suppose a laser don't
exist you just just a delay between your
eye and your fingers please delay that a
robot doesn't have to have so we end up
building in my research group like
systems that you know see things at a
kilohertz like a human eye would barely
hit a hundred Hertz so imagine things
that see stuff in slow motion like 10x
slow motion it will be very useful like
we talked a lot about autonomous car so
you know we don't get to see it but the
hundred lives are lost every day just in
the United States on traffic accidents
and many of them are like known cases
you know like the you're coming through
like like a ramp going into a highway
you hit somebody and you're off or you
know like you kind of get confused you
try to like swerve into the next lane
you go off the road and you crash
whatever and I'm I think if you had
enough computer in a car and a very fast
camera right at the time of an accident
you could use all compute you have like
you could shut down the infotainment
system and use that kind of computing
resources instead of rendering you use
it for the kind of artificial
intelligence that goes in there the
autonomy and you can you can either take
control of the car and bring it to a
full stop but even even if you can't do
that you can deliver what the human is
trying to do human is trying to change
the lane but goes off the road not being
able to do that with motor skills and
the eyes and you know you can get in
that and I was there's so many other
things that you can enable with what I
would call high throughput computing you
know data is coming in extremely fast
and in real time you have to process it
and the current CPUs have ever fast you
clock it are typically not enough you
need to build those computers from the
ground up so that they can ingest all
that data that I'm really interested in
just on that point really quick is the
currently what's the bottom like you
mentioned the delays in humans is it the
hardware so you work a lot with NVIDIA
hardware is it the hardware is it the
software I think it's both I think it's
both in fact they need to be
co-developed I think in the future I
mean that's a little bit what Nvidia
does sort of like they almost like build
the hardware and then they build the
neural networks and then they build the
hardware back and the neural networks
back and it goes back and forth but it's
that Co design and I think that you know
like we try to way back we try to build
a faster own that could use a camera
image to like track what's moving in
order to find where it is in the world
this typical sort of you know visual
inertial state estimation problems that
we would solve and you know we just kind
of realize that we're at the limit
sometimes of you know doing simple tasks
we're at the limit of the camera frame
rate because you know you really want to
track things you want the camera image
to be 90% kind of like or or some
somewhat the same from one frame to the
next and why are we at the limit of the
camera frame rate it's because camera
captures data it puts it into some
serial connection
it could be USB or like there's
something called camera serial interface
that we use a lot it puts into some
serial connection and copper wires can
only transmit so much data and you hit
the Shannon limit on copper wires and
you know you you hit yet another kind of
Universal limit that you can transfer
the data so you have to be much more
intelligent on how you capture those
pixels you can take compute and put it
right next to the pixels people are
governed all that you do how hard is to
get past the bottleneck of the copper
wire yeah you need to you need to do a
lot of parallel processing as you can
imagine the same thing happens in the
GPUs you know like the data is
transferred in parallel somehow it gets
into some parallel processing I think
that you know like now we're really kind
of diverted off into so many different
dimensions but great so its aggressive
light how do we make drones see many
more frame just a second
in order to enable aggressive fight
that's a super interesting problem
that's an interesting problem so but
like think about it you have you have
CPUs you clock them at you know several
gigahertz we don't clock them faster
largely because you know we run into
some heating issues and things like that
but another thing is that 3 gigahertz
clock light travels kind of like on the
order of a few inches or an inch that's
the size of a chip and so you pass a
clock cycle and as the clock signal is
going around in the chip you pass
another one and so trying to coordinate
that the design of the complexity of the
chip becomes so hard I mean we have hit
the fundamental limits of the universe
in so many things that we're designing I
don't know I realize that it's great but
like we can't make transistors smaller
because like quantum effects that
electrons start to tunnel around
we can't clock it faster
one of the reasons why is because like
the information doesn't travel faster in
the universe
yeah and we're limited by that same
thing with the laser scanner but so then
it becomes clear that you know the way
you organize the chip into a CPU or even
a GPU you now need to look at how to
redesign that if you're gonna stick with
silicon yes you could go do other things
too I mean there's that too but you
really almost need to take those
transistors put them in a different way
so that the information travels on those
transistors in a different way in a much
more way that is specific to the
high-speed cameras coming in and so
that's one of the things that that we
talk about quite a bit so drone racing
kind of really makes that embodies that
he embodies that and that's what is
exciting it's exciting for people you
know students like it it embodies all
those problems but going back we're
building coding code and other engine
and that engine I hope one day we'll be
just like how impactful seatbelts were
in in driving I hope so Wow or it could
enable your next generation autonomous
air taxis and things like that I mean it
sounds crazy but one day we may need to
purge that these things if you really
want to go from Boston to New York in
more than a half hours you may want to
fixed-wing aircraft most of these
companies that are kind of doing
Concorde flying cars they're focusing on
that but then how do you land it on top
of a building you may need to pull off
like kind of fast maneuvers for a robot
like perch land it's gonna go perch into
into a building if you want to do that
like you need these kinds of systems and
so drone racing you know it's being able
to go very faster than anything we can't
comprehend take an aircraft forget the
quadcopter we take your fixed-wing while
you're at it you might as well put some
like rocket engines in the back you just
light it you go through the gate and
everyone looks at it and just said what
just happened yeah and they would say
it's impossible for me to do that and
that's closing the same technology gap
that would you know one day steer cars
out of accidents so but
let's get back to the practical which is
sort of just getting the thing to work
in a race environment which is kind of
what the is another kind of exciting
thing which the DARPA challenge to the
desert did you know theoretically we had
autonomous vehicles but making them
successfully finish a race first of all
which nobody finished the first year and
then the second year just to get you
know to finish and really go at a
reasonable time is really difficult
engineering practically speaking
challenge so that let me ask about the
the Alpha pilot challenge is a I guess a
big prize potentially associated with it
but let me ask reminiscent of the DARPA
Days predictions you think anybody will
finish well not not soon I think that
depends on how you set up the race
course and so if the race course is a
slalom course I think people will kind
of do it but can you set up some course
like literally sunk or you get to design
it
there is the algorithm developer can you
set up some course so that you can beat
the best human when is that gonna happen
like that's not very easy even just
setting up some course if you let the
human that you're competing with set up
the course it becomes a worries a lot
harder hmm so how many in the space of
all possible courses are would humans
win and quad machines were a great
question let's get to that I want to
answer your other question which is like
the DARPA challenge days right what was
really hard I think I think we
understand we understood what we wanted
to build but still building things that
experimentation that iterated learning
that takes up a lot of time actually and
and so in my group for example in order
for us to be able to develop fast we
build like VR environments will take an
aircraft will put it in a motion capture
room big huge motion capture room and
we'll fly it in real time will render
other images and beam it back to the
drone that sounds kind of notionally
simple but it's actually hard because
now you're trying to fit all that data
through the air into the drone and so
you need to
do a few crazy things to make it happen
but once you do that then at least you
can try things if you crash into
something you didn't actually crash so
it's like the whole drone is in VR we
can do augmented reality and so on
and so I think at some point testing
becomes very important one of the nice
things about alpha pilot is that they
built the drone and they build a lot of
drones and and it's okay to crash in
fact I think maybe you know the viewers
may kind of like to see things that
suppose that potentially could be the
most exciting part it could be the
exciting part and I think you know as an
engineer it's a very different situation
to be in
like in academia a lot of my colleagues
who are actually in this race and
they're really great researchers but
I've seen them trying to do similar
things whereby they built this one drone
and you know some somebody with like a
face mask and a glows are going you know
right behind the drone is trying to hold
it if it if it falls down imagine you
don't have to do that I think that's one
of the nice things about auto pilot
challenge where you know we have this
drones and we're going to design the
courses in a way that will keep pushing
people up until the crashes start to
happen and you know we'll hopefully sort
of I don't think you want to tell people
crashing is okay if we want to be
careful here but because you know we
don't people to crash a lot but
certainly you know we want we want them
to push it so that you know everybody
crashes once or twice and and and you
know they're really pushing it to their
limits and that's where iterated
learning comes in as ever every crash is
a lesson is the lesson exactly so in
terms of the space of possible courses
how do you think about it in in the in
the war of the video versus machines or
do machines when we look at that quite a
bit I mean I think that you know you
will see quickly that like you can
design a course and you know in in in
certain courses like in the middle
somewhere if if you kind of run through
the course once you know the Machine
gets beaten pretty much consistently by
slightly but if you go through the
course like 10 times humans get beaten
very slightly but consistently so humans
at some point you know you get confused
you get tired and things like that
versus
machine is just executing the same line
of code tirelessly just going back to
the beginning and doing the same thing
exactly I think I think that kind of
thing happens and I realize sort of as
humans
there's the classical things you know
that everybody has realized like like if
you put in some sort of like strategic
thinking that's a little bit harder for
machines that I think sort of comprehend
precision is easy to do so that's what
they excel in and also sort of
repeatability is easier to do that's
what they excel in they can you can
build machines that excel in strategy as
well and beat humans that way too but
that's a lot harder to build I have a
million more questions but in the
interest of time last question yeah what
is the most beautiful idea you've come
across in robotics
well their simple equation experiment a
demo simulation piece of software what
just gives you pause that's an
interesting question I have done a lot
of work myself in decision making so
I've been interested in that area so you
know in robotics you have somehow the
field has split into like you know
there's people who would work on like
perception how robots perceive the
environment then how do you actually
make like decisions and there's people
also like how do you interact people
interact with drove us is a whole bunch
of different fields and and you know I I
have admittedly worked a lot on the more
control and decision-making
than the others and I think that you
know the one equation that has always
kind of baffled me is Bellman's equation
and so it's it's this person who have
realized like way back you know more
than half a century ago on like how do
you actually sit down and if you have
several variables that you're kind of
jointly trying to determine how do you
determine that
and there is one beautiful equation that
you know like today people do
reinforcement where we still use it and
and it's it's baffling to me because it
both kind of
tells you the simplicity because it's a
single equation that anyone can write
now we can teach it in the first course
on decision-making at the same time it
tells you how computation we how hard
the problem is I feel like my like a lot
of the things that I've done at MIT for
research has been kind of just this
fight against computational efficiency
things like how can we get it faster to
the point where we now got to like let's
just redesign this chip like maybe
that's the way but I think it talks
about how computationally hard certain
problems can be by nowadays what people
call curse of dimensionality and so as
the number of variables cannot grow the
number of decisions you can make grows
rapidly like if you have you know 100
variables each one of them take ten
values all possible assignments is more
than the number of atoms in the universe
it's just crazy and and that kind of
thinking is just embodied in that one
equation that I really like and the
beautiful balance between it being
theoretically optimal and somehow
practically speaking given the curse of
dimensionality nevertheless in practice
works among you know despite all those
challenges which is quite incredible
it's just quite incredible so you know I
would say that it's kind of like quite
baffling actually you know in a lot of
fields that we think about how little we
know you know like and so I think here
too you know we know that in the worst
case things are pretty hard but you know
in practice generally things work so
it's just kind of its kind of baffling
decision-making how little we know just
like how little we know about the
beginning of time how little we know
about you know our own future like if
you actually go into like from balanced
equation all the way down I mean there's
also how little we know about like
mathematics I mean we don't even know
the axioms are like consistent it's just
crazy yeah yeah I think a good lesson
the lesson there just as you said we
tend to focus on the worst case or the
the boundaries of everything we're
studying and then the average case seems
to somehow work out if you think about
life in general
we mess it up a bunch you know we
freaked out about a bunch of the
traumatic stuff but in the end it seems
to work out ok yeah that seems like a
good metaphor sir touch it thank you so
much for being a friend a colleague a
mentor that really appreciates it on and
talk to you like mice thank you thanks
thanks for listening to this
conversation with sir - karma and thank
you to a presenting sponsor cash app
please consider supporting the podcast
by downloading cash app and using code
lex podcast enjoy this podcast subscribe
my youtube review it with 5 stars an
apple podcast supported on patreon or
simply connect with me on Twitter at Lex
Friedman and now let me leave you with
some words from Hal 9000 from the movie
2001 a Space Odyssey I'm putting myself
to the fullest possible use which is all
I think that any conscious entity can
ever hope to do thank you for listening
and hope to see you next time
you