Kind: captions
Language: en
the following is a conversation with
dimitri dalgov
the cto of waymo which is an autonomous
driving company that started
as google's self-driving car project in
2009 and became waymo in 2016.
dimitri was there all along waymo is
currently leading in the fully
autonomous vehicle space
in that they actually have an at-scale
deployment of
publicly accessible autonomous vehicles
driving passengers around
with no safety driver with nobody
in the driver's seat this to me is an
incredible
accomplishment of engineering on one of
the most difficult
and exciting artificial intelligence
challenges of the 21st century
quick mention of a sponsor followed by
some thoughts related to the episode
thank you to trial labs a company that
helps businesses
apply machine learning to solve real
world problems
blinkist an app i use for reading
through summaries of books
better help online therapy with a
licensed professional
and cash app the app i use to send money
to friends
please check out the sponsors in the
description to get a discount and to
support this podcast
as a side note let me say that
autonomous and semi-autonomous driving
was the focus of my work at mit and it's
a problem space that i find
fascinating and full of open questions
from both a robotics
and a human psychology perspective
there's quite a bit
that i could say here about my
experiences in academia on this topic
that revealed to me let's say the
less admirable size of human beings but
i choose to focus on the positive on
solutions
i'm brilliant engineers like dimitri and
the team at waymo
who work tirelessly to innovate and to
build amazing technology
that will define our future because of
dimitri
and others like him i'm excited for this
future
and who knows perhaps i too will help
contribute something of value
to it if you enjoy this thing subscribe
on youtube review it with five stars and
up a podcast
follow on spotify support on patreon or
connect with me on twitter
at lex friedman and now here's my
conversation
with dmitry dolgov when did you first
fall in love with
robotics or even computer science more
in general computer science first
at a fairly young age and robotics
happened much later
um i i think my first
interesting introduction to computers
was
in the late 80s uh
when we got our first computer i think
was an
uh an ibm i think ibm it remember those
things that had like a turbo button in
the front you'd press it and you know
make the thing go faster did they
already have floppy disks
yeah yeah yeah like the the five point
four inch ones i think there's a bigger
inch so good when something then five
inches and three inches
yeah i think that was the five i don't i
maybe that was before that was the giant
plates and i didn't get that
uh but it was definitely not the not the
three inch ones
uh anyway so that that you know we got
that uh
computer i spent the first a few
months just you know playing video games
uh as you would expect
i got bored of that so i started messing
around
and trying to figure out how to make the
thing
do other stuff got into
exploring know programming and a couple
of years
later i got to a point where um i
actually wrote a game
a lot of games and a game developer a
japanese game developer
actually offered to buy it for me for
you know a few hundred bucks but you
know for for a kid
yeah in russia that's a big deal that's
a big deal yeah uh i did not take the
deal
wow integrity yeah i i instead uh
stupidity yes that was not the most
acute financial move that i made in my
life you know looking back at it now
uh i instead put it well you know i had
a reason i i put it online
uh it was what did you call it back in
the days it was a freeware i think
right it was not open source but you
could upload the binaries you put the
game online and the idea was that you
know people like it and then they you
know
contribute and they send you little
donations right so i did my quick math
of like you know my of course you know
thousands and millions of people are
going to play my game send me a couple
of bucks a piece you know
should definitely do that as i said not
not the best
remember what language it was what
programming it was about
which what pascal pascal and they had a
graphical component
so that text based yeah yeah it was uh
like
i think 320 by 200 whatever it was
i think that kind of the earlier that's
the cga resolution right
and i actually think the reason why this
company wanted to buy it is not like the
fancy graphics or the implementation
it was maybe the idea uh of my actual
game
the idea of the game okay well one of
the things
it's so funny i used to play this game
called golden axe
and the simplicity of the graphics
and something about the simplicity of
the music like
it still haunts me i don't know if
that's a childhood thing i don't know if
that's the same thing for call of duty
these days for young kids
but i still think that the
simple one the games are simple that
simple purity makes for
like allows your imagination to take
over and thereby creating a more magical
experience
like now with better and better graphics
it feels like
your imagination doesn't get to uh
create worlds
which is kind of interesting um it could
be just an
old man on a porch like waving at kids
these days that have no respect but
i still think that graphics almost get
in the way of the experience
i don't know flippy bird yeah
i don't know if the imagination gets
closed i don't
yeah but that that's more about games
that up like that's more like tetris
world where they
optimally masterfully
like create a fun
short-term dopamine experience versus
i'm more referring to like
role-playing games where there's like a
story you can live in it for
months or years um like uh
there's an elder scroll series which is
probably my favorite set of games
that was a magical experience and then
the graphics were terrible
the characters were all randomly
generated but they're i don't know
that's
it pulls you in there's a story it's
like an
interactive version of an elder scrolls
tolkien world and you get to live in it
i don't know
i miss it it's one of the things that
suck about being an
adult is there's no you have to live in
the real world as opposed to
the elder scrolls world you know
whatever brings you joy right
minecraft right minecraft is a great
example you create like it's not the
fancy graphics but it's the
creation of your own worlds yeah that
one is crazy
you know one of the pitches for being a
parent that people tell me
is that you can like use the excuse of
parenting to
to go back into the video game world and
like
like that's like you know father-son
father-daughter time
but really you just get to play video
games with your kids so anyway
at that time did you have any ridiculous
ambitious dreams of
where as a creator you might go as an
engineer did you
what did you think of yourself as as an
engineer as a tinkerer or did you want
to be like an astronaut
or something like that you know i'm
tempted to make something up about you
know robots
uh engineering or you know mysteries of
the universe but
that's not the actual memory that pops
into my mind uh when you when you ask me
about childhood dreams so i'll actually
share the real thing
uh when i was
maybe four or five years old i you know
as well do i
thought about you know what i wanted to
do when i grow up and i had this dream
of being
a traffic control cop you know they
don't have those today's i think but you
know
back in the 80s and you know in russia i
you probably are familiar with that legs
they had these
uh you know police officers they would
stand in the middle of an intersection
all day
and they would have their like striped
black and white batons that they would
use to you know control the flow of
traffic
and you know for whatever reason i was
strangely infatuated with this whole
process and like that that was my dream
uh that's what i wanted to do when i
grew up and you know my
parents uh both physics profs by the way
i think we're
you know a little concerned uh with that
level of ambition coming from their
child
yeah uh you know that age well that it's
an interesting i don't know if
you can relate but i very much love that
idea
i have a ocd nature that i think lends
itself
very close to the engineering mindset
which is you want to kind of
optimize you know solve a problem by
create creating an automated
solution like like a set of rules the
set of rules you could follow
and then thereby make it ultra efficient
i don't know if that's
it was it of that nature i certainly
have that there's like fact
like simcity and factory building games
all those kinds of things
kind of speak to that engineering
mindset or did you just like the uniform
i think it was more of the latter i
think it was the uniform and
you know the the striped baton that made
cars go
in the right direction but i guess you
know it is
i did end up uh i guess uh you know
working on the transportation industry
one way or another no uniform though but
that's right
maybe it was my you know deep inner
infatuation with
the you know traffic control batons that
led to this
career okay what uh when did you
when was the leap from programming to
robotics that happened later that was
after grad school
uh after and actually the most
self-driving cars was i think my first
real hands-on introduction to robotics
but i i never
really had that much hands-on experience
in school and training i you know worked
on applied math
and physics then in college i did more
of abstract computer science
and it was after grad school that i
really got
involved in robotics which was actually
self-driving cars and
you know that was a big big flip what uh
well grad school
so i went to grad school in michigan and
then i did a postdoc at stanford uh
which is
that was the postdoc where i got to play
with celebrating cars
yeah so we'll return there let's go back
to uh to moscow so i
you know for episode 100 i talked to my
dad and also i grew up with my dad
i guess uh
so i had to put up with him for many
years and uh
he he went to the fistiach
or mipt it's weird to say in english
because i've heard all this in russian
moscow institute of physics and
technology and to me that was like
i met some super interesting as a child
i met some super interesting characters
it felt to me like the greatest
university in the world
the most elite university in the world
and just the the people
that i met that came out of there were
like
not only brilliant but also special
humans
it seems like that place really tested
the soul
uh both like in terms of technically and
like spiritually
so that could be just the
romanticization of that place i'm not
sure but so maybe you can speak to it
but did is it correct to say that you
spent some time
at fistia yeah that's right six years i
got my bachelor's and master's and
physics and math there and it's actually
interesting because my
my dad actually both my parents uh went
there and
i think all the stories that i heard
like just like you alex
uh growing up about the place and you
know how interesting and special and
magical it was i think that was a
significant maybe the main reason
uh i wanted to go there uh for college
uh
enough so that i actually went back to
russia
from the us i graduated high school in
the us um you went
back there i went back there yeah that
wow exactly the reaction
most of my peers in college had but you
know
perhaps a little bit stronger that like
you know point me out as this crazy kid
were your parents supportive of that
yeah yeah i came to your previous
question
they uh they supported me and you know
letting me
kind of pursue my passions and the you
know things that
that's a bold move wow what was it like
there it was interesting you know
definitely
fairly hardcore on the fundamentals of
math
and physics and you know lots of good
memories
from you know from those times so okay
so stanford how did you get into
autonomous vehicles
i had the great fortune
and great honor to join stanford's darpa
urban challenge team
in 2006 there this was a third in the
sequence of the
darpa challenges their two grand
challenges
prior to that and then in 2007 they held
the darpa
urban challenge so you know i was doing
my postdoc i had i joined
the team and uh worked
on motion planning uh for you know
that competition so okay so for people
who might not know i know
from from a certain perspective
autonomous vehicles is a funny world
in a certain circle of people everybody
knows everything and then a certain
circle
uh nobody knows anything in terms of
general public
so it's interesting it's it's a good
question what to talk about but
i do think that the urban challenge is
worth revisiting it's a fun little
challenge
one that in first it like sparked so
much so many incredible
minds to focus on one of the hardest
problems of our time
in artificial intelligence so that's
it's a success from a perspective of a
single little challenge
but can you talk about like what did the
challenge involve
so were there pedestrians were there
other cars
what was the goal uh who was on the team
how long did it take any fun fun sort of
specs sure so the way the
the challenge was constructed in just a
little bit of background and as i
mentioned this was the
third uh competition in that series the
first two uh were the grand challenge
called the grand challenge the goal
there was to just
drive in a completely static environment
you know you had to drive in the desert
uh
that was very successful so then darpa
followed
with what they called the urban
challenge where the goal was to have you
know build vehicles that could operate
in more dynamic environments and share
them with other vehicles there were no
pedestrians
there but what darpa did is they took
over an abandoned
air force base and it was kind of like a
little fake city
that they built out there and they had
a bunch of uh robots uh you know cars
that were autonomous uh in there all at
the same time uh mixed in
with other vehicles driven by
professional uh drivers
and each car had a mission and so
there's a crude map that they received
at the beginning and they had a mission
and go you know here and then there and
over here
um and they kind of all were sharing
this environment at the same time they
interact to interact with each other
they had to interact with the human
drivers
there's this very first very rudimentary
um version of uh a self-driving car
that you know could operate on and on
yeah in an environment you know shared
with other dynamic actors
that as you said you're really in many
ways
you know kick started this whole
industry okay so
who was on the team and how did you do i
forget
uh we came in second uh
perhaps that was my contribution to the
team i think the stanford team came in
first in the darpa challenge uh but then
i joined the team and you know
you were the one with the bug in the
code i mean do you have sort of memories
of some particularly challenging things
or
you know one of the cool things it's not
a you know this isn't a product this
isn't the thing that
uh you know it there's you have a little
bit more freedom to experiment so you
can take risks and there's
uh so you can make mistakes uh so is
there interesting mistakes
is there interesting challenges that
stand out to you or some like taught you
um a good technical lesson or a good
philosophical lesson
from that time yeah uh you know
definitely definitely a very
memorable time not really a challenge
but like one of the
most vivid memories that i have from the
time
and i think that was actually one of the
days that
really got me hooked on this whole field
was
the first time i got to run my software
on the car and i was working on a part
of
our planning algorithm uh that had to
navigate in parking lots so it's you
know something that you know called free
space
motion planning so the very first
version of that uh
you know we tried on the car it was on
stanford's campus uh
in the middle of the night and you know
i had this little you know course
constructed with cones
uh in the middle of a parking lot so
we're there like 3 a.m you know by the
time we got the code to
you know you know compile and turn over
and you know it drove like i actually
did something quite reasonable
and you know it was of course very buggy
at the time
and had all kinds of problems but it was
pretty darn magical i remember going
back
and you know later at night trying to
fall asleep and just
being unable to fall asleep for you know
the rest of the night uh
just my mind was blown and yeah that
that's what i've been you know doing
ever since for more than a decade
uh in terms of challenges and uh you
know
interesting memories like on the day of
the competition i it was been pretty
nerve-wracking
i remember standing there with mike
montemerlo who was
the software lead and wrote most of the
code i think i did one little part
of the planner mike you know incredibly
that you know
pretty much the rest of it uh with with
you know a bunch of other incredible
people
but i remember standing on the day of
the competition uh
you know watching the car you know with
mike and your cars are
completely empty right they're all there
lined up in the beginning of the race
and then you know darpa sends them you
know on their mission one by one
so they leave and like you just they
have these sirens
they all had their different silence
silence right each iron had its own
personality if you will so you know off
they go and you don't see them you just
kind of and then every once in a while
they you know come a little bit closer
to where
the audience is and you can kind of hear
you know the sound of your car and you
know it seems to be moving along so that
you know gives you hope
and then you know it goes away and you
can't hear it for too long you start
getting anxious right just a little bit
like you know sending your kids to
college and like you know
kind of you invested in them you hope
you you you you you build it properly
but
like it's still anxiety-inducing uh so
that was
an incredibly uh fun few days
in terms of you know bugs as you
mentioned you know one
that was my bug that caused us the loss
of the first place
is still a debate that you occasionally
have with people on the cmu team scene
you came first
i should mention uh that you haven't
heard of them
but yeah no it's something you know it's
a small school it's it's
really a glitch that you know they
happen to succeed at something robotics
related very scenic though
most people go there for the scenery um
yeah
that's right it's a beautiful campus
unlike stanford so for people yeah
that's true i like stanford for people
who don't know cemu is one of the great
robotics and sort of artificial
intelligence universities in the world
cmu carnegie mellon university okay
sorry go ahead
good good psa so in the part
that i contributed to which was
navigating parking lots
and the way you know that part of the
mission worked is
yeah you in a parking lot you would get
from darpa
an outline of the map you can get this
you know giant polygon
that defined the perimeter of the
parking lot uh and there would be an
entrance and you know so maybe
multiple entries or access to it and
then you would get a goal
within that open space xy
you know heading where the car had to
park it had no information about the
optical selling obstacles that
the car might encounter there so it had
to navigate uh kind of completely free
space
from the entrance to the parking lot
into
that parking space and then uh
once parked there it had to
exit the parking lot while of course
encountering and reasoning about all the
obstacles that it encounters in real
time
so uh
our interpretation or at least my
interpretation of the rules was that you
had to reverse
out of the parking spot and that's what
our cars did even if there's no optical
in front
that's not what seam used car did and it
just kind of
drove right through so there's still a
debate and of course you know if you
stop and then reverse out and go out the
different way that cost you some time
right so there's still
a debate whether you know it was my poor
implementation that cost us extra time
or whether
it was you know cmu violating
an important rule of the competition and
you know i have my own uh opinion here
in terms of other bugs and like i i have
to apologize to mike montemerla
uh for sharing this on air but it is
actually uh one of the more memorable
ones
uh and it's something that's kind of
become a bit of a
a metaphor had a label in the industry
uh since then i think
at least in some circles it's called the
victory circle or victory lap
um and uh our cars
did that so in one of the missions in
the urban challenge in one of the
courses uh there was this big
oval right by the start and finish of
the race so darpa had
a lot of the missions would finish kind
of in that same location
and it was pretty cool because you could
see the cars come by and kind of finish
that part lag of the trip without that
leg of the mission and then you know go
on
and you know finish the rest of it uh
and other vehicles would you know come
hit
their waypoint and you know exit the
oval and
off they would go our car in the hand
which hit the checkpoint
and then it would do an extra lap around
the awful and only then you know
leave and go on its merry way so over
the course of you know the full day it
accumulated
uh some extra time and the problem was
that we had a bug where
it wouldn't you know start reasoning
about the next waypoint and plan around
to get to that next point until it hit
the previous one and in that particular
case by the time you hit the
that that one it was too late for us to
consider the next one and kind of
make a lane change so that every time it
would do like an extra lap so
that's the the stanford victory lap
oh there's there's i feel like there's
something philosophically profound in
there somehow but uh
i mean ultimately everybody is a winner
in that kind of competition
and it led to sort of famously to the
creation of uh
google self-driving car project and now
waymo
so can we uh give an overview of how is
way more born
how's the google self-driving car
project born what's the what is the
mission
what is the hope what is it is the
engineering kind of uh set of milestones
that it seeks to accomplish there's a
lot of questions in there
uh yeah i think you're right it kind of
the
urban challenge and the upper and
previous darpa grand challenges uh kind
of
led i think to a very large you know
degree to that next step and you know
larry and sergey um
uh larry page and sergey brin uh uh
google hunter scores
uh uh saw that competition and believed
in the technology
so now the google self-driving car
project was born
you know at that time and we started in
2009 it was a pretty small
group of us about a dozen people who
came together
uh to to work on on this project at
google
at that time we saw an
you know that incredible early result
in the darpa urban challenge i think
we're all incredibly excited
about where we got to and we believed in
the future of the technology but we
still had a very
rudimentary understanding of the problem
space so the
first goal of this project in 2009 was
to really
better understand what we're up against
and you know with that
goal in mind when we started the project
we created a few milestones for
ourselves
that maximized learnings
well the two milestones were you know uh
one was to drive a hundred thousand
miles
in autonomous mode which was at that
time you know orders of magnitude that
more than anybody has ever done and the
second milestone was to drive
10 routes uh each one was 100 miles long
they were specifically chosen to become
extra spicy you know extra complicated
and sample the full complexity
of the that that domain
um and you had to drive each one
from beginning to end with no
intervention no human intervention so
you get to the beginning of the course
uh you
you press the the button that include
engage in autonomy
and you had to you know go for 100 miles
you know beginning to end
uh with no interventions um and
it sampled again the full complexity of
driving conditions
some were on freeways we had one route
that went all through all the freeways
and all the
bridges in the bay area you know we had
some that went around lake tahoe and
kind of mountainous
roads we had some that drove through
dense urban
um environments like in downtown palo
alto
and through san francisco so it was
incredibly
uh interesting uh to work on
and it uh it took us
just under two years about a year and a
half a little bit more
to finish both of these milestones and
in that process
uh yeah hey it was an incredible amount
of fun probably the most
fun i had in my professional career and
because you're just learning so much you
are you know the goal here is to learn
and prototype you're not yet starting to
build a production system right so you
just
you were you know this is when you're
kind of you know working 24 7 and you're
hacking things together and you also
don't know
how hard this is i mean it's the point
like
so i mean that's an ambitious if i put
myself in that mindset even still
that's a really ambitious set of goals
like just those two
picking picking 10 different
difficult spicy challenges
and then having zero interventions so
like not saying gradually we're going to
like you know over a period of 10 years
we're going to have a bunch of roots and
gradually reduce the number of
interventions
you know would that literally says like
by as soon as possible we want to have
zero
and on hard roads so like to me if i was
facing that
it's unclear that whether that takes two
years or whether that takes 20 years
i mean under two i guess that speaks to
a really big
difference between doing something once
and having a prototype
uh where you are going after you know
learning about the problem
versus how you go about engineering a
product
that you know where you look at uh you
know you
properly do evaluation you look at
metrics you you know drive down and
you're confident that you can do that at
home
and i guess that's the you know why it
took a dozen people
uh you know 16 months or a little bit
more than that
uh back in 2009 and 2010
and with the technology of you know the
more than a decade ago
that amount of time to achieve that
milestone
of 10 routes 100 miles each and no
interventions
and you know it
took us a little bit longer to get to
you know a full
driverless product that customers use
that's another really important moment
is there some
memories of technical lessons or just
one like what did you learn about the
problem of driving from that experience
i mean we could we can now talk about
like what you learned from
modern day waymo but i feel like you may
have learned some profound things
in those early days even more so
because it feels like what waymo is now
is to trying to
you know how to do scale how to make
sure you create a product how to make
sure it's like safety and all those
things which is
all fascinating challenges but like you
were facing the more fundamental
philosophical problem of driving in
those early days like
what the hell is driving as an
autonomous
or maybe i'm again romanticizing it but
is it
is there uh is there some valuable
lessons you picked up over there
at those two years uh a ton
the most important one is probably that
we believe that it's doable and we've
gotten
uh far enough into the problem that uh
you know
we had a i think only a glimpse of the
true complexity uh
of the the domain yeah it's a little bit
like you know climbing a mountain where
you kind of
see the next peak and you think that's
kind of the summit but then you get to
that and you kind of see that that this
is just
the start of the journey uh but we've
tried
we've sampled enough of the problem
space and we've made
enough rapid uh success even you know
with technology
of 2009 2010 that it gave us confidence
to then you know pursue this as
a real product so okay so the next step
you mentioned the the milestones that
you had in the
in those two years what are the next
milestones that then led to the creation
of waymo and beyond
now it was a really interesting journey
and
waymo came a little bit later uh
then you know we completed those
milestones in 2010
that was the pivot when we decided to
focus
on actually building a product yeah
using this technology
uh the initial couple years after that
we were focused
on a freeway you know what you would
call a driver assist
uh maybe an l3 driver assist uh
program then around 2013 we've learned
enough uh about the space and the
thought
more deeply about you know the
product that we wanted to build that we
pivoted uh we pivoted towards
of this vision of you know building a
driver
and deploying it fully driverless
vehicles without a person and that
that's the path that we've been
on since then and uh very it was exactly
the right decision for us
so there was a moment where you also
considered like what is the right
trajectory here
what is the right role of automation in
the in the task of driving there's still
it wasn't from the early days obviously
you want to go fully autonomous
from the early days it was not i think
it was in 20 around 2013 maybe
that we've that became very clear
and we made that pivot and it also
became very clear uh
and that it's even the way you go
building
a driver assist system is you know
fundamentally different from how you go
building a fully driverless vehicle so
you know we've
uh pivoted towards the ladder
and that's what we've been working on
ever since and
so that was around 2013 then
there's sequence of really meaningful
for us really important
defining milestones since then in
the 2015
we had our first
actually the world's first fully
driverless
trade on uh public roads it was
in a custom-built vehicle that we had we
must have seen this we called them the
firefly that you know
funny-looking marshmallow looking thing
um and we
put a passenger uh his name was steve
mann a great
uh friend of our project from the early
days uh the the man happens to be
uh blind so we put him in that vehicle
uh the car had no steering wheel no
pedals it was an uncontrolled
environment
um you know no you know lead or chase
cars no police escorts um
and uh you know we did that trip a few
times in austin texas
so that was a really big milestone well
that was in austin yeah cool
okay um and you know we only but at that
time we're only it took a tremendous
amount of engineering it took a
tremendous amount of validation
uh to get to that point uh but you know
we
only did it a few times i only did that
it was a fixed route
it was not kind of a controlled
environment but it was a fixed route and
we only did a few times
uh then uh in
uh 2016 uh end of 2016 beginning of 2017
is
when we founded waymo uh the company
that's when
we kind of that was the next phase of
the project where
i wanted uh we believed in kind of the
commercial
uh vision of this technology and it made
sense to create an independent entity
you know within that
alphabet umbrella to pursue uh this
product
at scale beyond that in 2017 later in
was another really a huge
step for us really big milestone where
we started it was october
of 2017. where when we
started regular uh driverless
operations on public roads that first
day of operations we drove
uh in one day and that first day 100
miles and
you know driverless fashion and then
we've the most the most important thing
about that milestone was not that you
know 100 miles in one day
but that it was the start of kind of
regular ongoing
driverless operations can we say
driverless it means no driver
that's exactly right so on that first
day we actually had a mix and
up uh in some uh we didn't want to like
you know be on youtube on twitter that
same day so in
uh and many of the rides we had somebody
in the driver's seat but they could not
disengage like the car
it's not disengaged but actually on that
first day
uh some of the miles were driven and
just completely
empty driver's seat and this is the key
distinction that i think people don't
realize
it's you know that oftentimes when you
talk about autonomous vehicles
you're there's often a driver in the
seat that's ready to
uh to take over uh what's called a
safety driver
and then waymo is really one of the
only companies that i'm aware of or at
least as like boldly and
carefully and all and all that is
actually has
cases and now we'll talk about more and
more where there is
literally no driver so that that's
another
the the interesting case of where the
driver is not supposed to disengage
that's like a
nice middle ground if they're still
there but they're not supposed to
disengage
but really there's the case when there's
no
okay there's something magical about
there being nobody in the driver's seat
like just like to me you mentioned um
the first time you wrote some code for
free space navigation of the parking lot
that was like a magical moment
to me just sort of an as an observer of
robots
the first magical moment is
seeing an autonomous vehicle turn like
make a
left turn like apply
sufficient torque to the steering wheel
to where like there's a lot of rotation
and for some reason and there's nobody
in the driver's seat
for some reason that that communicates
that here's a being with power
that makes a decision there's something
about like the steering wheel because
we perhaps romanticize the notion of the
steering wheel it's so essential to the
our conception
our 20th century conception of a car and
it
turning the steering wheel with nobody
in driver's seat
that to me i think maybe to others
it's really powerful like this thing is
in control
and then there's this leap of trust that
you give like i'm gonna put my life
in the hands of this thing that's in
control so in that sense
when there's no but no driver in the
driver's seat
that's a magical moment for robots so i
i'm i gotten a chance to uh last year to
take a ride
in in a waymo vehicle and that that was
the magical moment there's like
nobody in the driver's seat it's it's
like the little details you would think
it doesn't matter whether it's a driver
or not
but like if there's no driver and the
steering wheel is turning on its own
i don't know that's magical it's
absolutely magical
like i you've taken many of these rights
in a completely empty car
no human in the car pulls up you know
you call it on your cell phone it pulls
up you get in
it takes you on its way there's nobody
uh in the car
but you right that's something called
you know fully driverless
our rider only mode of operation
uh yeah it it is
magical it is uh transformative this is
what we hear from our
uh writers it really changes your
experience and not like that that really
is what unlocks
the real potential of this technology uh
but you know
coming back to our journey uh you know
that was 2017 when we started
uh truly driverless operations then in
2018 we've launched our
public commercial service that we call
waymo one
in phoenix in 2019 we started
offering truly driverless rider only
rights
to our early writer population
of users and then you know 2020
has also been a pretty interesting year
uh one of the first ones
less about technology but more about the
maturing and the growth of
waymo as a company we raised our
first round of external financing uh
this year you know we were part of
alphabet so obviously we have
access to you know significant resources
but as kind of on the journey of waymo
maturing as a company it made sense for
us to you know partially go externally
uh uh and in this round so you know we
raised uh
about 3.2 billion dollars uh with from
you know that round
uh we've also you know uh started
putting
our fifth generation of our driver our
hardware
uh uh that is on the new vehicle but
it's also a qualitatively different
set of uh self-driving hardware uh
that's all
uh that is now on the jlr pace so that
was a very
important step for us the hardware specs
fifth generation
i think it'd be fun to maybe i apologize
if i'm interrupting but
maybe talk about maybe the generations
with a focus on what we're talking about
in the fifth generation in terms of
hardware specs
like what's on this car sure so we
separated out the actual car
that we are driving from the
self-driving hardware we put on it
um right now we have so this is as i
mentioned the fifth generation and we've
gone
through we started you know building our
own
hardware you know many many years ago
and
that firefly vehicle also had
the hardware suite that was mostly
designed engineered and built in-house
lidars are of one of the more important
components that we design and build from
the ground up
uh so on the fifth generation uh of
our uh drivers uh of our driving
hardware that we're
switching to right now uh we have
uh as with previous generations in terms
of sensing we have
lidars cameras and radars and when you
have
a pretty beefy computer that processes
all that information and makes you know
decisions
in real time on on board the car uh so
in all of the and it's really a
qualitative uh jump forward in terms of
the capabilities
and uh the various parameters and the
specs of the hardware
compared to what we had before and
compared to what you can kind of get off
the
of the shelf in the market today meaning
from fifth to fourth or from fifth to
first definitely from uh first to fifth
but also from the other
world's dumbest question definitely
definitely from fourth to fifth okay
as well as uh uh there's the
the last step is a big step forward so
everything's in-house
so like lidar's built in house and
and cameras are built in-house uh you
know it's
different you know we work with partners
there are some components uh
that you know we get from our
manufacturing and you know supply chain
partners
uh what exactly is in-house is a bit
different if you
we we do a lot of you know custom uh
design on
all of our sensing materials sliders
radars cameras
you know exactly there's lighters are
almost exclusively in-house and some of
the technologies that we have some of
the fundamental technologies there
are completely unique uh to weima uh
that is also largely true about radars
and cameras it's a little bit more of a
a mix in terms of what we do ourselves
versus what we get from
uh partners is there something uh super
sexy about the computer that you can
mention that's not top secret
like uh for people who enjoy computers
for i mean uh so there's there's a lot
of
machine learning involved but there's a
lot of just basic compute there's
you have to uh probably do a lot of
signal processing on all the different
sensors you have to integrate everything
has to be in real time there's probably
some kind of redundancy type of
situation
is there something interesting you can
say about the computer for the people
who love
hardware it does have all of the
characteristics all the properties that
you just mentioned
uh redundancy uh very beefy compute
for general processing as well as you
know inference
and ml models it is some of the more
sensitive stuff that you know i don't
want to get into for ip reasons but
yeah it can be shared a little bit
uh in terms of the specs of the sensors
that we have on the car you know we
actually shared some videos of
what our lighter seas lighters
see in the world we have 29 cameras we
have
five lighters we have six raiders on
these vehicles
and you can kind of get a feel for the
amount of data that they're producing
that all has to be processed in real
time
uh to you know do perception to do
complex reasoning
and kind of gives you some idea of how
beefy those computers are but i don't
want to get into specifics of exactly
how we build them
okay well let me try some more questions
that you can't get into the specifics of
like gpu wise
is that something you can get into you
know i know that google works
with tpus and so on i mean for machine
learning folks
it's kind of interesting or is there no
how do i ask it uh i've been talking to
people
in the government about ufos and they
don't answer any questions so this is
this is how i feel right now asking
about gpus
[Laughter]
but is there something interesting they
could reveal
or is it just you know uh yeah or would
leave it up to our imagination some of
the some of the compute is there any
i guess is there any fun trickery like i
talked to
chris lattner for a second time and he
was a key
person about tpus and there's a lot of
fun stuff going on in
google in terms of uh
hardware that optimizes for machine
learning is there something
you can reveal in terms of how much you
mentioned customization how much
customization there
is for hardware for machine learning
purposes
i'm going to be like that government you
know you that guy uh personally
audio foes
i i guess i you know will say that it's
really compute
is really important uh we have
very data hungry and compute hungry ml
models
of all over uh our stack and this is
where
you know both being part of alphabet as
well as designing our own sensors and
the entire hardware suite together
where on one hand you get access to like
really rich uh raw sensor
data that you can pipe from your sensors
uh
into your compute platform yeah and
build like build the whole pipe
from sensor raw sensor data to the big
compute as then have the massive compute
to process all that data
and this is where we're finding that
having a lot of control
of that that hardware part of the stack
is
really advantageous one of the
fascinating magical places to me
again might not be able to speak to the
details but
is the it is the other compute which is
like you know this we're just talking
about a single car
but the you know
the driving experience is a source of a
lot of fascinating data
and you have a huge amount of data
coming in on the car on the car
and you know the infrastructure of
storing some of that data
to then train or to analyze or so on
that's a fascinating like piece of it
that that i understand a single car i
don't understand how you pull it all
together in a nice way
is that something that you could speak
to in terms of the challenges of um
of seeing the network of cars and then
bringing the data back
and analyzing things that weren't that
like like edge cases of driving be able
to learn on them to improve the system
to
to see where things going wrong with
where things went right and analyze all
that kind of stuff
is there something interesting there in
the from an engineering perspective
oh there's an incredible uh
amount of really interesting work that's
happening there both
in the you know the real time operation
of the fleet of cars
and the information that they exchange
with each other in real time
to make better decisions as well uh
as on the kind of the off board
component where you have to deal with
massive amounts of data
for training your ml models evaluating
the male models
for simulating the entire system and for
you know evaluating your entire system
and this is where
and being part of alphabet has been once
again been tremendously
uh advantageous because we consume an
incredible amount of you know compute
for ml infrastructure we build a lot of
custom frameworks to you know get good
at you know
on data mining uh finding the
interesting edge cases for training and
for evaluation of the system
for both training and evaluating some
components and you know sub
uh parts of the system and various ml
models as well as the
uh evaluating the entire system and
simulation okay that first piece that
you mentioned that
cars communicating to each other
essentially i mean through perhaps
through a centralized point
but what uh that's fascinating too how
much does that help you like if you
imagine
like you know right now the number of
way more vehicles is
whatever x i don't know if you can talk
to what that number
but it's it's not in the hundreds of
millions yet
and imagine if the whole world is way
more vehicles
uh like that changes potentially the
power of connectivity like the more cars
you have
i guess actually if you look at phoenix
because there's enough vehicles
there's enough when there's like some
level of density
you can start to probably do some really
interesting stuff with
the fact that cars can negotiate can
be uh can communicate with each other
and thereby make decisions
is there something interesting there
that you can talk to about like how does
that help with the driving problem
from as compared to just a single car
solving the driving problem by itself
uh yeah it's it's a spectrum i uh first
to say that yeah
it's it helps uh and it helps in various
ways but it's not required
uh right now the way we build our system
engaged cars can operate independently
they can operate with no connectivity uh
so i think it is important that you know
you have
a fully uh autonomous you know fully
capable
uh driver uh that
computerized driver that each car has
then you know they do
share information and they share
information in real time it really
really helps right so the way we
do this today is uh you know whenever
one car
encounters something interesting in the
world whether it might be an accident or
a new construction zone that information
immediately gets
uh you know uploaded over the air and is
propagated to the rest of the fleet
so and that's kind of how we think about
maps as priors
in terms of the knowledge of our drivers
of our fleet of drivers that is
distributed across the fleet and it's
updated
in real time so that's one use case
you know you can imagine as the you know
the the density
of these vehicles go up that they can
exchange more information
in terms of what they're planning to do
uh and uh start
uh influencing how they interact with
each other uh as well as you know
potentially
sharing some observations right to help
with if you have enough density of these
vehicles where you know one car might be
seeing something that another is
relevant to another car
that is very dynamic you know it's not
part of kind of you're updating your
static prior
of the map of the world but it's more of
a dynamic information that could be
relevant to the
decisions that another cars make in real
time so you can see them exchanging that
information
and you can build on that but again i i
see that as
an advantage but it's you know not a
requirement
so what about the human in the loop so
uh when i got a chance to drive with a
ride in a waymo you know there's
customer service
[Laughter]
so like is somebody that's able to
dynamically
like tune in and uh help you out
what uh what role does the human play in
that picture that's a fascinating like
you know the idea of teleoperation be
able to remotely control a vehicle
so here what we're talking about is like
like frictionless uh like a human being
able to in a
in a frictionless way sort of help you
out i don't know if they're able to
actually control the vehicle
is that something you could talk to uh
yes okay uh to be clear
we don't do teleportation i'm going to
believe in teleoperation
for rare reasons that's not what we have
on our cars
we do as you mentioned have you know
version of you know customer support
uh you know we call it live health in
fact we find it that it's very
uh important for our rider experience
especially
if it's your first trip you've never
been in a fully driverless rider only
way more vehicle you get in there's
nobody there
right so you can imagine having all
kinds of you know questions in your head
like how this thing works
so we've put a lot of thought into kind
of guiding our
our writers our customers through that
experience especially for the first time
they get some information on the phone
uh if the fully driverless vehicle
is used to service their trip uh when
you get into the car
we have an in-car you know screen and
audio that kind of guides them and
explains
uh what to expect they also have a
button
that they can push that will connect
them to
you know a real life human being that
they can talk to
all right about this whole process so
that's one aspect of it um there is
i should mention that there is uh
another function that uh humans provide
uh to our
cars but it's not tele operation you can
think of it a little bit more like you
know fleet assistance
kind of like you know traffic control uh
that that you have
where our cars again they're responsible
on their own for making all of the
decisions all the driving decisions that
don't require connectivity
they you know anything that is safety or
latency critical
uh is done you know purely autonomously
by
on board uh our on onboard system uh but
there are situations where you know if
connectivity is available
uh can a car encounters a particularly
challenging situation you can imagine
like a super hairy
uh scene of an accident uh the cars will
do their best
they will recognize that it's an off
nominal situation they
will you know do their best to come up
you know with the right interpretation
the best course of action in that
scenario but if connectivity is
available they can ask
for confirmation from you know here mode
human
assistant to kind of confirm those
actions and perhaps
provide a little bit of kind of
contextual information and guidance
so october 8th was when you're talking
about the
was weimar launched the the
the fully self the public version
of its fully driverless that's right
term i think
service in phoenix is that october 8th
that's right
it was the introduction of fully
driverless rider only vehicles into our
you know public waymo one service okay
so that's that's amazing
so it's like anybody can get into waymo
in phoenix
oh that's right yeah so we previously
had
early people in our early writer program
uh
taking fully driverless rides in phoenix
and
uh just uh this a little while ago we
opened on october 8th we opened
that mode of operation to the public so
i can you know download the app and you
know go on the right
there is uh a lot more demand right now
uh for that servi