Kind: captions
Language: en
the following is a conversation with
dimitri dalgov
the cto of waymo which is an autonomous
driving company that started
as google's self-driving car project in
2009 and became waymo in 2016.
dimitri was there all along waymo is
currently leading in the fully
autonomous vehicle space
in that they actually have an at-scale
deployment of
publicly accessible autonomous vehicles
driving passengers around
with no safety driver with nobody
in the driver's seat this to me is an
incredible
accomplishment of engineering on one of
the most difficult
and exciting artificial intelligence
challenges of the 21st century
quick mention of a sponsor followed by
some thoughts related to the episode
thank you to trial labs a company that
helps businesses
apply machine learning to solve real
world problems
blinkist an app i use for reading
through summaries of books
better help online therapy with a
licensed professional
and cash app the app i use to send money
to friends
please check out the sponsors in the
description to get a discount and to
support this podcast
as a side note let me say that
autonomous and semi-autonomous driving
was the focus of my work at mit and it's
a problem space that i find
fascinating and full of open questions
from both a robotics
and a human psychology perspective
there's quite a bit
that i could say here about my
experiences in academia on this topic
that revealed to me let's say the
less admirable size of human beings but
i choose to focus on the positive on
solutions
i'm brilliant engineers like dimitri and
the team at waymo
who work tirelessly to innovate and to
build amazing technology
that will define our future because of
dimitri
and others like him i'm excited for this
future
and who knows perhaps i too will help
contribute something of value
to it if you enjoy this thing subscribe
on youtube review it with five stars and
up a podcast
follow on spotify support on patreon or
connect with me on twitter
at lex friedman and now here's my
conversation
with dmitry dolgov when did you first
fall in love with
robotics or even computer science more
in general computer science first
at a fairly young age and robotics
happened much later
um i i think my first
interesting introduction to computers
was
in the late 80s uh
when we got our first computer i think
was an
uh an ibm i think ibm it remember those
things that had like a turbo button in
the front you'd press it and you know
make the thing go faster did they
already have floppy disks
yeah yeah yeah like the the five point
four inch ones i think there's a bigger
inch so good when something then five
inches and three inches
yeah i think that was the five i don't i
maybe that was before that was the giant
plates and i didn't get that
uh but it was definitely not the not the
three inch ones
uh anyway so that that you know we got
that uh
computer i spent the first a few
months just you know playing video games
uh as you would expect
i got bored of that so i started messing
around
and trying to figure out how to make the
thing
do other stuff got into
exploring know programming and a couple
of years
later i got to a point where um i
actually wrote a game
a lot of games and a game developer a
japanese game developer
actually offered to buy it for me for
you know a few hundred bucks but you
know for for a kid
yeah in russia that's a big deal that's
a big deal yeah uh i did not take the
deal
wow integrity yeah i i instead uh
stupidity yes that was not the most
acute financial move that i made in my
life you know looking back at it now
uh i instead put it well you know i had
a reason i i put it online
uh it was what did you call it back in
the days it was a freeware i think
right it was not open source but you
could upload the binaries you put the
game online and the idea was that you
know people like it and then they you
know
contribute and they send you little
donations right so i did my quick math
of like you know my of course you know
thousands and millions of people are
going to play my game send me a couple
of bucks a piece you know
should definitely do that as i said not
not the best
remember what language it was what
programming it was about
which what pascal pascal and they had a
graphical component
so that text based yeah yeah it was uh
like
i think 320 by 200 whatever it was
i think that kind of the earlier that's
the cga resolution right
and i actually think the reason why this
company wanted to buy it is not like the
fancy graphics or the implementation
it was maybe the idea uh of my actual
game
the idea of the game okay well one of
the things
it's so funny i used to play this game
called golden axe
and the simplicity of the graphics
and something about the simplicity of
the music like
it still haunts me i don't know if
that's a childhood thing i don't know if
that's the same thing for call of duty
these days for young kids
but i still think that the
simple one the games are simple that
simple purity makes for
like allows your imagination to take
over and thereby creating a more magical
experience
like now with better and better graphics
it feels like
your imagination doesn't get to uh
create worlds
which is kind of interesting um it could
be just an
old man on a porch like waving at kids
these days that have no respect but
i still think that graphics almost get
in the way of the experience
i don't know flippy bird yeah
i don't know if the imagination gets
closed i don't
yeah but that that's more about games
that up like that's more like tetris
world where they
optimally masterfully
like create a fun
short-term dopamine experience versus
i'm more referring to like
role-playing games where there's like a
story you can live in it for
months or years um like uh
there's an elder scroll series which is
probably my favorite set of games
that was a magical experience and then
the graphics were terrible
the characters were all randomly
generated but they're i don't know
that's
it pulls you in there's a story it's
like an
interactive version of an elder scrolls
tolkien world and you get to live in it
i don't know
i miss it it's one of the things that
suck about being an
adult is there's no you have to live in
the real world as opposed to
the elder scrolls world you know
whatever brings you joy right
minecraft right minecraft is a great
example you create like it's not the
fancy graphics but it's the
creation of your own worlds yeah that
one is crazy
you know one of the pitches for being a
parent that people tell me
is that you can like use the excuse of
parenting to
to go back into the video game world and
like
like that's like you know father-son
father-daughter time
but really you just get to play video
games with your kids so anyway
at that time did you have any ridiculous
ambitious dreams of
where as a creator you might go as an
engineer did you
what did you think of yourself as as an
engineer as a tinkerer or did you want
to be like an astronaut
or something like that you know i'm
tempted to make something up about you
know robots
uh engineering or you know mysteries of
the universe but
that's not the actual memory that pops
into my mind uh when you when you ask me
about childhood dreams so i'll actually
share the real thing
uh when i was
maybe four or five years old i you know
as well do i
thought about you know what i wanted to
do when i grow up and i had this dream
of being
a traffic control cop you know they
don't have those today's i think but you
know
back in the 80s and you know in russia i
you probably are familiar with that legs
they had these
uh you know police officers they would
stand in the middle of an intersection
all day
and they would have their like striped
black and white batons that they would
use to you know control the flow of
traffic
and you know for whatever reason i was
strangely infatuated with this whole
process and like that that was my dream
uh that's what i wanted to do when i
grew up and you know my
parents uh both physics profs by the way
i think we're
you know a little concerned uh with that
level of ambition coming from their
child
yeah uh you know that age well that it's
an interesting i don't know if
you can relate but i very much love that
idea
i have a ocd nature that i think lends
itself
very close to the engineering mindset
which is you want to kind of
optimize you know solve a problem by
create creating an automated
solution like like a set of rules the
set of rules you could follow
and then thereby make it ultra efficient
i don't know if that's
it was it of that nature i certainly
have that there's like fact
like simcity and factory building games
all those kinds of things
kind of speak to that engineering
mindset or did you just like the uniform
i think it was more of the latter i
think it was the uniform and
you know the the striped baton that made
cars go
in the right direction but i guess you
know it is
i did end up uh i guess uh you know
working on the transportation industry
one way or another no uniform though but
that's right
maybe it was my you know deep inner
infatuation with
the you know traffic control batons that
led to this
career okay what uh when did you
when was the leap from programming to
robotics that happened later that was
after grad school
uh after and actually the most
self-driving cars was i think my first
real hands-on introduction to robotics
but i i never
really had that much hands-on experience
in school and training i you know worked
on applied math
and physics then in college i did more
of abstract computer science
and it was after grad school that i
really got
involved in robotics which was actually
self-driving cars and
you know that was a big big flip what uh
well grad school
so i went to grad school in michigan and
then i did a postdoc at stanford uh
which is
that was the postdoc where i got to play
with celebrating cars
yeah so we'll return there let's go back
to uh to moscow so i
you know for episode 100 i talked to my
dad and also i grew up with my dad
i guess uh
so i had to put up with him for many
years and uh
he he went to the fistiach
or mipt it's weird to say in english
because i've heard all this in russian
moscow institute of physics and
technology and to me that was like
i met some super interesting as a child
i met some super interesting characters
it felt to me like the greatest
university in the world
the most elite university in the world
and just the the people
that i met that came out of there were
like
not only brilliant but also special
humans
it seems like that place really tested
the soul
uh both like in terms of technically and
like spiritually
so that could be just the
romanticization of that place i'm not
sure but so maybe you can speak to it
but did is it correct to say that you
spent some time
at fistia yeah that's right six years i
got my bachelor's and master's and
physics and math there and it's actually
interesting because my
my dad actually both my parents uh went
there and
i think all the stories that i heard
like just like you alex
uh growing up about the place and you
know how interesting and special and
magical it was i think that was a
significant maybe the main reason
uh i wanted to go there uh for college
uh
enough so that i actually went back to
russia
from the us i graduated high school in
the us um you went
back there i went back there yeah that
wow exactly the reaction
most of my peers in college had but you
know
perhaps a little bit stronger that like
you know point me out as this crazy kid
were your parents supportive of that
yeah yeah i came to your previous
question
they uh they supported me and you know
letting me
kind of pursue my passions and the you
know things that
that's a bold move wow what was it like
there it was interesting you know
definitely
fairly hardcore on the fundamentals of
math
and physics and you know lots of good
memories
from you know from those times so okay
so stanford how did you get into
autonomous vehicles
i had the great fortune
and great honor to join stanford's darpa
urban challenge team
in 2006 there this was a third in the
sequence of the
darpa challenges their two grand
challenges
prior to that and then in 2007 they held
the darpa
urban challenge so you know i was doing
my postdoc i had i joined
the team and uh worked
on motion planning uh for you know
that competition so okay so for people
who might not know i know
from from a certain perspective
autonomous vehicles is a funny world
in a certain circle of people everybody
knows everything and then a certain
circle
uh nobody knows anything in terms of
general public
so it's interesting it's it's a good
question what to talk about but
i do think that the urban challenge is
worth revisiting it's a fun little
challenge
one that in first it like sparked so
much so many incredible
minds to focus on one of the hardest
problems of our time
in artificial intelligence so that's
it's a success from a perspective of a
single little challenge
but can you talk about like what did the
challenge involve
so were there pedestrians were there
other cars
what was the goal uh who was on the team
how long did it take any fun fun sort of
specs sure so the way the
the challenge was constructed in just a
little bit of background and as i
mentioned this was the
third uh competition in that series the
first two uh were the grand challenge
called the grand challenge the goal
there was to just
drive in a completely static environment
you know you had to drive in the desert
uh
that was very successful so then darpa
followed
with what they called the urban
challenge where the goal was to have you
know build vehicles that could operate
in more dynamic environments and share
them with other vehicles there were no
pedestrians
there but what darpa did is they took
over an abandoned
air force base and it was kind of like a
little fake city
that they built out there and they had
a bunch of uh robots uh you know cars
that were autonomous uh in there all at
the same time uh mixed in
with other vehicles driven by
professional uh drivers
and each car had a mission and so
there's a crude map that they received
at the beginning and they had a mission
and go you know here and then there and
over here
um and they kind of all were sharing
this environment at the same time they
interact to interact with each other
they had to interact with the human
drivers
there's this very first very rudimentary
um version of uh a self-driving car
that you know could operate on and on
yeah in an environment you know shared
with other dynamic actors
that as you said you're really in many
ways
you know kick started this whole
industry okay so
who was on the team and how did you do i
forget
uh we came in second uh
perhaps that was my contribution to the
team i think the stanford team came in
first in the darpa challenge uh but then
i joined the team and you know
you were the one with the bug in the
code i mean do you have sort of memories
of some particularly challenging things
or
you know one of the cool things it's not
a you know this isn't a product this
isn't the thing that
uh you know it there's you have a little
bit more freedom to experiment so you
can take risks and there's
uh so you can make mistakes uh so is
there interesting mistakes
is there interesting challenges that
stand out to you or some like taught you
um a good technical lesson or a good
philosophical lesson
from that time yeah uh you know
definitely definitely a very
memorable time not really a challenge
but like one of the
most vivid memories that i have from the
time
and i think that was actually one of the
days that
really got me hooked on this whole field
was
the first time i got to run my software
on the car and i was working on a part
of
our planning algorithm uh that had to
navigate in parking lots so it's you
know something that you know called free
space
motion planning so the very first
version of that uh
you know we tried on the car it was on
stanford's campus uh
in the middle of the night and you know
i had this little you know course
constructed with cones
uh in the middle of a parking lot so
we're there like 3 a.m you know by the
time we got the code to
you know you know compile and turn over
and you know it drove like i actually
did something quite reasonable
and you know it was of course very buggy
at the time
and had all kinds of problems but it was
pretty darn magical i remember going
back
and you know later at night trying to
fall asleep and just
being unable to fall asleep for you know
the rest of the night uh
just my mind was blown and yeah that
that's what i've been you know doing
ever since for more than a decade
uh in terms of challenges and uh you
know
interesting memories like on the day of
the competition i it was been pretty
nerve-wracking
i remember standing there with mike
montemerlo who was
the software lead and wrote most of the
code i think i did one little part
of the planner mike you know incredibly
that you know
pretty much the rest of it uh with with
you know a bunch of other incredible
people
but i remember standing on the day of
the competition uh
you know watching the car you know with
mike and your cars are
completely empty right they're all there
lined up in the beginning of the race
and then you know darpa sends them you
know on their mission one by one
so they leave and like you just they
have these sirens
they all had their different silence
silence right each iron had its own
personality if you will so you know off
they go and you don't see them you just
kind of and then every once in a while
they you know come a little bit closer
to where
the audience is and you can kind of hear
you know the sound of your car and you
know it seems to be moving along so that
you know gives you hope
and then you know it goes away and you
can't hear it for too long you start
getting anxious right just a little bit
like you know sending your kids to
college and like you know
kind of you invested in them you hope
you you you you you build it properly
but
like it's still anxiety-inducing uh so
that was
an incredibly uh fun few days
in terms of you know bugs as you
mentioned you know one
that was my bug that caused us the loss
of the first place
is still a debate that you occasionally
have with people on the cmu team scene
you came first
i should mention uh that you haven't
heard of them
but yeah no it's something you know it's
a small school it's it's
really a glitch that you know they
happen to succeed at something robotics
related very scenic though
most people go there for the scenery um
yeah
that's right it's a beautiful campus
unlike stanford so for people yeah
that's true i like stanford for people
who don't know cemu is one of the great
robotics and sort of artificial
intelligence universities in the world
cmu carnegie mellon university okay
sorry go ahead
good good psa so in the part
that i contributed to which was
navigating parking lots
and the way you know that part of the
mission worked is
yeah you in a parking lot you would get
from darpa
an outline of the map you can get this
you know giant polygon
that defined the perimeter of the
parking lot uh and there would be an
entrance and you know so maybe
multiple entries or access to it and
then you would get a goal
within that open space xy
you know heading where the car had to
park it had no information about the
optical selling obstacles that
the car might encounter there so it had
to navigate uh kind of completely free
space
from the entrance to the parking lot
into
that parking space and then uh
once parked there it had to
exit the parking lot while of course
encountering and reasoning about all the
obstacles that it encounters in real
time
so uh
our interpretation or at least my
interpretation of the rules was that you
had to reverse
out of the parking spot and that's what
our cars did even if there's no optical
in front
that's not what seam used car did and it
just kind of
drove right through so there's still a
debate and of course you know if you
stop and then reverse out and go out the
different way that cost you some time
right so there's still
a debate whether you know it was my poor
implementation that cost us extra time
or whether
it was you know cmu violating
an important rule of the competition and
you know i have my own uh opinion here
in terms of other bugs and like i i have
to apologize to mike montemerla
uh for sharing this on air but it is
actually uh one of the more memorable
ones
uh and it's something that's kind of
become a bit of a
a metaphor had a label in the industry
uh since then i think
at least in some circles it's called the
victory circle or victory lap
um and uh our cars
did that so in one of the missions in
the urban challenge in one of the
courses uh there was this big
oval right by the start and finish of
the race so darpa had
a lot of the missions would finish kind
of in that same location
and it was pretty cool because you could
see the cars come by and kind of finish
that part lag of the trip without that
leg of the mission and then you know go
on
and you know finish the rest of it uh
and other vehicles would you know come
hit
their waypoint and you know exit the
oval and
off they would go our car in the hand
which hit the checkpoint
and then it would do an extra lap around
the awful and only then you know
leave and go on its merry way so over
the course of you know the full day it
accumulated
uh some extra time and the problem was
that we had a bug where
it wouldn't you know start reasoning
about the next waypoint and plan around
to get to that next point until it hit
the previous one and in that particular
case by the time you hit the
that that one it was too late for us to
consider the next one and kind of
make a lane change so that every time it
would do like an extra lap so
that's the the stanford victory lap
oh there's there's i feel like there's
something philosophically profound in
there somehow but uh
i mean ultimately everybody is a winner
in that kind of competition
and it led to sort of famously to the
creation of uh
google self-driving car project and now
waymo
so can we uh give an overview of how is
way more born
how's the google self-driving car
project born what's the what is the
mission
what is the hope what is it is the
engineering kind of uh set of milestones
that it seeks to accomplish there's a
lot of questions in there
uh yeah i think you're right it kind of
the
urban challenge and the upper and
previous darpa grand challenges uh kind
of
led i think to a very large you know
degree to that next step and you know
larry and sergey um
uh larry page and sergey brin uh uh
google hunter scores
uh uh saw that competition and believed
in the technology
so now the google self-driving car
project was born
you know at that time and we started in
2009 it was a pretty small
group of us about a dozen people who
came together
uh to to work on on this project at
google
at that time we saw an
you know that incredible early result
in the darpa urban challenge i think
we're all incredibly excited
about where we got to and we believed in
the future of the technology but we
still had a very
rudimentary understanding of the problem
space so the
first goal of this project in 2009 was
to really
better understand what we're up against
and you know with that
goal in mind when we started the project
we created a few milestones for
ourselves
that maximized learnings
well the two milestones were you know uh
one was to drive a hundred thousand
miles
in autonomous mode which was at that
time you know orders of magnitude that
more than anybody has ever done and the
second milestone was to drive
10 routes uh each one was 100 miles long
they were specifically chosen to become
extra spicy you know extra complicated
and sample the full complexity
of the that that domain
um and you had to drive each one
from beginning to end with no
intervention no human intervention so
you get to the beginning of the course
uh you
you press the the button that include
engage in autonomy
and you had to you know go for 100 miles
you know beginning to end
uh with no interventions um and
it sampled again the full complexity of
driving conditions
some were on freeways we had one route
that went all through all the freeways
and all the
bridges in the bay area you know we had
some that went around lake tahoe and
kind of mountainous
roads we had some that drove through
dense urban
um environments like in downtown palo
alto
and through san francisco so it was
incredibly
uh interesting uh to work on
and it uh it took us
just under two years about a year and a
half a little bit more
to finish both of these milestones and
in that process
uh yeah hey it was an incredible amount
of fun probably the most
fun i had in my professional career and
because you're just learning so much you
are you know the goal here is to learn
and prototype you're not yet starting to
build a production system right so you
just
you were you know this is when you're
kind of you know working 24 7 and you're
hacking things together and you also
don't know
how hard this is i mean it's the point
like
so i mean that's an ambitious if i put
myself in that mindset even still
that's a really ambitious set of goals
like just those two
picking picking 10 different
difficult spicy challenges
and then having zero interventions so
like not saying gradually we're going to
like you know over a period of 10 years
we're going to have a bunch of roots and
gradually reduce the number of
interventions
you know would that literally says like
by as soon as possible we want to have
zero
and on hard roads so like to me if i was
facing that
it's unclear that whether that takes two
years or whether that takes 20 years
i mean under two i guess that speaks to
a really big
difference between doing something once
and having a prototype
uh where you are going after you know
learning about the problem
versus how you go about engineering a
product
that you know where you look at uh you
know you
properly do evaluation you look at
metrics you you know drive down and
you're confident that you can do that at
home
and i guess that's the you know why it
took a dozen people
uh you know 16 months or a little bit
more than that
uh back in 2009 and 2010
and with the technology of you know the
more than a decade ago
that amount of time to achieve that
milestone
of 10 routes 100 miles each and no
interventions
and you know it
took us a little bit longer to get to
you know a full
driverless product that customers use
that's another really important moment
is there some
memories of technical lessons or just
one like what did you learn about the
problem of driving from that experience
i mean we could we can now talk about
like what you learned from
modern day waymo but i feel like you may
have learned some profound things
in those early days even more so
because it feels like what waymo is now
is to trying to
you know how to do scale how to make
sure you create a product how to make
sure it's like safety and all those
things which is
all fascinating challenges but like you
were facing the more fundamental
philosophical problem of driving in
those early days like
what the hell is driving as an
autonomous
or maybe i'm again romanticizing it but
is it
is there uh is there some valuable
lessons you picked up over there
at those two years uh a ton
the most important one is probably that
we believe that it's doable and we've
gotten
uh far enough into the problem that uh
you know
we had a i think only a glimpse of the
true complexity uh
of the the domain yeah it's a little bit
like you know climbing a mountain where
you kind of
see the next peak and you think that's
kind of the summit but then you get to
that and you kind of see that that this
is just
the start of the journey uh but we've
tried
we've sampled enough of the problem
space and we've made
enough rapid uh success even you know
with technology
of 2009 2010 that it gave us confidence
to then you know pursue this as
a real product so okay so the next step
you mentioned the the milestones that
you had in the
in those two years what are the next
milestones that then led to the creation
of waymo and beyond
now it was a really interesting journey
and
waymo came a little bit later uh
then you know we completed those
milestones in 2010
that was the pivot when we decided to
focus
on actually building a product yeah
using this technology
uh the initial couple years after that
we were focused
on a freeway you know what you would
call a driver assist
uh maybe an l3 driver assist uh
program then around 2013 we've learned
enough uh about the space and the
thought
more deeply about you know the
product that we wanted to build that we
pivoted uh we pivoted towards
of this vision of you know building a
driver
and deploying it fully driverless
vehicles without a person and that
that's the path that we've been
on since then and uh very it was exactly
the right decision for us
so there was a moment where you also
considered like what is the right
trajectory here
what is the right role of automation in
the in the task of driving there's still
it wasn't from the early days obviously
you want to go fully autonomous
from the early days it was not i think
it was in 20 around 2013 maybe
that we've that became very clear
and we made that pivot and it also
became very clear uh
and that it's even the way you go
building
a driver assist system is you know
fundamentally different from how you go
building a fully driverless vehicle so
you know we've
uh pivoted towards the ladder
and that's what we've been working on
ever since and
so that was around 2013 then
there's sequence of really meaningful
for us really important
defining milestones since then in
the 2015
we had our first
actually the world's first fully
driverless
trade on uh public roads it was
in a custom-built vehicle that we had we
must have seen this we called them the
firefly that you know
funny-looking marshmallow looking thing
um and we
put a passenger uh his name was steve
mann a great
uh friend of our project from the early
days uh the the man happens to be
uh blind so we put him in that vehicle
uh the car had no steering wheel no
pedals it was an uncontrolled
environment
um you know no you know lead or chase
cars no police escorts um
and uh you know we did that trip a few
times in austin texas
so that was a really big milestone well
that was in austin yeah cool
okay um and you know we only but at that
time we're only it took a tremendous
amount of engineering it took a
tremendous amount of validation
uh to get to that point uh but you know
we
only did it a few times i only did that
it was a fixed route
it was not kind of a controlled
environment but it was a fixed route and
we only did a few times
uh then uh in
uh 2016 uh end of 2016 beginning of 2017
is
when we founded waymo uh the company
that's when
we kind of that was the next phase of
the project where
i wanted uh we believed in kind of the
commercial
uh vision of this technology and it made
sense to create an independent entity
you know within that
alphabet umbrella to pursue uh this
product
at scale beyond that in 2017 later in
was another really a huge
step for us really big milestone where
we started it was october
of 2017. where when we
started regular uh driverless
operations on public roads that first
day of operations we drove
uh in one day and that first day 100
miles and
you know driverless fashion and then
we've the most the most important thing
about that milestone was not that you
know 100 miles in one day
but that it was the start of kind of
regular ongoing
driverless operations can we say
driverless it means no driver
that's exactly right so on that first
day we actually had a mix and
up uh in some uh we didn't want to like
you know be on youtube on twitter that
same day so in
uh and many of the rides we had somebody
in the driver's seat but they could not
disengage like the car
it's not disengaged but actually on that
first day
uh some of the miles were driven and
just completely
empty driver's seat and this is the key
distinction that i think people don't
realize
it's you know that oftentimes when you
talk about autonomous vehicles
you're there's often a driver in the
seat that's ready to
uh to take over uh what's called a
safety driver
and then waymo is really one of the
only companies that i'm aware of or at
least as like boldly and
carefully and all and all that is
actually has
cases and now we'll talk about more and
more where there is
literally no driver so that that's
another
the the interesting case of where the
driver is not supposed to disengage
that's like a
nice middle ground if they're still
there but they're not supposed to
disengage
but really there's the case when there's
no
okay there's something magical about
there being nobody in the driver's seat
like just like to me you mentioned um
the first time you wrote some code for
free space navigation of the parking lot
that was like a magical moment
to me just sort of an as an observer of
robots
the first magical moment is
seeing an autonomous vehicle turn like
make a
left turn like apply
sufficient torque to the steering wheel
to where like there's a lot of rotation
and for some reason and there's nobody
in the driver's seat
for some reason that that communicates
that here's a being with power
that makes a decision there's something
about like the steering wheel because
we perhaps romanticize the notion of the
steering wheel it's so essential to the
our conception
our 20th century conception of a car and
it
turning the steering wheel with nobody
in driver's seat
that to me i think maybe to others
it's really powerful like this thing is
in control
and then there's this leap of trust that
you give like i'm gonna put my life
in the hands of this thing that's in
control so in that sense
when there's no but no driver in the
driver's seat
that's a magical moment for robots so i
i'm i gotten a chance to uh last year to
take a ride
in in a waymo vehicle and that that was
the magical moment there's like
nobody in the driver's seat it's it's
like the little details you would think
it doesn't matter whether it's a driver
or not
but like if there's no driver and the
steering wheel is turning on its own
i don't know that's magical it's
absolutely magical
like i you've taken many of these rights
in a completely empty car
no human in the car pulls up you know
you call it on your cell phone it pulls
up you get in
it takes you on its way there's nobody
uh in the car
but you right that's something called
you know fully driverless
our rider only mode of operation
uh yeah it it is
magical it is uh transformative this is
what we hear from our
uh writers it really changes your
experience and not like that that really
is what unlocks
the real potential of this technology uh
but you know
coming back to our journey uh you know
that was 2017 when we started
uh truly driverless operations then in
2018 we've launched our
public commercial service that we call
waymo one
in phoenix in 2019 we started
offering truly driverless rider only
rights
to our early writer population
of users and then you know 2020
has also been a pretty interesting year
uh one of the first ones
less about technology but more about the
maturing and the growth of
waymo as a company we raised our
first round of external financing uh
this year you know we were part of
alphabet so obviously we have
access to you know significant resources
but as kind of on the journey of waymo
maturing as a company it made sense for
us to you know partially go externally
uh uh and in this round so you know we
raised uh
about 3.2 billion dollars uh with from
you know that round
uh we've also you know uh started
putting
our fifth generation of our driver our
hardware
uh uh that is on the new vehicle but
it's also a qualitatively different
set of uh self-driving hardware uh
that's all
uh that is now on the jlr pace so that
was a very
important step for us the hardware specs
fifth generation
i think it'd be fun to maybe i apologize
if i'm interrupting but
maybe talk about maybe the generations
with a focus on what we're talking about
in the fifth generation in terms of
hardware specs
like what's on this car sure so we
separated out the actual car
that we are driving from the
self-driving hardware we put on it
um right now we have so this is as i
mentioned the fifth generation and we've
gone
through we started you know building our
own
hardware you know many many years ago
and
that firefly vehicle also had
the hardware suite that was mostly
designed engineered and built in-house
lidars are of one of the more important
components that we design and build from
the ground up
uh so on the fifth generation uh of
our uh drivers uh of our driving
hardware that we're
switching to right now uh we have
uh as with previous generations in terms
of sensing we have
lidars cameras and radars and when you
have
a pretty beefy computer that processes
all that information and makes you know
decisions
in real time on on board the car uh so
in all of the and it's really a
qualitative uh jump forward in terms of
the capabilities
and uh the various parameters and the
specs of the hardware
compared to what we had before and
compared to what you can kind of get off
the
of the shelf in the market today meaning
from fifth to fourth or from fifth to
first definitely from uh first to fifth
but also from the other
world's dumbest question definitely
definitely from fourth to fifth okay
as well as uh uh there's the
the last step is a big step forward so
everything's in-house
so like lidar's built in house and
and cameras are built in-house uh you
know it's
different you know we work with partners
there are some components uh
that you know we get from our
manufacturing and you know supply chain
partners
uh what exactly is in-house is a bit
different if you
we we do a lot of you know custom uh
design on
all of our sensing materials sliders
radars cameras
you know exactly there's lighters are
almost exclusively in-house and some of
the technologies that we have some of
the fundamental technologies there
are completely unique uh to weima uh
that is also largely true about radars
and cameras it's a little bit more of a
a mix in terms of what we do ourselves
versus what we get from
uh partners is there something uh super
sexy about the computer that you can
mention that's not top secret
like uh for people who enjoy computers
for i mean uh so there's there's a lot
of
machine learning involved but there's a
lot of just basic compute there's
you have to uh probably do a lot of
signal processing on all the different
sensors you have to integrate everything
has to be in real time there's probably
some kind of redundancy type of
situation
is there something interesting you can
say about the computer for the people
who love
hardware it does have all of the
characteristics all the properties that
you just mentioned
uh redundancy uh very beefy compute
for general processing as well as you
know inference
and ml models it is some of the more
sensitive stuff that you know i don't
want to get into for ip reasons but
yeah it can be shared a little bit
uh in terms of the specs of the sensors
that we have on the car you know we
actually shared some videos of
what our lighter seas lighters
see in the world we have 29 cameras we
have
five lighters we have six raiders on
these vehicles
and you can kind of get a feel for the
amount of data that they're producing
that all has to be processed in real
time
uh to you know do perception to do
complex reasoning
and kind of gives you some idea of how
beefy those computers are but i don't
want to get into specifics of exactly
how we build them
okay well let me try some more questions
that you can't get into the specifics of
like gpu wise
is that something you can get into you
know i know that google works
with tpus and so on i mean for machine
learning folks
it's kind of interesting or is there no
how do i ask it uh i've been talking to
people
in the government about ufos and they
don't answer any questions so this is
this is how i feel right now asking
about gpus
[Laughter]
but is there something interesting they
could reveal
or is it just you know uh yeah or would
leave it up to our imagination some of
the some of the compute is there any
i guess is there any fun trickery like i
talked to
chris lattner for a second time and he
was a key
person about tpus and there's a lot of
fun stuff going on in
google in terms of uh
hardware that optimizes for machine
learning is there something
you can reveal in terms of how much you
mentioned customization how much
customization there
is for hardware for machine learning
purposes
i'm going to be like that government you
know you that guy uh personally
audio foes
i i guess i you know will say that it's
really compute
is really important uh we have
very data hungry and compute hungry ml
models
of all over uh our stack and this is
where
you know both being part of alphabet as
well as designing our own sensors and
the entire hardware suite together
where on one hand you get access to like
really rich uh raw sensor
data that you can pipe from your sensors
uh
into your compute platform yeah and
build like build the whole pipe
from sensor raw sensor data to the big
compute as then have the massive compute
to process all that data
and this is where we're finding that
having a lot of control
of that that hardware part of the stack
is
really advantageous one of the
fascinating magical places to me
again might not be able to speak to the
details but
is the it is the other compute which is
like you know this we're just talking
about a single car
but the you know
the driving experience is a source of a
lot of fascinating data
and you have a huge amount of data
coming in on the car on the car
and you know the infrastructure of
storing some of that data
to then train or to analyze or so on
that's a fascinating like piece of it
that that i understand a single car i
don't understand how you pull it all
together in a nice way
is that something that you could speak
to in terms of the challenges of um
of seeing the network of cars and then
bringing the data back
and analyzing things that weren't that
like like edge cases of driving be able
to learn on them to improve the system
to
to see where things going wrong with
where things went right and analyze all
that kind of stuff
is there something interesting there in
the from an engineering perspective
oh there's an incredible uh
amount of really interesting work that's
happening there both
in the you know the real time operation
of the fleet of cars
and the information that they exchange
with each other in real time
to make better decisions as well uh
as on the kind of the off board
component where you have to deal with
massive amounts of data
for training your ml models evaluating
the male models
for simulating the entire system and for
you know evaluating your entire system
and this is where
and being part of alphabet has been once
again been tremendously
uh advantageous because we consume an
incredible amount of you know compute
for ml infrastructure we build a lot of
custom frameworks to you know get good
at you know
on data mining uh finding the
interesting edge cases for training and
for evaluation of the system
for both training and evaluating some
components and you know sub
uh parts of the system and various ml
models as well as the
uh evaluating the entire system and
simulation okay that first piece that
you mentioned that
cars communicating to each other
essentially i mean through perhaps
through a centralized point
but what uh that's fascinating too how
much does that help you like if you
imagine
like you know right now the number of
way more vehicles is
whatever x i don't know if you can talk
to what that number
but it's it's not in the hundreds of
millions yet
and imagine if the whole world is way
more vehicles
uh like that changes potentially the
power of connectivity like the more cars
you have
i guess actually if you look at phoenix
because there's enough vehicles
there's enough when there's like some
level of density
you can start to probably do some really
interesting stuff with
the fact that cars can negotiate can
be uh can communicate with each other
and thereby make decisions
is there something interesting there
that you can talk to about like how does
that help with the driving problem
from as compared to just a single car
solving the driving problem by itself
uh yeah it's it's a spectrum i uh first
to say that yeah
it's it helps uh and it helps in various
ways but it's not required
uh right now the way we build our system
engaged cars can operate independently
they can operate with no connectivity uh
so i think it is important that you know
you have
a fully uh autonomous you know fully
capable
uh driver uh that
computerized driver that each car has
then you know they do
share information and they share
information in real time it really
really helps right so the way we
do this today is uh you know whenever
one car
encounters something interesting in the
world whether it might be an accident or
a new construction zone that information
immediately gets
uh you know uploaded over the air and is
propagated to the rest of the fleet
so and that's kind of how we think about
maps as priors
in terms of the knowledge of our drivers
of our fleet of drivers that is
distributed across the fleet and it's
updated
in real time so that's one use case
you know you can imagine as the you know
the the density
of these vehicles go up that they can
exchange more information
in terms of what they're planning to do
uh and uh start
uh influencing how they interact with
each other uh as well as you know
potentially
sharing some observations right to help
with if you have enough density of these
vehicles where you know one car might be
seeing something that another is
relevant to another car
that is very dynamic you know it's not
part of kind of you're updating your
static prior
of the map of the world but it's more of
a dynamic information that could be
relevant to the
decisions that another cars make in real
time so you can see them exchanging that
information
and you can build on that but again i i
see that as
an advantage but it's you know not a
requirement
so what about the human in the loop so
uh when i got a chance to drive with a
ride in a waymo you know there's
customer service
[Laughter]
so like is somebody that's able to
dynamically
like tune in and uh help you out
what uh what role does the human play in
that picture that's a fascinating like
you know the idea of teleoperation be
able to remotely control a vehicle
so here what we're talking about is like
like frictionless uh like a human being
able to in a
in a frictionless way sort of help you
out i don't know if they're able to
actually control the vehicle
is that something you could talk to uh
yes okay uh to be clear
we don't do teleportation i'm going to
believe in teleoperation
for rare reasons that's not what we have
on our cars
we do as you mentioned have you know
version of you know customer support
uh you know we call it live health in
fact we find it that it's very
uh important for our rider experience
especially
if it's your first trip you've never
been in a fully driverless rider only
way more vehicle you get in there's
nobody there
right so you can imagine having all
kinds of you know questions in your head
like how this thing works
so we've put a lot of thought into kind
of guiding our
our writers our customers through that
experience especially for the first time
they get some information on the phone
uh if the fully driverless vehicle
is used to service their trip uh when
you get into the car
we have an in-car you know screen and
audio that kind of guides them and
explains
uh what to expect they also have a
button
that they can push that will connect
them to
you know a real life human being that
they can talk to
all right about this whole process so
that's one aspect of it um there is
i should mention that there is uh
another function that uh humans provide
uh to our
cars but it's not tele operation you can
think of it a little bit more like you
know fleet assistance
kind of like you know traffic control uh
that that you have
where our cars again they're responsible
on their own for making all of the
decisions all the driving decisions that
don't require connectivity
they you know anything that is safety or
latency critical
uh is done you know purely autonomously
by
on board uh our on onboard system uh but
there are situations where you know if
connectivity is available
uh can a car encounters a particularly
challenging situation you can imagine
like a super hairy
uh scene of an accident uh the cars will
do their best
they will recognize that it's an off
nominal situation they
will you know do their best to come up
you know with the right interpretation
the best course of action in that
scenario but if connectivity is
available they can ask
for confirmation from you know here mode
human
assistant to kind of confirm those
actions and perhaps
provide a little bit of kind of
contextual information and guidance
so october 8th was when you're talking
about the
was weimar launched the the
the fully self the public version
of its fully driverless that's right
term i think
service in phoenix is that october 8th
that's right
it was the introduction of fully
driverless rider only vehicles into our
you know public waymo one service okay
so that's that's amazing
so it's like anybody can get into waymo
in phoenix
oh that's right yeah so we previously
had
early people in our early writer program
uh
taking fully driverless rides in phoenix
and
uh just uh this a little while ago we
opened on october 8th we opened
that mode of operation to the public so
i can you know download the app and you
know go on the right
there is uh a lot more demand right now
uh for that service and then we have
capacity uh so we're kind of
uh managing that but that's exactly the
way you described it yeah well that's
interesting so there's more demand than
you can
you can handle like what uh
what has been uh reception so far like
what
i mean okay so you know that's this is
a product right that's a whole other
discussion of like how compelling of a
product it is
great but it's also like one of the most
kind of transformational technologies of
the 21st century
so there it's also like a tourist
attraction
like it's fun to you know to be a part
of it
so it'd be interesting to see like what
do people say what do people
uh what have been the feedback so far
you know still early days but so far the
feedback has been
uh incredible uh incredibly positive
they you know we asked them for feedback
during the ride we asked them for
feedback
uh after the ride as part of their trip
you know we asked them some questions we
asked them to
you know rate the performance of our
driver uh most
by far you know most of our drivers give
us five stars
in our app uh which is uh absolutely
great to see and yeah that's and we're
they're also giving us feedback on you
know things we can improve
uh and you know that's one of the main
reasons we're doing this is phoenix and
you know over the last couple of years
and every day today uh we are just
learning
a tremendous amount of new stuff from
our users there's there's no
substitute for actually doing the real
thing
actually having a fully driverless
product out there in the field with you
know users
uh that are actually paying us money to
get from point a to point b
so this is a legitimate like that's a
paid service that's right
and the idea is you use the app to go
from point a to point b
and then what what are the a's what are
the what's the freedom of the
of the starting and ending places it's
an area of geography where that service
is enabled it's a
you know decent size of geography of
territory it's actually larger than you
know the
size of san francisco uh and you know
within that
you have you know full freedom of you
know selecting where you want to go
you know of course there's some and you
on your app you get a map
you tell the car where you want to be
picked up
you know where you want you know the car
to pull over and pick you up and then
you tell it where you want to be dropped
off
all right and of course there's some
exclusions right you want to be you know
you uh
where in terms of where the car is
allowed to pull over right so
you know that you can't do but you know
besides that uh it's amazing
it's not like a fixed just would be very
i guess i don't know maybe that's what's
the question behind your question
but it's not a you know preset set of uh
yeah
so within the geographic constraints
with that within that area anywhere else
it can be you can be picked up and
dropped off anywhere that's right
and you know people use them on like all
kinds of trips they
we have and we have an incredible
spectrum of riders we i think the
youngest
actually have car seats them and we have
you know people taking their kids and
rides i think the youngest
riders we had on cars are one or two
years old you know and the full spectrum
of use cases people you can take them to
you know schools uh to you know go
grocery store shopping
to restaurants to bars you know run
errands you know go shopping et cetera
et cetera you can go to your office
right
like the full spectrum of use cases and
uh people
gonna use them in their daily lives to
get around
uh and we see all kinds of you know
really
interesting uh use cases and that that
that's providing us
incredibly valuable experience that
we then you know use to improve our
product so as somebody who's been on
done a few long rants
with joe rogan and others about the
toxicity of the internet and the
comments
and the negativity in the comments i'm
fascinated by feedback i
i believe that most people are
good and kind and intelligent and can
provide
like even in disagreement really
fascinating ideas so
on a product side it's fascinating to me
like how do you get
the richest possible user feedback
like to improve what's what are the
channels that
you use to measure because like you're
you're no longer that's one of the
magical things about autonomous vehicles
is it's not like it's frictionless
interaction with the human so like you
don't get to
you know it's just giving a ride so like
how do you get feedback from people to
in order to improve
uh yeah uh great question various
mechanisms uh so as part of the
normal flow we ask people for feedback
they as the car is driving around
we have on the phone and in the car and
to have a touchscreen
in the car you can actually click some
buttons and provide uh
real-time feedback on how the car is
doing and how the car is handling a
particular situation
you know both positive and negative so
that's one channel uh we have as we
discussed customer support or live help
where you know if a customer wants to
has a question
uh uh or he has some sort of concern
they can
talk to a person in real time so that
that is another mechanism that gives us
feedback uh at the end of a trip you
know we also ask them
how things went they give us comments
and you know star rating
and you know if it's uh we also you know
ask them
to explain what you know one one
well and you know what could be improved
and uh
we we have uh our writers providing you
know very rich
uh feedback there a lot the large
fraction is
uh very passionate and very excited
about this technology so we get really
good feedback
uh we also run uxr studies right you
know specific
and that are kind of more you know go
more in depth and we'll run both kind of
lateral and longitudinal studies um
where we have
you know deeper engagement uh with our
customers you know we have our
user experience research team tracking
over time and testing is about longitude
no it's cool
that's that's exactly right and you know
that's another really valuable uh
feedback uh source of feedback and you
we're just covering a tremendous amount
right
uh people go grocery stroping and they
like want to load
you know 20 bags of groceries in our
cars and like that that's one workflow
that you maybe don't you know think
about uh
you know getting just right when you're
building the driverless product
i have people like you know who uh
bike as part of their trip so they you
know bike somewhere then they get on our
cars they take a
part their bike they load into our
vehicle then go and that's you know how
they
you know where we want to pull over and
how that you know uh get in
and get out um uh process works uh
provides
very uh useful feedback in terms of you
know what makes a good
uh pickup and drop-off location uh we
get really valuable feedback
and in fact we had to um uh do some
really interesting work with
high definition maps and uh thinking
about
walking directions if you imagine you're
in a store right in some giant space
and then you know you want to be picked
up somewhere like if you just drop
a pin in the current location which is
maybe in the middle of a shopping mall
like what's the best
location for the car to come pick you up
and you can have simple heuristics where
you just kind of take your you know your
cleaning distance uh and find the
nearest
uh spot where the car can't pull over
that's closest to you but oftentimes
that's not the most convenient one you
know i have many anecdotes where that
heuristic
breaks in horrible ways i one example uh
that yeah i often mention is somebody
wanted to be
you know uh dropped off uh and
phoenix uh and you know we car picked a
location
uh that was close the closest to their
you know where the pin was dropped
on the map in terms of you know latitude
and longitude but it happened to be
on the other side of a parking lot that
had this row of cacti
and poor person had to like walk all
around the parking lot to get to where
they wanted to be
in 110 degree heat so that you know that
was about so then you know we took all
take all of these um all that feedback
from our users and
uh incorporate it into our system and
yeah and improve it
yeah i feel like that's like requires
agi to solve the problem
of like when you're which is a very
common case when you're in a big space
of some kind
like apartment building it doesn't
matter it's not some large space
and then you call the like the waymo
from there
right like so and you whatever it
doesn't matter right your
vehicle and like where is the pin
supposed to drop i feel like that's i
you don't think i think that requires a
gi i'm gonna
in order okay the alternative
which i think the google search engine
has taught
is like there's something really
valuable about
the perhaps slightly dumb answer but a
really powerful one which is like
what was done in the past by others like
what was the choice made by others
that seems to be like in terms of google
search when you have like billions of
searches
you can you could see which like when
they recommend
what you might possibly mean they
suggest based on
not some machine learning thing which
they also do but like on what
was successful for others in the past
and finding a thing that they were happy
with
is that integrated at all with waymo
like
what what pickups worked for others it
is
i i think you're exactly right so
there's uh real it's an interesting
problem
uh naive solutions uh
have uh interesting failure modes
uh so there's definitely
lots of things that can be done to
improve
uh and both learning
from you know what works what doesn't
work in actual heal from you know
getting richer data and
getting more information about the
environment and you know
richer maps but you're absolutely right
that there's something and there's some
properties of solutions that
uh in terms of the effect that they have
on users so much much much much better
than others right unpredictability
and understandability is important so
you can have maybe something that is not
quite as optimal but
is very natural and predictable to the
user and kind of works the same way
all the time and that matters that
matters a lot for the user experience
and but you know to get to the basics
the pretty fundamental property
is that the car actually arrives
where you told it right like you can
always you know change it see it on the
map and you can move it around if you
don't like it and
but like that property that the car
actually shows up reliably
yeah is critical which you know where uh
compared to some of the human uh driven
yes
analogs i think you know you can have
more unpredictability
it's actually uh the fact uh if if i
have uh
might do a little bit of a detail here
uh i think the fact that it's
you know your phone and the cars two
computers talking to each other uh can
lead to some
really interesting things we can do in
terms of the user interfaces
both in terms of function uh like the
car actually shows up
exactly where you told it uh you want it
to be but also some you know really
interesting things on the user interface
right as the car is driving as you
you know call it and it's on the way to
come and pick you up and of course you
get the position of the car
and the route on the map uh but and they
actually follow that route of course
uh but it can also share some really
interesting information about what it's
doing
so uh you know our cars uh as
they are coming to pick you up if it's
come if a car is coming up to a stop
sign
it will actually show you that like it's
there sitting because it's at a stop
sign or a traffic light it'll show you
that it's got you know sitting at a red
light
so you know they're like little things
uh right uh but
it i find those little touch uh touches
uh really interesting really magical
and it's just you know little things
like that that you can do to kind of
delight your users
you know this makes me think of um
there's some products that i just love
like
there's a there's a company called rev
uh rev.com where i like for this
podcast for example i can drag and drop
a video
and then they do all the captioning
uh it's humans doing the captioning but
they connect you good they they
automatic
automate everything of connecting you to
the humans and they do the captioning
and transcription it's all effortless
and like i remember when i first started
using them it was like
life is good like because it was so
painful to
to figure that out earlier uh the same
thing with uh
something called izotope rx this company
i use for cleaning up audio like
the sound cleanup they do it's like drag
and drop and it just cleans everything
up
very nicely uh another experience like
that i had with amazon one click
purchase first time i mean other places
do that now but
just the effortlessness of purchasing
making it frictionless
it kind of communicates to me like i'm a
fan of design i'm a fan of products
that you can just create a really
pleasant experience
the simplicity of it the elegance just
makes you fall in love with it
so on the do you think about this kind
of stuff i mean
that's exactly what we've been talking
about it's like the little details that
somehow make you fall in love with the
product
is that we went from like urban
challenge days where
where love was not part of the
conversation probably
and to to this point where there's uh
where there's human beings and you want
them to fall in love with the experience
is that something you're trying to
optimize for trying to think about like
how do you
how do you create experience that people
love absolutely
i think that's the vision is removing
any friction or complexity
from getting our
users our writers to where they want to
go and
making that as simple as possible and
then you know beyond that
on just transportation making you know
things and you know goods
get to their destination as seamlessly
as possible and talked about
you know a drag and drop experience
where you kind of express your intent
and then
you know it just magically happens and
for our riders that's
what we're trying to get to is you
download an app and you can
click and car shows up it's the same car
it's very predictable it's
a safe and high quality experience and
then
it gets you in a very reliable very
convenient
uh frictionless
way to where you want to be and along
the journey
i think we also want to like do a little
things to delight
our users like the ride-sharing
companies because they don't control the
experience i think
they can't make people fall in love
necessarily with the experience
or maybe they haven't put in the effort
but
i think it if i would just speak to the
ride-sharing experience i currently have
it's just very it's just very convenient
but there's a lot of room for like
falling in love with it
like we can speak to sort of car
companies car companies do this well you
can fall in love with a car
right and be like a loyal car person
like whatever
like i like bad ass hot rods i guess 69
corvette
and at this point you know you can't
really
cars are so owning a car is so 20th
century man
but is there something about the waymo
experience where you hope that people
will fall in love with because that
is that part of it or is it part of
is it just about making a convenient
ride
not ride sharing i don't know what the
right term is but just the convenient
eight to be
autonomous um transport
or like do you want them to fall in love
with waymo so maybe elaborate a little
bit
i mean almost like from a business
perspective i'm curious
like how
do you want to be in the background
invisible or do you want to be
uh like a source of joy that's
in very much in the foreground i want to
provide the best most enjoyable
transportation solution
uh and that means
building it building our product and
building our service in a way
that people do uh kind of use
in a very
seamless frictionless way in their in
their day-to-day lives
and i think that does mean uh you know
in some way falling in love
in that product right just kind of
becomes part of your routine i
uh it comes down my mind to
safety predictability of the experience
and um
privacy i think aspects of it right our
cars you get the same car you get very
predictable behavior
and that that is important and if you're
going to use it in your daily life
privacy and when you're in a car you can
do other things you're spending a bunch
just another space where you're spending
a significant
part of your life right so not having to
share it with
other people who you don't want to share
it with i think is uh
a very nice property uh maybe you want
to take a phone call or
do something else in the vehicle um and
you know
safety on the quality of the driving as
well as the physical safety
of you know not having so you know to
share that
ride is you know important to a lot of
people
what about the idea that when when
there's
somebody like a human driving and they
do a rolling stop on a stop sign like
sometimes like
you know you get an uber a lift or
whatever like human driver
and you know they can be a little bit
aggressive
as as drivers it feels like there is um
not all aggression is bad uh
now that may be a wrong again 20th
century conception of driving
maybe it's possible to create a driving
experience like
if you're in the back busy doing
something maybe
aggression is not a good thing it's a
very different kind of experience
perhaps
but it feels like in order to navigate
this world
you need to uh how do i
uh phrase this you need to kind of bend
the rules a little bit or at least like
test the rules
i don't know what language politicians
use to discuss this but uh
whatever language they use you like
flirt with the rules i don't know
but like you uh you sort of
uh have a bit of an aggressive way of
driving
that asserts your presence in this world
thereby making other vehicles and people
respect your presence
and thereby allowing you to sort of
navigate through intersections in a
timely fashion
i don't know if any of that made sense
but like how
does that fit into the experience of
driving autonomously is that
a lot of sales this is you're hitting a
very important point of
a number of behavioral components and
parameters that make your driving
feel you know assertive and natural and
comfortable predictable
um now our cars will follow rules right
they will do the safest thing possible
in all situations let you know be clear
on that
uh but if you think of really really
you know good drivers just you know
think about you know professional limo
drivers right they will follow the rules
they're very very smooth uh and yet
they're very efficient uh and but
they're assertive
uh they're comfortable for the people in
the vehicle
they're predictable for the uh other
people outside the vehicle that they
share the environment with
and that that's the kind of driver that
we want to build and you think
if maybe there's a sport analogy there
right yeah you can
do in very many sports the
true professionals are very efficient in
their movements
right they don't do like you know hectic
uh
flailing right they're you know smooth
and
precise right and they get the best
results so that's the kind of driver
that we want to build
in terms of you know aggressiveness yeah
you can like you know roll through the
stop signs you can do crazy lane changes
uh it typically doesn't get you to your
destination faster typically not the
safest or most predictable uh
very most comfortable thing to do and uh
but there is
a way to do both and that that that
that's what we're doing we're trying to
build a driver that is
uh safe comfortable smooth
and predictable yeah that's a really
interesting distinction i think in the
early days of autonomous vehicles
the vehicles felt cautious as opposed to
efficient
and and still probably but when i rode
in the waymo i mean there was
it was it was quite assertive it moved
pretty quickly
like um yeah and he's one of the
surprising feelings was that it actually
it went fast and it didn't feel like
awkwardly cautious than autonomous
vehicle like
like so i've also programmed autonomous
vehicles and everything i've ever
built was felt awkwardly either overly
aggressive
okay especially when it was my code or
uh like awkwardly cautious is the way i
would put it
and the waymo's vehicle felt
like uh assertive and i think efficient
as
like the right terminology here it
wasn't uh
and i also like the professional limo
driver because we often think like
you know an uber driver or a bus driver
or a taxi
this is the funny thing is people think
that taxi drivers are professionals
they i mean it's it's like that that's
like saying
me i'm a professional walker just
because i've been walking all my life
i think there's an art to it right
and if you take it seriously as an art
form
then there's a certain way that mastery
looks like
it's interesting to think about what
does mastery look like in driving
and perhaps what we associate with like
aggressiveness
is unnecessary like it's not part of the
experience of driving it's like
unnecessary fluff
that efficiency you could you can be
you can create a good driving experience
within the rules
that's uh i mean you're the first person
to tell me this so it's it's kind of
interesting i need to think about this
but that's exactly what it felt like
with waymo i kind of had this intuition
maybe it's the russian thing i don't
know
that you have to break the rules in life
to get anywhere
but maybe maybe it's possible that
that's not the case
in driving i have to think about that
but it certainly felt that way on the
streets of phoenix when i was there in
in waymo that
that that that was a very pleasant
experience and it wasn't frustrating in
that like
come on move already kind of feeling it
wasn't it that wasn't there
yeah i mean that's what that's what
we're going after yeah i don't think you
have to pick one i think
truly good driving and gives you both
efficiency assertiveness but also
comfort and predictability and you know
safety
uh and you know it's that's what
fundamental improvements in the
core capabilities truly unlock and
you can kind of think of it as you know
a precision and recall trade-off you
have certain capabilities of your model
and then it's very easy when you know
you have some curve of precision and
recoil you can move things around and
you can choose your operating point in
your training of precision versus recall
false positives versus false negatives
right but then and you know you can tune
things on that curve and be kind of more
cautious or more aggressive but then
aggressive is bad or you know cautious
is bad
but true capabilities come from actually
moving the whole curve
up right and then you are kind of on a
very
different plane of those trade-offs and
that that's what you know we're
trying to do here is to move the whole
curve up before i forget let's talk
about trucks
a little bit uh so i also got a chance
to check out some of the waymo truck
uh trucks i'm not sure if uh we want to
go
too much into that space but it's a
fascinating one so maybe we can mention
at least briefly
you know waymo is also not doing
autonomous trucking and uh how
different like philosophically and
technically is that whole space of
problems
it's one of our two big
products and uh you know commercial
applications
of our driver right right handling and
deliveries you know
we have waymo one and waymovia moving
people and moving goods
uh you know trucking is an example of
uh moving goods uh we've been uh working
on trucking since 2017. uh it
is uh
a very interesting space and your
question
how different is it it has this really
nice property that
the first order challenges like the
science
the hard engineering uh whether it's you
know hardware or you know onboard
software or
off-board software all of the you know
systems that you build
for you know training your ml models for
you know evaluating a retirement system
like those fundamentals
carry over the true challenges of
driving perception semantic
understanding
prediction decision making more planning
evaluation
uh the simulator ml infrastructure those
carry over
i think the data and the application and
kind of the the
domains might be different but the the
most difficult problems
uh all of that carries over between the
domains so
that that's very nice so that's how we
approach it we're kind of
build investing in the core the
technical core
and then there's specialization of and
uh
of that core technology to different
product lines to different commercial
applications
so on just to tease it apart a little
bit
uh on trucks so starting with the
hardware the configuration of the
sensors
is different right they're different
physically geometrically you know
different vehicles
uh so for example we have two of our
main laser
uh on the trucks on both sides so that
we have you know don't have the blind
spots
uh whereas on the jlr i-pace we have you
know one of it uh
sitting at the very top but the actual
sensors are uh almost the same
or largely uh the same so all of the
investment
that uh over the years we've put into
building our custom lighters custom
radars and pulling the whole system
together
that carries over very nicely uh then
you know on the perception side
uh the like the fundamental challenges
of
seeing understanding the world whether
it's you know object detection
classification
you know tracking semantic understanding
all that carries over now yes there's
some specialization when you're driving
on freeways
uh you know range becomes more important
the domain is a little bit different
but again the fundamentals carry over
very very nicely
same and i guess you get into prediction
or decision making
right the fundamentals of what it takes
to
predict what other people are going to
do to
find the long tail to improve your
system in that long tail
of behavior prediction and response that
carries over right and so on and so on
so i mean that's pretty exciting by the
way does waymovia include
using the the smaller vehicles for
transportation goods that's an
interesting distinction
so let's say there's three interesting
modes of operation so one is moving
humans one is moving goods
and one is like moving nothing zero
occupancy
meaning like you're going to the
destination
your your empty vehicle i mean it's it's
the third is the last wave that's the
entirety of it it's so less you know
exciting from the commercial perspective
[Laughter]
well i mean in terms of like if you
think about what's inside a vehicle as
it's moving
because it does you know some
significant fraction of the vehicle's
movement has to be
empty i mean it's kind of fascinating
maybe just on that small point is
is there different
control and like policies
that are applied for a zero occupancy
vehicle so vehicle with nothing in it
or is it just move as if there is a
person inside what was
with uh some subtle differences as a
first order approximation
there are no differences and if you
think about
you know safety and you know comfort and
quality of driving
only part of it you know
has to do with the people or the goods
inside of the vehicle
right but you don't want to be you know
you want to drive smoothly and as we
discussed
not for the purely funded benefit of you
know whatever you have inside the car
right it's also for the benefit of the
you know people outside kind of
feeding fitting uh naturally and
predictably into the whole environment
right
so you know yes there are some second
order uh things you can do it's gonna
change your route
and you know optimize maybe kind of your
fleet
things at the fleet scale and you would
take into account whether
some of your cars are actually
you know serving a useful trip whether
with people or with goods whereas you
know other cars
are you know driving completely empty
you know to that next
valuable trip that they're going to
provide but that those are
mostly second order effects okay cool so
phoenix
is uh is an incredible place and what
you've announced in phoenix is uh
it's kind of amazing but you know that's
just like one
city how do you take over the world
uh i mean i'm asking for a friend once
one step at a time
is that the cartoon pinky in the brain
yeah okay
but you know gradually is a true answer
so i think
the heart of your question is
what can you ask a better question than
i asked they asked a great question
to answer that one i i i'm you know just
gonna
you know phrase it in the terms that i
want to answer perfect
exactly right brilliant please
you know where are we today and you know
what happens next
uh and what does it take to go beyond
phoenix and was it what does it take
uh to get this technology to
more places and more people around the
world right
so our next
big area of focus is
exactly that larger scale
commercialization and
you know scaling up uh
if i think about
you know the main and your phoenix gives
us that
platform it gives us that foundation of
upon which we can build them and it's
there are few really
challenging aspects of this whole
problem that you have to pull together
in order to build the technology
in order to deploy it
uh into the field to go
from a driverless car to
a fleet of cars that are providing a
service
and then all the way to you know
commercialization so
uh and then you know this is what we
have in phoenix we've taken the
technology from
uh a proof point to an actual deployment
and have taken our driver you know from
you know one car to a fleet that can
provide a service um
beyond that if i think about what it
will take to
scale up and you know deploy in
you know more places with more customers
i
tend to think about uh three
main dimensions three main axes um of
scale
one is the core technology you know the
hardware
and software core capabilities of our
driver
the second dimension is
evaluation and deployment and the third
one
is the product commercial and
operational excellence so you can talk
you know
a bit about where we are along you know
each one of those three dimensions about
where we are today
and you know what has what will happen
next
um on you know the core technology on
you know the hardware and software and
together comprise our driver
we you know obviously have that
foundation
that is providing fully driverless trips
to our customers as we speak
in fact and we've learned a tremendous
amount from that so now what we're doing
is we are incorporating all those
lessons
into some pretty fundamental
improvements in our core technology both
on the hardware side and on the software
side
to build a more general more robust
solution
that then will enable us to massively
scale you know beyond phoenix
so on the hardware side
all of those lessons are now
incorporated into this fifth generation
hardware platform
that is you know uh being deployed right
now
and that's the platform the fourth
generation the thing that we have right
now driving in phoenix
it's good enough to operate operate
fully driverlessly you know
night and day in various speeds and
various conditions
but the fifth generation is the platform
upon which we want to go to massive
scale
we it in turn we've really made
qualitative improvements in terms of the
capability of the system
the simplicity of the architecture the
reliability of the redundancy
it is designed to be manufacturable at
very large scale and you know provides
the right unit economics
so that's that's the next big step for
us um on the hardware side
that's that's already there for scale
the version five
that's right is that uh coincidence or
should we look into it conspiracy theory
that's the same version as the pixel
phone
is that what's the harder they neither
confirm okay
all right cool so sorry so that's the
okay that's that axis what else
uh so similarly hardware is a very
discrete
jump but you know similar to the uh that
to how we're making that change from the
fourth generation hardware to the fifth
we're making similar improvements on the
software side
to make it more you know robust and more
general and allow us to kind of
quickly uh scale beyond phoenix so that
that's the first dimension of core
technology the second dimension is
evaluation and deployment
now how do you measure
your system how do you evaluate it how
do you build the release and deployment
process
where you know with confidence you can
you know regularly release new versions
of your driver
into a fleet how do you get good at it
so that
it is not you know a huge tax on your
researchers and engineers that you know
so you can how do you build all these
you know
processes the frameworks the simulation
the evaluation the data science the
validation
so that you know people can focus on
improving the system and kind of the
releases
just go out the door and get deployed
across the fleet so we've gotten really
good at that
in phoenix that's been a tremendously
difficult problem
but that's what we have in phoenix right
now that gives us that foundation
and now we're working on kind of
incorporating all the lessons that we've
learned
to make it more efficient to go to new
places you know scale up and just kind
of you know stamp things out
so that's that second dimension of
evaluation and deployment and the third
dimension
is product commercial and operational
excellence right and again phoenix there
is providing
uh an incredibly valuable platform you
know that's why we're doing things
end-to-end uh in phoenix we're learning
as you know we discussed a little
earlier today
a tremendous amount of really valuable
lessons from our users getting really
incredible feedback
uh and uh we'll continue to iterate on
that and
incorporate all those uh those lessons
into
making our product you know even better
and more convenient for our users
so you're converting this whole process
of phoenix
in phoenix into uh something that could
be copy and pasted elsewhere
so like uh perhaps you didn't think of
it that way when you were doing the
experimentation phoenix but
so how long did basically
you can correct me but you've i mean
it's still early days but you're taking
the full journey
in phoenix right as you were saying
of like what it takes to basically
automate i mean it's not the entirety of
phoenix right
but i imagine it can encompass
the entirety of phoenix that's some some
uh near-term date but that's not even
perhaps important like
as long as it's a large enough
geographic area so
what how copy-pastable
is that process currently and
how do like um
you know like when you copy and paste in
in uh in google
docs i think you know in or in word
you can like apply source formatting or
apply destination formatting
so how when you copy and paste uh the
phoenix
into like say boston
uh how do you apply the destination
formatting
like how much of the core of the entire
process
of bringing an actual public
transportation autonomous transportation
service to a city
is there in phoenix that you understand
enough to copy and paste into boston
or wherever um so we're not quite there
yet
we're not at a point where we're kind of
massively copy and pasting
all over the place uh but phoenix
what you know we did in phoenix and we
very intentionally have chosen phoenix
as
our first full deployment
uh area you know exactly for that reason
to kind of tease the problem apart
look at each dimension and focus on the
fundamentals of complexity and
de-risking
you know those dimensions and then
bringing the entire thing together to
get all the way
and force ourselves to learn all those
hard lessons on technology hardware and
software
on the evaluation deployment on you know
operating a service operating a business
using
uh actually you know um serving our
customers
all the way so that we're fully informed
about
the most difficult most important
challenges
to get us to that next step of massive
copy and pasting
as as you said and uh
[Music]
that's what we're doing right now we're
incorporating all those things that we
learned into
that next system that then will allow us
to kind of copy paste all over the place
and to massively scale to you know more
users and
more locations i mean you know just
talked a little bit about you know what
does that mean along those
different dimensions so on the hardware
side for example again it's that
uh switch from the fourth to the fifth
generation and the fifth generation is
designed to kind of have that property
can you say what other cities you're
thinking about like i'm thinking about
sorry we're in san francisco now i
thought i want to move to san francisco
but i'm thinking about moving to austin
um i don't know why people are not being
very nice about san francisco
currently for maybe it's a small it's
like maybe it's in vogue right now
but uh austin seems i visited there and
there was uh
i was in a walmart it's funny these
moments
like turn your life there's this very
nice
woman with kind eyes just like stopped
and said you look so handsome in that
tie honey
to me this has never happened to me in
my life but just the sweetness of this
woman
is something i've never experienced
certainly on the streets of boston but
even in san francisco where people
wouldn't
that's just not how they speak or think
i don't know there's a warmth too to
austin that love
and since waymo does have a little bit
of a history there
is that a possibility is this your
version of asking the question of like
you know dimitri i know you can't share
your commercial and deployment roadmap
but i'm thinking about moving to should
i cisco austin like in a blink twice if
you think i should move to him
yeah that's true this room you got me we
you know we've been
testing and all over the place i think
we've been testing more in
25 cities we drive in san francisco we
drive in
you know michigan for snow uh we we are
doing significant amount of testing in
the bay area including san francisco
which is not like because we're talking
about the very different thing which is
like
a full-on large geographic area
public service uh you can't share any
okay
what about moscow is that when is that
happening
take on yandex i'm not paying attention
to those folks
they're doing you know there's there's a
lot of fun
i mean maybe as a way of a question
you didn't speak to sort of like
policy or like is there tricky things
with government and so on
like is there other friction
that you've encountered except sort of
technological
friction of solving this very difficult
problem
is there other stuff that you have to
overcome
when when uh deploying a public service
in a city
that's interesting it's very important
so we we
put significant effort in uh
creating those partnerships and you know
those relationships
with governments at all levels you know
local governments municipalities you
know state level federal level uh we've
been engaged in very deep conversations
from the earliest days of our
you know projects uh whenever at all of
these levels you know whenever we go
to test uh or you know operate in a
new area you know we always lead with
with a conversation with the local
officials and but the result of that
that investment is that no
it's not challenges we have to overcome
it but it is a very important
that we continue to have this
conversation oh yeah
i love politicians too okay uh so mr
elon musk
said that uh lidar is a crutch
what are your thoughts
i wouldn't characterize it exactly that
way uh i know i think lighter is
very important uh it is a key sensor
uh that you know we use just like other
modalities and as we discussed
our cars use cameras uh lidars and
radars
they are all very important they are
at the kind of the physical level they
are very different they have very
different you know physical
characteristics
cameras are passive lighters and radars
are active you use different wavelengths
uh so that means they complement each
other uh
and very nicely and and together
combined they can be used to build a
much
safer and much more capable system so
you know to me it's more of a question
you know why the heck would you handicap
yourself and not use one
or more of those sensing modalities when
they you know undoubtedly just make your
system
uh more capable and safer
now it you know
what might make sense for one product uh
or one business might not make sense for
another one
so if you're talking about driver assist
technologies you make certain design
decisions and you make certain
trade-offs
and you make different ones if you are
you know building a driver uh that deep
deploy in fully driverless
vehicles uh and you know and lighter
specifically when this question comes up
i uh you know typically the criticisms
uh
that i hear or you know the
counterpoints
that cost and aesthetics
and like i i don't find either of those
honestly very compelling so on the cost
side
there's nothing fundamentally
prohibitive about you know the cost of
lighters you know radars used to be very
expensive uh before people start you
know
uh before people need certain balances
and technology and you started to
to manufacture them uh massive scale and
deploy them in vehicles right
uh similarly with lighters and this is
where the lidars that we have on our
cars especially the fifth generation
uh you know we've been able to make
some pretty qualitative discontinuous
jumps in terms of the fundamental
technology that allow us to
manufacture those things at very
significant scale
and add a fraction of the cost of you
know both
our previous generation as well as a
fraction of the cost of you know what
might be available
on the market you know off the shelf
right now and you know that improvement
will continue so i i think you know cost
is uh
not a real issue uh second one is uh you
know uh aesthetics
uh you know i don't think that's you
know a real issue either
uh um the beholder yeah
you can make lidar sexy again i think
you're exactly right i think it is sexy
like honestly i think foreign
you know i was actually somebody brought
this up to me um
i mean all forms of lidar even
uh even like the ones that are like big
you can make
look i mean it can make look beautiful
like
there's no sense in which you can't
integrate it into design
like there's all kinds of awesome
designs i don't think
small and humble is beautiful it could
be
like you know brutalism or like it could
be
uh like harsh corners i mean like i said
like hot rods like i don't like i don't
necessarily like like oh man i'm gonna
start so much controversy with this
i i don't like porsches
okay the porsche 911 like everyone says
the most beautiful
no it no it's like it's like a baby car
it doesn't make any sense
but everyone it's beauty's denied the
beholder you're already looking at me
like what's this kid talking about
you're happy to talk about you're
digging your own home the form and
function
and my take on the beauty of the
hardware that we put on our vehicles
you know i will not comment on a porsche
monologue
okay all right so but aesthetics fine
but there's an underlying like
philosophical question
behind the kind of lighter question is
like how much
of the problem can be solved with
uh computer vision with machine learning
so i think without sort of
disagreements and so on
it's nice to put uh it on the spectrum
because waymo is doing a lot of machine
learning as well
it's interesting to think how much of
driving if we look at five years
10 years 50 years down the road would
can be
learned in almost more and more and more
end-to-end way if we look at what tesla
is doing
with the as a machine learning problem
they're doing a multi-task learning
thing where it's just they break up
driving into a bunch of learning tasks
and they have one single neural network
and they're just collecting huge amounts
of data that's training that
i've recently hung out with george cotts
i don't know if you know george
uh i love him so much he's just an
entertaining human being we were off
mike talking about hunter s thompson
he's
he's the hunter that's thompson and
baton was driving okay so he
i didn't realize this with common ai but
they're like
really trying to do end to end they're
the machine
like looking at the machine learning
problem they're
really not doing multi-task learning but
it's
uh it's it's computing the drivable area
as a machine learning task
and hoping that like down the line
this level two system this driver
assistance
will eventually lead to allowing you to
have a fully autonomous vehicle
okay there's an underlying deep
philosophical question there technical
question
of how much of driving can be learned
so lidar is an effective tool today
uh for actually deploying a successful
service in phoenix
right that's safe that's reliable et
cetera et cetera but
uh the the question and i'm not saying
you can't do machine learning on lidar
but the the question is that like how
much of driving can be
learned eventually can we do fully
autonomous
that's learned yeah uh you know learning
is all over the place
and plays a key role in every part of
our system
i i as you said i would uh you know
decouple the sensing modalities
from the you know ml and the software
parts of it
lighter radar cameras like it's all
machine learning
all of the object detection
classification of course like that's
what you know these
modern deep nuts and continents are very
good at you feed them raw data
massive amounts of raw data um and you
know
that's actually what our custom build
lighters and raiders are really good at
and radars they don't just give you
point estimates of you know objects in
space they give you raw
like physical observations and then you
take all of that raw information you
know there's colors of the pixels
whether it's you know lighters returns
and some auxiliary information it's not
just distance
right and you know angle and distance is
much richer information that you get
from those returns
plus really rich information from the
radars you fuse it all together and you
feed it into those massive
ml models that then you know
lead to the best results in terms of you
know object uh deduction
classification you know state estimation
so there's a side interrupt but there is
a fusion
i mean that's something that people
didn't do for a very long time which is
like at the
sensor fusion level i guess like early
on fusing the
information together whether so that the
the sensory
information that the vehicle receives
from the different modalities
or even from different cameras is
combined before it is fed into the
machine learning models
uh yes i think this is one of the trends
you're seeing more of that you mentioned
end to end there's different
interpretations of antenna there's kind
of the
purest interpretation now i'm gonna like
have one model
that goes from raw sensor data to like
you know steering torque
and you know guest brakes that you know
that that's too much i don't think
that's the right way to do it
there's more you know smaller versions
of end to end
where you're you know kind of doing more
end-to-end learning or core training or
deep propagation
of kind of signals back and forth across
the different stages of your system
there's no really good ways it gets into
some fairly complex design choices where
on one hand you want modularity and the
compass
composite ability the composibility of
your system
but on the other hand you don't want to
create interfaces that are too narrow
or too brittle to engineered where
you're giving up on the generality of
the solution
or you're unable to properly propagate
signal you know reach signal forward and
losses and you know back so you can you
know optimize the whole system jointly
uh so i would decouple and i guess what
you're seeing in terms of the fusion
of the sensing data from different
modalities as well as kind of fusion
at in the temporal level going more from
you know frame by frame
yeah where you know you would have one
net that would do frame by frame
detection and camera and then you know
something that does frame by frame and
lighter and then radar and then you fuse
it you know in a weaker engineered way
later
like the field over the last you know
decade has been evolving in more kind of
joint fusion more end-to-end models that
are
solving some of these tasks you know
jointly and there's tremendous power in
that
and you know that that's that's that
that's the progression that kind of our
technology
our stack has been on as well now it's
your you know that so i would decouple
the kind of
sensing and how that information is used
from the role of ml in the entire stack
and you know i guess it's uh i there's
trade-offs
uh and you know modularity and how do
you inject
inductive bias into your system right
this is
uh there's tremendous power in being
able to do that
so you know we have there's no
part of our system that is not heavily
that does not heavily you know leverage
uh
data-driven development or a
state-of-the-art ml
but there's mapping there's a simulator
there's perception
you know object level you know
perception whether it's semantic
understanding prediction
decision making you know so forth and so
on um
it's and of course object detection and
classification like you're finding
pedestrians and cars and cyclists and
you know cones and signs and vegetation
and being very good at estimating kind
of detection classification and state
estimation
there's just stable stakes like like
that's step zero of this whole stack
you can be incredibly good at that
whether you use cameras or light as a
radar but they're just you know that's
stable stakes that's just stub zero
beyond that you get into the really
interesting challenges of semantic
understanding of the perception level
you get into scene level reasoning you
get into very deep problems
uh that have to do with prediction and
joint production and interaction so
interaction between all of the actors in
the environment pedestrian cyclists
other cars and you get into decision
making
right so how do you build a lot of
systems so uh we
leverage ml very heavily in all of these
components i do believe that the best
results
you achieve by kind of using a hybrid
approach
and having different types of ml
having different models with
different degrees of inductive bias that
you can have
and combining kind of model you know
free approaches with some you know model
based approaches and
some uh rule-based uh physics-based
uh systems so you know one example i can
give you is traffic lights
uh there's problem of the detection of
traffic light state and obviously that's
a great problem for you know computer
vision confidence are
you know that's their bread and butter
right that's how you build that
but then the interpretation of you know
of a traffic light that you're going to
need to learn that right you you
read you don't need to build something
you know complex ml model that you know
infers with some you know precision and
recall that red means stop
like it was a it's a very clear
engineered signal with very clear
semantics
right so you want to induce that bias
like how you induce that bias and that
whether you know it's a constraint
or a cost you know function in your
stack
but like it is important to be able to
inject that like clear semantic
signal into your stack and you know
that's what we do um
and but then the question of like and
that's when you
apply it to yourself when you're making
decisions whether you want to stop for a
red light
you know or not but if you think about
how other people
treat traffic lights we're back to the
ml version of that
because you know they're supposed to
stop for a red light but that doesn't
mean they will so then you're back in
the like very
uh heavy uh ml domain where you're
picking up on like very subtle keys
about
you know that have to do with the
behavior of objects pedestrians cyclists
cars and the whole entire configuration
of the scene
that allow you to make accurate
predictions on whether they will in fact
stop
or run a red light so it sounds like a
ready
for waymo like machine learning is a
huge part of the stack
so it's a huge part of like uh not just
so obviously the the first the level
zero or whatever you said which is like
just object detection of things that you
know with no that machine learning can
do but also starting to
to do prediction behavior and so on to
model the
what other or the other parties in the
scene entities in the scene are gonna do
so machine learning is more and more uh
playing a role in that
as well of course absolutely i think
we've been
and going back to the earliest days like
you know darpa
even the grand challenge and team was
leveraging you know machine learning
i was like pre you know image nut and it
was very different type of ml
but uh and i think actually that was
before my time but the stanford team
on during the grand challenge had a very
interesting machine learned
system that would you know use lighter
and camera
when driving in the desert and it we had
built the model
uh where it would kind of extend the
range
of free space reasoning so we get a
clear signal from lighter
and then it had a model that hey like
this stuff and camera kind of sort of
looks like this stuff and
lighter and i know this stuff and that
i've seen in lighter i'm very confident
there's free space so let me extend that
uh free space zone into the camera range
that would allow the vehicle to drive
faster
right and then we've been building on
top of that and kind of staying and
pushing the state of the art in a ml
in all kinds of different ml uh over the
years and in fact uh from the earlier
days i think you know 2010
is probably the year where google
uh maybe 2011 probably got got pretty
heavily involved
in uh machine learning uh kind of deep
nuts
uh and at that time was probably the
only company that was very heavily
investing
in kind of state-of-the-art ml and
self-driving cars
right and they they they go ahead you
know hand in hand
and we've been on that journey ever
since we're doing uh pushing
a lot of these areas uh in terms of
research you know at waymo and we
collaborate
very heavily with the researchers in
alphabet
and like all kinds of mel yeah supervise
the male unsupervised male
uh you know published some uh
interesting uh research papers in the
space uh
especially recently it's just super
super learning as well yeah so super
super active uh
of course there's you know kind of like
more uh mature stuff
like you know confidence for you know
object detection but there's some really
interesting really active uh
work that's happening in um kind of more
uh you know and bigger models and you
know models
that uh have more structure uh to them
uh you know not just you know large
bitmaps and reasonable temporal
sequences
and some of the interesting
breakthroughs that you've you know we've
seen
in language models right you know
transformers you know
you know gpd 3 and friends uh there's
some really interesting applications of
some of the core breakthroughs to those
problems
of you know behavior prediction as well
as you know decision making and planning
right you
think about it kind of the the behavior
how you know the path the trajectories
the the how people drive
and they have kind of a share a lot of
the fundamental structure
you know this problem there's you know
sequential
you know nature there's a lot of
structure uh in this representation
there is a strong locality kind of like
in sentences you know words that follow
each other
they're strongly connected but there's
also a kind of larger context that
doesn't have that locality and you also
see that in driving right what's
happening in the scene
as a whole has very strong implications
on
uh you know the kind of the next step in
that sequence where
whether you're predicting what other
people are going to do whether you're
making your own decisions
or whether in the simulator you're
building generative models of
you know humans walking cyclists riding
another car is driving oh that's that's
all really fascinating like how
it's fascinating to think that uh
transformer models and all this
all the breakthroughs in language and
nlp that might be applicable to like
driving at the higher level
at the behavioral level that's kind of
fascinating um let me ask about pesky
little creatures called pedestrians and
cyclists
they seem so humans are a problem if we
can get rid of them i would
um but unfortunately they're all sort of
a source of joy and love and beauty so
let's keep them around they're also our
customers oh for your perspective yes
yes
for sure there's some money very good
um but
uh i don't even know where i was going
oh yes pedestrians and cyclists
uh i you know
they're a fascinating injection into the
system of
uh uncertainty of um
of like a game theoretic dance of what
to do and and also
they have perceptions of their own
and they can tweet about your product so
you don't want to run them over
from that perspective uh i mean i don't
know i'm joking a lot but that
i think in seriousness like you know
pedestrians are complicated um
uh computer vision problem a complicated
behavioral problem
is there something interesting you could
say about what you've learned
from a machine learning perspective from
also an autonomous vehicle
and a product perspective about just
interacting with the humans in this
world
yeah just you know stayed on the record
we care deeply about the safety of
pedestrians
you know even the ones that don't have
twitter accounts um
thank you all right but you know not me
but yes i i'm glad i'm glad somebody
does okay uh but you know
in all seriousness safety of
uh vulnerable road users pedestrians or
cyclists is one of our highest
priorities
we do a tremendous amount of testing
and validation and put a very
significant emphasis
on you know the capabilities of our
systems that have to do with safety
around those unprotected vulnerable road
users
um you know cars just you know discussed
earlier in phoenix we have completely
empty cars completely driverless cars
you know
driving in this very large area and you
know some
people use them to you know go to school
so they'll drive through school zones
right
kids are kind of the very special class
of those vulnerable user road users
right you want to be
super super safe and super super
cautious around those so we take it very
very very seriously
um and you know what does it take uh to
uh be good at it uh you know
an incredible amount of uh
performance across your whole stack
you know starts with hardware and again
you want to use all
sensing modalities available to you
imagine driving on a residential road at
night
and kind of making a turn and you don't
have you know headlights covering some
part of the space and like you know
a kid might run out and you know
lighters are amazing at that
they see just as well in complete
darkness as they do during the day right
so just again it gives you that extra
uh uh you know margin
in terms of your capability and
performance and safety and quality
and in fact we oftentimes uh in these
kinds of situations we have our system
detect
something in some cases even earlier
than our trained operators
in the car might do especially in
conditions like you know
very dark nights um so starts with
sensing
then you know perception has to be
incredibly good and you have to be very
very good at
kind of detecting uh pedestrians
uh in all kinds of situations and all
kinds of environments including
people in weird poses uh people kind of
running around
and you know being partially occluded um
so you know that that's stop number one
then you have to have in
very high accuracy and very low latency
in terms of your reactions
to you know what you know these uh
actors might do right and we've put a
tremendous amount of engineering and
tremendous amount of validation in to
make sure our system performs
uh and you know oftentimes it does
require a very strong reaction
to do the same thing and we actually see
a lot of cases like that that's the long
tail of really
rare you know really uh kind of crazy
events
that contribute to the safety
around pedestrians like one one example
that comes to mind that we actually
happened uh in phoenix where we were uh
driving uh along and i think it was a 45
mile per hour road so in pretty high
speed traffic
and there was a sidewalk next to it and
there was a cyclist
on the sidewalk and as
uh we were in the right lane and right
next to the site so it was a multi-lane
road
so as we got close to the cyclist on the
sidewalk
uh it was a woman and she tripped and
fell just you know fell right into the
path of our vehicle
right um and our you know cart
uh uh you know this was actually with a
test driver our test drivers uh
uh did exactly the right thing uh they
kind of reacted and came to stop it
requires both very strong steering
and uh you know strong application of
the brake uh and then we simulated what
our system would have done in that
situation and it did
exactly the same thing it uh and that
that speaks to
all of those components of really good
uh state estimation and tracking
and like imagine you know a person on a
bike and they're falling over
and they're doing that right in front of
you right so you have to be real like
things are changing the appearance of
that whole
thing is changing right and the person
goes one way they're falling on the road
they're you know
being flat on the ground in front of you
you know the the bike goes flying the
other
direction like the two objects that used
to be one they're now you know
uh are splitting apart and the car has
to like detect all of that uh like
milliseconds matter
and it doesn't it's not good enough to
just break you have to like steer
and break and there's traffic around you
so like it all has to come together
and it was really great uh to see in
this case and other cases like that
that we're actually seeing in the wild
that our system is you know performing
exactly the way uh that we would have
liked and is able to you know avoid
uh collisions like this such an exciting
space for robotics
like in that split second to make
decisions of life and death
i don't know if the stakes are high in
the sense but it's also beautiful
that um um for somebody who loves
artificial intelligence
the possibility that an ai system might
be able to save a human life
that's kind of exciting as a as a
problem like to wake up
you get it's terrifying probably from
energy for an engineer to wake up
and to think about but it's also
exciting because it's like
it's it's in your hands let me try to
ask a question that's often brought up
about autonomous vehicles
and it might be fun to see if you have
anything anything interesting to say
which is about the trolley problem
so uh a trolley problem is a interesting
philosophical construct
of uh that highlights and there's many
others like it
of the difficult ethical decisions that
uh we humans
have before us in this complicated world
uh so
the specifically is the choice between
if you were forced to choose uh to kill
a group x of people versus a good why of
people like
one person if you didn't if you did
nothing you would kill one person but if
you would kill five people and if you
decide to swerve out of the way you
would only kill one person
do you do nothing or you choose to do
something you can construct all kinds of
sort of
ethical experiments of this kind that
um i i think at least on a positive note
inspire you to think about like
introspect
what are the the physics
of our morality and there's usually not
good
answers there i think it people love it
because it's just an exciting
thing to think about i think people who
build autonomous vehicles usually roll
their eyes
because uh this is not this
one as constructed this like literally
never comes up
in reality you never have to choose
between killing
one like one of two groups of people
but i wonder if you can speak to
is there some something interesting
to use an engineer of autonomous
vehicles that's within the trolley
problem
or maybe more generally are there
difficult
ethical decisions that you find that the
algorithm must make on the specific
version of the trial problem which one
would you do
if you're driving the question itself is
a profound
question because we humans ourselves
cannot answer and that's the very
point uh i guess i would kill both
um yeah humans i think you're exactly
right and that you know humans are not
particularly good
i think they kind of phrased as a like
what would a computer do but like
humans you know are not very good and i
actually often times
i think that you know freezing and kind
of not doing anything because
like you've taken a few extra
milliseconds to just process and then
you end up
like doing the worst of the possible
outcomes right so um i
i do think that as you've pointed out it
can be
a bit of a distraction and it can be a
bit of a kind of red herring i think
it's an interesting philosophy
discussion in the realm of uh philosophy
um right but in terms of what you know
how that affects the
actual engineering and deployment of
self-driving vehicles
i um it's not how you go about building
a system right we have talked about how
you engineer a system
how you go about evaluating the
different components
and you know the safety of the entire
thing how do you kind of inject
the you know various model based safety
based arguments and you're like yes you
reason it parts the system
you know you reason about the
probability of a collision the severity
of that collision right
and that is incorporated and there's you
know you have to properly reason about
uncertainty that flows through the
system right so
you know those uh um you know
factors definitely play a role in how
the cars don't behave but they have to
be more
of like the immersion behavior and what
you see like you're absolutely right
that these you know
clear uh theoretical problems that they
you know you you don't require that
in system and really kind of being back
to our previous discussion of like what
what you know what
what you know which one do you choose
well you know oftentimes
like you made a mistake earlier like you
shouldn't be in that situation
uh in the first place right and in
reality the system comes up
if you build a very good safe and
capable driver
you have enough uh you know clues uh in
the environment that
you drive defensively so you don't put
yourself in that situation right
and again you know it has you know this
if you go back to that analogy of you
know precision and recall like okay you
can make a
very hard trade-off of the i1 but like
neither answer is really good
but what instead you focus on is kind of
moving the whole curve up
and then you focus on building the right
capability and the right defensive
driving so that you know you don't put
yourself in a situation like this
i don't know if you have a good answer
for this but people love it when i ask
this question
about books um are there books
in um in your life that you've enjoyed
philosophical fiction
technical that had a big impact on you
as an engineer or as a human being
you know everything from science fiction
to a favorite textbook
is there three books that stand out that
you can think of
uh three books so i would uh you know
that impacted me
um i would say uh
this one is you probably know it well
um but and not generally well known
i i think in the u.s or kind of
internationally the master
and margarita it's uh one of
actually my favorite uh books um
it is you know by a russian it's a novel
by
russian author uh mikhail bulgakov and
it's just it's it's a great book and
it's one of those books that you can
like
reread your entire life and it's very
accessible you can read it as a kid
and like it's it you know it's that the
plot is interesting it's you know the
the devil
you know visiting the soviet union and
yeah but it
it like you read it reread it at
different stages of your life and
you yeah you enjoy it for different very
different reasons
and you keep finding like deeper and
deeper meaning uh and you know kind of
affected
you know hadn't definitely had an like
imprint on me
mostly from the probably kind of the
cultural stylistic
uh aspect like it makes you one of those
books that you know is good and makes
you think but also
has like this really you know silly
quirky dark sense of you know humor hey
casper is the russian so that's more
than maybe perhaps many other books
on that like slight no just out of
curiosity one of the saddest things is
i've read that book
in english did you by chance read it in
english or in russian
uh in russian only in russian uh and i
actually that that is a question i had
uh uh kind of pose to myself every once
in a while like i wonder how well it
translates
if it translates at all and there's the
language aspect of it and then there's
the cultural aspect so i
and actually i'm not sure if you know
either of those would so work well in
english
now i forget their names but so when the
covid lists a little bit i'm traveling
to paris
uh for for several reasons one it's just
i've never been to paris i want to go to
paris but
there's a the most famous translators
of uh destielski tolstoy of most of
russian literature
live there there's a couple they're
famous a man and a woman
and i'm going to sort of have a series
of conversations with them and in
preparation for that i'm
starting to read dusty sk in russian so
i'm really embarrassed to say that i
read
this everything i've read russian
literature of like
serious depth has been in english even
though
i can also read i mean obviously in
russian but
for some reason it seemed
uh in the optimization of life it seemed
the improper decision to do to read in
russian
like you know like i don't need to opt i
need to think in english not in russian
but now i'm changing my mind on that and
so the question of how well it
translates it's a really fundamental one
like it even with dostoyevsky
so from what i understand this death can
translate easier
uh others don't as much obviously the
poetry doesn't translate as well
i'm also the the music of a big fan of
vladimir wassotsky
he doesn't obviously translate well
people have tried
but mastermind i don't know i don't know
about that one i just know it in english
you know it's fun
fun as hell in english so uh so but it's
a curious question and i want to study
it
rigorously from both the machine
learning aspect and also because i
want to do a couple of interviews in
russia
that i'm still unsure
of how to properly conduct an interview
across a language barrier it's a
fascinating question
that ultimately communicates to an
american audience there's a few
russian people that i think are truly
special human beings
and i feel
like i sometimes encounter this with
some incredible scientists and maybe
you encounter this as well at some point
in your life that it feels like because
of the language barrier their ideas are
lost to history
it's a sad thing i think about like
chinese scientists or even authors that
like
that we don't in english-speaking world
don't get to appreciate
some like the depth of the culture
because it's lost in translation
and i feel like i would love to show
that to the world
like i'm i'm just some idiot but because
i have this
like at least some semblance of skill in
speaking russian
i feel like and i know how to record
stuff on a video camera
i feel like i want to catch like gregory
pearlman who's a mathematician i'm not
sure if you're familiar with him
yeah i want to talk to him like he's a
fascinating mind and to bring him to a
wider audience in english speaking
it'll be fascinating but that requires
to be rigorous about this question
of how well uh bulgakov translates
i mean i i know it's a it's a silly
concept but it's a fundamental one
because how do you translate and that's
that's the thing that uh google
translate is also facing
yeah uh as a as a more machine learning
problem but i
i wonder is a more bigger problem for ai
how do we capture the magic that's there
in the language
i i think that's a really interesting
really
challenging problem i if you do read it
master and margarita
in uh english uh sorry in russian i'd be
curious
get your uh opinion and i think part of
it is language but part of it's just you
know centuries of culture
that the cultures are different so it's
hard to
connect that but uh okay so that was my
first one right you had to know tomorrow
um the second one i would probably pick
the science fiction by the stragoski
brothers
uh you know it's up there with you know
isaac asimov and you know ray bradbury
uh and you know company uh the straguski
brothers kind of appealed more to me i
think more it made more of an impression
on me
uh growing up um can you i
apologize if i'm showing my complete
ignorance i'm so weak on sci-fi
which what what are they right oh um
uh roadside picnic um
[Music]
uh hard to be a god uh
uh beetle in an ant hill
uh monday starts on saturday like it's
it's not just science fiction it's also
like has very interesting you know
interpersonal
and societal questions and some of the
language is just
completely hilarious
that's the one that's right oh
interesting monday starts on saturday so
i need to read
okay oh boy you put that in the category
of science fiction
uh that one is i mean this was more of a
silly
you know humorous uh work i mean there
is kind of
profound too right science fiction right
is about you know this this research
institute and like
this it it has deep parallels to like
serious
research but the the setting of course
is that they're working on you know
magic
right and there's a lot of stuff so i i
i i that that's their style right
they go and you know other books are
very different right
you know hard to be a god right it's
about kind of this higher society being
injected into this primitive world and
how they operate there
like some of the very deep ethical you
know questions there
right and like they've got this spectrum
some as you know more about kind of more
uh adventure style but like i i enjoy
all of their books there's
probably a couple actually one i think
that they consider their most important
work
i think it's the snail on an on a a hill
i don't know exactly how sure how it
translates i tried reading a couple of
times i still don't get it
but everything else i fully enjoyed uh
and like for one of my birthdays as a
kid i got like their entire collection
like occupied a giant shelf in my room
and then like over the holidays i just
like
you know my parents couldn't drag me out
of the room and i read the whole thing
cover to cover
and it it uh i really enjoyed it
uh and that's it one more i thought for
the third one i you know maybe a little
bit darker
um uh but you know comes to mind is
orwell's 1984. uh
and i you know you asked what made an
impression on me and books that people
should read that one i think falls in
the category of
both now you know definitely it's one of
those books that you read and
you just kind of you know put it down
and you stare in space for a while
uh yeah you know that that that kind of
work uh i i think there's
you know lessons there people uh should
not ignore
and you know nowadays with like
everything that's happening in the world
i
i can't help it but you know have my
mind jump to some
you know parallels uh with what orwell
described and like there's this whole
you know concept of double think and
ignoring logic
and you know holding completely
contradictory opinions in your mind and
not have that not bother you and you
know sticking to the party line
yeah uh at all costs like you know
there's there's there's something
there if anything 2020 has taught me
and i'm a huge fan of animal farm which
is a kind of friendly
as a friend of 1984 by orwell
it's kind of another thought experiment
of how our society
may go in directions that we wouldn't
like it to go
but if if anything that's been
[Music]
kind of heartbreaking to an optimist
about 2020
is that that society is kind of fragile
like we have this this is a special
little experiment we have going on
and not it's not unbreakable like
we should be careful to like preserve
whatever special thing we have going on
i mean i think 1984 in these books brave
new world
they they're helpful in thinking like
stuff can go wrong in non-obvious ways
and it's like it's up to us to preserve
it
and it's like it's a responsibility it's
been weighing heavy on me because like
for some reason like uh
more than my mom follows me on twitter
and i feel like
i have i have like now somehow a
responsibility to
um to this world and it
dawned on me that like me and millions
of others
are like the little ants that maintain
this little colony
right so we have a responsibility not to
be uh i don't know what the right
analogy is but
i'll put a flamethrower to the place we
want to
not do that and there's interesting
complicated ways of doing that as 1984
shows
it could be through bureaucracy it could
be through incompetence it could be
through misinformation
it could be through division and
toxicity uh
i'm a huge believer in like that love
will be
the somehow the solution so
uh loving robots yeah
i i think you're exactly right
unfortunately i think it's uh less of a
flamethrower
type of next i think it's more of a in
many cases can be more of a slow boil
and that that's the danger let me ask uh
it's a fun thing to make a world-class
roboticist engineer
and leader uncomfortable with a
ridiculous question about
life what is the meaning of life
at dmitry from a robotics and a human
perspective
you only have a couple minutes or one
minute to answer so
i don't know if that makes it more
difficult or easier actually yeah
you know they're very tempted to
uh quote uh one of the stories
stories by uh uh isaac asimov actually
um actually titled appropriately titled
the last question uh
short story where you know the plot is
that you know humans build this super
computer you know this this this ai
intelligence and you know once
it's get power gets powerful enough they
pose this question to it you know
um how can the entropy in the universe
be reduced all right so your computer
replies
and as of yet insufficient information
to give a meaningful answer
right and then you know thousands of
years go by and they keep posing the
same question the computer you know
it gets more and more powerful and keeps
giving the same answer yeah as of yet
insufficient information to give a
meaningful answer or something along
those lines
right and then you know keeps you know
happening and
happening you fast forward like millions
of years into the future and you know
billions of years and like at some point
it's just the only entity in the
universe
it's like absorbed all humanity and all
knowledge in the universe and it like
keeps posing the same question to itself
and you know
finally it gets to the point where it is
able to answer that question but of
course at that point you know there's
you know the heat death of the universe
has occurred and that's the only entity
and there's nobody else to provide that
answer to so the only thing it can do is
to
you know answer it by demonstration so
it like you know recreates the
big bang right and resets the clock
right
but i i can try to give kind of a
a different version of the answer you
know maybe
uh not on the behalf of all humanity i
think that that might be a little
presumptuous for me to speak about the
meaning of life on
the behalf of all humans uh but at least
you know personally
uh it changes right i think if you think
about kind of what
uh gives uh
you know you and your life meaning and
purpose and kind of what drives you
um it seems to
change over time right and the the the
lifespan
of you know your existence uh you know
when
just when you just enter this this world
right it's all about kind of new
experiences
and you get like new smells new sounds
new emotions right
and like that's what's driving you right
you're experiencing
new amazing things right and that that's
magical right
that's pretty pretty pretty pretty
awesome right that gives you kind of
meaning
then you get a little bit older you
start more intentionally
uh learning about things right i guess
actually before you start intentionally
learning probably fun
fun is a thing that gives you kind of
meaning and purpose and purpose and the
thing you optimize for right
and like fun is good uh then you get you
know start learning and i guess that
this this
joy of comprehension and discovery
is another thing that you know gives you
meaning and purpose and drives you right
then
you know you learn enough stuff and
it you want to give some of it back
right and so impact and contributions
back to you know technology or society
uh uh people uh you know local or more
globally yeah
is becomes a new thing that you know
drives a lot of kind of your behavior
and something that gives you purpose and
that you derive you know positive
feedback from right
you know then you go and so on and so
forth you go through various stages of
life
if you have if you have kids
like that definitely changes your
perspective on things you know i have
three
that definitely flips some bits in your
head in terms of
you know what you care about and what
you optimize for and you know what
matters what doesn't matter right
so you know and so on and so forth right
and i i i
it seems to me that you know it's all of
those things and as
kind of you go through life um
you know you want these to be additive
right
new experiences fun learning
impact like you want you want to you
know be accumulating other you know i
don't want to you know
stop having fun or experiencing new
things and i think it's important that
it just kind of becomes uh additive as
opposed to a replacement or subtraction
but you know views probably as far as i
got but you know ask me in a few years i
might have one or two more to add to the
list
and before you know it time is up just
like it is for this conversation
uh but hopefully it was a fun ride it
was a huge honor to meet you as
you know i've been a fan of yours and a
fan of
google self-driving car and waymo for a
long time
i can't wait i mean it's one of the most
exciting if we look back in the 21st
century i truly believe it'll be one of
the most exciting things we
descendants of apes have created on this
earth so
i'm a huge fan and i can't wait to see
what you do
next thanks so much for talking today
thanks thanks for having me and it's a
also a huge fan doesn't work honestly
and uh i really enjoyed it thank you
thanks for listening to this
conversation with dimitri dalgov
and thank you to our sponsors trial labs
a company
that helps businesses apply machine
learning to solve real world problems
blinkist an app i use for reading
through summaries of books
better help online therapy with a
licensed professional
and cash app the app i use to send money
to friends
please check out these sponsors in the
description to get a discount
and to support this podcast if you
enjoyed this thing
subscribe on youtube review 5000 upper
podcast
follow on spotify support on patreon or
connect with me on twitter at lex
friedman
and now let me leave you with some words
from isaac
asimov science can amuse and fascinate
us all
but it is engineering that changes the
world
thank you for listening and hope to see
you next time
you