Kind: captions
Language: en
the following is a conversation with
Sergey Levine a professor at Berkeley
and a world-class researcher in deep
learning reinforcement learning robotics
and computer vision including the
development of algorithms for end-to-end
training of neural network policies that
combine perception and control scalable
algorithms for inverse reinforcement
learning and in general deep r.l
algorithms quick summary of the ads to
sponsors cash app and expressvpn please
consider supporting the podcast by
downloading cash app and using collects
pot cast and signing up at expressvpn
comm / flex pod click the links buy the
stuff it's the best way to support this
podcast and in general the journey I'm
on if you enjoy this thing subscribe on
YouTube review it with five stars an
apple podcast follow on Spotify
supported on patreon or connect with me
on Twitter at lex
friedman as usual i'll do a few minutes
of as now and never any ads in the
middle that can break the flow of the
conversation this show is presented by
cash app the number one finance app in
the App Store when you get it used colex
podcast cash app lets you send money to
friends buy bitcoin and invest in the
stock market with as little as one
dollar since cash app does fractional
share trading let me mention that the
order execution algorithm that works
behind the scenes to create the
abstraction of the fractional orders is
an algorithmic marvel so big props the
cash app engineers are taking a step up
to the next layer of abstraction over
the stock market making trading more
accessible for new investors and
diversification much easier so again if
you get cash out from the App Store
Google Play and use the code lex podcast
you get $10 and cash up will also donate
$10 the first an organization that is
helping to advanced robotics and stem
education for young people around the
world this show
is also sponsored by expressvpn get it
at expressvpn comm / Lex pod to support
this podcast and to get an extra three
months free on a one-year package I've
been using expressvpn for many years I
love it
I think expressvpn is the best VPN out
there they told me to say it but it
happens to be true my humble opinion it
doesn't lock your data it's crazy fast
and as easy to use literally just one
big power on button again it's probably
obvious to you but I should say it again
it's really important that they don't
log your data
it works on Linux and every other
operating system but Linux of course is
the best operating system shout out to
my favorite flavor
Ubuntu mottai 2004 once again get it at
expressvpn calm / relax pod to support
this podcast and to get an extra three
months free on a one-year package and
now here's my conversation sergey
Lavigne what's the difference between a
state-of-the-art human such as you and I
well I don't know if we qualify Stata
they're humans but a state-of-the-art
human and a state-of-the-art robot it's
a very interesting question
robot capability is it's kind of a I
think it's a very tricky thing to to
understand because there are some things
that are difficult that we wouldn't
think are difficult and some things that
are easy that we wouldn't think ever you
see and there's also a really big gap
between capabilities of robots in terms
of hardware and their physical
capability and capabilities of robots in
terms of what they can do autonomously
there is a little video that I think
robotics researchers really like to show
a special Robotics learning researchers
like myself from 2004 from Stanford
which demonstrates a prototype robot
called the PR one and the PR one was a
robot that was designed as a home
assistance robot and there's this
beautiful video showing the pr1 tidying
up a living room putting away toys and
at the end bringing a beer to the person
sitting on the couch which looks really
amazing and then the punch line is that
this
is entirely controlled by person yes so
you can so that in some ways the gap
between a state-of-the-art human
state-of-the-art robot if the robot has
a human brain is actually not that large
now obviously like human bodies are
sophisticated and very robust and
resilient in many ways but on the whole
if we're willing to like spend a bit of
money and do a bit of engineering we can
kind of close the hardware gap almost
but the intelligence gap that one is
very wide and when you say hardware you
you're referring to the physical sort of
the actuators the actual body the robot
is opposed to the hardware on which the
cognition the nervous the hardware of
the nervous system yes exactly I'm
referring to the body rather than the
mind so what so that means that the kind
of the work is cut out for us like while
we can still make the body better we
kind of know that the big bottleneck
right now is really the mind and how big
is that gap how big is the how big is
the difference in your in your sense of
ability to learn a bit ability to reason
ability to perceive the world between
humans and our best robots the gap is
very large and the gap becomes larger
the more unexpected events can happen in
the world so essentially the spectrum
along which you can measure the the size
of that gap is the spectrum of how open
the world is if you control everything
in the world very tightly if you put the
robot in like a factory and you tell it
where everything is and you rigidly
program its motion then it can do things
you know one might even say in a
superhuman way it can move faster it's
stronger it can lift up a car and things
like that but as soon as anything starts
to vary in the environment now it'll
trip up and if many many things vary
like they would like in your kitchen for
example then things are pretty much like
wide open now again we're gonna stick a
bit on the philosophical questions but
how much on the human side of the
cognitive abilities in your sense is
nature versus nurture so so how much of
it is product of evolution and how much
of it something we'll learn from sort of
scratch yeah well from the day were born
I'm going to read into your question as
asking about the implications of this
for AI really
by biologists I can't really like speak
authoritative also until in garnet if if
it's so if it's all about learning then
there's more hope for am so the way that
I look at this is that you know well
first of course biology is very messy
and it's if you ask the question how
does a person do something or has a
person's mind do something you come up
with a bunch of hypotheses and
oftentimes you can find support for many
different often conflicting hypotheses
one way that we can approach the
question of what the implication of this
for AI R is we can think about what's
sufficient so you know maybe a person is
from birth very very good at some things
like for example recognizing faces
there's a very strong evolutionary
pressure to do that if you can recognize
your mother's face then you're more
likely to survive and therefore people
are good at this but we can also ask
like what's what's the minimum
sufficient thing right and one of the
ways that we can study the minimal
sufficient thing is we could for example
see what people do in unusual situations
if you present them of things that
evolution couldn't have prepared them
for you know our daily lives actually do
this to us all the time we we didn't
evolve to deal with you know automobiles
and spaceflight and whatever so they're
all these situations that we can find
ourselves in and we do very well they're
like I can give you a joystick to
control a robotic arm which you've never
used before and you might be pretty bad
for the first couple of seconds but if I
tell you like your life depends on using
this robotic arm to like open this door
you'll probably manage it even though
you've never seen this device before you
even even ever used the joys to control
us and you'll kind of muddle through it
and that's not your evolved natural
ability that's your fear flexibility
your your adaptability and that's
exactly why our current robotic systems
really kind of fall flat but I wonder
how much general almost what we think of
as common sense
pre-trained models underneath all that
so that ability to adapt to a joystick
is requires you to have a kind of you
know I'm human so it's hard for me to
introspect all the knowledge I have
about the world but it seems like there
might be an iceberg underneath of the
amount of knowledge
you actually bring to the table now
that's kind of the open question there's
absolutely an iceberg of knowledge that
we bring to the table but I think it's
very likely that iceberg of knowledge is
actually built up over our lifetimes
because we have you know we have a lot
of prior experience to draw on and it
kind of makes sense that the right way
for us to you know to optimize our
efficiency our evolutionary fitness and
so on is to utilize all that experience
to build up the best iceberg we can get
and that's actually one you know well
that sounds an awful lot like what
machine learning actually does I think
that for modern machine learning it's
actually a really big challenge to take
this unstructured massive experience and
distill out something that looks like a
common sense understanding of the world
and perhaps part of that isn't it's not
because something about machine learning
itself is is broken or hard but because
we've been a little too rigid in
subscribing to a very supervised very
rigid notion of learning you know kind
of the input-output excess goes go to
why sort of model and maybe what we
really need to to do is to view the
world more as like a massive experience
that is not necessarily providing any
rigid supervision but sort of providing
many many instances of things that could
be and then you take that and you
distill it into some sort of common
sense understanding I see what you're
you're painting an optimistic beautiful
picture especially from the robotics
perspective because that means we just
need to invest in both better learning
algorithms figure out how we can get
access to more and more data for those
learning L goes to extract signal from
and then accumulate that iceberg of
knowledge it's a beautiful picture it's
a hopeful one I think it's potentially a
little bit more than just that and this
is this is where we perhaps reach the
limits of our current understanding but
one thing that I think that the research
community hasn't really resolved in a
satisfactory way is how much it matters
where that experience comes from like
you know do just like download
everything on the intranet and cram it
into essentially the 21st century analog
of the giant language model and then see
what happens or does it actually matter
whether your machine
experiences the world or in a sense that
actually attempts things observes the
outcome of its actions and kind of
augments the experience that way that it
chooses which parts of the world it gets
to interact with and observe and learn
from right it may be that the world is
so complex that simply obtaining a large
mass of sort of iid samples of the world
is is a very difficult way to go but if
you are actually interacting with the
world and essentially performing this
sort of hard- mining by attempting what
you think might work observing the
sometimes happy and sometimes sad
outcomes of that and augmenting your
understanding using that experience and
you're just doing this continually for
many years maybe that sort of data in
some sense is actually much more
favourable to obtaining a common sense
understanding well one reason we might
think that this is true is that you know
the what we associate with common sense
or lack of common sense is often
characterized by the ability to reason
about kind of counterfactual questions
like you know I if I were to you know
here I'm this bottle of water sitting on
the table everything is fine far knock
it over which I'm not going to do but if
I were to do that what would happen and
I know that nothing good would happen
from that but if I have a bad
understanding of the world I might think
that that's a good way for me to like
you know gain more utility if I actually
go about the daily life doing the things
that my current understanding of the
world suggests will give me high utility
in some ways I'll get exactly the the
right supervision to tell me not to do
those those bad things and to keep doing
the good things so there's a spectrum
between iid random walk through the
space of data and then there's and what
we humans do or I don't even know if we
do it through optimal but there might be
beyond what so this open question that
you raised where do you think systems
intelligent systems that would be able
to deal with this world fall can we do
pretty well by reading all of Wikipedia
sort of randomly sampling it like
language models do or do we have to be
exceptionally selective and intelligent
about which aspects of the wall we eat
chocolate so I think this is first an
open scientific problem and I don't have
like a clear answer but I can speculate
a little bit and what I would speculate
is that you don't need to be super super
careful I think it's less about like
being careful to avoid the useless stuff
and more about making sure that you hit
on the really important stuff so perhaps
it's okay if you spend part of your day
just you know guided by your curiosity
visiting interesting regions of the of
your state space but it's important for
you to you know every once in a while
make sure that you really try out the
solutions that your current model of the
world suggests might be effective and
observe whether those solutions are
working as you expect or not and perhaps
some of that is really essential to have
kind of a perpetual improvement loop
like this perpetual improvement loop is
really like but that's really the key
the key that's going to potentially
distinguish the best current methods
from the best methods of tomorrow in a
sense how important do you think is
exploration or total out-of-the-box
thinking exploration in this space is
you jump to totally different domain so
you kind of mentioned there's an
optimization problem you kind of kind of
explore the specifics of a particular
strategy whatever the thing you're
trying to solve how important is it to
explore totally outside of the
strategies they've been working for you
so far what's your intuition there yeah
I think it's a very problem dependent
kind of question and I think that that's
actually you know in some ways that
question gets at one of the big
differences between sort of the classic
formulation of a reinforcement learning
problem and some of the sort of more
open-ended reformulations of that
problem that have been explored in
recent years so classically
reinforcement learning is framed as a
problem of maximizing utility like any
kind of rational AI agent and then
anything you do is in service to
maximizing that utility but a very
interesting kind of way to look at
I'm not necessary saying that's the best
way to look at it but an interesting
alternative way to look at these
problems as as something where you first
get to explore the world
however you please and then afterwards
you will be tasked with doing something
and that might suggest to somewhat
different solutions so if you don't know
what you're going to be tasked with
doing and you just want to prepare
yourself optimally for whatever you're
uncertain future holds maybe then you
will choose to attain some sort of
coverage build up sort of an arsenal of
cognitive tools if you will such that
later on when someone tells you now your
job is to fetch the coffee for me you'll
be well prepared to undertake that task
and that you see that as the modern
formulation of the reinforcement
learning problem as the kind of the more
multi task the general intelligence kind
of formulation I think that's one
possible vision of where things might be
headed I don't think that's by any means
the mainstream or standard way of doing
things and it's not like if I had to but
I like it it's a beautiful vision so
maybe you actually take a step back what
is the goal of robotics what's the
general problem of robotics of trying to
solve you actually kind of painted two
pictures here one of the narrow one is
the general what in your view is the big
problem of robotics again ridiculously
philosophical questions I think that you
know maybe there are two ways I can
answer this question one is there's a
very pragmatic problem which was like
what would make robots what would sort
of maximize the usefulness of robots and
there the answer might be something like
a system where a system that can perform
whatever task a human user sets for it
you know within the physical constraints
of course if you tell it to teleport to
another planet but probably can't do
that but if you if you ask it to do
something that's within its physical
capability then potentially with a
little bit of additional training or a
little bit of additional trial and error
it ought to be able to figure it out in
much the same way as like a human tele
operator ought to figure out how to
drive the robot to do that that's kind
of a very pragmatic view of what it
would take to kind of solve the the
robotics problem if you will but I think
that there is a second answer and that
answer that the answer is a lot closer
to why I want to work on on robotics
which is that I think it's it's less
about what it would take to do a really
good job
in the world of robotics but more the
other way around what robotics can bring
to the table
to help us understand artificial
intelligence so your dream fundamentally
is to understand intelligence yes I
think that's the dream for many people
who actually work in this space I think
that there is there's something very
pragmatic and very useful about studying
robotics but I do think that a lot of
people that go into this field actually
you know the things that they draw
inspiration from are the potential for
robots to like help us learn about
intelligence and about ourselves that's
that's fascinating that robotics is
basically the space by which you can get
closer to understanding the fundamentals
of artificial intelligence so what is it
about robotics that's different from
some of the other approaches so if we
look at some of the early breakthroughs
in deep learning or in the computer
vision space and the natural language
processing there was really nice clean
benchmarks that a lot of people competed
on and thereby came out with a lot of
building ideas what's the fundamental
difference to you between computer
vision purely define an image net and
kind of the bigger robotics problem so
there are a couple of things one is that
with robotics you kind of have you kinda
have to take away many of the crutches
so you have to deal with with both the
the the particular problems of
perception control and so on but you
also have to deal with the integration
of those things and you know classically
we've always thought of the integration
as kind of a separate problem so a class
a kind of modular engineering approaches
that we solve individual subproblems
then wire them together and then the
whole thing works and one of the things
that we've been seeing over the last
couple of decades is that well maybe
studying the thing as a whole might lead
to just like very different solutions
now if we were to study the parts and
wire them together so the integrative
nature of robotics research helps us see
you know the different perspectives on
the problem another part of the answer
is that with robotics it it casts a
certain paradox into very clever relief
so this is sometimes referred to as more
of expert on the idea that in artificial
intelligence things that are very
hard for people can be very easy for
machines and vice versa things that are
very easy for people can be very hard
for machines so you know integral and
differential calculus is pretty
difficult to learn for people but if you
program a computer do it it can derive
derivatives and integrals for you all
day long without any trouble
whereas some things like you know
drinking from a cup of water very easy
for a person to do very hard for a robot
to deal with and sometimes when we see
such blatant discrepancies that give us
a really strong hint that we're missing
something important so if we really try
to zero in on those discrepancies we
might find that little bit that we're
missing and it's not that we need to
make machines better or worse at math
and better at drinking water but just
that by studying those discrepancies you
might find some new insight so that that
could be that could be in any space it
doesn't have to be robotics but you're
saying yeah I get it's kind of
interesting that robotics seems to have
a lot of those discrepancies so the the
the Hans more of a paradox is probably
referring to the space of the the
physical interaction I think you said
object manipulation walking all the kind
of stuff we do in the physical world
that well how do you make sense if you
were to try to disentangle the the
Marwick paradox like why is there such a
gap in our intuition about it why do you
think manipulating objects is so hard
from everything you've learned from
applying reinforcement learning in this
space yeah I think that one reason is
maybe that for many of the problems for
many of the other problems that we've
studied in AI and computer science and
so on the notion of input/output and
supervision is much much cleaner so
computer vision for example deals with
very complex inputs but it's
comparatively a bit easier at least up
to some level of abstraction to cast it
as a very tightly supervised problem
it's comparatively much much harder to
cast robotic manipulation as a very
tightly supervised problem you can do it
it just doesn't
work all that well so you could say that
well maybe we get a label data set where
we know exactly which motor commands to
send and then we train on that but for
various reasons that's not actually like
such a great solution and it also
doesn't seem to be even remotely similar
to how people and animals learn to do
things because we're not told by like
our parents here is how you fire your
muscles in order to walk we you know we
do get some guidance but the really
low-level detailed stuff we figure out
most of them our own and that's what you
mean by tightly coupled that every
single little sub action gets a
supervised signal of whether it's a good
one or not right so so while in computer
vision you could sort of imagine up to a
level of abstraction that maybe you know
somebody told you this is a car and this
is a cat and this is a dog in motor
control it's very clear that that was
not the case if we look I said of the
sub spaces of Robotics that again as you
said robotics integrates all of them
together and we'll get to see how this
beautiful mess into place but so there's
nevertheless still perception so it's
the the computer vision problem
broadly speaking understanding the
environment then there's also maybe you
can correct me on this kind of
categorization of the space then there's
prediction in trying to anticipate what
things are going to do into the future
in order for you to be able to act in
that world and then there's also this
game theoretic aspect of how your
actions will change the behavior of
others in this kind of space what and
this is bigger than reinforcement
learning this is just broadly looking at
the problem of Robotics what's the
hardest problem here or is there or is
what you said true that when you start
to look at all of them together that's
an int that's a whole nother thing like
you can't even say which one
individually is harder because all of
them together you should only be looking
at them all together I think when you
look at them all together some things
actually become easier and I think
that's actually pretty important so we
had you know back in 2014 we had some
work basically our first work on end to
end
enforced learning for robotic
manipulation skills from vision which
you know at the time was something that
seemed a little inflammatory and
controversial in the robotics world but
other than the the inflammatory and
controversial part of it
the point that we were actually trying
to make in that work is that for the
particular case of combining perception
and control you could actually do better
if you treat them together then if you
try to separate them and the way that we
try to demonstrate this as we picked a
fairly simple motor control task where a
robot had to insert a little red
trapezoid into a trapezoidal hole and we
had our separated solution which
involved first detecting the hole using
a pose detector and then actuated arm to
put it in and then our intent solution
which just mapped pixels to the torques
and one of the things we observed is
that if you use the intense solution
essentially the pressure on the
perception part of the model is actually
lower like it doesn't have to figure out
exactly where the thing is in 3d space
it just needs to figure out where it is
you know distributing the errors in such
a way that the horizontal difference
matters more than the vertical
difference because vertically just
pushes it down all the way until it
can't go any further and their
perceptual errors are a lot less harmful
whereas a perpendicular to the direction
of motion perceptual errors are much
more harmful so the point is that if you
combine these two things you can trade
off errors between the components
optimally to best accomplish the task
and the components can should be weaker
while still leading to better overall
performance as a profound idea I mean in
in the space of pegs and things like
that is quite simple it almost is
tempting to overlook but that's seems to
be at least intuitively an idea that
should generalize to basically all
aspects of perception control of course
when one strengthens the other yeah and
and we you know people who have studied
sort of perceptual heuristics in humans
and animals find things like that all
the time so one one very well-known
example this is something called the
gaze heuristic which is a little trick
that you can use to intercept a flying
object so if you want to catch a ball
for instance you could try to localize
it in 3d space estimate its velocity
estimate the effect of wind resistance
solve a complex system of differential
equations in your head or you can
maintain a running speed so the object
stays in the same position as in your
field of view so if it dips a little bit
you speed up if it rises a little bit
you slow down and if you follow the
simple rule you'll actually arrive at
exactly the place where the object lands
and you'll catch it and humans use it
when they play baseball human pilots use
it when they fly airplanes to figure out
if they're about to collide with
somebody frogs use this to catch insects
and so on and so on so this is something
that actually happens in nature and I'm
sure this is just one instance of it
that we were able to identify just
because it's you know that scientists
are able to identify that goes so
prevalent with our probably many others
do you ever just who can zoom in as we
talk about robotics they have a
canonical problem sort of a simple clean
beautiful representative problem in
robotics they you think about when
you're thinking about some of these
problems we talked about robotic
manipulation to me that seems
intuitively at least the robotics
community is converging towards that as
a space that's the canonical problem if
you agree that maybe you zoom in in some
particular aspect of that problem that
you just like like if we solve that
problem perfectly it'll unlock a major
step in towards human level intelligence
I don't think I have like a really great
answer to that and I think partly the
reason I don't have a great answer kind
of has to do with the it has to do with
the fact that the difficulty is really
in the flexibility and adaptability
rather than in doing a particular thing
really really well so it's hard to just
say like oh if you can I don't know like
shuffle a deck of cards as fast as like
a Vegas right a casino dealer then
you'll you'll be very proficient it's
really the ability to quickly figure out
how to do some arbitrary new thing well
enough so like you know to move on to
the next arbitrary thing but the the
source of newness and uncertainty have
you found problems in which it's easy to
generate new noonah sness messes yeah
new types of newness yeah so
a few years ago is so if you'd asked me
this question around like 2016 maybe I
would have probably said that robotic
grasping is a really great example of
that because it's a task with great
real-world utility like you will get a
lot of money if you can do it well when
is the robotic grasping picking up any
object with a robotic hand exactly so
you'll get a lot of money if you do it
well because lots of people want to run
warehouses with robots and it's highly
non-trivial because very different
objects will require very different
grasping strategies but actually since
then people have gotten really good at
building systems to solve this problem
as to the point where I'm not actually
sure how much more progress we can make
with that as like the main guiding thing
but it's kind of interesting to see the
kind of methods that have what actually
worked well in that space because a
robotic grasping classically used to be
regarded very much as kind of an almost
like a geometry problem so you people
who have studied the history of computer
vision will find this very familiar that
it's kind of in the same way that in the
early days of computer vision people
thought of it very much it's like an
inverse graphics thing in robotic
grasping people thought of it as an
inverse physics problem essentially you
look at what's in front of you figure
out the shapes then use your best
estimate of the laws of physics to
figure out where to put your fingers on
you pick up the thing and it turns out
that what works really well for robotic
grasping instantiated in many different
recent works including our own but also
ones from many other labs is to use
learning methods with some combination
of either exhaustive simulation or like
actual real-world trial-and-error and
turns out that those things actually
work really well and then you don't have
to worry about solving geometry problems
or physics problems so what are just by
the way and the grasping what are the
difficulties that have been worked on so
one is like the materials of things
maybe occlusions and the perception side
why is it such a difficult why is
picking stuff up such a difficult
problem yeah it's a difficult problem
because the number of things that you
might have to deal with or the variety
of things that you have to deal with is
extremely large
and oftentimes things that work for one
class of objects won't work for other
class of objects so if you if you get
really good at picking up boxes and now
you have to pick up plastic bags you
know you just need to employ a very
different strategy and there are many
properties of objects that are more than
just their geometry it has to do with
you know the bits that that are easier
to pick up the bits that are hard to
pick up the bits that are more flexible
the bits that will cause the thing to
pivot and Bend and drop out of your hand
versus the bits that resulted in I
secure grasp things that are flexible
things that if you pick them up the
wrong way they'll fall upside down and
the contents will spill out so there's
all these little details that come up
but the task is still kind of can be
characterized as one task like there's a
very clear notion of you did it or you
didn't do it so in terms of spilling
things there creeps in this notion that
starts the sound and feel like common
sense reasoning do you think solving the
general problem of Robotics requires
common sense reasoning requires general
intelligence this kind of human level
capability of you know like you said be
robust and deal with uncertainty but
also be able to sort of reason and
assimilate different pieces of knowledge
that you have yeah what do you what are
your thoughts on the needs of common
sense reasoning in the space of the
general robotics problem so I'm gonna
slightly dodge that question and say
that I think I think maybe actually it's
the other way around is that studying
robotics can help us understand how to
put common sense into our AI systems one
way to think about common sense is that
and and why our current systems might
lack common sense is that common sense
is a property is an emergent property of
actually having to interact with a
particular world a particular universe
and get things done in that universe so
you might think that for instance like a
an image captioning system maybe it
looks at pictures of the world and it
types out English sentences so it kind
of it kind of deals with our world
and then you can easily construct
situations where image captioning
systems do things that defy common sense
like give it a picture of a person
wearing fur coat and we'll say it's a
teddy bear but I think what's really
happening in those settings is that the
system doesn't actually live in our
world it lives in its own world that
consists of pixels and English sentences
and doesn't actually consist of like you
know having to put on a fur coat in the
winter so you don't get cold so perhaps
the the reason for the disconnect is
that the systems that we have now is
simply inhabit a different universe and
if we build AI systems that are forced
to deal with all of the messiness and
complexity of our universe maybe they
will have to acquire our common sense to
essentially maximize their utility
whereas the systems we're building now
don't have to do that they can take some
shortcut that's fascinating
you've a couple of times already sort of
reframed the role of robotics and this
whole thing and for some reason I don't
know if my way of thinking is common but
I thought like we need to understand and
solve intelligence in order to solve
robotics and you're kind of framing it
as no robotics is one of the best ways
to just study artificial intelligence
and build sort of like robotics is like
the right space in which you get to
explore some of the fundamental learning
mechanisms fundamental sort of
multimodal multitask aggregation of
knowledge mechanisms that are required
for general intelligence this really
interesting way to think about it but
let me ask about learning can the
general sort of robotics the epitome of
the robotics problem be solved purely
through learning perhaps and to end
learning sort of learning from scratch
as opposed to injecting human expertise
and rules and heuristics and so on I
think that in terms of the spirit of the
question I I would say yes I mean I
think that in though in some ways it may
be like an overly sharp dichotomy like
you know I think that in some ways when
we build algorithms we you know at some
point a person does something like yeah
there's always a person turned on the
computer first
you know implemented tensorflow but yeah
I think that in terms of the in terms of
the point that you're getting and I do
think the answer is yes I think that I
think that we can solve many problems
that have previously required meticulous
manual engineering through automated
optimization techniques and actually one
thing I will say on this topic is I
don't think this is actually a very
radical or very new idea I think people
have have been thinking about automated
optimization techniques as a way to do
control for a very very long time and in
some ways what's changed is really more
than aim so you know today we would say
that oh my robot does machine learning
it does reinforcement learning maybe in
the 1960s you'd say oh my robot is doing
optimal control and maybe the difference
between typing out a system of
differential equations and doing
feedback linearization versus training
and neural net it's not such a large
difference it's just you know pushing
the optimization deeper and deeper into
the thing well you think that were but
with the especially deep learning that
the accumulation of experiences in data
form to form deep representations starts
to feel like knowledge is supposed to
optimal control so this feels like
there's an accumulation of knowledge to
the learning process yes yeah so I think
that is a good point that one big
difference between learning based
systems and classic optimal control
systems is that learning based systems
and principle should get better and
better
the more they do something right and I
do think that that's actually a very
very powerful difference so if you look
back at the world of expert systems is
symbolic AI and so on of using logic to
accumulate expertise human expertise
human encoded expertise but do you think
that will have a role the some points
that the you know deep learning machine
learning reinforcement learning has been
in incredible results and breaks there
wasn't just inspired thousands maybe
millions of researchers but you know
there's this less popular now but it
used to be part of the idea of symbolic
AI do you think that will have a role
I think in some ways the kind of the the
descendants of symbolic I actually
already have a role so you know this is
the the highly biased history from my
perspective you say that well initially
we thought that rational decision-making
involves logical manipulation so you
have some model the world expressed in
term in terms of logic you have some
query like what action do I take in
order to for X to be true and then you
manipulate your logical symbolic
representation to get an answer what
that turned into somewhere in the 1990s
is well instead of building kind of
predicates and statements that have true
or false values will build probablistic
systems where things have probabilities
associated and probabilities of being
true and false not turning the Bayes
nets and that provided sort of a boost
to what we're really you know still
essentially logical inference systems
just probabilistic logical inference
systems and then people said well let's
actually learn the individual
probabilities inside these models and
then people said well let's not even
specify the nodes and the models let's
just put a big neural net in there but
in many ways I see these as actually can
descendants from the same idea it's
essentially instantiating rational
decision-making by means of some
inference process and learning by means
of an optimization process so so in a
sense I would say yes that it has a
place and in many ways that place is or
you know it already holds that place
it's already in there yeah it's just by
different it looks slightly different
than there was before yeah but but at
some there are some things that that we
can think about that make this a little
bit more obvious like if I train a big
neural net model to predict what will
happen in response to my robots actions
and then I run probablistic inference
meaning I invert that model to figure
out the actions that lead to some
plausible outcome like to me that seems
like a kind of logic you have a model of
the world it just happens to be
expressed by a neural net and you are
doing some inference procedure some sort
of manipulation on that model to figure
out you know the answer to a query that
you have it's the interpretability it's
the explained ability though that seems
to be lacking more so because the nice
thing about sort of experts
systems is you can follow the reasoning
of the system that to us mere humans is
somehow compelling it it would it's just
I don't know what to make of this fact
that there's a human desire for
intelligence systems to be able to
convey in a poetic way to us why made
the decisions it did like tell a
convincing story and perhaps that's like
a silly human thing like we shouldn't
expect that of intelligent systems like
we should be super happy that there is
intelligent systems out there but if I
were to sort of psychoanalyze the
researchers at the time I would say
expert systems connected to that part
that desire for AI researchers for
systems to be explainable
I mean maybe on that topic do you have a
hope that sort of inferences source of
learning based systems will be as
explainable as the dream was with expert
systems for example I think it's a very
complicated question because I think
that in some ways the question of
explain ability is kind of very closely
tied to the question of of like
performance like you know why do you
want your system to explain itself well
so that it's so that when it screws up
you can kind of figure out why it did it
right but it's nice but in some ways
that that's a much bigger problem extra
like your system might screw up and then
it might screw up at how it explains
itself or you might have some bugs
somewhere so that it's not actually
doing what was supposed to do so you
know maybe a good way to view that
problem is really as a problem as a
bigger problem of verification and
validation of which explained abilities
sort of what one component I see I just
see differently I see explained ability
you you put it beautifully I think you
actually summarized the field of
explained ability but to me there's
another aspect of explained ability
which is like storytelling that has
nothing to do with errors or with
like the the survey it doesn't it uses
errors as as elements of its story as
opposed to a fundamental need to be
explainable when errors occur it's just
that for other intelligence systems to
be in our world we seem to want to tell
each other stories and that that's true
in the political world is true in the
academic world and that I you know
neural networks are less capable of
doing that or perhaps they're equally
capable a storytelling storytelling may
be it doesn't matter what the
fundamentals of the system are you just
need to be a good storyteller maybe one
specific story I can tell you about in
that space is actually about some work
that was done by by my former
collaborator who's now a professor at
MIT named Jacob Andreas Jacob actually
works on natural language processing but
he had this idea to do a little bit of
work in reinforcement learning and how
on how natural language can basically
structure the internals of policies
trained with RL and one of the things he
did is he set up a model that attempts
to perform some tasks that's defined by
a reward function but the model reads in
a natural language instruction so this
is a pretty common thing to do in
instruction following so you tell it
like you know go to the Red House and
then supposed to go to the Red House but
then one of the things that Jacob did is
he treated that sentence not as a
command from a person but as a
representation of the internal kind of
state of the of the of the mind of this
policy essentially so that when it was
faced with a new task what it would do
is it would basically try to think of
possible language descriptions attempt
to do them and see if they led to the
right outcome so it would kind of think
out loud like you know I'm faced with
this new task what am I gonna do let me
go to the red house now that didn't work
let me go to the Blue Room or something
let me go to the green plant and once it
got some reward it would say oh go to
the green plant that's what's working
I'm gonna go to the green plant and then
you could look at the string that it
came up with and that was a description
of how it thought it should solve the
problem so you could do you could
basically incorporate language as
internal state and you can start getting
some handle on these kinds of things and
then what I was kind of trying to get to
is that also if you add to the reward
function
the convincing nough story hmm so I have
another reward signal of like people who
review that story how much they like it
I says that you you know and initially
that could be a hyper parameter or sort
of hard-coded heuristic type of thing
but it's an interesting notion of the
convincing 'no story becoming part of
the reward function the objective
function of the explained ability it's
in the world of sort of twitter and fake
news that might be a scary notion that
the the nature of truth may not be as
important as the convincing 'no some the
how convinced you are in telling the
story around the facts well let me ask
the the basic question you're one of the
world-class researchers in reinforcement
learning deeper and forceful learning
certainly in the robotic space
what is reinforcement learning i think
that reinforcement learning refers to
today is really just the kind of the
modern incarnation of learning based
control so classically reinforcement
learning has a much more narrow
definition which is that it's you know
literally learning from reinforcement
like the thing does something and then
it gets a reward or punishment but
really i think the way the term is used
today is it's used for for more broadly
to learning based control so some kind
of system that's supposed to be
controlling something and it uses data
to get better and what is control means
is action is the fundamental element
yeah it means making rational decisions
now and rational decisions are decisions
that maximize a measure of utility and
sequentially see many decisions time and
time and time again now like so it's
easier to see that kind of idea in the
space of maybe games in the space of
robotics
do you see is bigger than that is it
applicable like word were the limits of
the applicability of reinforcement
learning yeah so rational
decision-making is essentially the the
encapsulation of the AI problems you
didn't through a particular lens so any
problem that we would want a machine to
do intelligent machine can likely be
represented as a decision-making problem
you're classifying images is a
decision-making problem although not a
sequential one typically you know
controlling a chemical plant as a
decision-making problem deciding what
videos to recommend on YouTube is a
decision-making problem and one of the
really appealing things about
reinforcement learning is if it does
encapsulate the range of all these
decision-making problems perhaps working
on reinforcement learning is you know
one of the ways to reach a very broad
swath of AI problems but what what do
you use the fundament the difference
between reinforcement learning and maybe
supervised machine learning so the
reinforcement learning can be viewed as
a generalization of supervised machine
learning you can certainly cast
supervised learning as a reinforcement
learning problem you can just say your
loss function is the negative of your
reward but you have stronger assumptions
you have the assumption that someone
actually told you what the correct
answer was that your data was iid and so
on so you could view reinforcement
learning is essentially relaxing some of
those assumptions now that's not always
a very productive way to look at it
because if you actually have a
supervised learning problem you'll
probably solve it much more effectively
by using supervised learning methods
because it's easier but you can view
reinforcement as a journalist a tional
know for sure but they're fundamentally
that's a mathematical statement that's
absolutely correct but it seems that
reinforcement learning the kind of tools
we'll bring to the table today of today
so maybe down the line everything will
be a reinforcement learning problem just
like you said
image classification should be mapped to
a reinforcement learning problem but
today the tools and ideas the way we
think about them are different sort of
supervised learning has been used very
effectively to solve basic narrow AI
problems the reinforcement learning kind
of represents the dream of AI it's very
much so in the research space now in two
captivating the imagination of people
what we can do with intelligent systems
but it hasn't yet had as wide of an
impact as the supervised learning
approaches so that so that I my question
comes from more practical sense like
what do you see is the gap between the
more general reinforcement learning
and the very specific yes it's a
question decision-making with one
sequence one step in the sequence of the
supervised learning so for a practical
standpoint I think that one one thing
that is you know potentially a little
tough now and this is I think something
that we'll see this is a gap that we
might see closing over the next couple
of years is the ability of reinforcement
learning algorithms to effectively
utilize large amounts of prior data so
one of the reasons why it's a bit
difficult today to use reinforcement
learning for all the things that we
might want to use it for is that in most
of the settings where we want to do
rational decision-making it's a little
bit tough to just deploy some policy
that does crazy stuff and learns purely
through trial and error it's much easier
to collect a lot of data a lot of logs
of some other policy that you've got and
then maybe you you know if you can get a
good policy out of that then you deploy
it and let it kind of fine-tune a little
bit but algorithmically it's quite
difficult to do that so I think that
once we figure out how to get
reinforcement learning to bootstrap
effectively from large data sets then
we'll see very very rapid growth and
applications of these technologies so
this is what's referred to as off policy
reinforcement learning or offline RL or
batch RL and I think we're seeing a lot
of research right now that that's
bringing us closer and closer to that
can you maybe paint a picture of the
different methods she said
off policy what's value-based
reinforcement learning what's policy
based was modelled based with soft
policy on policy what are the different
categories of reinforcement yeah so one
way we can think about reinforcement
learning is that it's um it's in some
very fundamental way it's about learning
models that can answer kind of what-if
questions so what would happen if I take
this action that I haven't taken before
and you do that of course from
experience from data and oftentimes you
do it in a loop so you build a model
that answers these what-if questions use
it to figure out the best action you can
take and then go and try taking that and
see if the outcome agrees with what you
predicted
so the different kinds of techniques are
basically refer different ways of doing
it so model based methods answer a
question of
what state you would get basically what
would happen to the world if you were to
take a certain action value based
methods they answer the question of what
value you would get meaning what utility
you would get but in a sense they're not
really all that different because
they're both really just answering these
what-if questions now unfortunately for
us with current machine learning methods
answering what-if questions can be
really hard because they are really
questions about things that didn't
happen if you want to answer what-if
questions about things that did happen
you wouldn't need to learn model you
would just like repeat the thing that
worked before and that's really a big
part of why RL is a little bit tough so
if you have a purely on policy kind of
online process then you ask these
what-if questions you make some mistakes
then you're going to try doing those
mistake in things and then you observe
kind of the counter examples that'll
teach you not to do those things again
if you have a bunch of off policy data
and you just want to synthesize the best
pulse you can out of that data then you
really have to deal with the the
challenges of making these these
counterfactual what's the policy yeah a
policy is a model or some kind of
function that maps from observations of
the world to actions so in reinforcement
learning we often refer to the the
current configuration of the world as
the state so we say the state kind of
encompasses everything you need to fully
define where the world is at at the
moment and depending on how we formulate
the problem we might say you either get
to see the state or you get to see an
observation which is some snapshot some
piece of the state so policy is just
includes everything in it in order to be
able to act in this world yes and so
what is off policy mean if yeah so the
terms on policy and off policy refer to
how you get your data so if you get your
data from somebody else who was doing
some other stuff maybe you get your data
from some manually programmed a system
that was you know just running in the
world before that's referred to as off
policy data but if you got the data by
actually acting in the world based on
what your current policy thinks is good
we call that on policy data and
obviously on policy data is more useful
to you because if your current policy
makes some bad decisions you will I
you see that those decisions are bad off
policy data however might be much easier
to obtain because maybe that's all the
log data that you have from before so we
talked about new offline talked about
autonomous vehicles so you can envision
off policy kind of approaches in
robotics phases where there's really ton
of robots out there but they don't get
the luxury of being able to explore
based on reinforcement learning
framework so how do we make again open
question but how do we make our policy
methods work yeah so this is something
that has been kind of a big open problem
for a while and in the last few years
people have made a little bit of
progress on that you know I can tell you
about and it's not by any means solved
yet but I can tell you some of the
things that for example we've done to
try to address some of the challenges it
turns out that one really big challenge
with off policy reinforcement learning
is that you can't really trust your
models to give accurate predictions for
any possible action so if I've never
tried to if in my data said I never saw
somebody steering the car off the road
onto the sidewalk my value function or
my model is probably not going to
predict the right thing if I ask what
would happen if I were to steer the car
off the road onto the sidewalk so one of
the important things you have to do to
get off Paul crl to work is you have to
be able to figure out whether a given
action will result in a trustworthy
prediction or not and you can use kind
of distribution estimation methods kind
of density estimation methods to try to
figure that out so you could figure out
that well this action my model is
telling me that it's great but it looks
totally different from any action I've
taken before so I'm all it's probably
not correct and you can incorporate
regularization terms into your learning
objective that will essentially tell you
not to ask those questions that your
model is unable to answer what would
lead to breakthroughs in this space do
you think like well what's needed is
this a data set question do we need to
collect big benchmark data sets that
allow us to explore the space is it a
new kinds of methodologies like what's
your sense or maybe coming together in a
space of robotics and defining the
problem to do working on him I think
four off policy reinforced mooring in
particular it's very much an algorithms
question right now and you know this is
something that I think it's great
because now arounds question is you know
that that just takes some very smart
people to get together and think about
it really hard whereas if it was like a
data problem or hardware problem that
would take some serious engineering so
that's why I'm pretty excited about that
problem because I think that we're in a
position where we can make some real
progress on it just by coming up with
the right algorithms in terms of which
algorithms they could be you know that
the problems that their core are very
related to problems in you know things
like like causal inference right because
well you're really dealing with the
situations where you have a model a
statistical model that's trying to make
predictions about things that I hadn't
seen before
and if it's a if it's a model it's
generalizing properly that'll make good
predictions if it's a model that picks
up on spurious correlations that will
not generalize properly and then you can
you have an arsenal of tools you can use
you could for example figure out what
are the regions where it's trustworthy
or on the other hand you could try to
make it generalize better somehow or
some combination of the two is there
room for mixing sort of or most of it
like 90 95 percent is off policy you
already have the data set and then you
get to send the robot out to do a little
exploration like what what's that role
of mixing them together yeah absolutely
I think that this is something that you
actually might describe very well at the
beginning of the of our discussion when
you talk about the iceberg like this is
the iceberg that the 99% of your prior
experience that's your iceberg you'd use
that for all policy reinforcement
learning and then of course if you've
never you know opened that particular
kind of door with that particular lock
before then you have to go out and
fiddle with it a little bit and that's
that additional 1% to help you figure
out a new task and I think that's
actually like a pretty good recipe going
forward is this to you the most exciting
space of reinforcement learning now or
is there what's uh and maybe taking a
step back not just now but what's to use
the most beautiful idea apologize for
the romanticized question but the
beautiful idea or a concept in
reinforcement learning
in general I actually think that one of
the things that is a very beautiful idea
in reinforcement learning is just the
idea that you can obtain a near optimal
controller in your optimal policy
without actually having a complete model
of the world this is you know it's
something that feels perhaps kind of
obvious if you if you just hear the term
reinforcement learning or you think
about trial and error learning but from
a controls perspective it's a very weird
thing because classically you know we we
think about engineered systems and
controlling engineered systems as as the
problem of writing down some equations
and then figuring out given these
equations you know basically I solve for
X figure out the the thing that
maximizes its performance and the the
theory of reinforcement learning
actually gives us a mathematically
principled framework just think to
reason about you know optimizing some
quantity when you don't actually know
the equations that govern that system
and that I don't to me that actually
seems kind of kind of you know very
elegant not something that sort of
becomes immediately obvious at least in
the mathematical sense does it make
sense to you that it works at all well I
think it makes sense when you take some
time to think about it but it is a
little surprising well then then taking
a step into the more deeper
representations which is also very
surprising of sort of the richness of
the state space the space of
environments that this kind of approach
can operate in can you maybe say what is
deep reinforcement learning well deep
reinforcement learning simply refers to
taking reinforcement learning algorithms
and combining them with high capacity
neural net representations which is you
know kind of it might at first seem like
a pretty arbitrary thing just take these
two components and stick them together
but the reason that it's it's something
that has become so important in recent
years is that reinforcement learning it
kind of faces an exacerbated version of
a problem that has faced many other
machine learning too
so if you if we go back to like you know
the early 2000s or the late 90s we'll
see a lot of research on machine
learning methods that have some very
appealing mathematical properties like
they reduced a convex optimization
problems for instance but they require
very special inputs they require a
representation of the input that is
clean in some way like for example clean
in the sense that the classes in your
multi-class classification problems
separate linearly so they they have some
cases it's some kind of good
representation we call this a feature
representation and for a long time
people were very worried about features
in the world of supervised learning
because somebody had to actually build
those features so you couldn't just take
an image and plug it into your logistic
regression or your SVM or something
someone had to take that image and
process it using some handwritten code
and then neural nets came along and they
could actually learn the features and
suddenly we could apply learning
directly to the raw inputs which was
great for images but it was even more
great for all the other fields where
people hadn't come up with good features
yet and one of those fields actually
reinforced my learning because in
reinforcement learning the notion of
features if you don't use neural nets
and you have to design your own features
it's very very opaque like it's very
hard to imagine like let's say I'm
playing chess or go what is a feature
with which I can represent the value
function for go or even though the
optimal policy forego linearly I I don't
even know how to start thinking about it
and and people tried all sorts of things
that would write down you know an expert
chess player looks for whether the the
knight is in the middle of the board or
not so that's a feature is night in
middle of board and they would write
these like long lists of kind of
arbitrary made-up stuff and that was
really kind of getting us no way and
that's a little chess is a little more
accessible than the robotics problem
absolutely all right that's there's at
least experts in the different features
for chess but still like the neural
network there I did to me that's I mean
you put it eloquently and almost made it
seem like a natural step to add neural
networks but the fact that neural
networks are able to discover features
in the control problem it's very
interesting it's hopeful I'm not sure
what to think about it but it feels
hopeful that the control problem has
features to be learned
like I guess my question is is it
surprising to you how far the deep side
of deep reinforcement learning is able
to like what the space of problems has
been able to tackle from especially in
games with the Alpha star and and alpha
zero and just the the representation of
power there and in the robotic space and
what is your sense of the limits of this
representation power and the control
context I think that in regard to the
limits that here I think that one thing
that makes it a little hard to fully
answer this question is because in
settings where we would like to put push
these things to the limit we encounter
other bottlenecks so like the reason
that I can't get my robot to learn how
to like I don't know do the dishes in
the kitchen it's not because it's neural
net is not big enough it's because when
you try to actually do trial and error
learning you reinforce them a loner
directly in the real world where you
have the potential to gather these large
they're you know highly varied and
complex datasets you start running into
other problems like one problem you run
into very quickly it'll first sound like
a very pragmatic problem that actually
turns out to be a pretty deep scientific
problem take the robot put in your
kitchen have it try to learn to do the
dishes with trial and error it'll break
all your dishes and then we'll have no
more dishes to clean now you might think
this is a very practical issue but
there's something to this which is that
if you have a person trying to do this
you know a person will have some degree
of common sense they'll break one dish
it'll be a little more careful with the
next one and if they break all of them
they're gonna go and get more or
something like that so there's all sorts
of scaffolding that that comes very
naturally to us for our learning process
like you know if I have to learn
something through trial and error I have
a common sense to know that I have to
you know try multiple times if I screw
something up I ask for help or I recept
things or something like that and all
that it's kind of outside of the classic
reinforcement problem formulation there
are the things that are that can also be
categorizes
scaffolding but are very important like
for example where you get your award
function if I want to learn how to pour
a cup of water well how do I know if
I've done it correctly now that probably
requires an entire computer vision
system to be built just to determine
that and that seems a little bit
inelegant so there are all sorts of
things like this that start to come up
when we think through what we really
need to get reinforcement learning to
happen at scale in the real world and
any that many of these things actually
suggest a little bit of a shortcoming in
the problem formulation and a few deeper
questions that we have to resolve that's
really interesting I thought to like
David silver bought alpha zero and it
seems like there's no again the the we
haven't hit the limit at all in the
context when there is no broken dishes
so in the game in the case of go you can
it's really about just scaling compute
so again like the bottleneck is the
amount of money you're willing to invest
in compute and then maybe the different
the scaffolding around how difficult it
is to scale compute maybe but there
there's no limit and it's interesting
now we move to the real world and
there's the broken dishes they solved it
and the reward function like you
mentioned that's really nice of what how
do we push forward there do you think
there's there's this kind of sample
efficiency question that people bring up
or you know not having to break a
hundred thousand dishes is this an
algorithm question is this data
selection like question or what do you
think how do we how do we not break them
too many dishes yeah well one way we can
think about that is that maybe we need
to be better at reusing our data
building that that iceberg so perhaps
perhaps it's too much to hope that you
can have a machine that in isolation in
the vacuum without anything else can
just master complex tasks in like in
minutes the way that people do but
perhaps it also doesn't have to perhaps
what it really needs to do is have an
existence a lifetime where it does many
things and the previous things that it
has done prepare it to do new things
more
and you know the study of these kinds of
questions typically falls under
categories like multitask learning or
meta learning but they all fundamentally
deal with the same general theme which
is use experience for doing other things
to learn to do new things efficiently
and quickly so what do you think about
if you just look at one particular case
study of Tesla autopilot that has
quickly approaching towards a million
vehicles on the road where some
percentage of the time thirty forty
percent of the time is driven using the
computer vision multitask Hydra net
right and then the other percent that's
what they call it Hydra net the the
other percent is human controlled from
the human side how can we use that data
what's your sense like what's the signal
do you have ideas in this autonomous
vehicle space when people can lose their
lives you know it's a it's a safety
critical environment so how do we use
that data so I think that actually the
kind of problems that come up when we
want systems that are reliable and that
can kind of understand the limits of
their capabilities they're actually very
similar to the kind of problems that
come up when we have we're doing off
policy reinforcement learning so as I
mentioned before and off policy
reinforcement learning the big problem
is you need to know when you can trust
the predictions of your model because if
you if you're trying to evaluate some
pattern of behavior for which your model
doesn't give you an accurate prediction
then you shouldn't use that to to modify
your policy and it's actually very
similar to the problem that we're faced
when we actually then deploy that thing
and we want to decide whether we trust
it in the moment or not so perhaps we
just need to do a better job of figuring
out that part and that's a very deep
research question of course it's also a
question that a lot of people are
working on so I'm pretty optimistic that
we can make some progress on that over
the next few years what's the role of
simulation in reinforcement learning the
end deeper enforcement learning
reinforcement learning like how
essential is it it's been essential for
the breakthroughs so far for some
interesting breakthroughs do you think
it's a crutch that we rely on I mean
again it's can
throw off policy discussion but do you
think we can ever get rid of simulation
or do you think simulation will actually
take over will create more and more
realistic simulations that will allow us
to to solve actual real-world problems
like transfer the models will learn in
simulation from the walk-around yes I
think that simulation is a very
pragmatic tool that we can use to get a
lot of useful stuff to work right now
but I think that in the long run we will
need to build machines that can learn
from real data because that's the only
way that will get them to improve
perpetually because if we can't have our
machines learn from real data if they
have to rely on simulated data
eventually the simulator becomes the
bottleneck in fact this is a general
thing if your machine has any bottleneck
that is built by humans and that doesn't
improve from data it will eventually be
the thing that holds it back and if
you're entirely relying on your
simulator that'll be the bottleneck if
you're entirely really reliant on a
manually designed controller that's
going to be the bottleneck so simulation
is very useful it's very pragmatic but
it's not a substitute for being able to
utilize real experience and this is by
the way this is something that I think
is quite relevant now especially in the
context of some of the things we've
discussed because some of these kind of
scaffolding issues that I mentioned
things like the broken dishes and the
unknown reward function like these are
not problems that you would ever stumble
on when working in a purely simulated
kind of environment but they become very
apparent when we try to actually run
these things in the real world do you
throw a brief wrench into our discussion
let me ask do you think we're living in
a simulation oh I have no idea do you
think that's a useful thing to even
think about about the there the the
fundamental physics nature of reality or
another perspective the reason I think
the simulation hypothesis is interesting
is it's to think about how difficult is
it to create sort of a virtual reality
game type situation that will be
sufficiently convincing to us humans or
sufficiently enjoyable that would we
wouldn't want to leave that's actually a
practical engineering
and I I personally really enjoy virtual
reality but it's quite far away but I
kind of think about what would it take
for me to want to spend more time in
virtual reality versus the real world
and that's a that's a sort of a nice
clean question because at that point
we've reached if I want to live in a
virtual reality that means we're just a
few years away where majority of the
population lives in a virtual reality
and that's how we create the simulation
right you don't need to actually
simulate the you know the quantum
gravity and just every aspect of the of
the universe and that's a read that the
interesting question for reinforcement
learning too is if you want to make
sufficiently realistic simulations that
make it blend the difference between
sort of the real world and the
simulation there by just are the some of
the things we've been talking about kind
of the problems go away if we can create
actually interesting rich simulations
it's an interesting question and it
actually I think your question
casts your previous questions in a very
interesting light because in some ways
asking whether we can well the more
practical more kind of practical version
is like you know can we build simulators
that are good enough to train
essentially AI systems that will work in
the world and it's kind of interesting
to think about this about what this
implies if true it kind of implies that
it's easier to create the universe than
it is to create a brain and then it
seems like put this way it seems kind of
weird the aspect of the simulation most
interesting to me is the simulation of
other humans that seems to be a
complexity that makes the robotics
problem harder now I don't know if every
robotics person agrees with that notion
just as a quick aside what are your
thoughts about when the human enters the
picture of the robotics problem how does
that change the reinforcement learning
problem the the learning problem in
general yeah I think that's a it's a
kind of a complex question and I guess
my hope for a while had been that if we
build these
robotic learning systems that that are
multitask that utilize lots of prior
data and that learn from their own
experience the bit where they have to
interact with people will be perhaps
handled in much the same way as all the
other bits so if they have prior
experience in attracting with people and
they can learn from their own experience
of interacting with people for this new
task maybe that'll be enough now of
course there if it's not enough there
are many other things we can do and
there's quite a bit of research on that
in that area but I think it's worth a
shot to see whether the the the multi
agent interaction the the ability to
understand that other beings in the
world have their own goals and tensions
and thoughts and so on whether that kind
of understanding can emerge
automatically from simply learning to do
things with and maximize utility that
information arises from the data you've
said something about gravity sort of
that you don't need to explicitly inject
anything into the system they can be
learned from the data and gravity is an
example of something that could be
learned from data sort of like the
physics of the world like what what are
the limits of what we can learn from
data do you really do you think we can
so a very simple clean way to ask that
is do you really think we can learn
gravity from just data the idea the the
laws of gravity so it says something
that I think is a common kind of pitfall
when thinking about prior knowledge and
learning is to assume that just because
we know something then that it's better
to tell the Machine about that rather
than have it I regret out on its own in
many cases things that are important
that affect many of the events that the
Machine will experience are actually
pretty easy to learn like you know if
things if every time you drop something
it falls down like yeah you might not
get the you know you might get kind of
an in the Newton's version not Einsteins
version but it'll be pretty good and it
will probably be sufficient for you to
act rationally in the world because you
see the phenomena all the time so things
that are readily apparent from the data
we might not need to specify those by
hand it might actually be easier to let
the Machine figure
it just feels like that there might be a
space of many local local minima in
terms of theories of this world that we
would discover and get stuck on yeah of
course
Newtonian mechanics is not necessarily
easy to come by
yeah and well in fact in in some fields
of science for example human
civilizations itself full of these local
optima so for example if you think about
how people try to figure out biology and
medicine you know for the longest time
the kind of rules like the kind of
principles that serve us very well in
our day to day lives actually serve us
very poorly in understanding medicine
and biology we had kind of very
superstitious and weird ideas about how
the body worked until the advent of the
modern scientific method so that does
seem to be you know a failing of this
approach but it's also a failing of
human intelligence arguably maybe a
small aside but some you know the idea
of self play is fascinating
reinforcement learning sort of these
competitive and creating a competitive
context in which agents can play against
each other in a sort of at the same
skill level and thereby increasing each
other school it seems to be this kind of
self improving mechanism is
exceptionally powerful in the context
where it could be applied first of all
is that beautiful to you that this
mechanism work as well as it does and
also can be generalized to other context
like in the robotic space or anything
that's applicable to the real world I
think that it's a very interesting idea
and I suspect that the bottleneck to
actually generalizing it to the robotic
setting is actually gonna be the same as
as the bottleneck for everything else
that we need to be able to build
machines that can get better and better
through natural interaction with the
world and once we can do that then they
can go out and play with they can play
with each other they can play with
people they can play with the natural
environment but before we get there
we've got all these other problems we've
got we have to get out of the way
there's no shortcut around that you have
to interact with the national
environment
well because in in a self play setting
you still need a mediating mechanisms so
the the reason that you know self play
works for a board game is because the
rules of that board game mediate the
interaction between the agents so the
kind of intelligent behavior that will
emerge depends very heavily on the
nature of that mediating mechanism so on
the side of reward functions that's
coming up with good reward function
seems to be the thing that we associate
with general Intel like human beings
seem to value the idea of developing our
own reward functions of you know
arriving in meaning and so on and yet
for reinforcement learning we often kind
of specify that's the given what's your
sense of how we develop a reward for
good you know good reward functions yeah
I think that's a very complicated and
very deep question and you're completely
right that classically in reinforcement
learning this question has kind of been
treated as a non-issue that you sort of
treat the reward as this external thing
that comes from some other bit of your
biology and you can don't worry about it
and I do think that that's actually you
know a little bit of a mistake that we
shouldn't worry about it and we can
approach you in a few different ways we
can approach it for instance by thinking
of rewards as a communication medium we
can say well how does a person
communicate to a robot what its
objective is you can approach it also as
sort of more of an intrinsic motivation
medium you could say can we write down
kind of a general objective that leads
to good capability like for example can
you write down some objective such that
even in the absence of any other task if
you maximize that objective you'll sort
of learn useful things this is a
something that has sometimes been called
unsupervised reinforcement learning
which i think is a really fascinating
area of research especially today we've
done a bit of work on that recently one
of the things we've studied is whether
we can have some notion of of
unsupervised reinforcement learning by
means of you know information theoretic
quantities like for instance minimizing
a Bayesian measure of surprise this is
an idea that was you know pioneered
actually in the computational
neuroscience community by folks like
Carl Fritton
we've done some work recently that shows
that you can actually learn pretty
interesting skills by essentially
behaving in a way that allows you to
make accurate predictions about the
world it seems a little circular do the
things that will lead to you getting the
right answer for prediction but you can
you know by doing this you can sort of
discover stable niches in the world you
can discover that if you're playing
Tetris then correctly you know clearing
the rows will let you play Tetris for
longer and keep the board nice and clean
which sort of satisfies some desire for
order in the world and as a result to
get some degree of leverage over your
domain so we're exploring that pretty
actively is there a role for a human
notion of curiosity in itself being the
reward sort of discovering new things
about the war the world so one of the
things that I'm pretty interested in is
actually whether discovering new things
can actually be an emergent property of
some other objective that quantifies
capability so new things for the sake of
new things maybe it's not maybe might
not by itself be the right answer but
perhaps we can figure out an objective
for which discovering new things is
actually the natural consequence that's
something we're working on right now but
I don't have a clear answer for you
there yet that's still work-in-progress
you mean just as a security observation
to see sort of creative the patterns of
curiosity on the way to optimize for a
particular protector on the way to
optimize for a particular measure of
capability is is there ways to
understand or anticipate unexpected
unintended consequences of particular
reward functions
sort of anticipate the kind of
strategies that might be developed and
try to avoid highly detrimental strategy
yeah so classically this is something
that has been pretty hard in
reinforcement learning because it's
difficult for a designer to have good
intuition about you know what a learning
outcome will come up with when they give
it some objective there are ways to
mitigate that one way to mitigate it is
to actually define an objective that
says like don't do weird stuff
you can actually quantify you can say
just like don't enter situations that
have low probability under the
distribution of states you've seen
before it turns out that that's actually
one very good way to do off policy
reinforcement learning actually so we
can do some things like that if we
slowly venture in speaking about reward
functions into greater and greater
levels of intelligence there's a mr.
Russell thinks about this the alignment
of AI systems with us humans so how do
we ensure that AG AI systems align with
us humans it's a it's kind of a reward
function question of specifying the
behavior of AI systems such that their
success aligns with us with the broader
intended success interest of human
beings do you have thoughts on this they
have kind of concerns of where
reinforcement learning fits into this or
are you really focused on the current
moment of us being quite far away and
trying to solve the robotics problem I
don't have a great answer to this but
you know and I do think that this is a
problem that's that's important to
figure out for my part I'm actually a
bit more concerned about the other side
of the of this equation that you know
maybe rather than unintended
consequences for objectives that are
specified too well I'm actually more
worried right now about unintended
consequences for objectives that are not
optimized well enough which might become
a very pressing problem when we for
instance try to use these techniques for
safety critical systems like cars and
aircraft and so on I think at some point
we'll face the issue of objectives being
optimized too well but right now I think
we're more likely to face the issue of
them not being optimized well enough but
you don't think on intended consequence
can arise even when you're far from
optimality sort of like on the path to
it oh no I think I unattended
consequence can absolutely arise it's
just I think right now the bottleneck
for improving reliability safety and
things like that is more with systems
that like need to work better that the
optimize their objective better
you have thoughts concerns about
existential threats of human level
intelligence sort of if we put on our
hat of looking in ten twenty a hundred
five hundred years from now give
concerns about existential threats of AI
systems I think there are absolutely
existential threats for AI systems just
like there are for any powerful
technology but I think that the these
kinds of problems can take many forms
and and some of those forms will come
down to you know people with nefarious
intent some of them will come down to AI
systems that have some fatal flaws and
some of them will will of course come
down to AI systems that are too capable
in some way but among this set of
potential concerns I would actually be
much more concerned about the first two
right now and principally the one with
nefarious humans because you know just
through all of human history actress
that I Ferris humans that have been the
problem not the nefarious machines then
I am about the others and I think that
right now the best that I can do to make
sure things go well is to you know build
the best technology I can and also
hopefully promote responsible use of
that technology do you think RL systems
has something to teach us humans
you said nefarious humans getting us in
trouble I mean machine learning system
self in some ways have revealed to us
the ethical flaws in our data in that
same kind of wake and reinforce some
learning teach us about ourselves has it
taught something what have you learned
about yourself from trying to build
robots and reinforce the learning
systems I'm not sure what I've learned
about myself but maybe part of the
answer to your question might become a
little bit more apparent once we see
more widespread deployment of
reinforcement learning for decision
making support in you know in domains
like you know healthcare education
social media etc and I think we will see
some interesting stuff emerge there we
will see for instance what kind of
behaviors these systems come up with
in situations the where there is
interaction with humans and and where
they have you know possibility of
influencing human behavior I think we're
not quite there yet but maybe in the
next two years we'll see some
interesting stuff coming out in that
area
I hope outside the research because the
the exciting space where this could be
observed is sort of large companies that
deal with large data and I hope there's
some transparency and one of the things
it's unclear when I look at social
networks and just online is why an
algorithm did something or whether you
know even an algorithm was involved and
that'd be interesting as a formal
research perspective just to to observe
the results of algorithms to open up
that data or did these be sufficiently
transparent about the behavior of these
e-a systems in the real world what's
your sense I don't know if you looked at
the blog post bitter lesson by Irish
Sutton where it looks at serve the big
lesson of research in AI in
reinforcement learning is that simple
methods general methods that leverage
computation seem to work well so
basically don't try to do any kind of
fancy algorithms just wait for
computation and get fast do you share
this kind of intuition I think the high
level idea makes a lot of sense I'm not
sure that my takeaway would be that we
don't need to work on algorithms I think
that my takeaway would be that we should
work on general algorithms and actually
I think that this idea of needing to
better automate the acquisition of
experience in the real world actually
follows pretty naturally from Rich
Sutton's conclusion so if the claim is
that automated general methods plus data
leads to good results then it makes
sense that we should build general
methods and we should build the kind of
methods that we can deploy and get them
to go out there and like collect their
experience autonomously I think that you
know one place where I think that the
current state of things Falls a little
bit short of that is actually that the
going out there collecting the data
autonomously
which is easy to do in a simulator board
game but very hard to do in the real
world yeah it keeps coming back to this
one problem right it's uh so your mind
is focused there now in this real world
it just seems scary the step of
collecting the data and it seems unclear
to me how we can do it effectively well
you know it's seven billion people in
the world each of them had to do that at
some point in their lives
and we should leverage that experience
that they've all done the we should be
able to try to collect that kind of data
okay big questions maybe stepping back
through your life would book or books
technical or fiction or philosophical
had a big impact onion on the way you
saw the world I know he thought about in
the world your life in general hmm and
maybe what books if is different would
you recommend people consider reading on
their own intellectual journey it could
be within reinforcement learning but
could be very much bigger I don't know
if this is like a scientifically like
particularly meaningful answer but like
the honest answers that I I actually
found a lot of the work by Isaac Asimov
to be very inspiring when I was younger
I don't know if that has anything to do
with with AI necessarily you don't think
it had a ripple effect in your life
maybe it did but yeah I like I think
that a vision of a future where well
first of all artificial mice artificial
intelligence system artificial robotic
systems have you know kind of a big
place a big role in society and where we
try to imagine the sort of the the
limiting case of technological and
advancement and how that might play out
in in our future history but yeah I
think that the that was in some way
influential I don't really know how but
and I would recommend it I mean if
nothing else you'd be well entertained
did you first yourself like fall in love
with the idea of artificial intelligence
get captivated by this field so my
honest answer here is actually that I
only really started to think think about
it as a that's something that I might
want to do actually in graduate school
pretty light and a big part of that was
that until you know somewhere around
2009 2010 it just wasn't really high on
my priority list because I I didn't
think that it was something where we're
going to see very substantial advances
in my lifetime and you know maybe in
terms of my career the time when I
really decided I wanted to work on this
was when I actually took a seminar
course that was taught by Professor and
ring and you know at that point I of
course had some had like a decent
understanding of the technical things
involved but one of the things that
really resonated with me was when he
said in the opening lecture something to
the effect of like well he used to have
graduate students come to him and talk
about how they want to work on AI and he
would kind of chuckle and give them some
math problem to deal with but now he's
actually thinking that this is an area
where we might see like substantial
advances in our lifetime and that kind
of got me thinking because you know it's
an abstract sense yeah like you can kind
of imagine not but in a very real sense
when someone who had been working on
that kind of stuff their whole career
suddenly says that yeah like that had
that had some effect on me yeah this
might be a special moment in the history
of the field that this is where we might
see some some interesting breakthroughs
so in the space of advice somebody who's
interested in getting started and
machine learning or reinforcement
learning what advice would you give to
maybe an undergraduate student or maybe
even younger how what are the first
steps to take and further on what are
the stapes steps to take on that journey
so something that I think is important
to do is to is to not be afraid to like
spend time imagining the kind of outcome
that you might like to see so you know
one outcome might be a successful career
large paycheck or something or
state-of-the-art results in some
benchmark but hopefully that's not the
thing that's like the main driving force
for somebody but I I think that if
someone who's a student considering a
career in AI like takes a little while
sits down and thinks like what do I
really want to see what I want to see a
machine do what I want what do I want to
see a robot do what I want to do and
what I want to see a natural language
system just like imagine you know
imagine it almost like a commercial for
a future product or something or like
like something that you'd like to see in
the world and then actually sit down and
think about the steps that are necessary
to get there and hopefully that thing is
not a better number on imagenet
classification it's like it's probably
like an actual thing that we can't do
today that would be really awesome
whether it's a robot Butler or a you
know a really awesome healthcare
decision making support system whatever
it is that you find inspiring and I
think that thinking about that and then
backtracking from there and imagining
the steps needed to get there will
actually do much better research it'll
lead to rethinking the assumptions it'll
lead to working on the bottlenecks other
other people aren't working on and then
naturally to turn to you we've talked
about reward functions and you just give
an advice and looking forward I would
like to see what kind of change you
would like to make in the world what do
you think ridiculous big question what
do you think is the meaning of life what
is the meaning of your life what gives
you fulfillment purpose happiness and
meaning that's a very big question um
what's the reward function under which
you are operating yeah I think one thing
that does give you know if not meaning
at least satisfaction is some degree of
confidence that I'm working on a problem
that really matters I feel like it's
less important to me to like actually
solve a problem but it's it's quite nice
to take things to spend my time on that
I believe really matter and I I try
pretty hard to to look for that I don't
know if it's easy to answer this but if
you're successful what does that look
like what's the
they dream enough of course success is
built on top of success and you keep
going forever but what is the dream yeah
so one very concrete thing or maybe as
concrete as it's gonna get
here is is to see machines that actually
get better and better the you know the
longer they exists in the world and that
kind of seems like on the surface one
might even think that that's something
that we have today but I think we really
don't I think that there is unending
complexity in the universe and to date
all the machines that we've been able to
build don't sort of improve up to the
limit of that complexity they they hit a
wall somewhere maybe they hit a wall
because they're in a simulator that has
that is only a very limited very pale
imitation of the real world or they hit
a wall because they rely on a label
dataset but they never hit the wall of
like running out of stuff to see like
the did so you know I I'd like to build
a machine that that can go as far as
possible and that runs up against the
ceiling of the complexity of the
universe yes well I don't think there's
a better way to end it Sergey thank you
so much is a huge honor I can't wait to
see the amazing work they have to
publish and in education space in terms
of reinforcement learning thank you for
inspiring the world thank you for the
great research you do thank you thanks
for listening to this conversation with
Sergey levine and thank you to our
sponsors cash app and expressvpn please
consider supporting this podcast by
downloading cash app and using code lex
podcast and signing up at expressvpn
comm / lex pod click all the links buy
all the stuff it's the best way to
support this podcast and the journey I'm
on if you enjoy this thing subscribe on
YouTube review it five stars in a
podcast supported on patreon or connect
with me on Twitter at lex friedman
spelled somehow if you can figure out
how without using the letter e just FR
ID ma m and now let me leave you with
some words from Salvador Dali
intelligence without ambition is a bird
without wings thank you for listening
and hope to see you next time
you