Kind: captions
Language: en
let's just talk about maybe the control
problem so this idea of losing ability
to control the behavior and our AI
system so how do you see that how do you
see that coming about what do you think
we can do to manage it well so it
doesn't take a genius to realize that if
you make something that smarter than you
you might have a problem you know and
Turing Alan Turing you know wrote about
this and gave lectures about this you
know I think 1951 he did a lecture on
the radio and he basically says you know
once the machine thinking method starts
you know very quickly they'll outstrip
humanity and you know if we're lucky we
might be able to I think he says if we
may be able to turn off the power at
strategic moments but even so species
would be humbled yeah you actually I
think was wrong about that right here is
you you know if it's a sufficiently
intelligent machine is not going to let
you switch it off so it's actually in
competition with you so what do you
think is meant just for a quick tangent
if we shut off this super intelligent
machine that our species will be humbled
I think he means that we would realize
that we are inferior right that we we
only survive by the skin of our teeth
because we happen to get to the off
switch
just in time you know and if we hadn't
then we would have lost control over the
earth so do you are you more worried
when you think about the stuff about
super intelligent AI or are you more
worried about super powerful AI that's
not aligned with our values so the paper
clip scenarios kind of
I think so the main problem I'm working
on is is the control problem the the
problem of machines pursuing objectives
that are as you say not aligned with
human objectives and and this has been
there's been the way we've thought about
a eyes since the beginning you you build
a machine for optimizing and then you
put in some objective and it optimizes
right and and you know we we can think
of this as the the King Midas problem
right because if you know so King Midas
put in this objective right everything I
touch you turned to gold and the gods
you know that's like the machine they
said okay done you know you now have
this power and of course his food and
his drink and his family all turned to
gold and then he dies misery and
starvation and this is you know it's
it's a warning it's it's a failure mode
that pretty much every culture in
history has had some story along the
same lines you know there's the the
genie that gives you three wishes and
you know third wish is always you know
please undo the first two wishes because
I messed up
and you know and when office amuel wrote
his chest his checkup laying program
which learned to play checkers
considerably better than Arthur Samuel
could play and actually reached a pretty
decent standard
Norbert Wiener who was a one of the
major mathematicians of the 20th century
sort of the father of modern automation
control systems
you know he saw this and he basically
extrapolated you know as touring did and
said okay this is how we could lose
control and specifically that we have to
be certain that the purpose we put into
the machine is the purpose which we
really desire and the problem is we
can't do that right you mean we're not
it's a very difficult to encode so to
put our values on paper is really
difficult or you're just saying it's
impossible
your line is grating that's it so it's
it theoretically it's possible but in
practice it's extremely unlikely that we
could specify correctly in advance the
full range of concerns of humanity that
you talked about cultural transmission
of values I think is how humans to human
transmission the values happens right
what we learn yeah I mean as we grow up
we learn about the values that matter
how things how things should go what is
reasonable to pursue and what isn't
reasonable to pursue like machines can
learn in the same kind of way yeah so I
think that what we need to do is to get
away from this idea that you build an
optimizing machine and then you put the
objective into it
because if it's possible that you might
put in a wrong objective and we already
know this is possible because it's
happened lots of times right that means
that the machine should never take an
objective that's given as gospel truth
because once it takes them the the
objective is gospel truth alright then
it's the leaves that whatever actions
it's taking in pursuit of that objective
are the correct things to do so you
could be jumping up and down and saying
no you know no no no you're gonna
destroy the world but the machine knows
what the true objective is and is
pursuing it and tough luck to you you
know and this is not restricted to AI
right this is you know I think many of
the 20th century technologies right so
in statistics you you minimize a loss
function the loss function is
exogenously specified in control theory
you minimize a cost function in
operations research you maximize a
reward function and so on so in all
these disciplines this is how we
conceive of the problem and it's the
wrong problem because we cannot specify
with certainty the correct objective
right we need uncertainty we need the
machine to be uncertain about a
subjective what it is that it's post
it's my favorite idea of yours I've
heard you say somewhere well I shouldn't
pick favorites but it just sounds
beautiful we need to teach machines
humility yes I mean that's a beautiful
way to put it I love it that they're
humble oh yeah they know that they don't
know what it is they're supposed to be
doing and that those those objectives I
mean they exist they are within us but
we may not be able to explicate them we
may not even know you know how we want
our future to go so exactly and the
Machine you know a machine that's
uncertain is going to be deferential to
us so if we say don't do that
well now the machines learn something a
bit more about our true objectives
because something that it thought was
reasonable in pursuit of our objectives
turns out not to be so now it's learn
something so it's going to defer because
it wants to be doing what we really want
and you know that that point I think is
absolutely central to solving the
control problem and it's a different
kind of AI when you when you take away
this idea that the objective is known
then in fact a lot of the theoretical
frameworks that we're so familiar with
you know mark after
processes goal based planning you know
standard games research all of these
techniques actually become inapplicable
and you get a more complicated problem
because
because now
the interaction with the human becomes
part of the problem
because the human by making choices is
giving you more information about the
'true objective and that information
helps you achieve the objective better
and so that really means that you're
mostly dealing with game theoretic
problems where you've got the machine
and the human and they're coupled
together rather than a machine going off
by itself with a fixed objective which
is fascinating on the machine and the
human level that we when you don't have
an objective means you're together
coming up with an objective I mean
there's a lot of philosophy that you
know you could argue that life doesn't
really have meaning we we together agree
on what gives it meaning and we kind of
culturally create things that give why
the heck we are in this earth anyway we
together as a society create that
meaning and you have to learn that
objective and one of the biggest I
thought that's what you were gonna go
for a second one of the biggest troubles
we've run into outside of statistics and
machine learning and AI in just human
civilization is when you look at I came
from this I was born in the Soviet Union
and the history of the 20th century we
ran into the most trouble as humans when
there was a certainty about the
objective and you do whatever it takes
to achieve that objective whether you
talking about in Germany or communist
Russia oh yeah I guess I would say with
you know corporations in fact some
people argue that you know we don't have
to look forward to a time when AI
systems take over the world they already
have and they call corporations right
that corporations happen to be using
people as components right now but they
are effectively
algorithmic machines and they're
optimizing an objective which is
quarterly profit that isn't aligned with
overall well-being of the human race and
they are destroying the world they are
primarily responsible for our inability
to tackle climate change right so
I think that's one way of thinking about
what's going on with with cooperations
but I think the point you're making you
is valid that there are there are many
systems in the real world where we've
sort of prematurely fixed on the
objective and then decoupled the the
machine from those that's supposed to be
serving and I think you see this with
government right government is supposed
to be a machine that serves people but
instead it tends to be taken over by
people who have their own objective and
use government to optimize that
objective regardless of what people want
you