Kind: captions
Language: en
the following is a conversation with
peter norvig he's a director of research
at google
and the co-author with stuart russell of
the book artificial intelligence and
modern approach
that educated and inspired a whole
generation of researchers including
myself
to get into the field of artificial
intelligence
this is the artificial intelligence
podcast if you enjoy it
subscribe on youtube give it five stars
on itunes
support on patreon or simply connect
with me on twitter
at lex friedman spelled f-r-i-d-m-a-n
and now here's my conversation with
peter
norvig most researchers in the ai
community including myself
own all three editions red green and
blue of the uh
artificial intelligence a modern
approach it's a field-defining
textbook as many people are aware that
you wrote with stuart russell
how has the book changed and how have
you changed
in relation to it from the first edition
to the second to the third and now
fourth edition as you work on it yeah so
it's been a lot of years
a lot of changes one of the things
changing from the first
to maybe the second or third
was just the rise of uh computing power
right so i think in the in the first
edition
we said uh here's predicate logic
but uh that only goes so far because
pretty soon
you have millions of uh short little
predicate expressions and they couldn't
possibly fit in memory
uh so we're going to use first order
logic that's more concise
and then we quickly realized oh
predicate logic is pretty nice
because there are really fast sat
solvers and other things
and look there's only millions of
expressions and that fits easily into
memory or maybe even billions fit into
memory now
so that was a change of the type of
technology we needed
just because the hardware expanded even
to the second edition
resource constraints were loosened
significantly yeah
yeah and that was the early 2000s second
edition
right so 95 was the first and then 2000
2001 or so and then moving on from there
i think we're starting to see that again
with the
gpus and then more specific type of
machinery like the tpus and using custom
asics and so on
for deep learning so we're seeing
another advance in terms of the hardware
then i think another thing that we
especially notice this time around
is in all three of the first editions we
kind of said
well we're going to find ai as
maximizing expected utility
and you tell me your utility function
and now we've got
27 chapters worth of cool techniques for
how to optimize that
i think in this edition we're saying
more you know what
maybe that optimization part is the easy
part and the hard part
is deciding what is my utility function
what do i want
and if i'm a collection of agents or a
society
what do we want as a whole so you touch
that topic in this edition you get a
little bit more into utility
yeah that's really interesting on a
technical level we're almost pushing the
philosophical
i guess it is philosophical right so we
we've always had a philosophy chapter
which which i was uh
glad to that we were supporting
and now it's less kind of the uh
you know chinese room type argument and
more of these
uh ethical and societal type issues so
we get into
uh the issues of fairness and bias
and uh and just the issue of aggregating
utilities so how
do you encode human values into a
utility function is is this something
that you can do purely through data in a
learned way or is there
some systematic obviously there's no
good answers yet
there's just uh beginnings to this uh to
even opening
doors so there is no one answer yes
there are techniques uh to try to
learn that so we talk about inverse
reinforcement learning
right so reinforcement learning uh you
take some actions you get some rewards
and you figure out what actions you
should take
in inverse reinforcement learning you
observe somebody
taking actions and you figure out uh
well
that this must be what they were trying
to do if they did this action it must be
because they want it
of course there's restrictions to that
right so
lots of people take actions that are
self-destructive
uh where they're they're suboptimal in
certain ways so you don't want to learn
that
right you want to uh somehow learn the
uh the perfect actions
uh rather than the ones they actually
take so so that's a challenge
uh for that field then
another big part of it is just kind of
uh theoretical
of saying uh what can we accomplish and
so you look at like this
this work on the uh programs to
uh predict recidivism and decide uh
you know who should get parole or who
should get bail or whatever
uh and how are you gonna evaluate that
and
one of the big issues is fairness across
protected classes
protected classes being things like uh
sex and race and so on
and uh so two things you want is you
want to say well if i get a score of say
uh six out of ten then i want that to
mean the same
whether no matter what race i'm on yes
right so i want to have a
60 percent chance of reoccurring
uh regardless uh and the makers of the
one of the makers of a commercial
program to do that says that's what
we're trying to optimize and look
we achieved that we've we've reached
that kind of
balance and then on the other side you
also want to say
well if if it makes mistakes i want that
to affect
both sides of the protected class
equally
and it turns out they don't do that
right so they're
they're twice as likely to make a
mistake that would harm a black person
over a white person so that seems unfair
so you'd like to say well i want to
achieve both those goals
and then it turns out you do the
analysis and it's theoretically
impossible
to achieve both those goals so you have
to trade them off one against the other
so that analysis is really helpful to
know
what you can aim for and how much you
can get that you can't have everything
but the analysis certainly can't tell
you where should we make that
trade-off point but nevertheless then we
can
uh as humans deliberate where that
trade-off should be yeah so at least we
now we're
we're arguing an informed way we're not
asking for something impossible
we're saying uh here's where we are and
and here's what we aim for and
this strategy is better than that
strategy
so that's i would argue is a really
powerful and really important first step
but it's a doable one sort of removing
uh
undesirable degrees of bias in uh in
systems
in terms of protected classes and then
there's something i listen to your uh
commencement speech
or there's some fuzzier things like you
mentioned angry birds yeah
do you want do you want to create
systems that
feed the dopamine enjoyment that feed
that optimize for you returning to the
system enjoying
the moment of playing the game of
getting likes or whatever
this kind of thing or some kind of
long-term improvement
right is if are you even thinking about
that
that's ex that's really going to the
philosophical area
i think that's a really important issue
too certainly thinking about that i i
don't think about that
as a as an ai issue as much
but as you say you know the point is
we've
built this society in this
infrastructure
where we say we have a marketplace for
attention
and uh we've decided as a society that
we like things that are free
and so we want all uh apps on our phone
to be free
uh and that means they're all competing
for your attention and then eventually
they
they make some money some way through uh
ads or in-game sales or whatever
but they can only win by defeating all
the other apps by
instilling your attention and
we build a marketplace where it seems
like they're
working against you rather than working
with you
and i'd like to find a way where we can
change the playing field so we feel more
like well these things are on my side
yes they're letting me have some fun in
the short term but they're also helping
me
in the long term rather than competing
against me
and those aren't necessarily conflicting
objectives they're just
the incentives the direct current
incentives as we try to figure out this
whole new world
seem to be on uh the easier part of that
which is feeding the dopamine
the rush right but uh
let me take a quick step back
at the beginning of the artificial
intelligence and modern approach book of
writing
so here you are in the 90s when you
first
sat down with stuart to write the book
to cover
an entire field which is one of the only
books that successfully done that for ai
and
actually in a lot of other computer
science fields
you know it's a dif it's a it's a huge
undertaking so
it must have been quite daunting what
was that process like did you envision
that you would be
trying to cover the entire field
was there a systematic approach to it
that was more step by step
how did it feel so i guess it came about
you know
go to lunch with the other ai faculty at
berkeley
and we'd say uh you know the field is
changing
seems like the current books are a
little bit behind nobody's come out with
a new book recently we should do that
and everybody said yeah yeah that's a
great thing to do and we never did
anything
right and then i ended up heading off
to uh industry i went to uh sun labs so
i thought well that's the end of
my possible academic publishing career
but i met stuart again at a conference
like a year later
and said you know that book we were
always talking about you guys must be
half done with it by now right
he said well we keep talking we never do
anything so i said well you know we
should do it
and i think the reason is that
we all felt it was a time where the
field was changing
and that was in two ways
so you know the good old-fashioned ai
was based
uh primarily on boolean logic you had a
few
tricks to deal with uncertainty and it
was based
primarily on knowledge engineering then
the way you got something done is you
went out you interviewed an expert and
you wrote down by hand
everything they knew and we saw in
in 95 that the field was changing in
in two ways one we're moving more
towards probability
rather than boolean logic and we're
moving more towards machine learning
rather than knowledge engineering uh and
the other books uh
hadn't caught that way if they were
still in the uh
more in the in the old school although
so certainly they had part of that
on the way but we said if we start now
completely taking that point of view we
can have a different kind of book
and we were able to put that together
and uh what was literally the process if
you remember
did you start writing a chapter did you
outline
yeah i guess i guess we did an outline
and then we sort of assigned
chapters to each person at the time uh
i had moved to boston and stewart was in
berkeley
so basically uh we did it uh
uh over the internet and uh you know
that wasn't the same
as doing it today it meant you know
dial-up lines and telnetting in and
you know you you telnetted into
one shell and you type cat file name and
you hoped it was captured at the other
end and
certainly you're not sending uh images
and figures back and forth
right right that didn't work but you
know did you anticipate where the field
would go
from that day from from the 90s
did you see the
growth into learning-based methods into
data-driven methods
that followed in the future decades we
certainly thought
that learning was important i guess we
we missed it as uh being as important as
it
as it is today we missed this idea of
big data we missed that uh
uh the idea of deep learning hadn't been
invented yet
we could have uh taken the book from a
complete uh machine learning point of
view
right from the start we chose to do it
more from a point of view of we're going
to first develop
different types of representations and
we're going to talk
about different types of environments of
is it fully observable or partially
observable and is it
deterministic or stochastic and so on
and we
made it more complex along those axes
rather than
focusing on the machine learning axis
first
do you think you know there's some sense
in which the deep learning craze
is extremely successful for a particular
set of problems
and you know eventually it's going to in
the general case
hit challenges so in terms of the
difference between
perception systems and robots that have
to act in the world
do you think uh we're going to return to
ai modern approach type breadth
in addition five and six yeah
in uh in future decades do you think
deep learning will take its place as a
chapter and as
in his bigger uh view of ai yeah i think
we don't know yet how it's all going to
play out
so uh in the new edition uh we have a
chapter on deep learning
uh we got ian goodfellow to be the guest
author
for that chapter so he said he could
condense
his whole deep learning book into one
chapter i think he did a great job
we were also encouraged that he's you
know we gave him
the old neural net chapter and said
have fun with it modernize that and he
said you know half of that
was okay that certainly there's lots of
new things that have been developed
but some of the core was still the same
so i think we'll gain a better
understanding of what you can do there
i think we'll need to incorporate all
the things we can do with the other
technologies
right so deep learning started out
convolutional networks and very close to
perception
uh and has since moved to be uh
to be able to do more with actions and
some degree of
longer term planning but we need to do a
better job
with representation than reasoning and
one-shot learning and so on and
well i think we don't know yet how
that's going to play out
so do you think looking at the
some success but certainly uh
eventual demise the partial demise of
experts to symbolic
systems in the 80s do you think there is
kernels of wisdom
in the work that was done there with
logic and reasoning and so on
that will rise again in your view
so certainly i think the idea of
representation and reasoning
is crucial that you know sometimes you
just don't have enough data about the
world to learn
de novo so you've got to have
some idea of representation whether that
was programmed in or told or whatever
and then be able to take uh steps of
reasoning
i i think the problem uh with
the you know the good old-fashioned ai
was uh one we tried to base everything
on these uh
symbols that were atomic
and that's great if you're like trying
to define the properties of a triangle
right because they have necessary
insufficient conditions
uh but things in the real world don't
the real world is is messy and doesn't
have sharp edges
and atomic symbols do so that was a poor
match
and then the other aspect was that the
reasoning was universal and applied
anywhere which in some sense is good but
it also means there's no guidance
as to where to apply it and so you you
know you started getting these paradoxes
like
uh well if i have a mountain and i
remove
one grain of sand then it's still a
mountain and
but if i do that repeatedly at some
point it's not
right and with logic you know there's
nothing to stop
you from applying things uh repeatedly
but maybe with something like
deep learning and i don't really know
what the right name for it is
we could separate out those ideas so one
we could say
uh you know a mountain isn't just an
atomic
notion it it's some sort of something
like uh word embedding that
uh uh has a
a more complex representation yeah and
secondly we could somehow learn yeah
there's this rule that you can remove
one grain of sand
and you can do that a bunch of times but
you can't do it a near infinite amount
of times
but on the other hand when you're doing
induction on the integer
sure then it's fine to do it an infinite
number of times and
if we could uh somehow we have to learn
when these strategies are applicable
rather than having the strategies be
completely neutral
and available everywhere anytime you use
neural networks anytime you learn
from data or form representation from
data in an automated way
it's not very explainable as
to or it's not introspective to us
humans
in terms of uh how this neural network
sees the world
where why does it succeed so brilliantly
on so many in so many cases
and fail so miserably in surprising ways
and small
so what do you think is this is uh
the future there can simply more data
better data more organized data
solve that problem or is there
elements of symbolic systems that need
to be brought in which are a little bit
more explainable
yeah so i prefer to talk about
trust and uh validation and verification
rather than just about explainability
and then i think uh
explanations are one tool that you use
towards those goals
and i think it is important issue that
we don't want to use these systems
unless we
trust them and we want to understand
where they work and where they don't
work
and in an explanation can be part of
that
right so i apply for loan and i get
denied i want some explanation of why
and
you have in europe we have the gdpr that
says
you're required to be able to get that
but on the other hand
explanation alone is not enough right so
you know we're used to dealing with
people and with the
organizations and corporations and so on
and they can give you an explanation
then you have no guarantee that that
explanation relates to reality
right right so the bank can tell me well
you didn't get the loan
because you didn't have enough
collateral and that may be true or
it may be true that they just didn't
like my religion
or or something else i can't tell from
the explanation and that's
that's true whether the decision was
made by computer
or by a person so i want more
i do want to have the explanations and i
want to be able to uh have a
conversation to go back and forth
and said well you gave this explanation
but what about this
and what would have happened if this had
happened and uh
what would i need to change that so i
think a conversation is
is a better way to think about it than
just an explanation as a single output
and i think we need testing of various
kinds right so in order to know
was the decision really based on my
collateral or was it
based on my uh religion or skin color or
whatever
i can't tell if i'm only looking at my
case but if i look across all the cases
then i can detect a pattern all right
right so you want to have that kind of
capability
uh you want to have these adversarial
testing right so we thought we were
doing pretty good
at object recognition in images we said
look
we're at sort of pretty close to human
level performance on
imagenet and so on and then you start
seeing these
adversarial images and you say wait a
minute that part is nothing like
human performance okay you can mess with
it really easily you can mess with it
really easily
right and uh yeah you could do that to
humans too right so
in a different way perhaps right humans
don't know what color the dress was
right and so they're vulnerable to
certain attacks that are different
than the attacks on the on the machines
but the
you know the tax on the machines are so
striking uh they really change the way
you think about what we've done
right and the the way i think about it
is i think part of the problem is we're
seduced
by uh our low dimensional
metaphors right yeah so you know you
look like that phrase
you look in uh in a textbook and you say
okay now we've mapped out the space and
you know
uh cat is here and dog is here
and maybe there's a tiny little spot in
the middle where you can't tell the
difference but mostly we've got it all
covered
and if you believe that metaphor uh then
you say well we're nearly there
and uh you know there's only going to be
a couple adversarial images
but i think that's the wrong metaphor
and what you should really say is
it's not a 2d flat space that we've got
mostly covered
it's a million dimension space and a cat
is this string that goes out in this
crazy
bath and if you step a little bit off
the path in any direction you're in
nowhere's
land and you don't know what's going to
happen and so i think that's where we
are
and now we've got to deal with that so
uh it wasn't so much an explanation but
it was an understanding
of what the models are and what they're
doing and now we can start exploring how
do you fix that
yeah validating the robustness of the
system so on but take it back to the
this uh this word trust uh do you think
we're
a little too hard on our robots in terms
of uh the standards we apply
so you know
of uh there's a dance there's a
there's a there's a dance and nonverbal
and verbal communication between humans
you know if we apply the same kind of
standard in terms of humans
you know we trust each other pretty
quickly uh
you know you and i haven't met before
and there's some degree of trust
right that nothing's gonna go crazy
wrong and yet to ai
when we look at ai systems where we seem
to approach
uh through skepticism always always and
it's like they
have to prove through a lot of hard work
that they're even worthy of uh even
inkling of our trust do it
what do you what do you think about that
how how do we break that barrier close
that gap
i think that's right i think that's a
big issue uh just listening uh my friend
uh mark moffat is a naturalist
and he says uh the most amazing thing
about humans
is that you can walk into a coffee shop
or
a a busy street in a city
and there's lots of people around you
that you've never met before and you
don't kill each other
yeah he says chimpanzees cannot do that
yeah right
right if the chimpanzee's in a situation
where
here's some that aren't from my tribe
bad things happen especially in a coffee
shop there's delicious food around you
know
yeah yeah but but we humans have figured
that out yeah right
uh and you know for the most part for
the most part we still go to war
we still do terrible things uh but for
the most part we've learned to
trust each other and live together uh so
that's going to be important
for our uh our ai systems as well
and i th also i think uh you know a lot
of the emphasis
is on ai but in many cases
ai is part of the technology but isn't
really the main thing so a lot of
what we've seen is more due to
communications technology
than ai ai technology yeah you want to
make
these good decisions but the reason
we're able to have any kind of system at
all
is we've got the communication so that
we're collecting the data
and so that we can reach lots of people
around the world
i think that's a bigger change that
we're dealing with
speaking of reaching a lot of people
around the world on the side of
education
you've uh one of the many things in
terms of education you've done you
taught
the intro to artificial intelligence
course that signed up
100 160 000 students is one of the first
successful examples
and massive of a mooc massive open
online course
what did you learn from that experience
what do you think is the future
of moocs of education online yeah it was
a great fun
doing it particularly uh being
right at the start just because it was
exciting and new
but it also meant that we had less
competition
right so uh one of the things you hear
about uh well the problem with moocs is
uh the completion rates are are so low
so there must be a failure
and and i gotta admit i'm a prime
contributor right i've probably
started 50 different courses that i
haven't finished
but i got exactly what i wanted out of
them because i had never intended to
finish them i just wanted to
dabble in a little bit either to see the
topic matter or just to see the pedagogy
of how are they doing this class
so i guess the main thing i learned is
when i came in
i thought the challenge was
information saying if i'm just
take the stuff i want you to know and
i'm very clear
and explain it well then my job is done
and good things are going to happen and
then in doing the course
i learned well yeah you got to have the
information but really
the motivation is the most important
thing that
if students don't stick with it it
doesn't matter how good the content is
and i think being one of the first
classes
we were helped by uh sort of exterior
motivation
so we tried to do a good job of making
it enticing and setting
up ways for uh
you know the community to work with each
other to make it more motivating
but really a lot of it was hey this is a
new thing and i'm really excited to be
part of a new thing
and so the students brought their own
motivation
and so i think this is great because
there's lots of people around the world
who have never had this before
you know it would never have the
opportunity
to go to stanford and take a class or go
to mit or go to one of the other schools
but now we can bring that to them and if
they bring their own motivation
they can be successful in a way they
couldn't before
but that's really just the top tier of
people that are ready to do that the
rest of the people
just don't see or don't have their
motivation and don't see how if they
push through and were able to do it
what advantage that would get them so i
think we've got a long way to go
before we're able to do that and i think
it'll be some of it is
based on technology but more of it's
based on the idea of community that you
got to
actually get people together some of the
getting together can be done online i
think some of it really has to be done
in person to be able in order to build
that type of
community and trust you know there's an
intentional
mechanism that we've developed uh a
short attention span
especially younger people uh because
sort of
shorter and shorter videos online uh
there's a
whatever the the way the brain is
developing now
with people that have grown up with the
internet they have a quite a short
attention span so
and i would say i had the same when i
was growing up too
probably for different reasons so i
probably wouldn't have
learned as much as i have if i wasn't
forced to sit in a physical classroom
sort of bored sometimes falling asleep
but sort of forcing myself through that
process
to sometimes extremely difficult
computer science courses
what's the difference in your view
between in-person
education experience which you first of
all
yourself had and you yourself taught and
online education
and how do we close that gap if it's
even possible yeah
so i think there's two issues one is
whether it's in person or online so it's
sort of the physical
location and then the other is kind of
the
affiliation right so
you stuck with it in part because you
were in the classroom and you saw
everybody else was suffering right the
same way you were
but also because you were enrolled you
had paid tuition
sort of everybody was expecting you to
stick with it
society parents yeah peers
right and so those are two separate
things i mean you could certainly
imagine
i pay a huge amount of tuition and
everybody signed up and says yes you're
doing this
uh but then i'm in my room and my
classmates are in are in different rooms
right
we could have things set up that way so
it's not just the online versus offline
i think what's more important is the
commitment
that you've made and
certainly it is important to have that
kind of informal
you know i meet people outside of class
we talk together because we're all in it
together
i think that's uh really important both
in keeping your motivation and also
that's where
some of the most important learning goes
on so you want to have that
maybe you know especially now we start
getting into
higher bandwidths and augmented reality
and virtual reality you might be able to
get that without being in the same
physical place do you think it's
possible we'll see a course
at stanford for example that
for students enrolled students is only
online
in the near future who are literally
sort of that's part of the curriculum
and there is
no yeah so you're starting to see that i
know uh georgia tech
has a master's that's done that way
oftentimes
it's sort of they're creeping in in
terms of a master's program or sort of
um
further education considering the
constraints of students and so on
but i mean literally is it possible that
we just you know stanford mit berkeley
all these places go
online only in uh in the next few
decades
yeah probably not because you know
they've got a big commitment to
a physical campus sure right
there's a momentum that's both financial
and culturally right and
and then there are certain things that
just hard to do uh virtually right so
you know we're in a field uh where uh
if you have your own computer and your
own paper
and so on uh you can do the work
anywhere
uh but if you're in a biology lab or
something uh you know
you don't have all the right stuff at
home right
so our field programming you've also
done a lot of
you've done a lot of programming
yourself
in 2001 you wrote a great article about
programming called
teach yourself programming in 10 years
sort of response to
all the books that say teach yourself
programming in 21 days so if you're
giving advice to someone getting into
programming today
this is a few years since you've written
that article what's the best way to
undertake that journey
i think there's lots of different ways
and i think programming means more
things
now and i guess you know when i wrote
that article
i was thinking more about
becoming a professional software
engineer and i thought that's a
you know a sort of a career-long field
of study
but i think there's lots of things now
that people can do
where programming is a part of
solving what they want to solve without
achieving that professional level status
right so i'm not going to be going and
writing a million lines of code
but you know i'm a biologist or a
physicist or something
or even a historian and i've got some
data
and i want to ask a question of that
data and i think
for that you don't need 10 years right
so
there are many shortcuts to being able
to
answer those kinds of questions and and
you know you see today a lot of
emphasis on learning to code
teaching kids how to code uh i think
that's great
uh but i wish they would change the
message a little bit
right so i think code isn't the main
thing i don't really care if you know
the
syntax of javascript or if you can
connect these blocks together in this
visual language
but what i do care about is that you can
analyze a problem
you can think of a solution you can
carry out you know make a model
run that model test the model see the
results
verify that they're reasonable ask
questions and answer them
right so it's more modeling and problem
solving
and you use coding in order to do that
but it's not just learning coding for
its own sake that's really interesting
so
it's actually almost in many cases it's
learning to work with data
to extract something useful out of data
so when you say problem solving
you really mean taking some kind of
maybe collecting some kind of data set
cleaning it up and saying something
interesting about it which is useful in
all kinds of domains
and you know and i see myself
being stuck sometimes in kind of the the
old ways
right so you know be working on a
project
maybe with a younger employee and we say
oh well here's this new package that
could help solve this
problem and i'll go and i'll start
reading the manuals and you know i'll be
two hours into reading the manuals and
then
my colleague comes back and says i'm
done yeah you know i downloaded the
package i installed it i tried calling
some things
the first one didn't work the second one
work now i'm done
yeah and i say but i have 100 questions
about how does this work and how does
that work and they say
who cares right i don't need to
understand the whole thing i unders i
answered my question
it's a big complicated package i don't
understand the rest of it but
i got the right answer and i'm just it's
hard for me
to get into that mindset i want to
understand the whole thing and you know
if they wrote a manual
i should probably read it and but that's
not necessarily the right way i think i
have to get used to
dealing with more being more comfortable
with uncertainty and not knowing
everything yeah so i struggle with the
same instead of the
the spectrum between donald and don
knuth yeah
it was kind of the very you know before
he can say anything
about a problem he really has to get
down to the machine code assembly
yeah versus exactly what you said i've
have several students in my group that
uh you know
20 years old and they can solve almost
any problem within a few hours that
would take me probably weeks because i
would try to
as you said read the manual so
do you think the nature of mastery
you're you're mentioning biology
sort of outside disciplines applying
programming but computer scientists
so over time there's higher and higher
levels of abstraction available now
so with uh this week there's a the
tensorflow summit right so
if you're if you're not particularly
into deep learning but you're still a
computer scientist
uh you can accomplish an incredible
amount with uh tensorflow without really
knowing
any fundamental internals of machine
learning do you think the nature of
mastery
is is changing even for computer
scientists
like what it means to be an expert
programmer
yeah i think that's true you know we
never really should have focused on
programmer right because it's still it's
the skill
and what we really want to focus on is
the result so we
we built this uh ecosystem where the way
you can get stuff done
is by programming it yourself at least
when i started
that you know library functions meant
you had square root and that was about
it
right everything else you built from
scratch and then we built up an
ecosystem where
a lot of times well you can download a
lot of stuff that does
a big part of what you need and so now
it's more a question of
assembly rather than
[Music]
manufacturing and
that's a different way of looking at
problems from another perspective in
terms of mastery and
looking at programmers or people that
reason about problems in a computational
way
so google is you know the from the
hiring perspective from the perspective
of hiring or building a team of
programmers
how do you determine if someone's a good
programmer or
if somebody again yeah i want to deviate
from i want to move away from the word
programmer but
somebody who can solve problems of
large-scale data and so on
what's what's uh how do you build a team
like that through the interviewing
process
yeah and i and i think uh as a company
grows
uh you get more uh expansive
in the types of people you're looking
for right so i think you know
in the early days we'd interview people
and the question we were trying to ask
is uh how close are they to jeff dean
and most people were pretty far away but
we take the ones that were you know not
that far away
and so we got kind of a homogeneous
group of people who are really great
programmers
then as a company grows you say well we
don't want everybody
to be the same to have the same skill
set and so now
we're uh hiring uh biologists in our
health areas and we're hiring physicists
we're hiring
mechanical engineers we're hiring uh you
know
social scientists and ethnographers and
people with different backgrounds who
bring
different skills so you have mentioned
that you still may partake in code
reviews
given that you have a wealth of
experience as you've also mentioned
uh what errors do you often see
and tend to highlight in the code of
junior developers of people coming up
now
uh given your background from blisp
to a couple decades of programming yeah
that's a great question
you know sometimes i try to look at the
flexibility
of the design of yes
you know this api solves this problem
but
uh where is it going to go in the future
who else is going to want to call this
and uh you know are you making it easier
for them to do that it's a matter of
design is it documentation is it is it
uh sort of an amorphous thing you can't
really put it
it's just how it feels if you put
yourself in the shoes of a developer
would you use this kind of thing i think
it is how you feel
right and so yeah documentation is good
uh but it's but it's more a design
question right if you get the design
right then
people will figure it out whether the
documentation is good or not and
and if the design is wrong then it'll be
harder to use
how have uh you yourself changed as a
programmer
over the years as in in a way we
already started to say sort of you want
to read the manual you want to
understand the core of the
syntax to the how the language is
supposed to be used and so on
but what's the evolution been like from
the 80s 90s to today
i guess one thing is you don't have to
worry about
the small details of efficiency as much
as you used to
right so like i remember uh i did my
list book in the 90s and one of the
things i wanted to do
was say uh here's how you do an object
system
and uh basically we're going to make it
so each object is a hash table and you
look up the methods and here's how it
works and then i said
of course the real common lisp
object system is much more complicated
it's got all these efficiency
type issues and this is just a toy
nobody would do this in real life
and it turns out python pretty much did
exactly
what i said yeah and said uh objects are
just dictionaries and yeah they have a
few little uh tricks as well
but mostly you know the thing that
would have been 100 times too slow in
the 80s is now
plenty fast for most everything so you
had to as a programmer let go
of perhaps an obsession
that i remember coming up with of trying
to write
efficient code yeah that to say you know
what really matters
is the total time it takes to get the
project done
and most of that's going to be the
programmer time so if you're a little
bit less efficient but it makes it
easier to
understand and modify then that's the
right trade-off
so you've written quite a bit about lisp
your book on programming is in lisp
you you have a lot of code out there
that's in lisp
so myself and people who don't know what
lisp is should look it up
it's my favorite language for many ai
researchers it is a favorite language
the favorite language they never use
these days
so what part of the list do you find
most beautiful and powerful
so i think the beautiful part is the
simplicity that in
half a page you can define the whole
language
and other languages don't have that so
you feel like you can hold everything
in your head and
then you know a lot of people say well
then that's too simple you know here's
all these things i want to do
and you know my java or python or
whatever
has 100 or 200 or 300 different syntax
rules and don't i need all those
and lisp's answer was no we're only
going to give you
eight or so syntax rules but we're going
to allow you to define your own
and so that was a very powerful idea and
i think this idea of saying
i can start with my problem and with my
data
and then i can build the language i want
uh
for that problem and for that data and
then i can make lists define that
language
so you uh you're sort of uh mixing
levels and saying i'm simultaneously
a programmer in a language and a
language designer
and that allows a better match between
your problem
and your eventual code and i think lis
had done that better than other
languages yeah it's a very elegant
implementation of functional programming
but why do you think lisp has not had
the mass adoption and success of
languages like python
is it the parentheses is it all the
parentheses
yeah so i think a couple of things
so one was i think it was designed
for a single programmer or a small team
and a skilled programmer who
had the good taste to say well i'm i am
doing language design
and i have to make good choices and if
you make good choices
that's great if you make bad choices
you can hurt yourself and it can be hard
for other people on the team to
understand it
so i think there was a limit to the
scale
of the size of a project in terms of
number of people that lisp was good for
and as an industry we kind of grew
beyond that
i think it is in part the parentheses
you know one of the jokes is the acronym
for lisp is
lots of irritating silly parentheses
my acronym was lisp is syntactically
pure
saying all you need is parentheses and
atoms but i remember
you know so we had the the ai textbook
and
uh because we did it in the 90s we had
we had pseudocode in the book but then
we said well we'll have lisp online
because that's the language
of ai at the time and i remember some of
the students complaining
because they hadn't had lists before and
they didn't quite understand what was
going on and
i remember one student complained i
don't understand how this pseudocode
corresponds to this lisp and there was a
one-to-one correspondence
between the the symbols in the code in
the pseudocode and the only thing
difference was the parentheses
so i said it must be that for some
people a certain number of left
parentheses shuts off their brain
yeah it's very it's very possible in
that sense then python just goes the
other way
and so so that was the point at which i
said okay can't have only lisp that's a
language
because i you know i don't want to you
know you only got 10 or 12 or 15 weeks
or whatever it is to teach ai and i
don't want
to waste two weeks of that teaching lisp
so i say i got to have another language
java was the most popular language at
the time
i started doing that and then i said
it's really hard to
have a one-to-one correspondence between
the pseudocode and the java because
java's so verbose
so then i said i'm going to do a survey
and find the language that's most like
my pseudocode and turned out python
basically was
my pseudo code somehow i had channeled
uh guido and designed a pseudocode that
was the same as python although i hadn't
heard of python
uh at that point and from then on uh
that's what i've been using because it's
been a good match
so what's the story in python behind
pietudes your github repository with
puzzles and exercises and python is
pretty fun
yeah just it seems like fun uh you know
you know i like
uh doing puzzles and i like uh being an
educator
i did a class with udacity uh udacity
uh 212 i think it was it was basically
problem solving uh using python
and looking at different problems does
pie tubes feed that class
in terms of the exercises i was
wondering what that yeah so the class
the class came first
yeah some of the stuff that's in pi
tubes was write-ups of what was in the
class and then some of it was just
continuing to
uh to work on new problems
so what's the organizing madness of pi
tubes is it just the
collect a collection of cool exercises
just whatever i thought was fun okay
awesome
so you were the director of search
quality of google from
2001 to 2005. in the early days
uh when there's just a few employees and
when the company was growing
like crazy right so
i mean a google revolution has the way
we discover
share and aggregate knowledge
so just this is uh this is one of the
fundamental aspects of civilization
right is information
being shared and there's different
mechanisms throughout history but google
is just
10x improved that right and you're a
part of that
right people discovering that
information so what were some of the
challenges
on the philosophical or the technical
level in those early days
it definitely was an exciting time and
as you say we were
doubling in size every year
and the challenges were we wanted to get
the right answers
right and uh we had to figure out
what that meant we had to implement that
and we had to make it all
uh efficient and uh
we had to keep on testing and seeing if
we were delivering good answers
and now when you say good answers it
means whatever people are typing in in
terms of keywords in terms
that kind of thing that the the results
they get are ordered by the desirability
for them
of those results like they're like the
first thing they click on will likely be
the thing that they were
actually looking for right one of the
metrics we had was focused on the first
thing
uh some of it was focused on the whole
page so it was focused on
you know top three or so so we looked at
a lot of different metrics for
for how well we were doing and we broke
it down into
subclasses of you know maybe here's a
type of uh
of uh query that we're not doing well on
then we try to fix that
early on we started to realize that we
were in an adversarial
position right so we started thinking uh
well we're kind of like
the card catalog in the library right so
the books are here and we're off to the
side
and we're just reflecting what's there
and then we realized every time we make
a change the webmasters make a change
and it's uh game theoretic and so we had
to think not only of
is this the right move for us to make
now
but also if we make this move what's the
counter move going to be
is that going to get us into a work
worst place in which case we won't make
that move we'll make a different move
and did you find i mean i assume with
the popularity and the growth of the
internet that people were creating new
content
so you're almost helping guide the
creation yeah so that's certainly true
right so
we we definitely changed uh the
structure
of the network right so if you think
back you know in the in the very early
days
uh larry and sergey had the page rank
paper
and john kleinberg had this uh hubsan
authorities
model which says the web is made out of
these uh hubs which will be
my page of cool links about dogs or
whatever
and people would just list links uh and
then there'd be authorities which were
the ones uh the
page about dogs that most people link to
that doesn't happen anymore
people don't bother to say my page of
cool links
because we took over that function right
so so
we changed the way that worked did you
imagine back then that the internet
would be as massively vibrant as it is
today
i mean it was already growing quickly
but it's just another
i i don't know if you've ever if today
if you sit back and just look at the
internet
with wander the amount of content that's
just constantly being created constantly
being shared
unemployed yeah it's uh it's always been
surprising to me
i guess i'm not very good at predicting
the future in the future
okay and i remember you know being a
graduate student in
in 1980 or so and uh you know we had the
arpanet
and then there was this uh proposal to
uh commercialize it and have this
internet and this uh
uh crazy senator gore thought that might
be a good idea yeah and i remember
thinking oh come on you can't
you can't expect a commercial company to
understand
this technology they'll never be able to
do it yeah okay we can have this dot-com
domain but it won't go anywhere
so i was wrong al gore was right at the
same time
the nature of what it means to be a
commercial company has changed too
so google yeah isn't that it's founding
is different than uh
you know what companies were before i
think right so there's all these
uh business models that are so different
than what was possible back then
so in terms of predicting the future
what do you think it takes to build a
system
that approaches human level intelligence
you've talked about
of course that we you know we shouldn't
be so obsessed about creating human
level intelligence
just create systems that are very useful
for humans but what do you think it
takes
to uh to uh yeah approach that level
right
so certainly i don't think human level
intelligence is one thing
right so i think there's lots of
different tasks lots of different
capabilities
i also don't think uh that should be the
goal right so i you know
i wouldn't want to create a uh
calculator that could do multiplication
at human levels right
that would be a step backwards and so
for many things we should be aiming far
beyond human level
for other things maybe human level is a
good level to aim at
and for others we say well let's not
bother doing this because we already
have humans can take on those tasks
so as you say i like to focus on what
what's a useful tool
and and in some cases being on human
level
is an important part of crossing that
threshold to make the tool useful
so we see in things like these uh
personal assistants now
that you get either on your phone or on
a speaker that sits on the table
you want to be able to have a
conversation with those and
and i think as an industry we haven't
quite figured out what the right model
is for
what these things can do and we're
aiming towards well you just have a
conversation with them the way you can
with the person
right but we haven't delivered on that
model yet
right so you can ask it what's the
weather you can ask it
play some nice songs uh and
uh you know five or six other things and
then you run out of stuff that it can do
in terms of a deep meaningful connection
so you've mentioned the movie her as one
of your favorite ai movies
do you think it's possible for a human
being to fall in love with an ai system
ai assistant as you mentioned
so taking this big leap from uh what's
the weather to
you know having a deep connection yeah i
i think
uh as people that's what we love to do
and uh i was at a a showing of her
where we had a panel discussion and and
somebody asked me
uh what other movie do you think her is
similar to
and my answer was uh life of brian
which which is not a science fiction
movie uh
but both movies are about wanting to
believe in something that's not
necessarily real
yeah by the way for people don't know
it's monty python yeah yeah that's been
brilliantly put right so i mean i think
that's just the way we are we
we want to trust we want to believe we
want to fall in love
and uh it doesn't necessarily take that
much right so
you know my kids uh fell in love with
their teddy bear
right and the teddy bear was not very
interactive right so that's all us
yeah pushing our feelings onto
our devices and our things and i think
that that's what we like to do so we'll
continue to do that
so yeah as human beings will long for
that connection and just ai has to uh
do a little bit of work to uh to catch
us in the other end
yeah and certainly you know if you can
get to uh
dog level a lot of people have invested
a lot of uh
love in their pets and their pets some
some people
as i've been told in working with
autonomous vehicles have invested a lot
of love into their
inanimate cars yeah so it really doesn't
take much
so what is a good test
to linger on a topic that may be silly
or a little bit philosophical what is a
good test of intelligence in your view
is natural conversation like in the
touring test a good
a good test put another way what would
impress you yeah if you saw a computer
do it
these days yeah i mean i get impressed
all the time
right but like really impressive you
know go playing
uh starcraft playing uh those are all
pretty cool
you know and i think uh sure
conversation is important i think uh
you know we sometimes have these tests
where it's easy to fool
the system where you can have a chatbot
that can
have a conversation but you never uh it
never gets into a situation where it has
to be deep enough
that uh it really reveals itself
as being intelligent or not i think
uh you know turing suggested that
uh but i think if he were alive he'd say
you know i didn't really mean that
seriously
right yeah and i think uh and you know
this is just my opinion but but i think
turing's point was not that uh
this test of conversation is a good test
i think his point was
having a test is the right thing so
rather than
having the philosopher say oh no ai is
impossible
you should say well we'll just have a
test and then the result of that will
will tell us the answer and it doesn't
necessarily have to be a conversation
test
that's right and coming up a new better
test as the technology evolves is
probably the right way
do you worry as a lot of the general
public does
about not a lot but some vocal
uh part of the general public about the
existential threat of artificial
intelligence so
looking farther into the future as you
said most of us are not able to predict
much
so when shrouded in such mystery there's
a concern of
well you think you start thinking about
worst case is that something that
occupies your mind
space much so i certainly think about
uh threats i think about uh dangers uh
and i think uh any new technology uh
has positives and negatives and if it's
a powerful technology
it can be used for bad as well as for
good
so i'm certainly not worried about uh
the robot apocalypse
the terminator type scenarios i am
worried about
change in employment and
uh are we going to be able to react fast
enough to deal with that
i think we're you know we're already
seeing it today where a lot of people
are
are disgruntled about uh the way income
inequality is working
and uh and automation could help
accelerate those kinds of
problems i see powerful technologies can
always be used as weapons uh whether
they're
robots or drones or whatever uh some of
that
we're seeing due to ai a lot of it you
don't need ai
and i don't know what's uh what's a
worse threat if it's
an autonomous drone or it's uh
crispr technology becoming available or
we have lots of
threats to face and some of them involve
ai and some of them don't
so the threats that technology presents
are you for the most part optimistic
about technology also alleviating those
threats so creating new opportunities or
protecting us from the more detrimental
effects of these things yeah i don't
know it again it's hard to predict the
future
and uh yes as a success society
so far we've survived the nuclear
systems and other things of course uh
only societies that have survived are
having this conversation so uh maybe
that's a survivorship bias there yeah
what problem stands out to you as
exciting challenging
impactful to work on in the near future
for yourself
for the community in broadly so i
you know we talked about these uh
assistance in conversation i think
that's a great area
i think combining uh
common sense reasoning uh with
the power of data is a a great area
in which application in in conversation
relation or just broadly just
in general yeah as a programmer i'm
interested in
uh programming tools both in terms of
uh you know the current systems we have
today with with tensorflow and so on can
we make them
much easier to use for broader uh class
of people
and also can we apply uh machine
learning
to the more traditional type of
programming right so
you know when you go to google and you
uh
type in a query and you spell something
wrong it says did you mean
and the reason we're able to do that is
because lots of other people made a
similar error and then they corrected it
we should be able to go into our code
bases and our bug fix spaces
and when i type a line of code it should
be able to say
did you mean such and such if you type
this today you're probably going to
type in this bug fix tomorrow
yeah that's a really exciting
application of uh almost
uh an assistant for the coding
programming experience yeah
at every level so
i think i could safely speak for the
entire ai community
first of all for uh thank you for the
amazing work you've done
uh certainly for the amazing work you've
done with uh
ai a modern approach book yep i think
we're all looking forward very much for
the fourth edition and then the fifth
edition and so on
so uh peter thank you so much for
talking today yeah thank you pleasure
you